Skip to content

Utilize worker threads #282

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Siegrift opened this issue Apr 30, 2024 · 3 comments · Fixed by #288 or #291
Closed

Utilize worker threads #282

Siegrift opened this issue Apr 30, 2024 · 3 comments · Fixed by #288 or #291
Assignees

Comments

@Siegrift
Copy link
Collaborator

@bdrhn9 had a good idea that by spawning multiple workers we might leverage the additional CPU cores. This would also reduce the overhead of the "main" loop that currently does everything.

From the quick look there don't seem to be any downsides to implementing this.

@Siegrift
Copy link
Collaborator Author

Siegrift commented Apr 30, 2024

A big downside to this is that workers run in isolation. So for example, given we have 2 cores - we can't just operate an Airseeker on 10 chains and 5 per CPU. We would also need to have separate state and fetch Signed data for every worker. It's basically similar to having just a single Airseeker deployed.

What this could be used for is Signed data validation, because that is CPU heavy. And unfortunately, I don't see any other CPU intensive operations we could offload to a separate thread.

Related discussion: https://api3workspace.slack.com/archives/C05TQPT7PNJ/p1714514780191139

@Siegrift Siegrift changed the title Spawn multiple workers for chain processing Utilize worker threads Apr 30, 2024
@Siegrift
Copy link
Collaborator Author

Siegrift commented May 2, 2024

This was resolved by accident (by mentioning this issue in the #288 PR).

@Siegrift
Copy link
Collaborator Author

Siegrift commented May 7, 2024

Agreed on call to whitelist signed data URLs from the config as suggestion in the original Slack thread.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
1 participant