Skip to content

Distribute the scheduler #165

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jpsamaroo opened this issue Nov 17, 2020 · 0 comments
Open

Distribute the scheduler #165

jpsamaroo opened this issue Nov 17, 2020 · 0 comments

Comments

@jpsamaroo
Copy link
Member

jpsamaroo commented Nov 17, 2020

We're already pushing some extra work onto the worker nodes (mainly argument fetching and processor selection/load balancing), and it would be beneficial for large DAGs to move more work onto each worker. The main blocker is providing a way to split the DAG into multiple domains, where each domain is handled by a given thread on a given worker. With efficient Thunk serialization, we can then send a subgraph to each worker and let them process their own DAG without conflicts. We'll need to add a mechanism by which thunks automatically wait on their input thunks to complete before they attempt to download the output data; if possible, we can also have workers broadcast and shard their Chunks onto dependent workers as soon as the data is made available.

@jpsamaroo jpsamaroo changed the title Multithread and distribute the scheduler Distribute the scheduler Dec 5, 2021
@jpsamaroo jpsamaroo mentioned this issue Dec 5, 2021
6 tasks
@jpsamaroo jpsamaroo linked a pull request Dec 5, 2021 that will close this issue
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant