Skip to content

Add flexible cluster and worker management and lots of other stuff #13

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 65 commits into from
May 10, 2024

Conversation

oschulz
Copy link
Owner

@oschulz oschulz commented May 10, 2024

No description provided.

Copy link

codecov bot commented May 10, 2024

Codecov Report

Attention: Patch coverage is 71.53025% with 240 lines in your changes are missing coverage. Please review.

Project coverage is 70.00%. Comparing base (eb1cd88) to head (b8ad012).

Files Patch % Lines
src/workerpool.jl 70.80% 47 Missing ⚠️
src/custom_cluster_managers.jl 55.68% 39 Missing ⚠️
src/runworkers.jl 63.44% 34 Missing ⚠️
src/display.jl 0.00% 28 Missing ⚠️
src/onworkers.jl 71.42% 28 Missing ⚠️
src/procinit.jl 82.17% 23 Missing ⚠️
src/slurm.jl 0.00% 17 Missing ⚠️
ext/ParallelProcessingToolsThreadPinningExt.jl 0.00% 12 Missing ⚠️
src/exceptions.jl 69.23% 4 Missing ⚠️
src/waiting.jl 96.33% 4 Missing ⚠️
... and 2 more
Additional details and impacted files
@@             Coverage Diff             @@
##             main      #13       +/-   ##
===========================================
+ Coverage   48.31%   70.00%   +21.69%     
===========================================
  Files          11       19        +8     
  Lines         563     1227      +664     
===========================================
+ Hits          272      859      +587     
- Misses        291      368       +77     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@oschulz oschulz merged commit b949759 into main May 10, 2024
12 checks passed
@oschulz oschulz deleted the clusters-and-workers branch May 10, 2024 19:18
@oschulz
Copy link
Owner Author

oschulz commented May 10, 2024

CC @JBlaschke this should be the main implementation for now - will probably do another PR changing some things before I make a release.

@oschulz
Copy link
Owner Author

oschulz commented May 10, 2024

I also want to see if I can add support for Dagger Contexts (https://juliaparallel.org/Dagger.jl/dev/processors/#Dynamic-worker-pools) , to automatically add/remove workers there via the mechanisms in ParallelProcessingTools.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant