Skip to content

another locality optimization

Compare
Choose a tag to compare
@shashi shashi released this 07 May 09:52
· 1003 commits to master since this release

Scheduler now doesn't randomly pick a task when a worker doesn't have any task with some locally available data. It will instead stop using that worker and wait for some worker with the right affinity to come and pick up the task. This gives massive speedups in some JuliaDB datasets that are slow to serialize/deserialize.