Optimize remotecall pattern used in distributed loading

[amitmurthy]

Came across this possible optimization while I was looking at the messages sent between workers in #21718

Currently the fetch/wait calls are 1) executed serially and 2) are an extra network call as compared with using a @sync for p in pids; @async remotecall_fetch.... pattern.

Have replaced the @spawnat followed by fetch/wait calls with an async remotecall_fetch

[alyst]

sizehint!(Vector{Tuple{Int,Any}}(), nprocs())?

Looks like others is not required, should be ok to just skip myid() before @async block. One allocation less :)

[vtjnash]

I'm not sure I follow. Why is @sync for p in pids; @async remotecall_fetch.... more efficient than fetch.([@async remotecall.... for p in pids])? Is the idea that the former runs the task management locally, whereas the latter runs it distributed?

[amitmurthy]

for pids (2,3,4,5) :
@sync for p in pids; @async remotecall_fetch.... results in

Message remotecall_fetch from 1 to 2
Message remotecall_fetch from 1 to 3
Message remotecall_fetch from 1 to 4
Message remotecall_fetch from 1 to 5

# responses can come back in any order

Message remotecall_fetch_response from 3 to 1
Message remotecall_fetch_response from 2 to 1
Message remotecall_fetch_response from 4 to 1
Message remotecall_fetch_response from 5 to 1

fetch.([@async remotecall.... for p in pids]) results in

Message remotecall from 1 to 2
Message remotecall from 1 to 3
Message remotecall from 1 to 4
Message remotecall from 1 to 5

# The above calls return a future that are then "fetched" serially

Message fetch from 1 to 2
Message fetch_response from 2 to 1
Message fetch from 1 to 3
Message fetch_response from 3 to 1
Message fetch from 1 to 4
Message fetch_response from 4 to 1
Message fetch from 1 to 5
Message fetch_response from 5 to 1

In the first case, the results of the remotecall_fetch are pushed asynchronously from the worker nodes. In the second case they are fetched serially.

[amitmurthy]

Will merge after a green CI

[tkelman]

why the rush? it's been open less than a day, give people a chance to see it

[amitmurthy]

Sure, no issues.

[JeffBezanson]

Should use remotecall_wait, since the old version called wait.

[amitmurthy]

It is better to use a remotecall_fetch (after ensuring that the remote execution returns nothing) when we we do not need the result of the remote computation. Will avoid unnecessary storage and and clean-up of the result on the remote node.

[amitmurthy]

Will merge in a short while.