undefined function => stalled thunks

[kolia]

[mostly the same as #254, better diagnosis]

using Pkg
pkg"activate --temp"
pkg"add Dagger Tables"

using Dagger, Tables

using Distributed
addprocs(1)

@everywhere begin
    using Pkg
    Pkg.activate($(Pkg.project().path))
    Pkg.instantiate()
    using Dagger
end

f(x) = x

eager_thunks = map(1:4) do i
    Dagger.@spawn f(i)
end
#4-element Vector{Dagger.EagerThunk}:       
# EagerThunk (finished)                                                                                                                                     
# EagerThunk (running)
# EagerThunk (finished)                                                                                                                                     
# EagerThunk (finished)

d = delayed(f)(42)

compute(d)  # stalls

Seems likely that thunks[1] gets run on process pid 1, then thunks[2] gets run on the worker, and somehow the error you get from f not being defined there makes thunk[2] stall.

Seems likely that delayed(f)(42) gets scheduled on the worker, and exhibits the same behavior.

Interestingly, if instead of f(x) = x, I define f = identity, then delayed(f)(42) works, but thunk[2] stalls in the same way.

[jpsamaroo]

Ok, found the issue: deserialize on worker 2 throws the error on f being undefined (before we get into Dagger's "managed" code), but we use remote_do in fire_tasks, so we silently swallow the error. This only occurs for the second eager task, and for the delayed call, because those executed on worker 2; the rest of the tasks execute on worker 1, and thus complete successfully. Will have a fix up shortly.