mapping Dagger.@spawn with remote workers can cause one hung thunk

[kleinschmidt]

MWE:

using Pkg
pkg"activate --temp"
pkg"add Dagger Tables"

using Dagger, Tables

using Distributed
addprocs(1)

@everywhere begin
    using Pkg
    Pkg.activate($(Pkg.project().path))
    Pkg.instantiate()
    using Dagger
end

t = (a=rand(10), b=rand(10))
f(row) = row.a + row.b

thunks = map(Tables.rows(t)) do row
    Dagger.@spawn f(row)
end

Results consistently in

10-element Vector{Dagger.EagerThunk}:
 EagerThunk (finished)
 EagerThunk (running)
 EagerThunk (finished)
 EagerThunk (finished)
 EagerThunk (finished)
 EagerThunk (finished)
 EagerThunk (finished)
 EagerThunk (finished)
 EagerThunk (finished)
 EagerThunk (finished)

That is, the second thunk is always permanently left in running until the worker is killed and the job is rescheduled. Also occurs with 2 worker procs instead of one, but not with zero.

[kolia]

Can repro the MWE.

However I'm a little mystified as to why any of the thunks other than thunks[2] works at all, since f is only defined on pid 1 and hasn't been defined @everywhere.

When I define @everywhere f(row) = ..., it works.

[jpsamaroo]

@kolia thanks for figuring that out, that means that this is likely a bug in error propagation.

[kleinschmidt]

Another clue: when you use map(delayed(f), Tables.rows(t)) and then compute.(thunks) they ALL hang indefinitely. We're starting to suspect that this is due to an error not being handled properly and thus the job neither failing or returning.

[kolia]

see #255

[jpsamaroo]

Thanks for the reports @kleinschmidt and @kolia! This one has probably bitten many Dagger users (including me) in the butt without anyone realizing what was happening!

[kleinschmidt]

Thanks for the quick fix!