Skip to content

Eager scheduler error when @spawn inside a thunk #267

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
krynju opened this issue Aug 21, 2021 · 7 comments · Fixed by #281
Closed

Eager scheduler error when @spawn inside a thunk #267

krynju opened this issue Aug 21, 2021 · 7 comments · Fixed by #281
Labels
bug eager api scheduler

Comments

@krynju
Copy link
Member

krynju commented Aug 21, 2021 •
edited
Loading

Noticed this when working on the reduce function for the DTable
I've added the per column reduction within dtable partition parallelization, which requires spawning from inside of a thunk
Either when running a reduce function repeatedly or when using @btime I would sometimes gets this error

Let me know if this is any interesting, ill try to get a MWE if needed
this is happening on master

Error in eager scheduler:
Future can be set only once
Stacktrace:
  [1] error(s::String)
    @ Base .\error.jl:33
  [2] put!
    @ C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.8\Distributed\src\remotecall.jl:583 [inlined]
  [3] #put!#75
    @ c:\Users\krynjupc\.julia\dev\Dagger\src\thunk.jl:139 [inlined]
  [4] fill_registered_futures!(state::Dagger.Sch.ComputeState, node::Thunk, failed::Bool)
    @ Dagger.Sch c:\Users\krynjupc\.julia\dev\Dagger\src\sch\util.jl:17
  [5] finish_task!(ctx::Context, state::Dagger.Sch.ComputeState, node::Thunk, thunk_failed::Bool)
    @ Dagger.Sch c:\Users\krynjupc\.julia\dev\Dagger\src\sch\Sch.jl:756
  [6] (::Dagger.Sch.var"#80#88"{Context, Dagger.Sch.ComputeState, Thunk})()
    @ Dagger.Sch c:\Users\krynjupc\.julia\dev\Dagger\src\sch\Sch.jl:448
  [7] lock(f::Dagger.Sch.var"#80#88"{Context, Dagger.Sch.ComputeState, Thunk}, l::ReentrantLock)
    @ Base .\lock.jl:187
  [8] compute_dag(ctx::Context, d::Thunk; options::Dagger.Sch.SchedulerOptions)
    @ Dagger.Sch c:\Users\krynjupc\.julia\dev\Dagger\src\sch\Sch.jl:447
  [9] compute(ctx::Context, d::Thunk; options::Dagger.Sch.SchedulerOptions)
    @ Dagger c:\Users\krynjupc\.julia\dev\Dagger\src\compute.jl:31
 [10] (::Dagger.Sch.var"#45#46"{Context})()
    @ Dagger.Sch .\task.jl:411
Error in eager listener:
KeyError: key 2235 not found
Stacktrace:
  [1] getindex
    @ .\dict.jl:481 [inlined]
  [2] #41
    @ c:\Users\krynjupc\.julia\dev\Dagger\src\sch\dynamic.jl:188 [inlined]
  [3] map
    @ .\tuple.jl:239 [inlined]
  [4] _add_thunk!(ctx::Context, state::Dagger.Sch.ComputeState, task::Task, tid::Int64, ::Tuple{Dagger.var"#411#420", Tuple{Symbol, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, 
Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID}, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, Dagger.ThunkFuture, MemPool.DRef})
    @ Dagger.Sch c:\Users\krynjupc\.julia\dev\Dagger\src\sch\dynamic.jl:188
  [5] #invokelatest#2
    @ .\essentials.jl:718 [inlined]
  [6] invokelatest
    @ .\essentials.jl:716 [inlined]
  [7] #23
    @ c:\Users\krynjupc\.julia\dev\Dagger\src\sch\dynamic.jl:69 [inlined]
  [8] lock(f::Dagger.Sch.var"#23#27"{Tuple{Dagger.var"#411#420", Tuple{Symbol, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, 
Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID, Dagger.Sch.ThunkID}, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, Dagger.ThunkFuture, MemPool.DRef}, typeof(Dagger.Sch._add_thunk!), Context, Dagger.Sch.ComputeState, Task}, l::ReentrantLock)
    @ Base .\lock.jl:187
  [9] macro expansion
    @ c:\Users\krynjupc\.julia\dev\Dagger\src\sch\dynamic.jl:68 [inlined]
 [10] (::Dagger.Sch.var"#22#26"{Context, Dagger.Sch.ComputeState, Task, Distributed.RemoteChannel{Channel{Any}}, Distributed.RemoteChannel{Channel{Any}}})()
    @ Dagger.Sch .\task.jl:411
Stacktrace:
 [1] exec!(::Function, ::Dagger.Sch.SchedulerHandle, ::Function, ::Vararg{Any})
   @ Dagger.Sch c:\Users\krynjupc\.julia\dev\Dagger\src\sch\dynamic.jl:112
 [2] add_thunk!(::Function, ::Dagger.Sch.SchedulerHandle, ::Symbol, ::Vararg{Any}; future::Dagger.ThunkFuture, ref::MemPool.DRef, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ Dagger.Sch c:\Users\krynjupc\.julia\dev\Dagger\src\sch\dynamic.jl:185
 [3] eager_thunk()
   @ Dagger.Sch c:\Users\krynjupc\.julia\dev\Dagger\src\sch\eager.jl:81
 [4] macro expansion
   @ c:\Users\krynjupc\.julia\dev\Dagger\src\processor.jl:154 [inlined]
 [5] (::Dagger.var"#53#54"{typeof(Dagger.Sch.eager_thunk), Tuple{}, NamedTuple{(:sch_uid, :sch_handle, :processor, :utilization), Tuple{UInt64, Dagger.Sch.SchedulerHandle, Dagger.ThreadProc, UInt64}}})()
   @ Dagger .\threadingconstructs.jl:178
@krynju krynju mentioned this issue Aug 22, 2021
Merged
@jpsamaroo
Copy link
Member

An MWE would be appreciated! It's probably a simple oversight in the Future registration code.

@krynju
Copy link
Member Author

krynju commented Aug 24, 2021 •
edited
Loading

This worked, just run the loop a few times or increase the number of spawns in a.
env with threads

c = Dagger.@spawn 10+10
b = (x) -> x + 10
a = (x) -> x .+ fetch.([Dagger.spawn(b, x + i) for i in 1:100])

for i in 1:100
r = Dagger.@spawn a(c)
fetch(r)
end

result (different stacktrace, but this one and the one above happened in the original problem scenario)

Error in eager listener:
Future can be set only once
Stacktrace:
 [1] error(s::String)
   @ Base .\error.jl:33
 [2] put_future(rid::Distributed.RRID, v::MemPool.DRef, caller::Int64)
   @ Distributed C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.8\Distributed\src\remotecall.jl:590
 [3] call_on_owner(::Function, ::Distributed.Future, ::MemPool.DRef, ::Vararg{Any})
   @ Distributed C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.8\Distributed\src\remotecall.jl:514
 [4] put!(rr::Distributed.Future, v::MemPool.DRef)
   @ Distributed C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.8\Distributed\src\remotecall.jl:584
 [5] eager_thunk()
   @ Dagger.Sch c:\Users\krynjupc\.julia\dev\Dagger\src\sch\eager.jl:84
 [6] macro expansion
   @ c:\Users\krynjupc\.julia\dev\Dagger\src\processor.jl:154 [inlined]
 [7] (::Dagger.var"#53#54"{typeof(Dagger.Sch.eager_thunk), Tuple{}, NamedTuple{(:sch_uid, :sch_handle, :processor, :utilization), Tuple{UInt64, Dagger.Sch.SchedulerHandle, Dagger.ThreadProc, UInt64}}})()
   @ Dagger .\threadingconstructs.jl:178

@krynju
Copy link
Member Author

krynju commented Aug 30, 2021

Stumbled upon this issue again when working on groupby for the DTable (also nested spawn)
But only under load as the examples above

@jpsamaroo
Copy link
Member

Interestingly, Dagger just hangs for me when I get into the loop.

@krynju
Copy link
Member Author

krynju commented Aug 30, 2021 •
edited
Loading

Normally it also hangs for me once it prints the stacktrace (sometimes it doesn't even print the full one as in here #267 (comment)). After I ctrl-c out of it Dagger is still not responsive if you try doing anything else.

Are you on recent julia master?
If not then it's probably another issue that only appears after this was fixed #243

I can replicate it on linux as well, so not a windows thing for sure.

Another stacktrace using the mwe (this one looks like the original one from the initial comment):

Error in eager scheduler:
Future can be set only once
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:33
  [2] put_future(rid::Distributed.RRID, v::Tuple{Bool, Dagger.Chunk{Int64, MemPool.DRef, Dagger.ThreadProc, AnyScope}}, caller::Int64)
    @ Distributed /opt/julia/usr/share/julia/stdlib/v1.8/Distributed/src/remotecall.jl:590
  [3] call_on_owner(::Function, ::Distributed.Future, ::Tuple{Bool, Dagger.Chunk{Int64, MemPool.DRef, Dagger.ThreadProc, AnyScope}}, ::Vararg{Any})
    @ Distributed /opt/julia/usr/share/julia/stdlib/v1.8/Distributed/src/remotecall.jl:514
  [4] put!
    @ /opt/julia/usr/share/julia/stdlib/v1.8/Distributed/src/remotecall.jl:584 [inlined]
  [5] #put!#75
    @ ~/.julia/packages/Dagger/akmDx/src/thunk.jl:139 [inlined]
  [6] fill_registered_futures!(state::Dagger.Sch.ComputeState, node::Thunk, failed::Bool)
    @ Dagger.Sch ~/.julia/packages/Dagger/akmDx/src/sch/util.jl:17
  [7] finish_task!(ctx::Context, state::Dagger.Sch.ComputeState, node::Thunk, thunk_failed::Bool)
    @ Dagger.Sch ~/.julia/packages/Dagger/akmDx/src/sch/Sch.jl:756
  [8] (::Dagger.Sch.var"#80#88"{Context, Dagger.Sch.ComputeState, Thunk})()
    @ Dagger.Sch ~/.julia/packages/Dagger/akmDx/src/sch/Sch.jl:448
  [9] lock(f::Dagger.Sch.var"#80#88"{Context, Dagger.Sch.ComputeState, Thunk}, l::ReentrantLock)
    @ Base ./lock.jl:187
 [10] compute_dag(ctx::Context, d::Thunk; options::Dagger.Sch.SchedulerOptions)
    @ Dagger.Sch ~/.julia/packages/Dagger/akmDx/src/sch/Sch.jl:447
 [11] compute(ctx::Context, d::Thunk; options::Dagger.Sch.SchedulerOptions)
    @ Dagger ~/.julia/packages/Dagger/akmDx/src/compute.jl:31
 [12] (::Dagger.Sch.var"#45#46"{Context})()
    @ Dagger.Sch ./task.jl:411

@krynju
Copy link
Member Author

krynju commented Sep 2, 2021

New stacktrace, this one has some more info or it's a different case

Error in eager listener:
Future can be set only once
Stacktrace:
 [1] error(s::String)
   @ Base .\error.jl:33
 [2] put!(rr::Distributed.Future, v::MemPool.DRef)
   @ Distributed C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.8\Distributed\src\remotecall.jl:583
 [3] eager_thunk()
   @ Dagger.Sch c:\Users\krynjupc\.julia\dev\Dagger\src\sch\eager.jl:84
 [4] macro expansion
   @ c:\Users\krynjupc\.julia\dev\Dagger\src\processor.jl:154 [inlined]
 [5] (::Dagger.var"#53#54"{typeof(Dagger.Sch.eager_thunk), Tuple{}, NamedTuple{(:sch_uid, :sch_handle, :processor, :utilization), Tuple{UInt64, Dagger.Sch.SchedulerHandle, Dagger.ThreadProc, UInt64}}})()
   @ Dagger .\threadingconstructs.jl:178
ERROR: LoadError: ThunkFailedException (Thunk[5090](#446, (true, 27, ~Thunk[4800](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2184, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[4801](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2185, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[4803](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2186, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[4806](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2187, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[4810](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2188, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[4815](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2189, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[4820](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2190, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[4825](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2191, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[4830](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2192, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[4835](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2193, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[4840](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2194, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[4845](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2195, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[4850](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, 
OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2196, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[4855](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2197, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[4860](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2198, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[4865](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2199, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[4870](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2200, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[4875](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2201, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[4880](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2202, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[4885](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2203, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[4890](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2204, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[4895](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2205, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[4900](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2206, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[4905](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2207, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[4910](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2208, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[4915](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2209, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[4920](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2210, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[4925](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2211, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[4930](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2212, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[4935](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2213, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[4940](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2214, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[4945](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2215, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[4950](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2216, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[4955](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2217, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[4960](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2218, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[4965](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2219, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[4970](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2220, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[4975](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2221, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[4980](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2222, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[4985](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2223, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[4990](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2224, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[4995](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2225, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[5000](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2226, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[5005](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2227, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[5010](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2228, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[5015](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2229, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[5020](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2230, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[5025](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2231, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[5030](#441, (Dagger.Chunk{NamedTuple{(:a,), 
Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2232, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[5035](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2233, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[5040](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2234, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[5045](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2235, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[5050](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2236, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[5055](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2237, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[5060](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2238, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[5065](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2239, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[5070](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2240, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[5075](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2241, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[5080](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2242, 
0x00000000000000d0), OSProc(1), AnyScope(), false), :a)), ~Thunk[5085](#441, (Dagger.Chunk{NamedTuple{(:a,), Tuple{Vector{Char}}}, MemPool.DRef, OSProc, AnyScope}(NamedTuple{(:a,), Tuple{Vector{Char}}}, UnitDomain(), MemPool.DRef(1, 2243, 0x00000000000000d0), OSProc(1), AnyScope(), false), :a)))) failure):
MethodError: Cannot `convert` an object of type Nothing to an object of type MemPool.DRef
Closest candidates are:
  convert(::Type{T}, ::T) where T at essentials.jl:218
  MemPool.DRef(::Any, ::Any, ::Any) at C:\Users\krynjupc\.julia\packages\MemPool\N0esb\src\datastore.jl:8
Stacktrace:
  [1] Dagger.EagerThunk(uid::UInt64, future::Dagger.ThunkFuture, finalizer_ref::MemPool.DRef, thunk_ref::Nothing)
    @ Dagger c:\Users\krynjupc\.julia\dev\Dagger\src\thunk.jl:186
  [2] spawn(f::Function, args::Dagger.EagerThunk; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Dagger c:\Users\krynjupc\.julia\dev\Dagger\src\thunk.jl:230
  [3] spawn
    @ c:\Users\krynjupc\.julia\dev\Dagger\src\thunk.jl:225 [inlined]
  [4] _broadcast_getindex_evalf
    @ .\broadcast.jl:670 [inlined]
  [5] _broadcast_getindex
    @ .\broadcast.jl:643 [inlined]
  [6] _getindex
    @ .\broadcast.jl:667 [inlined]
  [7] _broadcast_getindex
    @ .\broadcast.jl:642 [inlined]
  [8] getindex
    @ .\broadcast.jl:597 [inlined]
  [9] copyto_nonleaf!(dest::Vector{Int64}, bc::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1}, Tuple{Base.OneTo{Int64}}, typeof(fetch), Tuple{Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1}, Nothing, typeof(Dagger.spawn), Tuple{Base.RefValue{typeof(Dagger.rowcount)}, Base.Broadcast.Extruded{Vector{Dagger.EagerThunk}, Tuple{Bool}, Tuple{Int64}}}}}}, iter::Base.OneTo{Int64}, state::Int64, count::Int64)
    @ Base.Broadcast .\broadcast.jl:1099
 [10] copy
    @ .\broadcast.jl:951 [inlined]
 [11] materialize(bc::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1}, Nothing, typeof(fetch), Tuple{Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1}, Nothing, typeof(Dagger.spawn), Tuple{Base.RefValue{typeof(Dagger.rowcount)}, Vector{Dagger.EagerThunk}}}}})
    @ Base.Broadcast .\broadcast.jl:904
 [12] (::Dagger.var"#446#461"{DTable})(::Bool, ::Int64, ::Vector{Pair{Char, Dagger.EagerThunk}}, ::Vararg{Vector{Pair{Char, Dagger.EagerThunk}}})
    @ Dagger c:\Users\krynjupc\.julia\dev\Dagger\src\table\groupby.jl:46
 [13] macro expansion
    @ c:\Users\krynjupc\.julia\dev\Dagger\src\processor.jl:154 [inlined]
 [14] (::Dagger.var"#53#54"{Dagger.var"#446#461"{DTable}, Tuple{Bool, Int64, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}, Vector{Pair{Char, Dagger.EagerThunk}}}, NamedTuple{(:sch_uid, :sch_handle, :processor, :utilization), Tuple{UInt64, Dagger.Sch.SchedulerHandle, Dagger.ThreadProc, UInt64}}})()
    @ Dagger .\threadingconstructs.jl:178
Stacktrace:
 [1] wait
   @ .\task.jl:322 [inlined]
 [2] fetch
   @ .\task.jl:337 [inlined]
 [3] execute!(::Dagger.ThreadProc, ::Function, ::Bool, ::Vararg{Any})
   @ Dagger c:\Users\krynjupc\.julia\dev\Dagger\src\processor.jl:157
 [4] do_task(to_proc::Dagger.ThreadProc, extra_util::UInt64, thunk_id::Int64, f::Function, data::Tuple{Bool, Int64, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, 
MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, 
MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, 
MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, 
MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, 
MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, 
MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, 
MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, 
MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, 
MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}}, send_result::Bool, persist::Bool, cache::Bool, meta::Bool, options::Dagger.Sch.ThunkOptions, ids::Vector{Int64}, ctx_vars::NamedTuple{(:log_sink, :profile), Tuple{Dagger.NoOpLog, Bool}}, sch_handle::Dagger.Sch.SchedulerHandle, uid::UInt64)
   @ Dagger.Sch c:\Users\krynjupc\.julia\dev\Dagger\src\sch\Sch.jl:962
 [5] macro expansion
   @ c:\Users\krynjupc\.julia\dev\Dagger\src\sch\Sch.jl:875 [inlined]
 [6] (::Dagger.Sch.var"#112#114"{Dagger.ThreadProc, Distributed.RemoteChannel{Channel{Any}}, Tuple{UInt64, Int64, Dagger.var"#446#461"{DTable}, Tuple{Bool, Int64, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}, Dagger.Chunk{Vector{Pair{Char, Dagger.EagerThunk}}, MemPool.DRef, Dagger.ThreadProc, AnyScope}}, Bool, Bool, Bool, Bool, Dagger.Sch.ThunkOptions, Vector{Int64}, NamedTuple{(:log_sink, :profile), Tuple{Dagger.NoOpLog, Bool}}, Dagger.Sch.SchedulerHandle, UInt64}})()
   @ Dagger.Sch .\task.jl:411
Stacktrace:
 [1] fetch(t::Dagger.ThunkFuture; proc::OSProc)
   @ Dagger c:\Users\krynjupc\.julia\dev\Dagger\src\thunk.jl:135
 [2] fetch
   @ c:\Users\krynjupc\.julia\dev\Dagger\src\thunk.jl:131 [inlined]
 [3] fetch(t::Dagger.EagerThunk)
   @ Dagger c:\Users\krynjupc\.julia\dev\Dagger\src\thunk.jl:193
 [4] groupby(d::DTable, col::Symbol; merge::Bool, chunksize::Int64)
   @ Dagger c:\Users\krynjupc\.julia\dev\Dagger\src\table\groupby.jl:84
 [5] top-level scope
   @ .\timing.jl:210 [inlined]
 [6] top-level scope
   @ c:\Users\krynjupc\.julia\dev\Dagger\proto.jl:0
in expression starting at c:\Users\krynjupc\.julia\dev\Dagger\proto.jl:9

This was referenced Sep 8, 2021
Merged
Merged
@krynju
Copy link
Member Author

krynju commented Sep 12, 2021

Future can be set only once fixed here: JuliaLang/julia#42167
Keyerror and improvement with hangs fixed here: #281

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug eager api scheduler
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix keyerror for eager api krynju/Dagger.jl
2 participants
@jpsamaroo @krynju