Trouble when using a sysimage with PythonCall

[ericphanson]

I'm having trouble when building a sysimage with PythonCall-- or rather, I can built it OK, but then when I use it, like

julia --project=myproject --sysimage=mysysimage.so -e "using Pkg; Pkg.instantiate()"

I get

#29 92.79 fatal: error thrown and no exception handler available.
#29 92.79 InitError(mod=:C, error=ErrorException("no environment in the LOAD_PATH depends on CondaPkg"))
#29 92.84 error at ./error.jl:33
#29 92.84 _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2247 [inlined]
#29 92.84 jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2429
#29 92.84 _resolve_top_env at /root/.julia/packages/CondaPkg/GoPlj/src/resolve.jl:16
#29 92.84 _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2247 [inlined]
#29 92.84 jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2429
#29 92.84 #resolve#17 at /root/.julia/packages/CondaPkg/GoPlj/src/resolve.jl:222
#29 92.84 _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2247 [inlined]
#29 92.84 jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2429
#29 92.84 resolve at /root/.julia/packages/CondaPkg/GoPlj/src/resolve.jl:209
#29 92.84 _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2247 [inlined]
#29 92.84 jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2429
#29 92.84 envdir at /root/.julia/packages/CondaPkg/GoPlj/src/env.jl:66
#29 92.85 init_context at /root/.julia/packages/PythonCall/Z6DIG/src/cpython/context.jl:56
#29 92.86 __init__ at /root/.julia/packages/PythonCall/Z6DIG/src/cpython/CPython.jl:21
#29 92.87 jfptr___init___72494 at /deps.so (unknown line)
#29 92.87 _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2247 [inlined]
#29 92.87 jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2429
#29 92.87 jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1788 [inlined]
#29 92.87 jl_module_run_initializer at /buildworker/worker/package_linux64/build/src/toplevel.c:73
#29 92.87 _finish_julia_init at /buildworker/worker/package_linux64/build/src/init.c:796
#29 92.87 julia_init at /buildworker/worker/package_linux64/build/src/init.c:730
#29 92.87 jl_repl_entrypoint at /buildworker/worker/package_linux64/build/src/jlapi.c:695
#29 92.87 main at /buildworker/worker/package_linux64/build/cli/loader_exe.c:42
#29 92.88 __libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
#29 92.88 _start at /usr/local/julia/bin/julia (unknown line)
#29 93.11 ERROR: Failed to precompile MyPackage [...redacted...]
#29 94.21 Stacktrace:
#29 94.21  [1] error(s::String)
#29 94.60    @ Base ./error.jl:33
#29 94.77  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, ignore_loaded_modules::Bool)
#29 94.77    @ Base ./loading.jl:1466
#29 94.77  [3] compilecache(pkg::Base.PkgId, path::String)
#29 94.78    @ Base ./loading.jl:1410
#29 94.78  [4] _require(pkg::Base.PkgId)
#29 94.78    @ Base ./loading.jl:1120
#29 94.78  [5] require(uuidkey::Base.PkgId)
#29 94.78    @ Base ./loading.jl:1013
#29 94.78  [6] require(into::Module, mod::Symbol)
#29 94.78    @ Base ./loading.jl:997

(please excuse the #29 92.79 stuff, I'm in a dockerfile here).

I believe the issue is that PythonCall's __init__ tries to resolve the CondaPkg environment, but when a module is in a sysimage, that __init__ occurs at startup time (i.e. when julia itself is starting up), and at that point in the process the package environment isn't setup yet... or something like that.

[cjdoris]

You're saying __init__ is being called before LOAD_PATH is set? Surely that can't be right, it's a documented variable in Base, and Base must have already initialised? Can you inject some code somewhere to print out the LOAD_PATH to check if this is the case?

[ericphanson]

I tried adding println(Base.load_path()) to the first line of PythonCall's __init__, but for some reason that didn't print:

julia> using PackageCompiler # in an environment with my dev'd copy of PythonCall and PackageCompiler

julia> create_sysimage(["PythonCall"]; sysimage_path="pythoncall2.so")
✔ [01m:45s] PackageCompiler: compiling incremental system image
ld: warning: could not create compact unwind for _julia_Dict_15870: stack subq instruction is too different from dwarf stack size
ld: warning: could not create compact unwind for _julia_Dict_15862: stack subq instruction is too different from dwarf stack size
ld: warning: could not create compact unwind for _julia_YY.236_57076: stack subq instruction is too different from dwarf stack size

julia> exit()

❯ julia --sysimage=pythoncall2.so
fatal: error thrown and no exception handler available.
InitError(mod=:C, error=ErrorException("no environment in the LOAD_PATH depends on CondaPkg"))
error at /Users/eph/pythoncall2.so (unknown line)
jl_apply_generic at /Users/eph/.julia/juliaup/julia-1.7.2+0~x64/lib/julia/libjulia-internal.1.7.dylib (unknown line)
_resolve_top_env at /Users/eph/.julia/packages/CondaPkg/GoPlj/src/resolve.jl:16
jl_apply_generic at /Users/eph/.julia/juliaup/julia-1.7.2+0~x64/lib/julia/libjulia-internal.1.7.dylib (unknown line)
#resolve#17 at /Users/eph/.julia/packages/CondaPkg/GoPlj/src/resolve.jl:222
jl_apply_generic at /Users/eph/.julia/juliaup/julia-1.7.2+0~x64/lib/julia/libjulia-internal.1.7.dylib (unknown line)
resolve at /Users/eph/.julia/packages/CondaPkg/GoPlj/src/resolve.jl:209
jl_apply_generic at /Users/eph/.julia/juliaup/julia-1.7.2+0~x64/lib/julia/libjulia-internal.1.7.dylib (unknown line)
envdir at /Users/eph/.julia/packages/CondaPkg/GoPlj/src/env.jl:66
init_context at /Users/eph/pythoncall2.so (unknown line)
__init__ at /Users/eph/pythoncall2.so (unknown line)
jfptr___init___48878 at /Users/eph/pythoncall2.so (unknown line)
jl_apply_generic at /Users/eph/.julia/juliaup/julia-1.7.2+0~x64/lib/julia/libjulia-internal.1.7.dylib (unknown line)
jl_module_run_initializer at /Users/eph/.julia/juliaup/julia-1.7.2+0~x64/lib/julia/libjulia-internal.1.7.dylib (unknown line)
_finish_julia_init at /Users/eph/.julia/juliaup/julia-1.7.2+0~x64/lib/julia/libjulia-internal.1.7.dylib (unknown line)
julia_init at /Users/eph/.julia/juliaup/julia-1.7.2+0~x64/lib/julia/libjulia-internal.1.7.dylib (unknown line)
jl_repl_entrypoint at /Users/eph/.julia/juliaup/julia-1.7.2+0~x64/lib/julia/libjulia-internal.1.7.dylib (unknown line)

[KristofferC]

Works for me:

❯ julia --project --sysimage="pythoncall.dylib"                                                                
    CondaPkg Found dependencies: /Users/kristoffercarlsson/.julia/packages/PythonCall/Z6DIG/CondaPkg.toml
    CondaPkg Removing environment
...

Due to the way these packages seem structured make sure that you are using the same project as that you created the sysimage with.

[ericphanson]

Due to the way these packages seem structured make sure that you are using the same project as that you created the sysimage with.

Ah ok, I was not doing that-- on purpose, because I want to build a sysimage w/ all dependencies but not my package code, so I don't need to rebuild the sysimage every time I change the code I'm working on. (Basically this strategy: https://github.com/beacon-biosignals/julia_pod/blob/94a91696ed68e97d7749ccb5659ed22c0d27b048/add_me_to_your_PATH/Dockerfile.template#L4-L16).

edit: but in my non-MWE, I was also copying over the CondaPkg.toml and including CondaPkg in the sysimage, so even though it's not the exact same project environment, it seems like it should be able to work still?

[cjdoris]

Did you ever resolve this? I guess like the error says you need to ensure that CondaPkg is actually a dependency of something in your LOAD_PATH.

[YuliyaCl]

The same error occurs when I'm trying to run app.exe (created with PackageCompiler.create_app)

[ericphanson]

Did you ever resolve this?

No, I just removed PythonCall from my sysimage as a workaround for now.

I guess like the error says you need to ensure that CondaPkg is actually a dependency of something in your LOAD_PATH.

Hm, I thought I had that, but maybe not.

[glennmoy]

Logging for posterity as I'm hitting the issue from the other direction: building a sysimage in julia then trying to execute it via juliacall.

MWE - inside a virtual env

$ julia --project=/path/to/venv/julia_env 

julia> using PackageCompiler; Pkg

julia> project = Pkg.project()

julia> create_sysimage(collect(keys(project.dependencies)); sysimage_path="./venv/julia_env/vmr-image.so")

juliapkg.json:

{
    "packages": {
        "<Application>": {
            "uuid": "XXX",
            "dev": true,
            "path": "/path/to/application"
        },
        "PythonCall": {
            "uuid": "6099a3de-0909-46bc-b1f4-468b9a2dfc0d",
        },
        "CondaPkg": {
            "uuid": "992eb4ea-22a4-4c89-a5bb-47a3300528ab",
        },
        etc.,
    }
}

and then trying to load from a python session:

$ python -X juliacall-sysimage=/path/to/venv/julia_env/vmr-image.so -X juliapkg-project=/path/to/venv/julia_env
>>> import juliapkg
>>> juliapkg.project()
'/path/to/venv/julia_env'
>>> import juliacall
fatal: error thrown and no exception handler available.
InitError(mod=:C, error=ErrorException("no environment in the LOAD_PATH (["/home/ubuntu/.julia/environments/v1.9/Project.toml", "/home/ubuntu/.julia-downloads/julia-1.9.1/share/julia/stdlib/v1.9"]) depends on CondaPkg"))
error at ./error.jl:35
_resolve_top_env at /home/ubuntu/.julia/packages/CondaPkg/osUdN/src/resolve.jl:16
#resolve#38 at /home/ubuntu/.julia/packages/CondaPkg/osUdN/src/resolve.jl:404
resolve at /home/ubuntu/.julia/packages/CondaPkg/osUdN/src/resolve.jl:384
envdir at /home/ubuntu/.julia/packages/CondaPkg/osUdN/src/env.jl:70 [inlined]
init_context at /home/ubuntu/.julia/packages/PythonCall/FCrXm/src/cpython/context.jl:63
__init__ at /home/ubuntu/.julia/packages/PythonCall/FCrXm/src/cpython/CPython.jl:21
jfptr___init___103567 at /home/ubuntu/projects/Maestro.jl/experiments/07_vanilla_map_reduce_sidecar2/venv/julia_env/vmr-image.so (unknown line)
_jl_invoke at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
jl_module_run_initializer at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/toplevel.c:75
_finish_julia_init at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/init.c:855
julia_init at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/init.c:804
ijl_init_with_image at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/jlapi.c:66 [inlined]
ijl_init_with_image at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/jlapi.c:55
unknown function (ip: 0x7f4d56ce1ff4)
unknown function (ip: 0x7f4d56ce1409)
_ctypes_callproc at /usr/lib/python3.8/lib-dynload/_ctypes.cpython-38-x86_64-linux-gnu.so (unknown line)
unknown function (ip: 0x7f4d56cfaae6)
_PyObject_MakeTpCall at python (unknown line)
_PyEval_EvalFrameDefault at python (unknown line)
_PyEval_EvalCodeWithName at python (unknown line)
_PyFunction_Vectorcall at python (unknown line)
_PyEval_EvalFrameDefault at python (unknown line)
_PyEval_EvalCodeWithName at python (unknown line)
PyEval_EvalCode at python (unknown line)
unknown function (ip: 0x601173)
unknown function (ip: 0x5c52ef)
PyVectorcall_Call at python (unknown line)
_PyEval_EvalFrameDefault at python (unknown line)
_PyEval_EvalCodeWithName at python (unknown line)
_PyFunction_Vectorcall at python (unknown line)
_PyEval_EvalFrameDefault at python (unknown line)
_PyFunction_Vectorcall at python (unknown line)
_PyEval_EvalFrameDefault at python (unknown line)
_PyFunction_Vectorcall at python (unknown line)
_PyEval_EvalFrameDefault at python (unknown line)
_PyFunction_Vectorcall at python (unknown line)
_PyEval_EvalFrameDefault at python (unknown line)
_PyFunction_Vectorcall at python (unknown line)
unknown function (ip: 0x5f3c40)
_PyObject_CallMethodIdObjArgs at python (unknown line)
PyImport_ImportModuleLevelObject at python (unknown line)
_PyEval_EvalFrameDefault at python (unknown line)
_PyEval_EvalCodeWithName at python (unknown line)
PyEval_EvalCode at python (unknown line)
unknown function (ip: 0x680000)
unknown function (ip: 0x68007e)
unknown function (ip: 0x4a44a3)
PyRun_InteractiveLoopFlags at python (unknown line)
PyRun_AnyFileExFlags at python (unknown line)
unknown function (ip: 0x4eeecd)
Py_BytesMain at python (unknown line)
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
_start at python (unknown line)

[github-actions]

This issue has been marked as stale because it has been open for 30 days with no activity. If the issue is still relevant then please leave a comment, or else it will be closed in 7 days.

[ericphanson]

I believe this is not stale (and autoclose bots are annoying).

Specifically, I think this is the current state:

• If I build the sysimage w/ the whole top-level environment and pythoncall, then it works OK
• If I build the sysimage without pythoncall, then add it dynamically after, then it also works
• but if I try to save time by adding pythoncall to the sysimage (so it is baked in at that level) but add the final environment later, that's where I run into this issue

That could be considered an acceptable state of affairs (esp. if pkgimages mean sysimages are less relevant) but it can be very confusing to run into and I think unless it's clearly documented somewhere (or fixed) the issue should stay open. (in my opinion).

[cjdoris]

I'm not quite sure what you mean by "final environment"?

TBH I'm really (really really) not an expert in sysimages so I can't help much. The error (as far as CondaPkg sees it) is that your LOAD_PATH (at the time the package is loaded) does not have any entries depending on CondaPkg - which should not happen because you're loading CondaPkg from somewhere in your environment. So presumably some sysimage thing is messing that up - at which point my expertise stops and you'll have to figure out why.

[cjdoris]

(I agree autoclose bots are annoying, but it's also annoying and stressful for me to have more open issues than I can deal with sat there with no activity. The bot is simply a way to spread the load a bit - either the person who wrote the issue can help out a bit to fix the problem, or if they don't care enough then the issue may as well be closed. I've updated the text the bot uses to be a little less cold.)

[cjdoris]

If you think the issue is alleviated by better documentation, feel free to open a PR on docs/src/faq.md.

[ericphanson]

that your LOAD_PATH (at the time the package is loaded)

I think this might be basically the issue.

I am also not a sysimage expert, but I think it could be that when you add a package into a sysimage, you're essentially loading the package as normal, and then later serializing out the entire state of your julia session, loaded packages and all. So it could be that __init__ time is when the package is being added to the sysimage, which isn't really when we actually want to use the package; we want to use it later when we start our julia session (with the sysimage).

So I think the code "fix" would be to not do this work at init time, but rather some other time (e.g. on the first call to any function that needs to do something w/ python).

I think the docs "fix" would be a FAQ question like you said, and maybe that's preferable.

[ericphanson]

Actually I'm not sure my explanation is accurate, re-reading the original issue again... Since the error occurs later, once we start to use the sysimage. So I am lost as to the exact issue. But maybe I can still try to add a FAQ entry