Precompilation fails with both PyCall and PythonCall

[cuihantao]

The MWE below fails to precompile:

__precompile__()

module Test1

using PyCall

using PythonCall

function __init__()

end

end # module

Error message:

[ Info: Precompiling Test1 [062e1887-703b-4a08-a5da-6524f6b6cb77]
┌ Warning: Package Test1 does not have PyCall in its dependencies:
│ - If you have Test1 checked out for development and have
│   added PyCall as a dependency but haven't updated your primary
│   environment's manifest file, try `Pkg.resolve()`.
│ - Otherwise you may need to report an issue with Test1
└ Loading PyCall into Test1 from project dependency, future warnings for Test1 are suppressed.
free(): invalid pointer

signal (6): Aborted
in expression starting at /home/hacui/repos/Test1/src/Test1.jl:10
gsignal at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
abort at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x7f147367e29d)
unknown function (ip: 0x7f147368632b)
unknown function (ip: 0x7f1473687b5b)
_Py_DECREF at /usr/local/src/conda/python-3.10.4/Include/object.h:500 [inlined]
_Py_XDECREF at /usr/local/src/conda/python-3.10.4/Include/object.h:567 [inlined]
_PyImport_LoadDynamicModuleWithSpec at /usr/local/src/conda/python-3.10.4/Python/importdl.c:244 [inlined]
_imp_create_dynamic_impl at /usr/local/src/conda/python-3.10.4/Python/import.c:2049 [inlined]
_imp_create_dynamic at /usr/local/src/conda/python-3.10.4/Python/clinic/import.c.h:330
cfunction_vectorcall_FASTCALL at /usr/local/src/conda/python-3.10.4/Objects/methodobject.c:430
do_call_core at /usr/local/src/conda/python-3.10.4/Python/ceval.c:5891 [inlined]
_PyEval_EvalFrameDefault at /usr/local/src/conda/python-3.10.4/Python/ceval.c:4277
_PyEval_EvalFrame at /usr/local/src/conda/python-3.10.4/Include/internal/pycore_ceval.h:46 [inlined]
_PyEval_Vector at /usr/local/src/conda/python-3.10.4/Python/ceval.c:5065 [inlined]
_PyFunction_Vectorcall at /usr/local/src/conda/python-3.10.4/Objects/call.c:342
_PyObject_VectorcallTstate at /usr/local/src/conda/python-3.10.4/Include/cpython/abstract.h:114 [inlined]
PyObject_Vectorcall at /usr/local/src/conda/python-3.10.4/Include/cpython/abstract.h:123 [inlined]
call_function at /usr/local/src/conda/python-3.10.4/Python/ceval.c:5867 [inlined]
_PyEval_EvalFrameDefault at /usr/local/src/conda/python-3.10.4/Python/ceval.c:4181
_PyEval_EvalFrame at /usr/local/src/conda/python-3.10.4/Include/internal/pycore_ceval.h:46 [inlined]
_PyEval_Vector at /usr/local/src/conda/python-3.10.4/Python/ceval.c:5065 [inlined]
_PyFunction_Vectorcall at /usr/local/src/conda/python-3.10.4/Objects/call.c:342
_PyObject_VectorcallTstate at /usr/local/src/conda/python-3.10.4/Include/cpython/abstract.h:114 [inlined]
PyObject_Vectorcall at /usr/local/src/conda/python-3.10.4/Include/cpython/abstract.h:123 [inlined]
call_function at /usr/local/src/conda/python-3.10.4/Python/ceval.c:5867 [inlined]
_PyEval_EvalFrameDefault at /usr/local/src/conda/python-3.10.4/Python/ceval.c:4198
_PyEval_EvalFrame at /usr/local/src/conda/python-3.10.4/Include/internal/pycore_ceval.h:46 [inlined]
_PyEval_Vector at /usr/local/src/conda/python-3.10.4/Python/ceval.c:5065 [inlined]
_PyFunction_Vectorcall at /usr/local/src/conda/python-3.10.4/Objects/call.c:342
_PyObject_VectorcallTstate at /usr/local/src/conda/python-3.10.4/Include/cpython/abstract.h:114 [inlined]
PyObject_Vectorcall at /usr/local/src/conda/python-3.10.4/Include/cpython/abstract.h:123 [inlined]
call_function at /usr/local/src/conda/python-3.10.4/Python/ceval.c:5867 [inlined]
_PyEval_EvalFrameDefault at /usr/local/src/conda/python-3.10.4/Python/ceval.c:4213
_PyEval_EvalFrame at /usr/local/src/conda/python-3.10.4/Include/internal/pycore_ceval.h:46 [inlined]
_PyEval_Vector at /usr/local/src/conda/python-3.10.4/Python/ceval.c:5065 [inlined]
_PyFunction_Vectorcall at /usr/local/src/conda/python-3.10.4/Objects/call.c:342
_PyObject_VectorcallTstate at /usr/local/src/conda/python-3.10.4/Include/cpython/abstract.h:114 [inlined]
PyObject_Vectorcall at /usr/local/src/conda/python-3.10.4/Include/cpython/abstract.h:123 [inlined]
call_function at /usr/local/src/conda/python-3.10.4/Python/ceval.c:5867 [inlined]
_PyEval_EvalFrameDefault at /usr/local/src/conda/python-3.10.4/Python/ceval.c:4213
_PyEval_EvalFrame at /usr/local/src/conda/python-3.10.4/Include/internal/pycore_ceval.h:46 [inlined]
_PyEval_Vector at /usr/local/src/conda/python-3.10.4/Python/ceval.c:5065 [inlined]
_PyFunction_Vectorcall at /usr/local/src/conda/python-3.10.4/Objects/call.c:342
_PyObject_VectorcallTstate at /usr/local/src/conda/python-3.10.4/Include/cpython/abstract.h:114 [inlined]
PyObject_Vectorcall at /usr/local/src/conda/python-3.10.4/Include/cpython/abstract.h:123 [inlined]
call_function at /usr/local/src/conda/python-3.10.4/Python/ceval.c:5867 [inlined]
_PyEval_EvalFrameDefault at /usr/local/src/conda/python-3.10.4/Python/ceval.c:4213
_PyEval_EvalFrame at /usr/local/src/conda/python-3.10.4/Include/internal/pycore_ceval.h:46 [inlined]
_PyEval_Vector at /usr/local/src/conda/python-3.10.4/Python/ceval.c:5065 [inlined]
_PyFunction_Vectorcall at /usr/local/src/conda/python-3.10.4/Objects/call.c:342
_PyObject_VectorcallTstate at /usr/local/src/conda/python-3.10.4/Include/cpython/abstract.h:114 [inlined]
object_vacall at /usr/local/src/conda/python-3.10.4/Objects/call.c:734
_PyObject_CallMethodIdObjArgs at /usr/local/src/conda/python-3.10.4/Objects/call.c:825
import_find_and_load at /usr/local/src/conda/python-3.10.4/Python/import.c:1521 [inlined]
PyImport_ImportModuleLevelObject at /usr/local/src/conda/python-3.10.4/Python/import.c:1622
import_name at /usr/local/src/conda/python-3.10.4/Python/ceval.c:5992 [inlined]
_PyEval_EvalFrameDefault at /usr/local/src/conda/python-3.10.4/Python/ceval.c:3695
_PyEval_EvalFrame at /usr/local/src/conda/python-3.10.4/Include/internal/pycore_ceval.h:46 [inlined]
_PyEval_Vector at /usr/local/src/conda/python-3.10.4/Python/ceval.c:5065
PyEval_EvalCode at /usr/local/src/conda/python-3.10.4/Python/ceval.c:1134
builtin_exec_impl at /usr/local/src/conda/python-3.10.4/Python/bltinmodule.c:1056 [inlined]
builtin_exec at /usr/local/src/conda/python-3.10.4/Python/clinic/bltinmodule.c.h:371
cfunction_vectorcall_FASTCALL at /usr/local/src/conda/python-3.10.4/Objects/methodobject.c:430
do_call_core at /usr/local/src/conda/python-3.10.4/Python/ceval.c:5891 [inlined]
_PyEval_EvalFrameDefault at /usr/local/src/conda/python-3.10.4/Python/ceval.c:4277
_PyEval_EvalFrame at /usr/local/src/conda/python-3.10.4/Include/internal/pycore_ceval.h:46 [inlined]
_PyEval_Vector at /usr/local/src/conda/python-3.10.4/Python/ceval.c:5065 [inlined]
_PyFunction_Vectorcall at /usr/local/src/conda/python-3.10.4/Objects/call.c:342
_PyObject_VectorcallTstate at /usr/local/src/conda/python-3.10.4/Include/cpython/abstract.h:114 [inlined]
PyObject_Vectorcall at /usr/local/src/conda/python-3.10.4/Include/cpython/abstract.h:123 [inlined]
call_function at /usr/local/src/conda/python-3.10.4/Python/ceval.c:5867 [inlined]
_PyEval_EvalFrameDefault at /usr/local/src/conda/python-3.10.4/Python/ceval.c:4181
_PyEval_EvalFrame at /usr/local/src/conda/python-3.10.4/Include/internal/pycore_ceval.h:46 [inlined]
_PyEval_Vector at /usr/local/src/conda/python-3.10.4/Python/ceval.c:5065 [inlined]
_PyFunction_Vectorcall at /usr/local/src/conda/python-3.10.4/Objects/call.c:342
_PyObject_VectorcallTstate at /usr/local/src/conda/python-3.10.4/Include/cpython/abstract.h:114 [inlined]
PyObject_Vectorcall at /usr/local/src/conda/python-3.10.4/Include/cpython/abstract.h:123 [inlined]
call_function at /usr/local/src/conda/python-3.10.4/Python/ceval.c:5867 [inlined]
_PyEval_EvalFrameDefault at /usr/local/src/conda/python-3.10.4/Python/ceval.c:4198
_PyEval_EvalFrame at /usr/local/src/conda/python-3.10.4/Include/internal/pycore_ceval.h:46 [inlined]
_PyEval_Vector at /usr/local/src/conda/python-3.10.4/Python/ceval.c:5065 [inlined]
_PyFunction_Vectorcall at /usr/local/src/conda/python-3.10.4/Objects/call.c:342
_PyObject_VectorcallTstate at /usr/local/src/conda/python-3.10.4/Include/cpython/abstract.h:114 [inlined]
PyObject_Vectorcall at /usr/local/src/conda/python-3.10.4/Include/cpython/abstract.h:123 [inlined]
call_function at /usr/local/src/conda/python-3.10.4/Python/ceval.c:5867 [inlined]
_PyEval_EvalFrameDefault at /usr/local/src/conda/python-3.10.4/Python/ceval.c:4213
_PyEval_EvalFrame at /usr/local/src/conda/python-3.10.4/Include/internal/pycore_ceval.h:46 [inlined]
_PyEval_Vector at /usr/local/src/conda/python-3.10.4/Python/ceval.c:5065 [inlined]
_PyFunction_Vectorcall at /usr/local/src/conda/python-3.10.4/Objects/call.c:342
_PyObject_VectorcallTstate at /usr/local/src/conda/python-3.10.4/Include/cpython/abstract.h:114 [inlined]
PyObject_Vectorcall at /usr/local/src/conda/python-3.10.4/Include/cpython/abstract.h:123 [inlined]
call_function at /usr/local/src/conda/python-3.10.4/Python/ceval.c:5867 [inlined]
_PyEval_EvalFrameDefault at /usr/local/src/conda/python-3.10.4/Python/ceval.c:4213
_PyEval_EvalFrame at /usr/local/src/conda/python-3.10.4/Include/internal/pycore_ceval.h:46 [inlined]
_PyEval_Vector at /usr/local/src/conda/python-3.10.4/Python/ceval.c:5065 [inlined]
_PyFunction_Vectorcall at /usr/local/src/conda/python-3.10.4/Objects/call.c:342
_PyObject_VectorcallTstate at /usr/local/src/conda/python-3.10.4/Include/cpython/abstract.h:114 [inlined]
object_vacall at /usr/local/src/conda/python-3.10.4/Objects/call.c:734
_PyObject_CallMethodIdObjArgs at /usr/local/src/conda/python-3.10.4/Objects/call.c:825
import_find_and_load at /usr/local/src/conda/python-3.10.4/Python/import.c:1521 [inlined]
PyImport_ImportModuleLevelObject at /usr/local/src/conda/python-3.10.4/Python/import.c:1622
builtin___import__ at /usr/local/src/conda/python-3.10.4/Python/bltinmodule.c:272
cfunction_call at /usr/local/src/conda/python-3.10.4/Objects/methodobject.c:543
_PyObject_MakeTpCall at /usr/local/src/conda/python-3.10.4/Objects/call.c:215
_PyObject_VectorcallTstate at /usr/local/src/conda/python-3.10.4/Include/cpython/abstract.h:112 [inlined]
_PyObject_VectorcallTstate at /usr/local/src/conda/python-3.10.4/Include/cpython/abstract.h:99 [inlined]
_PyObject_CallFunctionVa at /usr/local/src/conda/python-3.10.4/Objects/call.c:485 [inlined]
PyObject_CallFunction at /usr/local/src/conda/python-3.10.4/Objects/call.c:507
PyImport_Import at /usr/local/src/conda/python-3.10.4/Python/import.c:1827
PyImport_Import at /home/hacui/.julia/packages/PythonCall/XgP8G/src/cpython/pointers.jl:297 [inlined]
macro expansion at /home/hacui/.julia/packages/PythonCall/XgP8G/src/Py.jl:131 [inlined]
pyimport at /home/hacui/.julia/packages/PythonCall/XgP8G/src/concrete/import.jl:11
init_consts at /home/hacui/.julia/packages/PythonCall/XgP8G/src/concrete/consts.jl:181
#187 at /home/hacui/.julia/packages/PythonCall/XgP8G/src/PythonCall.jl:86
unknown function (ip: 0x7f141a66924f)
with_gil at /home/hacui/.julia/packages/PythonCall/XgP8G/src/cpython/gil.jl:10 [inlined]
with_gil at /home/hacui/.julia/packages/PythonCall/XgP8G/src/cpython/gil.jl:9 [inlined]
__init__ at /home/hacui/.julia/packages/PythonCall/XgP8G/src/PythonCall.jl:85
unknown function (ip: 0x7f141a6397df)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2247 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2429
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1788 [inlined]
jl_module_run_initializer at /buildworker/worker/package_linux64/build/src/toplevel.c:73
jl_init_restored_modules at /buildworker/worker/package_linux64/build/src/dump.c:2215
_include_from_serialized at ./loading.jl:768
_require_search_from_serialized at ./loading.jl:854
_require at ./loading.jl:1097
require at ./loading.jl:1013
require at ./loading.jl:997
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2247 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2429
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1788 [inlined]
call_require at /buildworker/worker/package_linux64/build/src/toplevel.c:426 [inlined]
eval_import_path at /buildworker/worker/package_linux64/build/src/toplevel.c:463
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:689
jl_eval_module_expr at /buildworker/worker/package_linux64/build/src/toplevel.c:196 [inlined]
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:673
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:830
jl_toplevel_eval_in at /buildworker/worker/package_linux64/build/src/toplevel.c:944
eval at ./boot.jl:373 [inlined]
include_string at ./loading.jl:1196
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2247 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2429
_include at ./loading.jl:1253
include at ./Base.jl:418 [inlined]
include_package_for_output at ./loading.jl:1318
jfptr_include_package_for_output_24389.clone_1 at /home/hacui/packages/julias/julia-1.7/lib/julia/sys.so (unknown line)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2247 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2429
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1788 [inlined]
do_call at /buildworker/worker/package_linux64/build/src/interpreter.c:126
eval_value at /buildworker/worker/package_linux64/build/src/interpreter.c:215
eval_stmt_value at /buildworker/worker/package_linux64/build/src/interpreter.c:166 [inlined]
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:587
jl_interpret_toplevel_thunk at /buildworker/worker/package_linux64/build/src/interpreter.c:731
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:885
jl_toplevel_eval_in at /buildworker/worker/package_linux64/build/src/toplevel.c:944
eval at ./boot.jl:373 [inlined]
eval at ./client.jl:453
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2247 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2429
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1788 [inlined]
do_call at /buildworker/worker/package_linux64/build/src/interpreter.c:126
eval_value at /buildworker/worker/package_linux64/build/src/interpreter.c:215
eval_stmt_value at /buildworker/worker/package_linux64/build/src/interpreter.c:166 [inlined]
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:587
jl_interpret_toplevel_thunk at /buildworker/worker/package_linux64/build/src/interpreter.c:731
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:885
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:830
jl_toplevel_eval_in at /buildworker/worker/package_linux64/build/src/toplevel.c:944
eval at ./boot.jl:373 [inlined]
exec_options at ./client.jl:268
_start at ./client.jl:495
jfptr__start_38732.clone_1 at /home/hacui/packages/julias/julia-1.7/lib/julia/sys.so (unknown line)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2247 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2429
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1788 [inlined]
true_main at /buildworker/worker/package_linux64/build/src/jlapi.c:559
jl_repl_entrypoint at /buildworker/worker/package_linux64/build/src/jlapi.c:701
main at /buildworker/worker/package_linux64/build/cli/loader_exe.c:42
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
_start at /home/hacui/packages/julias/julia-1.7/bin/julia (unknown line)
Allocations: 7835414 (Pool: 7831080; Big: 4334); GC: 9
ERROR: Failed to precompile Test1 [062e1887-703b-4a08-a5da-6524f6b6cb77] to /home/hacui/.julia/compiled/v1.7/Test1/jl_DYxy5e.
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:33
 [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, ignore_loaded_modules::Bool)
   @ Base ./loading.jl:1466
 [3] compilecache(pkg::Base.PkgId, path::String)
   @ Base ./loading.jl:1410
 [4] _require(pkg::Base.PkgId)
   @ Base ./loading.jl:1120
 [5] require(uuidkey::Base.PkgId)
   @ Base ./loading.jl:1013
 [6] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:997

julia> 

[pedromxavier]

Hi @cuihantao, have you included those packages via ] add PyCall PythonCall in your Project.toml file?

[cuihantao]

@pedromxavier I did add these to the package. The Project.toml file reads like

name = "Test1"
uuid = "062e1887-703b-4a08-a5da-6524f6b6cb77"
authors = ["Hantao Cui <[email protected]>"]
version = "0.1.0"

[deps]
PyCall = "438e738f-606a-5dbb-bf0a-cddfbfd45ab0"
PythonCall = "6099a3de-0909-46bc-b1f4-468b9a2dfc0d"

Did you have any success replicating the issue? I can be readily replicated using ]generate Test1, ]activate ./Test, ]add PythonCall PyCall, and then using PythonCall; using PyCall.

[cjdoris]

Yeah I can reproduce this on Linux. The issue is that PyCall installs Python 3.9 and PythonCall installs Python 3.10, and the ABIs clash.

Both PyCall and PythonCall load libpython with RTLD_GLOBAL, which means they share a global symbol namespace, hence they clash when trying to use symbols with the same name from different versions of Python. Probably if one or the other used RTLD_LOCAL this issue wouldn't arise, but I'm not sure what the implications of this are.

In the meantime, you can work around it by ensuring both packages use the same Python. You can do one of:

• ENV["JULIA_PYTHONCALL_EXE"] = "@PyCall" to force PythonCall to use PyCall's Python. But this opts out of any automatic dependency management.
• ENV["PYTHON"] = PythonCall.C.CTX.exe_path; Pkg.build("PyCall") to force PyCall to use PythonCall's Python. But you'll need to do this each time you change project because PythonCall by default uses a different Python per project.

[cjdoris]

Using RTLD_LOCAL is not possible (third point here: https://docs.python.org/3.10/whatsnew/3.8.html#changes-in-the-c-api) so you'll have to use one of the above methods on Linux. I'll document this.

[cuihantao]

Thank you!