Build failure on Linux/ARM64

[rgobbel]

Running on Ubuntu 18.04, on an NVIDIA AGX Xavier. Some of this may be related to #39612, but I think there's more than that going on.

Base  ───────────── 64.912324 seconds
ArgTools  ───────── 13.429038 seconds
Artifacts  ────────  0.200953 seconds
Base64  ───────────  0.314328 seconds
CRC32c  ───────────  0.013255 seconds
FileWatching  ─────  0.198042 seconds
Libdl  ────────────  0.003328 seconds
Logging  ──────────  0.065492 seconds
Mmap  ─────────────  0.171325 seconds
NetworkOptions  ───  0.135897 seconds
SHA  ──────────────  0.411277 seconds
Serialization  ────  0.775204 seconds
Sockets  ──────────  0.652390 seconds
Unicode  ──────────  0.011515 seconds
DelimitedFiles  ───  0.208232 seconds
LinearAlgebra  ──── 17.310527 seconds
Markdown  ─────────  2.466062 seconds
Printf  ───────────  0.739326 seconds
Random  ───────────  1.091169 seconds
Tar  ──────────────  0.571908 seconds
Dates  ────────────  4.815496 seconds
Distributed  ──────  1.811784 seconds
Future  ───────────  0.010328 seconds
InteractiveUtils  ─  1.468970 seconds
LibGit2  ──────────  2.856807 seconds
Profile  ──────────  0.605529 seconds
SparseArrays  ─────  6.637348 seconds
UUIDs  ────────────  0.031589 seconds
REPL  ───────────── 11.367567 seconds
SharedArrays  ─────  0.295236 seconds
Statistics  ───────  0.348245 seconds
SuiteSparse  ──────  1.553806 seconds
TOML  ─────────────  0.166104 seconds
Test  ─────────────  0.524904 seconds
LibCURL  ──────────  0.917544 seconds
Downloads  ────────  1.068550 seconds

signal (11): Segmentation fault
in expression starting at /nvme/home/gobbel/src/julia/usr/share/julia/stdlib/v1.7/Pkg/src/REPLMode/command_declarations.jl:3
__aarch64_sync_cache_range at /workspace/srcdir/gcc-8.1.0/libgcc/config/aarch64/sync-cache.c:54
_ZN4llvm3sys6Memory26InvalidateInstructionCacheEPKvm at /nvme/home/gobbel/src/julia/usr/bin/../lib/libLLVM-11jl.so (unknown line)
Allocations: 97008282 (Pool: 96985129; Big: 23153); GC: 129
Pkg  ────────────── 11.911898 seconds
LazyArtifacts  ────  0.004044 seconds
Stdlibs total  ──── 85.209436 seconds
Sysimage built. Summary:
Total ─────── 150.128497 seconds 
Base: ───────  64.912324 seconds 43.2378%
Stdlibs: ────  85.209436 seconds 56.7577%
    JULIA usr/lib/julia/sys-o.a
Invalid instruction at 0x7f9638b0fc: 0xd5380015

signal (4): Illegal instruction
in expression starting at none:0
gotoblas_dynamic_init at /nvme/home/gobbel/src/julia/usr/bin/../lib/libopenblas64_.so (unknown line)
gotoblas_init at /nvme/home/gobbel/src/julia/usr/bin/../lib/libopenblas64_.so (unknown line)
unknown function (ip: 0x7fa7fd7a33)
unknown function (ip: 0x7fa7fd7a33)
Allocations: 2677 (Pool: 2666; Big: 11); GC: 0
ERROR: LoadError: failed process: Process('/nvme/home/gobbel/src/julia/usr/bin/julia -O0 --sysimage /nvme/home/gobbel/src/julia/usr/lib/julia/sys.ji --startup-file=no -Cnative -e 'pushfirst!(DEPOT_PATH, "/tmp/jl_v8PiQf");
Base.PRECOMPILE_TRACE_COMPILE[] = "/tmp/jl_yVS9gn";
Base.compilecache(Base.PkgId("__PackagePrecompilationStatementModule"), "/tmp/jl_v8PiQf/__PackagePrecompilationStatementModule/src/__PackagePrecompilationStatementModule.jl")
# NOTE: these were moved to the end of Base.jl. TODO: move back here.
# # Used by Revise & its dependencies
# while true  # force inference
# delete!(push!(Set{Module}(), Base), Main)
# m = first(methods(+))
# delete!(push!(Set{Method}(), m), m)
# empty!(Set())
# push!(push!(Set{Union{GlobalRef,Symbol}}(), :two), GlobalRef(Base, :two))
# (setindex!(Dict{String,Base.PkgId}(), Base.PkgId(Base), "file.jl"))["file.jl"]
# (setindex!(Dict{Symbol,Vector{Int}}(), [1], :two))[:two]
# (setindex!(Dict{Base.PkgId,String}(), "file.jl", Base.PkgId(Base)))[Base.PkgId(Base)]
# (setindex!(Dict{Union{GlobalRef,Symbol}, Vector{Int}}(), [1], :two))[:two]
# (setindex!(IdDict{Type, Union{Missing, Vector{Tuple{LineNumberNode, Expr}}}}(), missing, Int))[Int]
# Dict{Symbol, Union{Nothing, Bool, Symbol}}(:one => false)[:one]
# Dict(Base => [:(1+1)])[Base]
# Dict(:one => [1])[:one]
# Dict("abc" => Set())["abc"]
# pushfirst!([], sum)
# get(Base.pkgorigins, Base.PkgId(Base), nothing)
# sort!([1,2,3])
# unique!([1,2,3])
# cumsum([1,2,3])
# append!(Int[], BitSet())
# isempty(BitSet())
# delete!(BitSet([1,2]), 3)
# deleteat!(Int32[1,2,3], [1,3])
# deleteat!(Any[1,2,3], [1,3])
# Core.svec(1, 2) == Core.svec(3, 4)
# # copy(Core.Compiler.retrieve_code_info(Core.Compiler.specialize_method(which(+, (Int, Int)), [Int, Int], Core.svec())))
# any(t->t[1].line > 1, [(LineNumberNode(2,:none),:(1+1))])
# break   # end force inference
# end
using Artifacts, Base.BinaryPlatforms, Libdl
artifacts_toml = abspath(joinpath(Sys.STDLIB, "Artifacts", "test", "Artifacts.toml"))
# cd(() -> (name = "c_simple"; @artifact_str(name)), dirname(artifacts_toml))
artifacts = Artifacts.load_artifacts_toml(artifacts_toml)
platforms = [Artifacts.unpack_platform(e, "c_simple", artifacts_toml) for e in artifacts["c_simple"]]
best_platform = select_platform(Dict(p => triplet(p) for p in platforms))
dlopen("libjulia", RTLD_LAZY | RTLD_DEEPBIND)

', ProcessSignaled(4)) [0]

Stacktrace:
  [1] pipeline_error
    @ ./process.jl:525 [inlined]
  [2] run(::Cmd; wait::Bool)
    @ Base ./process.jl:440
  [3] run
    @ ./process.jl:438 [inlined]
  [4] (::Main.anonymous.var"#1#5"{Set{String}, String})(prec_path::String)
    @ Main.anonymous /nvme~/src/julia/contrib/generate_precompile.jl:257
  [5] mktempdir(fn::Main.anonymous.var"#1#5"{Set{String}, String}, parent::String; prefix::String)
    @ Base.Filesystem ./file.jl:734
  [6] mktempdir
    @ ./file.jl:732 [inlined]
  [7] generate_precompile_statements()
    @ Main.anonymous /nvme~/src/julia/contrib/generate_precompile.jl:240
  [8] top-level scope
    @ /nvme~/src/julia/contrib/generate_precompile.jl:392
  [9] eval(m::Module, e::Any)
    @ Core ./boot.jl:369
 [10] top-level scope
    @ /nvme~/src/julia/contrib/generate_precompile.jl:6
in expression starting at /nvme/home/gobbel/src/julia/contrib/generate_precompile.jl:3
*** This error is usually fixed by running `make clean`. If the error persists, try `make cleanall`. ***
sysimage.mk:86: recipe for target '/nvme/home/gobbel/src/julia/usr/lib/julia/sys-o.a' failed
make[1]: *** [/nvme/home/gobbel/src/julia/usr/lib/julia/sys-o.a] Error 1
Makefile:88: recipe for target 'julia-sysimg-release' failed
make: *** [julia-sysimg-release] Error 2
:~/src/julia$ date
Mon Mar  1 12:07:47 PST 2021
~/src/julia$ git status
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean

[Keno]

There seem to be two separate things going on here:

> signal (11): Segmentation fault
> in expression starting at /nvme/home/gobbel/src/julia/usr/share/julia/stdlib/v1.7/Pkg/src/REPLMode/command_declarations.jl:3
> __aarch64_sync_cache_range at /workspace/srcdir/gcc-8.1.0/libgcc/config/aarch64/sync-cache.c:54
> _ZN4llvm3sys6Memory26InvalidateInstructionCacheEPKvm at /nvme/home/gobbel/src/julia/usr/bin/../lib/libLLVM-11jl.so (unknown line)

This is a segfault while trying to invalidate the cache. I'm a bit surprised by this. This function hardly touches memory. It's possible of course for the cache invalidation instructions to generate a trap, but I don't think they do on other Aarch64 platforms.

> Invalid instruction at 0x7f9638b0fc: 0xd5380015
> 
> signal (4): Illegal instruction
> in expression starting at none:0
> gotoblas_dynamic_init at /nvme/home/gobbel/src/julia/usr/bin/../lib/libopenblas64_.so (unknown line)

This looks like an issue with openblas's architecture detection code.

[ViralBShah]

Is this on Julia master or the release-1.6 branch?

[rgobbel]

Is this on Julia master or the release-1.6 branch?

After doing:

git pull
make distcleanall
git clean -xfd

v1.6.0 now builds without errors. Julia master (after the same thorough pre-build clean) now fails in a different way than described above:

Downloads  ────────  1.022166 seconds

signal (11): Segmentation fault
in expression starting at /nvme/home/gobbel/src/julia/usr/share/julia/stdlib/v1.7/Pkg/src/REPLMode/command_declarations.jl:3
__aarch64_sync_cache_range at /workspace/srcdir/gcc-8.1.0/libgcc/config/aarch64/sync-cache.c:54
_ZN4llvm3sys6Memory26InvalidateInstructionCacheEPKvm at /nvme/home/gobbel/src/julia/usr/bin/../lib/libLLVM-11jl.so (unknown line)
Allocations: 97829096 (Pool: 97806293; Big: 22803); GC: 129
Segmentation fault (core dumped)
*** This error might be fixed by running `make clean`. If the error persists, try `make cleanall`. ***
sysimage.mk:67: recipe for target '/nvme/home/gobbel/src/julia/usr/lib/julia/sys.ji' failed
make[1]: *** [/nvme/home/gobbel/src/julia/usr/lib/julia/sys.ji] Error 1
Makefile:82: recipe for target 'julia-sysimg-ji' failed
make: *** [julia-sysimg-ji] Error 2

I've tried this with GCC versions 7 and 8, both with the same result.

Happy to try any other suggested environment changes.

[ViralBShah]

Perhaps worth pinging @vchuravy

[ViralBShah]

I just tried on an aarch64 box, and master builds fine. Of course, not all arms are the same, but it would be helpful to get some more information here about the arm chip.

[rgobbel]

I'm doing this on an NVIDIA AGX Xavier:

gobbel@xavier:~/src/julia$ lscpu
Architecture:        aarch64
Byte Order:          Little Endian
CPU(s):              8
On-line CPU(s) list: 0-7
Thread(s) per core:  1
Core(s) per socket:  2
Socket(s):           4
Vendor ID:           Nvidia
Model:               0
Model name:          ARMv8 Processor rev 0 (v8l)
Stepping:            0x0
CPU max MHz:         2265.6001
CPU min MHz:         115.2000
BogoMIPS:            62.50
L1d cache:           64K
L1i cache:           128K
L2 cache:            2048K
L3 cache:            4096K
Flags:               fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp

gobbel@xavier:~/src/julia$ cat /proc/cpuinfo 
processor	: 0
model name	: ARMv8 Processor rev 0 (v8l)
BogoMIPS	: 62.50
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp
CPU implementer	: 0x4e
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0x004
CPU revision	: 0
MTS version	: 51035886

Apparently because of the way that NVIDIA chose to deal with the Linux kernel device tree, this system is stuck at Ubuntu 18.04, with kernel 4.9.201-tegra. I have another AGX Xavier that I'm going to try to update, after which I can try the build with an up-to-date kernel.

I tried splitting up the loading of the Pkg commands (initialize compound_declarations with empty vectors, then append command declarations one at a time) to see if I could get a better sense of where it was failing, and was surprised to see that after that change, it gets all the way through Pkg setup. After that, it runs into a problem with OpenBLAS:

Invalid instruction at 0x7f90c390fc: 0xd5380015

signal (4): Illegal instruction
in expression starting at none:0
gotoblas_dynamic_init at /nvme/home/gobbel/src/julia/usr/bin/../lib/libopenblas64_.so (unknown line)
gotoblas_init at /nvme/home/gobbel/src/julia/usr/bin/../lib/libopenblas64_.so (unknown line)
unknown function (ip: 0x7fa298ca33)
unknown function (ip: 0x7fa298ca33)
Allocations: 2674 (Pool: 2665; Big: 9); GC: 0
ERROR: LoadError: failed process: Process(`/nvme/home/gobbel/src/julia/usr/bin/julia -O0 --sysimage /nvme/home/gobbel/src/julia/usr/lib/julia/sys.ji --startup-file=no -Cnative -e 'pushfirst!(DEPOT_PATH, "/tmp/jl_bLZe9W");
Base.PRECOMPILE_TRACE_COMPILE[] = "/tmp/jl_4vISTL";
Base.compilecache(Base.PkgId("__PackagePrecompilationStatementModule"), "/tmp/jl_bLZe9W/__PackagePrecompilationStatementModule/src/__PackagePrecompilationStatementModule.jl")
# NOTE: these were moved to the end of Base.jl. TODO: move back here.
# # Used by Revise & its dependencies
# while true  # force inference
# delete!(push!(Set{Module}(), Base), Main)
# m = first(methods(+))
# delete!(push!(Set{Method}(), m), m)
# empty!(Set())
# push!(push!(Set{Union{GlobalRef,Symbol}}(), :two), GlobalRef(Base, :two))
# (setindex!(Dict{String,Base.PkgId}(), Base.PkgId(Base), "file.jl"))["file.jl"]
# (setindex!(Dict{Symbol,Vector{Int}}(), [1], :two))[:two]
# (setindex!(Dict{Base.PkgId,String}(), "file.jl", Base.PkgId(Base)))[Base.PkgId(Base)]
# (setindex!(Dict{Union{GlobalRef,Symbol}, Vector{Int}}(), [1], :two))[:two]
# (setindex!(IdDict{Type, Union{Missing, Vector{Tuple{LineNumberNode, Expr}}}}(), missing, Int))[Int]
# Dict{Symbol, Union{Nothing, Bool, Symbol}}(:one => false)[:one]
# Dict(Base => [:(1+1)])[Base]
# Dict(:one => [1])[:one]
# Dict("abc" => Set())["abc"]
# pushfirst!([], sum)
# get(Base.pkgorigins, Base.PkgId(Base), nothing)
# sort!([1,2,3])
# unique!([1,2,3])
# cumsum([1,2,3])
# append!(Int[], BitSet())
# isempty(BitSet())
# delete!(BitSet([1,2]), 3)
# deleteat!(Int32[1,2,3], [1,3])
# deleteat!(Any[1,2,3], [1,3])
# Core.svec(1, 2) == Core.svec(3, 4)
# # copy(Core.Compiler.retrieve_code_info(Core.Compiler.specialize_method(which(+, (Int, Int)), [Int, Int], Core.svec())))
# any(t->t[1].line > 1, [(LineNumberNode(2,:none),:(1+1))])
# break   # end force inference
# end
using Artifacts, Base.BinaryPlatforms, Libdl
artifacts_toml = abspath(joinpath(Sys.STDLIB, "Artifacts", "test", "Artifacts.toml"))
# cd(() -> (name = "HelloWorldC"; @artifact_str(name)), dirname(artifacts_toml))
artifacts = Artifacts.load_artifacts_toml(artifacts_toml)
platforms = [Artifacts.unpack_platform(e, "HelloWorldC", artifacts_toml) for e in artifacts["HelloWorldC"]]
best_platform = select_platform(Dict(p => triplet(p) for p in platforms))
dlopen("libjulia", RTLD_LAZY | RTLD_DEEPBIND)

'`, ProcessSignaled(4)) [0]

Stacktrace:
  [1] pipeline_error
    @ ./process.jl:525 [inlined]
  [2] run(::Cmd; wait::Bool)
    @ Base ./process.jl:440
  [3] run
    @ ./process.jl:438 [inlined]
  [4] (::Main.anonymous.var"#1#5"{Set{String}, String})(prec_path::String)
    @ Main.anonymous /nvme~/src/julia/contrib/generate_precompile.jl:257
  [5] mktempdir(fn::Main.anonymous.var"#1#5"{Set{String}, String}, parent::String; prefix::String)
    @ Base.Filesystem ./file.jl:740
  [6] mktempdir (repeats 2 times)
    @ ./file.jl:738 [inlined]
  [7] generate_precompile_statements()
    @ Main.anonymous /nvme~/src/julia/contrib/generate_precompile.jl:240
  [8] top-level scope
    @ /nvme~/src/julia/contrib/generate_precompile.jl:409
  [9] eval(m::Module, e::Any)
    @ Core ./boot.jl:369
 [10] top-level scope
    @ /nvme~/src/julia/contrib/generate_precompile.jl:6
in expression starting at /nvme/home/gobbel/src/julia/contrib/generate_precompile.jl:3
*** This error is usually fixed by running `make clean`. If the error persists, try `make cleanall`. ***
sysimage.mk:86: recipe for target '/nvme/home/gobbel/src/julia/usr/lib/julia/sys-o.a' failed
make[1]: *** [/nvme/home/gobbel/src/julia/usr/lib/julia/sys-o.a] Error 1
Makefile:88: recipe for target 'julia-sysimg-release' failed
make: *** [julia-sysimg-release] Error 2

[vchuravy]

Ok this looks like a two part issue:

1. The OpenBLAS issue looks like Illegal instruction (core dumped) on import for numpy 1.19.5 on ARM64 numpy/numpy#18131 and might have been fixed by Label the assembly part of the ARMV8 dynamic arch detection as volatile OpenMathLib/OpenBLAS#3060, we might want to selectively backport that.

signal (4): Illegal instruction
in expression starting at none:0
gotoblas_dynamic_init at /nvme/home/gobbel/src/julia/usr/bin/../lib/libopenblas64_.so (unknown line)
gotoblas_init at /nvme/home/gobbel/src/julia/usr/bin/../lib/libopenblas64_.so (unknown line)

2. The issue in LLVM looks odd. Can you run that through GDB and dump the instruction it is segfaulting on?
   It is coming from https://github.com/gcc-mirror/gcc/blob/releases/gcc-8/libgcc/config/aarch64/sync-cache.c and it would be good to know which instruction causes the issue, but there is very little we can immediately do about that (except if it got fixed in subsequent GCC versions)

[vchuravy]

The only recent change to that file is gcc-mirror/gcc@761e6bb, e.g. GCC 10.1/10.2

[ViralBShah]

Maybe report it to NVIDIA if there was a way to, I suppose.

[maleadt]

This isn't only a build failure, but also manifests with our binaries of course:

$ ./jl
Invalid instruction at 0x7f96faf0fc: 0xd5380015

signal (4): Illegal instruction
in expression starting at none:0
gotoblas_dynamic_init at /home/tim/.cache/jl/installs/bin/linux/aarch64/1.7/julia-1.7-latest-linux-aarch64/bin/../lib/julia/libopenblas64_.so (unknown line)
gotoblas_init at /home/tim/.cache/jl/installs/bin/linux/aarch64/1.7/julia-1.7-latest-linux-aarch64/bin/../lib/julia/libopenblas64_.so (unknown line)
unknown function (ip: 0x7fac026a33)
unknown function (ip: 0x7fac026a33)
Allocations: 2719 (Pool: 2709; Big: 10); GC: 0
Illegal instruction (core dumped)

Given how ARMv8 on Linux is Tier 1, maybe we should put this on the milestone? I can have a look at backporting and testing the necessary patches next week.

About the __aarch64_sync_cache_range issue: I've been seeing this forever on my specific ARM devkit (a Jetson AGX), but it's somehow never fatal (despite printing a fatal error). Here, too, it seems to continue just fine after it.

[maleadt]

I'm actually stuck on the cache invalidation segfault trying to fix the OpenBLAS-related issue (I don't remember it actually getting in the way of compiling Julia on this system). Anyway, the full back-trace is as follows:

Thread 1 "julia" received signal SIGSEGV, Segmentation fault.
__aarch64_sync_cache_range (base=0x7fac5faf90, end=0x7fac5fafe0) at /workspace/srcdir/gcc-11.1.0/libgcc/config/aarch64/sync-cache.c:61
61      /workspace/srcdir/gcc-11.1.0/libgcc/config/aarch64/sync-cache.c: No such file or directory.
(gdb) bt
#0  __aarch64_sync_cache_range (base=0x7fac5faf90, end=0x7fac5fafe0) at /workspace/srcdir/gcc-11.1.0/libgcc/config/aarch64/sync-cache.c:61
#1  0x0000007fb487737c in llvm::sys::Memory::InvalidateInstructionCache(void const*, unsigned long) () from /home/tim/julia/usr/bin/../lib/libLLVM-12jl.so
#2  0x0000007fb7918438 in (anonymous namespace)::ROAllocator<false>::finalize (this=0x555562dc70) at /home/tim/julia/src/cgmemmgr.cpp:529
#3  (anonymous namespace)::DualMapAllocator<false>::finalize (this=0x555562dc70) at /home/tim/julia/src/cgmemmgr.cpp:662
#4  0x0000007fb7919e1c in (anonymous namespace)::RTDyldMemoryManagerJL::finalizeMemory (this=0x555562ca60, ErrMsg=<optimized out>) at /home/tim/julia/src/cgmemmgr.cpp:890
#5  0x0000007fb61aaaf4 in llvm::RuntimeDyldImpl::finalizeAsync(std::unique_ptr<llvm::RuntimeDyldImpl, std::default_delete<llvm::RuntimeDyldImpl> >, llvm::unique_function<void (llvm::object::OwningBinary<llvm::object::ObjectFile>, std::unique_ptr<llvm::RuntimeDyld::LoadedObjectInfo, std::default_delete<llvm::RuntimeDyld::LoadedObjectInfo> >, llvm::Error)>, llvm::object::OwningBinary<llvm::object::ObjectFile>, std::unique_ptr<llvm::RuntimeDyld::LoadedObjectInfo, std::default_delete<llvm::RuntimeDyld::LoadedObjectInfo> >)::{lambda(llvm::Expected<std::map<llvm::StringRef, llvm::JITEvaluatedSymbol, std::less<llvm::StringRef>, std::allocator<std::pair<llvm::StringRef const, llvm::JITEvaluatedSymbol> > > >)#1}::operator()(llvm::Expected<std::map<llvm::StringRef, llvm::JITEvaluatedSymbol, std::less<llvm::StringRef>, std::allocator<std::pair<llvm::StringRef const, llvm::JITEvaluatedSymbol> > > >) ()
   from /home/tim/julia/usr/bin/../lib/libLLVM-12jl.so
#6  0x0000007fb61aaecc in void llvm::detail::UniqueFunctionBase<void, llvm::Expected<std::map<llvm::StringRef, llvm::JITEvaluatedSymbol, std::less<llvm::StringRef>, std::allocator<std::pair<llvm::StringRef const, llvm::JITEvaluatedSymbol> > > > >::CallImpl<llvm::RuntimeDyldImpl::finalizeAsync(std::unique_ptr<llvm::RuntimeDyldImpl, std::default_delete<llvm::RuntimeDyldImpl> >, llvm::unique_function<void (llvm::object::OwningBinary<llvm::object::ObjectFile>, std::unique_ptr<llvm::RuntimeDyld::LoadedObjectInfo, std::default_delete<llvm::RuntimeDyld::LoadedObjectInfo> >, llvm::Error)>, llvm::object::OwningBinary<llvm::object::ObjectFile>, std::unique_ptr<llvm::RuntimeDyld::LoadedObjectInfo, std::default_delete<llvm::RuntimeDyld::LoadedObjectInfo> >)::{lambda(llvm::Expected<std::map<llvm::StringRef, llvm::JITEvaluatedSymbol, std::less<llvm::StringRef>, std::allocator<std::pair<llvm::StringRef const, llvm::JITEvaluatedSymbol> > > >)#1}>(void*, llvm::Expected<std::map<llvm::StringRef, llvm::JITEvaluatedSymbol, std::less<llvm::StringRef>, std::allocator<std::pair<llvm::StringRef const, llvm::JITEvaluatedSymbol> > > >&) ()
   from /home/tim/julia/usr/bin/../lib/libLLVM-12jl.so
#7  0x0000007fb6183df8 in void llvm::detail::UniqueFunctionBase<void, llvm::Expected<llvm::DenseMap<llvm::orc::SymbolStringPtr, llvm::JITEvaluatedSymbol, llvm::DenseMapInfo<llvm::orc::SymbolStringPtr>, llvm::detail::DenseMapPair<llvm::orc::SymbolStringPtr, llvm::JITEvaluatedSymbol> > > >::CallImpl<(anonymous namespace)::JITDylibSearchOrderResolver::lookup(std::set<llvm::StringRef, std::less<llvm::StringRef>, std::allocator<llvm::StringRef> > const&, llvm::unique_function<void (llvm::Expected<std::map<llvm::StringRef, llvm::JITEvaluatedSymbol, std::less<llvm::StringRef>, std::allocator<std::pair<llvm::StringRef const, llvm::JITEvaluatedSymbol> > > >)>)::{lambda(llvm::Expected<llvm::DenseMap<llvm::orc::SymbolStringPtr, llvm::JITEvaluatedSymbol, llvm::DenseMapInfo<llvm::orc::SymbolStringPtr>, llvm::detail::DenseMapPair<llvm::orc::SymbolStringPtr, llvm::JITEvaluatedSymbol> > >)#1}>(void*, llvm::Expected<llvm::DenseMap<llvm::orc::SymbolStringPtr, llvm::JITEvaluatedSymbol, llvm::DenseMapInfo<llvm::orc::SymbolStringPtr>, llvm::detail::DenseMapPair<llvm::orc::SymbolStringPtr, llvm::JITEvaluatedSymbol> > >&) () from /home/tim/julia/usr/bin/../lib/libLLVM-12jl.so
#8  0x0000007fb6119b9c in llvm::orc::AsynchronousSymbolQuery::handleComplete() () from /home/tim/julia/usr/bin/../lib/libLLVM-12jl.so
#9  0x0000007fb61310d4 in llvm::orc::ExecutionSession::OL_completeLookup(std::unique_ptr<llvm::orc::InProgressLookupState, std::default_delete<llvm::orc::InProgressLookupState> >, std::shared_ptr<llvm::orc::AsynchronousSymbolQuery>, std::function<void (llvm::DenseMap<llvm::orc::JITDylib*, llvm::DenseSet<llvm::orc::SymbolStringPtr, llvm::DenseMapInfo<llvm::orc::SymbolStringPtr> >, llvm::DenseMapInfo<llvm::orc::JITDylib*>, llvm::detail::DenseMapPair<llvm::orc::JITDylib*, llvm::DenseSet<llvm::orc::SymbolStringPtr, llvm::DenseMapInfo<llvm::orc::SymbolStringPtr> > > > const&)>) () from /home/tim/julia/usr/bin/../lib/libLLVM-12jl.so
#10 0x0000007fb61315d4 in llvm::orc::InProgressFullLookupState::complete(std::unique_ptr<llvm::orc::InProgressLookupState, std::default_delete<llvm::orc::InProgressLookupState> >) () from /home/tim/julia/usr/bin/../lib/libLLVM-12jl.so
#11 0x0000007fb6122a8c in llvm::orc::ExecutionSession::OL_applyQueryPhase1(std::unique_ptr<llvm::orc::InProgressLookupState, std::default_delete<llvm::orc::InProgressLookupState> >, llvm::Error) ()
   from /home/tim/julia/usr/bin/../lib/libLLVM-12jl.so
#12 0x0000007fb6129874 in llvm::orc::ExecutionSession::lookup(llvm::orc::LookupKind, std::vector<std::pair<llvm::orc::JITDylib*, llvm::orc::JITDylibLookupFlags>, std::allocator<std::pair<llvm::orc::JITDylib*, llvm::orc::JITDylibLookupFlags> > > const&, llvm::orc::SymbolLookupSet, llvm::orc::SymbolState, llvm::unique_function<void (llvm::Expected<llvm::DenseMap<llvm::orc::SymbolStringPtr, llvm::JITEvaluatedSymbol, llvm::DenseMapInfo<llvm::orc::SymbolStringPtr>, llvm::detail::DenseMapPair<llvm::orc::SymbolStringPtr, llvm::JITEvaluatedSymbol> > >)>, std::function<void (llvm::DenseMap<llvm::orc::JITDylib*, llvm::DenseSet<llvm::orc::SymbolStringPtr, llvm::DenseMapInfo<llvm::orc::SymbolStringPtr> >, llvm::DenseMapInfo<llvm::orc::JITDylib*>, llvm::detail::DenseMapPair<llvm::orc::JITDylib*, llvm::DenseSet<llvm::orc::SymbolStringPtr, llvm::DenseMapInfo<llvm::orc::SymbolStringPtr> > > > const&)>) ()
   from /home/tim/julia/usr/bin/../lib/libLLVM-12jl.so
#13 0x0000007fb6181a54 in (anonymous namespace)::JITDylibSearchOrderResolver::lookup(std::set<llvm::StringRef, std::less<llvm::StringRef>, std::allocator<llvm::StringRef> > const&, llvm::unique_function<void (llvm::Expected<std::map<llvm::StringRef, llvm::JITEvaluatedSymbol, std::less<llvm::StringRef>, std::allocator<std::pair<llvm::StringRef const, llvm::JITEvaluatedSymbol> > > >)>) () from /home/tim/julia/usr/bin/../lib/libLLVM-12jl.so
#14 0x0000007fb61ab26c in llvm::RuntimeDyldImpl::finalizeAsync(std::unique_ptr<llvm::RuntimeDyldImpl, std::default_delete<llvm::RuntimeDyldImpl> >, llvm::unique_function<void (llvm::object::OwningBinary<llvm::object::ObjectFile>, std::unique_ptr<llvm::RuntimeDyld::LoadedObjectInfo, std::default_delete<llvm::RuntimeDyld::LoadedObjectInfo> >, llvm::Error)>, llvm::object::OwningBinary<llvm::object::ObjectFile>, std::unique_ptr<llvm::RuntimeDyld::LoadedObjectInfo, std::default_delete<llvm::RuntimeDyld::LoadedObjectInfo> >) () from /home/tim/julia/usr/bin/../lib/libLLVM-12jl.so
#15 0x0000007fb61aba24 in llvm::jitLinkForORC(llvm::object::OwningBinary<llvm::object::ObjectFile>, llvm::RuntimeDyld::MemoryManager&, llvm::JITSymbolResolver&, bool, llvm::unique_function<llvm::Error (llvm::object::ObjectFile const&, llvm::RuntimeDyld::LoadedObjectInfo&, std::map<llvm::StringRef, llvm::JITEvaluatedSymbol, std::less<llvm::StringRef>, std::allocator<std::pair<llvm::StringRef const, llvm::JITEvaluatedSymbol> > >)>, llvm::unique_function<void (llvm::object::OwningBinary<llvm::object::ObjectFile>, std::unique_ptr<llvm::RuntimeDyld::LoadedObjectInfo, std::default_delete<llvm::RuntimeDyld::LoadedObjectInfo> >, llvm::Error)>) () from /home/tim/julia/usr/bin/../lib/libLLVM-12jl.so
#16 0x0000007fb6182150 in llvm::orc::RTDyldObjectLinkingLayer::emit(std::unique_ptr<llvm::orc::MaterializationResponsibility, std::default_delete<llvm::orc::MaterializationResponsibility> >, std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer> >) () from /home/tim/julia/usr/bin/../lib/libLLVM-12jl.so
#17 0x0000007fb6148d68 in llvm::orc::IRCompileLayer::emit(std::unique_ptr<llvm::orc::MaterializationResponsibility, std::default_delete<llvm::orc::MaterializationResponsibility> >, llvm::orc::ThreadSafeModule) ()
   from /home/tim/julia/usr/bin/../lib/libLLVM-12jl.so
#18 0x0000007fb615330c in llvm::orc::BasicIRLayerMaterializationUnit::materialize(std::unique_ptr<llvm::orc::MaterializationResponsibility, std::default_delete<llvm::orc::MaterializationResponsibility> >) ()
   from /home/tim/julia/usr/bin/../lib/libLLVM-12jl.so
#19 0x0000007fb612aac0 in llvm::orc::ExecutionSession::materializeOnCurrentThread(std::unique_ptr<llvm::orc::MaterializationUnit, std::default_delete<llvm::orc::MaterializationUnit> >, std::unique_ptr<llvm::orc::MaterializationResponsibility, std::default_delete<llvm::orc::MaterializationResponsibility> >) () from /home/tim/julia/usr/bin/../lib/libLLVM-12jl.so
#20 0x0000007fb6128df4 in std::_Function_handler<void (std::unique_ptr<llvm::orc::MaterializationUnit, std::default_delete<llvm::orc::MaterializationUnit> >, std::unique_ptr<llvm::orc::MaterializationResponsibility, std::default_delete<llvm::orc::MaterializationResponsibility> >), void (*)(std::unique_ptr<llvm::orc::MaterializationUnit, std::default_delete<llvm::orc::MaterializationUnit> >, std::unique_ptr<llvm::orc::MaterializationResponsibility, std::default_delete<llvm::orc::MaterializationResponsibility> >)>::_M_invoke(std::_Any_data const&, std::unique_ptr<llvm::orc::MaterializationUnit, std::default_delete<llvm::orc::MaterializationUnit> >&&, std::unique_ptr<llvm::orc::MaterializationResponsibility, std::default_delete<llvm::orc::MaterializationResponsibility> >&&) () from /home/tim/julia/usr/bin/../lib/libLLVM-12jl.so
#21 0x0000007fb61295a8 in llvm::orc::ExecutionSession::dispatchOutstandingMUs() () from /home/tim/julia/usr/bin/../lib/libLLVM-12jl.so
#22 0x0000007fb61310dc in llvm::orc::ExecutionSession::OL_completeLookup(std::unique_ptr<llvm::orc::InProgressLookupState, std::default_delete<llvm::orc::InProgressLookupState> >, std::shared_ptr<llvm::orc::AsynchronousSymbolQuery>, std::function<void (llvm::DenseMap<llvm::orc::JITDylib*, llvm::DenseSet<llvm::orc::SymbolStringPtr, llvm::DenseMapInfo<llvm::orc::SymbolStringPtr> >, llvm::DenseMapInfo<llvm::orc::JITDylib*>, llvm::detail::DenseMapPair<llvm::orc::JITDylib*, llvm::DenseSet<llvm::orc::SymbolStringPtr, llvm::DenseMapInfo<llvm::orc::SymbolStringPtr> > > > const&)>) () from /home/tim/julia/usr/bin/../lib/libLLVM-12jl.so
#23 0x0000007fb61315d4 in llvm::orc::InProgressFullLookupState::complete(std::unique_ptr<llvm::orc::InProgressLookupState, std::default_delete<llvm::orc::InProgressLookupState> >) () from /home/tim/julia/usr/bin/../lib/libLLVM-12jl.so
#24 0x0000007fb6122a8c in llvm::orc::ExecutionSession::OL_applyQueryPhase1(std::unique_ptr<llvm::orc::InProgressLookupState, std::default_delete<llvm::orc::InProgressLookupState> >, llvm::Error) ()
   from /home/tim/julia/usr/bin/../lib/libLLVM-12jl.so
---Type <return> to continue, or q <return> to quit---
#25 0x0000007fb6129874 in llvm::orc::ExecutionSession::lookup(llvm::orc::LookupKind, std::vector<std::pair<llvm::orc::JITDylib*, llvm::orc::JITDylibLookupFlags>, std::allocator<std::pair<llvm::orc::JITDylib*, llvm::orc::JITDylibLookupFlags> > > const&, llvm::orc::SymbolLookupSet, llvm::orc::SymbolState, llvm::unique_function<void (llvm::Expected<llvm::DenseMap<llvm::orc::SymbolStringPtr, llvm::JITEvaluatedSymbol, llvm::DenseMapInfo<llvm::orc::SymbolStringPtr>, llvm::detail::DenseMapPair<llvm::orc::SymbolStringPtr, llvm::JITEvaluatedSymbol> > >)>, std::function<void (llvm::DenseMap<llvm::orc::JITDylib*, llvm::DenseSet<llvm::orc::SymbolStringPtr, llvm::DenseMapInfo<llvm::orc::SymbolStringPtr> >, llvm::DenseMapInfo<llvm::orc::JITDylib*>, llvm::detail::DenseMapPair<llvm::orc::JITDylib*, llvm::DenseSet<llvm::orc::SymbolStringPtr, llvm::DenseMapInfo<llvm::orc::SymbolStringPtr> > > > const&)>) ()
   from /home/tim/julia/usr/bin/../lib/libLLVM-12jl.so
#26 0x0000007fb6129f00 in llvm::orc::ExecutionSession::lookup(std::vector<std::pair<llvm::orc::JITDylib*, llvm::orc::JITDylibLookupFlags>, std::allocator<std::pair<llvm::orc::JITDylib*, llvm::orc::JITDylibLookupFlags> > > const&, llvm::orc::SymbolLookupSet const&, llvm::orc::LookupKind, llvm::orc::SymbolState, std::function<void (llvm::DenseMap<llvm::orc::JITDylib*, llvm::DenseSet<llvm::orc::SymbolStringPtr, llvm::DenseMapInfo<llvm::orc::SymbolStringPtr> >, llvm::DenseMapInfo<llvm::orc::JITDylib*>, llvm::detail::DenseMapPair<llvm::orc::JITDylib*, llvm::DenseSet<llvm::orc::SymbolStringPtr, llvm::DenseMapInfo<llvm::orc::SymbolStringPtr> > > > const&)>) ()
   from /home/tim/julia/usr/bin/../lib/libLLVM-12jl.so
#27 0x0000007fb612a25c in llvm::orc::ExecutionSession::lookup(std::vector<std::pair<llvm::orc::JITDylib*, llvm::orc::JITDylibLookupFlags>, std::allocator<std::pair<llvm::orc::JITDylib*, llvm::orc::JITDylibLookupFlags> > > const&, llvm::orc::SymbolStringPtr, llvm::orc::SymbolState) () from /home/tim/julia/usr/bin/../lib/libLLVM-12jl.so
#28 0x0000007fb612a650 in llvm::orc::ExecutionSession::lookup(llvm::ArrayRef<llvm::orc::JITDylib*>, llvm::orc::SymbolStringPtr, llvm::orc::SymbolState) () from /home/tim/julia/usr/bin/../lib/libLLVM-12jl.so
#29 0x0000007fb612a884 in llvm::orc::ExecutionSession::lookup(llvm::ArrayRef<llvm::orc::JITDylib*>, llvm::StringRef, llvm::orc::SymbolState) () from /home/tim/julia/usr/bin/../lib/libLLVM-12jl.so
#30 0x0000007fb78bdb70 in JuliaOJIT::addModule (this=0x5555629390, M=...) at /home/tim/julia/src/jitlayers.cpp:779
#31 0x0000007fb78be6c0 in jl_add_to_ee (m=...) at /home/tim/julia/src/jitlayers.cpp:1058
#32 jl_add_to_ee (M=..., NewExports=..., Queued=..., ToMerge=..., depth=depth@entry=1) at /home/tim/julia/src/jitlayers.cpp:1102
#33 0x0000007fb78bf18c in jl_add_to_ee (NewExports=..., M=...) at /home/tim/julia/src/jitlayers.cpp:1124
#34 _jl_compile_codeinst (codeinst=codeinst@entry=0x7fa93b2830, src=<optimized out>, world=world@entry=29642) at /home/tim/julia/src/jitlayers.cpp:154
#35 0x0000007fb78c0004 in jl_generate_fptr (mi=mi@entry=0x7faa095460, world=world@entry=29642) at /home/tim/julia/src/jitlayers.cpp:350
#36 0x0000007fb782e378 in jl_compile_method_internal (mi=mi@entry=0x7faa095460, world=world@entry=29642) at /home/tim/julia/src/gf.c:1978
#37 0x0000007fb7830b1c in jl_compile_method_internal (world=29642, mi=0x7faa095460) at /home/tim/julia/src/gf.c:2245
#38 _jl_invoke (world=29642, mfunc=<optimized out>, nargs=127, args=0x7fffff8d68, F=0x7fa9ee9e60) at /home/tim/julia/src/gf.c:2237
#39 jl_apply_generic (F=0x7fa9ee9e60, args=0x7fffff8d68, nargs=127) at /home/tim/julia/src/gf.c:2427
#40 0x0000007fb7851fe8 in jl_apply (nargs=2, args=0x7fffff8d60) at /home/tim/julia/src/julia.h:1768
#41 do_call (args=args@entry=0x7fb0e21e38, nargs=nargs@entry=2, s=s@entry=0x7fffff90a0) at /home/tim/julia/src/interpreter.c:125
#42 0x0000007fb7851bb8 in eval_value (e=e@entry=0x7faf125310, s=0x7fffff90a0, s@entry=0x7fffff9030) at /home/tim/julia/src/interpreter.c:214
#43 0x0000007fb7852678 in eval_stmt_value (s=0x7fffff9030, stmt=<optimized out>) at /home/tim/julia/src/interpreter.c:165
#44 eval_body (stmts=<optimized out>, s=s@entry=0x7fffff90a0, ip=2, ip@entry=0, toplevel=toplevel@entry=1) at /home/tim/julia/src/interpreter.c:597
#45 0x0000007fb7853514 in jl_interpret_toplevel_thunk (m=m@entry=0x7fa7d2ae60, src=0x7faa6e7a90) at /home/tim/julia/src/interpreter.c:727
#46 0x0000007fb786f508 in jl_toplevel_eval_flex (m=m@entry=0x7fa7d2ae60, e=<optimized out>, fast=fast@entry=1, expanded=expanded@entry=1) at /home/tim/julia/src/toplevel.c:885
#47 0x0000007fb786fcb0 in jl_eval_module_expr (ex=<optimized out>, parent_module=0x7fb7a46048 <jl_world_counter>) at /home/tim/julia/src/toplevel.c:196
#48 jl_toplevel_eval_flex (m=m@entry=0x7fa636b0d0, e=<optimized out>, fast=fast@entry=1, expanded=expanded@entry=0) at /home/tim/julia/src/toplevel.c:673
#49 0x0000007fb786f83c in jl_toplevel_eval_flex (m=0x7fa636b0d0, m@entry=0x7fb304fa60, e=e@entry=0x73a8, fast=fast@entry=1, expanded=expanded@entry=0) at /home/tim/julia/src/toplevel.c:830
#50 0x0000007fb78710d4 in jl_toplevel_eval (m=m@entry=0x7fb304fa60, v=v@entry=0x73a8) at /home/tim/julia/src/toplevel.c:894
#51 0x0000007fb7871200 in jl_toplevel_eval_in (m=0x7fb304fa60, ex=0x73a8) at /home/tim/julia/src/toplevel.c:944

So this looks like something with our JIT's memory manager. I'm entirely unfamiliar with this code, any pointers @Keno?

[vchuravy]

On what instruction are you segfaulting? What CPU specifically is it?