Errors when loading parallel packages on same machine

[RossBoylan]

See also #16778. That problem involves multiple machines; I get very similar errors using, e.g., -p 4 on a single machine.

Originally I think the packages/cache were a bit out of date, and when I ran in parallel things went wrong and rebuilt everywhere. If they were all rebuilding in the same spot, as they seemed to, it's easy to imagine that would cause problems. I also had a single CPU copy of julia running at the same time (idle) throughout. I was running under ESS.

After that I started up a single cpu version, did a Pkg.update(), and ran my code. Everything seemed OK.

However, I then ran a parallel julia, and continued to get errors. The included code is all in a module and connect to a Postgresql DB to get some data.

Here are excerpts from the sessions, first my initial run

   | | | | | | |/ _' |  |
   | | |_| | | | (_| |  |  Version 0.4.6-pre+28 (2016-04-22 00:59 UTC)
  _/ |\__'_|_|_|\__'_|  |  Commit 022917e (44 days old release-0.4)
 |__/                   |  x86_64-linux-gnu

 WARNING: Terminal not fully functional
 julia>
 julia> nprocs()
 5

 julia> now()
 2016-06-05T00:04:52  # Pacific time US
julia> @everywhere include("trouble.jl")
 INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/Debug.ji for module Debug.
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/PDMats.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/PDMats.ji
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/PDMats.ji
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/PDMats.ji
 WARNING: Module PDMats uuid did not match cache file
   This is likely because module PDMats does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Distributions.ji
 WARNING: Module PDMats uuid did not match cache file
   This is likely because module PDMats does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Distributions.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/StatsFuns.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/StatsFuns.ji
 WARNING: Module PDMats uuid did not match cache file
   This is likely because module PDMats does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Distributions.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/StatsFuns.ji
 WARNING: Module PDMats uuid did not match cache file
   This is likely because module PDMats does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Distributions.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/StatsFuns.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/ArrayViews.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/ArrayViews.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/ArrayViews.ji
 WARNING: replacing module Debug
 WARNING: replacing module Debug
 WARNING: replacing module Debug
 WARNING: replacing module Debug
 WARNING: Module ArrayViews uuid did not match cache file
   This is likely because module ArrayViews does not support  precompilation but is imported by a module that does.
WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/StatsBase.ji
 WARNING: Module ArrayViews uuid did not match cache file
   This is likely because module ArrayViews does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/StatsBase.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/ArrayViews.ji
 WARNING: Module ArrayViews uuid did not match cache file
   This is likely because module ArrayViews does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/StatsBase.ji
 INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/Distributions.ji for module Distributions.
 WARNING: Module ArrayViews uuid did not match cache file
   This is likely because module ArrayViews does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/StatsBase.ji
 INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/PDMats.ji for module PDMats.
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/SortingAlgorithms.ji
 WARNING: Module SortingAlgorithms uuid did not match cache file
   This is likely because module SortingAlgorithms does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/DataFrames.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/DataArrays.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/SortingAlgorithms.ji
 WARNING: Module SortingAlgorithms uuid did not match cache file
   This is likely because module SortingAlgorithms does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/DataFrames.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/SortingAlgorithms.ji
 WARNING: Module SortingAlgorithms uuid did not match cache file
   This is likely because module SortingAlgorithms does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/DataFrames.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/DataArrays.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/DataArrays.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/SortingAlgorithms.ji
 WARNING: Module SortingAlgorithms uuid did not match cache file
   This is likely because module SortingAlgorithms does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/DataFrames.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/DataArrays.ji
 INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/StatsFuns.ji for module StatsFuns.
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/GZip.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Docile.ji
WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/GZip.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Docile.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/GZip.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Docile.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/GZip.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Docile.ji
 INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/ArrayViews.ji for module ArrayViews.
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/JSON.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/ImmutableArrays.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/JSON.ji
 INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/StatsBase.ji for module StatsBase.
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/JSON.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/ImmutableArrays.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/JSON.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/ImmutableArrays.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/ImmutableArrays.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Gadfly.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Codecs.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/FixedPointNumbers.ji
 WARNING: Module FixedPointNumbers uuid did not match cache file
   This is likely because module FixedPointNumbers does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Colors.ji
 WARNING: Module FixedPointNumbers uuid did not match cache file
   This is likely because module FixedPointNumbers does not support  precompilation but is imported by a module that does.
WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/ColorTypes.ji
 WARNING: both DataArrays and StatsBase export "autocor"; uses of it in module DataFrames must be qualified
 WARNING: both DataArrays and StatsBase export "inverse_rle"; uses of it in module DataFrames must be qualified
 WARNING: both DataArrays and StatsBase export "rle"; uses of it in module DataFrames must be qualified
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Gadfly.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Codecs.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/FixedPointNumbers.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Gadfly.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Codecs.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/FixedPointNumbers.ji
 WARNING: Module FixedPointNumbers uuid did not match cache file
   This is likely because module FixedPointNumbers does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Colors.ji
 WARNING: Module FixedPointNumbers uuid did not match cache file
   This is likely because module FixedPointNumbers does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/ColorTypes.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Gadfly.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Codecs.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/FixedPointNumbers.ji
 WARNING: Module FixedPointNumbers uuid did not match cache file
   This is likely because module FixedPointNumbers does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Colors.ji
 WARNING: Module FixedPointNumbers uuid did not match cache file
   This is likely because module FixedPointNumbers does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/ColorTypes.ji
 WARNING: Module FixedPointNumbers uuid did not match cache file
   This is likely because module FixedPointNumbers does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Colors.ji
 WARNING: Module FixedPointNumbers uuid did not match cache file
   This is likely because module FixedPointNumbers does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/ColorTypes.ji
 WARNING: both DataArrays and StatsBase export "autocor"; uses of it in module DataFrames must be qualified
 WARNING: both DataArrays and StatsBase export "inverse_rle"; uses of it in module DataFrames must be qualified
 WARNING: both DataArrays and StatsBase export "rle"; uses of it in module DataFrames must be qualified
 WARNING: both DataArrays and StatsBase export "autocor"; uses of it in module DataFrames must be qualified
 WARNING: both DataArrays and StatsBase export "inverse_rle"; uses of it in module DataFrames must be qualified
 WARNING: both DataArrays and StatsBase export "rle"; uses of it in module DataFrames must be qualified
 WARNING: both DataArrays and StatsBase export "autocor"; uses of it in module DataFrames must be qualified
 WARNING: both DataArrays and StatsBase export "inverse_rle"; uses of it in module DataFrames must be qualified
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Iterators.ji
 WARNING: Module Iterators uuid did not match cache file
   This is likely because module Iterators does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Compose.ji
 WARNING: Module StatsFuns uuid did not match cache file
   This is likely because module StatsFuns does not support  precompilation but is imported by a module that does.
 WARNING: Module StatsFuns uuid did not match cache file
   This is likely because module StatsFuns does not support  precompilation but is imported by a module that does.
 WARNING: Module StatsFuns uuid did not match cache file
   This is likely because module StatsFuns does not support  precompilation but is imported by a module that does.
 WARNING: node state is inconsistent: node 2 failed to load cache from /home/ross/.julia/lib/v0.4/Distributions.ji
 WARNING: Module StatsFuns uuid did not match cache file
   This is likely because module StatsFuns does not support  precompilation but is imported by a module that does.
 WARNING: node state is inconsistent: node 3 failed to load cache from /home/ross/.julia/lib/v0.4/Distributions.ji
 WARNING: node state is inconsistent: node 4 failed to load cache from /home/ross/.julia/lib/v0.4/Distributions.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Iterators.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Iterators.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Iterators.ji
 WARNING: node state is inconsistent: node 5 failed to load cache from /home/ross/.julia/lib/v0.4/Distributions.ji
 WARNING: Module Iterators uuid did not match cache file
   This is likely because module Iterators does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Compose.ji
 WARNING: Module Iterators uuid did not match cache file
   This is likely because module Iterators does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Compose.ji
 WARNING: Module Iterators uuid did not match cache file
   This is likely because module Iterators does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Compose.ji
 WARNING: replacing module FastGaussQuadrature
 WARNING: replacing module FastGaussQuadrature
 WARNING: replacing module FastGaussQuadrature
 WARNING: replacing module FastGaussQuadrature
 WARNING: Module FixedPointNumbers uuid did not match cache file
   This is likely because module FixedPointNumbers does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Cairo.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Graphics.ji
 INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/DataFrames.ji for module DataFrames.
 WARNING: Module FixedPointNumbers uuid did not match cache file
   This is likely because module FixedPointNumbers does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Cairo.ji
 WARNING: Module FixedPointNumbers uuid did not match cache file
   This is likely because module FixedPointNumbers does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Cairo.ji
 WARNING: Module FixedPointNumbers uuid did not match cache file
   This is likely because module FixedPointNumbers does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Cairo.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
WARNING: both DataArrays and StatsBase export "rle"; uses of it in module DataFrames must be qualified
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Graphics.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Graphics.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Graphics.ji
 INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/DataArrays.ji for module DataArrays.
 WARNING: Module Iterators uuid did not match cache file
   This is likely because module Iterators does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Showoff.ji
 WARNING: Module Iterators uuid did not match cache file
   This is likely because module Iterators does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Showoff.ji
 WARNING: Module Iterators uuid did not match cache file
   This is likely because module Iterators does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Showoff.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Hexagons.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Hexagons.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Hexagons.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Contour.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/FixedSizeArrays.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Contour.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Contour.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/FixedSizeArrays.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/FixedSizeArrays.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Loess.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Loess.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Loess.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/WoodburyMatrices.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/WoodburyMatrices.ji
 WARNING: Module WoodburyMatrices uuid did not match cache file
   This is likely because module WoodburyMatrices does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/KernelDensity.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Calculus.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/WoodburyMatrices.ji
 WARNING: Module WoodburyMatrices uuid did not match cache file
   This is likely because module WoodburyMatrices does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/KernelDensity.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Calculus.ji
 WARNING: Module WoodburyMatrices uuid did not match cache file
   This is likely because module WoodburyMatrices does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/KernelDensity.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Calculus.ji
 WARNING: Module Calculus uuid did not match cache file
   This is likely because module Calculus does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/DualNumbers.ji
 WARNING: Module Calculus uuid did not match cache file
   This is likely because module Calculus does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/DualNumbers.ji
 WARNING: Module Calculus uuid did not match cache file
   This is likely because module Calculus does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/DualNumbers.ji
 WARNING: Module DualNumbers uuid did not match cache file
   This is likely because module DualNumbers does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Optim.ji
 WARNING: Module DualNumbers uuid did not match cache file
   This is likely because module DualNumbers does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Optim.ji
 WARNING: Module DualNumbers uuid did not match cache file
   This is likely because module DualNumbers does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Optim.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Grid.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Grid.ji
 WARNING: Module Compat uuid did not match cache file
   This is likely because module Compat does not support  precompilation but is imported by a module that does.
 WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Grid.ji
 INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/GZip.ji for module GZip.
 INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/SortingAlgorithms.ji for module SortingAlgorithms.
 INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/Docile.ji for module Docile.
 WARNING: Module Docile uuid did not match cache file
   This is likely because module Docile does not support  precompilation but is imported by a module that does.
 WARNING: Module Docile uuid did not match cache file
   This is likely because module Docile does not support  precompilation but is imported by a module that does.
 WARNING: Module Docile uuid did not match cache file
   This is likely because module Docile does not support  precompilation but is imported by a module that does.
WARNING: node state is inconsistent: node 2 failed to load cache from /home/ross/.julia/lib/v0.4/DataFrames.ji
 WARNING: Module Docile uuid did not match cache file
   This is likely because module Docile does not support  precompilation but is imported by a module that does.
 WARNING: node state is inconsistent: node 3 failed to load cache from /home/ross/.julia/lib/v0.4/DataFrames.ji
 WARNING: node state is inconsistent: node 4 failed to load cache from /home/ross/.julia/lib/v0.4/DataFrames.ji
 WARNING: node state is inconsistent: node 5 failed to load cache from /home/ross/.julia/lib/v0.4/DataFrames.ji
 WARNING: replacing module DBI
 WARNING: replacing module DBI
 WARNING: replacing module DBI
 WARNING: replacing module DBI
 INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/JSON.ji for module JSON.
 WARNING: replacing module PostgreSQL
 WARNING: replacing module PostgreSQL
 WARNING: replacing module PostgreSQL
 WARNING: replacing module PostgreSQL
 INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/ImmutableArrays.ji for module ImmutableArrays.
 WARNING: replacing module ImmutableArrays
 WARNING: replacing module ImmutableArrays
 WARNING: replacing module ImmutableArrays
 WARNING: replacing module ImmutableArrays
 INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/Gadfly.ji for module Gadfly.
 INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/Codecs.ji for module Codecs.
 INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/FixedPointNumbers.ji for module FixedPointNumbers.
 INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/Colors.ji for module Colors.
 INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/ColorTypes.ji for module ColorTypes.
 INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/Iterators.ji for module Iterators.
 INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/Compose.ji for module Compose.
 INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/Cairo.ji for module Cairo.
 INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/Graphics.ji for module Graphics.
 INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/Showoff.ji for module Showoff.
 INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/Hexagons.ji for module Hexagons.
 INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/Contour.ji for module Contour.
 INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/FixedSizeArrays.ji for module FixedSizeArrays.
 INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/Loess.ji for module Loess.
 INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/WoodburyMatrices.ji for module WoodburyMatrices.
 INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/KernelDensity.ji for module KernelDensity.
 INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/Calculus.ji for module Calculus.
 INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/DualNumbers.ji for module DualNumbers.
 INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/Optim.ji for module Optim.
 INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/Grid.ji for module Grid.
 WARNING: Module StatsFuns uuid did not match cache file
   This is likely because module StatsFuns does not support  precompilation but is imported by a module that does.
 WARNING: Module StatsFuns uuid did not match cache file
   This is likely because module StatsFuns does not support  precompilation but is imported by a module that does.
 WARNING: node state is inconsistent: node 2 failed to load cache from /home/ross/.julia/lib/v0.4/Gadfly.ji
 WARNING: node state is inconsistent: node 3 failed to load cache from /home/ross/.julia/lib/v0.4/Gadfly.ji
 WARNING: Module StatsFuns uuid did not match cache file
   This is likely because module StatsFuns does not support  precompilation but is imported by a module that does.
 WARNING: node state is inconsistent: node 4 failed to load cache from /home/ross/.julia/lib/v0.4/Gadfly.ji
 WARNING: Module StatsFuns uuid did not match cache file
   This is likely because module StatsFuns does not support  precompilation but is imported by a module that does.
 WARNING: node state is inconsistent: node 5 failed to load cache from /home/ross/.julia/lib/v0.4/Gadfly.ji
 ERROR: On worker 5:
 LoadError: LoadError: LoadError: LoadError: UndefVarError: CairoSurface not defined
  in include_string at loading.jl:282
  in include_from_node1 at ./loading.jl:323
  in include_string at loading.jl:282
  in include_from_node1 at ./loading.jl:323
  in require at ./loading.jl:259
  in include_string at loading.jl:282
  in include_from_node1 at ./loading.jl:323
  in require at ./loading.jl:259
  in include_string at loading.jl:282
  in include_from_node1 at ./loading.jl:323
  in eval at ./sysimg.jl:14
  in anonymous at multi.jl:1394
  in anonymous at multi.jl:923
  in run_work_thunk at multi.jl:661
  [inlined code] from multi.jl:923
  in anonymous at task.jl:63
 while loading /home/ross/.julia/v0.4/Compose/src/cairo_backends.jl, in expression starting on line 45
 while loading /home/ross/.julia/v0.4/Compose/src/Compose.jl, in expression starting on line 17
 while loading /home/ross/.julia/v0.4/Gadfly/src/Gadfly.jl, in expression starting on line 9
 while loading /home/ross/PCORI/trouble.jl, in expression starting on line 10
  in remotecall_fetch at multi.jl:747
  in remotecall_fetch at multi.jl:750
  in anonymous at multi.jl:1396
  in sync_end at ./task.jl:413
  in anonymous at multi.jl:1405

 julia> include("/home/ross/PCORI/trouble.jl")

 WARNING: replacing module RB
 RB

 julia> @everywhere include("trouble.jl")
 WARNING: replacing module RB
 WARNING: replacing module RB
 WARNING: replacing module RB
 WARNING: replacing module RB
 WARNING: replacing module RB
 WARNING: could not import Gadfly.plot into RB

And after the cleanup with one CPU:

 julia> nprocs()
 4
julia> @everywhere include("trouble.jl")
 WARNING: replacing module Debug
 WARNING: replacing module Debug
 WARNING: replacing module Debug
 WARNING: replacing module Distributions
 WARNING: replacing module Distributions
 WARNING: replacing module Distributions
 WARNING: replacing module FastGaussQuadrature
 WARNING: replacing module FastGaussQuadrature
 WARNING: replacing module FastGaussQuadrature
 WARNING: replacing module DataFrames
 WARNING: Method definition dump(Base.IO, DataArrays.AbstractDataArray{T<:Any, 1}, Int64, Any) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:358 overwritten in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:358.
 WARNING: Method definition vcat(Array{Void, 1}) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:677 overwritten in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:677.
 WARNING: Method definition describe(AbstractArray) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:406 overwritten in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:406.
 WARNING: Method definition describe(Any, AbstractArray{#T<:Number, N<:Any}) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:408 overwritten in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:408.
 WARNING: Method definition describe(Any, AbstractArray{#T<:Any, N<:Any}) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:425 overwritten in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:425.
 WARNING: replacing module DataFrames
 WARNING: Method definition dump(Base.IO, DataArrays.AbstractDataArray{T<:Any, 1}, Int64, Any) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:358 overwritten in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:358.
 WARNING: Method definition vcat(Array{Void, 1}) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:677 overwritten in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:677.
 WARNING: Method definition describe(AbstractArray) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:406 overwritten in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:406.
 WARNING: Method definition describe(Any, AbstractArray{#T<:Number, N<:Any}) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:408 overwritten in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:408.
 WARNING: Method definition describe(Any, AbstractArray{#T<:Any, N<:Any}) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:425 overwritten in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:425.
 WARNING: replacing module DataFrames
 WARNING: Method definition dump(Base.IO, DataArrays.AbstractDataArray{T<:Any, 1}, Int64, Any) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:358 overwritten in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:358.
 WARNING: Method definition vcat(Array{Void, 1}) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:677 overwritten in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:677.
 WARNING: Method definition describe(AbstractArray) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:406 overwritten in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:406.
 WARNING: Method definition describe(Any, AbstractArray{#T<:Number, N<:Any}) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:408 overwritten in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:408.
 WARNING: Method definition describe(Any, AbstractArray{#T<:Any, N<:Any}) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:425 overwritten in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:425.
 WARNING: replacing module DBI
 WARNING: replacing module DBI
 WARNING: replacing module DBI
 WARNING: replacing module PostgreSQL
 WARNING: replacing module PostgreSQL
 WARNING: replacing module PostgreSQL
 WARNING: replacing module ImmutableArrays
 WARNING: replacing module ImmutableArrays
 WARNING: replacing module ImmutableArrays
 WARNING: replacing module Gadfly
 WARNING: New definition
     chain(Gadfly.Data...) at /home/ross/.julia/v0.4/Gadfly/src/data.jl:57
 is ambiguous with:
     chain(Gadfly.Data...) at /home/ross/.julia/v0.4/Gadfly/src/data.jl:57.
 To fix, define
     chain()
 before the new definition.
 WARNING: replacing module Gadfly
 WARNING: New definition
     chain(Gadfly.Data...) at /home/ross/.julia/v0.4/Gadfly/src/data.jl:57
 is ambiguous with:
     chain(Gadfly.Data...) at /home/ross/.julia/v0.4/Gadfly/src/data.jl:57.
 To fix, define
     chain()
 before the new definition.
 WARNING: replacing module Gadfly
 WARNING: New definition
     chain(Gadfly.Data...) at /home/ross/.julia/v0.4/Gadfly/src/data.jl:57
is ambiguous with:
     chain(Gadfly.Data...) at /home/ross/.julia/v0.4/Gadfly/src/data.jl:57.
 To fix, define
     chain()
 before the new definition.
 ERROR: On worker 2:
 LoadError: MethodError: `prepare` has no method matching prepare(::PostgreSQL.PostgresDatabaseHandle, ::ASCIIString)
 Closest candidates are:
   prepare(::PostgreSQL.PostgresDatabaseHandle, ::AbstractString)
   prepare(::DBI.DatabaseHandle, ::AbstractString)
  in include_string at loading.jl:282
  in include_from_node1 at ./loading.jl:323
  in eval at ./sysimg.jl:14
  in anonymous at multi.jl:1394
  in anonymous at multi.jl:923
  in run_work_thunk at multi.jl:661
  [inlined code] from multi.jl:923
  in anonymous at task.jl:63
 while loading /home/ross/PCORI/trouble.jl, in expression starting on line 15
  in remotecall_fetch at multi.jl:747
  in remotecall_fetch at multi.jl:750
  in anonymous at multi.jl:1396
  in sync_end at ./task.jl:413
  in anonymous at multi.jl:1405

[tkelman]

See also #13684. You should probably load everything and do recompilation in serial first, before trying to load that code on parallel workers.

[RossBoylan]

Thanks for the reference. Note that the second log in my original report was after I loaded and recompiled in serial; there were still problems.

[RossBoylan]

This seems to have been closed with the message

The binary build on each client needs to be the same, not just the source checkout.

But the bug title and the very first line of my report says that these errors happened while running on a single machine.

So I think this bug should be re-opened.

[tkelman]

I'd look into what's causing the module replacement and method overwrite warnings, resolve those and see if you still have an issue.

[RossBoylan]

How?

The warnings only occur in parallel mode; when I ran on a single process immediately before there were no complaints (that is, once I did the package update).

[RossBoylan]

If the needs-more-info tag is directed at me, I don't understand what more info I should provide.

[tkelman]

Self-contained code that allows others to reproduce the problem.

[RossBoylan]

The errors appear to concern that state of my package cache; are you saying I should upload the cache?

The sequence that led to these problems began running parallel jobs when the cache needed to be updated, and reproducing that would require an update to the external packages. I suppose with some fiddling I could create a local "upstream" package and tweak it.

Apart from the question of origins, I'm currently in this weird state in which Pkg.update() runs fine as a single process, but loading in parallel fails.
Ross

---

From: Tony Kelman [[email protected]]
Sent: Thursday, June 16, 2016 10:25 AM
To: JuliaLang/julia
Cc: Boylan, Ross; Author
Subject: Re: [JuliaLang/julia] Errors when loading parallel packages on same machine (#16788)

Self-contained code that allows others to reproduce the problem.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com//issues/16788#issuecomment-226554689, or mute the threadhttps://github.com/notifications/unsubscribe/ACZgMkzakr_xNoDl51u6VvR87K3Fd8kUks5qMYcmgaJpZM4IvMiD.

[wildart]

Pkg.update is not designed to run in parallel. Running it in parallel mode could result in unidentified state of the global package cache.

[tkelman]

Sounds like you need to clear out your cache, have it rebuild in serial, and if the problems go away there's not really an actionable bug right now, other than perhaps the feature request of implementing a package-cache lock file.

[RossBoylan]

How do I clear the cache? I don't see a command in http://docs.julialang.org/en/release-0.4/manual/packages/. Do I just deleted the whole ~/.julia/v0.4, or maybe ~/julia/v0.4/.cache and the directory it points to?

BTW, I did not explicitly run Pkg.update() in parallel, but, as the first transcript in my original report shows, some kind of update operation was triggered when I loaded my module in parallel. I'm guessing it was basically a Pkg.update().

[tkelman]

delete everything under ~/.julia/lib/v0.4

[RossBoylan]

~/.julia/lib/v0.4/.cache is a symlink to ~/.julia/.cache. Should I delete ~/.julia/.cache as well, or perhaps the material underneath it? I'd hate to delete the cache without actually deleting it:)

[tkelman]

.cache is actually a git cache, not what we're referring to here, and you shouldn't need to touch it - we mean just the .ji package precompilation "cache," sorry for the unclear wording

[RossBoylan]

Short version: clearing the cache didn't help. The following code is sufficient for me to get errors in parallel on one machine, even though I get no errors when not in parallel:

module RB

using Debug
using Distributions
using FastGaussQuadrature
using DataFrames
using DBI
using PostgreSQL
using ImmutableArrays
using Gadfly

import Gadfly.plot
end

Here are the errors:

 julia> nprocs()
 4

!julia> @everywhere include("trouble2.jl")
 WARNING: replacing module Debug
 WARNING: replacing module Debug
 WARNING: replacing module Debug
 WARNING: replacing module Distributions
 WARNING: replacing module Distributions
 WARNING: replacing module Distributions
 WARNING: replacing module FastGaussQuadrature
 WARNING: replacing module FastGaussQuadrature
 WARNING: replacing module FastGaussQuadrature
 WARNING: replacing module DataFrames
 WARNING: replacing module DataFrames
 WARNING: Method definition vcat(Array{Void, 1}) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:677 overwritten i\
 n module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:677.
 WARNING: Method definition dump(Base.IO, DataArrays.AbstractDataArray{T<:Any, 1}, Int64, Any) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractda\
 taframe/abstractdataframe.jl:358 overwritten in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:358.
 WARNING: Method definition describe(AbstractArray) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:406 overwritte\
 n in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:406.
 WARNING: Method definition describe(Any, AbstractArray{#T<:Number, N<:Any}) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractda\
 taframe.jl:408 overwritten in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:408.
 WARNING: Method definition describe(Any, AbstractArray{#T<:Any, N<:Any}) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataf\
 rame.jl:425 overwritten in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:425.
 WARNING: Method definition vcat(Array{Void, 1}) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:677 overwritten i\
 n module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:677.
 WARNING: Method definition dump(Base.IO, DataArrays.AbstractDataArray{T<:Any, 1}, Int64, Any) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractda\
 taframe/abstractdataframe.jl:358 overwritten in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:358.
 WARNING: Method definition describe(AbstractArray) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:406 overwritte\
 n in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:406.
 WARNING: Method definition describe(Any, AbstractArray{#T<:Number, N<:Any}) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractda\
 taframe.jl:408 overwritten in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:408.
 WARNING: Method definition describe(Any, AbstractArray{#T<:Any, N<:Any}) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataf\
 rame.jl:425 overwritten in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:425.
 WARNING: replacing module DataFrames
 WARNING: Method definition vcat(Array{Void, 1}) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:677 overwritten i\
 n module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:677.
 WARNING: Method definition dump(Base.IO, DataArrays.AbstractDataArray{T<:Any, 1}, Int64, Any) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractda\
 taframe/abstractdataframe.jl:358 overwritten in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:358.
 WARNING: Method definition describe(AbstractArray) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:406 overwritte\
 n in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:406.
 WARNING: Method definition describe(Any, AbstractArray{#T<:Number, N<:Any}) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractda\
 taframe.jl:408 overwritten in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:408.
 WARNING: Method definition describe(Any, AbstractArray{#T<:Any, N<:Any}) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataf\
 rame.jl:425 overwritten in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:425.
 WARNING: replacing module DBI
 WARNING: replacing module DBI
 WARNING: replacing module DBI
 WARNING: replacing module PostgreSQL
 WARNING: replacing module PostgreSQL
 WARNING: replacing module PostgreSQL
 WARNING: replacing module ImmutableArrays
 WARNING: replacing module ImmutableArrays
 WARNING: replacing module ImmutableArrays
 WARNING: replacing module Gadfly
 WARNING: replacing module Gadfly
 WARNING: replacing module Gadfly
 WARNING: New definition
     chain(Gadfly.Data...) at /home/ross/.julia/v0.4/Gadfly/src/data.jl:57
 is ambiguous with:
     chain(Gadfly.Data...) at /home/ross/.julia/v0.4/Gadfly/src/data.jl:57.
 To fix, define
     chain()
 before the new definition.
 WARNING: New definition
     chain(Gadfly.Data...) at /home/ross/.julia/v0.4/Gadfly/src/data.jl:57
 is ambiguous with:
     chain(Gadfly.Data...) at /home/ross/.julia/v0.4/Gadfly/src/data.jl:57.
 To fix, define
     chain()
 before the new definition.
 WARNING: New definition
     chain(Gadfly.Data...) at /home/ross/.julia/v0.4/Gadfly/src/data.jl:57
 is ambiguous with:
     chain(Gadfly.Data...) at /home/ross/.julia/v0.4/Gadfly/src/data.jl:57.
 To fix, define
     chain()
 before the new definition.

History

First I deleted everything under ~/.julia/v0.4/ and issued Pkg.add to get the packages back. I did Pkg.update() and load of my original trouble.jl in serial mode and everything was fine. Then I started a parallel version, did @everywhere include, and got errors.

Then I realized I had deleted the wrong directory, and deleted everything under ~/.julia/lib/v0.4/. There were no julia's running at the time. I repeated the single user load (except using the stuff trouble2.jl shown above); no error messages. Then I did the parallel load reported above.

[tkelman]

Thanks. I only see warnings there?

[RossBoylan]

Correct. I noticed that and thought perhaps everything was OK. But when I load the real code, which gets some data from a database, I get

julia> @everywhere include("trouble.jl")
 WARNING: replacing module Debug
 WARNING: replacing module Debug
 WARNING: replacing module Debug
 WARNING: replacing module Debug
 WARNING: replacing module Distributions
 WARNING: replacing module Distributions
 WARNING: replacing module Distributions
 WARNING: replacing module Distributions
 WARNING: replacing module FastGaussQuadrature
 WARNING: replacing module FastGaussQuadrature
 WARNING: replacing module FastGaussQuadrature
 WARNING: replacing module FastGaussQuadrature
 ERROR: On worker 3:
 LoadError: LoadError: LoadError: UndefVarError: Postgres not defined
  in include_string at loading.jl:282
  in include_from_node1 at ./loading.jl:323
  in include_string at loading.jl:282
  in include_from_node1 at ./loading.jl:323
  in require at ./loading.jl:259
  in include_string at loading.jl:282
  in include_from_node1 at ./loading.jl:323
  in eval at ./sysimg.jl:14
  in anonymous at multi.jl:1394
  in anonymous at multi.jl:923
  in run_work_thunk at multi.jl:661
  [inlined code] from multi.jl:923
  in anonymous at task.jl:63
 while loading /home/ross/.julia/v0.4/PostgreSQL/src/dbi_impl.jl, in expression starting on line 3
 while loading /home/ross/.julia/v0.4/PostgreSQL/src/PostgreSQL.jl, in expression starting on line 17
 while loading /home/ross/PCORI/trouble.jl, in expression starting on line 8
  in remotecall_fetch at multi.jl:747
  in remotecall_fetch at multi.jl:750
  in anonymous at multi.jl:1396

 ...and 1 other exceptions.

  in sync_end at ./task.jl:413
  in anonymous at multi.jl:1405

The extra code is

conn = connect(Postgres, "localhost", "pcori", "xxx", "pcori")
stmt = prepare(conn, "SELECT isim::int, id::int, x::int, t::double precision, y::int, regular::int, censor::int, wait::double precision from obs2619 where isim=1")
result = execute(stmt)
# my previous attempt used fetchdf(stmt)                                                                                                                                
obs = fetchdf(result)

stmt = prepare(conn, "SELECT iter::int,  negloglike::double precision, beta_0::double precision, beta_g::double precision, beta_t::double precision, beta_i::double pre\
cision, lnsigma_0::double precision, lnsigma_t::double precision, atanhrho::double precision, delta_0::double precision, delta_y ::double precision, lnnu::double preci\
sion from iter where  iterset=2")
result = execute(stmt)
iter = fetchdf(result)
finish(stmt)

disconnect(conn)

although I note the line number in the traceback refers to line 8, using PostgreSQL, which is in the test code I sent.

[RossBoylan]

The using directives and the calls such as connect(Postgres, "localhost", "pcori", "xxx", "pcori") are within the RB module, so I don't see how bindings in Main matter. But perhaps I misunderstand how using works. Do I need to qualify references to the module, PostgreSQL in the case of the last error above, explicitly? Use some other kind of import method?

[RossBoylan]

Regarding the warning messages displayed by my simple test code: does the fact that I don't see them in serial model mean they don't occur, or could this just be a difference in whether warnings are displayed in parallel vs serial mode?

[RossBoylan]

JuliaLang/Distributed.jl#20 and related bugs suggested trying an additional include just in the main process. This eliminated all apparent problems:

# trouble.jl is the header code given above plus additional substantive code
 julia> include("/home/ross/PCORI/trouble.jl")

 RB

 julia> @everywhere include("trouble.jl")
 WARNING: replacing module RB
# and nothing else

I'm delighted to be able to run parallel code.

I remain baffled by the behavior. For example, why are the errors and warnings from the child processes affected by what I do in the main process?

Outstanding items:

1. protection against multiple updates to the same directory. This affects parallel runs on one machine (my case), multiple independent runs on one machine (me also when I had a couple of julia interactive shells open), and parallel runs on multiple machines sharing a common file system. Note that "updates" includes both explicit calls to Pkg.update() and the implicit updates that occur when reading in code like my sample code, i.e., include a file that has using directives.
2. using loads modules on workers but does not put exported bindings in Main Distributed.jl#20 and related
3. Other? It's not clear to me if the previous 2 items account for all the misbehavior I've seen.

[wildart]

When you develop a concurrent application you should account for various race conditions in your code. It's a bit naive to expect single process application to work as multiprocess one without any modification. Even though the Julia language provide a friendly framework for multiprocessor computations that eliminates thread-level races, process-level data races could occur when data is modified and accessed simultaneously by different processes. Always check your code and packages so they are correctly work in multiprocessor environments.

[RossBoylan]

julia regenerates the cache somewhat behind the back of user code; it seems to me desirable that it do so in a way that is parallel-safe.

[StefanKarpinski]

We probably should do some file locking here. The main issue is actually a cross-platform implementation and API for file locking: #7176. It still doesn't seem like we'll be able to rely on libuv for this, so someone would need to implement it in Julia.

[wildart]

I think if we move from a in-file storage to a directory storage which would contain manifests of installed packages the parallel installation can be easily solved.

[RossBoylan]

Another scenario: multiple architectures with a shared home directory.
Not my original problem, or even an anticipated scenario for me, but maybe something to consider.

[mauro3]

I bumped into this too running parallel scripts on a single machine. An easy way to get to the repeated warnings is including a file containing, e.g.:

import StatsBase

then do

$ julia -p2
               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: https://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.6.1 (2017-10-24 22:15 UTC)
 _/ |\__'_|_|_|\__'_|  |  
|__/                   |  x86_64-pc-linux-gnu

julia> @everywhere begin
           include("tmp-load.jl")
       end
WARNING: Method definition quantile(AbstractArray{#s26, N} where N where #s26<:Real) in module StatsBase at /home/mauro/.julia/v0.6/StatsBase/src/scalarstats.jl:191 overwritten in module StatsBase at /home/mauro/.julia/v0.6/StatsBase/src/scalarstats.jl:191.                                                                                                                                                                                                                            
WARNING: Method definition stdm(AbstractArray{T, N} where N where T<:Real, AbstractArray{T, N} where N where T<:Real, Int64) in module StatsBase at /home/mauro/.julia/v0.6/StatsBase/src/moments.jl:141 overwritten in module StatsBase at /home/mauro/.julia/v0.6/StatsBase/src/moments.jl:141.
WARNING: Method definition #stdm(Array{Any, 1}, typeof(Base.stdm), AbstractArray{T, N} where N where T<:Real, AbstractArray{T, N} where N where T<:Real, Int64) in module StatsBase overwritten in module StatsBase.                          
WARNING: replacing module StatsBase.                                                                                                                                                                                                          
WARNING: Method definition quantile(AbstractArray{#s26, N} where N where #s26<:Real) in module StatsBase at /home/mauro/.julia/v0.6/StatsBase/src/scalarstats.jl:191 overwritten in module StatsBase at /home/mauro/.julia/v0.6/StatsBase/src/scalarstats.jl:191.
WARNING: Method definition stdm(AbstractArray{T, N} where N where T<:Real, AbstractArray{T, N} where N where T<:Real, Int64) in module StatsBase at /home/mauro/.julia/v0.6/StatsBase/src/moments.jl:141 overwritten in module StatsBase at /home/mauro/.julia/v0.6/StatsBase/src/moments.jl:141.
WARNING: Method definition #stdm(Array{Any, 1}, typeof(Base.stdm), AbstractArray{T, N} where N where T<:Real, AbstractArray{T, N} where N where T<:Real, Int64) in module StatsBase overwritten in module StatsBase.                          
WARNING: replacing module StatsBase.                                                                                                                                                                                                          

I also got the error reported above, but I don't have a MWE. I also get the UndefVarError:

## lots of warnings here: Method-redefinitions and replacing modules
...
WARNING: Method definition pygui(Bool) in module PyPlot at /home/werderm/.julia/v0.6/PyPlot/src/init.jl:214 overwritten in module PyPlot at /home/werderm/.julia/v0.6/PyPlot/src/init.jl:214.
WARNING: replacing module PyPlot.
ERROR: LoadError: On worker 2:
UndefVarError: iv not defined
eval at ./boot.jl:235
deserialize_global_from_main at ./distributed/clusterserialize.jl:158
foreach at ./abstractarray.jl:1733
deserialize at ./distributed/clusterserialize.jl:56
handle_deserialize at ./serialize.jl:726
deserialize at ./serialize.jl:637
handle_deserialize at ./serialize.jl:681
deserialize at ./serialize.jl:637
deserialize_datatype at ./serialize.jl:997
handle_deserialize at ./serialize.jl:677
deserialize at ./serialize.jl:637
handle_deserialize at ./serialize.jl:684
deserialize_msg at ./distributed/messages.jl:98
message_handler_loop at ./distributed/process_messages.jl:161
process_tcp_streams at ./distributed/process_messages.jl:118
JuliaLang/julia#99 at ./event.jl:73
Stacktrace:
 [1] #remotecall_fetch#141(::Array{Any,1}, ::Function, ::Function, ::Base.Distributed.Worker, ::Base.Distributed.RRID, ::Vararg{Any,N} where N) at ./distributed/remotecall.jl:354
 [2] remotecall_fetch(::Function, ::Base.Distributed.Worker, ::Base.Distributed.RRID, ::Vararg{Any,N} where N) at ./distributed/remotecall.jl:346
 [3] #remotecall_fetch#144(::Array{Any,1}, ::Function, ::Function, ::Int64, ::Base.Distributed.RRID, ::Vararg{Any,N} where N) at ./distributed/remotecall.jl:367
 [4] remotecall_fetch(::Function, ::Int64, ::Base.Distributed.RRID, ::Vararg{Any,N} where N) at ./distributed/remotecall.jl:367
 [5] call_on_owner(::Function, ::Future, ::Int64, ::Vararg{Int64,N} where N) at ./distributed/remotecall.jl:440
 [6] wait(::Future) at ./distributed/remotecall.jl:455
 [7] sync_end() at ./task.jl:274
 [8] include_from_node1(::String) at ./loading.jl:576
 [9] include(::String) at ./sysimg.jl:14
...

Interestingly & luckily, running the script again in the same julia session makes the error and warnings disappear.