Environment reinitialization if failed

[johnnychen94]

Because CondaPkg doesn't verify the name/version when pkg> conda add xxx, it is very likely that users hit a broken CondaPkg.toml file and thus using PythonCall fails. I'm wondering if there's a way to quickly fix this in the same Julia session?

MWE:

julia> versioninfo()
Julia Version 1.7.1
Commit ac5cc99908 (2021-12-22 19:35 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: 12th Gen Intel(R) Core(TM) i9-12900K
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, goldmont)
Environment:
  JULIA_CONDAPKG_EXE = /home/jc/miniconda3/bin/conda

(@v1.7) pkg> activate --temp
  Activating new project at `/tmp/jl_UJo19L`

julia> using CondaPkg

(jl_UJo19L) pkg> conda add nothingness

(jl_UJo19L) pkg> conda status
CondaPkg Status /tmp/jl_UJo19L/CondaPkg.toml
Not Resolved (resolve first for more information)
Packages
  nothingness

(jl_UJo19L) pkg> add PythonCall
    Updating registry at `~/.julia/registries/General.toml`
   Resolving package versions...
    Updating `/tmp/jl_UJo19L/Project.toml`
  [6099a3de] + PythonCall v0.6.1
  ...

julia> using PythonCall
    CondaPkg Found dependencies: /tmp/jl_UJo19L/CondaPkg.toml
    CondaPkg Found dependencies: /home/jc/.julia/environments/v1.7/CondaPkg.toml
    CondaPkg Found dependencies: /home/jc/.julia/packages/PythonCall/Z6DIG/CondaPkg.toml
    CondaPkg Creating environment
             │ /home/jc/miniconda3/bin/conda
             │ create
             │ -y
             │ -p /tmp/jl_UJo19L/.CondaPkg/env
             │ --override-channels
             │ --no-channel-priority
             │ nothingness
             │ python >=3.5,<4
             └ -c conda-forge
Collecting package metadata (current_repodata.json): done
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

  - nothingness

julia> shutil = pyimport("shutil")
ERROR: UndefVarError: pyimport not defined
Stacktrace:
 [1] top-level scope
   @ REPL[8]:1

For instance, I'd like to hotfix this via:

(jl_UJo19L) pkg> conda rm nothingness

(jl_UJo19L) pkg> conda status
CondaPkg Status /tmp/jl_UJo19L/CondaPkg.toml (empty)
Not Resolved (resolve first for more information)

julia> using PythonCall

julia> shutil = pyimport("shutil")
ERROR: UndefRefError: access to undefined reference
Stacktrace:
 [1] PyUnicode_DecodeUTF8
   @ ~/.julia/packages/PythonCall/Z6DIG/src/cpython/pointers.jl:297 [inlined]
 [2] pystr_fromUTF8
   @ ~/.julia/packages/PythonCall/Z6DIG/src/concrete/str.jl:1 [inlined]
 [3] pystr_fromUTF8
   @ ~/.julia/packages/PythonCall/Z6DIG/src/concrete/str.jl:2 [inlined]
 [4] pystr
   @ ~/.julia/packages/PythonCall/Z6DIG/src/concrete/str.jl:10 [inlined]
 [5] Py
   @ ~/.julia/packages/PythonCall/Z6DIG/src/Py.jl:138 [inlined]
 [6] macro expansion
   @ ~/.julia/packages/PythonCall/Z6DIG/src/Py.jl:128 [inlined]
 [7] pyimport(m::String)
   @ PythonCall ~/.julia/packages/PythonCall/Z6DIG/src/concrete/import.jl:11
 [8] top-level scope
   @ REPL[14]:1

but it errors unless I restart a new Julia.

[cjdoris]

If a Julia module is imported but its __init__ function fails, you invariably need to restart your Julia session. This is common to many packages which need to set up some state when they start.

You can always do pkg> conda resolve yourself to install the dependencies. This way you can fix any problems before doing using PythonCall.

[johnnychen94]

You can always do pkg> conda resolve yourself to install the dependencies.

That works but is still inconvenient I think. Is there a way to dynamically add/rm a package? For instance, the following steps currently fail because CondaPkg tries to drop the current environment.

julia> np = pyimport("numpy")
ERROR: Python: ModuleNotFoundError: No module named 'numpy'
Python stacktrace: none
Stacktrace:
 [1] pythrow()
   @ PythonCall ~/.julia/packages/PythonCall/Z6DIG/src/err.jl:94
 [2] errcheck
   @ ~/.julia/packages/PythonCall/Z6DIG/src/err.jl:10 [inlined]
 [3] pyimport(m::String)
   @ PythonCall ~/.julia/packages/PythonCall/Z6DIG/src/concrete/import.jl:11
 [4] top-level scope
   @ REPL[2]:1

(@v1.7) pkg> conda add numpy

(@v1.7) pkg> conda resolve -f
    CondaPkg Found dependencies: /home/jc/.julia/environments/v1.7/CondaPkg.toml
    CondaPkg Found dependencies: /home/jc/.julia/packages/PythonCall/Z6DIG/CondaPkg.toml
    CondaPkg Removing environment
             │ /home/jc/miniconda3/bin/conda
             │ remove
             │ -y
             │ -p /home/jc/.julia/environments/v1.7/.CondaPkg/env
             └ --all

CondaEnvironmentError: cannot remove current environment. deactivate and run conda remove again

[cjdoris]

You're not supposed to add or remove packages while you've got Python running. This is because the versions of packages installed can become incompatible with the versions you already have loaded.

So in the case of PythonCall you should use CondaPkg to get your dependencies right before loading PythonCall, and then not touch them.

[johnnychen94]

Okay, I see there's no intent to provide another solution than "restarting the entire Julia session" so I'm closing this.

Just one last word. I opened this issue because this is something that PyCall allows, e.g.,

julia> using PyCall

julia> seaborn= pyimport("seaborn")
ERROR: PyError (PyImport_ImportModule

The Python package seaborn could not be imported by pyimport. Usually this means
that you did not install seaborn in the Python version being used by PyCall.

...

# switch to other terminal and install seaborn, then switch back
julia> seaborn = pyimport("seaborn")
PyObject <module 'seaborn' from '/home/jc/miniconda3/envs/conda_jl/lib/python3.9/site-packages/seaborn/__init__.py'>

[cjdoris]

Sure you can do that, but it's not recommended for the reasons above. It will probably actually be fine 99% of the time in interactive use, but CondaPkg is intentionally conservative because it's primary purpose is to create Conda environments reproducibly.

I'll think about making the interactive experience a bit nicer - maybe a flag somewhere which just adds a package to the environment without reinstalling everything.