Error during Module initialization

[pedromxavier]

I'm currently using PythonCall within a new Julia package. The environment is configured according to the docs. Indeed, PythonCall is was added as a dependency, CondaPkg.toml is present and so on.
Locally it works like a charm, but during standard Julia CI it fails to initialize due to pathof(PythonCall) returning nothing. It then triggers a Method Error by calling dirname(::Nothing). In fact, according to pathof's docs:

help?> pathof
search: pathof

  pathof(m::Module)

  Return the path of the m.jl file that was used to import
  module m, or nothing if m was not imported from a package.

  Use dirname to get the directory part and basename to get the
  file name part of the path.

I'm not really certain about init_juliacall semantics, but I think this is a corner case to be addressed.
https://github.com/cjdoris/PythonCall.jl/blob/d193f567c6a4ddde84ecb0ca3a9a36c8335820d1/src/juliacall.jl#L20

Also, any tips or workarounds are appreciated!

[cjdoris]

I've occasionally had the same issue locally, but it seemed to occur at random - restarting Julia always fixed it.

PythonCall is always loaded from a package though, so pathof should always return a string, so this seems like a bug?

Do you have a way to reproduce this reliably?

[cjdoris]

I guess I should really be using @__DIR__ instead. I'll do that.

[pedromxavier]

I've occasionally had the same issue locally, but it seemed to occur at random - restarting Julia always fixed it.

PythonCall is always loaded from a package though, so pathof should always return a string, so this seems like a bug?

Do you have a way to reproduce this reliably?

TL;DR: I think this issue may affect every chained dependecy in the form Package -> Other Package -> PythonCall

Yes, indeed. I should've made this explict but I was in a hurry. The problem showed up in two situations and I guess it has to do with caching of some sort.

The first one happens randomly when the environment was clean and the project is built for the first time. The reliable situation happens when using PythonCall on the innermost package on a Pkg.develop-nested environment. I don't know if this extends to Pkg.add but I guess yes. I think that nesting has nothing to do with it, any locally-dependent package would trigger this.

Summary:

MainPackage.jl (uses InnerPackage via ]dev)
├── src
│   ├── InnerPackge.jl (uses Pythoncall via ]add)

I've just solved it by building and testing InnerPackage before instead of lazily wating for it to be built as MainPackage's dep.

[pedromxavier]

I think this happens because from Package's perspective, PythonCall is not imported from a package as it is from OtherPackage's.

When running locally this can be fixed by running again but keeping this will mess with many CI workflows around.

[cjdoris]

Interesting! Anyway I've fixed this on the main branch now, using @__DIR__ instead of pathof.