pyjl (#512)

* change julia wrapper hierarchy

* remove RawValue, TypeValue and ModuleValue from juliacall

* add JlBase.__init__ and store nothing/true/false specially

* more general numeric operators

* rounding methods, jl_to_py, remove jlwrap numbers

* update release notes

* remove references to nonexistent types

* add jl_callback

* simpler iteration

* simplify callbacks (pyfunc etc)

* change to pyjl(x)

* rename ArrayValue to JlArray etc

* __init__ for all

* mixins for consistency

* improve pymacro docstring

* jlwrap tests

* remove old tests

* pyjlset tests

* pyjldict tests

* pyjlio tests

* pyjlarray tests

* pyjlvector tests

* pyjl hash, == and iter

* add JlContainer

* eliminate mixins and add contains and clear to JlCollection

* fix keys, values, items

* fix tests

* tests for JlCollection

* update jlwrap docs

* Jl tests

* JlDict tests

* PySet tests

* JlIO tests

* more JlIO tests

* rename julia wrapper classes

* enable CI on v1 branch

* Jl.jl_eval fix return type

* adapt python tests to latest version

* propagate seval -> jl_eval

---------

Co-authored-by: Christopher Doris <github.com/cjdoris>

Author: cjdoris

.github/workflows/docs.yml

@@ -4,6 +4,7 @@ on:
   push:
     branches:
     - main
+    - v1
     tags:
     - '*'
   pull_request:

.github/workflows/tests.yml

@@ -4,9 +4,11 @@ on:
   pull_request:
     branches:
     - main
+    - v1
   push:
     branches:
     - main
+    - v1
     tags:
     - '*'
 

docs/src/conversion-to-julia.md

@@ -11,7 +11,7 @@ From Python, the arguments to a Julia function will be converted according to th
 | From                                                                                                         | To                                                          |
 | :----------------------------------------------------------------------------------------------------------- | :---------------------------------------------------------- |
 | **Top priority (wrapped values).**                                                                           |                                                             |
-| `juliacall.AnyValue`                                                                                         | `Any`                                                       |
+| `juliacall.Jl`                                                                                         | `Any`                                                       |
 | **Very high priority (arrays).**                                                                             |                                                             |
 | Objects satisfying the buffer or array interface (inc. `bytes`, `bytearray`, `array.array`, `numpy.ndarray`) | `PyArray`                                                   |
 | **High priority (canonical conversions).**                                                                   |                                                             |

docs/src/conversion-to-python.md

@@ -23,16 +23,13 @@ From Python, this occurs when converting the return value of a Julia function.
 | Standard integer range (`AbstractRange{T}`, `T` a standard integer) | `range`                                                 |
 | `Date`, `Time`, `DateTime` (from `Dates`)                           | `date`, `time`, `datetime` (from `datetime`)            |
 | `Second`, `Millisecond`, `Microsecond`, `Nanosecond` (from `Dates`) | `timedelta` (from `datetime`)                           |
-| `Number`                                                            | `juliacall.NumberValue`, `juliacall.ComplexValue`, etc. |
-| `AbstractArray`                                                     | `juliacall.ArrayValue`, `juliacall.VectorValue`         |
-| `AbstractDict`                                                      | `juliacall.DictValue`                                   |
-| `AbstractSet`                                                       | `juliacall.SetValue`                                    |
-| `IO`                                                                | `juliacall.BufferedIOValue`                             |
-| `Module`                                                            | `juliacall.ModuleValue`                                 |
-| `Type`                                                              | `juliacall.TypeValue`                                   |
-| Anything else                                                       | `juliacall.AnyValue`                                    |
+| `AbstractArray`                                                     | `juliacall.JlArray`, `juliacall.JlVector`               |
+| `AbstractDict`                                                      | `juliacall.JlDict`                                      |
+| `AbstractSet`                                                       | `juliacall.JlSet`                                       |
+| `IO`                                                                | `juliacall.JlBinaryIO`                                  |
+| Anything else                                                       | `juliacall.Jl`                                          |
 
-See [here](@ref julia-wrappers) for an explanation of the `juliacall.*Value` wrapper types.
+See [here](@ref julia-wrappers) for an explanation of the `juliacall.Jl*` wrapper types.
 
 ## [Custom rules](@id jl2py-conversion-custom)
 
@@ -44,11 +41,3 @@ object, then also define `ispy(::T) = true`.
 ```@docs
 PythonCall.ispy
 ```
-
-Alternatively, if you define a wrapper type (a subtype of
-[`juliacall.AnyValue`](#juliacall.AnyValue)) then you may instead define `pyjltype(::T)` to
-be that type.
-
-```@docs
-PythonCall.pyjltype
-```

docs/src/faq.md

@@ -20,7 +20,7 @@ Related issues: [#201](https://github.com/JuliaPy/PythonCall.jl/issues/201), [#2
 
 ## Issues when Numpy arrays are expected
 
-When a Julia array is passed to Python, it is wrapped as a [`ArrayValue`](#juliacall.ArrayValue).
+When a Julia array is passed to Python, it is wrapped as a [`JlArray`](#juliacall.JlArray).
 This type satisfies the Numpy array interface and the buffer protocol, so can be used in
 most places where a numpy array is valid.
 

docs/src/juliacall-reference.md

@@ -5,7 +5,7 @@
 `````@customdoc
 juliacall.Main - Constant
 
-The Julia `Main` module, as a [`ModuleValue`](#juliacall.ModuleValue).
+The Julia `Main` module, as a [`Jl`](#juliacall.Jl).
 
 In interactive scripts, you can use this as the main entry-point to JuliaCall:
 ```python
@@ -20,18 +20,6 @@ The modules `Base`, `Core` and `PythonCall` are also available.
 
 ## Utilities
 
-`````@customdoc
-juliacall.convert - Function
-
-```python
-convert(T, x)
-```
-
-Convert `x` to a Julia object of type `T`.
-
-You can use this to pass an argument to a Julia function of a specific type.
-`````
-
 `````@customdoc
 juliacall.newmodule - Function
 
@@ -45,74 +33,70 @@ A new module with the given name.
 ## [Wrapper types](@id julia-wrappers)
 
 Apart from a few fundamental immutable types, all Julia values are by default converted into
-Python to some [`AnyValue`](#juliacall.AnyValue) object, which wraps the original value, but
-giving it a Pythonic interface.
-
-Subclasses of [`AnyValue`](#juliacall.AnyValue) provide additional Python semantics. For
-example a Julia vector is converted to a [`VectorValue`](#juliacall.VectorValue) which
-satisfies the Python sequence interface and behaves very similar to a list.
-
-There is also a [`RawValue`](#juliacall.RawValue) object, which gives a stricter
-"Julia-only" interface, documented below. These types all inherit from `ValueBase`:
-
-- `ValueBase`
-  - [`RawValue`](#juliacall.RawValue)
-  - [`AnyValue`](#juliacall.AnyValue)
-    - [`NumberValue`](#juliacall.NumberValue)
-      - `ComplexValue`
-      - `RealValue`
-        - `RationalValue`
-        - `IntegerValue`
-    - [`ArrayValue`](#juliacall.ArrayValue)
-      - [`VectorValue`](#juliacall.VectorValue)
-    - [`DictValue`](#juliacall.DictValue)
-    - [`SetValue`](#juliacall.SetValue)
-    - [`IOValue`](#juliacall.IOValue)
-      - `BinaryIOValue`
-      - `TextIOValue`
-    - [`ModuleValue`](#juliacall.ModuleValue)
-    - [`TypeValue`](#juliacall.TypeValue)
+Python to a [`Jl`](#juliacall.Jl) object, which wraps the original value and gives it a
+Pythonic interface.
+
+Other wrapper classes provide more specific Python semantics. For example a Julia vector can
+be converted to a [`JlVector`](#juliacall.JlVector) which satisfies the Python sequence
+interface and behaves very similar to a list.
+
+- `JlBase`
+  - [`Jl`](#juliacall.Jl)
+  - [`JlCollection`](#juliacall.JlCollection)
+    - [`JlArray`](#juliacall.JlArray)
+      - [`JlVector`](#juliacall.JlVector)
+    - [`JlDict`](#juliacall.JlDict)
+    - [`JlSet`](#juliacall.JlSet)
+  - [`JlIOBase`](#juliacall.JlIOBase)
+    - `JlBinaryIO`
+    - `JlTextIO`
 
 `````@customdoc
-juliacall.AnyValue - Class
+juliacall.Jl - Class
 
-Wraps any Julia object, giving it some basic Python semantics. Subtypes provide extra
-semantics.
+Wraps any Julia object, giving it some basic Python semantics.
 
 Supports `repr(x)`, `str(x)`, attributes (`x.attr`), calling (`x(a,b)`), iteration,
 comparisons, `len(x)`, `a in x`, `dir(x)`.
 
-Calling, indexing, attribute access, etc. will convert the result to a Python object
-according to [this table](@ref jl2py). This is typically a builtin Python type (for
-immutables) or a subtype of `AnyValue`.
+Calling, indexing, attribute access, etc. will always return a `Jl`. To get the result
+as an ordinary Python object, you can use the `.jl_to_py()` method.
 
-Attribute access can be used to access Julia properties as well as normal class members. In
-the case of a name clash, the class member will take precedence. For convenience with Julia
-naming conventions, `_b` at the end of an attribute is replaced with `!` and `_bb` is
-replaced with `!!`.
+Attribute access (`x.attr`) can be used to access Julia properties except those starting
+and ending with `__` (since these are Python special methods) or starting with `jl_` or
+`_jl_` (which are reserved by `juliacall` for Julia-specific methods).
 
 ###### Members
-- `_jl_raw()`: Convert to a [`RawValue`](#juliacall.RawValue). (See also [`pyjlraw`](@ref).)
-- `_jl_display()`: Display the object using Julia's display mechanism.
-- `_jl_help()`: Display help for the object.
+- `jl_callback(*args, **kwargs)`: Calls the Julia object with the given arguments.
+  Unlike ordinary calling syntax, the arguments are passed as `Py` objects instead of
+  being converted.
+- `jl_display()`: Display the object using Julia's display mechanism.
+- `jl_eval(expr)`: If the object is a Julia `Module`, evaluates the given expression.
+- `jl_help()`: Display help for the object.
+- `jl_to_py()`: Convert to a Python object using the [usual conversion rules](@ref jl2py).
 `````
 
 `````@customdoc
-juliacall.NumberValue - Class
+juliacall.JlCollection - Class
+
+Wraps any Julia collection. It is a subclass of `collections.abc.Collection`.
 
-This wraps any Julia `Number` value. It is a subclass of `numbers.Number` and behaves
-similar to other Python numbers.
+Julia collections are arrays, sets, dicts, tuples, named tuples, refs, and in general
+anything which is a collection of values in the sense that it supports functions like
+`iterate`, `in`, `length`, `hash`, `==`, `isempty`, `copy`, `empty!`.
 
-There are also subtypes `ComplexValue`, `RealValue`, `RationalValue`, `IntegerValue` which
-wrap values of the corresponding Julia types, and are subclasses of the corresponding
-`numbers` ABC.
+It supports `in`, `iter`, `len`, `hash`, `bool`, `==`.
+
+###### Members
+- `clear()`: Empty the collection in-place.
+- `copy()`: A copy of the collection.
 `````
 
 `````@customdoc
-juliacall.ArrayValue - Class
+juliacall.JlArray - Class
 
 This wraps any Julia `AbstractArray` value. It is a subclass of
-`collections.abc.Collection`.
+`juliacall.JlCollection`.
 
 It supports zero-up indexing, and can be indexed with integers or slices. Slicing returns a
 view of the original array.
@@ -130,22 +114,20 @@ copy of the original array.
 ###### Members
 - `ndim`: The number of dimensions.
 - `shape`: Tuple of lengths in each dimension.
-- `copy()`: A copy of the array.
 - `reshape(shape)`: A reshaped view of the array.
 - `to_numpy(dtype=None, copy=True, order="K")`: Convert to a numpy array.
 `````
 
 `````@customdoc
-juliacall.VectorValue - Class
+juliacall.JlVector - Class
 
-This wraps any Julia `AbstractVector` value. It is a subclass of `juliacall.ArrayValue` and
+This wraps any Julia `AbstractVector` value. It is a subclass of `juliacall.JlArray` and
 `collections.abc.MutableSequence` and behaves similar to a Python `list`.
 
 ###### Members
 - `resize(size)`: Change the length of the vector.
 - `sort(reverse=False, key=None)`: Sort the vector in-place.
 - `reverse()`: Reverse the vector.
-- `clear()`: Empty the vector.
 - `insert(index, value)`: Insert the value at the given index.
 - `append(value)`: Append the value to the end of the vector.
 - `extend(values)`: Append the values to the end of the vector.
@@ -156,65 +138,23 @@ This wraps any Julia `AbstractVector` value. It is a subclass of `juliacall.Arra
 `````
 
 `````@customdoc
-juliacall.DictValue - Class
+juliacall.JlDict - Class
 This wraps any Julia `AbstractDict` value. It is a subclass of `collections.abc.Mapping` and
 behaves similar to a Python `dict`.
 `````
 
 `````@customdoc
-juliacall.SetValue - Class
+juliacall.JlSet - Class
 This wraps any Julia `AbstractSet` value. It is a subclass of `collections.abc.Set` and
 behaves similar to a Python `set`.
 `````
 
 `````@customdoc
-juliacall.IOValue - Class
+juliacall.JlIOBase - Class
 
 This wraps any Julia `IO` value. It is a subclass of `io.IOBase` and behaves like Python
 files.
 
-There are also subtypes `BinaryIOValue` and `TextIOValue`, which are subclasses of
+There are also subtypes `JlBinaryIO` and `JlTextIO`, which are subclasses of
 `io.BufferedIOBase` (buffered bytes) and `io.TextIOBase` (text).
 `````
-
-`````@customdoc
-juliacall.ModuleValue - Class
-This wraps any Julia `Module` value.
-
-It is the same as [`AnyValue`](#juliacall.AnyValue) except for one additional convenience
-method:
-- `seval([module=self], code)`: Evaluates the given code (a string) in the given module.
-`````
-
-`````@customdoc
-juliacall.TypeValue - Class
-
-This wraps any Julia `Type` value.
-
-It is the same as [`AnyValue`](#juliacall.AnyValue) except that indexing is used to access
-Julia's "curly" syntax for specifying parametric types:
-
-```python
-from juliacall import Main as jl
-# equivalent to Vector{Int}() in Julia
-jl.Vector[jl.Int]()
-```
-`````
-
-`````@customdoc
-juliacall.RawValue - Class
-
-Wraps any Julia value with a rigid interface suitable for generic programming.
-
-Supports `repr(x)`, `str(x)`, attributes (`x.attr`), calling (`x(a,b)`), `len(x)`, `dir(x)`.
-
-This is very similar to [`AnyValue`](#juliacall.AnyValue) except that indexing, calling,
-etc. will always return a `RawValue`.
-
-Indexing with a tuple corresponds to indexing in Julia with multiple values. To index with a
-single tuple, it will need to be wrapped in another tuple.
-
-###### Members
-- `_jl_any()`: Convert to a [`AnyValue`](#juliacall.AnyValue) (or subclass). (See also
-  [`pyjl`](@ref).)
-`````

docs/src/juliacall.md

@@ -50,11 +50,11 @@ import juliacall
 jl = juliacall.newmodule("SomeName")
 ```
 
-Julia modules have a special method `seval` which will evaluate a given piece of code given
+Julia modules have a special method `jl_eval` which will evaluate a given piece of code given
 as a string in the module. This is most frequently used to import modules:
 ```python
 from array import array
-jl.seval("using Statistics")
+jl.jl_eval("using Statistics")
 x = array('i', [1, 2, 3])
 jl.mean(x)
 # 2.0
@@ -64,7 +64,7 @@ jl.cor(x, y)
 ```
 
 What to read next:
-- The main functionality of this package is in `AnyValue` objects, which represent Julia
+- The main functionality of this package is in `Jl` objects, which represent Julia
   objects, [documented here](@ref julia-wrappers).
 - If you need to install Julia packages, [read here](@ref julia-deps).
 - When you call a Julia function, such as `jl.rand(...)` in the above example, its

docs/src/pycall.md

@@ -26,7 +26,7 @@ Furthermore, `pyconvert` can be extended to support more types, whereas `convert
 
 Both packages allow conversion of Julia values to Python: `PyObject(x)` in PyCall, `Py(x)` in PythonCall.
 
-Whereas both packages convert numbers, booleans, tuples and strings to their Python counterparts, they differ in handling other types. For example PyCall converts `AbstractVector` to `list` whereas PythonCall converts `AbstractVector` to `juliacall.VectorValue` which is a sequence type directly wrapping the Julia value - this has the advantage that mutating the Python object also mutates the original Julia object.
+Whereas both packages convert numbers, booleans, tuples and strings to their Python counterparts, they differ in handling other types. For example PyCall converts `AbstractVector` to `list` whereas PythonCall converts `AbstractVector` to `juliacall.JlVector` which is a sequence type directly wrapping the Julia value - this has the advantage that mutating the Python object also mutates the original Julia object.
 
 Hence with PyCall the following does not mutate the original array `x`:
 ```julia

docs/src/pythoncall-reference.md

@@ -88,11 +88,13 @@ conversion to Python, unless the value is immutable and has a corresponding Pyth
 
 ```@docs
 pyjl
-pyjlraw
 pyisjl
 pyjlvalue
 pybinaryio
 pytextio
+pyjlarray
+pyjlset
+pyjldict
 ```
 
 ## Arithmetic

docs/src/releasenotes.md

@@ -2,6 +2,17 @@
 
 ## Unreleased (v1)
 * `PythonCall.GC` is now more like `Base.GC`: `enable(true)` replaces `enable()`, `enable(false)` replaces `disable()`, and `gc()` is added.
+* Breaking changes to Julia wrapper types:
+  * Classes renamed: `ValueBase` to `JlBase`, `AnyValue` to `Jl`, `ArrayValue` to `JlArray`, etc.
+  * Classes removed: `RawValue`, `ModuleValue`, `TypeValue`, `NumberValue`, `ComplexValue`, `RealValue`, `RationalValue`, `IntegerValue`.
+  * `Jl` now behaves similar to how `RawValue` behaved before. In particular, most methods on `Jl` now return a `Jl` instead of an arbitrary Python object.
+  * `juliacall.Pkg` removed (you can import it yourself).
+  * `juliacall.convert` removed (use `juliacall.Jl` instead).
+  * Methods renamed: `_jl_display()` to `jl_display()`, `_jl_help()` to `jl_help()`, etc.
+  * Methods removed: `_jl_raw()`.
+  * `pyjl(x)` now always returns a `juliacall.Jl` (it used to select a wrapper type if possible).
+  * `pyjltype(x)` removed.
+* New functions: `pyjlarray`, `pyjldict`, `pyjlset`.
 
 ## Unreleased
 * `Serialization.serialize` can use `dill` instead of `pickle` by setting the env var `JULIA_PYTHONCALL_PICKLE=dill`.

examples/flux.ipynb

@@ -31,14 +31,14 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "jl.seval(\"using Flux\")\n",
+    "jl.jl_eval(\"using Flux\")\n",
     "model = jl.Chain(\n",
     "    jl.Dense(1, 10, jl.relu),\n",
     "    jl.Dense(10, 10, jl.relu),\n",
     "    jl.Dense(10, 10, jl.relu),\n",
     "    jl.Dense(10, 1),\n",
     ")\n",
-    "loss = jl.seval(\"m -> (x, y) -> Flux.Losses.mse(m(x), y)\")(model)"
+    "loss = jl.jl_eval(\"m -> (x, y) -> Flux.Losses.mse(m(x), y)\")(model)"
    ]
   },
   {