pyclass

[yakir12]

I'm trying to implement a Python class using PythonCall.pyclass.

The following works as expected:

using PythonCall

Person = pyclass("Person", (); print_random = print ∘ rand)
@pyexec (Person = Person) => """
p1 = Person()
p1.print_random() 
"""
# prints a random number

But any attempt to include methods that accept arguments referencing the instance of the class itself:

Person = pyclass("Person", (); __init__ = (self) -> (self.state = 0))
@pyexec (Person = Person) => """
p1 = Person()
"""

fails:

ERROR: Python: TypeError: Julia: MethodError: no method matching (::var"#5#6")()
Closest candidates are:
  (::var"#5#6")(!Matched::Any) at REPL[15]:1
Python stacktrace:
 [1] __call__
   @ /home/yakir/.julia/packages/PythonCall/Z6DIG/src/jlwrap/any.jl:167:30
 [2] <module>
   @ REPL[16]:1:1
Stacktrace:
 [1] pythrow()
   @ PythonCall ~/.julia/packages/PythonCall/Z6DIG/src/err.jl:94
 [2] errcheck
   @ ~/.julia/packages/PythonCall/Z6DIG/src/err.jl:10 [inlined]
 [3] pycallargs(f::Py, args::Py)
   @ PythonCall ~/.julia/packages/PythonCall/Z6DIG/src/abstract/object.jl:153
 [4] pycall(::Py, ::Py, ::Vararg{Py}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ PythonCall ~/.julia/packages/PythonCall/Z6DIG/src/abstract/object.jl:171
 [5] pycall
   @ ~/.julia/packages/PythonCall/Z6DIG/src/abstract/object.jl:161 [inlined]
 [6] #_#11
   @ ~/.julia/packages/PythonCall/Z6DIG/src/Py.jl:330 [inlined]
 [7] Py
   @ ~/.julia/packages/PythonCall/Z6DIG/src/Py.jl:330 [inlined]
 [8] pyexec(::Type{Nothing}, code::Py, globals::Module, locals::NamedTuple{(:Person,), Tuple{Py}})
   @ PythonCall ~/.julia/packages/PythonCall/Z6DIG/src/concrete/code.jl:98
 [9] top-level scope
   @ ~/.julia/packages/PythonCall/Z6DIG/src/concrete/code.jl:146

More context

I'm trying to use the picamera Python module and process each frame in Julia. The following MWE however segfaults:

ENV["JULIA_PYTHONCALL_EXE"] = "/usr/bin/python3" # I have to include this due to https://github.com/cjdoris/PythonCall.jl/issues/120

using PythonCall

SideEffect = pyclass("SideEffect", (), 
                   __init__ = () -> nothing,
                   write = s -> println(rand()),
                   flush = () -> print("done")
                  )

@pyexec (SideEffect = SideEffect) => """
import picamera
with picamera.PiCamera() as camera:
    camera.start_recording(SideEffect(), format='h264') # start_recording calls SideEffect.write for each frame and SideEffect.flush at the end
    camera.wait_recording(5)
    camera.stop_recording()
"""

[cjdoris]

__init__ = pymethod(self -> self.state=0) should work.

What's happening is that those methods you were defining are callable wrappers around Julia values, but are not proper Python functions, and only proper Python functions become class methods, everything else are just ordinary class members. The pymethod function converts any callable to a proper Python function.

[cjdoris]

Possibly pyclass could be changed to automatically do this wrapping for you on any Julia functions.

[yakir12]

Wrapping functions in pymethod fixed the toy example:

Person = pyclass("Person", (); 
                 __init__ = pymethod((self) -> (self.state = 0; nothing)), 
                 add = pymethod((self, n) -> (self.state += n)), 
                 print = pymethod((self) -> print(self.state))
                )
@pyexec (Person = Person) => """
p1 = Person()
p1.add(4)
p1.print() # prints 4
"""

pyclass could be changed to automatically do this wrapping for you on any Julia functions

Yeah, that would make it easier to use I think (could apply the wrapping automatically depending on the type of the arguments).

But my main use case still segfaults:

ENV["JULIA_PYTHONCALL_EXE"] = "/usr/bin/python3" # I have to include this due to https://github.com/cjdoris/PythonCall.jl/issues/120

using PythonCall

function finit(self)
    self.i = 0
    nothing # aparently, __init__ functions need to return "None"
end
function fwrite(self, s)
    self.i += 1
    println(self.i)
end
fflush(self) = print("Done")

SideEffect = pyclass("SideEffect", (), 
                     __init__ = pymethod(finit),
                     write = pymethod(fwrite),
                     flush = pymethod(fflush)
                  )

@pyexec (SideEffect = SideEffect) => """
import picamera
with picamera.PiCamera() as camera:
    camera.start_recording(SideEffect(), format='h264')
    camera.wait_recording(5)
    camera.stop_recording()
"""

I realize this might be a bit hard to debug and outside this specific issue, but do you have any idea what might be going on...? The relevant documentation for picamera is here if that helps... Really appreciate any insight you might have.

[yakir12]

I imagine the following is near impossible to replicate without an RPI + Pi Camera, but here is a leaned down example:
This works:

@pyexec """
import picamera

class SideEffect(object):
    def write(self, s):
        print('a')

with picamera.PiCamera() as camera:
    camera.start_recording(SideEffect(), format='h264')
    camera.wait_recording(5)
    camera.stop_recording()
"""

while the following, which is nearly identical, segfaults:

SideEffect = pyclass("SideEffect", (), write = pymethod((self, s) -> print("a")))

@pyexec (SideEffect = SideEffect) => """
import picamera

with picamera.PiCamera() as camera:
    camera.start_recording(SideEffect(), format='h264')
    camera.wait_recording(5)
    camera.stop_recording()
"""

[cjdoris]

I can't really help without a MWE that works on Windows, Mac or Linux I'm afraid.

As a quick check, the following does work for me, so something non-trivial is going on:

SideEffect = pyclass(....)
@pyexec SideEffect => "SideEffect().write(12)"

[kdheepak]

If you want to hack it, you can dynamically add any function as a method to the class.

>>> from types import MethodType

>>> class Foo:
...     pass
... 

>>> f = Foo()
 
>>> def method(self):
...     print("hello world")
... 

>>> Foo.method = MethodType(method, Foo)

>>> f.method()
hello world

[yakir12]

I really appreciate the ideas (and I'll see if there is some mechanism to test this on systems that are not an RPI+PiCam), but unfortunately, the following also segfaults:

@pyexec """
global SideEffect
class SideEffect(object):
    pass
"""

fun = pymethod((self, s) -> println("frame"))

@pyexec (write = fun) => """
from types import MethodType
import picamera

SideEffect.write = MethodType(write, SideEffect)

with picamera.PiCamera() as camera:
    camera.start_recording(SideEffect(), format='h264')
    camera.wait_recording(5)
    camera.stop_recording()
"""

Let me know @kdheepak if this implementation wasn't what you meant.

[kdheepak]

I'm not particularly familiar with the Python C API but maybe you can try this instead:

@pyexec """
global SideEffect
class SideEffect(object):
    pass
"""

fun = pymethod((self, s) -> println("frame"))

@pyexec (write = fun) => """
from types import MethodType
import picamera

SideEffect.write = write

with picamera.PiCamera() as camera:
    camera.start_recording(SideEffect(), format='h264')
    camera.wait_recording(5)
    camera.stop_recording()
"""

pymethod seems to be calling PyInstanceMethod_New (CPython documentation):

https://github.com/cjdoris/PythonCall.jl/blob/781d7fc7954d8fbc5f68db244c1640e5bb412589/src/concrete/method.jl#L8

But my suggestion was for a Python function. I'm not seeing a pyfunc equivalent to pymethod in the documentation of this package though.

Does it segfault when you try running the code without using the picamera package?

[cjdoris]

Try the version with MethodType but without wrapping the Julia function with pymethod? I believe MethodType can take any callable.

[yakir12]

you can try this instead

Yeah, it segfaulted as well.

without wrapping the Julia function with pymethod

Segfault too.

I'm not seeing a pyfunc equivalent to pymethod

Indeed, might a pyfunc implementation fix this you think?

Does it segfault when you try running the code without using the picamera package?

Nope...

[yakir12]

The one constant is if the class was defined outside Python then it segfaults.

[cjdoris]

How about if you add a pure python method to the class using MethodType?

[yakir12]

Specifically

@pyexec """
from types import MethodType
import picamera

class SideEffect(object):
    pass

def write(self, s):
    print("frame")

SideEffect.write = MethodType(write, SideEffect)

with picamera.PiCamera() as camera:
    camera.start_recording(SideEffect(), format='h264')
    camera.wait_recording(5)
    camera.stop_recording()
"""

Works.

[yakir12]

I can't really help without a MWE that works on Windows, Mac or Linux I'm afraid.

I found https://pypi.org/project/fake-rpi/ and while it does fake picamera it's really basic (with just 3 of the library's methods, neither of which are start_recording), so I doubt it'll do much good...

[cjdoris]

I meant to define the class from Julia but the method from Python.

[cjdoris]

Also can you post the stack trace?

[yakir12]

I meant to define the class from Julia but the method from Python

This defeats the purpose a bit since the whole point is to define the write function in Julia.

Also can you post the stack trace?

For sure, I didn't up to now cause it's so sparse:

julia> include("tmp3.jl")

signal (11): Segmentation fault
in expression starting at /home/yakir/dance/tmp3.jl:29
Allocations: 9119279 (Pool: 9117777; Big: 1502); GC: 15
Segmentation fault

Would me generating one of those rr reports (never done it before but I could try) help?

[cjdoris]

I hadn't heard of rr before but it looks great, should be very helpful.

[yakir12]

Ooof, building Julia (which is required for running the rr tool) on RPI is not trivial: every version I'm trying to build segfaults. This is unrelated to what we're doing here, it's apparently common on ARMv7 32-bit (which is tier 3).

[yakir12]

I'll just update that while I don't know why the segfaults occur, I've managed to retrieve frames from the Raspberry Pi Camera in sufficient speeds and dimensions using https://www.raspberrypi.com/documentation/accessories/camera.html#libcamera-vid and thus avoiding Python's picamera package altogether.

Another independent development that might solve this for future users is that Raspbian now comes in 64 bit (see news here https://www.raspberrypi.com/news/raspberry-pi-os-64-bit/). This is huge in and of itself, but it might also completely solve these kind of mysterious sefaults. I might be able to test this (in which case I'll be sure to report back here).

[cjdoris]

Ok great, yeah let me know how that goes. I'll leave this ticket open for the small change to pyclass, but won't do anything on the segfaults in light of it being a tier 3 platform.

[cjdoris]

I have actually gone and removed pyclass entirely, you should use pytype now instead (on main branch, not yet released).

My rationale is that in the future I'd like a pyclass function which supports metaclasses and the like, but it would need to have a different function signature, so I'm reserving the name for future use. Anyway it was just a simple bit of sugar around pytype.

I have made pytype a bit friendlier now though, so defining classes that way should be quite simple - you just need to wrap any methods in pyfunc. See the docstring. What's nice is that you can define the docstring or signature of the function at the same time to get better help or introspection.