Get ctypes working

[tiran]

• build libffi from https://github.com/hoodmane/libffi-emscripten
• cp /opt/libffi-emscripten/lib/libffi.a /emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/pic/
• cp /opt/libffi-emscripten/include/ffi* /emsdk/upstream/emscripten/cache/sysroot/include/
• Build with ./build-python-emscripten-node.sh --enable-wasm-dynamic-linking
• Add *shared*\n_ctypes _ctypes/_ctypes.c _ctypes/callbacks.c _ctypes/callproc.c _ctypes/stgdict.c _ctypes/cfield.c -lffi to Module/Setup.local

[pmp-p]

cp /python-wasm/libffi-emscripten/target/lib/libffi.* /emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/pic/
cp /python-wasm/libffi-emscripten/target/include/ffi* /emsdk/upstream/emscripten/cache/sysroot/include/

Maybe that fixture could be avoided with https://bugs.python.org/issue14527 + its pr python/cpython#20451 using pkg-config + setting PKG_CONFIG_PATH

Why not add a specific target libpython.wasm to the makefile built with -s SIDE_MODULE=1 ? that would be --enable-shared
( i think both static and dynamic libpython are usefull, -fPIC everywhere is harmless )

[pmp-p]

just saw python/cpython#32253 and remind of problem i had with ctypes
iirc v8 cannot load synchronously any wasm library with a size >32KiB ( unlike node or firefox), so it would probably need when targeting browser to add --use-preload-plugins somewhere in assets packaging

[hoodmane]

v8 cannot load any wasm library with a size >32KiB

Of course you mean it refuses to synchronously load large wasm libraries.

--use-preload-plugins somewhere in assets packaging

I would encourage people to package their Python side modules as wheels and compile them themselves rather than using preload plugins. This requires more setup code, but it gives much better control and there are is much more mature tooling around wheels and zip files than around emscripten file_packager which is basically write-only and cannot be modified by humans or tools after creation.

[pmp-p]

compile them themselves rather than using preload plugins.

but emscripten_run_preload_plugins and FS.createPreloadedFile have serious limitations. I don't see how you would take out the .so from the wheel and dlopen it synchronously unless you stay in a worker or firefox ( which both bring other serious limitations ).

i just tried with BrowserFS with ZipFS/OverlayFS/InMemoryFS and the only way i've found was to copy file into MEMFS and preload from there (required for image/audio/wasm ), the gap beetween Node and browser is to big there.

[hoodmane]

See here:
https://github.com/pyodide/pyodide/blob/main/src/js/load-package.ts#L227

[pmp-p]

so basically it would be preferable to hook fopen and rewrite the preloading logic instead of trying to rely on emscripten FS ? anyway i guess that would be required to make a wasi shim for the browser.

[hoodmane]

I'm a bit confused about what the filesystem has to do with it? I'm not doing anything crazy like hooking fopen. Emscripten has the function loadWebAssemblyModule which loads a Wasm Module from a buffer. It doesn't do any filesystem operations. You can get the buffer to it however you like.

[pmp-p]

what the filesystem has to do with it?

the low level fonction for preloading audio/images/wasm ( The 3 of them not only 1 like in load-package.ts ) is FS.createPreloadedFile but it does not seem to support handling blob urls ( i tried with various in memory fs ).

if you don't hook the filesytem how would a C extension be able to dlopen ( or load an image take SDL2_image port ) on the fly. "import" is just the tip of the iceberg.

ref: https://emscripten.org/docs/api_reference/Filesystem-API.html#FS.createPreloadedFile

[hoodmane]

Don't use FS.createPreloadedFile, use Module.loadWebAssemblyModule.

[hoodmane]

Right okay, so the point is that Emscripten's dlopen looks in Module.preloadedWasm for the compiled wasm and if it finds it there it doesn't do any filesystem operations. So what you need to do is

const module = await Module.loadWebAssemblyModule(wasmBuffer, {
      loadAsync: true,
      nodelete: true,
      allowUndefined: true,
});
Module.preloadedWasm[dynamic_lib_path] = module;

where dynamic_lib_path is the path that you want to dlopen later.

[pmp-p]

thx @hoodmane
i've found 3 globals ( not in Module but in window , i guess Module is for workers ) that govern preloading : these are preloadedWasm, preloadedImages, preloadedAudios
It allowed me to load directly from a mounted zip by preloading a temp copy in MEMFS, instantiate and then alter the table to set the correct vfs path so C extensions or ctypes/ffi calls can load the files without further assistance.

edit/ it was made from C with one call to https://emscripten.org/docs/api_reference/emscripten.h.html#c.emscripten_run_preload_plugins + a python call to run_script ( eval for js ) to fix the table after the yield. possibly a job for importlib.invalidate_caches()

[hoodmane]

not in Module but in window

Really you should be building with -s MODULARIZE=1 and then you will find these in Module. Not ideal to allow the huge number of Emscripten variables to be dumped into global scope.

[pmp-p]

you should be building with -s MODULARIZE=1

that seems a very good idea !

@tiran that would need to be set at the makefile stage of -o python.html Programs/python.o of browser target ?

[tiran]

MODULARIZE requires changes to our browser-ui. My JS skills are on total noob level. The worker is failing with some messages related to promise and onMessage:

Error: Promised response from onMessage listener went out of scope

[pmp-p]

Error: Promised response from onMessage listener went out of scope

same js level there that sounds scary, i'm wary about worker since they have very choppy input and no audio, and worse i'm quite sure i have read somewhere they do not fit well with dynamic linking ( or was it threading? )

[hoodmane]

Workers are great, should always be used in production since otherwise Python will block the UI. The debugging experience is worse though. Workers work correctly with both dynamic linking and threading, but dynamic linking and threading can't be used at the same time.

[pmp-p]

@hoodmane indeed but i fear your point of view is a bit too Pyodide centric which is a bit too "serious" for broad audience.
Workers are not great for 3D / Audio and reactive input like multitouch ( mostly gaming use cases Panda3D / Pygame etc ... )

i advocate "python for everyone, everywhere" so maybe there should be one UI with python as worker and one without :)

[hoodmane]

I think they are even more important for games and reactive input because otherwise that input cannot be processed while Python code is running. We keep Pyodide's console in the main thread because it makes debugging easier.

[hoodmane]

For a game, I think the correct design would look like:
Main UI thread listens to input events and serializes them into a SharedArrayBuffer.
Game main loop periodically polls the SharedArrayBuffer and updates game state accordingly.

You could probably make it work on the main thread with window.requestAnimationFrame (which is the browser's way of doing something once every frame):

function step(){
	iteratePythonGameState();
	window.requestAnimationFrame(step);
}
window.requestAnimationFrame(step);

and then probably there is enough time between each step to process input events.

Running the game in the main thread isn't considered to be "browser best practices", but using a worker is indeed a lot of extra trouble. I am hoping to improve the situation with https://github.com/hoodmane/synclink which admittedly still won't make it easy to understand but at least will limit how much code you have to write.

[hoodmane]

maybe there should be one UI with python as worker and one without

If you are putting it into Python I would recommend not running it in a worker. Pyodide has decided that making a full UI that runs in a worker is out of scope for us, I think it's also a bit out of scope for CPython. Compare: CPython has a basic repl, IPython is a package.
python/cpython#32284

[pmp-p]

Actually the design you describe is close to the one on main thread, but worker sneak in more serialization overhead and some flaws.

js call requestAnimationFrame
wasm
{
- game consume directly ALL events+io in emscripten queue cherry picking without serializing unwanted events.
- game logic and ctypes can make synchronous call to window.document if required.worker cannot.
- game draws with next vsync as a deadline, skip details if too late

}
browser
{

• draw batch is processed by javascript.
• audio batch is processed by javascript.
• catch all events+io and put them in emscripten queue.

}
loop again.

[hoodmane]

• game logic and ctypes can make synchronous call to window.document if required.worker cannot.

Yeah the goal of hoodmane/synclink is to allow synchronous calls to main thread from the worker. But it obviously introduces some complexity and limitations.

[AlmogBaku]

Does anyone have an idea how this could be accomplished with WASI? since WASI doesn't support dynamic linking

[pmp-p]

@AlmogBaku

well as discussed on discord i may have one but it's as simple as it's horribly slow : move the dlfcn implementation (temporarily as wasi may gain dlfcn someday ) in cpython layer and use an asynchronous wasm vm from cpython ( pywasm (pure python) or pywasm3 (very portable C) ) to load extra modules