✨ Support Proxy/Copying shared array data so we can use runAsync with data from resize plugin and similar.

[FieldMarshallVague]

What feature or enhancement are you suggesting?

Basically, I'm running into this problem:

#2520 (comment)

I'm getting a segmentation fault when running my tflite model inside a worklet that I'm passing the output of the resize plugin. I've tried copying the data etc., but it all ends up the same way. Segfault, SIGSEG, dead 11, something something. I'm looking at patching the plugin myself, but this is way outside my experience and I don't know what I don't know (if you know what I mean).

I know it's possible to use runAtTargetFps, but that is janky and the sync approach is bad UX (either jank on target FPS or just <5fps in my case - using fast-tflite).

It would be great if the fix mrousavy suggests in the above link could be implemented. I guess proxying would be ideal, if that means the frame data doesn't have to be duplicated?

If I make any progress on this myself, I'll report back here, but I've never done anything on Android native before (and guessing this might also be an issue on iOS?).

I don't seem to be able to select 'both' in the platform dropdown, but I'm assuming this either isn't an issue on iOS or would also need to be done on that, too?

What Platforms whould this feature/enhancement affect?

Android

Alternatives/Workarounds

(as above) I know it's possible to use runAtTargetFps, but that is janky and all other approaches are bad UX (either jank on target FPS or just <10fps in my case).

Additional information

LOGS:

02-13 11:31:15.723   900   900 W BufferQueueLayer: [SurfaceView - com.myapp.myapp/com.myapp.myapp.MainActivity#0] Timestamp 9162465852700 seems implausible relative to expectedPresent 1932519565600
02-13 11:31:15.729 24627 24782 I ResizePlugin: Rotation: 270
02-13 11:31:15.729 24627 24782 I ResizePlugin: Mirror not specified, defaulting to: false
02-13 11:31:15.729 24627 24782 I ResizePlugin: Target scale: 576 x 320
02-13 11:31:15.729 24627 24782 I ResizePlugin: Cropping to 640 x 355 at (0, 63)
02-13 11:31:15.729 24627 24782 I ResizePlugin: Target Format: RGB
02-13 11:31:15.729 24627 24782 I ResizePlugin: Target DataType: UINT8
02-13 11:31:15.729 24627 24782 I ResizePlugin: Allocating _argbBuffer Buffer with size 1228800...
02-13 11:31:15.730 24627 24782 I ResizePlugin: Converting YUV 4:2:0 -> ARGB 8888...
02-13 11:31:15.730 24627 24782 I ResizePlugin: Cropping [0, 0 @ 640x480] ARGB buffer to [0, 63 @ 640x355]...
02-13 11:31:15.730 24627 24782 I ResizePlugin: Allocating _cropBuffer Buffer with size 908800...
02-13 11:31:15.731 24627 24782 I ResizePlugin: Scaling [0, 0 @ 640x355] ARGB buffer to [0, 0 @ 576x320]...
02-13 11:31:15.731 24627 24782 I ResizePlugin: Allocating _scaleBuffer Buffer with size 737280...
02-13 11:31:15.732 24627 24782 I ResizePlugin: Rotating ARGB buffer by 270 degrees...
02-13 11:31:15.732 24627 24782 I ResizePlugin: Allocating _rotatedBuffer Buffer with size 737280...
02-13 11:31:15.733 24627 24782 I ResizePlugin: Converting ARGB Buffer to Pixel Format 0...
02-13 11:31:15.733 24627 24782 I ResizePlugin: Allocating _customFormatBuffer Buffer with size 552960...
02-13 11:31:15.733 24627 24782 I SharedArray: Wrapping Java ByteBuffer with size 552960...
02-13 11:31:15.734 24627 24782 F libc    : Fatal signal 11 (SIGSEGV), code 2 (SEGV_ACCERR), fault addr 0x11ba82da7d in tid 24782 (mqt_v_js), pid 24627 (myapp.myapp)

• I agree to follow this project's Code of Conduct
• I searched for similar feature requests in this repository and found none.

[maintenance-hans]

Guten Tag, Hans here! 🍻

I appreciate your detailed feedback and the reference to the existing issue. However, I noticed that you did not provide any logs related to the segmentation fault you are experiencing. Logs from adb logcat on Android are essential for troubleshooting this issue. Please gather those logs and post them here, so mrousavy can better understand zhe problem and assist you accordingly.

Also, if you would like to support mrousavy and ensure faster attention to issues, consider becoming a sponsor of ze project here.

Looking forward to your updates!

Note: If you think I made a mistake, please ping @mrousavy to take a look.

[FieldMarshallVague]

^^ Added above, but FWIW, the issue I linked to has logs which are exemplary of the problem. I've had the same as well as different errors, but they all refer to the same underlying issue, afaict.

[mrousavy]

This is way more complicated than it sounds. Check out Nitro's ArrayBuffer implementation to understand the constraints.

[FieldMarshallVague]

This is way more complicated than it sounds. Check out Nitro's ArrayBuffer implementation to understand the constraints.

@mrousavy Thanks for letting me know, and the tip. I'd already started to realise it might be more than I could handle, and this pretty much confirms it (very new to native and no cpp experience). Feel free to close this.