Skip to content

Can not get proper touch event data in backend #4577

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
3 tasks done
bin-san opened this issue Apr 5, 2025 · 12 comments · May be fixed by #4579
Open
3 tasks done

Can not get proper touch event data in backend #4577

bin-san opened this issue Apr 5, 2025 · 12 comments · May be fixed by #4579
Labels
analysis Status: Requires team/community input bug Type/scope: A problem that needs fixing 🌿 intermediate Difficulty: Requires moderate understanding 🟡 medium Priority: Relevant, but not essential

Comments

@bin-san
Copy link

bin-san commented Apr 5, 2025

First Check

  • I added a very descriptive title here.
  • This is not a Q&A. I am sure something is wrong with NiceGUI or its documentation.
  • I used the GitHub search to find a similar issue and came up empty.

Example Code

from nicegui import ui 

label = ui.label('touch data')

with ui.element('div').style('width:80vw;height:80vh; background-color:lightblue;') as toucharea:
    toucharea.on('touchmove', lambda e: print(e) and label.set_text(f"{e.args['touches']['0']}"))

ui.run(host='0.0.0.0', port=8000)

Description

start the app. open the app in a browser of a touch screen device that is connected to the same network. move your finger across the touch area, no touch position is shown.

NiceGUI Version

2.13.0

Python Version

3.12

Browser

Chrome

Operating System

Android

Additional Context

No response

@evnchn
Copy link
Collaborator

evnchn commented Apr 5, 2025

Am getting plenty of touch position if I do DevTools on Desktop, turn on mobile simulation, then drag my mouse (which behaves like a finger touching the screen)

Image

May I know which browser you are using? I hope it is not Safari...

@evnchn
Copy link
Collaborator

evnchn commented Apr 5, 2025

Works fine on Samsung Internet on Android, Safari on iOS.

Definitely not a NiceGUI issue, otherwise all 3 successful cases we've seen so far would be broken.

You really need to check your browser, and note that "The touchmove event only works on touch screens"


There may be tricky cases such as a touchscreen which simply drags a mouse across (occurs with some cheap Windows tablets on AliExpress, big interactive displays, or when over a Remote Desktop connection with a client which cannot send touch input. I have had the displeasure to encounter all 3 scenarios at least once)

@bin-san
Copy link
Author

bin-san commented Apr 5, 2025

Look at the event['touches']['0'] its an empty dict in the event printed in the terminal... likewise all the key storing touch position are empty dict, it should contain 'clientX' and 'clientY'

@evnchn
Copy link
Collaborator

evnchn commented Apr 5, 2025

Ah, so that's the core issue here. Thank you for pointing out that. 😅

But at least, we know the event handler "touchmove" is being triggered. Investigation follows.

@evnchn
Copy link
Collaborator

evnchn commented Apr 5, 2025

Now, if you are urgent for a solution, the following would work without any changes to NiceGUI:

from nicegui import ui 

label = ui.label('touch data')

ui.on('mytouch', lambda e: label.set_text(f"{e.args}"))

with ui.element('div').style('width:80vw;height:80vh; background-color:lightblue;') as toucharea:
    toucharea.on('touchmove', js_handler='(e) => { console.log(e); emitEvent("mytouch", e.touches[0]); }')

ui.run(port=8080)
Image

Still investigating...

@evnchn
Copy link
Collaborator

evnchn commented Apr 5, 2025

Well, that's a tough nut to crack:

Image

JSON.stringify doesn't stringify the Touch event

@evnchn
Copy link
Collaborator

evnchn commented Apr 5, 2025

Got a fix in #4579

But not sure if there are other tricky objects like Touch

But it won't make itself to release quickly, so js_handler is your best friend for now.

@falkoschindler
Copy link
Contributor

Note that the test code print(e) and label.set_text(f"{e.args['touches']['0']}") will never set the label text because print returns None.

Minimum reproduction:

ui.card().classes('w-80 h-80').on('touchmove', lambda e: print(e.args['touches']['0']))

@falkoschindler falkoschindler added bug Type/scope: A problem that needs fixing 🟡 medium Priority: Relevant, but not essential 🌿 intermediate Difficulty: Requires moderate understanding analysis Status: Requires team/community input labels Apr 6, 2025
@falkoschindler
Copy link
Contributor

Note that we can reproduce the problem with developer tools like this: https://stackoverflow.com/a/36877125/3419103

I wonder why there isn't any JavaScript error when serializing the Touch object. And how does it help to assembly the object like in PR #4579?

@evnchn
Copy link
Collaborator

evnchn commented Apr 6, 2025

When JSON.stringify serializes a Touch object, it just returns '{}'

#4579 patches the serialization by introducing advance_serialize which is known-good for Touch and putting it in the replacer for big JSON.stringify call.

@falkoschindler
Copy link
Contributor

Ok, I finally understand. With some debug code in stringifyEventArgs I reproduced the problem with seemingly unproblematic Touch objects:

const touch = filtered.touches[0];
const json = JSON.stringify(touch, (k, v) => (v instanceof Node || v instanceof Window ? undefined : v));
console.log(touch, json);

They look like regular objects with regular properties. But apparently the properties are "non-enumerable". Cursor explains it like this:

JSON.stringify() follows an algorithm similar to:

  1. It starts with the object's own enumerable properties
  2. It ignores properties that are functions, undefined, or symbols
  3. It handles circular references by throwing an error
  4. It processes other objects recursively following the same rules

This is why your Touch objects are serializing as empty objects ({}). The Touch interface properties are defined as non-enumerable in the DOM API, so JSON.stringify() skips them entirely even though you can access them directly with code like touch.clientX.
This behavior is actually consistent with other JavaScript enumeration mechanisms like for...in loops and Object.keys().

I'm still not very happy with a special treatment for Touch objects. But it looks like we have no other choice.

@evnchn
Copy link
Collaborator

evnchn commented Apr 11, 2025

I believe, though we may not be exactly 100% happy, we can't be more happier than the current solution which handles Touch objects specially.

If we implement a custom JSON.stringify which serializes all objects including tricky ones properly, the behavior may be wildly divergent from the JSON.stringify. Devs who would like to, given an input, figure out more-or-less how NiceGUI would serialize it, have no choice but to include the NiceGUI browser core JS and actually try it.

Now, we know, the serializer:

  • Matches JSON.stringify
  • Except when it's a Touch, where it dumps all keys

Predictable and simple. What more can you ask for.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
analysis Status: Requires team/community input bug Type/scope: A problem that needs fixing 🌿 intermediate Difficulty: Requires moderate understanding 🟡 medium Priority: Relevant, but not essential
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants