Pointer up event is ignored when screen rotating, causing touch pointer to stay

[yume-chan]

• I have read the FAQ.
• I have searched in existing issues.

Environment

• OS: Windows 11
• scrcpy version: 2.4
• installation method: Windows release
• device model:
• Android version: [e.g. 10]

Describe the bug

When device screen is rotating, there is a time window when the client sends touch events with incorrect screen size. These touch events are ignored by the server due to the incorrect screen size.

Normally this is fine, although the pointer stays on device screen, next time the mouse button is pressed, it's replaced by the new state. Except when the client side has a touch screen, because each finger has a different ID, the finger with ignored touch up event will stay permanently on device screen, causing incorrect behavior in apps. The only solution is to restart Scrcpy.

This can happen for example when user taps the full screen button in a video player, causing the screen to rotate. Or by releasing finger while using Alt + R to rotate the screen.

[rom1v]

Thank you for the report. I will think about it.

[yume-chan]

Looking at #5894 I'm thinking how about sending all pointer events in portrait orientation.

When device rotates, the (effective) screen size (in portrait orientation) won't change, so the event can still be accepted. As long as the client's orientation information matches what it displays, the pointer positions are still correct even if orientations on client and server don't match.

However this would still not work when screen resolution really changes (like wm size or foldable devices). For wm size the proportional position should be same, but for foldable devices the absolute position should be same. (need to get screen physical size and counter the density change?)

[rom1v]

Looking at #5894 I'm thinking how about sending all pointer events in portrait orientation.

Semantically, it would probably be wrong.

If you click on the scrcpy window while the device is in portrait mode, but you inject the event when the actual device is in landscape mode, the click will occur at a "wrong" location (the activity window will have already rotated).

Better to ignore the click in that case I think.

[yume-chan]

When the client is in portrait mode, it sends touch coordinates as-is, but when server receives those events, it knows the device is in landscape mode, so it will transform (rotate) those coordinates before injecting to system.

The coordinates will reflect where the user touches on the screen, it doesn't care whether it's the correct UI element the user want. For example in this case, when user taps the fullscreen button and the screen rotates to landscape, the fullscreen button moves to another location and now the video player is under the finger, but the app will still receive several touch events to that location, before the user release their finger.

[rom1v]

The coordinates will reflect where the user touches on the screen, it doesn't care whether it's the correct UI element the user want.

This is true: typically, even without any rotation, if there's a high delay between the scrcpy window and the device, the click has been performed on a content "in the past".

But for the specific case where the device rotation has changed, the click performed on the old orientation are almost always "unexpected" if we inject them in the new orientation. That's why I think these should be ignored (even if in theory that can cause minor disagreements, like a click pressed but not released, but I think the state is reset by Android on rotation anyway). In any case, that was the reason why I decided ignored it in the first place.