Fix mouse grab behavior on Android #16203

PatrickStankard · 2024-02-06T20:25:19Z

Description

In #14016, @schellingb posted a branch (https://github.com/schellingb/RetroArch/tree/android_grab_mouse) that implemented a grab_mouse interface for Android. I forked and tested it extensively on an Android 10 device, and ended up refactoring the RetroActivityFuture a bit so that it worked as expected.

I ran into a few things: looking at the ADB logs, I noticed that, for some reason, enabling "Immersive Mode" and attempting to call requestPointerCapture on the window's decor view was inconsistently working. One of the only references I could find relating to this behavior was this ImmersiveActivity example from the Android source code, where they used a Handler to toggle the UI changes with a 300ms delay. I ended up doing the same thing in the RetroActivityFuture, using a Handler to toggle UI changes (including mouse capture and "immersion mode") with a 300ms delay, and it seems to work pretty well.

I also made the input_auto_mouse_grab config setting default to true, for Android only, based on the conversation in that issue. That way, mouse capture works by default, but you can still disable the setting if (for some reason) you don't want the mouse to be captured by RetroArch.

I tested this using the Mame 2003-Plus and FinalBurn Neo cores, directly through the RetroArch APK, and indirectly through the Daijishou front-end, which uses am start to launch the RetroActivityFuture activity. Everything works as expected!

~~Controlling the menu UI with a mouse in Android still doesn't seem to work, however - someone with more experience working in the RetroArch codebase might have an easier time fixing that.~~ Fixed in e60f04c

Related Issues

#14016

Reviewers

@schellingb - thank you very much for co-authoring this in your android_grab_mouse branch, and for implementing grab_mouse, otherwise I wouldn't have known where to start!

PatrickStankard · 2024-02-06T21:06:00Z

pkg/android/phoenix/src/com/retroarch/browser/retroactivity/RetroActivityFuture.java

-      // If HIDEMOUSE parameters is provided then hide the mourse cursor
-      // This requires NVIDIA Android extensions (available on NVIDIA Shield), if they are not
-      // available then nothing will be done
-      if (retro.hasExtra("HIDEMOUSE")) hideMouseCursor();


I forgot to mention - I removed this because it doesn't seem like it's being used anywhere, and it's not referenced anywhere in documentation or anything. This PR uses the input_auto_mouse_grab config value to determine if it should hide/capture the mouse, which I think is better anyway. Happy to restore it if anyone disagrees, though

schellingb · 2024-02-07T00:35:35Z

Hi there.

Thanks for pushing this forward! I kinda lost interest after seeing the remaining issues and the overall inconsistent behavior on Android with a mouse attached versus a desktop PC running RetroArch in fullscreen.

I also made the input_auto_mouse_grab config setting default to true, for Android only, based on the conversation in that issue. That way, mouse capture works by default, but you can still disable the setting if (for some reason) you don't want the mouse to be captured by RetroArch.

Is this a solution or a workaround?
I was hoping for no need for this as it isn't needed on desktop platforms.
Still, this is probably better than having to explain users to need to toggle the setting if they want to use a mouse. Although existing RetroArch installations won't get the setting switched automatically.

Controlling the menu UI with a mouse in Android still doesn't seem to work, however - someone with more experience working in the RetroArch codebase might have an easier time fixing that.

Last time I noticed the menu does react to mouse movement and clicks but it seems utterly confused about the mouse position or its movement. For me it just jerked the menu around when I tried it back then, is this still the same?

Does anyone know how other (non-desktop) platforms handle a mouse? I'm interested in how it runs in something like DRM/KMS, which is Linux without a desktop manager. For example, does Lakka on a RaspberryPi support a USB mouse to control the menu and games?

Edit: Also wanted to add thanks to @worleydl who initially took a stab at implementing the interface, otherwise I wouldn't have known where to start :-)

PatrickStankard · 2024-02-07T01:49:47Z

@schellingb I think this commit (e218a96) might have the behavior you're looking for, I'm gonna test it and let you know

PatrickStankard · 2024-02-07T08:44:16Z

@schellingb controlling the menu with a mouse works now, in this commit: cd47e27 🎉

The issue was that android_input_state wasn't handling RARCH_DEVICE_MOUSE_SCREEN.

I also tested the commit to always capture the mouse on focus, regardless of the input_auto_mouse_grab config setting (e218a96). It works, but I'm on the fence about having that setting essentially be a no-op for all non-windowed platforms. I'm worried about someone (for some reason) wanting to keep it disabled, and then this change would prevent that...

We could remove the config option entirely for non-windowed platforms as a part of this PR, but it feels like there's a lot going on here already. What do you think?

TyroneSlothrop · 2024-02-07T21:19:20Z

This is a long shot but is there anything specific that made the initial report Android-only aside from the device it was tested on? I believe the same issue exists on iOS and wondering if it could be ported: #15483 (I'm also willing to test).

PatrickStankard · 2024-02-07T22:22:37Z

@TyroneSlothrop unfortunately everything here is Android-specific (input drivers, frontend drivers, etc), so it can't be ported to iOS. Looking at the input drivers for Cocoa, it seems like iOS is set up for mouse support... I'll ask some questions in that issue you posted

LuanVSO · 2024-02-07T23:37:24Z

thank you for working on this.
just tested it and i have some feedback

the mouse sensitivity is way too low (on the menu atleast), even turning the android mouse speed setting to the max didn't help
touch input is still broken if you turn on "mouse support" on the menu (though it was already broken previously anyway)

Screen_Recording_20240207_200915_RetroArch.mp4

PatrickStankard · 2024-02-09T15:43:15Z

@LuanVSO thank you for the video - I'm trying to see if it's something I can do to speed up the cursor, but I haven't had any luck yet...

@schellingb I ended up reverting that "always capture the mouse regardless of the input_auto_mouse_grab config value" commit ( e218a96), the way it is setup now makes it easy when debugging mouse behavior like #16203 (comment)

PatrickStankard · 2024-02-10T02:19:49Z

@LuanVSO I think I figured out the pointer speed issue, can you give the build on this commit a try? a4bb625

PatrickStankard · 2024-02-10T02:23:19Z

input/drivers/android_input.c

 {
-   /* Adjust mouse speed based on ratio


I removed the code here because it didn't seem to accurately set the mouse speed - it made it significantly slower compared to what you'd expect it, based on the Android system mouse speed settings: #16203 (comment)

Since the user can adjust their mouse speed in Android settings, and/or (sometimes) within core options, I don't think it's really necessary

LuanVSO · 2024-02-10T02:42:42Z

@LuanVSO I think I figured out the pointer speed issue, can you give the build on this commit a try? a4bb625

Mouse speed is fixed now
touch behavior is better as well
even navigating the menu on samsung dex works great now

But

when the mouse is captured it can't go past about 60% of the screen horizontally
it appears that the position of the last touch event gets "saved" and the position the the cursor keeps fliping between the mouse position and the last touch event (it keeps clicking stuff too)

Screen_Recording_20240209_232858_RetroArch.mp4

PatrickStankard · 2024-02-10T04:10:37Z

@LuanVSO this build uses the actual width/height from the video resolution, give it a shot: 9a1c30a

I haven't looked into any of the touch behavior yet, I'll take a look at some point

LuanVSO · 2024-02-10T04:48:33Z

can move the mouse without issues now

LibretroAdmin · 2024-02-14T00:17:07Z

Hi, should we merge the PR in its current state?

LuanVSO · 2024-02-15T02:12:03Z

The mouse capture is good, the remaining issues are marginally related but weren't introduced by this pr.
I think it's fine to merge it.

PatrickStankard · 2024-02-15T12:23:24Z

@LibretroAdmin I’m good with merging it in it’s current state, I know @schellingb didn’t want the final result to depend on having the input_auto_mouse_grab config value, but I think we could try and make that change downstream from this PR. I think having it default to true for Android (going forward) is good enough for now, and people can always enable it themselves if they already have a config

Bluscream · 2024-03-04T14:43:01Z

I would love to have this so i can finally start enjoying some point and click adventures :)

Makes mouse grabbing and 'Game Focus' work on Android with a real mouse Properly handle relative mouse motion events on Android (SDK 28 and newer)

…300ms delay

…late_deltas

PatrickStankard · 2024-03-04T14:56:00Z

@LibretroAdmin I have been using this branch to play point-and-click DOS games and trackball games on Android for a few weeks now, and it works really well. I think it's good to be merged

i30817 · 2024-03-28T17:28:21Z

Honestly mouse should be captured by default in all platforms. It's the cause of a lot of issues in desktops too when using cores hiding the mouse in windowed RA or fake Fullscreen (vast majority of desktops these days) iirc. There are worse problems (saw a lot of funny behaviors in KMS\DRM Linux because it bypasses the hardware quirks library libinput usually loaded by gnome\window manager), but even if everything else is working, not having it on can cause bugs in those cases... as this pr showed for Android and others showed for Wayland (mouse capture was a no op in Wayland until a recent series of fix commits, so that 'just' allowed mouse capture to do its job IF turned on).

* Add grab_mouse interface for Android Makes mouse grabbing and 'Game Focus' work on Android with a real mouse Properly handle relative mouse motion events on Android (SDK 28 and newer) * Enable workflow_dispatch on CI Android * Update android_mouse_calculate_deltas callsites * Add RETRO_DEVICE_MOUSE to android_input_get_capabilities * Use Handler to trigger UI events (toggle mouse, immersive mode) with 300ms delay * Enable input_auto_mouse_grab by default for Android * Handle RARCH_DEVICE_MOUSE_SCREEN in Android input driver * Add android.hardware.type.pc to manifest * Don't attempt to set pointer speed via scaling in android_mouse_calculate_deltas * Keep x/y values within viewport resolution for screen mouse * Use video_driver_get_size to get width/height --------- Co-authored-by: Bernhard Schelling <14200249+schellingb@users.noreply.github.com>

PatrickStankard commented Feb 6, 2024

View reviewed changes

PatrickStankard force-pushed the android_grab_mouse branch from 0159e46 to e218a96 Compare February 7, 2024 01:42

PatrickStankard force-pushed the android_grab_mouse branch 7 times, most recently from 2ba7750 to cd47e27 Compare February 7, 2024 08:42

PatrickStankard force-pushed the android_grab_mouse branch from fb08393 to a4bb625 Compare February 10, 2024 02:13

PatrickStankard commented Feb 10, 2024

View reviewed changes

PatrickStankard force-pushed the android_grab_mouse branch from f1df68a to 9a1c30a Compare February 10, 2024 03:50

PatrickStankard force-pushed the android_grab_mouse branch from 9a1c30a to 4f4bdfe Compare February 20, 2024 20:25

schellingb added 2 commits March 4, 2024 09:54

Add grab_mouse interface for Android

717ef38

Makes mouse grabbing and 'Game Focus' work on Android with a real mouse Properly handle relative mouse motion events on Android (SDK 28 and newer)

Enable workflow_dispatch on CI Android

0018c19

PatrickStankard added 9 commits March 4, 2024 09:54

Update android_mouse_calculate_deltas callsites

d22f379

Add RETRO_DEVICE_MOUSE to android_input_get_capabilities

224834c

Use Handler to trigger UI events (toggle mouse, immersive mode) with …

eefc1ef

…300ms delay

Enable input_auto_mouse_grab by default for Android

a8e52fe

Handle RARCH_DEVICE_MOUSE_SCREEN in Android input driver

1156114

Add android.hardware.type.pc to manifest

6b22d29

Don't attempt to set pointer speed via scaling in android_mouse_calcu…

500280f

…late_deltas

Keep x/y values within viewport resolution for screen mouse

485bd4b

Use video_driver_get_size to get width/height

b5936db

PatrickStankard force-pushed the android_grab_mouse branch from 4f4bdfe to b5936db Compare March 4, 2024 14:54

LibretroAdmin merged commit 5452999 into libretro:master Mar 19, 2024
27 checks passed

PatrickStankard mentioned this pull request Mar 27, 2024

Mouse Grab not enabled for Android #14016

Open

PatrickStankard deleted the android_grab_mouse branch April 16, 2024 16:13

schellingb mentioned this pull request Jul 28, 2024

[Feature Request] physical mouse support for Android version? schellingb/dosbox-pure#522

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix mouse grab behavior on Android #16203

Fix mouse grab behavior on Android #16203

PatrickStankard commented Feb 6, 2024 •

edited

Loading

PatrickStankard Feb 6, 2024 •

edited

Loading

schellingb commented Feb 7, 2024 •

edited

Loading

PatrickStankard commented Feb 7, 2024

PatrickStankard commented Feb 7, 2024 •

edited

Loading

TyroneSlothrop commented Feb 7, 2024 •

edited

Loading

PatrickStankard commented Feb 7, 2024

LuanVSO commented Feb 7, 2024

PatrickStankard commented Feb 9, 2024 •

edited

Loading

PatrickStankard commented Feb 10, 2024

PatrickStankard Feb 10, 2024

LuanVSO commented Feb 10, 2024 •

edited

Loading

PatrickStankard commented Feb 10, 2024

LuanVSO commented Feb 10, 2024

LibretroAdmin commented Feb 14, 2024

LuanVSO commented Feb 15, 2024

PatrickStankard commented Feb 15, 2024

Bluscream commented Mar 4, 2024

PatrickStankard commented Mar 4, 2024

i30817 commented Mar 28, 2024 •

edited

Loading

Fix mouse grab behavior on Android #16203

Fix mouse grab behavior on Android #16203

Conversation

PatrickStankard commented Feb 6, 2024 • edited Loading

Description

Related Issues

Reviewers

PatrickStankard Feb 6, 2024 • edited Loading

Choose a reason for hiding this comment

schellingb commented Feb 7, 2024 • edited Loading

PatrickStankard commented Feb 7, 2024

PatrickStankard commented Feb 7, 2024 • edited Loading

TyroneSlothrop commented Feb 7, 2024 • edited Loading

PatrickStankard commented Feb 7, 2024

LuanVSO commented Feb 7, 2024

PatrickStankard commented Feb 9, 2024 • edited Loading

PatrickStankard commented Feb 10, 2024

PatrickStankard Feb 10, 2024

Choose a reason for hiding this comment

LuanVSO commented Feb 10, 2024 • edited Loading

PatrickStankard commented Feb 10, 2024

LuanVSO commented Feb 10, 2024

LibretroAdmin commented Feb 14, 2024

LuanVSO commented Feb 15, 2024

PatrickStankard commented Feb 15, 2024

Bluscream commented Mar 4, 2024

PatrickStankard commented Mar 4, 2024

i30817 commented Mar 28, 2024 • edited Loading

PatrickStankard commented Feb 6, 2024 •

edited

Loading

PatrickStankard Feb 6, 2024 •

edited

Loading

schellingb commented Feb 7, 2024 •

edited

Loading

PatrickStankard commented Feb 7, 2024 •

edited

Loading

TyroneSlothrop commented Feb 7, 2024 •

edited

Loading

PatrickStankard commented Feb 9, 2024 •

edited

Loading

LuanVSO commented Feb 10, 2024 •

edited

Loading

i30817 commented Mar 28, 2024 •

edited

Loading