Add more information for pinch gestures on mobile

[brentfpage]

Description

• the SDL_PinchFingerEvent struct defined in include/SDL3/SDL_events.h has been expanded to include the midpoint of the finger positions comprising the pinch gesture (focus) as well as the x and y displacements between the finger positions (span). Including this information allows a programmer to implement a zoom in response to the gesture that a.) keeps focus centered on the screen and that b.) zooms the y axis more rapidly if the user's fingers are more vertically aligned.
• the updated declaration and definition of SDL_SendPinch in src/events/(SDL_touch_c.h and SDL_touch.c) accommodate this new focus and span information
• Capturing the additional pinch info in Java/Android
  • when a pinch event is detected in android-project/app/src/main/java/org/libsdl/app/SDLSurface.java , now the focus and span info is forwarded on to downstream functions in addition to scale, while previously only scale was forwarded. This forwarding function is specifically onNativePinchUpdate
  • the Java Native interface in android-project/app/src/main/java/org/libsdl/app/SDLActivity.java and src/core/android/SDL_android.c that connects onNativePinchUpdate to SDL_SendPinch has been expanded to include focus and span
• Calls to SDL_SendPinch in non-Android user interface frameworks
  • src/video/cocoa/SDL_cocoawindow.m
  • src/video/uikit/SDL_uikitview.m
  • src/video/wayland/SDL_waylandevents.c
  • src/video/x11/SDL_x11xinput2.c
    have been adjusted so that event.pinch.focus_x, event.pinch.focus_y, event.pinch.span_x, and event.pinch.span_y are set to zero by SDL_SendPinch. Existing uses of pinch events will only involve event.pinch.scale, so these changes are non-breaking. On the other hand, it may be surprising to a programmer that focus and span are zero considering that they exist. I am unfamiliar with these UI frameworks, so won't attempt to add focus and span info to their SDL_SendPinch calls, but some glances at these frameworks' APIs suggests this would be possible to some extent.

Coordinate system transformations:

In SDLSurface.java, the Android gesture detector returns span and focus info in screen pixel coordinates. For consistency with what is done in that same file for simple touch events, I convert span and focus to normalized screen coordinates before passing them on to SDL_SendPinch/onNativePinchUpdate. But, I convert them back to pixel screen coordinates in SDL_SendPinch, again to be consistent with what is done for touch events (in SDL_SendTouch)

To be precise, the pinch info transformations are very nearly inverses of one another, but for reasons I'm unfamiliar with, the original conversion to normalized screen coordinates divides by (pixel_count - 1) instead of (pixel_count), while the subsequent conversion back to screen pixel coordinates multiplies by pixel_count. The final change in coordinates by a multiplicative factor pixel_count/(pixel_count - 1) is minuscule for all devices.

[maia-s]

This breaks binary compatibility, because the windowID field of SDL_PinchFingerEvent changes offset. An app compiled against an older SDL would get a garbage window id if dynamically linked to an SDL library with this patch.

[brentfpage]

Does the newest commit fix the issue?

[sezero]

Does the newest commit fix the issue?

No: windowID should stay after scale

[brentfpage]

My bad – fixed now

[slouken]

This generally seems like a good idea. 0 could be a valid value, so we should probably use -1 if the value isn't available, and document that.

The other platforms should include some data, so let's try to fill those in if we can. @Kontrabant, do you have any insight here?

[Kontrabant]

• libinput, and thus XInput2 and Wayland, give the origin and the angle of the pinch, but no direct span information.
• It looks like Apple devices can return the touches associated with the gesture, which would allow calculating the origin and angle as well.

Would it make sense to send the angle of the gesture instead of the span? It would still fulfill the use case of zooming more rapidly on on one axis, and would work everywhere. Most likely, if the span were returned, anything wanting to use it for accelerating a zoom on one axis would end up deriving the angle anyway.

[brentfpage]

Would it make sense to send the angle of the gesture instead of the span? ... Most likely, if the span were returned, anything wanting to use it for accelerating a zoom on one axis would end up deriving the angle anyway.

I agree that would make sense. For my use case, I computed the cos/sin of the angle from the span information. So, it does not detract from the functionality if SDL returns the angle instead of the span info.

It looks like Apple devices can return the touches associated with the gesture, which would allow calculating the origin and angle as well.

I'll try to implement that.

[brentfpage]

Some wrinkles:

• the "rotation" that libinput provides is actually relative to the previous pinch event update. E.g., if the fingers start at 1:00 and 4:00 and move to 1:30 and 4:30, then rotation=15°
• libinput also provides dx and dy, but it turns out these report changes in the position of the "logical center" of the pinch gesture, also relative to the previous pinch event update
• it seems libinput doesn't provide access to the (x, y) of the "logical center" (although it looks to be possibly available in wayland?). This limitation I think is consistent with libinput's pinch gestures being designed for trackpad use. On a trackpad, essentially only relative motion is important.
• for x11 and wayland, I resorted to filling span_(x,y) and focus_(x,y) with a dummy value: -1. I also updated the doc of the SDL_PinchFingerEvent struct to explain this -1 value.

On the bright side:

• uikit includes touch locations in its pinch event objects. I used these locations to fill in the span_(x,y) and focus_(x,y) info in the calls to SDL_SendPinch for uikit. I decided to stick with using span_x and span_y because 1.) there's no platform that natively provides the angle and 2.) computing the angle would require the use of trig functions, which seems desirable to avoid if possible

Cocoa/AppKit also provides finger locations, but from the comments about NULL windows in the Cocoa/AppKit file that I listed, I gather that they're locations on a trackpad. Adding the associated 'span' and 'focus' info to SDL_PinchFingerEvent would compel use of 0->1 normalization for these values because then they would have different coordinated spaces (screen or touchpad) depending on the context. Is it worth it? Or maybe it's acceptable to use pixel screen coordinates for mobile/touchscreen contexts (uikit and android) and 0->1 touchpad coordinates for desktop contexts (Cocoa/AppKit)? For now, I set the Cocoa/Appkit span/focus values to the dummy value: -1.

[icculus]

These need to be added to the end of the struct, so we don't break ABI compatibility. Put them under windowID, please.

[icculus]

(you mentioned this was fixed...did it not get pushed?)

[brentfpage]

I don't know how that happened. I figure I checkout'ed an old commit without paying close enough attention. Fixed again now.

[slouken]

@brentfpage, you're planning to switch this to use angles which will enable this feature for X11 and so forth, right? @Kontrabant, am I reading that right?

[brentfpage]

@brentfpage, you're planning to switch this to use angles which will enable this feature for X11 and so forth, right?

I don't think it can be done. The angle associated with libinput pinch events is a rotation relative to the previous frame. I'm guessing that only this information is available because libinput is not intended for touchscreen use – just trackpad use. It may make sense to have separate pinch events for trackpads and touchscreens. It's not as though the current pinch API is written in stone – it was added something like last october if memory serves.

[slouken]

We should document that these values are only sent on mobile devices. Once that's done, @Kontrabant, do you have any more feedback? Do you think this is a good addition?

[Kontrabant]

It's unfortunate that libinput and desktop Macs work differently here, but this otherwise seems useful on mobile devices and looks good to me.

[brentfpage]

I made it clear now in the documentation for SDL_PinchFingerEvent that focus_(x/y) and span_(x/y) are only available on mobile devices, and that in other contexts they are set to -1.