Davide Spano
[email protected]
[email protected]
CNR-ISTI, HIIS Laboratory, Via G. Moruzzi 1
56124 Pisa, Italy
Gestures
Davide Spano
Gesture Interfaces
16 May 2012
Remotes (1)
 Nintendo Wiimote, 2006
 Buttons
 Three-axis linear accelerometer
 IR camera
 Three axis gyroscope extension (Wii Motion Plus)
 Game market opened to casual gamers
Davide Spano
Gesture Interfaces
16 May 2012
Remotes (2)
 Playstation Move, 2010
 Buttons
 Spherical RGB light emitter plus camera
 Three-axis accelerometer and gyroscope
 Compass
Davide Spano
Gesture Interfaces
16 May 2012
Floor Devices (1)
 Dance pads
 Konami Dance Dance Revolution (1998)
 Huge directional pads
 Buttons
Davide Spano
Gesture Interfaces
16 May 2012
Floor Devices (2)
 Wii Balance Board
 Four pressure sensors
 Snowboard games
 Fitness games
Davide Spano
Gesture Interfaces
16 May 2012
Motion Tracking (1)
 CamSpace, 2009
 Object recognition
 No haptic feedback
Davide Spano
Gesture Interfaces
16 May 2012
Motion Tracking (2)
 Microsoft Kinect, 2009
 RGB camera
 Depth sensor
 Microphone array
 Still no haptic feedback
Davide Spano
Gesture Interfaces
16 May 2012
Multitouch
 Two or more touch-sensed position on the screen
 Patents already on 1984
 iPhone 2007
 Support in Mac OS X, Windows 7, Ubuntu …
Davide Spano
Gesture Interfaces
16 May 2012
Gestures Overview
 Gesture types:
 Static
 Dynamic
 Spatio-temporal
 Conceptual:
 Gestures are continuous
 Gestures exploit time and space as arguments
 Gestures enable single user parallel interaction
 Need for a suitable programming paradigm
Davide Spano
Gesture Interfaces
16 May 2012
The Midas Touch
Out
Of
range
Mouse
Tracking
Touches,
Skeleton
Not
Tracking
Davide Spano
Engaged
Accelerometers,
Gyroscope
Tracking
Gesture Interfaces
Tracking
16 May 2012
Current Programming Model
 Do gestures really fit this pattern?
Davide Spano
Gesture Interfaces
16 May 2012
Multitouch Frameworks
Davide Spano
Gesture
Events
Touch
Events
Slide
Start
Pinch
Move
Rotate
End
Gesture Interfaces
16 May 2012
Example: Pinch for zoom (1)
Davide Spano
Gesture Interfaces
16 May 2012
Example: Pinch for zoom (2)
Davide Spano
Start
Move
Move
End
Gesture Interfaces
16 May 2012
Full body tracking
Davide Spano
Gesture Interfaces
16 May 2012
Microsoft Kinect SDK
Davide Spano
Gesture Interfaces
16 May 2012
Primesense NITE Controls
 State of the Art toolkit for full body gesture interaction
 Reusable controls (recognizers) for NUI
 Control list
 Push detector
 Swipe detector
 Steady detector
 Wave detector
 Circle detector
 Selectable slider (1D/2D)
 All based on the observer pattern
Davide Spano
Gesture Interfaces
16 May 2012
Circle Control Sample
Davide Spano
Gesture Interfaces
16 May 2012
One Event is not enough!
Handler
Gesture
Pan
Handler
Handler
Handler
1
1
1
Start
Samples
Davide Spano
Gesture Interfaces
Move
End
Touch ID
16 May 2012
Gesture Definition Meta Model
 Declarative
 Compositional
 Ground Terms (basic building blocks)
 Composition Operators
 Behaviour Association
 Entire Gesture
 Any other sub-component
 Abstract Gesture Recognition Support
 Formally defined through Petri Nets
Davide Spano
Gesture Interfaces
16 May 2012
Break…
http://www.foxtrot.com/
Davide Spano
Gesture Interfaces
16 May 2012
Gesture Support Abstraction
 Feature: an observable property
𝑓 ∈ ℝ𝑛
 Gesture Support: a set of features
𝐺𝑆 = 𝑓1 , 𝑓2 , … , 𝑓𝑚
𝑚
𝐺𝑆 ∈ ℝ𝑘 𝑓𝑖 ∈ ℝ𝑛𝑖
𝑛𝑖 = 𝑘
𝑖=0
 Gesture Support State: feature values at a given time
𝐺𝑆𝑖 = 𝑓1 𝑡𝑖 , 𝑓2 𝑡𝑖 , … , 𝑓𝑚 𝑡𝑖
𝑡𝑖 ∈ ℝ
 Gesture Support Sequence: gesture support state values
through the time
𝑆 = 𝐺𝑆1 , 𝐺𝑆 , … , 𝐺𝑆𝑛
𝑛 ∈ℕ
Davide Spano
Gesture Interfaces
16 May 2012
Ground Terms and Composition
Operators
 Low-level device events as ground terms
 Optional conditions on support state
 Composition operators
Davide Spano
Gesture Interfaces
16 May 2012
Multitouch Gesture Support
 Features:
 𝑝𝑖
2D position of n touches, in pixel coordinates
 If a touch is not currently detected 𝑝𝑖 = ⊥, ⊥
 𝑡𝑖𝑚𝑒 current detection time
 Multi-Touch recognition ground terms:
 𝑆𝑡𝑎𝑟𝑡𝑖 = 𝑝𝑖 [𝑝𝑖 (𝑡 − 1) = (⊥, ⊥) ∧ 𝑝𝑖 (𝑡) ≠ (⊥, ⊥)]
 𝑀𝑜𝑣𝑒𝑖 = 𝑝𝑖 [ 𝑝𝑖 (𝑡 − 1) ≠ (⊥, ⊥) ∧ 𝑝𝑖 (𝑡) ≠ (⊥, ⊥)]
 𝐸𝑛𝑑𝑖 = 𝑝𝑖 [ 𝑝𝑖 (𝑡 − 1) ≠ (⊥ , ⊥) ∧ 𝑝𝑖 (𝑡) = (⊥ , ⊥)]
Davide Spano
Gesture Interfaces
16 May 2012
Multitouch Gesture Description
 Tap
𝑆𝑡𝑎𝑟𝑡𝑖 >> 𝐸𝑛𝑑𝑖
 Double Tap
𝑆𝑡𝑎𝑟𝑡𝑖 ≫ 𝐸𝑛𝑑𝑖 ≫ 𝑆𝑡𝑎𝑟𝑡𝑖 𝑝𝑜𝑠 ∧ 𝑡𝑖𝑚𝑒𝐷𝑖𝑓𝑓 ≫ 𝐸𝑛𝑑𝑖
 Pan
 Slide
𝑆𝑡𝑎𝑟𝑡𝑖 ≫ 𝑀𝑜𝑣𝑒𝑖∗ [> 𝐸𝑛𝑑𝑖
𝑆𝑡𝑎𝑟𝑡𝑖 ≫ 𝑀𝑜𝑣𝑒𝑖∗ 𝑙𝑖𝑛𝑒𝑎𝑟 ∧ 𝑠𝑝𝑒𝑒𝑑 ≫ 𝐸𝑛𝑑𝑖
 Pinch
𝑆𝑡𝑎𝑟𝑡𝑖 |=|𝑆𝑡𝑎𝑟𝑡𝑗 ≫ 𝑀𝑜𝑣𝑒𝑖∗ || 𝑀𝑜𝑣𝑒𝑗∗
[> 𝐸𝑛𝑑𝑖 |=| 𝐸𝑛𝑑𝑗
 Rotate
𝑆𝑡𝑎𝑟𝑡𝑖 |=|𝑆𝑡𝑎𝑟𝑡𝑗 ≫
𝑀𝑜𝑣𝑒𝑖∗ [c]|| 𝑀𝑜𝑣𝑒𝑗∗ [𝑐] [> 𝐸𝑛𝑑𝑖 |=| 𝐸𝑛𝑑𝑗
Davide Spano
Gesture Interfaces
16 May 2012
Proton (1)
 Kin et al. Proton: Multitouch Gestures as Regular
Expressions, CHI 2012
 Multitouch Gestures Described as Regular Expression
 Ground Terms
𝑂𝑇𝑦𝑝𝑒
𝐸𝑇𝐼𝐷
𝐸 ∈ 𝐷, 𝑀, 𝑈
 Composition operators

| *
 Matcher for handling ambiguity
 One gesture at time assumption
 Tablature graphical representation for gestures
Davide Spano
Gesture Interfaces
16 May 2012
Proton (2)
Davide Spano
Gesture Interfaces
16 May 2012
Full Body Gesture Platform
 Skeleton tracked as a set of joints
 2 user full tracking + 4 only body centre
 Features:
 Joint (user id, joint id, position, orientation)
 Current time
Davide Spano
Gesture Interfaces
16 May 2012
Full Body Example (1)
Front
Davide Spano
Not Front
Gesture Interfaces
Right hand
Both hands
16 May 2012
Full Body Example
𝐹𝑟𝑜𝑛𝑡 ≫ 𝑅𝑖𝑔ℎ𝑡𝐻𝑎𝑛𝑑
Davide Spano
𝐵𝑜𝑡ℎ𝐻𝑎𝑛𝑑𝑠
Gesture Interfaces
∗
[> 𝑁𝑜𝑡𝐹𝑟𝑜𝑛𝑡
16 May 2012
A different programming model
 Describe declaratively the gesture
 Create new ones using the building blocks
 Reuse existing ones through composition
 Code the event handlers
 Attach them to the events at the desired granularity
 It is called only when it is needed (no further checks)
 Temporal description and interface behaviour are
completely separated
 Complex Gestures can be stored as classes
 They can still expose their low level events
Davide Spano
Gesture Interfaces
16 May 2012
Library Support
Davide Spano
Gesture Interfaces
16 May 2012
Multitouch with Full Body
Experiment
 Hands point tracking 𝑙 = (𝑥𝑙 , 𝑦𝑙 , 𝑧𝑙 )
𝑟 = (𝑥𝑟 , 𝑦𝑟 , 𝑧𝑟 )
 Basic Touches Description
 𝑆𝑡𝑎𝑟𝑡1 = 𝑟 𝑧𝑟 𝑡 − 1 > 𝑘 ∧ 𝑧𝑟 𝑡 ≤ 𝑘
 𝑆𝑡𝑎𝑟𝑡2 = 𝑙 𝑧𝑙 𝑡 − 1 > 𝑘 ∧ 𝑧𝑙 𝑡 ≤ 𝑘
 𝑀𝑜𝑣𝑒1 = 𝑟 𝑧𝑟 𝑡 − 1 ≤ 𝑘 ∧ 𝑧𝑟 𝑡 ≤ 𝑘
 𝑀𝑜𝑣𝑒2 = 𝑙 𝑧𝑙 𝑡 − 1 ≤ 𝑘 ∧ 𝑧𝑙 𝑡 ≤ 𝑘
 𝐸𝑛𝑑1 = 𝑟 𝑧𝑟 𝑡 − 1 ≤ 𝑘 ∧ 𝑧𝑟 𝑡 > 𝑘
 𝐸𝑛𝑑2 = 𝑙 𝑧𝑙 𝑡 − 1 ≤ 𝑘 ∧ 𝑧𝑙 𝑡 > 𝑘
Davide Spano
Gesture Interfaces
16 May 2012
Proof of Concept Applications
Davide Spano
Gesture Interfaces
16 May 2012
Demo Videos
 Parallel Interaction with iPhone video
 “Multitouch” with Kinect video
 Same Gesture Definition!
Davide Spano
Gesture Interfaces
16 May 2012
Conclusion
http://www.mojocomic.com/
Davide Spano
Gesture Interfaces
16 May 2012
References
 Spano, L.D. A Model Based Approach for gesture interfaces
(EICS 2011) pdf
 Kin K., Hartmann B., DeRose T., Agrawala M. Proton:
Multitouch Gestures as Regular Expressions pdf
 Kinect for Windows
 OpenNI
 Primesense NITE
Davide Spano
Gesture Interfaces
16 May 2012