Conceptio › Archive › Google Patents

Google Patentsopen access

Gesture based user interfaces, apparatuses and systems using eye tracking, head … — Perceptive Devices Llc (US11402902B2)

Perceptive Devices Llc · Google Patents

Google Patents · Patents · License: Open Access

patent, google patents, intellectual property, US11402902B2, Perceptive Devices Llc, Uday Parshionikar, en, 2022

ABSTRACT

Abstract

User interaction concepts, principles and algorithms for gestures involving facial expressions, motion or orientation of body parts, eye gaze, tightening muscles, mental activity, and other user actions are disclosed. User interaction concepts, principles and algorithms for enabling hands-free and voice-free interaction with electronic devices are disclosed. Apparatuses, systems, computer implementable methods, and non-transient computer storage media storing instructions, implementing the disclosed concepts, principles and algorithms are disclosed. Gestures for systems using eye gaze and head tracking that can be used with augmented, mixed or virtual reality, mobile or desktop computing are disclosed. Use of periods of limited activity and consecutive user actions in orthogonal axes is disclosed. Generation of command signals based on start and end triggers is disclosed. Methods for coarse as well as fine modification of objects are disclosed.

Description

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No. 16/726,350 filed Dec. 24, 2019 entitled âGESTURE BASED USER INTERFACES, APPARATUSES AND SYSTEMS USING EYE TRACKING, HEAD TRACKING, HAND TRACKING, FACIAL EXPRESSIONS AND OTHER USER ACTIONSâ; which is a continuation-in-part of U.S. patent application Ser. No. 15/921,632 filed Mar. 14, 2018 entitled âGESTURE CONTROL VIA EYE TRACKING, HEAD TRACKING, FACIAL EXPRESSIONS AND OTHER USER ACTIONSâ; which is a continuation-in-part of U.S. patent application Ser. No. 14/897,657 filed Dec. 11, 2015 entitled âSYSTEMS, METHODS, APPARATUSES, COMPUTER READABLE MEDIUM FOR CONTROLLING ELECTRONIC DEVICESâ, which claims priority to PCT Application Serial No. PCT/US14/43529, filed Jun. 20, 2014 entitled âSYSTEMS, METHODS, APPARATUSES, COMPUTER READABLE MEDIUM FOR CONTROLLING ELECTRONIC DEVICESâ, which claims priority to U.S. Provisional Patent Application Ser. No. 61/837,215, filed Jun. 20, 2013 entitled âMultipurpose Controllers using Sensors, Heuristics for User Intent, Computer Vision, Multiple OMDs, ODEs and POLAsâ, the disclosures of which are all expressly incorporated herein by reference for all they contain.

U.S. patent application Ser. No. 15/921,632 is also a continuation-in-part of U.S. patent application Ser. No. 15/469,456 filed Mar. 24, 2017 entitled âGESTURE BASED USER INTERFACES, APPARATUSES AND CONTROL SYSTEMSâ, which is a continuation-in-part of U.S. patent application Ser. No. 14/897,657 filed Dec. 11, 2015 entitled âSYSTEMS, METHODS, APPARATUSES, COMPUTER READABLE MEDIUM FOR CONTROLLING ELECTRONIC DEVICESâ, which claims priority to PCT Application Serial No. PCT/US14/43529, filed Jun. 20, 2014 entitled âSYSTEMS, METHODS, APPARATUSES, COMPUTER READABLE MEDIUM FOR CONTROLLING ELECTRONIC DEVICESâ, which claims priority to U.S. Provisional Patent Application Ser. No. 61/837,215, filed Jun. 20, 2013 entitled âMultipurpose Controllers using Sensors, Heuristics for User Intent, Computer Vision, Multiple OMDs, ODEs and POLAsâ, the disclosures of which are all expressly incorporated herein by reference for all they contain. U.S. patent application Ser. No. 15/469,456 also claims priority to U.S. Provisional Patent Application Ser. No. 62/313,042 filed on Mar. 24, 2016 entitled âGestures Based User Interfaces, Apparatuses and Control Systemsâ and U.S. Provisional Patent Application Ser. No. 62/427,006 filed on Nov. 28, 2016 entitled âGestures Based User Interfaces, Apparatuses and Control Systemsâ, the disclosures of which are all also expressly incorporated herein by reference for all they contain.

U.S. patent application Ser. No. 15/921,632 also claims priority to U.S. Provisional Patent Application Ser. No. 62/470,872 filed on Mar. 14, 2017 entitled âGestures Based User Interfaces, Apparatuses and Control Systemsâ, U.S. Provisional Patent Application Ser. No. 62/537,482 filed on Jul. 27, 2017 entitled âGestures Based User Interfaces, Apparatuses and Control Systemsâ, U.S. Provisional Patent Application Ser. No. 62/589,228 filed on Nov. 21, 2017 entitled âGestures Based User Interfaces, Apparatuses and Control Systemsâ, U.S. Provisional Patent Application Ser. No. 62/626,253 filed on Feb. 5, 2018 entitled âGestures Based User Interfaces, Apparatuses and Control Systemsâ, and U.S. Provisional Patent Application Ser. No. 62/630,253 filed on Feb. 14, 2018 entitled âGestures Based User Interfaces, Apparatuses and Control Systemsâ, the disclosures of which are all also expressly incorporated herein by reference for all they contain.

U.S. patent application Ser. No. 16/726,350 is also a continuation-in-part of U.S. patent application Ser. No. 16/201,776 filed Nov. 27, 2018 entitled âGESTURE BASED USER INTERFACES, APPARATUSES AND CONTROL SYSTEMSâ, which is a continuation-in-part of U.S. patent application Ser. No. 14/897,657 filed Dec. 11, 2015 entitled âSYSTEMS, METHODS, APPARATUSES, COMPUTER READABLE MEDIUM FOR CONTROLLING ELECTRONIC DEVICESâ, which claims priority to PCT Application Serial No. PCT/US14/43529, filed Jun. 20, 2014 entitled âSYSTEMS, METHODS, APPARATUSES, COMPUTER READABLE MEDIUM FOR CONTROLLING ELECTRONIC DEVICESâ, which claims priority to U.S. Provisional Patent Application Ser. No. 61/837,215, filed Jun. 20, 2013 entitled âMultipurpose Controllers using Sensors, Heuristics for User Intent, Computer Vision, Multiple OMDs, ODEs and POLAsâ, the disclosures of which are all expressly incorporated herein by reference for all that they contain.

U.S. patent application Ser. No. 16/201,776 is also a continuation-in-part of U.S. patent application Ser. No. 15/469,456 filed Mar. 24, 2017 entitled âGESTURE BASED USER INTERFACES, APPARATUSES AND CONTROL SYSTEMSâ, which is a continuation-in-part of U.S. patent application Ser. No. 14/897,657 filed Dec. 11, 2015 entitled âSYSTEMS, METHODS, APPARATUSES, COMPUTER READABLE MEDIUM FOR CONTROLLING ELECTRONIC DEVICESâ, which claims priority to PCT Application Serial No. PCT/US14/43529, filed Jun. 20, 2014 entitled âSYSTEMS, METHODS, APPARATUSES, COMPUTER READABLE MEDIUM FOR CONTROLLING ELECTRONIC DEVICESâ, which claims priority to U.S. Provisional Patent Application Ser. No. 61/837,215, filed Jun. 20, 2013 entitled âMultipurpose Controllers using Sensors, Heuristics for User Intent, Computer Vision, Multiple OMDs, ODEs and POLAsâ. U.S. patent application Ser. No. 15/469,456 also claims priority to U.S. Provisional Patent Application Ser. No. 62/313,042 filed on Mar. 24, 2016 entitled âGestures Based User Interfaces, Apparatuses and Control Systemsâ and U.S. Provisional Patent Application Ser. No. 62/427,006 filed on Nov. 28, 2016 entitled âGestures Based User Interfaces, Apparatuses and Control Systemsâ, the disclosures of which are all also expressly incorporated herein by reference for all that they contain.

U.S. patent application Ser. No. 16/201,776 also claims priority to U.S. Provisional Patent Application Ser. No. 62/626,253 filed on Feb. 5, 2018 entitled âGestures Based User Interfaces, Apparatuses and Control Systemsâ, and U.S. Provisional Patent Application Ser. No. 62/630,253 filed on Feb. 14, 2018 entitled âGestures Based User Interfaces, Apparatuses and Control Systemsâ, the disclosures of which are all also expressly incorporated herein by reference for all that they contain.

This disclosure is related to U.S. patent application Ser. No. 13/418,331 filed Mar. 12, 2012 entitled âMultipurpose Controller for Electronic Devices, Facial Expressions Management and Drowsiness Detectionâ, U.S. patent application Ser. No. 14/054,789 filed Oct. 15, 2013 entitled âMultipurpose Controllers and Methodsâ, and U.S. patent application Ser. No. 15/695,283 filed Sep. 5, 2017 entitled âMultipurpose controllers and methodsâ, the disclosures of which are all hereby expressly incorporated by reference for all that they contain.

Any information in any material (e.g., a United States patent, United States patent application, book, article, etc.) that has been incorporated by reference herein, is only incorporated by reference to the extent that no conflict exists between such information and the other statements and drawings set forth herein. In the event of such conflict, including a conflict that would render invalid any claim herein or seeking priority hereto, then any such conflicting information in such incorporated by reference material is specifically not incorporated by reference herein.

BACKGROUND

Efforts have been made for many years to provide diverse means of controlling/communicating with electronic devices. Some of the means of control involve use of controllers to control/communicate with electronic devices. Other means/methods seek to eliminate the need to hold and/or touch controllers to control electronic devices. They involve communicating intent by means of gestures performed using hands, arms, legs, face and other body parts. Voice commands can also be used to communicate with electronic devices. Communication via brain waves is also possible. Each of these methods have limitations, however, one of the common concerns can be detecting and/or confirming user intention behind actions performed by the user of the electronic device(s).

SUMMARY

This application includes disclosure of methods, systems, apparatuses as well as principles/algorithms that can be implemented using computer executable instructions stored on computer readable mediums, for defining user gestures, performing user gestures, interpreting user actions, detecting user intent, confirming user intent and communicating user intent when communicating with electronic devices. A method of representation of user gestures via a symbolic language is also disclosed. Disclosed user gestures include user actions that can involve actions using eyes, head, facial expression, fingers, hands, arms, and other parts of body, verbal actions and mental actions that can be detected by monitoring brain waves. Many of the disclosed principles can enable hands-free and/or voice-free control of devices including those used in the fields of accessibility, Augmented/Mixed/Virtual Reality, gaming, desktop and mobile computing, and others. However, the disclosures are not limited to hands-free or voice-free principles of control over electronic devices. Multiple principles, concepts and user gestures are disclosed that allow for quick and large motions of OOI via eye gaze, as well as precise motions and accurate placement of OOI using other user actions including head motion and hand gestures are disclosed.

Concept of TMB (Time and Magnitude Bounded) user actions including motions, positions, expressions and other actions is disclosed. Use of TMB user actions for conveying and detecting user intent is disclosed.

Concept of Modifier Action is disclosed. A designated modifier action performed just prior to a user gesture can change the interpretation of that user gesture. For example, a user gesture for Left Click command when preceded by a specified âRâ action, generates a Right Click instead. The designated Body motion or position in substantially one particular axis before a user gesture for one type of click causes a different type of click. Click gesture can comprise a TMB facial expression. Body motion can be head motion, possibly with time and magnitude bounds and possibly preceded by a POLA. The modifier action can be a body motion that is unidirectional or in form of a shape that can be open or closed or in shape of letter of alphabet and can be performed clockwise or anticlockwise.

A user gesture for a swipe command is disclosed. A user gesture for a swipe command can comprise a TMB motion or position of a body part, possibly followed by a period of No Motion (possibly of minimum duration) occurring within designated time period. The body part can be head. The direction of swipe can be in accordance to the direction of the motion or position of the body part.

Use of sequential TMB user actions (such as motions or positions) in orthogonal direction in user gestures is disclosed. Combination of TMB motion or position actions in orthogonal axes, performed sequentially, can lead to generation of command signals. These combinations can be followed by POLA. There can be a POLA between some of the consecutive TMB actions (that are performed along orthogonal axes). There can be VLWPs between some of the consecutive TMB actions (that are performed along orthogonal axes).

Moving back or forth in the X axis can cause a Zoom in or out command signals to be generated, if a designated user action is detected to be active during the translational motion. The designated user action can be a facial expression. Rotating the head can also generate Zoom in/out command signals, if a designated user action is detection to be active during the head rotations.

A generic user gesture for manipulations of an Object of Interest (OOI) is disclosed. A head rotation or translation performed by the user can cause rotation or translation of the OOI on a display screen, possibly when performed upon detection of a designated trigger user action. The designated trigger user action can be a facial expression, and can be followed by a FLBP and that can be further followed by a period of No Motion. The designated trigger user action can also be tensing of designated muscles.

Note: In this document, the term âdisplay screenâ can refer to a physical display screen as well as any mechanism (such as a retinal projection mechanism) used to display virtual objects in a virtual 2D, 3D or multi-dimensional space that can be seen by the user.

Concept of Gesture Wake up Sequences (GWS) is disclosed. GWS can be used to activate the processing of certain designated target user gestures in a control system. These GWS's can be as simple as a period of No Motion, or a POLA, possibly combined with a VLWP (possibly with designated time bounds), or can be any suitable sequence of user actions. This VLWP can possibly wait for the first action of a previously defined target user gesture that needs to be processed by the system. GWS can be performed before a defined target user gesture that needs processing. After a target user gesture's processing is complete, the control system can stop processing other gestures that need a GWS, until another GWS is encountered. Some GWS can be composed of a TMB user action, optionally by a GWS and a POLA. Requirement to perform GWS before certain user gestures can be automatically imposed by the system based on ambient conditions, such as nature and pattern of motions experienced by the user or controller in conditions.

Concept of Session Wake up Sequences is disclosed. Certain user gestures can be used as Session Wake up Sequences (SWS) wherein there are used to start processing of other user gestures used to generate command signals. Once a SWS is performed, the control system can process user gestures for a designated amount of time from the time when the SWS was performed, and/or for at least designated amount of time from start/end of the SWS or start/end of the last user gesture processed once this SWS was performed.

Concept of Modes is disclosed. The command signals generated by the control system in response to performance of a particular user gesture can change based the active mode. Different sequences of user actions can be used to activate (start) or deactivate (end) a control system mode.

Use of a TMB motions performed with the head in Yaw, Pitch or Roll axis is disclosed for use of start of generating signals for modification of an object of interest. User gestures using Roll action in start triggers disclosed. User gestures without Roll as part of start triggers also disclosed.

Use of POLAs in ascertaining user intent behind user actions is disclosed.

Use of âLâ shaped gestures disclosed. Use of insertion of an orthogonal action to an existing user gesture or sequence of user actions is disclosed. Use of orthogonal actions to start definition of user gestures disclosed. Starting and ending user gestures with two or more actions that are in orthogonal axes is disclosed, possibly preceded or followed by a POLA. Embodiments that insert a POLA, FLBP, VLWP between the orthogonal actions are disclosed.

Use of user gestures comprising head position or motion along with eye gaze based control is disclosed. Use of facial expressions along with eye gaze based control system is disclosed. Activation of OOI Motion based on eye blink or wink in an eye gaze based control system is also disclosed.

Concept of PCE/PCM Stickiness, Dwell Park and OOI Stickiness is disclosed. User feedback on Dwell Park and OOI Stickiness is disclosed. OOI Motion/Modification Disabling Events (ODE) to stop generation of command signals for modification of an OOI is disclosed.

Use of POLAs as start as well as end triggers is disclosed. Method for provision of user feedback related to performance of various user actions in a user gesture, including level of detected user action, status of POLA, detection status of various body parts being tracker, and level of PCE/PCM, is disclosed. This includes visual feedback around the OOI.

CROSS REFERENCE TO RELATED APPLICATIONS

BACKGROUND

SUMMARY

Use of POLAs in ascertaining user intent behind user actions is disclosed.

Principles in definition and use of steady eye gaze before and during performance of other user actions, as a confirmation of user intent of those user actions, are disclosed. Eye gaze steadiness can be measured using a combination of displacement of the point of interest on the display screen, displacement of the eye gaze vector, magnitude of velocity of the point of interest on the display screen and magnitude of velocity of the eye gaze vector.

Concept of warping an Object of Interest (OOI) is disclosed. The warping can be based on combination of head motion, facial expressions, hand gestures, and any other user actions.

Concept of Post Warp Period (PWP) is disclosed. Use of additional OOI Modification Driver (OMD) actions in PWP is disclosed. Conditional use of OMD based on factors such as change in eye gaze, presence/absence of active facial expressions, programmatic states, input mechanisms' state, and other user actions is disclosed.

Variations related to measurement of change in eye gaze are disclosed. Iteration based calculations for change in eye gaze is disclosed. Calculation of change in eye gaze based on a designated event, wherein designated event can include OOI warp, motion of OOI and other suitable actions, is disclosed.

Combination of multiple user actions in formation of OOI Warp start triggers is disclosed, including combination of head motion and eye gaze displacement.

OOI Warping without PWP phase is disclosed.

Concept of chained OOI warping wherein an end trigger of one warp serves as the start trigger of a subsequent warp, is disclosed.

OOI Warping based on Hand Gestures and OOI Modification Signals based on Hand Gestures is disclosed. Changing hand gestures during PWP is disclosed. Influence of changes in hand gesture on OOI Modification Signals during the Post Warp Period is disclosed.

Generation of Helper Signals (including Zoom signals) during Post Warp Period is disclosed.

Gestures made using eyes are disclosed.

Enabling dwell clicking, wink/blink clicking based on facial expressions is disclosed.

Detection of accidental selections is disclosed.

POLA based user gestures providing option to select from multiple commands is disclosed.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a Head Coordinate System;

FIG. 2 illustrates an exemplary User Gesture Recognition Process Flowchart in one Embodiment;

FIG. 3A illustrates exemplary Body Actions Represented by Symbols including âY>â (Right Yaw), âP>â (Down Pitch) and in particular shows an example of motion/position of a part of the user's body as experienced/detected by the sensors that can map to the âY>P>â representation; FIG. 3B illustrates exemplary Body Actions Represented by Symbols including âY>â, âP>â and â#â (time periods of motion where the (absolute) magnitudes of specified motion types are continuously within corresponding specified motion thresholds/ranges) and in particular shows an observed motion pattern for a user gesture that can map to the âY>#P>â representation;

FIG. 4 illustrates exemplary Time Bound User Actions Represented by Symbols Including â[<S>]â;

FIG. 5 illustrates exemplary Time and Magnitude Bounded User Actions;

FIG. 6 illustrates exemplary Periods of No Motion;

FIG. 7 illustrates an exemplary User Action Pattern Matching â{YP}â;

FIG. 8A illustrates exemplary Y (yaw) and R (roll) Body Action Patterns over time to illustrate how VLWP (represented by the symbol âËâ) works, and in particular shows a pictorial representation of a portion of âYËRâ user gesture; FIG. 8B illustrates exemplary Y and R Body Action Patterns over time to illustrate how VLWP works, and in particular shows a pictorial representation where the R value falls outside the MNT range within the time bounds for VLWP, âËâ; FIG. 8C illustrates exemplary Y and R Body Action Patterns over time to illustrate how VLWP works, and in particular shows a pictorial representation where the R value does not fall outside the MNT range within the time bounds for VLWP, âHâ;

FIG. 9 illustrates an exemplary Direction of Eye Gaze versus Direction of Head Pointing in an Embodiment;

FIG. 10 illustrates an exemplary embodiment Using POLAs for Start as well as End Triggers while Generating Signals, and in particular part (a) of this Figure illustrates Body Motion over time that can be used to determine if the user is performing or not performing a POLA with the designated body part; part (b) of this Figure illustrates OOI Motion/Modification signals that can start being generated in accordance to the Body Motion; and part (c) of this Figure illustrates a further variation where while the OOI Modification signals are in accordance to the Body Motion, their magnitude is not directly proportional to the Body Motion;

FIG. 11 illustrates an exemplary embodiment of Visual Feedback Provided to User including Body Part Detection Status Indicator, PCE Level Indicator and Stop/Dwell Indicator;

FIG. 12 illustrates an exemplary embodiment of Visual Feedback Provided to User including Body Part Detection Status Indicator, PCE/PCM Level Indicator, Stop/Dwell Indicator and User Action Indicators;

FIG. 13 illustrates an exemplary embodiment of Visual Feedback Provided to User including Body Part Detection Status Indicator, PCE/PCM Level Indicator and User Action Indicators when Monitored Body Part is in Motion;

FIG. 14 illustrates an exemplary embodiment of Visual Feedback Provided to User by Changing the OOI (for Indicating Level of PCE/PCM);

FIG. 15 illustrates an exemplary embodiment of Visual Feedback Provided to User by Changing the OOI (for Indicating Level of PCE/PCM and Body Motion);

FIG. 16A illustrates an exemplary embodiment of OOI Stickiness Indicator with no indicator around the OOI when the magnitude of body motion is below the MNT; FIG. 16B illustrates an exemplary embodiment of OOI Stickiness Indicator with a circular indicator around the OOI when the magnitude of detected body motion is higher than the MNT, but still quite small compared to the start motion threshold; FIG. 16C illustrates an exemplary embodiment of OOI Stickiness Indicator with a more prominent visual indicator than FIG. 16B around the OOI, indicating higher detected magnitude of body motion but still lower than the start motion threshold; FIG. 16D illustrates an exemplary embodiment of OOI Stickiness Indicator with a more prominent visual indicator than FIG. 16C around the OOI, indicating higher detected magnitude of body motion but still lower than the start motion threshold; FIG. 16E illustrates an exemplary embodiment of OOI Stickiness Indicator with a more prominent visual indicator than FIG. 16D around the OOI, indicating higher detected magnitude of body motion but still lower than the start motion threshold; FIG. 16F illustrates an exemplary embodiment of OOI Stickiness Indicator with a full ring visual indicator when body motion equals or exceeds the start motion threshold, indicating the start of OOI motion signal generation;

FIG. 17A illustrates an exemplary embodiment of Dwell Park Progress Meter with a full ring around the OOI when the OOI is in motion and the magnitude of the body motion is above the end motion threshold; FIG. 17B illustrates an exemplary embodiment of Dwell Park Progress Meter where the size of the visual indicator is smaller than FIG. 17A indicating the body motion fell below the end motion threshold and continues to stay below it; FIG. 17C illustrates an exemplary embodiment of Dwell Park Progress Meter where the size of the visual indicator is smaller than FIG. 17B indicating the body motion continues to stay below the end motion threshold for a longer duration than in FIG. 17B ; FIG. 17D illustrates an exemplary embodiment of Dwell Park Progress Meter where the size of the visual indicator is smaller than FIG. 17C indicating the body motion continues to stay below the end motion threshold for a longer duration than in FIG. 17C ; FIG. 17E illustrates an exemplary embodiment of Dwell Park Progress Meter where the size of the visual indicator is smaller than FIG. 17D indicating the body motion continues to stay below the end motion threshold for a longer duration than in FIG. 17D ; and FIG. 17F illustrates an exemplary embodiment of Dwell Park Progress Meter where the visual indicator disappears indicating the body motion continues to stay below the end motion threshold for a duration equal to or longer than required to complete a POLA;

FIG. 18 illustrates an exemplary embodiment showing Points of Interest on a User's Face;

FIG. 19 illustrates an exemplary embodiment showing Bounding Box of the Mouth of the User;

FIG. 20 illustrates an exemplary embodiment Schematic Representation of Shape of the User's Mouth;

FIG. 21 illustrates an exemplary embodiment showing Schematic Representation of the User's Hand Attempting to Make a Pointing Gesture;

FIG. 22 illustrates an exemplary embodiment of a Controller Worn by the User;

FIG. 23 illustrates an exemplary embodiment of a Controller that can be Worn by the User;

FIG. 24 illustrates an exemplary embodiment of a Controller wherein the Controller is comprised in a Head-Worn Device;

FIG. 25 illustrates an exemplary Flow Diagram of Operation for an embodiment of a Controller; and

FIG. 26 illustrates a Schematic Layout of Functional Components of an exemplary Controller embodiment.

FIG. 27 illustrates an exemplary embodiment that uses the concept of PCE Stickiness and POLAs as ODE.

FIG. 28 illustrates an exemplary Eye Gaze Tolerance zone for measuring steadiness of Eye Gaze, wherein the Eye Gaze Tolerance zone is centered at the CPOI.

FIG. 29 illustrates an exemplary variation of Smile Facial Expression in relation to steadiness measurement of user's Eye Gaze.

FIG. 30 illustrates an exemplary Eye Gaze Tolerance zone for measuring steadiness of Eye Gaze, wherein the Eye Gaze Tolerance zone is centered at the location of OOI.

FIG. 31 illustrates an exemplary Head Motion pattern to trigger OOI Warp based on Head Motion in an embodiment.

FIG. 32 illustrates an exemplary OOI Warps based on Blinks in an embodiment.

FIG. 33 illustrates an exemplary OOI Modification Signal Generation based on detection of an Active Facial Expression.

FIG. 34 illustrates an exemplary generation of OOI Modification signals in accordance to Head Motion versus Eye Gaze based on CEGV Threshold, in some embodiments.

FIG. 35 illustrates an exemplary generation of OOI Modification signals in accordance to Head Motion versus Eye Gaze based on active FE active status and Mouse Button status, in some embodiments.

FIG. 36 illustrates an exemplary ebCEGV Tolerance Zone, wherein designated event is the last OOI Warp and the ebCEGV Tolerance Zone is centered around CPOI at the designated event.

FIG. 37 illustrates an exemplary ebCEGV Tolerance Zone, wherein designated event is motion of OOI and the ebCEGV Tolerance Zone is centered around the last location of the OOI.

FIG. 38A illustrates an exemplary ebCEGV Tolerance Zone, wherein designated event is motion of OOI and the ebCEGV Tolerance Zone is centered around the last location of the OOI and wherein the OOI Warp Start Trigger is based on combination of Head Motion and Eye gaze, and in particular shows the CPOI (Calculated Point of Interest) for iteration âi+1â is outside the ebCEGV Tolerance Zone;

FIG. 38B illustrates an exemplary ebCEGV Tolerance Zone, wherein designated event is motion of OOI and the ebCEGV Tolerance Zone is centered around the last location of the OOI and wherein the OOI Warp Start Trigger is based on combination of Head Motion and Eye gaze, and in particular shows the CPOI for iteration âi+1â is within the ebCEGV Tolerance Zone.

FIG. 39 illustrates an exemplary Head Motion and corresponding OOI Modification Signals based on the principle of OOI warping, wherein there is no Post Warp Period.

FIG. 40A illustrates an exemplary embodiment where the OOI Warp Start Trigger is based on a Hand Gesture, and wherein OOI Modification Signals during the Post Warp Period are also generated based on a Hand Gesture;

FIG. 40B illustrates an exemplary embodiment where the OOI Warp Start Trigger is based on a Hand Gesture, and wherein OOI Modification Signals during the Post Warp Period are also generated based on a Hand Gesture and shows motion of user's hand between iterations âiâ and âi+1â.

FIG. 41A illustrates an exemplary embodiment using multiple Hand Gestures as OOI Warp Start Triggers and OMD during Post Warp Period;

FIG. 41B shows an exemplary formula used to calculate OOI motion when user performs a hand gesture with only index finger raised;

FIG. 41C shows an exemplary formula used to calculate OOI motion when user performs a hand gesture with two fingers raised;

FIG. 41D shows an exemplary formula used to calculate OOI motion when user performs a hand gesture with three fingers raised;

FIG. 41E shows an exemplary formula used to calculate OOI motion when user performs a hand gesture with four fingers raised;

FIG. 41F shows an exemplary formula used to calculate OOI motion when user performs a hand gesture with five fingers raised.

FIG. 42 illustrates Eye Gaze Calculation Error in an exemplary embodiment, at the moment of OOI Warp.

FIG. 43 illustrates an exemplary Facial Expression Signals and OOI Modification signals in an embodiment that generates Helper Signals (PMA Zoom signals) during PWP.

FIG. 44A illustrates an exemplary Progressive Magnification Area (PMA) on a display screen for an embodiment that generates Helper Signals (PMA Zoom signals) during PWP before the helper signals have started to be generated;

FIG. 44B illustrates the exemplary PMA of FIG. 44A wherein the generated PMA Zoom signals caused 1.5Ã magnification of the PMA;

FIG. 44C illustrates the exemplary PMA of FIGS. 44A and 44B wherein the generated PMA Zoom signals caused further magnification of the PMA;

FIG. 44D illustrates the exemplary PMA of FIGS. 44A-44C wherein the generated PMA Zoom signals caused still further magnification of the PMA.

FIG. 45 illustrates an exemplary embodiment in the form of wearable Augmented/Mixed Reality Eye Glasses that uses diverse user actions including Eye gaze, Head motion and Hand gestures.

FIG. 46 illustrates an exemplary embodiment of a Control System that uses diverse user actions including Eye gaze, Head motion and Hand gestures without the need for wearables.

FIG. 47 depicts OOI Motion heuristics with the Eyeball/Gaze tracking in an embodiment of the controller.

FIG. 48 depicts Click and Drag heuristics with the Eyeball/Gaze tracking in an embodiment.

FIG. 49 depicts âPCE Falling Too Fastâ heuristics in an embodiment.

FIG. 50 depicts âPCE Rising Againâ heuristics in an embodiment.

FIG. 51 shows an example of the control flow that can be included in the Control Software of an embodiment of the Controller.

FIG. 52 shows an example of Heuristics of POLA based Multi-Command gesture in an embodiment of the Controllerâ Part 1.

FIG. 53 shows an example of Heuristics of POLA based Multi-Command gesture in an embodiment of the Controllerâ Part 2.

FIG. 54 shows an example Options Menu in an example of Heuristics of POLA based Multi-Command gesture in an embodiment of the Controller.

FIG. 55 shows another example Options Menu in an example of Heuristics of POLA based Multi-Command gesture in an embodiment of the Controller.

FIG. 56 shows an example of Heuristics of POLA based Multi-Command gesture in an embodiment of the Controllerâ Part 3.

FIG. 57 shows an example of Heuristics of POLA based Multi-Command gesture in an embodiment of the Controllerâ Part 4.

FIG. 58 shows an example of Heuristics of POLA based Multi-Command gesture in an embodiment of the Controllerâ Part 5.

FIG. 59 shows an illustrative example of a POLA in an embodiment of the controller.

FIG. 60 depicts an illustrative example of PCE Stickiness and use of POLAs as ODE in an embodiment of the controller.

FIG. 61 shows an illustrative depiction of an embodiment using POLA as a mechanism for detecting user intent.

FIG. 62 shows an exemplary illustration of an embodiment when the OOI motion is dependent on both the Eye gaze as well as Head motion as the OMD.

LIST OF TABLES

Table 1âAn illustrative Embodiment of Gesture based User Interface (that can be used as part of a Control System).

Table 2âIllustration of Easy Motion ModeâFirst Embodiment.

Table 3âIllustration of Easy Motion ModeâSecond Embodiment.

Table 4âExemplary Embodiments of Start Trigger (that can be used to start generation of OOI Attribute Modification signals).

Table 5âAn illustrative embodiment of gestures based User Interface that can be implemented without the use of a PCE or PCM.

Table 6âEmbodiment of a User Interface using User Gestures with Prominence of Roll Motion/Position Actions.

Table 7âEmbodiment of a User Interface using User Gestures that can be used with Smart Glasses and other Head Worn Devices (including but not limited to Head/Ear Phones, Ear Buds, Eye Wear, Augmented Reality or Virtual Reality Devices), as well as other Wearables (such as wrist bands) as well as Hand Held controllers.

DETAILED DESCRIPTION

The embodiments of the present invention described below are not intended to be exhaustive or to limit the invention to the precise forms disclosed in the following detailed description. Rather, the embodiments are chosen and described so that others skilled in the art may appreciate and understand the principles and practices of the present invention.

While exemplary embodiments incorporating the principles of the present invention have been disclosed herein above, the present invention is not limited to the disclosed embodiments. Instead, this application is intended to cover any variations, uses, or adaptations of the invention using its general principles. Further, this application is intended to cover such departures from the present disclosure as come within known or customary practice in the art to which this invention pertains.

The term âelectronic deviceâ is used to designate any devices that can have a microprocessor and that can be communicated with. A microprocessor can include one or more processors, memory and programmable input/output peripherals. A controller can include one or more microprocessors and/or memory with instructions that can help control or communicate with electronic devices.

This document discloses user interface concepts, principles and techniques that can be translated into software algorithms to provide a rich functionality, convenience, flexibility and ease-of-use to users. Further, the disclosed concepts/principles/techniques can lead to easier implementation of the gesture recognition algorithms. Note that these concepts, techniques and principles can be used with controllers described in the above referenced patent applications as well as any other devices or systems that can track user's head/face/body motions, facial expressions, and other actions to control or communicate with any electronic devices. Note that this document uses the term âElectronic Deviceâ as defined in the above-mentioned patent applications. Further, the UI concepts and principles described herein can be used to not only control an electronic device distinct from the controller, but also the controller and/or the controlling system itself. For the purpose of simplicity, the rest of the document will use the term âcontrollerâ to include âcontrolling systemsâ as well. Further, it is also understood that controllers themselves can be electronic devices; therefore, any mention of a controller âcontrolling/communicating with an electronic deviceâ can also include the controller generating signals for its own consumption.

The principles disclosed can be used with hand held and body worn controllers, traditional computing devices such as desktop and laptop computers, smart TVs, mobile computing devices such as tablets and smart phones, Augmented/Virtual/Mixed Reality devices, industrial machinery, medical systems, home appliances, electrical lighting systems, as well as with systems where the user's body or body part can be used for providing input. Body parts used for user actions prescribed to perform user gestures can include, but are not limited to, head, facial muscles, part of the face, jaws, tongue, eyes, ears, throat, neck, fingers, hands, arms, torso, chest, abdomen, shoulders, legs, feet, toes, and any muscles or tissues a user can have control or influence over.

A user gesture can be defined as a combination of user actions. User actions can be any actions that are performed by the user for the purpose of communicating with or controlling an electronic device. These user actions can be body actions that can include motions of various body parts, facial expressions, actions to orient and hold various body parts in certain poses/positions/orientations, as well as other bodily actions. Holding the eye gaze steady or moving the eye gaze can also be considered a body action. Some embodiments can also use actions performed by the user such as speech/speaking, holding breath/inhaling/exhaling, tensing of muscles/body parts (that may or may not be detected externally, such as jaw muscles, abdominal muscles, arm and leg muscles, anal sphincter, etc.), and so on as body actions. User actions such as entering meditative or attentive state, consciously relaxing the body with or without meditation, (mentally) imagining, visualizing, remembering or intending particular actions (e. g. pushing or pulling, lifting or sinking imaginary, virtual or real objects), experiences or scenarios (which can be detected by analyzing brainwaves or other biometric information), deep breathing, inhaling, exhaling, holding breath, etc. can also be used as user actions in defining user gestures. A user gesture can require certain user actions to be performed in a specified sequence, and can require other user actions to be performed concurrently/simultaneously with each other. User gestures can be recognized and translated by the controller or control system into signals to communicate with and/or control an electronic device. Some user gestures can be recognized and translated into signals to control the controller/control system itself. Signals generated in response to some user gestures may be stored in the control system or controlled device for indefinite amount of time and that stored signal information can be retrieved when required. User actions performed as part of a user gesture can serve various purposes in a specified user gesture. Following are some types of user actions based on the purpose they can fulfill in a user gesture.

a. Actions Enabling/Disabling Generation of Signals (AEGS)

b. Actions Influencing Attributes of Generated Signals being or to be generated (AIAGS)

c. Actions that Confirm User Intent (ACUI)

d. Actions that are Demarcators (i.e. help demarcate one part of user gesture from another, or even help demarcate one user gesture from another)

e. Actions with Multiple Purposes (AMP) (i.e. they can fulfill a combination of multiple purposes simultaneously)

Note: A particular user action can serve different purposes (and thereby can be viewed as having different types) when it is used in different types of user gestures. Further, a particular user action can occur multiple times within a user gesture and can be specified to have different purpose(s) (type/types) during different occurrences.

The use of Primary Control Expressions (PCEs) (possibly along with other user actions) to achieve control of electronic devices is disclosed. PCEs are designated facial expressions that can be used in definition of user gestures that are designed to communicate with or control electronic devices. PCEs can be used as AEGS in various user gestures. For example, PCEs are AEGS in Object of Interest (OOI) Motion and Click-and-Drag Heuristics. However, the role of PCE can be viewed as AMP in the Selection Heuristic as the PCE alone enables the generation of signals as well as cause that generation. Various facial expressions include, but are not limited to, smile, frown (with eyebrow or mouth), eyebrow motion, jaw drops, teeth clenches, closing/opening mouth, puffing cheeks, pouting, nose wiggles, ear wiggles, opening/closing eyes, blinking, winking and other motions of the facial muscles. Note that in some cultures, âfrownâ means contracting the brow where eyebrows can come closer together and the forehead can appear wrinkled. Whereas in other cultures, âfrownâ can be an expression of mouth where corners of the mouth can be pulled or curled downwards. Therefore, for clarity, we will distinguish between the two kinds of frowns as âeyebrow frownâ or âmouth frownâ as and when needed; otherwise the term frown will be used to refer to either of them or both.

The concept of Primary Control Motion (PCM) is similar to the concept of PCE. While PCEs can be facial expressions, PCMs can be designated body motions or pose/position/orientations of a designated set of one or more body parts. PCMs can include designated combination(s) or sequence(s) of body motions that can include motions of the entire head, eyeballs, hands, fingers, arms, shoulders, torso, legs, feet, toes, etc. Note that motions of the entire head such as head nods, head tilts, side to side head motions or head rolls, etc. are considered to be head/body motions and not facial expressions. Motion of the eyeballs is also considered to be body motion and not a facial expression. However, motion of eyelids such as opening/closing of eyes, blinking and winking are considered facial expressions.

Similarly, motion of eyebrows such as eyebrow raises, furrowing of eyebrows and other eyebrow motions are considered facial expressions. Just as PCEs, PCMs are accorded special significance when communicating with electronic devices. A PCM or a PCE can be used as an enabler, trigger, modifier, or even as a specific command, while communicating with an electronic device. PCE and PCM can also comprise actions such as entering meditative/attentive states, tensing specified muscles (such as periauricular muscles, jaw muscles, arm muscles, chest muscles, abdominal muscles, perianal muscles, pelvis floor muscles, leg muscles, etc.), relaxing, deep breathing, holding breath, etc. as these actions can be used to signify user intention and thereby can be used in heuristics explained (as PCEs or PCMs). PCEs and PCMs can be used as AEGS as well as ACUI.

A general rule of thumb for distinguishing PCM from PCE can be to consider if the designated user action involves rigid body motion of body parts versus non-rigid body motion. If the user action involves rigid body motion (that is where the shape of the individual designated parts do not change during the motion) then that can be considered to be PCM; e.g. motion of head/eye balls/fingers/forearm/arm, opening or closing of hand into a fist, making gestures with hands (such as pointing with index finger, pinching gesture with index finger and thumb, wiggling a finger, shooting gesture with a hand, stop gesture with the hand, making a Vulcan salute, etc.) and so on. As an example, when the user makes a âpointing with the index finger gestureâ, the individual parts of the hand and finger (such as phalanges, metacarpals, etc.) can be considered to be each going through a rigid body motion to change the overall configuration of the hand. On the other hand, if the user action involves non-rigid body motion, such as changing shape of the mouth (by smiling, frowning, pouting, opening/closing the mouth, etc.), changing shape of the cheek muscles, changing opening of the eye/squinting/winking/blinking, raising eye brows, furrowing of the eye brows, etc., those actions can be considered to be facial expressions and be designated as PCE. Having said the above, PCEs and PCMs can be considered completely equivalent to each other when it comes to performing designated functions in user gestures and can be used interchangeably in various heuristics and user gestures.

A designated sequence of multiple user actions can also be used as a PCE or a PCM, a Start Trigger, an End Trigger, an ODE, a Gesture Wakeup Sequence, a Session Wakeup Sequence, etc. For example, a pair of smiles or blinks or eyebrow twitches performed within a maximum specified time duration can be considered to be a PCE. Similarly, a smile followed by a blink when performed within a maximum specified time duration can be also considered together to be a PCE. Any number of facial expressions or other body actions can be combined to create a variety of PCEs or PCMs, various triggers, wake up sequences, ODEs, STHS, ETHS, etc. Then each of these could be used in any of the heuristics disclosed in this as well as referenced documents (e.g. OOI Modification, Selection, Click and Drag, OOI Warping, and so on).

Any heuristics (explained in this as well as the referenced patent applications) can be implemented in a controller/control system by means of multiple user gestures. For example, the selection heuristics can be implemented in one embodiment using a first user gesture that uses a smile facial expression as the Primary Control Expression (PCE) as well as another user gesture that uses an eyebrow raise facial expression as the PCE, and so on. Note that PCEs and PCMs can be considered as AEGS. Further, the Selection and the Click-and-Drag Heuristics could be modified to generate different signals in place of the selection signals. For example, when playing a g

CLAIMS

Claims ( 38 )

I claim:

1. A computer implemented method of controlling an electronic device by a user, the computer implemented method comprising:

receiving eye gaze information (Eye Info) indicative of eye gaze of the user;

receiving at least one of

i. facial expression information (FE Info) indicative of one or more facial expressions on the user's face, wherein said FE Info is different from said Eye Info;

ii. head information (Head Info) indicative of at least one of motion or position of the user's head, wherein said Head Info is different from said Eye Info and said FE Info; and

iii. body information (Body Info) indicative of at least one of motion and position of one or more body parts of the user, wherein said Body Info is different from said Eye Info, said FE Info and said Head Info; and

waiting for detection of a period of limited activity (POLA) based on said Eye Info, wherein the POLA comprises the Eye Info being steady within a specified range for at least a specified minimum time duration;

only after detection of the POLA:

(1) start comparing at least one of said FE Info, Head Info and Body Info with one or more predefined user gestures to detect a first matching portion of a recognized user gesture, wherein said recognized user gesture is one of said one or more predefined user gestures, and wherein said first matching portion is disposed at start of said recognized user gesture and said first matching portion comprises all or part of said recognized user gesture;

(2) if said first matching portion of said recognized user gesture is detected within a specified waiting period, generating command signals for the electronic device based on at least one of said FE Info, Head Info and Body Info until said recognized user gesture is no longer detected; and

(3) if said first matching portion of said recognized user gesture is not detected within the specified waiting period, restart said waiting for detection of a POLA step.

2. The computer-implemented method of claim 1 , wherein the electronic device is worn on the user's body.

3. The computer-implemented method of claim 2 , wherein a MEMS sensor provides at least a portion of at least one of said FE Info, said Head Info and said Body Info.

4. The computer-implemented method of claim 1 , wherein said generated command signals modify an object of interest (OOI) affected by the electronic device.

5. The computer-implemented method of claim 4 , wherein said OOI is a virtual object.

6. The computer-implemented method of claim 1 , wherein said generated command signals comprise a mouse selection command.

7. The computer-implemented method of claim 1 , wherein said one or more facial expressions of the user consist of at least one of a smile, jaw motion, eyebrow motion, puffing action, sipping action, puffing a cheek, sucking a cheek, opening mouth, closing mouth, pouting, frowning, pulling a corner of the lips, puckering lips, puffing or sucking cheeks, inflating nostrils, sticking out tongue, and moving tongue.

8. The computer-implemented method of claim 1 , wherein said generating command signals for the electronic device is suppressed if said Eye Info is outside a specified eye gaze zone.

9. The computer implemented method of claim 1 , wherein one or more user interface elements on the electronic device are activated when said Eye Info indicates that the user is looking in a specified direction in relation to the electronic device.

10. A system for a user to control an electronic device, the system comprising:

at least one first input configured to receive at least one of

i. facial expression information (FE Info) indicative of one or more facial expressions on the face of the user,

ii. head information (Head Info) indicative of at least one of motion or position of the user's head, wherein said Head Info is different from said FE Info, and

iii. body information (Body Info) indicative of at least one of motion and position of one or more body parts of the user, wherein said Body Info is different from said FE Info and said Head Info;

at least one second input configured to receive eye gaze information (Eye Info) indicative of eye gaze of the user, wherein said Eye Info is different from said FE Info, said Head Info and said Body Info;

a processor configured to detect a period of limited activity (POLA) based on said Eye Info; and only after detection of said POLA, to compare at least one of said FE Info, said Head Info and said Body Info with one or more predefined user gestures to detect a first matching portion of a recognized user gesture, wherein said recognized user gesture is one of said one or more predefined user gestures; if said first matching portion of said recognized user gesture is detected within a specified waiting period, to generate command signals for the electronic device based on at least one of said FE Info, said Head Info and said Body Info until said recognized user gesture is no longer detected; and if said first matching portion of said recognized user gesture is not detected within the specified waiting period, to wait for detection of a subsequent POLA,

wherein the POLA comprises said Eye Info being steady within a specified range-for at least a specified minimum time duration, and wherein said first matching portion is disposed at start of said recognized user gesture and said first matching portion comprises all or part of said recognized user gesture.

11. The system of claim 10 , wherein the electronic device is worn on the user's body.

12. The system of claim 11 , wherein the system further comprises a MEMS sensor, wherein said MEMS sensor provides at least a portion of at least one of said FE Info, said Head Info and said Body Info.

13. The system of claim 10 , wherein said generated command signals modify an object of interest (OOI) affected by the electronic device.

14. The system of claim 13 , wherein said OOI is a virtual object.

15. The system of claim 10 , wherein said generated command signals comprise a mouse selection command.

16. The system of claim 10 , wherein said one or more facial expressions of the user consist of at least one of smiling, jaw motion, eyebrow motion, puffing or sucking action, opening mouth, closing mouth, pouting, frowning, pulling a corner of the lips, puckering lips, puffing a cheek, sucking a cheek, puffing action, sipping action, inflating nostrils, sticking out the tongue, and moving the tongue.

17. The system of claim 10 , wherein said generating command signals for the electronic device is suppressed if said Eye Info is outside a specified eye gaze zone.

18. The system of claim 10 , wherein one or more user interface elements on the electronic device are activated when said Eye Info indicates that the user is looking in a specified direction in relation to the electronic device.

19. A non-transitory computer readable medium comprising one or more programs configured to be executed by one or more processors to enable a user to communicate with an electronic device, said one or more programs causing performance of a method comprising:

receiving eye gaze information (Eye Info) indicative of eye gaze of the user;

receiving at least one of

i. facial expression information (FE Info) indicative of one or more facial expressions on the user's face, wherein said FE Info is different from said Eye Info;

ii. head information (Head Info) indicative of at least one of motion or position of the user's head, wherein said Head Info is different from said Eye Info and said FE Info; and

waiting for detection of a period of limited activity (POLA) based on said Eye Info, wherein said POLA comprises said Eye Info being steady within a specified range for at least a specified minimum time duration;

only after detection of said POLA:

(1) start comparing at least one of said FE Info, said Head Info and said Body Info with one or more predefined user gestures to detect a first matching portion of a recognized user gesture, wherein said recognized user gesture is one of said one or more predefined user gestures, and wherein said first matching portion is disposed at start of said recognized user gesture and said first matching portion comprises all or part of said recognized user gesture;

(2) if said first matching portion of said recognized user gesture is detected within a waiting period, generating command signals for the electronic device based on at least one of said FE Info, said Head Info and said Body Info until said recognized user gesture is no longer detected; and

(3) if said first matching portion of said recognized user gesture is not detected within the waiting period, restart said waiting for detection of a POLA step.

20. The non-transitory computer readable medium of claim 19 , wherein the electronic device is worn on the user's body.

21. The non-transitory computer readable medium of claim 19 , wherein an image sensor provides at least a portion of at least one of said FE Info, said Head Info and said Body Info.

22. The non-transitory computer readable medium of claim 19 , wherein said generated command signals modify an object of interest (OOI) affected by the electronic device.

23. The non-transitory computer readable medium of claim 22 , wherein said OOI is a virtual object.

24. The non-transitory computer readable medium of claim 19 , wherein said generated command signals comprise a mouse selection command.

25. The non-transitory computer readable medium of claim 19 , wherein said one or more facial expressions of the user consist of at least one of a smile, jaw motion, eyebrow motion, puffing action, sipping action, sucking a cheek, puffing a cheek, opening mouth, closing mouth, pouting, frowning, pulling a corner of the lips, puckering lips, puffing or sucking cheeks, inflating nostrils, sticking tongue out, and moving tongue.

26. The non-transitory computer readable medium of claim 19 , wherein said generating command signals for the electronic device is suppressed if said Eye Info is outside a specified eye gaze zone.

27. The non-transitory computer readable medium of claim 19 , wherein one or more user interface elements on the electronic device are activated when said Eye Info indicates that the user is looking in a specified direction in relation to the electronic device.

28. An apparatus for a user to control an electronic device, the apparatus comprising:

memory;

a processor configured to:

receive eye gaze information (Eye Info) indicative of eye gaze of the user;

receive at least one of

i. facial expression information (FE Info) indicative of one or more facial expressions on the user's face, wherein said FE Info is different from said Eye Info;

ii. head information (Head Info) indicative of at least one of motion or position of the user's head, wherein said Head Info is different from said Eye Info and said FE Info; and

detect a period of limited activity (POLA) based on said Eye Info, wherein the POLA comprises said Eye Info being steady within a specified range-for at least a specified minimum time duration;

only after detection of said POLA:

(1) compare at least one of said FE Info, said Head Info and said Body Info with one or more predefined user gestures to detect a first matching portion of a recognized user gesture, wherein said recognized user gesture is one of said one or more predefined user gestures, and wherein said first matching portion is disposed at start of said recognized user gesture and said first matching portion comprises all or part of said recognized user gesture;

(2) if said first matching portion of said recognized user gesture is detected within a waiting period, generate command signals for the electronic device based on at least one of said FE Info, said Head Info and said Body Info until said recognized user gesture is no longer detected; and

(3) if said first matching portion of said recognized user gesture is not detected within the waiting period, waiting for detection of a subsequent POLA; and

a first sensor configured to provide at least a portion of at least one of said FE Info, Head Info, said Eye Info and said Body Info.

29. The apparatus of claim 28 , wherein the electronic device is worn on the user's body.

30. The apparatus of claim 29 , wherein said first sensor comprises a MEMS sensor.

31. The apparatus of claim 28 , wherein said generated command signals modify an object of interest (OOI) affected by the electronic device.

32. The apparatus of claim 31 , wherein said OOI is a virtual object.

33. The apparatus of claim 28 , wherein said generated command signals comprise a mouse selection command.

34. The apparatus of claim 28 , wherein said one or more facial expressions of the user consist of at least one of smiling, jaw motion, eyebrow motion, puffing or sucking action, opening mouth, closing mouth, pouting, frowning, pulling a corner of the lips, puckering lips, puffing action, sipping action, puffing a cheek, sucking a cheek, inflating nostrils, sticking out tongue and moving tongue.

35. The apparatus of claim 29 , wherein said apparatus further comprises an eye tracking system.

36. The apparatus of claim 28 , wherein said apparatus is in form of eyewear.

37. The apparatus of claim 29 , wherein said apparatus is configured to be a part of the electronic device.

38. The apparatus of claim 28 , wherein one or more user interface elements on the electronic device are activated when said Eye Info indicates that the user is looking in a specified direction in relation to the electronic device.

US17/141,903

2013-06-20

2021-01-05

Gesture based user interfaces, apparatuses and systems using eye tracking, head tracking, hand tracking, facial expressions and other user actions

Active

US11402902B2

( en )

Priority Applications (4)

Application Number

Priority Date

Filing Date

Title

US17/141,903

US11402902B2

( en )

2013-06-20

2021-01-05

Gesture based user interfaces, apparatuses and systems using eye tracking, head tracking, hand tracking, facial expressions and other user actions

US17/879,083

US11977677B2

( en )

2013-06-20

2022-08-02

Gesture based user interfaces, apparatuses and systems using eye tracking, head tracking, hand tracking, facial expressions and other user actions

US18/656,489

US12158987B2

( en )

2014-06-20

2024-05-06

Gesture based user interfaces, apparatuses and systems using eye tracking, head tracking, hand tracking, facial expressions and other user actions

US18/966,049

US20250165066A1

( en )

2015-12-11

2024-12-02

Gesture based user interfaces, apparatuses and systems using eye tracking, head tracking, hand tracking, facial expressions and other user actions

Applications Claiming Priority (15)

Application Number

Priority Date

Filing Date

Title

US201361837215P

2013-06-20

PCT/US2014/043529

WO2014205422A2

( en )

2013-06-20

2014-06-20

Systems, methods, apparatuses, computer readable medium for controlling electronic devices

US201514897657A

2015-12-11

US201662313042P

2016-03-24

US201662427006P

2016-11-28

US201762470872P

2017-03-14

US15/469,456

US10137363B2

( en )

2013-06-20

2017-03-24

Gesture based user interfaces, apparatuses and control systems

US201762537482P

2017-07-27

US201762589228P

2017-11-21

US201862626253P

2018-02-05

US201862630253P

2018-02-14

US15/921,632

US10558272B2

( en )

2013-06-20

2018-03-14

Gesture control via eye tracking, head tracking, facial expressions and other user actions

US16/201,776

US20190265802A1

( en )

2013-06-20

2018-11-27

Gesture based user interfaces, apparatuses and control systems

US16/726,350

US10884493B2

( en )

2013-06-20

2019-12-24

Gesture based user interfaces, apparatuses and systems using eye tracking, head tracking, hand tracking, facial expressions and other user actions

US17/141,903

US11402902B2

( en )

2013-06-20

2021-01-05

Gesture based user interfaces, apparatuses and systems using eye tracking, head tracking, hand tracking, facial expressions and other user actions

Related Parent Applications (1)

Application Number

Title

Priority Date

Filing Date

US16/726,350

Continuation

US10884493B2

( en )

2013-06-20

2019-12-24

Gesture based user interfaces, apparatuses and systems using eye tracking, head tracking, hand tracking, facial expressions and other user actions

Related Child Applications (1)

Application Number

Title

Priority Date

Filing Date

US17/879,083

Continuation

US11977677B2

( en )

2013-06-20

2022-08-02

Gesture based user interfaces, apparatuses and systems using eye tracking, head tracking, hand tracking, facial expressions and other user actions

Publications (2)

Publication Number

Publication Date

US20210157402A1

US20210157402A1 ( en )

2021-05-27

US11402902B2

true

US11402902B2 ( en )

2022-08-02

Family

ID=71837511

Family Applications (3)

Application Number

Title

Priority Date

Filing Date

US16/726,350

Active

US10884493B2

( en )

2013-06-20

2019-12-24

Gesture based user interfaces, apparatuses and systems using eye tracking, head tracking, hand tracking, facial expressions and other user actions

US17/141,903

Active

US11402902B2

( en )

2013-06-20

2021-01-05

Gesture based user interfaces, apparatuses and systems using eye tracking, head tracking, hand tracking, facial expressions and other user actions

US17/879,083

Active

US11977677B2

( en )

2013-06-20

2022-08-02

Gesture based user interfaces, apparatuses and systems using eye tracking, head tracking, hand tracking, facial expressions and other user actions

Family Applications Before (1)

Application Number

Title

Priority Date

Filing Date

US16/726,350

Active

US10884493B2

( en )

2013-06-20

2019-12-24

Gesture based user interfaces, apparatuses and systems using eye tracking, head tracking, hand tracking, facial expressions and other user actions

Family Applications After (1)

Application Number

Title

Priority Date

Filing Date

US17/879,083

Active

US11977677B2

( en )

2013-06-20

2022-08-02

Gesture based user interfaces, apparatuses and systems using eye tracking, head tracking, hand tracking, facial expressions and other user actions

Country Status (1)

Country

Link

( 3 )

US10884493B2

( en )

Cited By (3)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

WO2024110791A1

( en )

2022-11-22

2024-05-30

Sangireddy Sisirnath

A method for enabling quality of learning, communication, and a cognitive support system for user interaction using a mixed reality device

US12495980B2

( en )

2023-03-13

2025-12-16

Google Llc

Detecting heart rate variability using a hearable

US12629037B2

( en )

2023-06-09

2026-05-19

Google Llc

Audioplethysmography calibration

Families Citing this family (88)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US8866809B2

( en )

2008-09-30

2014-10-21

Apple Inc.

System and method for rendering dynamic three-dimensional appearing imagery on a two-dimensional user interface

CA2896985A1

( en )

2013-01-03

2014-07-10

Meta Company

Extramissive spatial imaging digital eye glass for virtual or augmediated vision

US12504816B2

( en )

2013-08-16

2025-12-23

Meta Platforms Technologies, Llc

Wearable devices and associated band structures for sensing neuromuscular signals using sensor pairs in respective pods with communicative pathways to a common processor

US10188309B2

( en )

2013-11-27

2019-01-29

North Inc.

Systems, articles, and methods for electromyography sensors

US11921471B2

( en )

2013-08-16

2024-03-05

Meta Platforms Technologies, Llc

Systems, articles, and methods for wearable devices having secondary power sources in links of a band for providing secondary power in addition to a primary power source

US20150124566A1

( en )

2013-10-04

2015-05-07

Thalmic Labs Inc.

Systems, articles and methods for wearable electronic devices employing contact sensors

US12511014B1

( en )

2014-03-01

2025-12-30

sigmund lindsay clements

Public devices connected to augmented reality glasses having gaze and secondary input sensors

US10600213B2

( en )

2016-02-27

2020-03-24

Focal Sharp, Inc.

Method and apparatus for color-preserving spectrum reshape

US12554325B2

( en )

2016-07-25

2026-02-17

Meta Platforms Technologies, Llc

Methods and apparatuses for low latency body state prediction based on neuromuscular data

CN108268120B

( en )

2016-12-29

2020-07-28

åæ¹å¨è§ææ¯è¡ä»½æéå¬å¸

Image data processing method and device based on VR or AR and security inspection system

US10558278B2

( en )

2017-07-11

2020-02-11

Apple Inc.

Interacting with an electronic device through physical movement

KR102374404B1

( en )

2017-07-25

2022-03-15

ì¼ì±ì ìì£¼ìíì¬

Device and method for providing content

US10957069B2

( en )

2017-09-29

2021-03-23

Tobii Ab

Head pose estimation from local eye region

EP3697297A4

( en )

2017-10-19

2020-12-16

Facebook Technologies, Inc.

SYSTEMS AND METHODS FOR IDENTIFYING BIOLOGICAL STRUCTURES ASSOCIATED WITH NEUROMUSCULAR SOURCE SIGNALS

US11144125B2

( en )

2017-12-07

2021-10-12

First-Light Usa, Llc

Hands-free switch system

US11961494B1

( en )

2019-03-29

2024-04-16

Meta Platforms Technologies, Llc

Electromagnetic interference reduction in extended reality environments

US12579768B2

( en )

2018-01-25

2026-03-17

Meta Platforms Technologies, Llc

Wearable electronic devices, extended reality systems including neuromuscular sensors, and methods for generating text from speech input and modifying the generated text based on neuromuscular data

US11907423B2

( en )

2019-11-25

2024-02-20

Meta Platforms Technologies, Llc

Systems and methods for contextualized interactions with an environment

CN118567472A

( en )

2018-05-04

2024-08-30

è°·ææéè´£ä»»å¬å¸

Adaptive automatic assistant based on detected mouth movement and/or gaze

EP3608174B1

( en )

2018-08-10

2021-03-24

Lg Electronics Inc.

Vehicle display system for vehicle

CN113423341B

( en )

2018-11-27

2024-12-03

åå¹³å°ææ¯æéå¬å¸

Method and apparatus for automatic calibration of wearable electrode sensor systems

US11275778B2

( en )

2018-12-04

2022-03-15

International Business Machines Corporation

Content marshaling using biometric data

CN110164440B

( en )

2019-06-03

2022-08-09

äº¤äºæªæ¥(åäº¬)ç§ææéå¬å¸

Voice interaction awakening electronic device, method and medium based on mouth covering action recognition

US11948401B2

( en )

2019-08-17

2024-04-02

Nightingale.ai Corp.

AI-based physical function assessment system

US11175809B2

( en )

2019-08-19

2021-11-16

Capital One Services, Llc

Detecting accessibility patterns to modify the user interface of an application

US11526212B1

( en )

2019-09-25

2022-12-13

Amazon Technologies, Inc.

System to determine don/doff of wearable device

US12148120B2

( en )

2019-12-18

2024-11-19

Ati Technologies Ulc

Frame reprojection for virtual reality and augmented reality

US11327651B2

( en )

2020-02-12

2022-05-10

Facebook Technologies, Llc

Virtual keyboard based on adaptive language model

US11463437B2

( en )

2020-03-04

2022-10-04

International Business Machines Corporation

Device activation verification

JP2021157277A

( en )

2020-03-25

2021-10-07

ã½ãã¼ã°ã«ã¼ãæ ªå¼ä¼ç¤¾

Information processing apparatus, information processing method, and program

CN113496171A

( en )

2020-04-03

2021-10-12

åäº¬å°ç±³ç§»å¨è½¯ä»¶æéå¬å¸

Gesture detection method and device, mobile terminal and storage medium

WO2021260831A1

( en )

2020-06-24

2021-12-30

æ¥æ¬é»ä¿¡é»è©±æ ªå¼ä¼ç¤¾

Information input device

US11553313B2

( en )

2020-07-02

2023-01-10

Hourglass Medical Llc

Clench activated switch system

US12380798B2

( en )

2020-07-02

2025-08-05

Hourglass Medical Llc

Switch system for operating a controlled device

EP3937139A1

( en )

2020-07-06

2022-01-12

Technische UniversitÃ¤t MÃ¼nchen

Method of aligning virtual and real objects

US11157081B1

( en )

2020-07-28

2021-10-26

Shenzhen Yunyinggu Technology Co., Ltd.

Apparatus and method for user interfacing in display glasses

US11656681B2

( en )

2020-08-31

2023-05-23

Hypear, Inc.

System and method for determining user interactions with visual content presented in a mixed reality environment

US11341543B2

( en )

2020-08-31

2022-05-24

HYPE AR, Inc.

System and method for generating visual content associated with tailored advertisements in a mixed reality environment

US11281291B1

( en )

2020-10-20

2022-03-22

Rovi Guides, Inc.

Methods and systems of extended reality environment interaction based on eye motions

US11392198B2

( en )

2020-10-20

2022-07-19

ROVl GUIDES, INC.

Methods and systems of extended reality environment interaction based on eye motions

US11609629B2

( en )

2020-10-20

2023-03-21

Rovi Guides, Inc.

Methods and systems of extended reality environment interaction based on eye motions

US11320903B1

( en )

2020-10-20

2022-05-03

Rovi Guides, Inc.

Methods and systems of extended reality environment interaction based on eye motions

US11747896B2

( en )

2020-10-20

2023-09-05

Rovi Guides, Inc.

Methods and systems of extended reality environment interaction based on eye motions

US12373029B2

( en )

2020-10-20

2025-07-29

Adeia Guides Inc.

Methods and systems of extended reality environment interaction based on eye motions

EP4001568B1

( en )

2020-11-17

2024-05-01

Aptiv Technologies AG

Pinch detection based on estimated pinch force

EP4001567B1

( en )

2020-11-17

2024-05-29

Aptiv Technologies AG

Method and device for detecting potential pinches

EP4001566B1

( en )

2020-11-17

2024-05-29

Aptiv Technologies AG

Pinch detection during motor restart

EP4001565B1

( en )

2020-11-17

2023-12-27

Aptiv Technologies Limited

Method and device for detecting potential pinches

US12172073B2

( en )

2021-01-29

2024-12-24

Andrea Joy Brucculeri

Method, apparatus, GUIs and APIs for a user experience design related to hands-free gaming accessibility

WO2022173558A1

( en )

2021-02-12

2022-08-18

Hourglass Medical Llc

Clench-control accessory for head-worn devices

US11868531B1

( en )

2021-04-08

2024-01-09

Meta Platforms Technologies, Llc

Wearable device providing for thumb-to-finger-based input gestures detected based on neuromuscular signals, and systems and methods of use thereof

WO2022225912A1

( en )

2021-04-21

2022-10-27

Hourglass Medical Llc

Methods for voice blanking muscle movement controlled systems

US11747919B1

( en )

2021-05-14

2023-09-05

Apple Inc.

Multi-input for rotating and translating crown modules

US12449907B2

( en )

2021-05-19

2025-10-21

Apple Inc.

Navigating user interfaces using a cursor

CN113223344B

( en )

2021-05-25

2022-08-23

æ¹åæ±½è½¦å·¥ç¨èä¸å¦é¢

Big data-based professional teaching display system for art design

FR3123794B1

( en )

2021-06-14

2024-03-15

Centre Hospitalier Regional Univ De Tours

COMMUNICATION INTERFACE ADAPTED BASED ON A COGNITIVE ASSESSMENT FOR PATIENTS DEPRIVED OF SPEECH

WO2023278138A1

( en )

2021-06-28

2023-01-05

Dathomir Laboratories Llc

Methods and systems for changing a display based on user input and gaze

WO2023004063A1

( en )

2021-07-21

2023-01-26

Dolby Laboratories Licensing Corporation

Screen interaction using eog coordinates

US20230069764A1

( en )

2021-08-24

2023-03-02

Meta Platforms Technologies, Llc

Systems and methods for using natural gaze dynamics to detect input recognition errors

US12159352B2

( en )

2021-08-30

2024-12-03

Bsset Llc

Extended reality movement platform

US11758350B2

( en )

2021-09-24

2023-09-12

Apple Inc.

Posture transition detection and classification using linked biomechanical model

US12307012B2

( en )

2021-10-27

2025-05-20

Google Llc

Response to sounds in an environment based on correlated audio and user events

CN118475899A

( en )

2021-12-14

2024-08-09

è¹æå¬å¸

User interaction and eye tracking with text embedded elements

EP4198948A1

( en )

2021-12-20

2023-06-21

BIC Violex Single Member S.A.

Personalized lecture generator

USD1056924S1

( en )

2021-12-22

2025-01-07

Mcmaster-Carr Supply Company

Display screen or portion thereof with animated graphical user interface

US20230244768A1

( en )

2022-02-01

2023-08-03

Matthew Edward Natividad Healey

Systems and methods for intent-based device unlocking

US11768544B2

( en )

2022-02-01

2023-09-26

Microsoft Technology Licensing, Llc

Gesture recognition based on likelihood of interaction

US12579758B2

( en )

2022-02-08

2026-03-17

Apple Inc.

Devices, methods, and graphical user interfaces for interacting with virtual objects using hand gestures

CN114911341B

( en )

2022-04-21

2024-10-01

ä¸å½äººæ°è§£æ¾åå½é²ç§æå¤§å¦

Target selection method and system based on secondary triggering of eye gestures

US12216829B2

( en )

2022-04-26

2025-02-04

Oura Health Oy

Ring-inputted commands

US12386428B2

( en )

2022-05-17

2025-08-12

Apple Inc.

User interfaces for device controls

US12130965B2

( en )

2022-07-07

2024-10-29

Plume Design, Inc.

Ring enabling its wearer to enter control commands

US12536265B2

( en )

2022-07-18

2026-01-27

Bank Of America Corporation

Systems and methods for performing non-contact authorization verification for access to a network

US12353664B1

( en )

2022-09-28

2025-07-08

Amazon Technologies, Inc.

Registering selections of graphical elements on electronic displays using time-of-flight sensors

WO2024076458A1

( en )

2022-10-04

2024-04-11

Hourglass Medical Llc

Eyeglasses nose piece switch

WO2023039616A2

( en )

2022-11-02

2023-03-16

Futurewei Technologies, Inc.

Device having optical see-through 3d display

CN115686223B

( en )

2022-11-21

2025-07-22

åæ¹èåéä¿¡æéå¬å¸

Unmanned equipment control method and system for mixed interaction of eyes and teeth

US12198528B2

( en )

2022-12-27

2025-01-14

The Adt Security Corporation

Alarm system facial recognition

WO2024176149A1

( en )

2023-02-21

2024-08-29

Aavaa Inc.

Method and system for determining facial movements

US12311256B2

( en )

2023-02-28

2025-05-27

Sony Interactive Entertainment Inc.

Using eye-tracking / eye metrics to detect VR eye-strain-levels, then using adaptive rendering to lower levels of eye-strain

CN116246792B

( en )

2023-03-15

2026-03-24

Modeling methods, devices, electronic equipment and storage media

US12307018B2

( en )

2023-04-03

2025-05-20

Sony Interactive Entertainment Inc.

Hand gesture magnitude analysis and gearing for communicating context-correct communication

KR20240151533A

( en )

2023-04-11

2024-10-18

íëëª¨ë¹ì¤ ì£¼ìíì¬

Mouse control method and apparatus using gaze recognition

WO2025072676A1

( en )

2023-09-29

2025-04-03

Apple Inc.

Replying to events on a first device by commands using secondary device gestures

US12340013B1

( en )

<span i

Gesture based user interfaces, apparatuses and systems using eye tracking, head … — Perceptive Devices Llc (US11402902B2)

Related documents