ABSTRACT
Abstract
Described are various embodiments of a pupil tracking system and method, and digital display device and digital image rendering system and method using same. In one embodiment, a computer-implemented method for dynamically adjusting rendering of a digital image using a light field display comprises: sequentially acquiring a user pupil location; digitally computing a velocity thereof over time; digitally comparing the velocity with a designated threshold pupil velocity; digitally rendering the digital image via the light field display in accordance with a maintained light field viewing zone geometry digitally defined in respect of a previously acquired user pupil location unless the velocity is above the designated threshold pupil velocity.
Description
CROSS REFERENCE TO RELATED APPLICATIONS
This application is a Continuation of International Application No. PCT/IB2020/053035 filed Mar. 31, 2020, which claims priority to Canadian Patent Application No. 3,038,584 filed Apr. 1, 2019, and to U.S. Provisional Patent Application No. 62/929,599 filed Nov. 1, 2019, the entire disclosure of each of which are incorporated herein by reference.
FIELD OF THE DISCLOSURE
The present disclosure relates to eye tracking and digital displays, and, in particular, to a pupil tracking system and method, and digital display device and digital image rendering system and method using same.
BACKGROUND
Gaze tracking technologies are currently being applied in different fields, for example, in the context of display content engagement tracking, or in tracking a user's attention and/or distraction in different contexts such as while driving a vehicle. One may generally define two broad categories of gaze tracking technologies. The first category generally relies on projecting near-IR light on a user's face and detecting corneo-scleral reflections (i.e. glints) on the user's eye to do so-called bright and/or dark pupil tracking. Different products of this type are available, for example TOBII (http://www.tobii.com) provides a range of products using such technology. Another broad category includes computer vision methods that rely on extracting facial features from digital images or videos. Examples of products for computer vision facial feature extraction include Face++ (https://www.faceplusplus.com) or the open source facial feature extraction library OpenFace (https://github.com/TadasB altrusaitis/OpenFace).
Using these techniques, a user's gaze direction can be monitored in real-time and put in context to monitor what draw's the user's attention over time.
This background information is provided to reveal information believed by the applicant to be of possible relevance. No admission is necessarily intended, nor should be construed, that any of the preceding information constitutes prior art or forms part of the general common knowledge in the relevant art.
SUMMARY
The following presents a simplified summary of the general inventive concept(s) described herein to provide a basic understanding of some aspects of the disclosure. This summary is not an extensive overview of the disclosure. It is not intended to restrict key or critical elements of embodiments of the disclosure or to delineate their scope beyond that which is explicitly or implicitly described by the following description and claims.
In accordance with one aspect, there is provided a computer-implemented method, automatically implemented by one or more digital processors, for dynamically adjusting a digital image to be rendered on a digital display based on a corresponding viewer pupil location, the method comprising: sequentially acquiring a user pupil location; digitally computing from at least some said sequentially acquired user pupil location an estimated physical trajectory and/or velocity of said user pupil location over time; digitally predicting from said estimated physical trajectory and/or velocity a predicted user pupil location for a projected time; and digitally adjusting the digital image to be rendered at said projected time based on said predicted user pupil location.
In accordance with another aspect, there is provided a computer-readable medium having instructions stored thereon to be automatically implemented by one or more processors to dynamically adjust a digital image to be rendered based on a corresponding viewer pupil location by: sequentially acquiring a user pupil location; digitally computing from at least some said sequentially acquired user pupil location an estimated physical trajectory and/or velocity of said user pupil location over time; digitally predicting from said estimated trajectory and/or velocity a predicted user pupil location for a projected time; and digitally adjusting the digital image to be rendered at said projected time based on said predicted user pupil location.
In accordance with another aspect, there is provided a digital display device operable to automatically adjust a digital image to be rendered thereon, the device comprising: a digital display medium; a hardware processor; and a pupil tracking engine operable by said hardware processor to automatically: receive as input sequential user pupil locations; digitally compute from said sequential user pupil locations an estimated physical trajectory of said user pupil location over time; and digitally predict from said estimated trajectory a predicted user pupil location for a projected time; wherein said hardware processor is operable to adjust the digital image to be rendered via said digital display medium at said projected time based on said predicted user pupil location.
In accordance with another aspect, there is provided a computer-implemented method, automatically implemented by one or more digital processors, for dynamically adjusting rendering of a digital image using a light field display, the method comprising: sequentially acquiring a user pupil location; digitally computing from at least some said sequentially acquired user pupil location a velocity of said user pupil location over time; digitally comparing said velocity with a designated threshold pupil velocity; digitally rendering the digital image via the light field display in accordance with a maintained light field viewing zone geometry digitally defined in respect of a previously acquired user pupil location unless said velocity is above said designated threshold pupil velocity; and upon said velocity exceeding said designated threshold pupil velocity, digitally adjusting a rendering geometry of the digital image via the light field display so to correspondingly adjust said light field viewing zone geometry to correspond to a newly acquired user pupil location.
In accordance with another aspect, there is provided a computer-readable medium having instructions stored thereon to be automatically implemented by one or more processors to dynamically adjust rendering of a digital image using a light field display by: sequentially acquiring a user pupil location; digitally computing from at least some said sequentially acquired user pupil location a velocity of said user pupil location over time; digitally comparing said velocity with a designated threshold pupil velocity; digitally rendering the digital image via the light field display in accordance with a maintained light field viewing zone geometry digitally defined in respect of a previously acquired user pupil location unless said velocity is above said designated threshold pupil velocity; and upon said velocity exceeding said designated threshold pupil velocity, digitally adjusting a rendering geometry of the digital image via the light field display so to correspondingly adjust said light field viewing zone geometry to correspond to a newly acquired user pupil location.
In accordance with another aspect, there is provided a digital display device operable to automatically adjust a digital image to be rendered thereon, the device comprising: a light field display; a hardware processor; and a pupil tracking engine operable by said hardware processor to automatically receive as input sequential user pupil locations, digitally compute from at least some said sequential user pupil locations a velocity of said user pupil location over time, and digitally compare said velocity with a designated threshold pupil velocity; wherein said hardware processor is operable to digitally render the digital image via the light field display in accordance with a maintained light field viewing zone geometry digitally defined in respect of a previously acquired user pupil location unless said velocity is above said designated threshold pupil velocity, and upon said velocity exceeding said designated threshold pupil velocity, digitally adjust a rendering geometry of the digital image via the light field display so to correspondingly adjust said light field viewing zone geometry to correspond to a newly acquired user pupil location.
One embodiment further comprises digitally adjusting a rendering geometry of the digital image via the light field display so to correspondingly adjust said light field viewing zone geometry to correspond to a function of a newly acquired user pupil location upon a designated condition for movement of said light field viewing zone geometry is met.
In one embodiment, the designated condition for movement of said viewing zone comprises at least one of said user pupil location crossing a defined boundary of said maintained light field viewing zone geometry, said maintained light field viewing zone geometry remaining static for a prescribed period of time, or said velocity is greater than a distinct predetermined threshold.
In one embodiment, the function is an interpolation of said newly acquired user pupil location and said maintained light field viewing zone geometry.
In one embodiment, the function is a function of time since said designated condition for movement was met.
In one embodiment, the interpolation is calculated for a designated period of time after said designated condition was met.
In one embodiment, the designated period of time is between about 0.02 s and 1 s.
In one embodiment, the threshold velocity is between 0.02 m/s and 1 m/s.
In one embodiment, the threshold velocity is approximately 0.1 m/s.
In one embodiment, digitally rendering the digital image via the light field display comprises: digitally mapping the digital image on an adjusted image plane designated to provide the user with a designated image perception adjustment; associating adjusted image pixel data with at least some of said pixels according to said mapping; and rendering said adjusted image pixel data via said pixels thereby rendering a perceptively adjusted version of the digital image.
Other aspects, features and/or advantages will become more apparent upon reading of the following non-restrictive description of specific embodiments thereof, given by way of example only with reference to the accompanying drawings.
BRIEF DESCRIPTION OF THE FIGURES
Several embodiments of the present disclosure will be provided, by way of examples only, with reference to the appended drawings, wherein:
FIG. 1 is a schematic representation of a predicted pupil location calculated using a predictive pupil tracking process based on previously acquired pupil locations, according to one embodiment;
FIG. 2 is schematic representation of a pupil location in three-dimensional space, according to one embodiment;
FIG. 3 is a process flow diagram of a predictive pupil tracking method, according to one embodiment;
FIG. 4 is a schematic representation of an effective pupil tracking frequency increased using a predictive pupil tracking process such as that sown in FIG. 3 , according to one embodiment;
FIGS. 5A and 5B are schematic representations of acquired pupil location sequences and forecast pupil locations predicted therefrom, in accordance with at least one embodiment;
FIG. 6 is a process flow diagram illustrating an operational mode of a predictive pupil tracking method, in accordance with at least one of the various embodiments;
FIG. 7 is a process flow diagram illustrating another operational mode of a predictive pupil tracking method, in accordance with at least one of the various embodiments;
FIG. 8 is a process flow diagram of an illustrative ray-tracing rendering process, in accordance with one embodiment;
FIGS. 9 and 10 are process flow diagrams of exemplary input constant parameters and variables, respectively, for the ray-tracing rendering process of FIG. 8 , in accordance with one embodiment;
FIGS. 11A to 11C are schematic diagrams illustrating certain process steps of FIG. 8 ;
FIG. 12 is process flow diagram of an illustrative ray-tracing rendering process, in accordance with another embodiment;
FIGS. 13A to 13D are schematic diagrams illustrating certain process steps of FIG. 12 ; and
FIG. 14 is a schematic state diagram of a predictive pupil tracking system, in accordance with one embodiment.
Elements in the several figures are illustrated for simplicity and clarity and have not necessarily been drawn to scale. For example, the dimensions of some of the elements in the figures may be emphasized relative to other elements for facilitating understanding of the various presently disclosed embodiments. Also, common, but well-understood elements that are useful or necessary in commercially feasible embodiments are often not depicted in order to facilitate a less obstructed view of these various embodiments of the present disclosure.
DETAILED DESCRIPTION
Various implementations and aspects of the specification will be described with reference to details discussed below. The following description and drawings are illustrative of the specification and are not to be construed as limiting the specification. Numerous specific details are described to provide a thorough understanding of various implementations of the present specification. However, in certain instances, well-known or conventional details are not described in order to provide a concise discussion of implementations of the present specification.
Various apparatuses and processes will be described below to provide examples of implementations of the systems and methods disclosed herein. No implementation described below limits any claimed implementation and any claimed implementations may cover processes or apparatuses that differ from those described below. The claimed implementations are not limited to apparatuses or processes having all of the features of any one apparatus or process described below or to features common to multiple or all of the apparatuses or processes described below. It is possible that an apparatus or process described below is not an implementation of any claimed subject matter.
Furthermore, numerous specific details are set forth in order to provide a thorough understanding of the implementations described herein. However, it will be understood by those skilled in the relevant arts that the implementations described herein may be practiced without these specific details. In other instances, well-known methods, procedures and components have not been described in detail so as not to obscure the implementations described herein.
In this specification, elements may be described as âconfigured toâ perform one or more functions or âconfigured forâ such functions. In general, an element that is configured to perform or configured for performing a function is enabled to perform the function, or is suitable for performing the function, or is adapted to perform the function, or is operable to perform the function, or is otherwise capable of performing the function.
It is understood that for the purpose of this specification, language
CROSS REFERENCE TO RELATED APPLICATIONS
This application is a Continuation of International Application No. PCT/IB2020/053035 filed Mar. 31, 2020, which claims priority to Canadian Patent Application No. 3,038,584 filed Apr. 1, 2019, and to U.S. Provisional Patent Application No. 62/929,599 filed Nov. 1, 2019, the entire disclosure of each of which are incorporated herein by reference.
FIELD OF THE DISCLOSURE
The present disclosure relates to eye tracking and digital displays, and, in particular, to a pupil tracking system and method, and digital display device and digital image rendering system and method using same.
BACKGROUND
Gaze tracking technologies are currently being applied in different fields, for example, in the context of display content engagement tracking, or in tracking a user's attention and/or distraction in different contexts such as while driving a vehicle. One may generally define two broad categories of gaze tracking technologies. The first category generally relies on projecting near-IR light on a user's face and detecting corneo-scleral reflections (i.e. glints) on the user's eye to do so-called bright and/or dark pupil tracking. Different products of this type are available, for example TOBII (http://www.tobii.com) provides a range of products using such technology. Another broad category includes computer vision methods that rely on extracting facial features from digital images or videos. Examples of products for computer vision facial feature extraction include Face++ (https://www.faceplusplus.com) or the open source facial feature extraction library OpenFace (https://github.com/TadasB altrusaitis/OpenFace).
Using these techniques, a user's gaze direction can be monitored in real-time and put in context to monitor what draw's the user's attention over time.
This background information is provided to reveal information believed by the applicant to be of possible relevance. No admission is necessarily intended, nor should be construed, that any of the preceding information constitutes prior art or forms part of the general common knowledge in the relevant art.
SUMMARY
The following presents a simplified summary of the general inventive concept(s) described herein to provide a basic understanding of some aspects of the disclosure. This summary is not an extensive overview of the disclosure. It is not intended to restrict key or critical elements of embodiments of the disclosure or to delineate their scope beyond that which is explicitly or implicitly described by the following description and claims.
In accordance with one aspect, there is provided a computer-implemented method, automatically implemented by one or more digital processors, for dynamically adjusting a digital image to be rendered on a digital display based on a corresponding viewer pupil location, the method comprising: sequentially acquiring a user pupil location; digitally computing from at least some said sequentially acquired user pupil location an estimated physical trajectory and/or velocity of said user pupil location over time; digitally predicting from said estimated physical trajectory and/or velocity a predicted user pupil location for a projected time; and digitally adjusting the digital image to be rendered at said projected time based on said predicted user pupil location.
In accordance with another aspect, there is provided a computer-readable medium having instructions stored thereon to be automatically implemented by one or more processors to dynamically adjust a digital image to be rendered based on a corresponding viewer pupil location by: sequentially acquiring a user pupil location; digitally computing from at least some said sequentially acquired user pupil location an estimated physical trajectory and/or velocity of said user pupil location over time; digitally predicting from said estimated trajectory and/or velocity a predicted user pupil location for a projected time; and digitally adjusting the digital image to be rendered at said projected time based on said predicted user pupil location.
In accordance with another aspect, there is provided a digital display device operable to automatically adjust a digital image to be rendered thereon, the device comprising: a digital display medium; a hardware processor; and a pupil tracking engine operable by said hardware processor to automatically: receive as input sequential user pupil locations; digitally compute from said sequential user pupil locations an estimated physical trajectory of said user pupil location over time; and digitally predict from said estimated trajectory a predicted user pupil location for a projected time; wherein said hardware processor is operable to adjust the digital image to be rendered via said digital display medium at said projected time based on said predicted user pupil location.
In accordance with another aspect, there is provided a computer-implemented method, automatically implemented by one or more digital processors, for dynamically adjusting rendering of a digital image using a light field display, the method comprising: sequentially acquiring a user pupil location; digitally computing from at least some said sequentially acquired user pupil location a velocity of said user pupil location over time; digitally comparing said velocity with a designated threshold pupil velocity; digitally rendering the digital image via the light field display in accordance with a maintained light field viewing zone geometry digitally defined in respect of a previously acquired user pupil location unless said velocity is above said designated threshold pupil velocity; and upon said velocity exceeding said designated threshold pupil velocity, digitally adjusting a rendering geometry of the digital image via the light field display so to correspondingly adjust said light field viewing zone geometry to correspond to a newly acquired user pupil location.
In accordance with another aspect, there is provided a computer-readable medium having instructions stored thereon to be automatically implemented by one or more processors to dynamically adjust rendering of a digital image using a light field display by: sequentially acquiring a user pupil location; digitally computing from at least some said sequentially acquired user pupil location a velocity of said user pupil location over time; digitally comparing said velocity with a designated threshold pupil velocity; digitally rendering the digital image via the light field display in accordance with a maintained light field viewing zone geometry digitally defined in respect of a previously acquired user pupil location unless said velocity is above said designated threshold pupil velocity; and upon said velocity exceeding said designated threshold pupil velocity, digitally adjusting a rendering geometry of the digital image via the light field display so to correspondingly adjust said light field viewing zone geometry to correspond to a newly acquired user pupil location.
In accordance with another aspect, there is provided a digital display device operable to automatically adjust a digital image to be rendered thereon, the device comprising: a light field display; a hardware processor; and a pupil tracking engine operable by said hardware processor to automatically receive as input sequential user pupil locations, digitally compute from at least some said sequential user pupil locations a velocity of said user pupil location over time, and digitally compare said velocity with a designated threshold pupil velocity; wherein said hardware processor is operable to digitally render the digital image via the light field display in accordance with a maintained light field viewing zone geometry digitally defined in respect of a previously acquired user pupil location unless said velocity is above said designated threshold pupil velocity, and upon said velocity exceeding said designated threshold pupil velocity, digitally adjust a rendering geometry of the digital image via the light field display so to correspondingly adjust said light field viewing zone geometry to correspond to a newly acquired user pupil location.
One embodiment further comprises digitally adjusting a rendering geometry of the digital image via the light field display so to correspondingly adjust said light field viewing zone geometry to correspond to a function of a newly acquired user pupil location upon a designated condition for movement of said light field viewing zone geometry is met.
In one embodiment, the designated condition for movement of said viewing zone comprises at least one of said user pupil location crossing a defined boundary of said maintained light field viewing zone geometry, said maintained light field viewing zone geometry remaining static for a prescribed period of time, or said velocity is greater than a distinct predetermined threshold.
In one embodiment, the function is an interpolation of said newly acquired user pupil location and said maintained light field viewing zone geometry.
In one embodiment, the function is a function of time since said designated condition for movement was met.
In one embodiment, the interpolation is calculated for a designated period of time after said designated condition was met.
In one embodiment, the designated period of time is between about 0.02 s and 1 s.
In one embodiment, the threshold velocity is between 0.02 m/s and 1 m/s.
In one embodiment, the threshold velocity is approximately 0.1 m/s.
In one embodiment, digitally rendering the digital image via the light field display comprises: digitally mapping the digital image on an adjusted image plane designated to provide the user with a designated image perception adjustment; associating adjusted image pixel data with at least some of said pixels according to said mapping; and rendering said adjusted image pixel data via said pixels thereby rendering a perceptively adjusted version of the digital image.
Other aspects, features and/or advantages will become more apparent upon reading of the following non-restrictive description of specific embodiments thereof, given by way of example only with reference to the accompanying drawings.
BRIEF DESCRIPTION OF THE FIGURES
Several embodiments of the present disclosure will be provided, by way of examples only, with reference to the appended drawings, wherein:
FIG. 1 is a schematic representation of a predicted pupil location calculated using a predictive pupil tracking process based on previously acquired pupil locations, according to one embodiment;
FIG. 2 is schematic representation of a pupil location in three-dimensional space, according to one embodiment;
FIG. 3 is a process flow diagram of a predictive pupil tracking method, according to one embodiment;
FIG. 4 is a schematic representation of an effective pupil tracking frequency increased using a predictive pupil tracking process such as that sown in FIG. 3 , according to one embodiment;
FIGS. 5A and 5B are schematic representations of acquired pupil location sequences and forecast pupil locations predicted therefrom, in accordance with at least one embodiment;
FIG. 6 is a process flow diagram illustrating an operational mode of a predictive pupil tracking method, in accordance with at least one of the various embodiments;
FIG. 7 is a process flow diagram illustrating another operational mode of a predictive pupil tracking method, in accordance with at least one of the various embodiments;
FIG. 8 is a process flow diagram of an illustrative ray-tracing rendering process, in accordance with one embodiment;
FIGS. 9 and 10 are process flow diagrams of exemplary input constant parameters and variables, respectively, for the ray-tracing rendering process of FIG. 8 , in accordance with one embodiment;
FIGS. 11A to 11C are schematic diagrams illustrating certain process steps of FIG. 8 ;
FIG. 12 is process flow diagram of an illustrative ray-tracing rendering process, in accordance with another embodiment;
FIGS. 13A to 13D are schematic diagrams illustrating certain process steps of FIG. 12 ; and
FIG. 14 is a schematic state diagram of a predictive pupil tracking system, in accordance with one embodiment.
Elements in the several figures are illustrated for simplicity and clarity and have not necessarily been drawn to scale. For example, the dimensions of some of the elements in the figures may be emphasized relative to other elements for facilitating understanding of the various presently disclosed embodiments. Also, common, but well-understood elements that are useful or necessary in commercially feasible embodiments are often not depicted in order to facilitate a less obstructed view of these various embodiments of the present disclosure.
DETAILED DESCRIPTION
Various implementations and aspects of the specification will be described with reference to details discussed below. The following description and drawings are illustrative of the specification and are not to be construed as limiting the specification. Numerous specific details are described to provide a thorough understanding of various implementations of the present specification. However, in certain instances, well-known or conventional details are not described in order to provide a concise discussion of implementations of the present specification.
Various apparatuses and processes will be described below to provide examples of implementations of the systems and methods disclosed herein. No implementation described below limits any claimed implementation and any claimed implementations may cover processes or apparatuses that differ from those described below. The claimed implementations are not limited to apparatuses or processes having all of the features of any one apparatus or process described below or to features common to multiple or all of the apparatuses or processes described below. It is possible that an apparatus or process described below is not an implementation of any claimed subject matter.
Furthermore, numerous specific details are set forth in order to provide a thorough understanding of the implementations described herein. However, it will be understood by those skilled in the relevant arts that the implementations described herein may be practiced without these specific details. In other instances, well-known methods, procedures and components have not been described in detail so as not to obscure the implementations described herein.
In this specification, elements may be described as âconfigured toâ perform one or more functions or âconfigured forâ such functions. In general, an element that is configured to perform or configured for performing a function is enabled to perform the function, or is suitable for performing the function, or is adapted to perform the function, or is operable to perform the function, or is otherwise capable of performing the function.
It is understood that for the purpose of this specification, language of âat least one of X, Y, and Zâ and âone or more of X, Y and Zâ may be construed as X only, Y only, Z only, or any combination of two or more items X, Y, and Z (e.g., XYZ, XY, YZ, ZZ, and the like). Similar logic may be applied for two or more items in any occurrence of âat least one . . . â and âone or more . . . â language.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.
Throughout the specification and claims, the following terms take the meanings explicitly associated herein, unless the context clearly dictates otherwise. The phrase âin one of the embodimentsâ or âin at least one of the various embodimentsâ as used herein does not necessarily refer to the same embodiment, though it may. Furthermore, the phrase âin another embodimentâ or âin some embodimentsâ as used herein does not necessarily refer to a different embodiment, although it may. Thus, as described below, various embodiments may be readily combined, without departing from the scope or spirit of the innovations disclosed herein.
In addition, as used herein, the term âorâ is an inclusive âorâ operator, and is equivalent to the term âand/or,â unless the context clearly dictates otherwise. The term âbased onâ is not exclusive and allows for being based on additional factors not described, unless the context clearly dictates otherwise. In addition, throughout the specification, the meaning of âa,â âan,â and âtheâ include plural references. The meaning of âinâ includes âinâ and âon.â
As used in the specification and claims, the singular forms âaâ, âanâ and âtheâ include plural references unless the context clearly dictates otherwise.
The term âcomprisingâ as used herein will be understood to mean that the list following is non-exhaustive and may or may not include any other additional suitable items, for example one or more further feature(s), component(s) and/or element(s) as appropriate.
The systems and methods described herein provide, in accordance with different embodiments, different examples of a pupil tracking system and method, wherein one or more previously acquired pupil (center) locations can be used to generate and predict one or more future pupil (center) locations, compute an average or current pupil displacement velocity and/or trajectory, or other pupil displacement dynamics as may be relevant to the application at hand. In doing so, in accordance with some embodiments or applications, a corresponding rendering of a perceived image that relies at least in part on pupil tracking inputs, can now take into account not only one or more of a current, past and/or future predicted pupil location and/or gaze direction, but also a past, current and/or future predicted pupil location trajectory and/or velocity, which can ultimately result in providing an increase in the effective rate of pupil tracking (and related image re-rendering), a reduction in re-rendering jitteriness for predictively fixated (and/or pre- and/or post-fixated) pupil dynamics despite ongoing pupil movement capture, and/or other like rendering dynamic improvements. For example, in some such embodiments, a digital display device and digital image rendering system and method are provided that rely, at least in part, on pupil tracking to adjust an output image thereof. For example, an image to be displayed can be adjusted, at least in part, as a function of a tracked user pupil location. In accordance with some of the herein-described embodiments, an output image can therefore be adjusted not only as a function of an available user pupil location, but also or alternatively as a function an acquired and/or predicted user pupil location, trajectory and/or velocity, for example, where an image refresh rate is higher than a pupil tracking rate, and/or to apply a variable rate to image re-rendering and/or to a rendering geometry adjustment mechanism applied to account for pupil displacement (e.g. within a context of a lightfield display or like user-specific directional view generating display devices).
For instance, while existing gaze tracking applications rely on real-time pupil location acquisitions to monitor a user's gaze direction in evaluating what is currently drawing their attention, such gaze tracking systems and methods are typically either insufficiently rapid or precise to support real-time applications requiring high resolution and high accuracy pupil location tracking. For example, the trade-off for operating real-time gaze trackers (e.g. trackers operating on a timescale in the order of roughly 100 ms) is generally a low spatial accuracy, which may nonetheless suffice to monitor a general user gaze direction, whereas higher accuracy solutions will typically be much slower. Accordingly, current solutions are not generally amenable to address applications where both a higher temporal resolution and spatial accuracy may be required, e.g. where current gaze tracking solutions would generate prohibitive lag times and/or adversely impact a user experience. Furthermore, while predictive eye tracking can result in increased tracking and corresponding image rendering rates for improved spatial image rendering geometry accuracy, predictive eye tracking techniques as described herein may also allow for such high precision, high accuracy pupil-specific image rendering processes to accommodate different view modes, for example, to dynamically adjust pupil displacement impacts on image rendering based on acquired and predicted pupil dynamics, e.g. as a viewer alternates between moving and fixated view periods, as will be described in greater detail below.
For example, in accordance with some of the embodiments herein described, pupil location tracking and/or prediction may play an important role in light field display systems, wherein a rendered image(s) provides an optimal viewing experience in a defined region(s) of viewing space, herein referred to as a view zone, or viewing zone. In such embodiments, applying predictive techniques based on acquired pupil locations and derived pupil velocity/trajectory considerations can result in a significantly improved viewer experience whereby a relatively fixated gaze can be recognized by virtue of reduced pupil velocities or likewise recognizable fixated pupil location dynamics (e.g. constrained trajectory, limited displacement amplitudes, recognizable behavioural pupil dynamics for a particular activity such as reading, etc.), thus invoking a âfixatedâ (and/or pre- and/or post-fixated) viewing mode or state in which an image rendering geometry is not as often updated for pupil location, thus significantly reducing potentially perceived image jitteriness and/or stability. Comparatively, where captured pupil locations are suggestive of significant pupil displacements, the pupil tracking system and correlated image rendering process can migrate to a âmovingâ mode whereby image rendering dynamics and geometries are more rapidly updated to accommodate such movement.
For example, in some of the herein-described embodiments, a pupil tracking system and method is implemented for the purposes of applying adaptive image corrections or adjustments in a light field display system or device, whereby acquisition of a temporally and spatially accurate pupil location, in three-dimensions, is important in the delivery of a positive user experience. For example, certain embodiments involve the provision of corrective image rendering through light field shaping optics so to correct for a user's reduced visual acuity. An exemplary application for the herein-described embodiments is described in Applicant's U.S. Pat. No. 10,394,322, Applicant's co-pending U.S. patent application Ser. Nos. 16/510,673, 16/569,137, and 16/551,572, the entire contents of each of which are hereby incorporated herein by reference. An example drawn therefrom is also described below, in accordance with one embodiment. In such embodiments, high pupil location accuracy may be appreciated to ensure desired image corrections are adequately generated while minimizing the production of optical artefacts that may otherwise be distracting to the viewer. Given the high spatial resolution considered to implement such corrections, a high temporal sensitivity can also be addressed as slight displacements in the viewer's pupils may bring forth significant changes in ray tracing, or like vision correction computations, applied to compute the various optical views provided through the light-field display and its impact on image correction and focused image rendering. As the viewer's eyes can readily perceive fluctuations within a temporal range of a few dozen milliseconds, a temporal pupil tracking resolution may be required in this order, in some embodiments, to ensure a quality user experience. Namely, pupil tracking outputs may be preferred on timescales similar to, or in the order of, an image refresh rate, so to ensure that appropriate image rendering is provided to provide the desired visual compensation without introducing adverse visual effects or delays. Conversely, and in accordance with some embodiments, where pupil displacements are tracked and/or predicted to remain within a relatively confined viewing zone, for example as prescribed or bounded by display hardware, optics and/or viewer positioning, a rendering geometry of the lightfield display may be maintained so not to overly refresh, for example, ray tracing and/or view zone pixel allocations, thereby reducing or minimizing perceived image rendering jitteriness that could otherwise be perceived due to an oversensitive pupil tracking and image rendering system. Indeed, a viewer identifiable as being within a fixated or static view configuration (i.e. where pupil displacements are predictively contained within or reasonably around a designated view zone, eye box, etc.), may ultimately have a better viewing experience if image rendering dynamics/geometries are not as frequently updated, for instance, favouring image rendering stability over spatial accuracy. A highly spatially and temporally sensitive system may nonetheless be preferred where the viewer's fixated mode migrates to a moving mode, in which pupil tracking and rendering accuracy and precision may be of greater importance to an effective viewer experience.
Given the temporal constraints and considerations noted above, predictive pupil tracking can be implemented, in accordance with some of the herein-described embodiments, so to mitigate delayed optical effects that may impact a viewer's experience and consequently provide for a better overall user experience, while also or alternatively mitigating jittery optical/image rendering effects that may be perceived when a viewer is otherwise mostly in a static or fixated viewing state.
The following will provide different examples of pupil tracking and correlated image rendering techniques that rely on acquired and/or predicted pupil locations, velocities and/or trajectories to improve a user experience, as introduced above.
With reference to FIG. 1 , and in accordance with one exemplary embodiment, a predictive pupil tracking system, generally referred to using the numeral 100 , will now be described. In the illustrated embodiment of FIG. 1 , the system 100 relies on one or more pupil tracking devices or systems 105 to output a current pupil location. These may include, without limitation, any system using corneo-scleral reflections (i.e. glints) on the user's eye, from one or more IR or near-IR light sources or the like (for either bright and/or dark pupil tracking); or computer vision-based methods using feature recognition applied to an image of the user's face obtained via a digital camera of the like.
Note that different devices using different technologies may be used in combination, for example, to leverage computation efficiencies in tracking and/or monitoring a user's eye and/or pupil location in different environments, and/or to provide metrics by which system accuracies can be evaluated, and different approaches weighted accordingly to provide higher overall system accuracies. Furthermore, different techniques may be implemented, for example, to reduce overall system power consumption, computational load, reduce hardware load requirements and/or reduce the viewer's exposure to various light probes (e.g. IR, Near-IR probes) typically used in glint-based pupil locating processes. For example, machine vision implementations may be relied upon at a first level to adequately locate and track facial features such as the user's eyes, pupils and pupil centers, whereas higher-resolution glint-based techniques may be layered thereon (e.g. via IR/NIR illumination) to refine and/or confirm machine vision results at a lower frequency, thus reducing IR/NIR emissions which may be unfavourable in certain conditions but may otherwise be required in other low lighting conditions. Similarly, different spatial estimation techniques may be applied to, again, reduce computational load by, for example, estimating pupil center locations using machine vision techniques by predominantly tracking eye locations (which are easier to track in general) and confirming pupil locations and/or centers at lower refresh rates. These and other techniques may be considered herein without departing from the general scope and nature of the present disclosure.
With continued reference to FIG. 1 , generally, device(s) 105 is(are) operable to provide a sequence of pupil center positional data 109 of a user (e.g. 3D position of the pupil center) in real-time or near real-time. For instance, where different techniques are used to computed pupil center locations 109 , these different outputs may be combined, averaged and/or otherwise statistically compiled to produce pupil center location information useable in subsequent steps. For example, in some embodiments, a machine-vision based approach may be used to first estimate a location of the pupils. This estimation may rely on various facial feature identification and/or extraction techniques, for example, but not limited to, by searching for and/or identifying the curvature of the eye(s), the dark pupil centers in contract with the sclera, etc., in combination, for example, with one or more glint-based techniques that, for example, may be constrained to previously machine-identified eye/pupil regions and/or be used a confirmation, validation or recalibration of such techniques. In some examples, past pupil locations may not only be used, directly or indirectly through one or more encoded variations or transformations thereof, to output predictive pupil location information, but also to seed pupil location measurements, for example, in the context of a machine vision pupil search algorithm or the like.
With continued reference to FIG. 1 , the system 100 uses, at least in part, data 109 as an input to a Prediction Engine 113 configured to analyze and generate therefrom one or more temporally predictive pupil locations 119 based on characteristic patterns automatically derived and interpreted from input data 109 . For instance, one or more predictive data modeling techniques may be used by Prediction Engine 113 to extract one or more parameters representative of monitored real-time pupil location variation, and generate or construct therefrom a mathematical representation or model operable to output predictive pupil locations 119 . Some of these techniques will be discussed below, without limitation.
In some embodiments, one or more temporally predictive modeling methods (statistical or otherwise) can be used by Prediction Engine 113 to generate a predictive pupil location sequence 119 . These may include, but are not limited to: moving averages, exponential smoothing, linear and/or non-linear regressions, spline interpolation, Box-Jenkins forecasting methods, Kalman Filters, alpha-beta filters, non-parametric models such as Gaussian Process Models and/or neural networks (including convolutional, recurrent or recursive neural networks). Other filters may also or alternatively include a weighted median filter, or the like. Generally, any amount of previously generated pupil location data, and/or data derived therefrom (e.g. velocity, acceleration, displacement trends or patterns, etc.) may be used in the estimation or extrapolation of the pupil center location to produce predictably reliable results. In some cases, a trajectory model (e.g. probable pupil location as a function time) from past data points may be extrapolated or projected beyond the last data point (pupil center location) to obtain an estimated trajectory (as a function of time) of (probable) future pupil locations. Moreover, any number of estimated locations may be generated from the estimated trajectory while waiting for the next true pupil center location measurement, which can then be relied upon to refine the estimated trajectory and iteratively apply appropriate correction thereto to output ongoing predictive pupil location data. As noted above, while a predicted future pupil location may be used to predictively induce a corresponding image rendering process (e.g. to predictively output an appropriate image rendering geometry and/or perspective), acquired pupil tracking data may also or otherwise be used to compute a current or predicted pupil trajectory, and/or again consider a current or average pupil velocity, so to effectively predict the likelihood that the viewer's pupil will sufficiently move within a forecasted time period to warrant impacting/adjusting current image rendering parameters.
In some embodiments, each pupil center location obtained from the pupil tracking device or system 105 may also comprise measurement errors associated therewith. These errors, if present, may be used by Prediction Engine 113 when generating the estimated pupil center sequence 113 . The methods for incorporating such measurement errors in the modelling methods described above are well known in the art.
As shown in FIG. 2 , and in accordance with one embodiment, a pupil location is the three- dimensional position 212 of the pupil center 215 measured from a reference point 218 . While the pupil moves slightly within the eye depending on where a user is focusing his/her gaze, the head and body of the user itself may move as well. Within the context of a vision correction application, or other 3D light field image perception adjustment applications, the pupil location in three dimensional space is generally set relative to a location of a light field display screen such that, in some embodiments, appropriate ray tracing processes can be implemented to at least partially govern how light emanated from each display pixel (of interest) is appropriately channeled through a corresponding light field shaping layer and relayed to the viewer's pupil. Naturally, as a viewer's pupil location changes relative to the display, so will corrective or otherwise adjusted pixel data change to adjust the output pixelated image accordingly. Accordingly, the light field display will generally include, or be associated with, related pupil tracking hardware such as one or more light sources (e.g. IR/NIR) and/or cameras (visible, IR, NIR) and related pupil tracking firmware/software. Further details in respect of one illustrative embodiment will be described below.
With reference now to FIG. 3 , and in accordance with one exemplary embodiment, a predictive pupil tracking method using system 100 described above, and generally referred to using the numeral 300 , will now be described. The above-described system 100 uses a sequence of pupil locations to generate predictive estimations of future pupil locations. As noted above, it will be appreciated that other direct, derived or transformed pupil location data may be used to this end. For simplicity, the following examples will focus on predictive trajectory models based on a time-ordered series of previously stored pupil locations.
The system described may thus be leveraged to complement or improve these pupil-tracking systems by generating one or more future pupil locations while another system or device is waiting for the eye or pupil tracking systems to acquire/compute a new location. Thus, the method described herein may provide for an improved frequency at which pupil locations are provided as output to another system or method. For instance, output of a current pupil location may be delayed due to processing load and/or lag times, resulting in the output, in some applications, of somewhat stale data that, for example, when processed within the context of highly sensitive light field rendering applications (that will invariably introduce their own computational lag), result in the provision of a reduced viewer experience. Conversely, viewer experience may also or otherwise be adversely affected if pupil-tracking systems perceive a user pupil to have shifted, for instance through digitization of user pupil positions, error in pupil location measurements, or minor spurious pupil movements from an otherwise stationary user. Such a phenomenon may result in a re-rendering of an image or adjustment of an image rendering geometry, in a situation where user experience may be improved, for instance, by not adjusting pixel data at all. Namely, an image rendered with the intent of providing a designated image perception for a given input pupil location may be unsatisfactorily rendered for the viewer if the viewer's pupil location changed significantly, or erroneously perceived to have changed, while image rendering computations were being implemented. Accordingly, computational lag times, combined with the generally high refresh rates required to provide an enjoyable viewer experience, may introduce undesirable effects given at times noticeable pupil location changes, or a light field display refreshes unnecessarily due to inaccurate instantaneous perception of movement. Using predictive pupil location data in light field rendering applications, as considered herein, may thus mitigate issues common with the use of otherwise misleading pupil location data.
Accordingly, the systems and methods described herein may be used to advantage in light field rendering methods or systems in which the pupil center position of a user is used to generate a light field image via a light field capable display or the like. Indeed, the predictive pupil tracking method described herein, according to some embodiments, may make use of past pupil positional data to improve the speed or frequency at which the pupil center position, which may be a moving target, is available to a light field ray tracing algorithm, or like light field rendering process. Since the light field rendering embodiments described above rely, in part, on having an accurate pupil center location, the speed or frequency at which the pupil positional information is extracted by the pupil tracker may become a bottleneck for the light field rendering algorithm. A 60 Hz digital display (most phone displays, for example) will have a refresh rate of about 15 ms, whereas higher frequency displays (e.g. 120 Hz displays) have much faster refresh rates, which imposes significant constraints on the computation and output of accurate pupil tracking data, particularly when combined with computation loads involved in most light field rendering applications. For instance, for an optimal light field output experience, a rendered light field should be refreshed at or around the display screen's refresh rate. This refresh rate should naturally align with a current location of the user's pupil at that time and thus, benefits from a predictive pupil tracking approach that can extrapolate, from current data, where the pupil will actually be when the screen next refreshes to render a new light field output. Otherwise, the lack of temporal accuracy may lead to a reduced visual experience. Conversely, the importance of a high refresh rate for many applications in which a user is moving may unduly prioritise computational resources for image refreshing when a user is substantially stationary, or pupils are moving at low velocity, which, for at least the abovementioned reasons, can also adversely affect user experience. Available computational power may thus be leveraged instead to predict or estimate, based on previous known (e.g. measured) pupil center locations, an estimated future location of the pupil center and selectively use this estimation to update the light field image, as appropriate, while waiting for the next true pupil center location measurement, thereby resulting in a smoother viewing experience.
Coming back to FIG. 3 , a pupil location iterative refresh cycle is started at step 305 . The method first checks at step 309 if, at this time, an actual measured pupil location is available from the one or more pupil tracking device or system 105 . If this is the case, the method outputs the measured pupil location at step 313 . If this is not the case, then at step 317 , the method checks to see if enough prior pupil center locations (as measured by one or more pupil tracking device or system 105 ) have been recorded to provide enough data for prediction engine 113 to provide an accurate predicted one or more future pupil locations. If this is not the case, then the method goes back to step 305 . If enough data is available, then the method uses, at step 321 , Prediction Engine 113 to generate the most probable trajectory (position as a function of time) of future pupil locations. It may then, at step 325 , extract one or more future pupil locations from this trajectory, which are then fed back as output (step 313 ). The method loops back to step 305 once more. Therefore, the method as described above, may ensure that measured pupil locations are outputted and used as soon as possible, while relying on Prediction Engine 113 to generate data points in between.
Similarly, predictive pupil tracking data can be used to accommodate predefined light field rendering lags, for example, where a pupil location is required early on in light field rendering computations (e.g. ray tracing) to output corrective or adaptive pixel data for rendering. Accordingly, rather than to compute ray traces, for example, on the basis of a current pupil location output, such computations may rely on a predictive location so that, when the corrected or adjusted image is finally computed and ready for display, the user's pupil is most likely now located at the predicted location and thus in an ideal location to best view the rendered image. A predictive location may also be identified as one in which the image currently being displayed requires no further adjustment (i.e. the user's pupil is most likely already located in or around an ideal location to best view the rendered image), for example if the user pupil is stationary or moving slowly. In such a situation, light field rendering computations may be bypassed altogether for a time in favour of saving computational resources or improving user experience. These and other time lapse, lags and synchronization considerations may readily apply in different embodiments, as will be readily appreciated by the skilled artisan.
FIG. 4 shows an exemplary schematic diagram relating a consecutive sequence of pupil location measurements with a corresponding time sequence (by a single unit of time for simplicity). Hence, the sequence from N to N+1 implies a time difference of one unit. Therefore, by using past pupil locations (N, Nâ1, Nâ2, etc.) to generate a most probable future pupil location at time T+½ (for example), the frequency at which pupil locations are available is effectively increased by a factor of two. Likewise, a predictable pupil location may be forecasted when addressing higher computation load processes.
FIG. 5A shows the positional change corresponding to the time sequence illustrated in FIG. 4 . The skilled technician will understand that the use of a 2D representation is only for demonstration purposes and that an additional depth component can also normally be used. As explained above, each point (Tâ2, Tâ1 and T) represents a sequence of measured pupil center locations, separated in time. At time T, while waiting for the next measurement (the result of which will be available at time T+1), previous measurements (N, Nâ1, and Nâ2 from times T, Tâ1 and Tâ2 in this example) may be used to generate an estimated trajectory 510 of probable future pupil center location and extract therefrom an estimated future pupil location 520 at time T+½.
As will be appreciated by the skilled artisan, gaze or pupil tracking comprises an important element of many light field display systems, such as those comprising an array of light field shaping elements (e.g. microlens arrays, apertures, and the like), which may produce the highest quality images within a specific region(s) of space, or a view zone. User experience may therefore be improved when an image is rendered taking into account a user pupil location or predicted location. Referencing again FIG. 5A , a light field image rendered at time T may therefore be optimally viewed within a view zone 530 . A view zone geometry may be defined by the light field display components and/or light field shaping element sizes and/or geometries. One skilled in the art will therefore readily appreciate that while the view zone 530 is represented with a boundary 540 that is represented as circular in FIG. 5A , such a boundary may be hexagonal, rectangular, stretched hexagonal, etc., and is not limited to two dimensions. In this example, if the pupil location at time T is utilized to render an image for a moving viewer, who will then view the image at the pupil location at time T+½, the viewer may not receive a high quality image at time T+½, as the pupil location may then lie outside of the view zone for which the image was optimally rendered. However, by estimating the trajectory 510 of the user's pupil over time, a prediction engine, such as that described above as element 113 of FIG. 1 , may, in accordance with at least one embodiment, estimate pupil location coordinates at time T+½ in order to project an image corresponding to a view zone that may encompass the predicted pupil location 520 , thereby providing a more positive viewing experience.
Similarly, FIG. 5B highlights yet another embodiment in which a prediction engine 113 may improve viewer experience. In this example, a user pupil location follows an initial trajectory similar to that shown in FIG. 5A , as denoted by the pupil locations, in order, Tâ5, Tâ4, and Tâ3. However, in this example, in contrast to that of FIG. 5A , a user pupil slows in its movement after Tâ3. In this example, the user pupil may be measured as having a trajectory and/or velocity small enough that its position 522 at time T+½ may still lie within a boundary 542 of the view zone 532 produced at time T. In this case, and in accordance with at least one embodiment, adjusting an image rendering geometry (e.g. geometrically re-allocating pixel values based on a distinctly computed optimal view zone) so re-render a digital image (e.g. for a static image) or impact rendering of future time-sequenced images (e.g. for a dynamic image) may not correspond to an improvement of user experience, but may even be detrimental thereto. For at least the reasons discussed above, it may be beneficial to therefore not refresh and/or re-render a display geometry in favour of providing a stable image geometry if a prediction engine 113 predicts a pupil location 522 that will not significantly deviate in space from previous recorded locations.
Accordingly, a prediction engine such as that depicted in FIG. 1 as herein described may utilise a number of pupil positions and/or velocity data, or calculated values related thereto, to improve user experience. In accordance with at least one embodiment, it may be sufficient to measure or calculate a user pupil velocity in order to predict that an image re-rendering may be unnecessary, if, for instance, a predicted pupil location is within an existing view zone. Such a prediction may be performed using said velocity, as well as optionally
CLAIMS
Claims ( 27 )
What is claimed is:
1. A computer-implemented method, automatically implemented by one or more digital processors, for reducing jitteriness or increasing stability in a perceived light field image projected via a light field display, the method comprising:
sequentially acquiring a user pupil location;
digitally computing from at least some said sequentially acquired user pupil location a velocity of said user pupil location over time;
digitally comparing said velocity with a designated threshold pupil velocity, wherein velocities below said designated threshold are associated with a relatively fixated state;
digitally rendering the light field image via the light field display in accordance with a maintained spatially-defined light field viewing zone geometry digitally defined in respect of a previously acquired user pupil location to project the light field image within a maintained spatially-defined light field viewing zone in accordance with said previously acquired user pupil location so to reduce jitteriness or increase stability of the perceived light field image, unless said velocity is above said designated threshold pupil velocity; and
upon said velocity exceeding said designated threshold pupil velocity, digitally adjusting a rendering geometry of the light field image via the light field display so to correspondingly adjust said maintained spatially-defined light field viewing zone geometry to project the light field image within an adjusted spatially-defined light field viewing zone in accordance with a newly acquired user pupil location.
2. The computer-implemented method of claim 1 , further comprising digitally adjusting said rendering geometry of the light field image via the light field display so to correspondingly adjust said maintained spatially-defined light field viewing zone geometry to correspond to a function of said newly acquired user pupil location upon a designated condition for movement of said maintained spatially-defined light field viewing zone geometry being met.
3. The computer-implemented method of claim 2 , wherein said designated condition for movement of said maintained spatially-defined light field viewing zone geometry comprises at least one of said user pupil location crossing a defined boundary of said maintained spatially-defined light field viewing zone geometry, said maintained light field viewing zone geometry remaining static for a prescribed period of time, or said velocity is greater than a distinct predetermined threshold.
4. The computer-implemented method of claim 2 , wherein said function is an interpolation of said newly acquired user pupil location and said maintained spatially-defined light field viewing zone geometry.
5. The computer-implemented method of claim 2 , wherein said function is a function of time since said designated condition for movement was met.
6. The computer-implemented method of claim 4 , where said interpolation is calculated for a designated period of time after said designated condition was met.
7. The computer-implemented method of claim 6 , wherein said designated period of time is between about 0.02 s and 1 s.
8. The computer-implemented method of claim 1 , wherein said threshold velocity is between 0.02 m/s and 1 m/s.
9. The computer-implemented method of claim 8 , wherein said threshold velocity is approximately 0.1 m/s.
10. The computer-implemented method of claim 1 , wherein said digitally rendering the light field image via the light field display comprises:
digitally mapping a digital image on an adjusted image plane designated to provide the user with a designated image perception adjustment;
associating adjusted image pixel data with at least some of said pixels according to said mapping; and
rendering said adjusted image pixel data via said pixels thereby rendering said light field image corresponding to a perceptively adjusted version of the digital image.
11. A non-transitory computer-readable medium having instructions stored thereon to be automatically implemented by one or more processors to reduce jitteriness or increase stability in a perceived light field image projected via a light field display by:
sequentially acquiring a user pupil location;
digitally computing from at least some said sequentially acquired user pupil location a velocity of said user pupil location over time;
digitally comparing said velocity with a designated threshold pupil velocity, wherein velocities below said designated threshold are associated with a relatively fixated state;
digitally rendering the light field image via the light field display in accordance with a maintained spatially-defined light field viewing zone geometry digitally defined in respect of a previously acquired user pupil location to project the light field image within a maintained spatially-defined light field viewing zone in accordance with said previously acquired user pupil location so to reduce jitteriness or increase stability of the perceived light field image, unless said velocity is above said designated threshold pupil velocity; and
upon said velocity exceeding said designated threshold pupil velocity, digitally adjusting a rendering geometry of the light field image via the light field display so to correspondingly adjust said maintained spatially-defined light field viewing zone geometry to project the light field image within an adjusted spatially-defined light field viewing zone in accordance with a newly acquired user pupil location.
12. The non-transitory computer-readable medium of claim 11 , further comprising digitally adjusting said rendering geometry of the light field image via the light field display so to correspondingly adjust said spatially-defined light field viewing zone geometry to correspond to a function of said newly acquired user pupil location upon a designated condition for movement of said spatially-defined light field viewing zone geometry is met.
13. The non-transitory computer-readable medium of claim 12 , wherein said designated condition for movement of said spatially-defined light field viewing zone geometry comprises at least one of said user pupil location crossing a defined boundary of said maintained spatially-defined light field viewing zone geometry, said maintained spatially-defined light field viewing zone geometry remaining static for a prescribed period of time, or said velocity is greater than a distinct predetermined threshold.
14. The non-transitory computer-readable medium of claim 12 , wherein said function is an interpolation of said newly acquired user pupil location and said maintained spatially-defined light field viewing zone geometry.
15. The non-transitory computer-readable medium of claim 12 , wherein said function is a function of time since said designated condition for movement was met.
16. The non-transitory computer-readable medium of claim 14 , where said interpolation is calculated for a designated period of time after said designated condition was met.
17. The non-transitory computer-readable medium of claim 16 , wherein said designated period of time is between about 0.02 s and 1 s.
18. The non-transitory computer-readable medium of claim 11 , wherein said threshold velocity is between 0.02 m/s and 1 m/s.
19. The non-transitory computer-readable medium of claim 18 , wherein said threshold velocity is approximately 0.1 m/s.
20. The non-transitory computer-readable medium of claim 11 , wherein said digitally rendering the light field image via the light field display comprises instructions for:
digitally mapping a digital image on an adjusted image plane designated to provide the user with a designated image perception adjustment;
associating adjusted image pixel data with at least some of said pixels according to said mapping; and
rendering said adjusted image pixel data via said pixels thereby rendering said light field image corresponding to a perceptively adjusted version of the digital image.
21. A digital display device operable to automatically adjust a light field image to be rendered thereon, the device comprising:
a light field display;
a hardware processor; and
a pupil tracking engine operable by said hardware processor to automatically:
receive as input sequential user pupil locations;
digitally compute from at least some said sequential user pupil locations a velocity of said user pupil location over time; and
digitally compare said velocity with a designated threshold pupil velocity, wherein velocities below said designated threshold are associated with a relatively fixated state;
wherein said hardware processor is operable to reduce jitteriness or increase stability in a perceived light field image by:
digitally rendering the light field image via the light field display in accordance with a maintained spatially-defined light field viewing zone geometry digitally defined in respect of a previously acquired user pupil location to project the light field image within a maintained spatially defined light field viewing zone in accordance with said previously acquired user pupil location so to reduce jitteriness or increase stability of the perceived light field image, unless said velocity is above said designated threshold pupil velocity; and
upon said velocity exceeding said designated threshold pupil velocity, digitally adjusting a rendering geometry of the light field image via the light field display so to correspondingly adjust said maintained spatially-defined light field viewing zone geometry to project the light field image within an adjusted spatially-defined light field viewing zone in accordance with a newly acquired user pupil location.
22. The digital display device of claim 21 , further comprising digitally adjusting said rendering geometry of the light field image via the light field display so to correspondingly adjust said maintained spatially-defined light field viewing zone geometry to correspond to a function of said newly acquired user pupil location upon a designated condition for movement of said maintained spatially-defined light field viewing zone geometry is met.
23. The digital display device of claim 22 , wherein said designated condition for movement of said maintained spatially-defined viewing zone geometry comprises at least one of said user pupil location crossing a defined boundary of said maintained spatially-defined light field viewing zone geometry, said maintained spatially-defined light field viewing zone geometry remaining static for a prescribed period of time, or said velocity is greater than a distinct predetermined threshold.
24. The digital display device of claim 22 , wherein said function is an interpolation of said newly acquired user pupil location and said maintained spatially-defined light field viewing zone geometry.
25. The digital display device of claim 22 , wherein said function is a function of time since said designated condition for movement was met.
26. The digital display device of claim 24 , where said interpolation is calculated for a designated period of time after said designated condition was met.
27. The digital display of claim 21 , wherein said hardware processor is operable to digitally render the light field image via the light field display by:
digitally mapping a digital image on an adjusted image plane designated to provide the user with a designated image perception adjustment;
associating adjusted image pixel data with at least some of said pixels according to said mapping; and
rendering said adjusted image pixel data via said pixels thereby rendering said light field image corresponding to a perceptively adjusted version of the digital image.
US17/239,385
2019-04-01
2021-04-23
Pupil tracking system and method, and digital display device and digital image rendering system and method using same
Active
US11385712B2
( en )
Priority Applications (2)
Application Number
Priority Date
Filing Date
Title
US17/239,385
US11385712B2
( en )
2019-04-01
2021-04-23
Pupil tracking system and method, and digital display device and digital image rendering system and method using same
US17/831,273
US11644897B2
( en )
2019-04-01
2022-06-02
User tracking system using user feature location and method, and digital display device and digital image rendering system and method using same
Applications Claiming Priority (6)
Application Number
Priority Date
Filing Date
Title
CA3038584A
CA3038584A1
( en )
2019-04-01
2019-04-01
Pupil tracking system and method, and digital display device and digital image rendering system and method using same
CACA3038584
2019-04-01
CA3038584
2019-04-01
US201962929599P
2019-11-01
2019-11-01
PCT/IB2020/053035
WO2020201999A2
( en )
2019-04-01
2020-03-31
Pupil tracking system and method, and digital display device and digital image rendering system and method using same
US17/239,385
US11385712B2
( en )
2019-04-01
2021-04-23
Pupil tracking system and method, and digital display device and digital image rendering system and method using same
Related Parent Applications (1)
Application Number
Title
Priority Date
Filing Date
PCT/IB2020/053035
Continuation
WO2020201999A2
( en )
2019-04-01
2020-03-31
Pupil tracking system and method, and digital display device and digital image rendering system and method using same
Related Child Applications (1)
Application Number
Title
Priority Date
Filing Date
US17/831,273
Continuation-In-Part
US11644897B2
( en )
2019-04-01
2022-06-02
User tracking system using user feature location and method, and digital display device and digital image rendering system and method using same
Publications (2)
Publication Number
Publication Date
US20210271319A1
US20210271319A1 ( en )
2021-09-02
US11385712B2
true
US11385712B2 ( en )
2022-07-12
Family
ID=72664371
Family Applications (1)
Application Number
Title
Priority Date
Filing Date
US17/239,385
Active
US11385712B2
( en )
2019-04-01
2021-04-23
Pupil tracking system and method, and digital display device and digital image rendering system and method using same
Country Status (3)
Country
Link
US
( 1 )
US11385712B2
( en )
EP
( 1 )
EP3948402B1
( en )
WO
( 1 )
WO2020201999A2
( en )
Cited By (1)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US20230025540A1
( en )
*
2020-06-11
2023-01-26
Guangzhou Automobile Group Co., Ltd.
Method for visually tracking gaze point of human eye, vehicle early warning method and device
Families Citing this family (18)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US11615285B2
( en )
2017-01-06
2023-03-28
Ecole Polytechnique Federale De Lausanne (Epfl)
Generating and identifying functional subnetworks within structural networks
CN115842907A
( en )
*
2018-03-27
2023-03-24
京ä¸æ¹ç§æéå¢è¡ä»½æéå ¬å¸
Rendering method, computer product and display device
US11663478B2
( en )
2018-06-11
2023-05-30
Inait Sa
Characterizing activity in a recurrent artificial neural network
US11972343B2
( en )
2018-06-11
2024-04-30
Inait Sa
Encoding and decoding information
US12412072B2
( en )
2018-06-11
2025-09-09
Inait Sa
Characterizing activity in a recurrent artificial neural network
US11893471B2
( en )
2018-06-11
2024-02-06
Inait Sa
Encoding and decoding information and artificial neural networks
US11652603B2
( en )
2019-03-18
2023-05-16
Inait Sa
Homomorphic encryption
US11569978B2
( en )
2019-03-18
2023-01-31
Inait Sa
Encrypting and decrypting information
US11580401B2
( en )
2019-12-11
2023-02-14
Inait Sa
Distance metrics and clustering in recurrent neural networks
US11816553B2
( en )
2019-12-11
2023-11-14
Inait Sa
Output from a recurrent neural network
US11651210B2
( en )
2019-12-11
2023-05-16
Inait Sa
Interpreting and improving the processing results of recurrent neural networks
US11797827B2
( en )
*
2019-12-11
2023-10-24
Inait Sa
Input into a neural network
US11494883B1
( en )
*
2020-12-16
2022-11-08
Meta Platforms Technologies, Llc
Image correction
US12380599B2
( en )
2021-09-13
2025-08-05
Inait Sa
Characterizing and improving of image processing
CN115002444B
( en )
*
2022-05-26
2024-06-04
京ä¸æ¹ç§æéå¢è¡ä»½æéå ¬å¸
Display module, display method thereof, display device and virtual display equipment
WO2023233207A1
( en )
*
2022-06-02
2023-12-07
Evolution Optiks Limited
User tracking system and method, and digital display device and digital image rendering system and method using same
CN115113733B
( en )
*
2022-07-07
2025-11-07
ä¸æçµåï¼ä¸å½ï¼ç åä¸å¿
Information generation method and device
JP7576064B2
( en )
*
2022-08-25
2024-10-30
ç¢å´ç·æ¥æ ªå¼ä¼ç¤¾
Vehicle display device
Citations (23)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US6669651B1
( en )
*
2000-02-24
2003-12-30
Matsushita Electric Works, Ltd.
Non-invasive brain function examination
US7572008B2
( en )
2002-11-21
2009-08-11
Tobii Technology Ab
Method and installation for detecting and following an eye and the gaze direction thereof
US8066375B2
( en )
2005-10-10
2011-11-29
Tobii Technology Ab
Eye tracker having an extended span of operating distances
US8120577B2
( en )
2005-10-28
2012-02-21
Tobii Technology Ab
Eye tracker with visual feedback
US8342687B2
( en )
2009-10-08
2013-01-01
Tobii Technology Ab
Eye-tracking using a GPU
US8824779B1
( en )
2011-12-20
2014-09-02
Christopher Charles Smyth
Apparatus and method for determining eye gaze from stereo-optic views
US20140327771A1
( en )
*
2013-05-01
2014-11-06
Nvidia Corporation
System, method, and computer program product for displaying a scene as a light field
US8885882B1
( en )
2011-07-14
2014-11-11
The Research Foundation For The State University Of New York
Real time eye tracking for human computer interaction
US20150223684A1
( en )
2014-02-13
2015-08-13
Bryson Hinton
System and method for eye tracking
US20150338915A1
( en )
*
2014-05-09
2015-11-26
Eyefluence, Inc.
Systems and methods for biomechanically-based eye signals for interacting with real and virtual objects
US20160148050A1
( en )
2014-11-24
2016-05-26
Hyundai Motor Company
Apparatus and method for detecting eyes
US20160166146A1
( en )
*
2014-12-11
2016-06-16
Icspi Corp.
Eye-Tracking System and Method Therefor
US20160335475A1
( en )
2014-02-04
2016-11-17
Fraunhofer-Gesellschaft Zur Foerderung Der Angewandten Forschung E.V.
3d image analyzer for determining the gaze direction
US20170011492A1
( en )
2013-03-04
2017-01-12
Tobii Ab
Gaze and saccade based graphical manipulation
US20170353717A1
( en )
*
2014-12-29
2017-12-07
Beijing Zhigu Tech Co., Ltd.
Light field display control methods and apparatus, and light field display devices
WO2018000020A1
( en )
2016-06-29
2018-01-04
Seeing Machines Limited
Systems and methods for performing eye gaze tracking
US20180008141A1
( en )
2014-07-08
2018-01-11
Krueger Wesley W O
Systems and methods for using virtual reality, augmented reality, and/or a synthetic 3-dimensional information for the measurement of human ocular performance
US20180011533A9
( en )
2014-05-09
2018-01-11
Google Inc.
Systems and methods for biomechanically-based eye signals for interacting with real and virtual objects
US20180136722A1
( en )
2015-09-04
2018-05-17
Sony Interactive Entertainment Inc.
Apparatus and method for dynamic graphics rendering based on saccade detection
US20180143684A1
( en )
2014-02-21
2018-05-24
Tobii Ab
Apparatus and method for robust eye/gaze tracking
US20180181809A1
( en )
2016-12-28
2018-06-28
Nvidia Corporation
Unconstrained appearance-based gaze estimation
US20180206771A1
( en )
2009-03-30
2018-07-26
Tobii Ab
Eye closure detection using structured illumination
US20180224933A1
( en )
2014-01-24
2018-08-09
Tobii Ab
Gaze driven interaction for a vehicle
2020
2020-03-31
EP
EP20781789.1A
patent/EP3948402B1/en
active
Active
2020-03-31
WO
PCT/IB2020/053035
patent/WO2020201999A2/en
not_active
Ceased
2021
2021-04-23
US
US17/239,385
patent/US11385712B2/en
active
Active
Patent Citations (26)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US6669651B1
( en )
*
2000-02-24
2003-12-30
Matsushita Electric Works, Ltd.
Non-invasive brain function examination
US7572008B2
( en )
2002-11-21
2009-08-11
Tobii Technology Ab
Method and installation for detecting and following an eye and the gaze direction thereof
US8066375B2
( en )
2005-10-10
2011-11-29
Tobii Technology Ab
Eye tracker having an extended span of operating distances
US8120577B2
( en )
2005-10-28
2012-02-21
Tobii Technology Ab
Eye tracker with visual feedback
US20180206771A1
( en )
2009-03-30
2018-07-26
Tobii Ab
Eye closure detection using structured illumination
US8342687B2
( en )
2009-10-08
2013-01-01
Tobii Technology Ab
Eye-tracking using a GPU
US8885882B1
( en )
2011-07-14
2014-11-11
The Research Foundation For The State University Of New York
Real time eye tracking for human computer interaction
US8824779B1
( en )
2011-12-20
2014-09-02
Christopher Charles Smyth
Apparatus and method for determining eye gaze from stereo-optic views
US20170011492A1
( en )
2013-03-04
2017-01-12
Tobii Ab
Gaze and saccade based graphical manipulation
US20140327771A1
( en )
*
2013-05-01
2014-11-06
Nvidia Corporation
System, method, and computer program product for displaying a scene as a light field
US20180224933A1
( en )
2014-01-24
2018-08-09
Tobii Ab
Gaze driven interaction for a vehicle
US20160335475A1
( en )
2014-02-04
2016-11-17
Fraunhofer-Gesellschaft Zur Foerderung Der Angewandten Forschung E.V.
3d image analyzer for determining the gaze direction
US20150223684A1
( en )
2014-02-13
2015-08-13
Bryson Hinton
System and method for eye tracking
US20180143684A1
( en )
2014-02-21
2018-05-24
Tobii Ab
Apparatus and method for robust eye/gaze tracking
US20160085302A1
( en )
*
2014-05-09
2016-03-24
Eyefluence, Inc.
Systems and methods for biomechanically-based eye signals for interacting with real and virtual objects
US20160062459A1
( en )
*
2014-05-09
2016-03-03
Eyefluence, Inc.
Systems and methods for biomechanically-based eye signals for interacting with real and virtual objects
US20160274660A1
( en )
*
2014-05-09
2016-09-22
Eyefluence, Inc.
Systems and methods for biomechanically-based eye signals for interacting with real and virtual objects
US20180011533A9
( en )
2014-05-09
2018-01-11
Google Inc.
Systems and methods for biomechanically-based eye signals for interacting with real and virtual objects
US20150338915A1
( en )
*
2014-05-09
2015-11-26
Eyefluence, Inc.
Systems and methods for biomechanically-based eye signals for interacting with real and virtual objects
US20180008141A1
( en )
2014-07-08
2018-01-11
Krueger Wesley W O
Systems and methods for using virtual reality, augmented reality, and/or a synthetic 3-dimensional information for the measurement of human ocular performance
US20160148050A1
( en )
2014-11-24
2016-05-26
Hyundai Motor Company
Apparatus and method for detecting eyes
US20160166146A1
( en )
*
2014-12-11
2016-06-16
Icspi Corp.
Eye-Tracking System and Method Therefor
US20170353717A1
( en )
*
2014-12-29
2017-12-07
Beijing Zhigu Tech Co., Ltd.
Light field display control methods and apparatus, and light field display devices
US20180136722A1
( en )
2015-09-04
2018-05-17
Sony Interactive Entertainment Inc.
Apparatus and method for dynamic graphics rendering based on saccade detection
WO2018000020A1
( en )
2016-06-29
2018-01-04
Seeing Machines Limited
Systems and methods for performing eye gaze tracking
US20180181809A1
( en )
2016-12-28
2018-06-28
Nvidia Corporation
Unconstrained appearance-based gaze estimation
Non-Patent Citations (8)
* Cited by examiner, â Cited by third party
Title
Fuhl W. et al. " PupilNet v2.0: Convolutional Neural Network for CPU based real time Robust Pupil Detection ", arXiv: 1711.00112, 2017.
Kar A. & Corcoran P., " A Review and Analysis of Eye-Gaze Estimation Systems, Algorithms and Performance Evaluation Methods in Consumer Platforms ", IEEE Access, vol. 5, pp. 16495-16519, 2017.
Kim M., Wang O., Ng N., Convolutional Neural Network Architectures for Gaze Estimation on Mobile Device, Course Project Report, CS231n: Convolutional Neural Networks for Visual Recognition, 2017.
Koko & Florez " Cost Effective IR-Free Eye Tracking on Mobile Device ", Senior Research Proposal, Department of Mathematics and Computer Science, Stetson University, Florida, 2016.
Krafka K. et al. " Eye Tracking for Everyone ", 2016 IEEE Conference on Computer Vision and Pattern Recognition, IEEE, pp. 2176-2184, 2016.
Naqvi R. A. et al., " Deep Learning-Based Gaze Detection System for Automobile Drivers Using a NIR Camera Sensor ", Sensor, doi: 10.3390/s18020456, pp. 1-34, 2018.
Santini T. et al. " PuRe: Robust pupil detection for real-time pervasive eye tracking ", arXiv:1712.08900v1, 2017.
Swirski et al. " Robust real-time pupil tracking in highly off-axis images ", Proceedings of the Symposium on Eye Tracking Research and Applications, pp. 173-176, 2012.
Cited By (2)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US20230025540A1
( en )
*
2020-06-11
2023-01-26
Guangzhou Automobile Group Co., Ltd.
Method for visually tracking gaze point of human eye, vehicle early warning method and device
US12304471B2
( en )
*
2020-06-11
2025-05-20
Guangzhou Automobile Group Co., Ltd.
Method for visually tracking gaze point of human eye, vehicle early warning method and device
Also Published As
Publication number
Publication date
WO2020201999A3
( en )
2021-04-22
EP3948402B1
( en )
2023-12-06
EP3948402A2
( en )
2022-02-09
WO2020201999A2
( en )
2020-10-08
US20210271319A1
( en )
2021-09-02
EP3948402A4
( en )
2022-12-07
EP3948402C0
( en )
2023-12-06
Similar Documents
Publication
Publication Date
Title
EP3948402B1
( en )
2023-12-06
Pupil tracking system and method, and digital display device and digital image rendering system and method using same
US10699373B1
( en )
2020-06-30
Light field display, adjusted pixel rendering method therefor, and vision correction system and method using same
US10884495B2
( en )
2021-01-05
Light field display, adjusted pixel rendering method therefor, and vision correction system and method using same
US20230161409A1
( en )
2023-05-25
Reducing head mounted display power consumption and heat generation through predictive rendering of content
US20220394234A1
( en )
2022-12-08
System and method for implementing a viewer-specific image perception adjustment within a defined view zone, and vision correction system and method using same
US11899205B2
( en )
2024-02-13
Digital display device comprising a complementary light field display or display portion, and vision correction system and method using same
US11644897B2
( en )
2023-05-09
User tracking system using user feature location and method, and digital display device and digital image rendering system and method using same
US11308685B2
( en )
2022-04-19
Rendering computer-generated reality text
US20230334909A1
( en )
2023-10-19
Multi-wavelength biometric imaging system
CA3134669A1
( en )
2020-10-08
Pupil tracking system and method, and digital display device and digital image rendering system and method using same
US12277623B2
( en )
2025-04-15
Attention-driven rendering for computer-generated objects
CA3045261A1
( en )
2020-12-05
Digital display device comprising a complementary light field display portion, and vision correction system and method using same
WO2023233207A1
( en )
2023-12-07
User tracking system and method, and digital display device and digital image rendering system and method using same
US20250097402A1
( en )
2025-03-20
Interpolation of reprojected content
US20240402799A1
( en )
2024-12-05
Eye tracking loss mitigations
CN121564294A
( en )
2026-02-24
Virtual reality picture rendering method and device
CA3040952A1
( en )
2020-10-23
Selective light field display, pixel rendering method therefor, and vision correction system and method using same
WO2025072037A1
( en )
2025-04-03
Camera focusing for video passthrough systems
Legal Events
Date
Code
Title
Description
2021-04-23
FEPP
Fee payment procedure
Free format text : ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY
2021-04-29
FEPP
Fee payment procedure
Free format text : ENTITY STATUS SET TO SMALL (ORIGINAL EVENT CODE: SMAL); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY
2021-07-15
STPP
Information on status: patent application and granting procedure in general
Free format text : NON FINAL ACTION MAILED
2021-08-13
AS
Assignment
Owner name : EVOLUTION OPTIKS LIMITED, BARBADOS
Free format text : ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:LUSSIER, GUILLAUME;EL-MONAJJED, KHALED;REEL/FRAME:057177/0061
Effective date : 20200817
2021-10-26
STPP
Information on status: patent application and granting procedure in general
Free format text : RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER
<time itemprop="date" datetime="2021-1