Google Patentsopen access

Methods and systems for creating virtual and augmented reality — Magic Leap, Inc. (US10203762B2)

ABSTRACT

Abstract

Configurations are disclosed for presenting virtual reality and augmented reality experiences to users. The system may comprise an image capturing device to capture one or more images, the one or more images corresponding to a field of the view of a user of a head-mounted augmented reality device, and a processor communicatively coupled to the image capturing device to extract a set of map points from the set of images, to identify a set of sparse points and a set of dense points from the extracted set of map points, and to perform a normalization on the set of map points.

Description

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application claims priority to U.S. Provisional Patent App. Ser. No. 62/012,273 filed on Jun. 14, 2014 entitled “METHODS AND SYSTEMS FOR CREATING VIRTUAL AND AUGMENTED REALITY,” . This application is a continuation-in-part of U.S. patent application Ser. No. 14/331,218 filed on Jul. 14, 2014 entitled “PLANAR WAVEGUIDE APPARATUS WITH DIFFRACTION LENSING ELEMENT(S) AND SYSTEM EMPLOYING SAME,”. This application is cross-related to U.S. patent application Ser. No. 14/555,585 filed on Nov. 27, 2014 entitled “VIRTUAL AND AR SYSTEMS AND METHODS,” U.S. patent application Ser. No. 14/690,401 filed on Apr. 18, 2015 entitled “SYSTEMS AND METHOD FOR AUGMENTED REALITY” and to U.S. patent application Ser. No. 14/205,126 filed on Mar. 11, 2014 entitled “SYSTEM AND METHOD FOR AUGMENTED AND VIRTUAL REALITY,”. The content of the aforementioned patent applications are hereby expressly incorporated by reference in their entirety for all purposes.

BACKGROUND

Modern computing and display technologies have facilitated the development of systems for so called “virtual reality” or “augmented reality” experiences, wherein digitally reproduced images or portions thereof are presented to a user in a manner wherein they seem to be, or may be perceived as, real. A virtual reality, or “VR”, scenario typically involves presentation of digital or virtual image information without transparency to other actual real-world visual input; an augmented reality, or “AR”, scenario typically involves presentation of digital or virtual image information as an augmentation to visualization of the actual world around the user. For example, an augmented reality scene may allow a user of AR technology may see one or more virtual objects super-imposed on or amidst real world objects (e.g., a real-world park-like setting featuring people, trees, buildings in the background, etc.).

The human visual perception system is very complex, and producing a VR or AR technology that facilitates a comfortable, natural-feeling, rich presentation of virtual image elements amongst other virtual or real-world imagery elements is challenging. Traditional stereoscopic wearable glasses generally feature two displays that are configured to display images with slightly different element presentation such that a three-dimensional perspective is perceived by the human visual system. Such configurations have been found to be uncomfortable for many users due to a mismatch between vergence and accommodation which may be overcome to perceive the images in three dimensions. Indeed, some users are not able to tolerate stereoscopic configurations.

Although a few optical configurations (e.g., head-mounted glasses) are available (e.g., GoogleGlass®, Occulus Rift®, etc.), none of these configurations is optimally suited for presenting a rich, binocular, three-dimensional augmented reality experience in a manner that will be comfortable and maximally useful to the user, in part because prior systems fail to address some of the fundamental aspects of the human perception system, including the photoreceptors of the retina and their interoperation with the brain to produce the perception of visualization to the user.

The human eye is an exceedingly complex organ, and typically comprises a cornea, an iris, a lens, macula, retina, and optic nerve pathways to the brain. The macula is the center of the retina, which is utilized to see moderate detail. At the center of the macula is a portion of the retina that is referred to as the “fovea”, which is utilized for seeing the finest details of a scene, and which contains more photoreceptors (approximately 120 cones per visual degree) than any other portion of the retina.

The human visual system is not a passive sensor type of system; it actively scans the environment. In a manner somewhat akin to use of a flatbed scanner to capture an image, or use of a finger to read Braille from a paper, the photoreceptors of the eye fire in response to changes in stimulation, rather than constantly responding to a constant state of stimulation. Thus, motion is required to present photoreceptor information to the brain.

Indeed, experiments with substances such as cobra venom, which has been utilized to paralyze the muscles of the eye, have shown that a human subject will experience blindness if positioned with eyes open, viewing a static scene with venom-induced paralysis of the eyes. In other words, without changes in stimulation, the photoreceptors do not provide input to the brain and blindness is experienced. It is believed that this is at least one reason that the eyes of normal humans have been observed to move back and forth, or dither, in side-to-side motion, also known as “microsaccades”.

As noted above, the fovea of the retina contains the greatest density of photoreceptors. While it is typically perceived that humans have high-resolution visualization capabilities throughout a field of view, in actuality humans only a small high-resolution center that is mechanically swept around almost constantly, along with a persistent memory of the high-resolution information recently captured with the fovea. In a somewhat similar manner, the focal distance control mechanism of the eye (e.g., ciliary muscles operatively coupled to the crystalline lens in a manner wherein ciliary relaxation causes taut ciliary connective fibers to flatten out the lens for more distant focal lengths; ciliary contraction causes loose ciliary connective fibers, which allow the lens to assume a more rounded geometry for more close-in focal lengths) dithers back and forth by approximately ¼ to ½ diopter to cyclically induce a small amount of “dioptric blur” on both the close side and far side of the targeted focal length. This is utilized by the accommodation control circuits of the brain as cyclical negative feedback that helps to constantly correct course and keep the retinal image of a fixated object approximately in focus.

The visualization center of the brain also gains valuable perception information from the motion of both eyes and components thereof relative to each other. Vergence movements (e.g., rolling movements of the pupils toward or away from each other to converge the lines of sight of the eyes to fixate upon an object) of the two eyes relative to each other are closely associated with focusing (or “accommodation”) of the lenses of the eyes. Under normal conditions, changing the focus of the lenses of the eyes, or accommodating the eyes, to focus upon an object at a different distance will automatically cause a matching change in vergence to the same distance, under a relationship known as the “accommodation-vergence reflex.” Likewise, a change in vergence will trigger a matching change in accommodation, under normal conditions. Working against this reflex (as is the case with most conventional stereoscopic AR or VR configurations) is known to produce eye fatigue, headaches, or other forms of discomfort in users.

Movement of the head, which houses the eyes, also has a key impact upon visualization of objects. Humans tend to move their heads to visualize the world around them, and are often are in a fairly constant state of repositioning and reorienting the head relative to an object of interest. Further, most people prefer to move their heads when their eye gaze needs to move more than about 20 degrees off center to focus on a particular object (e.g., people do not typically like to look at things “from the corner of the eye”). Humans also typically scan or move their heads in relation to sounds—to improve audio signal capture and utilize the geometry of the ears relative to the head. The human visual system gains powerful depth cues from what is called “head motion parallax”, which is related to the relative motion of objects at different distances as a function of head motion and eye vergence distance. In other words, if a person moves his head from side to side and maintains fixation on an object, items farther out from that object will move in the same direction as the head, and items in front of that object will move opposite the head motion. These may be very salient cues for where objects are spatially located in the environment relative to the person. Head motion also is utilized to look around objects, of course.

Further, head and eye motion are coordinated with the “vestibulo-ocular reflex”, which stabilizes image information relative to the retina during head rotations, thus keeping the object image information approximately centered on the retina. In response to a head rotation, the eyes are reflexively and proportionately rotated in the opposite direction to maintain stable fixation on an object. As a result of this compensatory relationship, many humans can read a book while shaking their head back and forth. Interestingly, if the book is panned back and forth at the same speed with the head approximately stationary, the same generally is not true—the person is not likely to be able to read the moving book. The vestibulo-ocular reflex is one of head and eye motion coordination, and is generally not developed for hand motion. This paradigm may be important for AR systems, because head motions of the user may be associated relatively directly with eye motions, and an ideal system preferably will be ready to work with this relationship.

Indeed, given these various relationships, when placing digital content (e.g., 3-D content such as a virtual chandelier object presented to augment a real-world view of a room; or 2-D content such as a planar/flat virtual oil painting object presented to augment a real-world view of a room), design choices may be made to control behavior of the objects. For example, a 2-D oil painting object may be head-centric, in which case the object moves around along with the user's head (e.g., as in a GoogleGlass® approach). In another example, an object may be world-centric, in which case it may be presented as though it is part of the real world coordinate system, such that the user may move his head or eyes without moving the position of the object relative to the real world.

Thus when placing virtual content into the augmented reality world presented with an AR system, choices are made as to whether the object should be presented as world centric, body-centric, head-centric or eye centric. In head-centric approaches, the virtual object stays in position in the real world so that the user may move his body, head, eyes around it without changing its position relative to the real world objects surrounding it, such as a real world wall. In body-centric approaches, a virtual element may be fixed relative to the user's torso, so that the user can move his head or eyes without moving the object, but that is slaved to torso movements, In head centric approaches, the displayed object (and/or display itself) may be moved along with head movements, as described above in reference to GoogleGlass®). In eye-centric approaches, as in a “foveated display” configuration, as is described below, content is slewed around as a function of the eye position.

With world-centric configurations, it may be desirable to have inputs such as accurate head pose measurement, accurate representation and/or measurement of real world objects and geometries around the user, low-latency dynamic rendering in the augmented reality display as a function of head pose, and a generally low-latency display.

The U.S. patent applications listed above present systems and techniques to work with the visual configuration of a typical human to address various challenges in virtual reality and augmented reality applications. The design of these virtual reality and/or AR systems presents numerous challenges, including the speed of the system in delivering virtual content, quality of virtual content, eye relief of the user, size and portability of the system, and other system and optical challenges.

The systems and techniques described herein are configured to work with the visual configuration of the typical human to address these challenges.

SUMMARY

Embodiments of the present invention are directed to devices, systems and methods for facilitating virtual reality and/or augmented reality interaction for one or more users. In one aspect, a system for displaying virtual content is disclosed.

In one aspect, an augmented reality system comprises an image capturing device to capture one or more images, the one or more images corresponding to a field of the view of a user of a head-mounted augmented reality device, and a processor communicatively coupled to the image capturing device to extract a set of map points from the set of images, to identify a set of sparse points and a set of dense points from the extracted set of map points, and to perform a normalization on the set of map points.

Additional and other objects, features, and advantages of the invention are described in the detail description, figures and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings illustrate the design and utility of various embodiments of the present invention. It should be noted that the figures are not drawn to scale and that elements of similar structures or functions are represented by like reference numerals throughout the figures. In order to better appreciate how to obtain the above-recited and other advantages and objects of various embodiments of the invention, a more detailed description of the present inventions briefly described above will be rendered by reference to specific embodiments thereof, which are illustrated in the accompanying drawings. Understanding that these drawings depict only typical embodiments of the invention and are not therefore to be considered limiting of its scope, the invention will be described and explained with additional specificity and detail through the use of the accompanying drawings in which:

FIG. 1 illustrates a system architecture of an augmented reality (AR) system interacting with one or more servers, according one illustrated embodiment.

FIG. 2 illustrates a detailed view of a cell phone used as an AR device interacting with one or more servers, according to one illustrated embodiment.

FIG. 3 illustrates a plan view of an example AR device mounted on a user's head, according to one illustrated embodiment.

FIGS. 4A-4D illustrate one or more embodiments of various internal processing components of the wearable AR device.

FIGS. 5A-5H illustrate embodiments of transmitting focused light to a user through a transmissive beamsplitter substrate.

FIGS. 6A and 6B illustrate embodiments of coupling a lens element with the transmissive beamsplitter substrate of FIGS. 5A-5H .

FIGS. 7A and 7B illustrate embodiments of using one or more waveguides to transmit light to a user.

FIGS. 8A-8Q illustrate embodiments of a diffractive optical element (DOE).

FIGS. 9A and 9B illustrate a wavefront produced from a light projector, according to one illustrated embodiment.

FIG. 10 illustrates an embodiment of a stacked configuration of multiple transmissive beamsplitter substrate coupled with optical elements, according to one illustrated embodiment.

FIGS. 11A-11C illustrate a set of beamlets projected into a user's pupil, according to the illustrated embodiments.

FIGS. 12A and 12B illustrate configurations of an array of microprojectors, according to the illustrated embodiments.

FIGS. 13A-13M illustrate embodiments of coupling microprojectors with optical elements, according to the illustrated embodiments.

FIGS. 14A-14F illustrate embodiments of spatial light modulators coupled with optical elements, according to the illustrated embodiments.

<figref idrefs

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application claims priority to U.S. Provisional Patent App. Ser. No. 62/012,273 filed on Jun. 14, 2014 entitled “METHODS AND SYSTEMS FOR CREATING VIRTUAL AND AUGMENTED REALITY,” . This application is a continuation-in-part of U.S. patent application Ser. No. 14/331,218 filed on Jul. 14, 2014 entitled “PLANAR WAVEGUIDE APPARATUS WITH DIFFRACTION LENSING ELEMENT(S) AND SYSTEM EMPLOYING SAME,”. This application is cross-related to U.S. patent application Ser. No. 14/555,585 filed on Nov. 27, 2014 entitled “VIRTUAL AND AR SYSTEMS AND METHODS,” U.S. patent application Ser. No. 14/690,401 filed on Apr. 18, 2015 entitled “SYSTEMS AND METHOD FOR AUGMENTED REALITY” and to U.S. patent application Ser. No. 14/205,126 filed on Mar. 11, 2014 entitled “SYSTEM AND METHOD FOR AUGMENTED AND VIRTUAL REALITY,”. The content of the aforementioned patent applications are hereby expressly incorporated by reference in their entirety for all purposes.

BACKGROUND

Modern computing and display technologies have facilitated the development of systems for so called “virtual reality” or “augmented reality” experiences, wherein digitally reproduced images or portions thereof are presented to a user in a manner wherein they seem to be, or may be perceived as, real. A virtual reality, or “VR”, scenario typically involves presentation of digital or virtual image information without transparency to other actual real-world visual input; an augmented reality, or “AR”, scenario typically involves presentation of digital or virtual image information as an augmentation to visualization of the actual world around the user. For example, an augmented reality scene may allow a user of AR technology may see one or more virtual objects super-imposed on or amidst real world objects (e.g., a real-world park-like setting featuring people, trees, buildings in the background, etc.).

The human visual perception system is very complex, and producing a VR or AR technology that facilitates a comfortable, natural-feeling, rich presentation of virtual image elements amongst other virtual or real-world imagery elements is challenging. Traditional stereoscopic wearable glasses generally feature two displays that are configured to display images with slightly different element presentation such that a three-dimensional perspective is perceived by the human visual system. Such configurations have been found to be uncomfortable for many users due to a mismatch between vergence and accommodation which may be overcome to perceive the images in three dimensions. Indeed, some users are not able to tolerate stereoscopic configurations.

Although a few optical configurations (e.g., head-mounted glasses) are available (e.g., GoogleGlass®, Occulus Rift®, etc.), none of these configurations is optimally suited for presenting a rich, binocular, three-dimensional augmented reality experience in a manner that will be comfortable and maximally useful to the user, in part because prior systems fail to address some of the fundamental aspects of the human perception system, including the photoreceptors of the retina and their interoperation with the brain to produce the perception of visualization to the user.

The human eye is an exceedingly complex organ, and typically comprises a cornea, an iris, a lens, macula, retina, and optic nerve pathways to the brain. The macula is the center of the retina, which is utilized to see moderate detail. At the center of the macula is a portion of the retina that is referred to as the “fovea”, which is utilized for seeing the finest details of a scene, and which contains more photoreceptors (approximately 120 cones per visual degree) than any other portion of the retina.

The human visual system is not a passive sensor type of system; it actively scans the environment. In a manner somewhat akin to use of a flatbed scanner to capture an image, or use of a finger to read Braille from a paper, the photoreceptors of the eye fire in response to changes in stimulation, rather than constantly responding to a constant state of stimulation. Thus, motion is required to present photoreceptor information to the brain.

Indeed, experiments with substances such as cobra venom, which has been utilized to paralyze the muscles of the eye, have shown that a human subject will experience blindness if positioned with eyes open, viewing a static scene with venom-induced paralysis of the eyes. In other words, without changes in stimulation, the photoreceptors do not provide input to the brain and blindness is experienced. It is believed that this is at least one reason that the eyes of normal humans have been observed to move back and forth, or dither, in side-to-side motion, also known as “microsaccades”.

As noted above, the fovea of the retina contains the greatest density of photoreceptors. While it is typically perceived that humans have high-resolution visualization capabilities throughout a field of view, in actuality humans only a small high-resolution center that is mechanically swept around almost constantly, along with a persistent memory of the high-resolution information recently captured with the fovea. In a somewhat similar manner, the focal distance control mechanism of the eye (e.g., ciliary muscles operatively coupled to the crystalline lens in a manner wherein ciliary relaxation causes taut ciliary connective fibers to flatten out the lens for more distant focal lengths; ciliary contraction causes loose ciliary connective fibers, which allow the lens to assume a more rounded geometry for more close-in focal lengths) dithers back and forth by approximately ¼ to ½ diopter to cyclically induce a small amount of “dioptric blur” on both the close side and far side of the targeted focal length. This is utilized by the accommodation control circuits of the brain as cyclical negative feedback that helps to constantly correct course and keep the retinal image of a fixated object approximately in focus.

The visualization center of the brain also gains valuable perception information from the motion of both eyes and components thereof relative to each other. Vergence movements (e.g., rolling movements of the pupils toward or away from each other to converge the lines of sight of the eyes to fixate upon an object) of the two eyes relative to each other are closely associated with focusing (or “accommodation”) of the lenses of the eyes. Under normal conditions, changing the focus of the lenses of the eyes, or accommodating the eyes, to focus upon an object at a different distance will automatically cause a matching change in vergence to the same distance, under a relationship known as the “accommodation-vergence reflex.” Likewise, a change in vergence will trigger a matching change in accommodation, under normal conditions. Working against this reflex (as is the case with most conventional stereoscopic AR or VR configurations) is known to produce eye fatigue, headaches, or other forms of discomfort in users.

Movement of the head, which houses the eyes, also has a key impact upon visualization of objects. Humans tend to move their heads to visualize the world around them, and are often are in a fairly constant state of repositioning and reorienting the head relative to an object of interest. Further, most people prefer to move their heads when their eye gaze needs to move more than about 20 degrees off center to focus on a particular object (e.g., people do not typically like to look at things “from the corner of the eye”). Humans also typically scan or move their heads in relation to sounds—to improve audio signal capture and utilize the geometry of the ears relative to the head. The human visual system gains powerful depth cues from what is called “head motion parallax”, which is related to the relative motion of objects at different distances as a function of head motion and eye vergence distance. In other words, if a person moves his head from side to side and maintains fixation on an object, items farther out from that object will move in the same direction as the head, and items in front of that object will move opposite the head motion. These may be very salient cues for where objects are spatially located in the environment relative to the person. Head motion also is utilized to look around objects, of course.

Further, head and eye motion are coordinated with the “vestibulo-ocular reflex”, which stabilizes image information relative to the retina during head rotations, thus keeping the object image information approximately centered on the retina. In response to a head rotation, the eyes are reflexively and proportionately rotated in the opposite direction to maintain stable fixation on an object. As a result of this compensatory relationship, many humans can read a book while shaking their head back and forth. Interestingly, if the book is panned back and forth at the same speed with the head approximately stationary, the same generally is not true—the person is not likely to be able to read the moving book. The vestibulo-ocular reflex is one of head and eye motion coordination, and is generally not developed for hand motion. This paradigm may be important for AR systems, because head motions of the user may be associated relatively directly with eye motions, and an ideal system preferably will be ready to work with this relationship.

Indeed, given these various relationships, when placing digital content (e.g., 3-D content such as a virtual chandelier object presented to augment a real-world view of a room; or 2-D content such as a planar/flat virtual oil painting object presented to augment a real-world view of a room), design choices may be made to control behavior of the objects. For example, a 2-D oil painting object may be head-centric, in which case the object moves around along with the user&#39;s head (e.g., as in a GoogleGlass® approach). In another example, an object may be world-centric, in which case it may be presented as though it is part of the real world coordinate system, such that the user may move his head or eyes without moving the position of the object relative to the real world.

Thus when placing virtual content into the augmented reality world presented with an AR system, choices are made as to whether the object should be presented as world centric, body-centric, head-centric or eye centric. In head-centric approaches, the virtual object stays in position in the real world so that the user may move his body, head, eyes around it without changing its position relative to the real world objects surrounding it, such as a real world wall. In body-centric approaches, a virtual element may be fixed relative to the user&#39;s torso, so that the user can move his head or eyes without moving the object, but that is slaved to torso movements, In head centric approaches, the displayed object (and/or display itself) may be moved along with head movements, as described above in reference to GoogleGlass®). In eye-centric approaches, as in a “foveated display” configuration, as is described below, content is slewed around as a function of the eye position.

With world-centric configurations, it may be desirable to have inputs such as accurate head pose measurement, accurate representation and/or measurement of real world objects and geometries around the user, low-latency dynamic rendering in the augmented reality display as a function of head pose, and a generally low-latency display.

The U.S. patent applications listed above present systems and techniques to work with the visual configuration of a typical human to address various challenges in virtual reality and augmented reality applications. The design of these virtual reality and/or AR systems presents numerous challenges, including the speed of the system in delivering virtual content, quality of virtual content, eye relief of the user, size and portability of the system, and other system and optical challenges.

The systems and techniques described herein are configured to work with the visual configuration of the typical human to address these challenges.

SUMMARY

Embodiments of the present invention are directed to devices, systems and methods for facilitating virtual reality and/or augmented reality interaction for one or more users. In one aspect, a system for displaying virtual content is disclosed.

In one aspect, an augmented reality system comprises an image capturing device to capture one or more images, the one or more images corresponding to a field of the view of a user of a head-mounted augmented reality device, and a processor communicatively coupled to the image capturing device to extract a set of map points from the set of images, to identify a set of sparse points and a set of dense points from the extracted set of map points, and to perform a normalization on the set of map points.

Additional and other objects, features, and advantages of the invention are described in the detail description, figures and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings illustrate the design and utility of various embodiments of the present invention. It should be noted that the figures are not drawn to scale and that elements of similar structures or functions are represented by like reference numerals throughout the figures. In order to better appreciate how to obtain the above-recited and other advantages and objects of various embodiments of the invention, a more detailed description of the present inventions briefly described above will be rendered by reference to specific embodiments thereof, which are illustrated in the accompanying drawings. Understanding that these drawings depict only typical embodiments of the invention and are not therefore to be considered limiting of its scope, the invention will be described and explained with additional specificity and detail through the use of the accompanying drawings in which:

FIG. 1 illustrates a system architecture of an augmented reality (AR) system interacting with one or more servers, according one illustrated embodiment.

FIG. 2 illustrates a detailed view of a cell phone used as an AR device interacting with one or more servers, according to one illustrated embodiment.

FIG. 3 illustrates a plan view of an example AR device mounted on a user&#39;s head, according to one illustrated embodiment.

FIGS. 4A-4D illustrate one or more embodiments of various internal processing components of the wearable AR device.

FIGS. 5A-5H illustrate embodiments of transmitting focused light to a user through a transmissive beamsplitter substrate.

FIGS. 6A and 6B illustrate embodiments of coupling a lens element with the transmissive beamsplitter substrate of FIGS. 5A-5H .

FIGS. 7A and 7B illustrate embodiments of using one or more waveguides to transmit light to a user.

FIGS. 8A-8Q illustrate embodiments of a diffractive optical element (DOE).

FIGS. 9A and 9B illustrate a wavefront produced from a light projector, according to one illustrated embodiment.

FIG. 10 illustrates an embodiment of a stacked configuration of multiple transmissive beamsplitter substrate coupled with optical elements, according to one illustrated embodiment.

FIGS. 11A-11C illustrate a set of beamlets projected into a user&#39;s pupil, according to the illustrated embodiments.

FIGS. 12A and 12B illustrate configurations of an array of microprojectors, according to the illustrated embodiments.

FIGS. 13A-13M illustrate embodiments of coupling microprojectors with optical elements, according to the illustrated embodiments.

FIGS. 14A-14F illustrate embodiments of spatial light modulators coupled with optical elements, according to the illustrated embodiments.

FIGS. 15A-15C illustrate the use of a wedge type waveguides along with a plurality of light sources, according to the illustrated embodiments.

FIGS. 16A-16O illustrate embodiments of coupling optical elements to optical fibers, according to the illustrated embodiments.

FIG. 17 illustrates a notch filter, according to one illustrated embodiment.

FIG. 18 illustrates a spiral pattern of a fiber scanning display, according to one illustrated embodiment.

FIGS. 19A-19N illustrate occlusion effects in presenting a darkfield to a user, according to the illustrated embodiments.

FIGS. 20A-20O illustrate embodiments of various waveguide assemblies, according to the illustrated embodiments.

FIGS. 21A-21N illustrate various configurations of DOEs coupled to other optical elements, according to the illustrated embodiments.

FIGS. 22A-22Y illustrate various configurations of freeform optics, according to the illustrated embodiments.

FIG. 23 illustrates a top view of components of a simplified individual AR device.

FIG. 24 illustrates an example embodiment of the optics of the individual AR system.

FIG. 25 illustrates a system architecture of the individual AR system, according to one embodiment.

FIG. 26 illustrates a room based sensor system, according to one embodiment.

FIG. 27 illustrates a communication architecture of the augmented reality system and the interaction of the augmented reality systems of many users with the cloud.

FIG. 28 illustrates a simplified view of the passable world model, according to one embodiment.

FIG. 29 illustrates an example method of rendering using the passable world model, according to one embodiment.

FIG. 30 illustrates a high level flow diagram for a process of recognizing an object, according to one embodiment.

FIG. 31 illustrates a ring buffer approach employed by object recognizers to recognize objects in the passable world, according to one embodiment.

FIG. 32 illustrates an example topological map, according to one embodiment.

FIG. 33 illustrates a high level flow diagram for a process of localization using the topological map, according to one embodiment.

FIG. 34 illustrates a geometric map as a connection between various keyframes, according to one embodiment.

FIG. 35 illustrates an example embodiment of the topological map layered on top of the geometric map, according to one embodiment.

FIG. 36 illustrates a high level flow diagram for a process of performing a wave propagation bundle adjust, according to one embodiment.

FIG. 37 illustrates map points and render lines from the map points to the keyframes as seen through a virtual keyframe, according to one embodiment.

FIG. 38 illustrates a high level flow diagram for a process of finding map points based on render rather than search, according to one embodiment.

FIG. 39 illustrates a high level flow diagram for a process of rendering a virtual object based on a light map, according to one embodiment.

FIG. 40 illustrates a high level flow diagram for a process of creating a light map, according to one embodiment.

FIG. 41 depicts a user-centric light map, according to one embodiment

FIG. 42 depicts an object-centric light map, according to one embodiment.

FIG. 43 illustrates a high level flow diagram for a process of transforming a light map, according to one embodiment.

FIG. 44 illustrates a library of autonomous navigation definitions or objects, according to one embodiment.

FIG. 45 illustrates an interaction of various autonomous navigation objects, according to one embodiment.

FIG. 46 illustrates a stack of autonomous navigation definitions or objects, according to one embodiment.

FIGS. 47A-47B illustrate using the autonomous navigation definitions to identify emotional states, according to one embodiment.

FIG. 48 illustrates a correlation threshold graph to be used to define an autonomous navigation definition or object, according to one embodiment.

FIG. 49 illustrates a system view of the passable world model, according to one embodiment.

FIG. 50 illustrates an example method of displaying a virtual scene, according to one embodiment.

FIG. 51 illustrates a plan view of various modules of the AR system, according to one illustrated embodiment.

FIG. 52 illustrates an example of objects viewed by a user when the AR device is operated in an augmented reality mode, according to one illustrated embodiment.

FIG. 53 illustrates an example of objects viewed by a user when the AR device is operated in a virtual mode, according to one illustrated embodiment.

FIG. 54 illustrates an example of objects viewed by a user when the AR device is operated in a blended virtual interface mode, according to one illustrated embodiment.

FIG. 55 illustrates an embodiment wherein two users located in different geographical locations each interact with the other user and a common virtual world through their respective user devices, according to one embodiment.

FIG. 56 illustrates an embodiment wherein the embodiment of FIG. 55 is expanded to include the use of a haptic device, according to one embodiment.

FIG. 57A-57B illustrates an example of mixed mode interfacing, according to one or more embodiments.

FIG. 58 illustrates an example illustration of a user&#39;s view when interfacing the AR system, according to one embodiment.

FIG. 59 illustrates an example illustration of a user&#39;s view showing a virtual object triggered by a physical object when the user is interfacing the system in an augmented reality mode, according to one embodiment.

FIG. 60 illustrates one embodiment of an augmented and virtual reality integration configuration wherein one user in an augmented reality experience visualizes the presence of another user in a virtual realty experience.

FIG. 61 illustrates one embodiment of a time and/or contingency event based augmented reality experience configuration.

FIG. 62 illustrates one embodiment of a user display configuration suitable for virtual and/or augmented reality experiences.

FIG. 63 illustrates one embodiment of local and cloud-based computing coordination.

FIG. 64 illustrates various aspects of registration configurations, according to one illustrated embodiment.

FIG. 65 illustrates an example scenario of interacting with the AR system, according to one embodiment.

FIG. 66 illustrates another perspective of the example scenario of FIG. 65 , according to another embodiment.

FIG. 67 illustrates yet another perspective view of the example scenario of FIG. 65 , according to another embodiment.

FIG. 68 illustrates a top view of the example scenario according to one embodiment.

FIG. 69 illustrates a game view of the example scenario of FIGS. 65-68 , according to one embodiment.

FIG. 70 illustrates a top view of the example scenario of FIGS. 65-68 , according to one embodiment.

FIG. 71 illustrates an augmented reality scenario including multiple users, according to one embodiment.

FIGS. 72A-72B illustrate using a smartphone or tablet as an AR device, according to one embodiment.

FIG. 73 illustrates an example method of using localization to communicate between users of the AR system, according to one embodiment.

FIGS. 74A-74B illustrates an example office scenario of interacting with the AR system, according to one embodiment.

FIG. 75 illustrates an example scenario of interacting with the AR system in a house, according to one embodiment.

FIG. 76 illustrates another example scenario of interacting with the AR system in a house, according to one embodiment.

FIG. 77 illustrates another example scenario of interacting with the AR system in a house, according to one embodiment.

FIGS. 78A-78B illustrate yet another example scenario of interacting with the AR system in a house, according to one embodiment.

FIGS. 79A-79E illustrate another example scenario of interacting with the AR system in a house, according to one embodiment.

FIGS. 80A-80O illustrate another example scenario of interacting with the AR system in a virtual room, according to one embodiment.

FIG. 81 illustrates another example user interaction scenario, according to one embodiment.

FIG. 82 illustrates another example user interaction scenario, according to one embodiment.

FIGS. 83A-83B illustrates yet another example user interaction scenario, according to one or more embodiments.

FIGS. 84A-84C illustrates the user interacting with the AR system in a virtual space, according to one or more embodiments.

FIGS. 85A-85C illustrates various user interface embodiments.

FIGS. 86A-86C illustrates other embodiments to create a user interface, according to one or more embodiments.

FIGS. 87A-87C illustrates other embodiments to create and move a user interface, according to one or more embodiments.

FIGS. 88A-88C illustrates user interfaces created on the user&#39;s hand, according to one or more embodiments.

FIGS. 89A-89J illustrate an example user shopping experience with the AR system, according to one or more embodiments.

FIG. 90 illustrates an example library experience with the AR system, according to one or more embodiments.

FIGS. 91A-91F illustrate an example healthcare experience with the AR system, according to one or more embodiments.

FIG. 92 illustrates an example labor experience with the AR system, according to one or more embodiments.

FIGS. 93A-93L illustrate an example workspace experience with the AR system, according to one or more embodiments.

FIG. 94 illustrates another example workspace experience with the AR system, according to one or more embodiments.

FIGS. 95A-95E illustrates another AR experience, according to one or more embodiments.

FIGS. 96A-96D illustrates yet another AR experience, according to one or more embodiments.

FIGS. 97A-97H illustrates a gaming experience with the AR system, according to one or more embodiments.

FIGS. 98A-98D illustrate a web shopping experience with the AR system, according to one or more embodiments.

FIG. 99 illustrates a block diagram of various games in a gaming platform, according to one or more embodiments.

FIG. 100 illustrates a variety of user inputs to communicate with the augmented reality system, according to one embodiment.

FIG. 101 illustrates LED lights and diodes tracking a movement of the user&#39;s eyes, according to one embodiment.

FIG. 102 illustrates a Purkinje image, according to one embodiment.

FIG. 103 illustrates a variety of hand gestures that may be used to communicate with the augmented reality system, according to one embodiment.

FIG. 104 illustrates an example totem, according to one embodiment.

FIG. 105A-105C illustrate other example totems, according to one or more embodiments.

FIG. 106A-106C illustrate other totems that may be used to communicate with the augmented reality system.

FIGS. 107A-107D illustrates other example totems, according to one or more embodiments.

FIGS. 108A-1080 illustrate example embodiments of ring and bracelet totems, according to one or more embodiments.

FIGS. 109A-109C illustrate more example totems, according to one or more embodiments.

FIGS. 110A-110B illustrate a charms totem and a keychain totem, according to one or more embodiments.

FIG. 111 illustrates a high level flow diagram for a process of determining user input through a totem, according to one embodiment.

FIG. 112 illustrates a high level flow diagram for a process of producing a sound wavefront, according to one embodiment.

FIG. 113 is a block diagram of components used to produce a sound wavefront, according to one embodiment.

FIG. 114 is an example method of determining sparse and dense points, according to one embodiment.

FIG. 115 is a block diagram of projecting textured light, according to one embodiment.

FIG. 116 is an example block diagram of data processing, according to one embodiment.

FIG. 117 is a schematic of an eye for gaze tracking, according to one embodiment.

FIG. 118 shows another perspective of the eye and one or more cameras for gaze tracking, according to one embodiment.

FIG. 119 shows yet another perspective of the eye and one or more cameras for gaze tracking, according to one embodiment.

FIG. 120 shows yet another perspective of the eye and one or more cameras for gaze tracking, according to one embodiment.

FIG. 121 shows a translational matrix view for gaze tracking, according to one embodiment.

FIG. 122 illustrates an example method of gaze tracking, according to one embodiment.

FIGS. 123A-123D illustrate a series of example user interface flows using avatars, according to one embodiment.

FIGS. 124A-124M illustrate a series of example user interface flows using extrusion, according to one embodiment.

FIGS. 125A-125M illustrate a series of example user interface flows using gauntlet, according to one embodiment.

FIGS. 126A-126L illustrate a series of example user interface flows using grow, according to one embodiment.

FIGS. 127A-127E illustrate a series of example user interface flows using brush, according to one embodiment.

FIGS. 128A-128P illustrate a series of example user interface flows using fingerbrush, according to one embodiment.

FIGS. 129A-129M illustrate a series of example user interface flows using pivot according to one embodiment.

FIGS. 130A-130I illustrate a series of example user interface flows using strings, according to one embodiment.

FIGS. 131A-131I illustrate a series of example user interface flows using spiderweb, according to one embodiment.

FIG. 132 is a plan view of various mechanisms by which a virtual object relates to one or more physical objects.

FIG. 133 is a plan view of various types of AR rendering, according to one or more embodiments.

FIG. 134 illustrates various types of user input in an AR system, according to one or more embodiments.

FIGS. 135A-135J illustrates various embodiments pertaining to using gestures in an AR system, according to one or more embodiments.

FIG. 136 illustrates a plan view of various components for a calibration mechanism of the AR system, according to one or more embodiments.

FIG. 137 illustrates a view of an AR device on a user&#39;s face, the AR device having eye tracking cameras, according to one or more embodiments.

FIG. 138 illustrates an eye identification image of the AR system, according to one or more embodiments.

FIG. 139 illustrates a retinal image taken with an AR system, according to one or more embodiments.

FIG. 140 is a process flow diagram of an example method of generating a virtual user interface, according to one illustrated embodiment.

FIG. 141 is another process flow diagram of an example method of generating a virtual user interface based on a coordinate frame, according to one illustrated embodiment.

FIG. 142 is a process flow diagram of an example method of constructing a customized user interface, according to one illustrated embodiment.

FIG. 143 is a process flow diagram of an example method of retrieving information from the passable world model and interacting with other users of the AR system, according to one illustrated embodiment.

FIG. 144 is a process flow diagram of an example method of retrieving information from a knowledge based in the cloud based on received input, according to one illustrated embodiment.

FIG. 145 is a process flow diagram of an example method of calibrating the AR system, according to one illustrated embodiment.

DETAILED DESCRIPTION

Various embodiments will now be described in detail with reference to the drawings, which are provided as illustrative examples of the invention so as to enable those skilled in the art to practice the invention. Notably, the figures and the examples below are not meant to limit the scope of the present invention. Where certain elements of the present invention may be partially or fully implemented using known components (or methods or processes), only those portions of such known components (or methods or processes) that are necessary for an understanding of the present invention will be described, and the detailed descriptions of other portions of such known components (or methods or processes) will be omitted so as not to obscure the invention. Further, various embodiments encompass present and future known equivalents to the components referred to herein by way of illustration.

In the foregoing specification, the invention has been described with reference to specific embodiments thereof. It will, however, be evident that various modifications and changes may be made thereto without departing from the broader spirit and scope of the invention. For example, the above-described process flows are described with reference to a particular ordering of process actions. However, the ordering of many of the described process actions may be changed without affecting the scope or operation of the invention. The specification and drawings are, accordingly, to be regarded in an illustrative rather than restrictive sense.

Disclosed are methods and systems for generating virtual and/or augmented reality. In order to provide a realistic and enjoyable virtual reality (VR) or augmented reality (AR) experience, virtual content may be strategically delivered to the user&#39;s eyes in a manner that is respectful of the human eye&#39;s physiology and limitations. The following disclosure will provide various embodiments of such optical systems that may be integrated into an AR system. Although most of the disclosures herein will be discussed in the context of AR systems, it should be appreciated that the same technologies may be used for VR systems also, and the following embodiments should not be read as limiting.

The following disclosure will provide details on various types of systems in which AR users may interact with each other through a creation of a map that comprises comprehensive information about the physical objects of the real world in real-time. The map may be advantageously consulted in order to project virtual images in relation to known real objects. The following disclosure will provide various approaches to understanding information about the real world, and using this information to provide a more realistic and enjoyable AR experience. Additionally, this disclosure will provide various user scenarios and applications in which AR systems such as the ones described herein may be realized.

System Overview

In one or more embodiments, the AR system 10 comprises a computing network 5 , comprised of one or more computer servers 11 connected through one or more high bandwidth interfaces 15 . The servers 11 in the computing network may or may not be co-located. The one or more servers 11 each comprise one or more processors for executing program instructions. The servers may also include memory for storing the program instructions and data that is used and/or generated by processes being carried out by the servers 11 under direction of the program instructions.

The computing network 5 communicates data between the servers 11 and between the servers and one or more user devices 12 over one or more data network connections 13 . Examples of such data networks include, without limitation, any and all types of public and private data networks, both mobile and wired, including for example the interconnection of many of such networks commonly referred to as the Internet. No particular media, topology or protocol is intended to be implied by the figure.

User devices are configured for communicating directly with computing network 5 , or any of the servers 11 . Alternatively, user devices 12 communicate with the remote servers 11 , and, optionally, with other user devices locally, through a specially programmed, local gateway 14 for processing data and/or for communicating data between the network 5 and one or more local user devices 12 .

As illustrated, gateway 14 is implemented as a separate hardware component, which includes a processor for executing software instructions and memory for storing software instructions and data. The gateway has its own wired and/or wireless connection to data networks for communicating with the servers 11 comprising computing network 5 . Alternatively, gateway 14 can be integrated with a user device 12 , which is worn or carried by a user. For example, the gateway 14 may be implemented as a downloadable software application installed and running on a processor included in the user device 12 . The gateway 14 provides, in one embodiment, one or more users access to the computing network 5 via the data network 13 .

Servers 11 each include, for example, working memory and storage for storing data and software programs, microprocessors for executing program instructions, graphics processors and other special processors for rendering and generating graphics, images, video, audio and multi-media files. Computing network 5 may also comprise devices for storing data that is accessed, used or created by the servers 11 .

Software programs running on the servers and optionally user devices 12 and gateways 14 , are used to generate digital worlds (also referred to herein as virtual worlds) with which users interact with <figure-callout id="12" label="user devices" filenames="

CLAIMS

Claims ( 22 )

What is claimed is:

1. An augmented reality display system, comprising:

an image capturing device of the augmented reality display system to capture at least one image, wherein

the image capturing device comprises one or more image capturing sensors,

at least a portion of the at least one image is perceived within a field of view of a user, and

the at least one image captures at least one gesture that is created by the user and interacts with virtual content projected by the augmented reality display system to the user; and

a processor coupled directly with no intervening elements or indirectly with one or more intervening elements to the image capturing device to recognize the at least one gesture as at least one recognized gesture, the processor configured to recognize the at least one gesture as the at least one recognized gesture is further configured to:

identify a plurality of candidate gestures and a plurality of computational utilization or expense requirements for gesture recognition of the at least one gesture;

determine whether the at least one image includes one or more identifiable depth points at least by performing a line search for the at least one image with one or more lines or line segments;

determine an order of processing in which a plurality of analysis nodes is executed to perform respective gesture identification processes on the at least one gesture with respect to the plurality of candidate gestures based at least in part upon the plurality of computational resource utilization or expense requirements;

during one or more earlier stages in the order of processing, generate one or more reduced sets of candidate gestures from the plurality of candidate gestures at least by executing one or more first gesture identification processes of the respective gesture identification processes that analyze the at least one image with a first analysis node to remove one or more candidate gestures from the plurality of candidate gestures based at least in part upon a first computational resource utilization or expense requirement of the plurality of computational resource utilization or expense requirements; and

during one or more later stages in the order of processing, determine and recognize the at least one gesture from the one or more reduced sets of candidate gestures as the at least one recognized gesture at least by analyzing the at least one image based at least in part on the one or more reduced sets of candidate gestures with a second analysis node corresponding to a second computational expense criterion and at least by executing one or more second gesture identification processes on the at least one gesture and the one or more reduced sets of candidate gestures, wherein

the one or more second gesture identification processes consume a larger amount of processing power than the one or more first gesture identification processes; and

the processor further configured to determine a user input based at least in part on the at least one recognized gesture.

2. The augmented reality display system of claim 1 , wherein the processor is configured to generate a scoring value for a set of points identified for the at least one gesture based at least in part on comparison between the set of points and predetermined gestures and to recognize the at least one gesture when the scoring value exceeds a threshold value.

3. The augmented reality display system of claim 1 , further comprising a database to store predetermined gestures, wherein the computational resource utilization or expense requirement includes reducing or minimizing computational resource utilization for the gesture recognition for the at least one gesture, the first computational resource or expense requirement corresponds to a relatively lower computation resource utilization, and the second computational resource or expense requirement corresponds to a relatively higher computational resource utilization when compared to the first computational resource or expense requirement.

4. The augmented reality display system of claim 3 , further comprising a networked memory to access the database of predetermined gestures.

5. The augmented reality display system of claim 1 , wherein the processor is further configured to recognize the at least one gesture that comprises a hand gesture or motion or a finger gesture or a finger motion.

6. The augmented reality display system of claim 1 , wherein the augmented reality display system comprises a user wearable apparatus to display a virtual world as well as at least a portion of a physical environment in which the user is located.

7. The augmented reality display system of claim 1 , where the processor is further configured to recognize the at least one gesture that comprises an inter-finger interaction.

8. The augmented reality display system of claim 1 , wherein the processor is further configured to recognize the at least one gesture comprising at least one of inter-finger interactions, pointing, tapping, or rubbing.

9. The augmented reality display system of claim 1 , further comprising a spatial light modulator that is coupled directly without intervening elements or indirectly with one or more elements to the processor, and the processor controls the spatial light modulator in a manner such that one or more virtual objects are displayed to the user based at least in part on the user input.

10. The augmented reality display system of claim 9 , further comprising a virtual user interface to receive the user input or a user interaction with the virtual user interface or with the one or more virtual objects.

11. A method for determining user input, comprising:

capturing an image corresponding to a field of view of a user through an augmented reality system, wherein

the image comprises a gesture image of at least one gesture that is created by the user and interacts with virtual content projected by the augmented reality system to the user, and

at least a portion of the image is perceived by the user within the field of view provided by the augmented reality system;

identifying a plurality of candidate gestures and a plurality of computational utilization or expense requirements for gesture recognition of the at least one gesture;

determining whether the image includes one or more identifiable depth points at least by performing a line search for the image with one or more lines or line segments;

determining an order of processing in which a plurality of analysis nodes is executed to perform respective gesture identification processes on the at least one gesture with respect to the plurality of candidate gestures based at least in part upon a plurality of computational resource utilization or expense requirements;

during one or more earlier stages in the order of processing, generating one or more reduced sets of candidate gestures from the plurality of candidate gestures at least by executing one or more first gesture identification processes of the respective gesture identification processes that analyze the at least one image with a first analysis node to remove one or more candidate gestures from the plurality of candidate gestures based at least in part upon a first computational resource utilization or expense requirement of the plurality of computational resource utilization or expense requirements;

during one or more later stages in the order of processing, determining and recognizing the at least one gesture from the one or more reduced sets of candidate gestures as the at least one recognized gesture at least by analyzing the at least one image based at least in part on the one or more reduced sets of candidate gestures with a second analysis node corresponding to a second computational resource or expense requirement and at least by executing one or more second gesture identification processes on the at least one gesture and the one or more reduced sets of candidate gestures, wherein

the one or more second gesture identification processes consume a larger amount of processing power than the one or more first gesture identification processes; and

determining a user input based in part or in whole upon the at least one recognized gesture.

12. The method of claim 11 , further comprising generating a scoring value for a set of points for the at least one gesture based in part or in whole on results of comparing the set of points to a first set of points associated with a database including predetermined gestures.

13. The method of claim 12 , further comprising recognizing the at least one gesture when the scoring value exceeds a threshold value.

14. The method of claim 11 , further comprising overlaying a virtual world with at least a portion of a physical environment in which the user is located.

15. The method of claim 14 , further comprising accessing a networked memory to access a database including predetermined gestures.

16. The method of claim 11 , determining the user input further comprising recognizing the at least one gesture that comprises a hand gesture, a hand motion, a finger gesture, or a finger motion.

17. The method of claim 11 , further comprising capturing and recognizing a sensory input from the user or from a physical environment in which the user is located.

18. The method of claim 11 , determining the user input further comprising recognizing the at least one gesture that comprises an inter-finger interaction.

19. The method of claim 11 , determining the user input further comprising recognizing the at least one gesture that comprises at least one of inter-finger interactions, pointing, tapping, or rubbing.

20. The method of claim 11 , further comprising displaying one or more virtual objects to the user based at least in part on the user input.

21. The method of claim 20 , further comprising recognizing a user interaction with the one or more virtual objects or with a virtual world in which the one or more virtual objects are displayed in a virtual user interface based in part or in whole upon the at least one gesture.

22. The method of claim 11 , further comprising:

generating, at a processor of the augmented reality system, a scoring value for a set of points identified for the at least one gesture based at least in part on comparison between the set of points and predetermined gestures; and

recognizing the at least one gesture as the at least one recognized gesture when the scoring value exceeds a threshold value.

US14/738,877

2014-03-11

2015-06-13

Methods and systems for creating virtual and augmented reality

Active

US10203762B2

( en )

Priority Applications (5)

Application Number

Priority Date

Filing Date

Title

US14/738,877

US10203762B2

( en )

2014-03-11

2015-06-13

Methods and systems for creating virtual and augmented reality

US16/200,290

US10852838B2

( en )

2014-06-14

2018-11-26

Methods and systems for creating virtual and augmented reality

US17/074,256

US11507193B2

( en )

2014-06-14

2020-10-19

Methods and systems for creating virtual and augmented reality

US18/046,219

US11995244B2

( en )

2014-06-14

2022-10-13

Methods and systems for creating virtual and augmented reality

US18/635,985

US20240272724A1

( en )

2014-06-14

2024-04-15

Methods and systems for creating virtual and augmented reality

Applications Claiming Priority (6)

Application Number

Priority Date

Filing Date

Title

US14/205,126

US10629003B2

( en )

2013-03-11

2014-03-11

System and method for augmented and virtual reality

US201462012273P

2014-06-14

2014-06-14

US14/331,218

US9671566B2

( en )

2012-06-11

2014-07-14

Planar waveguide apparatus with diffraction element(s) and system employing same

US14/555,585

US9791700B2

( en )

2013-11-27

2014-11-27

Virtual and augmented reality systems and methods

US14/690,401

US10262462B2

( en )

2014-04-18

2015-04-18

Systems and methods for augmented and virtual reality

US14/738,877

US10203762B2

( en )

2014-03-11

2015-06-13

Methods and systems for creating virtual and augmented reality

Related Parent Applications (1)

Application Number

Title

Priority Date

Filing Date

US14/331,218

Continuation-In-Part

US9671566B2

( en )

2012-06-11

2014-07-14

Planar waveguide apparatus with diffraction element(s) and system employing same

Related Child Applications (1)

Application Number

Title

Priority Date

Filing Date

US16/200,290

Continuation

US10852838B2

( en )

2014-06-14

2018-11-26

Methods and systems for creating virtual and augmented reality

Publications (2)

Publication Number

Publication Date

US20160026253A1

US20160026253A1 ( en )

2016-01-28

US10203762B2

true

US10203762B2 ( en )

2019-02-12

Family

ID=55166744

Family Applications (1)

Application Number

Title

Priority Date

Filing Date

US14/738,877

Active

US10203762B2

( en )

2014-03-11

2015-06-13

Methods and systems for creating virtual and augmented reality

Country Status (1)

Country

Link

US

( 1 )

US10203762B2

( en )

Cited By (98)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20190025595A1

( en )

*

2016-01-15

2019-01-24

Meleap Inc.

Image display system, method for controlling image display system, image distribution system and head-mounted display

US20190113310A1

( en )

*

2017-09-15

2019-04-18

Tactacam LLC

Weapon sighted camera system

US10346141B1

( en )

*

2016-03-16

2019-07-09

Amazon Technologies, Inc.

Deterministic execution for visually developed operations

US10448342B1

( en )

*

2018-10-19

2019-10-15

Motorola Mobility Llc

Aggregate transmit power limiting on uncoordinated multiple transmitter device

US10462451B1

( en )

*

2017-05-12

2019-10-29

Facebook Technologies, Llc

Asymmetric structured light source

US10509228B1

( en )

*

2017-12-20

2019-12-17

Facebook Technologies, Llc

Low field myopia for artificial reality systems

US10567732B2

( en )

*

2017-02-06

2020-02-18

Robotemi Ltd

Method and device for stereoscopic vision

CN110824611A

( en )

*

2019-11-21

2020-02-21

歌尔股份有限公司

Uniformity compensation method, optical waveguide system and augmented reality device

US10600189B1

( en )

*

2016-12-06

2020-03-24

Apple Inc.

Optical flow techniques for event cameras

US10607353B2

( en )

*

2018-08-30

2020-03-31

Facebook Technologies, Llc

Structured light depth sensing

US10636157B2

( en )

*

2017-04-26

2020-04-28

Mitutoyo Corporation

Method and system for calculating a height map of a surface of an object from an image stack in scanning optical 2.5D profiling of the surface by an optical system

US10681183B2

( en )

2014-05-28

2020-06-09

Alexander Hertel

Platform for constructing and consuming realm and object featured clouds

US20200202625A1

( en )

*

2018-12-21

2020-06-25

Curious Company, LLC

Body pose message system

US10803668B2

( en )

2018-09-06

2020-10-13

Curious Company, LLC

Controlling presentation of hidden information

US10818088B2

( en )

2018-07-10

2020-10-27

Curious Company, LLC

Virtual barrier objects

US10846520B2

( en )

*

2017-12-12

2020-11-24

Zi-Nan Wang

Simulated sandtray system

US10856253B1

( en )

2020-01-31

2020-12-01

Dell Products, Lp

System and method for beamsteering acquisition and optimization in an enhanced reality environment

US10872584B2

( en )

2019-03-14

2020-12-22

Curious Company, LLC

Providing positional information using beacon devices

US10901291B1

( en )

2017-12-20

2021-01-26

Facebook Technologies, Llc

Bifocal optical assembly for a head-mounted display

US10924661B2

( en )

*

2019-05-02

2021-02-16

International Business Machines Corporation

Generating image capture configurations and compositions

US10936146B2

( en )

2019-02-22

2021-03-02

Microsoft Technology Licensing, Llc

Ergonomic mixed reality step-by-step instructions tethered to 3D holograms in real-world locations

US10979700B2

( en )

*

2018-03-27

2021-04-13

Canon Kabushiki Kaisha

Display control apparatus and control method

US10991162B2

( en )

2018-12-04

2021-04-27

Curious Company, LLC

Integrating a user of a head-mounted display into a process

US11009704B2

( en )

*

2016-06-20

2021-05-18

Akonia Holographies LLC

Pupil expansion

US11103763B2

( en )

2018-09-11

2021-08-31

Real Shot Inc.

Basketball shooting game using smart glasses

US20210287453A1

( en )

*

2020-03-13

2021-09-16

Magic Leap, Inc.

Three dimensional diorama for spatial computing assets

US20210304624A1

( en )

*

2020-03-26

2021-09-30

Seiko Epson Corporation

Method for setting target flight path of aircraft, target flight path setting system, and program for setting target flight path

US11141645B2

( en )

2018-09-11

2021-10-12

Real Shot Inc.

Athletic ball game using smart glasses

US11182044B2

( en )

*

2019-06-01

2021-11-23

Apple Inc.

Device, method, and graphical user interface for manipulating 3D objects on a 2D screen

US20210374382A1

( en )

*

2018-03-16

2021-12-02

Inveox Gmbh

Automated identification, orientation and sample detection of a sample container

US20210381806A1

( en )

*

2020-06-06

2021-12-09

Battelle Memorial Institute

High-definition electrical stimulation for enhanced spatial awareness and target alignment in weapon aiming applications

US11210857B2

( en )

2019-09-26

2021-12-28

The Toronto-Dominion Bank

Systems and methods for providing an augmented-reality virtual treasure hunt

US11226459B2

( en )

2018-02-13

2022-01-18

Apple Inc.

Integrated photonics device having integrated edge outcouplers

US11231319B1

( en )

2019-09-09

2022-01-25

Apple Inc.

Athermal wavelength stability monitor using a detraction grating

US11282248B2

( en )

2018-06-08

2022-03-22

Curious Company, LLC

Information display by overlay on an object

US11308576B2

( en )

2018-01-15

2022-04-19

Microsoft Technology Licensing, Llc

Visual stylization on stereoscopic images

US11308846B2

( en )

2020-03-13

2022-04-19

Apple Inc.

Electronic devices with color compensation

US11313695B2

( en )

2018-09-27

2022-04-26

Phiar Technologies, Inc.

Augmented reality navigational indicator

US20220130147A1

( en )

*

2019-02-22

2022-04-28

Fogale Nanotech

Method and device for monitoring the environment of a robot

US11321914B1

( en )

*

2018-01-10

2022-05-03

Amazon Technologies, Inc.

System for generating a navigational map of an environment

US20220155910A1

( en )

*

2020-11-16

2022-05-19

Samsung Electronics Co., Ltd.

Method for displaying user interface and electronic device therefor

US11423619B2

( en )

2020-03-25

2022-08-23

Volvo Car Corporation

System and method for a virtual showroom

US11422625B2

( en )

2019-12-31

2022-08-23

Human Mode, L.L.C.

Proxy controller suit with optional dual range kinematics

US11448518B2

( en )

*

2018-09-27

2022-09-20

Phiar Technologies, Inc.

Augmented reality navigational overlay

US20220319044A1

( en )

*

2021-04-02

2022-10-06

Toshiba Tec Kabushiki Kaisha

Information processing apparatus and information processing method

US11500154B1

( en )

2019-10-18

2022-11-15

Apple Inc.

Asymmetric optical power splitting system and method

US11513392B1

( en )

2021-09-23

2022-11-29

Apple Inc.

Direct-lit backlight units with optical films

WO2022256189A1

( en )

*

2022-05-20

2022-12-08

Innopeak Technology, Inc.

Hand gesture detection methods and systems with optimized hand detection

US11525967B1

( en )

2018-09-28

2022-12-13

Apple Inc.

Photonics integrated circuit architecture

US11533580B2

( en )

2019-04-30

2022-12-20

Apple Inc.

Locating content in an environment

US11620797B2

( en )

2021-08-05

2023-04-04

Bank Of America Corporation

Electronic user interface with augmented detail display for resource location

US11682175B2

( en )

2020-08-24

2023-06-20

Fd Ip &amp; Licensing Llc

Previsualization devices and systems for the film industry

US11699269B2

( en )

2021-08-25

2023-07-11

Bank Of America Corporation

User interface with augmented work environments

US11756392B2

( en )

2020-06-17

2023-09-12

Apple Inc.

Portable electronic device having a haptic button assembly

US11762470B2

( en )

2016-05-10

2023-09-19

Apple Inc.

Electronic device with an input device having a haptic engine

US11778856B2

( en )

2019-05-15

2023-10-03

Apple Inc.

Electronic device having emissive display with light recycling

US11825375B2

( en )

2019-04-30

2023-11-21

Apple Inc.

Locating content in an environment

US11822083B2

( en )

2019-08-13

2023-11-21

Apple Inc.

Display system with time interleaving

US11858199B2

( en )

2021-10-18

2024-01-02

NEXA3D Inc.

Methods and systems for photocuring liquid with reduced heat generation using a digital light processing (DLP) light source

US11875492B1

( en )

2023-05-01

2024-01-16

Fd Ip &amp; Licensing Llc

Systems and methods for digital compositing

US11881678B1

( en )

2019-09-09

2024-01-23

Apple Inc.

Photonics assembly with a photonics die stack

US20240036646A1

( en )

*

2022-07-28

2024-02-01

Ntt Docomo, Inc.

Controlling a user interface with a trackpad and a smart watch

US20240033930A1

( en )

*

2020-12-10

2024-02-01

Mitsubishi Electric Corporation

Remote control manipulator system and remote control assistance system

US11914201B2

( en )

2021-09-23

2024-02-27

Apple Inc.

Mechanisms that transfer light between layers of multi-chip photonic assemblies

US20240087238A1

( en )

*

2022-06-21

2024-03-14

T-Mobile Innovations Llc

System and method for extended reality processing in a local wireless environment with reduced latency

US11934579B2

( en )

2019-05-30

2024-03-19

Hewlett-Packard Development Company, L.P.

Haptic device activation for virtual events that exceed view boundary

US11950022B1

( en )

2020-04-24

2024-04-02

Apple Inc.

Head-mounted devices with forward facing cameras

US20240146699A1

( en )

*

2022-11-02

2024-05-02

Truist Bank

Secure packet record in multi-source vr environment

WO2024031109A3

( en )

*

2023-07-19

2024-05-10

Futurewei Technologies, Inc.

Audio-visual information system for smart eyewear

US11980813B2

( en )

2021-05-04

2024-05-14

Ztag, Inc.

System and method of using a virtual focal point in real physical game

US12028348B2

( en )

2022-07-07

2024-07-02

Bank Of America Corporation

System and method for performing interactions across geographical regions within a metaverse

US12039142B2

( en )

2020-06-26

2024-07-16

Apple Inc.

Devices, methods and graphical user interfaces for content applications

US20240323342A1

( en )

*

2023-03-21

2024-09-26

Apple Inc.

Electronic Device with Reliable Passthrough Video Fallback Capability and Hierarchical Failure Detection Scheme

US12111207B2

( en )

2022-09-23

2024-10-08

Apple Inc.

Despeckling in optical measurement systems

US12111210B2

( en )

2021-07-08

2024-10-08

Apple Inc.

Light source modules for noise mitigation

US12135429B2

( en )

2022-12-08

2024-11-05

Meta Platforms Technologies, Llc

Pupil-steering for three-dimensional (3D) resolution enhancement in single photon avalanche diode (SPAD) eye tracking (ET)

US12153726B1

( en )

*

2023-06-30

2024-11-26

Adobe Inc.

Integrating text of a document into an extended reality environment

US12169902B2

( en )

2021-09-21

2024-12-17

Apple Inc.

Methods and systems for composing and executing a scene

US12218479B2

( en )

2020-07-20

2025-02-04

Apple Inc.

Photonic integrated circuits with controlled collapse chip connections

US20250050236A1

( en )

*

2021-12-03

2025-02-13

Sony Interactive Entertainment Inc.

Communication system

US12230224B1

( en )

2022-08-25

2025-02-18

Apple Inc.

Displays with content-specific headroom

US12254551B2

( en )

2022-07-13

2025-03-18

Fd Ip &amp; Licensing Llc

Method and application for animating computer generated images

US12265657B2

( en )

2020-09-25

2025-04-01

Apple Inc.

Methods for navigating user interfaces

US12309163B2

( en )

2023-04-25

2025-05-20

Bank Of America Corporation

System and method for managing metaverse instances

US12326971B1

( en )

2024-10-03

2025-06-10

Bansen Labs, Llc

System and method for facilitating adaptive recentering in virtual reality environments

US20250201153A1

( en )

*

2023-12-18

2025-06-19

Apple Inc.

Head-Mounted Device with Content Dimming for Masking Noise

US12347348B2

( en )

2020-09-25

2025-07-01

Apple Inc.

Structured display shutdown for video pass-through electronic devices

US12348856B2

( en )

2022-02-11

2025-07-01

Samsung Electronics Co., Ltd.

Method and device for obtaining image of object

US12406454B2

( en )

2016-03-31

2025-09-02

Magic Leap, Inc.

Interactions with 3D virtual objects using poses and multiple-dof controllers

US12432330B2

( en )

2022-09-19

2025-09-30

Apple Inc.

Head-mounted electronic device with reliable passthrough video fallback capability

US12445759B2

( en )

2018-09-25

2025-10-14

Apple Inc.

Haptic output system

US12474788B2

( en )

2015-10-20

2025-11-18

Magic Leap, Inc.

Selecting virtual objects in a three-dimensional space

US12549436B2

( en )

2021-08-10

2026-02-10

International Business Machines Corporation

Internet of things configuration using eye-based controls

US12567364B1

( en )

2023-12-05

2026-03-03

Apple Inc.

Displays with locally boosted contrast

US12591313B2

( en )

2023-06-27

2026-03-31

Samsung Electronics Co., Ltd.

Widget interaction for extended reality (XR) applications

US12591297B2

( en )

2017-04-19

2026-03-31

Magic Leap, Inc.

Multimodal task execution and text editing for a wearable system

US12608890B2

( en )

2022-04-20

2026-04-21

Apple Inc.

Obstructed objects in a three-dimensional environment

US12620187B2

( en )

2022-05-17

2026-05-05

Apple Inc.

Systems, methods, and user interfaces for generating a three-dimensional virtual representation of an object

Families Citing this family (1566)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

GB0522968D0

( en )

2005-11-11

2005-12-21

Popovich Milan M

Holographic illumination device

GB0718706D0

( en )

2007-09-25

2007-11-07

Creative Physics Ltd

Method and apparatus for reducing laser speckle

US11726332B2

( en )

2009-04-27

2023-08-15

Digilens Inc.

Diffractive projection apparatus

US9335604B2

( en )

2013-12-11

2016-05-10

Milan Momcilo Popovich

Holographic waveguide display

US9720899B1

( en )

2011-01-07

2017-08-01

Narrative Science, Inc.

Automatic generation of narratives from data using communication goals and narrative analytics

US10185477B1

( en )

2013-03-15

2019-01-22

Narrative Science Inc.

Method and system for configuring automatic generation of narratives from data

US9274349B2

( en )

2011-04-07

2016-03-01

Digilens Inc.

Laser despeckler based on angular diversity

US10670876B2

( en )

2011-08-24

2020-06-02

Digilens Inc.

Waveguide laser illuminator incorporating a despeckler

EP2995986B1

( en )

2011-08-24

2017-04-12

Rockwell Collins, Inc.

Data display

WO2016020630A2

( en )

2014-08-08

2016-02-11

Milan Momcilo Popovich

Waveguide laser illuminator incorporating a despeckler

US9606992B2

( en )

*

2011-09-30

2017-03-28

Microsoft Technology Licensing, Llc

Personal audio/visual apparatus providing resource management

WO2013102759A2

( en )

2012-01-06

2013-07-11

Milan Momcilo Popovich

Contact image sensor using switchable bragg gratings

WO2013163347A1

( en )

2012-04-25

2013-10-31

Rockwell Collins, Inc.

Holographic wide angle display

US9417660B2

( en )

2012-04-25

2016-08-16

Kopin Corporation

Collapsible head set computer

US9456744B2

( en )

2012-05-11

2016-10-04

Digilens, Inc.

Apparatus for eye tracking

US20130339859A1

( en</spa

Related documents

• Augmented reality system with spatialized audio tied to user manipulated … — Magic Leap, Inc. (US11231770B2)#653963
• Virtual reality system using multiple force arrays for a solver — DeepMotion, Inc. (US10657696B2)#653961
• High-efficiency fuel cell system with carbon dioxide capture assembly and … — Fuelcell Energy, Inc. (US10170782B2)#653965
• Integrated electrochemical cell power generation and electrolysis systems — Czero, Inc. (US20250087711A1)#653966
• Generating augmented reality experiences utilizing physical objects to … — Meta Platforms Technologies, Llc (US20260037214A1)#653960