ABSTRACT
Abstract
An augmented reality system includes left and right transparent eyepieces through which a user can view the real world and which serve to couple imagewise modulated light into the user's eyes in order to display virtual content components of a mixed reality experience. The system further includes left and right speakers and a facility for user spatial manipulation of virtual objects. The system produces spatialized audio that has a virtual sound source position fixed to one or more virtual objects that are spatially manipulated by the user. Thus the system provides more realistic visual and auditory presentation of virtual components of a mixed reality experience.
Description
RELATED APPLICATIONS
This patent application is a continuation of U.S. patent application Ser. No. 15/937,721 filed on Mar. 27, 2018, which claims priority to U.S. Provisional Patent Application No. 62/477,976 filed Mar. 28, 2017, which are hereby incorporated by reference in their entirety.
FIELD OF THE INVENTION
The invention pertains to augmented reality systems.
BACKGROUND
Significant steps in the development of display technology include: the magic lantern, film based motion pictures, television, and modern digital video display technologies implemented in personal computers and mobile devices. Recently virtual reality headgear has been introduced. Such virtual reality headgear includes a pair of displays positioned close to and in front of the user's eyes and lenses intervening between the displays and the user's eyes to allow the user to focus on the displays. When using such virtual reality headgear the user is completely immersed in a virtual world and substantially cut off from the real world. Protracted use of virtual reality systems can induce âvirtual reality sicknessâ which may be due to flaws in the realisticness of the sensation of the virtual environment. For example the sensory stimulus received via the vestibular system which is dependent on the user's dynamics in the real world may conflict with sensory stimulus of motion in the virtual world. It is believed that the human perception systems are highly attuned to integrating disparate, visual, auditory, touch, taste, smell and vestibular stimuli, and contradictory sensations causes sensory discord which can cause the human physiological systems to transition to a low functionally (virtual reality sickness) state that inhibits further engagement in the causative activity.
It would be desirable to provide a presentation of virtual content with achieves higher fidelity to the sensory perception that would be achieved if the virtual content were real.
SUMMARY
One aspect of the subject matter disclosed herein includes providing augmented reality system that includes headgear that includes a left eye display that includes a left transparent eyepiece and a right eye display that includes a right transparent eyepiece. The transparent eyepieces are used to couple images of virtual content into the user's field of view while simultaneously allowing the user see and be grounded (e.g., have a comfortable feeling of being oriented and balanced) in the real world. The headgear also includes a left speaker and a right speaker. The system also includes a handheld controller. A tracking subsystem that allows the relative translation offset and orientation of the handheld controller with respect to the headgear is also included in the system. The system also includes at least one processor coupled to the left eye display, the right eye display, the left speaker, the right speaker and the tracking system. The at least one processor could be included in the headgear, the handheld controller, and/or a separate unit. The at least one processor operates a system including, at least the headgear and the handheld controller. Left and right stereoscopic images of a first type virtual object are projected through, respectively, the left transparent eyepiece and the right transparent eyepieces. In certain cases the left and right displays may be configured to allow the curvature of the wave front of light reaching the user's eyes to be altered based on the distance to the virtual objects. The wave front curvature provided by the left and right displays may be quantized such that a limited number of values are available or may be continuously variable. The first type virtual object is maintained substantially fixed relative to the handheld controller. Because the virtual object is fixed to the handheld controller, by moving the handheld controller the user can move the first type virtual object and see the first type virtual object move relative to the real world through the transparent eyepieces. The first type virtual object emits a sound. The sound is emitted from a point in space occupied by the virtual object. The point in space from which the sound is emitted moves along with the virtual object under the control of the user through the facility of the handheld controller. As the point in space is moved an appropriate Head Related Transfer Function (HRTF) for the coordinates of the point in space relative to the user's head is used to process the sound emitted by the object and the stereo output of the HRTF is output through the left and right speakers. In this manner the user hears a sound that is substantially realistically spatialized so as to sound as though emanating from the position to which the user has moved the virtual object.
In an alternative to the above described aspect, the handheld controller is not used. Rather, a machine vision subsystem (of the augmented reality system) including a camera tracks a user's hand and the spatialized sound emitting virtual object is, at least temporarily, fixed to the user's hand. The augmented reality system can also be configured to recognize certain gestures (e.g., a throwing motion) that direct that the virtual object be decoupled from the user's hand. After such decoupling the spatialized audio will continue to be emitted from the moving position of the virtual object.
An additional aspect of the subject matter disclosed herein involves a second type virtual object the motion of which is defined in an inertial motion fixed to a real world environment in which a user of the augmented reality system is situated. A spatialized sound for the second type virtual object has a virtual source point that is occupied by the second type of virtual object. The second type virtual object may for example obey Newton's laws of motion with some imposed preprogrammed mass. For special applications, in addition to the mass, an electric charge or magnetic dipole moment may also be assigned to the object so that it can be made to properly interact with additional real or virtual objects that have an electric charge or magnetic dipole moment. The user can interact with the second type virtual object using his or her hands via the agency of hand gesture recognition implemented in the system. For example the user could push or take hold of the second virtual object with a prearranged hand gesture that the system is programmed to recognize. By pushing the second type virtual object the user can impart a velocity to the second virtual object which will then continue in accordance with Newton's laws and may subsequently be subject to further manipulation by the user and/or interact with other virtual objects or real objects. For example the virtual object can be virtual ball and the user may be playing a game of handball using the virtual ball and a real wall. The real wall may be part of a mixed reality object that also includes a virtual representation of the wall. The virtual ball may emit a âwhooshâ sound that corresponds the sound of an object moving at high speed through the air. The whoosh sound is spatialized and has a virtual sound source that is occupied by the virtual ball. A second spatialized bounce sound with a virtual source position occupied by the virtual ball may be emitted when the virtual ball bounces off of an augmented reality wall that includes a real wall component and its associated virtual wall representation component. While the second type virtual object is in motion, in order to maintain its motion (e.g., per Newton's 1 st law) relative an inertial reference frame fixed to the environment of the user, even in the case that the user rotates his/her head along with the headgear, the motion of the user's head must be tracked and an alteration of the images of the second type virtual object displayed through the eyepieces must be performed in order to compensate for the motion of the user's head. The nature of the alteration bears further explanation. The images are altered to maintain coordinates of the second type virtual object as defined in the inertial reference frame tied to the local environment (including any intended motion within the inertial reference frame) unaffected by the rotation of the headgear along with the user's head.
BRIEF DESCRIPTION OF THE DRAWINGS
The drawings illustrate the design and utility of preferred embodiments of the present invention, in which similar elements are referred to by common reference numerals. In order to better appreciate how the above-recited and other advantages and objects of the present inventions are obtained, a more particular description of the present inventions briefly described above will be rendered by reference to specific embodiments thereof, which are illustrated in the accompanying drawings. Understanding that these drawings depict only typical embodiments of the invention and are not therefore to be considered limiting of its scope, the invention will be described and explained with additional specificity and detail through the use of the accompanying drawings in which:
FIG. 1 shows an augmented reality system according to an embodiment of the invention;
FIG. 2 is a top view of wearable headgear of the system shown in FIG. 1 positioned on a user's head;
FIG. 3 is a front view of the wearable headgear shown in FIGS. 2-3 ;
FIG. 4 is an edge on view from the top of a right eyepiece and a schematic representation of a right source imagewise modulated light of the augmented reality headgear shown in FIGS. 1-3 .
FIG. 5 is a block diagram of the system shown in FIGS. 1-3 according to an embodiment in the invention;
FIG. 6 shows certain components of an electromagnetic tracking system including a receive side set of three orthogonal solenoids along with a transmit side set of three orthogonal solenoids that can be used for tracking relative coordinates in the systems shown in FIGS. 1-5 according to an embodiment of the invention;
FIG. 7 is a flowchart of a method of operating the system shown in FIGS. 1-6 according to embodiments of the invention;
FIG. 8 is a flowchart of a method of generating spatialized audio that may be used in the method shown in FIG. 7 and FIG. 14 ;
FIG. 9 is a flowchart of a method of using optical sensors to collect environmental information that is used to enhance the realisticness of audio signals associated with virtual content according to embodiments of the invention;
FIG. 10 is a flowchart of a method of using environmental information that has been collected by the method shown in FIG. 9 to generate spatialized audio in the methods shown in FIG. 7 and FIG. 14 according to embodiments of the invention;
FIGS. 11-12 depict a user using the system described above with reference to FIGS. 1-10 and experiencing spatialized audio simulated to be emanating from a first virtual object that the user is manipulating with a controller such as shown in FIG. 1 ;
FIG. 13 depicts a user using the system described above with reference to FIGS. 1-10 and experiencing spatialized audio emanating from a second virtual object that the user is manipulating with a controller such as shown in FIG. 1 ;
FIG. 14 is a flowchart of a method of operating an augmented reality system according to another embodiment of the invention; and
FIG. 15 depicts a user using an augmented reality system that is operating according to the method shown in FIG. 14 .
FIG. 16 schematically represents coupling of an environmental sound to a person's two ears.
DETAILED DESCRIPTION
FIG. 1 shows an augmented reality system 100 according to an embodiment of the invention. As shown in FIG. 1 , the system 100 includes augmented reality headgear 102 , a handheld controller 104 , and an auxiliary unit 106 . The augmented reality headgear 102 includes a left (user's left) transparent waveguide set eyepiece (herein below âleft eyepieceâ) 108 and a right transparent waveguide set eyepiece (herein below âright eyepieceâ) 110 . Each eyepiece
108 , 110 includes surface diffractive optical elements for controlling the flow of imagewise modulated light. In particular, the left eyepiece 108 includes a left incoupling grating set 112 (a first of which is visible in FIG. 1 and FIG. 3 ), a left orthogonal pupil expansion (OPE) grating set 114 (a first of which is visible in FIG. 1 and FIG. 3 ) and a left exit (output) pupil expansion (EPE) grating set 116 (a first of which is visible in FIG. 1 and FIG. 3 ). Similarly the right eyepiece 110 includes a right incoupling grating set 118 , a right OPE grating set 120 and a right EPE grating set 122 . Imagewise modulated light is transferred via the incoupling gratings
112 , 118 , OPEs
114 , 120 and EPE
116 , 122 to a user's eye. Alternatively, in lieu of the incoupling grating sets
RELATED APPLICATIONS
This patent application is a continuation of U.S. patent application Ser. No. 15/937,721 filed on Mar. 27, 2018, which claims priority to U.S. Provisional Patent Application No. 62/477,976 filed Mar. 28, 2017, which are hereby incorporated by reference in their entirety.
FIELD OF THE INVENTION
The invention pertains to augmented reality systems.
BACKGROUND
Significant steps in the development of display technology include: the magic lantern, film based motion pictures, television, and modern digital video display technologies implemented in personal computers and mobile devices. Recently virtual reality headgear has been introduced. Such virtual reality headgear includes a pair of displays positioned close to and in front of the user's eyes and lenses intervening between the displays and the user's eyes to allow the user to focus on the displays. When using such virtual reality headgear the user is completely immersed in a virtual world and substantially cut off from the real world. Protracted use of virtual reality systems can induce âvirtual reality sicknessâ which may be due to flaws in the realisticness of the sensation of the virtual environment. For example the sensory stimulus received via the vestibular system which is dependent on the user's dynamics in the real world may conflict with sensory stimulus of motion in the virtual world. It is believed that the human perception systems are highly attuned to integrating disparate, visual, auditory, touch, taste, smell and vestibular stimuli, and contradictory sensations causes sensory discord which can cause the human physiological systems to transition to a low functionally (virtual reality sickness) state that inhibits further engagement in the causative activity.
It would be desirable to provide a presentation of virtual content with achieves higher fidelity to the sensory perception that would be achieved if the virtual content were real.
SUMMARY
One aspect of the subject matter disclosed herein includes providing augmented reality system that includes headgear that includes a left eye display that includes a left transparent eyepiece and a right eye display that includes a right transparent eyepiece. The transparent eyepieces are used to couple images of virtual content into the user's field of view while simultaneously allowing the user see and be grounded (e.g., have a comfortable feeling of being oriented and balanced) in the real world. The headgear also includes a left speaker and a right speaker. The system also includes a handheld controller. A tracking subsystem that allows the relative translation offset and orientation of the handheld controller with respect to the headgear is also included in the system. The system also includes at least one processor coupled to the left eye display, the right eye display, the left speaker, the right speaker and the tracking system. The at least one processor could be included in the headgear, the handheld controller, and/or a separate unit. The at least one processor operates a system including, at least the headgear and the handheld controller. Left and right stereoscopic images of a first type virtual object are projected through, respectively, the left transparent eyepiece and the right transparent eyepieces. In certain cases the left and right displays may be configured to allow the curvature of the wave front of light reaching the user's eyes to be altered based on the distance to the virtual objects. The wave front curvature provided by the left and right displays may be quantized such that a limited number of values are available or may be continuously variable. The first type virtual object is maintained substantially fixed relative to the handheld controller. Because the virtual object is fixed to the handheld controller, by moving the handheld controller the user can move the first type virtual object and see the first type virtual object move relative to the real world through the transparent eyepieces. The first type virtual object emits a sound. The sound is emitted from a point in space occupied by the virtual object. The point in space from which the sound is emitted moves along with the virtual object under the control of the user through the facility of the handheld controller. As the point in space is moved an appropriate Head Related Transfer Function (HRTF) for the coordinates of the point in space relative to the user's head is used to process the sound emitted by the object and the stereo output of the HRTF is output through the left and right speakers. In this manner the user hears a sound that is substantially realistically spatialized so as to sound as though emanating from the position to which the user has moved the virtual object.
In an alternative to the above described aspect, the handheld controller is not used. Rather, a machine vision subsystem (of the augmented reality system) including a camera tracks a user's hand and the spatialized sound emitting virtual object is, at least temporarily, fixed to the user's hand. The augmented reality system can also be configured to recognize certain gestures (e.g., a throwing motion) that direct that the virtual object be decoupled from the user's hand. After such decoupling the spatialized audio will continue to be emitted from the moving position of the virtual object.
An additional aspect of the subject matter disclosed herein involves a second type virtual object the motion of which is defined in an inertial motion fixed to a real world environment in which a user of the augmented reality system is situated. A spatialized sound for the second type virtual object has a virtual source point that is occupied by the second type of virtual object. The second type virtual object may for example obey Newton's laws of motion with some imposed preprogrammed mass. For special applications, in addition to the mass, an electric charge or magnetic dipole moment may also be assigned to the object so that it can be made to properly interact with additional real or virtual objects that have an electric charge or magnetic dipole moment. The user can interact with the second type virtual object using his or her hands via the agency of hand gesture recognition implemented in the system. For example the user could push or take hold of the second virtual object with a prearranged hand gesture that the system is programmed to recognize. By pushing the second type virtual object the user can impart a velocity to the second virtual object which will then continue in accordance with Newton's laws and may subsequently be subject to further manipulation by the user and/or interact with other virtual objects or real objects. For example the virtual object can be virtual ball and the user may be playing a game of handball using the virtual ball and a real wall. The real wall may be part of a mixed reality object that also includes a virtual representation of the wall. The virtual ball may emit a âwhooshâ sound that corresponds the sound of an object moving at high speed through the air. The whoosh sound is spatialized and has a virtual sound source that is occupied by the virtual ball. A second spatialized bounce sound with a virtual source position occupied by the virtual ball may be emitted when the virtual ball bounces off of an augmented reality wall that includes a real wall component and its associated virtual wall representation component. While the second type virtual object is in motion, in order to maintain its motion (e.g., per Newton's 1 st law) relative an inertial reference frame fixed to the environment of the user, even in the case that the user rotates his/her head along with the headgear, the motion of the user's head must be tracked and an alteration of the images of the second type virtual object displayed through the eyepieces must be performed in order to compensate for the motion of the user's head. The nature of the alteration bears further explanation. The images are altered to maintain coordinates of the second type virtual object as defined in the inertial reference frame tied to the local environment (including any intended motion within the inertial reference frame) unaffected by the rotation of the headgear along with the user's head.
BRIEF DESCRIPTION OF THE DRAWINGS
The drawings illustrate the design and utility of preferred embodiments of the present invention, in which similar elements are referred to by common reference numerals. In order to better appreciate how the above-recited and other advantages and objects of the present inventions are obtained, a more particular description of the present inventions briefly described above will be rendered by reference to specific embodiments thereof, which are illustrated in the accompanying drawings. Understanding that these drawings depict only typical embodiments of the invention and are not therefore to be considered limiting of its scope, the invention will be described and explained with additional specificity and detail through the use of the accompanying drawings in which:
FIG. 1 shows an augmented reality system according to an embodiment of the invention;
FIG. 2 is a top view of wearable headgear of the system shown in FIG. 1 positioned on a user's head;
FIG. 3 is a front view of the wearable headgear shown in FIGS. 2-3 ;
FIG. 4 is an edge on view from the top of a right eyepiece and a schematic representation of a right source imagewise modulated light of the augmented reality headgear shown in FIGS. 1-3 .
FIG. 5 is a block diagram of the system shown in FIGS. 1-3 according to an embodiment in the invention;
FIG. 6 shows certain components of an electromagnetic tracking system including a receive side set of three orthogonal solenoids along with a transmit side set of three orthogonal solenoids that can be used for tracking relative coordinates in the systems shown in FIGS. 1-5 according to an embodiment of the invention;
FIG. 7 is a flowchart of a method of operating the system shown in FIGS. 1-6 according to embodiments of the invention;
FIG. 8 is a flowchart of a method of generating spatialized audio that may be used in the method shown in FIG. 7 and FIG. 14 ;
FIG. 9 is a flowchart of a method of using optical sensors to collect environmental information that is used to enhance the realisticness of audio signals associated with virtual content according to embodiments of the invention;
FIG. 10 is a flowchart of a method of using environmental information that has been collected by the method shown in FIG. 9 to generate spatialized audio in the methods shown in FIG. 7 and FIG. 14 according to embodiments of the invention;
FIGS. 11-12 depict a user using the system described above with reference to FIGS. 1-10 and experiencing spatialized audio simulated to be emanating from a first virtual object that the user is manipulating with a controller such as shown in FIG. 1 ;
FIG. 13 depicts a user using the system described above with reference to FIGS. 1-10 and experiencing spatialized audio emanating from a second virtual object that the user is manipulating with a controller such as shown in FIG. 1 ;
FIG. 14 is a flowchart of a method of operating an augmented reality system according to another embodiment of the invention; and
FIG. 15 depicts a user using an augmented reality system that is operating according to the method shown in FIG. 14 .
FIG. 16 schematically represents coupling of an environmental sound to a person's two ears.
DETAILED DESCRIPTION
FIG. 1 shows an augmented reality system 100 according to an embodiment of the invention. As shown in FIG. 1 , the system 100 includes augmented reality headgear 102 , a handheld controller 104 , and an auxiliary unit 106 . The augmented reality headgear 102 includes a left (user's left) transparent waveguide set eyepiece (herein below âleft eyepieceâ) 108 and a right transparent waveguide set eyepiece (herein below âright eyepieceâ) 110 . Each eyepiece
108 , 110 includes surface diffractive optical elements for controlling the flow of imagewise modulated light. In particular, the left eyepiece 108 includes a left incoupling grating set 112 (a first of which is visible in FIG. 1 and FIG. 3 ), a left orthogonal pupil expansion (OPE) grating set 114 (a first of which is visible in FIG. 1 and FIG. 3 ) and a left exit (output) pupil expansion (EPE) grating set 116 (a first of which is visible in FIG. 1 and FIG. 3 ). Similarly the right eyepiece 110 includes a right incoupling grating set 118 , a right OPE grating set 120 and a right EPE grating set 122 . Imagewise modulated light is transferred via the incoupling gratings
112 , 118 , OPEs
114 , 120 and EPE
116 , 122 to a user's eye. Alternatively, in lieu of the incoupling grating sets 112 , 118 , OPE grating sets 114 , 120 and EPE grating sets 116 , 122 the eyepieces
108 , 110 include refractive and reflective features for controlling the coupling of imagewise modulated light to the user's eyes.
A left source of imagewise modulated light 124 is optically coupled into the left eyepiece 108 through the left incoupling grating set 112 and a right source of imagewise modulated light 126 is optically coupled into the right eyepiece 110 through the right incoupling grating set 118 . The sources of imagewise modulated light 124 , 126 can, for example, take the form of optical fiber scanners, projectors including electronic light modulators such as, for example, Digital Light Processing (DLP) chips, or Liquid Crystal on Silicon (LCoS) modulators, or emissive displays, such as for example, micro Light Emitting Diode (μLED) or micro Organic Light Emitting Diode (μOLED) panels coupled into the incoupling grating sets 112 , 118 using one or more lenses per side. The input coupling grating sets 112 , 118 deflect light from the sources of imagewise modulated light 124 , 126 to angles above the critical angle for Total Internal Reflection (TIR) for the eyepieces
108 , 110 . The OPE grating sets 114 , 120 incrementally deflect light propagating by TIR down toward the EPE grating sets 116 , 122 . The EPE grating sets 116 , 122 incrementally couple light out toward the user's face including the user's eyes' pupils. The left eyepiece 108 in combination with the left source of imagewise modulated light 124 constitutes a left display and the right eyepiece 110 in combination with the right source of imagewise modulated light 126 constitutes a right display.
The auxiliary unit 106 can include a battery ( 526 , FIG. 5 ) to provide energy to operate the system 100 , and can include a processor ( 516 , FIG. 5 ) for executing programs to operate the system 100 . As shown the auxiliary unit 106 includes a clip 128 that is useful for attaching the auxiliary unit 106 to a user's belt. Alternatively the auxiliary unit 106 can have a different form factor.
The augmented reality headgear 102 also includes a left temple arm 130 and a right temple arm 132 . The left temple arm 130 includes a left temple speaker port 134 and the right temple arm 132 includes a right temple speaker port 136 . An orthogonal coil electromagnetic receiver 138 is accommodated in the left temple piece although it could be located elsewhere in the headgear 102 . An Inertial Measurement Unit (IMU) 140 is accommodated in the right temple arm 132 although it could be located elsewhere in the headgear 102 . The headgear 102 also includes a left depth (e.g., time-of-flight) camera 142 and a right depth camera 144 . The depth cameras
142 , 144 are suitably oriented in different directions so as to together cover a wider field of view.
The handheld controller 104 includes a grip portion 146 and a top 148 that includes a plurality of buttons 150 . The buttons 150 may also be used as an optical tracking target for tracking six degrees of freedom (3 degrees of translation and 3 degrees of rotation) of motion handheld controller 104 . Additionally, as discussed further below the handheld controller 104 can include additional components for tracking six degrees of motion of the handheld controller 104 relative to the headgear 102 . The auxiliary unit 106 is coupled to the headgear 102 through a multiconduit cable 152 which can, for example, include electrical wires and fiber optics. Wireless connections between the auxiliary unit 106 and the headgear 102 can also be used.
FIG. 2 is a top view of wearable headgear 102 of the system 100 shown in FIG. 1 positioned on a user's head 202 . The user's left eye 204 is positioned to look through the left eyepiece 108 and the user's right eye 206 is positioned to look through the right eyepiece 110 . The left speaker port 134 is positioned next to the user's left ear 208 and the right speaker port 136 is positioned next to the user's right ear 210 . FIG. 3 is a front view of the wearable headgear shown in FIGS. 2-3 .
According to certain embodiments each of the left eyepiece 108 and the right eyepiece 110 includes a stack of multiple waveguides 402 ( FIG. 4 ). For example each eyepiece
108 , 110 can individual waveguides dedicated to each of multiple (e.g., red, blue and green) color channels. Additionally each eyepiece
108 , 110 can include multiple sets of waveguides, with each set including individual waveguides for handling different (e.g., red, blue and green) color channels with the sets being differentiated by being configured to impart different wave front curvature to emitted light. The wave front curvature would typically be diverging (convex) toward the user's eyes so as to correspond to a virtual object position spaced in front of the user by a distance corresponding to the reciprocal of wave front curvature. As shown in FIG. 1 and FIG. 3 the EPE gratings in the EPE grating set 122 include curved grating grooves. The curved grating grooves serve to impart the above described wave front curvature by altering the Poynting vector of exiting light across each EPE.
Stereoscopically adjusted left and right eye imagery is output to the user through the imagewise light modulators
124 , 126 and the eyepieces
108 , 110 in order to contribute to the perception that displayed content is three dimensional. Selecting a waveguide through which to output imagery to best match the wave front curvature with which a virtual object is displayed to match the distance indicated by the stereoscopic left and right images aids in increasing the realisticness of the three dimensional imagery and helps to avoid a sensory conflict between the depth perception cues due the difference between the left and right eye imagery and the autonomic accommodation (object distance dependent focus) of the human eye. While not wishing to be bound to any particular theory of operation of the augmented reality system 100 , it is believe that such sensory conflicts as they exist in prior virtual and augmented reality system may be one source of virtual reality sickness.
FIG. 4 is an edge on view from the top of the right eyepiece 110 and a schematic representation of the right source imagewise modulated light 126 of the augmented reality headgear 102 shown in FIGS. 1-3 . Although not shown the structure of the left eyepiece 108 is a mirror image of the structure of the right eyepiece 110 . As shown in FIG. 4 the stack of waveguides 402 includes a first subset of three waveguides 404 and a second subset of three waveguides 406 . The two subsets of waveguides
404 , 406 are differentiated by having different EPE gratings (not shown in FIG. 4 ) which have different grating line curvatures to impart different wave front curvature to exiting light. Within each of the subsets of waveguides
404 , 406 each waveguide can be used to couple a different spectral channel (e.g., one of red, green and blue spectral channels) to the user's right eye 206 .
FIG. 5 is a block diagram of the system 100 shown in FIGS. 1-3 according to an embodiment in the invention. As shown in FIG. 5 the handheld controller 104 (also known as a âtotemâ) includes a totem-to-headgear six degree of freedom (6 DOF) totem subsystem 502 and the augmented reality headgear 102 includes a totem-to- headgear 6 DOF headgear subsystem 504 . The 6 DOF totem subsystem 502 and the 6 DOF headgear subsystem 504 cooperate to determine six coordinates of the handheld controller 104 relative to the augmented reality headgear 102 . The six coordinates include three translation offsets and three degrees of freedom for rotation (orientation) coordinates. The six degrees of freedom may be expressed relative to a coordinate system fixed to the headgear 102 . The three translation offsets may be expressed as X, Y, and Z offsets in such a coordinate system. And the rotation degrees of freedom may be expressed as sequence of yaw, pitch and roll rotations or as a Quaternion orientation. According to one approach the 6 DOF headgear system 504 , the depth cameras 142 , 144 (alternatively non-depth camera(s)) included in the headgear 102 and an optical target, e.g., in the form of buttons 150 as described above, or in the form of dedicated indicia included in the handheld controller 104 are used for 6 DOF tracking. Alternatively the handheld controller 104 can include a camera and the headgear 102 can include an optical target that together are used for optical tracking. According to another approach which is discussed more fully below the headgear 102 and the handheld controller 104 each include a set of three orthogonally oriented solenoids which are used to wirelessly send and receive signals. By measuring the relative magnitude of the signals received in each of the coils used for receiving, the 6 DOF of the handheld controller 104 relative to the augmented reality headgear 102 may be determined. Various techniques for determining the relative position and orientation between two sets of three orthogonal coils are known in the art of motion tracking and may be used to track the relative position and orientation of the hand controller 104 with respect to the headgear 102 . (Additionally, 6 DOF totem subsystem 502 can include an Inertial Measurement Unit (IMU) that is useful to provide improved accuracy and/or more timely information on rapid movements of the handheld controller 104 .
In order more fully realize the illusion of virtual content in the augmented reality system 100 it is useful that the motion or stationary status of each virtual object be defined in an inertial reference frame fixed to the user's environment notwithstanding the fact that the user along with the augmented reality headgear 102 that is generating the virtual content may be ambulating and pivoting their head. For example, if a virtual person is seated on a real chair in front of the user, the virtual person should remain seated when the user rotates their head 10° to the left, and not suddenly be shifted out of the chair into space adjacent to the chair. Similarly if a virtual person is walking at a steady pace across a real room the virtual person's steady walk should be maintained notwithstanding the user walking toward the virtual person (unless the virtual person is specifically programmed to react in a different manner). To maintain virtual objects motion as defined relative to the environment fixed inertial reference frame the augmented reality headgear tracks the user's head orientation (termed âhead poseâ) and position with respect to the local environment (e.g., position within a room in which the user is located). The user's head pose and position can be determined processing imagery from the depth cameras
142 , 144 using a Simultaneous Localization and Mapping (SLAM) and visual odometry procedure. As shown in FIG. 5 the depth cameras
142 , 144 are coupled to a SLAM/ visual odometry block 506 . The SLAM/ visual odometry block 506 implementation can, for example, include a programmed processor which is a form of electronic circuitry. An additional source of information on the user's head pose and location is obtained from a headgear Inertial Measurement Unit (IMU) 508 . Information from the IMU 508 can be integrated with information from the SLAM/ visual odometry block 506 to provide improved accuracy and/or more timely information on rapid adjustments of the user's head pose and position.
Coordinates of the handheld controller 104 as determined by the 6 DOF totem subsystem 502 and the 6 DOF headgear subsystem 504 can be transformed to a coordinate system fixed to the user's environment which is also used as a coordinate system for defining the 6 DOF of virtual content.
The depth cameras
142 , 144 are also coupled to and supply 3D imagery to a hand gesture tracker 510 . The hand gesture tracker 510 is configured to match 3D imagery received from the depth cameras
142 , 144 to stored patterns representing for each of multiple hand gestures that are recognizable by the hand gesture tracker 510 .
The headgear 102 also includes a left speaker 512 that is acoustically coupled to the user's left ear 208 through the left speaker port 134 and a right speaker 514 that is acoustically coupled to the user's right ear 206 through the right speaker port 136 when the headgear 102 is in use.
The auxiliary unit 106 includes a processor 516 (or alternatively multiple processors) that is coupled to and receives data from the headgear's 6 DOF headgear subsystem 504 , the IMU 508 , the SLAM/ visual odometry block 506 , and the hand gesture tracker 510 . The processor 516 can take the form of electronic circuitry. The processor 516 is also coupled to and can send control signals to the 6 DOF totem system 502 . The processor 516 may be coupled to the 6 DOF totem system wirelessly, as the handheld controller 104 may be untethered. The processor 516 can receive 6 DOF information regarding the relative orientation of the handheld controller 104 to the headgear 102 from the 6 DOF totem subsystem 502 or the 6 DOF headgear subsystem 504 .
Within the auxiliary unit 106 the processor 516 is coupled to an audio visual content memory 518 , a Graphical Processing Unit (GPU) scene render 520 and a Digital Signal Processor (DSP) audio spatializer 522 . The audio visual content memory 518 , the Graphical Processing Unit (GPU) scene render 520 and the Digital Signal Processor (DSP) audio spatializer 522 may take the form of electronic circuitry. The DSP audio spatializer 522 is coupled to a Head Related Transfer Function (HRTF) memory 524 . The GPU scene renderer 520 includes a left channel output coupled to the left source of imagewise modulated light 124 and a right channel output coupled to the right source of imagewise modulated light 126 . GPU scene renderer 520 outputs stereoscopic image data to the sources of imagewise modulated light 124 , 126 . The DSP audio spatializer 522 is coupled to and outputs left and right channel audio to, respectively, the left speaker 512 and the right speaker 514 . The audio spatializer 522 is responsive to information received from the processor 516 as to the direction from the user of the system 100 to sound emitting virtual content, the movement of which is being controlled by the user via the handheld controller 104 or with the user's hand. Based on the
CLAIMS
Claims ( 19 )
What is claimed is:
1. A system comprising: a handheld controller; a wearable head device including: a left eye display comprising a left eyepiece; a right eye display comprising a right eyepiece; a left side speaker; and a right side speaker; first tracking components; and circuitry coupled to the left eye display, the right eye display, the left side speaker, the right side speaker, and the first tracking components, wherein the circuitry is configured to perform a method comprising: identifying, via the first tracking components, a position of the handheld controller relative to the wearable head device; detecting, via the first tracking components, one or more surfaces of a nearby environment; and outputting spatialized audio that includes a left channel audio signal that is output to the left side speaker and a right channel audio signal that is output to the right side speaker, the spatialized audio associated with a virtual sound source location that is substantially fixed relative to the handheld controller, the virtual sound source located a first distance from a first surface of the one or more detected surfaces, wherein: the method further comprises displacing a virtual sound source image from the virtual sound source by an amount equal to twice the first distance, and one or more of the left channel audio signal and the right channel audio signal is based on the virtual sound source image that simulates an acoustic reflection off of the first surface of the one or more detected surfaces.
2. The system of claim 1 , wherein detecting one or more surfaces of a nearby environment comprises determining a distance between the wearable head device and the one or more surfaces.
3. The system of claim 1 , wherein detecting one or more surfaces of a nearby environment comprises determining a surface property associated with each of the detected one or more surfaces.
4. The system of claim 3 , wherein each surface property associated with the one or more detected surfaces is associated with a different acoustic property.
5. The system of claim 1 , wherein the virtual sound source image is located behind the first surface relative to the virtual sound source.
6. The system of claim 1 , wherein one or more of the left channel audio signal and the right channel audio signal is based on a second virtual sound source image that simulates an acoustic reflection of the first virtual sound source image off of a second surface of the one or more detected surfaces.
7. The system of claim 6 , wherein the second virtual sound source image is located behind the second surface relative to the virtual sound source.
8. The system of claim 1 , wherein the first tracking components comprise one or more depth cameras.
9. The system of claim 8 , wherein the one or more depth cameras are mounted to the wearable head device.
10. The system of claim 1 , wherein the virtual sound source location is disposed a pre-determined distance from the handheld controller.
11. The system of claim 1 , wherein the circuitry is further configured to receive, via the first tracking components, one or more signals indicative of an orientation of the handheld controller relative to the wearable head device.
12. The system of claim 1 , wherein the circuitry is further configured to generate the left channel audio signal and the right channel audio signal by applying at least one head related transfer function to a mono audio signal.
13. The system of claim 1 , further comprising:
second tracking components coupled to the circuitry,
wherein the circuitry is further configured to receive, via the second tracking components, one or more signals indicative of a position and/or an orientation of the wearable head device relative to a real world environment.
14. A system comprising: a wearable head device including: a left eye display comprising an eyepiece; a right eye display comprising an eyepiece; a left side speaker; and a right side speaker; a hand tracking machine vision subsystem; and circuitry coupled to the left eye display, the right eye display, the left side speaker, the right side speaker, and the hand tracking machine vision subsystem, wherein the circuitry is configured to perform a method comprising: identifying, via the hand tracking machine vision subsystem, a position of a hand of a user relative to the wearable head device; outputting spatialized audio that includes a left channel audio signal that is output to the left side speaker and a right channel audio signal that is output to the right side speaker, the spatialized audio associated with a virtual sound source location that is substantially fixed relative to the hand of the user, the virtual sound source located a first distance from a first surface of the one or more detected surfaces, wherein: the method further comprises displacing a virtual sound source image from the virtual sound source by an amount equal to twice the first distance, and one or more of the left channel audio signal and the right channel audio signal is based on the virtual sound source image that simulates an acoustic reflection off of the first surface of the one or more detected surfaces.
15. The system of claim 14 , wherein the circuitry is further configured to identify, via the hand tracking machine vision subsystem, a gesture of the hand and further configured to adjust at least one kinematic parameter of the virtual object in response to identifying the gesture.
16. The system of claim 14 , wherein the circuitry is configured to generate the left channel audio signal and the right channel audio signal by applying at least one head related transfer function to a mono audio signal.
17. A system comprising: a wearable head device including: a left eye display comprising an eyepiece; a right eye display comprising an eyepiece; a left side speaker; and a right side speaker; one or more receivers; and circuitry coupled to the left eye display, the right eye display, the left side speaker, the right side speaker, and the one or more receivers, wherein the circuitry is configured to perform a method comprising: determining a location of a virtual object relative to the wearable head device; outputting spatialized audio that includes a left channel audio signal that is output to the left side speaker and a right channel audio signal that is output to the right side speaker, the spatialized audio associated with a virtual sound source location that is substantially fixed relative to the virtual object, the virtual sound source located a first distance from a first surface of the one or more detected surfaces; and receiving user input through the one or more receivers and in response to receiving the user input through the one or more receivers, change a location of the virtual object and change the at least one virtual sound source location, and wherein: the method further comprises displacing a virtual sound source image from the virtual sound source by an amount equal to twice the first distance, and one or more of the left channel audio signal and the right channel audio signal is based on the virtual sound source image that simulates an acoustic reflection off of the first surface of the one or more detected surfaces.
18. The system of claim 17 , wherein the one or more receivers comprise one or more of a hand tracking machine vision subsystem and a handheld controller.
19. The system of claim 17 , wherein the circuitry is configured to generate the left channel audio signal and the right channel audio signal by applying at least one head related transfer function to a mono audio signal.
US16/920,321
2017-03-28
2020-07-02
Augmented reality system with spatialized audio tied to user manipulated virtual object
Active
US11231770B2
( en )
Priority Applications (1)
Application Number
Priority Date
Filing Date
Title
US16/920,321
US11231770B2
( en )
2017-03-28
2020-07-02
Augmented reality system with spatialized audio tied to user manipulated virtual object
Applications Claiming Priority (3)
Application Number
Priority Date
Filing Date
Title
US201762477976P
2017-03-28
2017-03-28
US15/937,721
US10747301B2
( en )
2017-03-28
2018-03-27
Augmented reality system with spatialized audio tied to user manipulated virtual object
US16/920,321
US11231770B2
( en )
2017-03-28
2020-07-02
Augmented reality system with spatialized audio tied to user manipulated virtual object
Related Parent Applications (1)
Application Number
Title
Priority Date
Filing Date
US15/937,721
Continuation
US10747301B2
( en )
2017-03-28
2018-03-27
Augmented reality system with spatialized audio tied to user manipulated virtual object
Publications (2)
Publication Number
Publication Date
US20200333876A1
US20200333876A1 ( en )
2020-10-22
US11231770B2
true
US11231770B2 ( en )
2022-01-25
Family
ID=63670336
Family Applications (2)
Application Number
Title
Priority Date
Filing Date
US15/937,721
Active
2038-06-06
US10747301B2
( en )
2017-03-28
2018-03-27
Augmented reality system with spatialized audio tied to user manipulated virtual object
US16/920,321
Active
US11231770B2
( en )
2017-03-28
2020-07-02
Augmented reality system with spatialized audio tied to user manipulated virtual object
Family Applications Before (1)
Application Number
Title
Priority Date
Filing Date
US15/937,721
Active
2038-06-06
US10747301B2
( en )
2017-03-28
2018-03-27
Augmented reality system with spatialized audio tied to user manipulated virtual object
Country Status (9)
Country
Link
US
( 2 )
US10747301B2
( en )
EP
( 1 )
EP3602252A4
( en )
JP
( 3 )
JP7175281B2
( en )
KR
( 1 )
KR102716952B1
( en )
CN
( 1 )
CN110476139B
( en )
AU
( 2 )
AU2018244316B2
( en )
CA
( 1 )
CA3058171A1
( en )
IL
( 1 )
IL269533B2
( en )
WO
( 1 )
WO2018183390A1
( en )
Cited By (3)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US20220391169A1
( en )
*
2018-09-26
2022-12-08
Apple Inc.
Spatial management of audio
US11809784B2
( en )
2018-09-28
2023-11-07
Apple Inc.
Audio assisted enrollment
US12340631B2
( en )
2022-06-05
2025-06-24
Apple Inc.
Providing personalized audio
Families Citing this family (48)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
ES2992065T3
( en )
2016-08-16
2024-12-09
Insight Medical Systems Inc
Sensory augmentation systems in medical procedures
EP3667464B1
( en )
*
2016-11-11
2024-02-14
Telefonaktiebolaget LM Ericsson (publ)
Supporting an augmented-reality software application
US10747301B2
( en )
2017-03-28
2020-08-18
Magic Leap, Inc.
Augmented reality system with spatialized audio tied to user manipulated virtual object
US10429949B2
( en )
*
2017-06-30
2019-10-01
Htc Corporation
User interaction apparatus and method
GB2567012B
( en )
*
2017-10-02
2021-05-12
Advanced Risc Mach Ltd
Motion Sensing
KR102122600B1
( en )
2018-03-07
2020-06-12
ë§¤ì§ ë¦½, ì¸ì½í¬ë ì´í°ë
Visual tracking of peripheral devices
US11617050B2
( en )
2018-04-04
2023-03-28
Bose Corporation
Systems and methods for sound source virtualization
US11875012B2
( en )
*
2018-05-25
2024-01-16
Ultrahaptics IP Two Limited
Throwable interface for augmented reality and virtual reality environments
EP3861768A4
( en )
*
2018-10-05
2021-12-08
Magic Leap, Inc.
INTERAURAL TIME DIFFERENCE CROSSFADER FOR BINAURAL AUDIO RESTITUTION
CN116320907B
( en )
*
2018-10-05
2026-02-06
å¥è·å ¬å¸
Near field audio rendering
US11047691B2
( en )
*
2018-10-31
2021-06-29
Dell Products, L.P.
Simultaneous localization and mapping (SLAM) compensation for gesture recognition in virtual, augmented, and mixed reality (xR) applications
US11287526B2
( en )
*
2018-11-21
2022-03-29
Microsoft Technology Licensing, Llc
Locating spatialized sounds nodes for echolocation using unsupervised machine learning
WO2020139588A1
( en )
2018-12-24
2020-07-02
Dts, Inc.
Room acoustics simulation using deep learning image analysis
US12086221B2
( en )
*
2019-02-01
2024-09-10
Sony Group Corporation
Multi-factor authentication for virtual reality
CN111949112A
( en )
*
2019-05-14
2020-11-17
Oppo广ä¸ç§»å¨éä¿¡æéå ¬å¸
Object interaction method and apparatus, system, computer readable medium and electronic device
US11486961B2
( en )
*
2019-06-14
2022-11-01
Chirp Microsystems
Object-localization and tracking using ultrasonic pulses with reflection rejection
US11030459B2
( en )
*
2019-06-27
2021-06-08
Intel Corporation
Methods and apparatus for projecting augmented reality enhancements to real objects in response to user gestures detected in a real environment
US11774757B2
( en )
*
2019-07-22
2023-10-03
Apple Inc.
Optical module for head-mountable device
US11448883B2
( en )
2019-09-27
2022-09-20
Microsoft Technology Licensing, Llc
Non-linear angular momentum metasurface
US11467406B2
( en )
*
2019-09-27
2022-10-11
Microsoft Technology Licensing, Llc
Field of view expanding system
CN115053270B
( en )
*
2019-12-09
2026-01-09
å¥è·å ¬å¸
Systems and methods for operating a head-mounted display system based on user identity.
EP3839699A1
( en )
*
2019-12-19
2021-06-23
Koninklijke KPN N.V.
Augmented virtuality self view
US11205296B2
( en )
*
2019-12-20
2021-12-21
Sap Se
3D data exploration using interactive cuboids
USD959476S1
( en )
2019-12-20
2022-08-02
Sap Se
Display system or portion thereof with a virtual three-dimensional animated graphical user interface
USD959447S1
( en )
2019-12-20
2022-08-02
Sap Se
Display system or portion thereof with a virtual three-dimensional animated graphical user interface
USD959477S1
( en )
2019-12-20
2022-08-02
Sap Se
Display system or portion thereof with a virtual three-dimensional animated graphical user interface
KR102318698B1
( en )
*
2019-12-27
2021-10-28
주ìíì¬ ë¯¹ì
Method and program for creating virtual space where virtual objects are arranged based on spherical coordinate system
WO2021163373A1
( en )
*
2020-02-14
2021-08-19
Magic Leap, Inc.
3d object annotation
WO2021178454A1
( en )
*
2020-03-02
2021-09-10
Magic Leap, Inc.
Immersive audio platform
KR102334091B1
( en )
*
2020-03-20
2021-12-02
주ìíì¬ ì½í´ë¦¬ì´ë·ìì´ìì´
Augmented reality device for audio identification and control method thereof
JP7560568B2
( en )
*
2020-03-24
2024-10-02
ãã¸ã㯠ãªã¼ãï¼ ã¤ã³ã³ã¼ãã¬ã¤ããã
Systems and methods for virtual and augmented reality
KR20210123198A
( en )
2020-04-02
2021-10-13
주ìíì¬ ì ì´ë
Argumented reality based simulation apparatus for integrated electrical and architectural acoustics
KR102199591B1
( en )
*
2020-04-02
2021-01-07
주ìíì¬ ì ì´ë
Argumented reality based simulation apparatus for integrated electrical and architectural acoustics
US11356795B2
( en )
*
2020-06-17
2022-06-07
Bose Corporation
Spatialized audio relative to a peripheral device
US11712797B2
( en )
*
2020-09-11
2023-08-01
Fanuc Corporation
Dual hand detection in teaching from demonstration
US12179350B2
( en )
*
2020-09-11
2024-12-31
Fanuc Corporation
Dual arm robot teaching from dual hand human demonstration
CN112181138B
( en )
*
2020-09-16
2022-05-03
éå²å°é¸ççç§ææéå ¬å¸
Adaptive intelligent head-hand VR system and method
CN112367426B
( en )
*
2020-11-09
2021-06-04
Oppo广ä¸ç§»å¨éä¿¡æéå ¬å¸
Virtual object display method and device, storage medium and electronic equipment
KR20220077452A
( en )
2020-12-02
2022-06-09
íë겸
Group experience type augmented reality providing system using vision tracking technology
US11496852B2
( en )
*
2020-12-03
2022-11-08
Snap Inc.
Head-related transfer function
KR20230112729A
( en )
*
2020-12-07
2023-07-27
ì¤ë ì¸ì½í¬ë ì´í°ë
Augmented reality spatial audio experience
US11836301B2
( en )
*
2021-08-10
2023-12-05
Qualcomm Incorporated
Electronic device for tracking objects
US11956409B2
( en )
*
2021-08-23
2024-04-09
Tencent America LLC
Immersive media interoperability
WO2023025695A1
( en )
*
2021-08-23
2023-03-02
Analog Devices International Unlimited Company
Method of calculating an audio calibration profile
US12591406B2
( en )
*
2023-02-02
2026-03-31
Universal City Studios Llc
System and method for generating interactive media
US20250036344A1
( en )
*
2023-07-24
2025-01-30
Google Llc
Wearable display with coupled mobile interface module
US20250039626A1
( en )
*
2023-07-26
2025-01-30
Ambit Microsystems (Shanghai) Ltd.
Method and system for simulating spatial sound in augmented reality and computer-readable storage medium
WO2026014659A1
( en )
*
2024-07-12
2026-01-15
주ìíì¬ ê¸±ì¤ë¡íí¸
Wearable device for providing sound setting, and operating method thereof
Citations (78)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US20010055397A1
( en )
*
1996-07-17
2001-12-27
American Technology Corporation
Parametric virtual speaker and surround-sound system
US6349001B1
( en )
*
1997-10-30
2002-02-19
The Microoptical Corporation
Eyeglass interface system
US6757068B2
( en )
*
2000-01-28
2004-06-29
Intersense, Inc.
Self-referenced tracking
US20050174470A1
( en )
*
2004-02-06
2005-08-11
Olympus Corporation
Head-mounted camera
US20050195277A1
( en )
*
2004-03-04
2005-09-08
Olympus Corporation
Image capturing apparatus
US20050248852A1
( en )
*
2004-05-06
2005-11-10
Olympus Corporation
Head-mounted display apparatus
US20070029899A1
( en )
*
2005-08-03
2007-02-08
Seiko Epson Corporation
Electrostatic ultrasonic transducer, ultrasonic speaker, audio signal reproduction method, electrode manufacturing method for use in ultrasonic transducer, ultrasonic transducer manufacturing method, superdirective acoustic system, and display device
US20070291949A1
( en )
*
2006-06-14
2007-12-20
Matsushita Electric Industrial Co., Ltd.
Sound image control apparatus and sound image control method
US20080297437A1
( en )
*
2007-05-31
2008-12-04
Canon Kabushiki Kaisha
Head mounted display and control method therefor
US20090310802A1
( en )
2008-06-17
2009-12-17
Microsoft Corporation
Virtual sound source positioning
US20100020229A1
( en )
*
2007-04-30
2010-01-28
General Electric Company
Wearable personal video/audio device method and system
US20100033427A1
( en )
*
2002-07-27
2010-02-11
Sony Computer Entertainment Inc.
Computer Image and Audio Processing of Intensity and Input Devices for Interfacing with a Computer Program
US20100079356A1
( en )
*
2008-09-30
2010-04-01
Apple Inc.
Head-mounted display apparatus for retaining a portable electronic device with display
US20110194029A1
( en )
*
2010-02-05
2011-08-11
Kopin Corporation
Touch sensor for controlling eyewear
US20110221656A1
( en )
*
2010-02-28
2011-09-15
Osterhout Group, Inc.
Displayed content vision correction with electrically adjustable lens
US20120242560A1
( en )
*
2011-03-24
2012-09-27
Seiko Epson Corporation
Head-mounted display device and control method for the head-mounted display device
US20120314871A1
( en )
*
2011-06-13
2012-12-13
Yasuyuki Koga
Information processing apparatus, information processing method, and program
US20130083018A1
( en )
*
2011-09-30
2013-04-04
Kevin A. Geisner
Personal audio/visual system with holographic objects
US20130182064A1
( en )
*
2012-01-18
2013-07-18
Harman Becker Automotive Systems Gmbh
Method for operating a conference system and device for a conference system
US20130217488A1
( en )
*
2012-02-21
2013-08-22
Radu Mircea COMSA
Augmented reality system
US20130236040A1
( en )
*
2012-03-08
2013-09-12
Disney Enterprises, Inc.
Augmented reality (ar) audio with position and action triggered virtual sound effects
US20130328762A1
( en )
*
2012-06-12
2013-12-12
Daniel J. McCulloch
Controlling a virtual object with a real controller device
US20130335573A1
( en )
*
2012-06-15
2013-12-19
Qualcomm Incorporated
Input method designed for augmented reality goggles
US20140006026A1
( en )
*
2012-06-29
2014-01-02
Mathew J. Lamb
Contextual audio ducking with situation aware devices
US20140328505A1
( en )
*
2013-05-02
2014-11-06
Microsoft Corporation
Sound field adaptation based upon user tracking
US20140362110A1
( en )
*
2013-06-08
2014-12-11
Sony Computer Entertainment Inc.
Systems and methods for customizing optical representation of views provided by a head mounted display based on optical prescription of a user
US20140375531A1
( en )
*
2013-06-24
2014-12-25
Ray Latypov
Method of roviding to the user an image from the screen of the smartphome or tablet at a wide angle of view, and a method of providing to the user 3d sound in virtual reality
US20150022444A1
( en )
*
2012-02-06
2015-01-22
Sony Corporation
Information processing apparatus, and information processing method
US20150067580A1
( en )
*
2013-09-02
2015-03-05
Lg Electronics Inc.
Wearable device and method of outputting content thereof
US20150138065A1
( en )
*
2013-11-21
2015-05-21
Nvidia Corporation
Head-mounted integrated interface
US20150205126A1
( en )
*
2013-11-27
2015-07-23
Magic Leap, Inc.
Virtual and augmented reality systems and methods
US20150235426A1
( en )
*
2014-02-18
2015-08-20
Merge Labs, Inc.
Remote control augmented motion data capture
WO2015123771A1
( en )
2014-02-18
2015-08-27
Sulon Technologies Inc.
Gesture tracking and control in augmented and virtual reality
US9163933B1
( en )
*
2014-03-28
2015-10-20
Rockwell Collins, Inc.
Wearable head tracking system
US20150301787A1
( en )
*
2014-04-18
2015-10-22
Magic Leap, Inc.
Systems and methods for generating sound wavefronts in augmented or virtual reality systems
US20150319530A1
( en )
2012-12-18
2015-11-05
Nokia Technologies Oy
Spatial Audio Apparatus
US9182594B2
( en )
*
2013-01-29
2015-11-10
Seiko Epson Corporation
Head-mounted image display apparatus
US20150346495A1
( en )
*
2014-05-30
2015-12-03
Magic Leap, Inc.
Methods and system for creating focal planes in virtual and augmented reality
US20150348328A1
( en )
*
2014-06-03
2015-12-03
Seiko Epson Corporation
Head-mounted display device, method of controlling head-mounted display device, information transmitting and receiving system, and computer program
US20150355811A1
( en )
*
2013-01-08
2015-12-10
Ayotle Sas
Methods and systems for controlling a virtual interactive surface and interactive display systems
US20150352437A1
( en )
*
2014-06-09
2015-12-10
Bandai Namco Games Inc.
Display control method for head mounted display (hmd) and image generation device
US20150382131A1
( en )
*
2014-06-26
2015-12-31
Audi Ag
Method for operating a virtual reality system and virtual reality system
US20150379349A1
( en )
2012-03-22
2015-12-31
Google Inc.
Staredown to Produce Changes in Information Density and Type
US20150382130A1
( en )
*
2014-06-27
2015-12-31
Patrick Connor
Camera based adjustments to 3d soundscapes
US20160026253A1
( en )
*
2014-03-11
2016-01-28
Magic Leap, Inc.
Methods and systems for creating virtual and augmented reality
US20160035140A1
( en )
*
2014-07-29
2016-02-04
Sony Computer Entertainment Europe Limited
Image processing
US20160085302A1
( en )
2014-05-09
2016-03-24
Eyefluence, Inc.
Systems and methods for biomechanically-based eye signals for interacting with real and virtual objects
US20160093105A1
( en )
*
2014-09-30
2016-03-31
Sony Computer Entertainment Inc.
Display of text information on a head-mounted display
US20160187970A1
( en )
*
2013-06-11
2016-06-30
Sony Computer Entertainment Europe Limited
Head-mountable apparatus and system
US20160206957A1
( en )
2015-01-20
2016-07-21
Disney Enterprises, Inc.
Tracking specific gestures relative to user movement
US20160212272A1
( en )
*
2015-01-21
2016-07-21
Sriram Srinivasan
Spatial Audio Signal Processing for Objects with Associated Audio Content
US20160227338A1
( en )
*
2015-01-30
2016-08-04
Gaudi Audio Lab, Inc.
Apparatus and a method for processing audio signal to perform binaural rendering
US20160320847A1
( en )
2014-01-03
2016-11-03
Harman International Industries, Incorporated
Gesture interactive wearable spatial audio system
US9557566B2
( en )
*
2012-03-07
2017-01-31
Seiko Epson Corporation
Head-mounted display device and control method for the head-mounted display device
US20170039774A1
( en )
*
2014-04-14
2017-02-09
Tremolant Inc.
Augmented Reality Communications
US20170076503A1
( en )
*
2015-09-16
2017-03-16
Bandai Namco Entertainment Inc.
Method for generating image to be displayed on head tracking type virtual reality head mounted display and image generation device
US20170078825A1
( en )
*
2015-09-16
2017-03-16
Magic Leap, Inc.
Head pose mixing of audio files
US9630105B2
( en )
*
2013-09-30
2017-04-25
Sony Interactive Entertainment Inc.
Camera based safety mechanisms for users of head mounted displays
US9774979B1
( en )
*
2016-03-03
2017-09-26
Google Inc.
Systems and methods for spatial audio adjustment
US20170357334A1
( en )
*
2016-06-09
2017-12-14
Alexandru Octavian Balan
Modular extension of inertial controller for six dof mixed reality input
<tr itemprop="backwardR