ABSTRACT
Abstract
An apparatus for providing a virtual or augmented reality experience, includes: a screen, wherein the screen is at least partially transparent for allowing a user of the apparatus to view an object in an environment surrounding the user; a surface detector configured to detect a surface of the object; an object identifier configured to obtain an orientation and/or an elevation of the surface of the object, and to make an identification for the object based on the orientation and/or the elevation of the surface of the object; and a graphic generator configured to generate an identifier indicating the identification for the object for display by the screen, wherein the screen is configured to display the identifier.
Description
RELATED APPLICATION DATA
The present application is a continuation of U.S. patent application Ser. No. 17/211,502 filed on Mar. 24, 2021, which claims priority to, and the benefit of, U.S. Provisional Patent Application No. 62/994,159, filed on Mar. 24, 2020. The entire disclosures of the above applications are expressly incorporated by reference herein.
FIELD
The present disclosure relates to connected mobile computing systems, methods, and configurations, and more specifically to mobile computing systems, methods, and configurations featuring at least one wearable component which may be utilized for virtual and/or augmented reality operation.
BACKGROUND
Modern computing and display technologies have facilitated the development of âmixed realityâ (MR) systems for so called âvirtual realityâ (VR) or âaugmented realityâ (AR) experiences, wherein digitally reproduced images or portions thereof are presented to a user in a manner wherein they seem to be, or may be perceived as, real. A VR scenario typically involves presentation of digital or virtual image information without transparency to actual real-world visual input. An AR scenario typically involves presentation of digital or virtual image information as an augmentation to visualization of the real world around the user (i.e., transparency to real-world visual input). Accordingly, AR scenarios involve presentation of digital or virtual image information with transparency to the real-world visual input.
MR systems may generate and display color data, which increases the realism of MR scenarios. Many of these MR systems display color data by sequentially projecting sub-images in different (e.g., primary) colors or âfieldsâ (e.g., Red, Green, and Blue) corresponding to a color image in rapid succession. Projecting color sub-images at sufficiently high rates (e.g., 60 Hz, 120 Hz, etc.) may deliver a smooth color MR scenario in a user's mind.
Various optical systems generate images, including color images, at various depths for displaying MR (VR and AR) scenarios. Some such optical systems are described in U.S. Utility patent application Ser. No. 14/555,585 filed on Nov. 27, 2014, the contents of which are hereby expressly and fully incorporated by reference in their entirety, as though set forth in full.
MR systems may employ wearable display devices (e.g., head-worn displays, helmet-mounted displays, or smart glasses) that are at least loosely coupled to a user's head, and thus move when the user's head moves. If the user's head motions are detected by the display device, the data being displayed can be updated (e.g., âwarpedâ) to take the change in head pose (i.e., the orientation and/or location of user's head) into account.
As an example, if a user wearing a head-worn display device views a virtual representation of a virtual object on the display and walks around an area where the virtual object appears, the virtual object can be rendered for each viewpoint, giving the user the perception that they are walking around an object that occupies real space. If the head-worn display device is used to present multiple virtual objects, measurements of head pose can be used to render the scene to match the user's dynamically changing head pose and provide an increased sense of immersion.
Head-worn display devices that enable AR provide concurrent viewing of both real and virtual objects. With an âoptical see-throughâ display, a user can see through transparent (e.g., semi-transparent or full-transparent) elements in a display system to view directly the light from real objects in an environment. The transparent element, often referred to as a âcombiner,â superimposes light from the display over the user's view of the real world, where light from by the display projects an image of virtual content over the see-through view of the real objects in the environment. A camera may be mounted onto the head-worn display device to capture images or videos of the scene being viewed by the user.
Current optical systems, such as those in MR systems, optically render virtual content. Content is âvirtualâ in that it does not correspond to real physical objects located in respective positions in space. Instead, virtual content only exist in the brains (e.g., the optical centers) of a user of the head-worn display device when stimulated by light beams directed to the eyes of the user.
In some cases, a head-worn image display device may display virtual objects with respect to a real environment, and/or may allow a user to place and/or manipulate virtual objects with respect to the real environment. In such cases, the image display device may be configured to localize the user with respect to the real environment, so that virtual objects may be correctly displaced with respect to the real environment.
It is desirable that mixed reality, or augmented reality, near-eye displays be lightweight, low-cost, have a small form-factor, have a wide virtual image field of view, and be as transparent as possible. In addition, it is desirable to have configurations that present virtual image information in multiple focal planes (for example, two or more) in order to be practical for a wide variety of use-cases without exceeding an acceptable allowance for vergence-accommodation mismatch.
SUMMARY
Methods and apparatuses for providing virtual content, such as virtual object, for display by one or more screens of one or more image display devices (worn by one or more users) are described herein. In some embodiments, the virtual content may be displayed so that it appears to be in a physical environment as viewed by a user through the screen. The virtual content may be provided for interaction by users in the same physical environment, or by users in different environments (e.g., in different rooms). New techniques for identifying objects in the environment, for defining a virtual space to place virtual items, and for inserting virtual items for interaction by one or more users, are described herein.
An apparatus for providing a virtual or augmented reality experience, includes: a screen, wherein the screen is at least partially transparent for allowing a user of the apparatus to view an object in an environment surrounding the user; a surface detector configured to detect a surface of the object; an object identifier configured to obtain an orientation and/or an elevation of the surface of the object, and to make an identification for the object based on the orientation and/or the elevation of the surface of the object; and a graphic generator configured to generate an identifier indicating the identification for the object for display by the screen, wherein the screen is configured to display the identifier.
Optionally, the object identifier is configured to identify the object as a wall if the orientation of the surface of the object is substantially vertical, and if the elevation of the surface of the object is above an elevation threshold.
Optionally, the object identifier is configured to identify the object as a floor if the orientation of the surface of the object is substantially horizontal, and if the elevation of the surface of the object is below an elevation threshold.
Optionally, the object is configured to identify the object as furniture if the orientation of the surface of the object is substantially horizontal, and if the elevation of the surface of the object is above a first elevation threshold and is below a second elevation threshold.
Optionally, the screen is configured to display the identifier at a location in the screen, such that when the user views the object in the environment through the screen, the identifier will be in a spatial relationship with respect to the object.
Optionally, the object identifier is configured to obtain an input indicating a selection of the object for which the identification of the object is to be determined.
Optionally, the input comprises a user input generated via a controller component, the user input indicating the selection of the object.
Optionally, the user input indicates a cursor position in the screen, and wherein the object identifier is configured to determine the object in the environment being selected based on the cursor position.
Optionally, the user input indicates an orientation of the controller component, and wherein the object identifier is configured to determine the object in the environment being selected based on a direction of pointing by the controller component towards the object in the environment.
Optionally, the apparatus further includes a camera, wherein the object identifier is configured to select the object for identification based on a presence of an image of the object in a camera image provided by the camera.
Optionally, the object identifier is configured to select the object automatically.
Optionally, the object identifier is configured to select the object in response to the object being presence in a sequence of camera images that comprise the camera image within a duration exceeding a time threshold.
Optionally, the apparatus further includes a space definer configured to define a virtual space.
Optionally, the space definer is configured to define a virtual wall for the virtual space.
Optionally, the virtual wall is offset from a real physical wall in the environment surrounding the user.
Optionally, the virtual wall is aligned with, or intersects, a real physical wall in the environment surrounding the user.
Optionally, the screen is configured to display a wall identifier at a location in the screen, such that when the user views the virtual wall, the wall identifier will be in a spatial relationship with respect to the virtual wall.
Optionally, the space definer is configured to define a plurality of virtual walls for the virtual space; and wherein the screen is configured to display wall identifiers for the respective virtual walls.
Optionally, the space definer is configured to define a corner for the virtual space.
Optionally, the space definer is configured to define a wall edge for the virtual space.
Optionally, the space definer is configured to obtain a user input generated via a controller component, the user input indicating a selection of a feature in the environment for defining at least a part of the virtual space.
Optionally, the feature in the environment comprises a wall, a wall corner, an edge, or any combination of the foregoing.
Optionally, the user input indicates a cursor position in the screen.
Optionally, the user input indicates an orientation of the controller component, and wherein the selection of the feature in the environment is based on a direction of pointing by the controller component towards the feature in the environment.
Optionally, the apparatus further includes a camera, wherein the apparatus is configured to select a feature in the environment, for defining at least a part of the virtual space, based on a presence of an image of the feature in a camera image provided by the camera.
Optionally, the apparatus is configured to select the feature in the environment automatically.
Optionally, the apparatus is configured to select the feature in response to the feature being presence in a sequence of camera images that includes the camera image within a duration exceeding a time threshold.
Optionally, the screen is configured to display a pointer based on an orientation of a controller component, and wherein the apparatus further comprises a pointer configurator, wherein the pointer configurator is configured to adjust a configuration of the pointer displayed in the screen, and/or to present an interaction graphic in association with the pointer.
Optionally, the pointer configurator is configured to adjust the configuration of the pointer and/or to present the interaction graphic in response to an interaction between the pointer and a virtual object.
Optionally, the graphic generator is also configured to provide a virtual content for interaction by the user and an additional user, and wherein the screen is configured to display the virtual content.
Optionally, the apparatus is configured to connect the user and the additional user to a same virtual space so that the user and the additional user can interact with the virtual content at the same virtual space.
Optionally, the graphic generator is configured to provide the virtual content for interaction by the user and the additional user in different respective rooms.
Optionally, the graphic generator is configured to provide the virtual content for interaction by the user and the additional user in a same room.
Optionally, the apparatus is configured to register the user and the additional user to the environment so that the user and the additional user can interact with the virtual content while they are in the environment.
Optionally, the apparatus is configured to register the user and the additional user to the environment by determining one or more anchor points that are associated with a location of the user and a location of the additional user.
A method performed by an apparatus that is configured to provide a virtual or augmented reality experience, includes: detecting, by a surface detector, a surface of an object in an environment surrounding a user of the apparatus; obtaining, by
RELATED APPLICATION DATA
The present application is a continuation of U.S. patent application Ser. No. 17/211,502 filed on Mar. 24, 2021, which claims priority to, and the benefit of, U.S. Provisional Patent Application No. 62/994,159, filed on Mar. 24, 2020. The entire disclosures of the above applications are expressly incorporated by reference herein.
FIELD
The present disclosure relates to connected mobile computing systems, methods, and configurations, and more specifically to mobile computing systems, methods, and configurations featuring at least one wearable component which may be utilized for virtual and/or augmented reality operation.
BACKGROUND
Modern computing and display technologies have facilitated the development of âmixed realityâ (MR) systems for so called âvirtual realityâ (VR) or âaugmented realityâ (AR) experiences, wherein digitally reproduced images or portions thereof are presented to a user in a manner wherein they seem to be, or may be perceived as, real. A VR scenario typically involves presentation of digital or virtual image information without transparency to actual real-world visual input. An AR scenario typically involves presentation of digital or virtual image information as an augmentation to visualization of the real world around the user (i.e., transparency to real-world visual input). Accordingly, AR scenarios involve presentation of digital or virtual image information with transparency to the real-world visual input.
MR systems may generate and display color data, which increases the realism of MR scenarios. Many of these MR systems display color data by sequentially projecting sub-images in different (e.g., primary) colors or âfieldsâ (e.g., Red, Green, and Blue) corresponding to a color image in rapid succession. Projecting color sub-images at sufficiently high rates (e.g., 60 Hz, 120 Hz, etc.) may deliver a smooth color MR scenario in a user's mind.
Various optical systems generate images, including color images, at various depths for displaying MR (VR and AR) scenarios. Some such optical systems are described in U.S. Utility patent application Ser. No. 14/555,585 filed on Nov. 27, 2014, the contents of which are hereby expressly and fully incorporated by reference in their entirety, as though set forth in full.
MR systems may employ wearable display devices (e.g., head-worn displays, helmet-mounted displays, or smart glasses) that are at least loosely coupled to a user's head, and thus move when the user's head moves. If the user's head motions are detected by the display device, the data being displayed can be updated (e.g., âwarpedâ) to take the change in head pose (i.e., the orientation and/or location of user's head) into account.
As an example, if a user wearing a head-worn display device views a virtual representation of a virtual object on the display and walks around an area where the virtual object appears, the virtual object can be rendered for each viewpoint, giving the user the perception that they are walking around an object that occupies real space. If the head-worn display device is used to present multiple virtual objects, measurements of head pose can be used to render the scene to match the user's dynamically changing head pose and provide an increased sense of immersion.
Head-worn display devices that enable AR provide concurrent viewing of both real and virtual objects. With an âoptical see-throughâ display, a user can see through transparent (e.g., semi-transparent or full-transparent) elements in a display system to view directly the light from real objects in an environment. The transparent element, often referred to as a âcombiner,â superimposes light from the display over the user's view of the real world, where light from by the display projects an image of virtual content over the see-through view of the real objects in the environment. A camera may be mounted onto the head-worn display device to capture images or videos of the scene being viewed by the user.
Current optical systems, such as those in MR systems, optically render virtual content. Content is âvirtualâ in that it does not correspond to real physical objects located in respective positions in space. Instead, virtual content only exist in the brains (e.g., the optical centers) of a user of the head-worn display device when stimulated by light beams directed to the eyes of the user.
In some cases, a head-worn image display device may display virtual objects with respect to a real environment, and/or may allow a user to place and/or manipulate virtual objects with respect to the real environment. In such cases, the image display device may be configured to localize the user with respect to the real environment, so that virtual objects may be correctly displaced with respect to the real environment.
It is desirable that mixed reality, or augmented reality, near-eye displays be lightweight, low-cost, have a small form-factor, have a wide virtual image field of view, and be as transparent as possible. In addition, it is desirable to have configurations that present virtual image information in multiple focal planes (for example, two or more) in order to be practical for a wide variety of use-cases without exceeding an acceptable allowance for vergence-accommodation mismatch.
SUMMARY
Methods and apparatuses for providing virtual content, such as virtual object, for display by one or more screens of one or more image display devices (worn by one or more users) are described herein. In some embodiments, the virtual content may be displayed so that it appears to be in a physical environment as viewed by a user through the screen. The virtual content may be provided for interaction by users in the same physical environment, or by users in different environments (e.g., in different rooms). New techniques for identifying objects in the environment, for defining a virtual space to place virtual items, and for inserting virtual items for interaction by one or more users, are described herein.
An apparatus for providing a virtual or augmented reality experience, includes: a screen, wherein the screen is at least partially transparent for allowing a user of the apparatus to view an object in an environment surrounding the user; a surface detector configured to detect a surface of the object; an object identifier configured to obtain an orientation and/or an elevation of the surface of the object, and to make an identification for the object based on the orientation and/or the elevation of the surface of the object; and a graphic generator configured to generate an identifier indicating the identification for the object for display by the screen, wherein the screen is configured to display the identifier.
Optionally, the object identifier is configured to identify the object as a wall if the orientation of the surface of the object is substantially vertical, and if the elevation of the surface of the object is above an elevation threshold.
Optionally, the object identifier is configured to identify the object as a floor if the orientation of the surface of the object is substantially horizontal, and if the elevation of the surface of the object is below an elevation threshold.
Optionally, the object is configured to identify the object as furniture if the orientation of the surface of the object is substantially horizontal, and if the elevation of the surface of the object is above a first elevation threshold and is below a second elevation threshold.
Optionally, the screen is configured to display the identifier at a location in the screen, such that when the user views the object in the environment through the screen, the identifier will be in a spatial relationship with respect to the object.
Optionally, the object identifier is configured to obtain an input indicating a selection of the object for which the identification of the object is to be determined.
Optionally, the input comprises a user input generated via a controller component, the user input indicating the selection of the object.
Optionally, the user input indicates a cursor position in the screen, and wherein the object identifier is configured to determine the object in the environment being selected based on the cursor position.
Optionally, the user input indicates an orientation of the controller component, and wherein the object identifier is configured to determine the object in the environment being selected based on a direction of pointing by the controller component towards the object in the environment.
Optionally, the apparatus further includes a camera, wherein the object identifier is configured to select the object for identification based on a presence of an image of the object in a camera image provided by the camera.
Optionally, the object identifier is configured to select the object automatically.
Optionally, the object identifier is configured to select the object in response to the object being presence in a sequence of camera images that comprise the camera image within a duration exceeding a time threshold.
Optionally, the apparatus further includes a space definer configured to define a virtual space.
Optionally, the space definer is configured to define a virtual wall for the virtual space.
Optionally, the virtual wall is offset from a real physical wall in the environment surrounding the user.
Optionally, the virtual wall is aligned with, or intersects, a real physical wall in the environment surrounding the user.
Optionally, the screen is configured to display a wall identifier at a location in the screen, such that when the user views the virtual wall, the wall identifier will be in a spatial relationship with respect to the virtual wall.
Optionally, the space definer is configured to define a plurality of virtual walls for the virtual space; and wherein the screen is configured to display wall identifiers for the respective virtual walls.
Optionally, the space definer is configured to define a corner for the virtual space.
Optionally, the space definer is configured to define a wall edge for the virtual space.
Optionally, the space definer is configured to obtain a user input generated via a controller component, the user input indicating a selection of a feature in the environment for defining at least a part of the virtual space.
Optionally, the feature in the environment comprises a wall, a wall corner, an edge, or any combination of the foregoing.
Optionally, the user input indicates a cursor position in the screen.
Optionally, the user input indicates an orientation of the controller component, and wherein the selection of the feature in the environment is based on a direction of pointing by the controller component towards the feature in the environment.
Optionally, the apparatus further includes a camera, wherein the apparatus is configured to select a feature in the environment, for defining at least a part of the virtual space, based on a presence of an image of the feature in a camera image provided by the camera.
Optionally, the apparatus is configured to select the feature in the environment automatically.
Optionally, the apparatus is configured to select the feature in response to the feature being presence in a sequence of camera images that includes the camera image within a duration exceeding a time threshold.
Optionally, the screen is configured to display a pointer based on an orientation of a controller component, and wherein the apparatus further comprises a pointer configurator, wherein the pointer configurator is configured to adjust a configuration of the pointer displayed in the screen, and/or to present an interaction graphic in association with the pointer.
Optionally, the pointer configurator is configured to adjust the configuration of the pointer and/or to present the interaction graphic in response to an interaction between the pointer and a virtual object.
Optionally, the graphic generator is also configured to provide a virtual content for interaction by the user and an additional user, and wherein the screen is configured to display the virtual content.
Optionally, the apparatus is configured to connect the user and the additional user to a same virtual space so that the user and the additional user can interact with the virtual content at the same virtual space.
Optionally, the graphic generator is configured to provide the virtual content for interaction by the user and the additional user in different respective rooms.
Optionally, the graphic generator is configured to provide the virtual content for interaction by the user and the additional user in a same room.
Optionally, the apparatus is configured to register the user and the additional user to the environment so that the user and the additional user can interact with the virtual content while they are in the environment.
Optionally, the apparatus is configured to register the user and the additional user to the environment by determining one or more anchor points that are associated with a location of the user and a location of the additional user.
A method performed by an apparatus that is configured to provide a virtual or augmented reality experience, includes: detecting, by a surface detector, a surface of an object in an environment surrounding a user of the apparatus; obtaining, by an object identifier, an orientation and/or an elevation of the surface of the object; identifying, by the object identifier, the object based on the orientation and/or the elevation of the surface of the object; and generating, by a graphic generator, an identifier for the identified object for display by a screen, wherein the screen is at least partially transparent for allowing the user of the apparatus to view the object in the environment surrounding the user.
Optionally, the object is identified as a wall if the orientation of the surface of the object is substantially vertical, and if the elevation of the surface of the object is above an elevation threshold.
Optionally, the object is identified as a floor if the orientation of the surface of the object is substantially horizontal, and if the elevation of the surface of the object is below an elevation threshold.
Optionally, the object is identified as furniture if the orientation of the surface of the object is substantially horizontal, and if the elevation of the surface of the object is above a first elevation threshold and is below a second elevation threshold.
Optionally, the identifier is displayed at a location in the screen, such that when the user views the object in the environment through the screen, the identifier will be in a spatial relationship with respect to the object.
Optionally, the method further includes obtaining an input indicating a selection of the object for which the identification of the object is to be determined.
Optionally, the input comprises a user input generated via a controller component, the user input indicating the selection of the object.
Optionally, the user input indicates a cursor position in the screen, and wherein the method further comprises determining the object in the environment being selected based on the cursor position.
Optionally, the user input indicates an orientation of the controller component, and wherein the method further comprises determining the object in the environment being selected based on a direction of pointing by the controller component towards the object in the environment.
Optionally, the method further includes selecting the object for identification based on a presence of an image of the object in a camera image provided by a camera of the apparatus.
Optionally, object is selected automatically for identification.
Optionally, the object is selected in response to the object being presence in a sequence of camera images that comprise the camera image within a duration exceeding a time threshold.
Optionally, the method further includes defining a virtual space.
Optionally, the act of defining the virtual space comprises defining a virtual wall for the virtual space.
Optionally, the virtual wall is offset from a real physical wall in the environment surrounding the user.
Optionally, the virtual wall is aligned with, or intersects, a real physical wall in the environment surrounding the user.
Optionally, the method further includes displaying, by the screen, a wall identifier at a location in the screen, such that when the user views the virtual wall, the wall identifier will be in a spatial relationship with respect to the virtual wall.
Optionally, the act of defining the virtual space comprises defining a plurality of virtual walls for the virtual space; and wherein the method further comprises displaying, by the screen, wall identifiers for the respective virtual walls.
Optionally, the act of defining the virtual space comprises defining a corner for the virtual space.
Optionally, the act of defining the virtual space comprises defining a wall edge for the virtual space.
Optionally, the method further includes obtaining a user input generated via a controller component, the user input indicating a selection of a feature in the environment for defining at least a part of the virtual space.
Optionally, the feature in the environment comprises a wall, a wall corner, an edge, or any combination of the foregoing.
Optionally, the user input indicates a cursor position in the screen.
Optionally, the user input indicates an orientation of the controller component, and wherein the selection of the feature in the environment is based on a direction of pointing by the controller component towards the feature in the environment.
Optionally, the method further includes selecting a feature in the environment for defining at least a part of the virtual space, wherein the act of selecting is performed based on a presence of an image of the feature in a camera image provided by a camera of the apparatus.
Optionally, the feature in the environment is selected automatically.
Optionally, the feature is selected in response to the feature being presence in a sequence of camera images that includes the camera image within a duration exceeding a time threshold.
Optionally, the method further includes displaying, by the screen, a pointer based on an orientation of a controller component, and wherein the method further comprises adjusting a configuration of the pointer displayed in the screen, and/or presenting an interaction graphic in association with the pointer.
Optionally, the act of adjusting the configuration of the pointer and/or the act of presenting the interaction graphic, is performed in response to an interaction between the pointer and a virtual object.
Optionally, the method further includes providing a virtual content for interaction by the user and an additional user.
Optionally, the method further includes connecting the user and the additional user to a same virtual space so that the user and the additional user can interact with the virtual content at the same virtual space.
Optionally, the virtual content is provided for interaction by the user and the additional user in different respective rooms.
Optionally, the virtual content is provided for interaction by the user and the additional user in a same room.
Optionally, the method further includes registering the user and the additional user to the environment so that the user and the additional user can interact with the virtual content while they are in the environment.
Optionally, the act of registering the user and the additional user to the environment comprises determining one or more anchor points that are associated with a location of the user and a location of the additional user.
A processor-readable non-transitory medium stores a set of instructions, an execution of which by a processing unit will cause a method to be performed, the processing unit being a part of an apparatus that is configured to provide a virtual or augmented reality experience, the method comprising: detecting, by a surface detector, a surface of an object in an environment surrounding a user of the apparatus; obtaining, by an object identifier, an orientation and/or an elevation of the surface of the object; identifying, by the object identifier, the object based on the orientation and/or the elevation of the surface of the object; and generating, by a graphic generator, an identifier for the identified object for display by a screen, wherein the screen is at least partially transparent for allowing the user of the apparatus to view the object in the environment surrounding the user.
An apparatus for providing an augmented reality experience, includes: a screen, wherein the screen is at least partially transparent for allowing a user of the apparatus to view an object in an environment surrounding the user; a space definer configured to obtain an input, and to define a virtual space based on the input, wherein the space definer is configured to obtain the input while the screen is being worn by the user; and a graphic generator configured to provide a virtual content for display by the screen, wherein the screen is configured to display the virtual content in the virtual space.
Optionally, the space definer is configured to define a virtual wall for the virtual space.
Optionally, the virtual wall is offset from a real physical wall in the environment surrounding the user.
Optionally, the virtual wall is aligned with, or intersects, a real physical wall in the environment surrounding the user.
Optionally, the screen is configured to display a wall identifier at a location in the screen, such that when the user views the virtual wall, the wall identifier will be in a spatial relationship with respect to the virtual wall.
Optionally, the space definer is configured to define a plurality of virtual walls for the virtual space; and wherein the screen is configured to display wall identifiers for the respective virtual walls.
Optionally, the space definer is configured to define a corner for the virtual space.
Optionally, the space definer is configured to define a wall edge for the virtual space.
Optionally, the space definer is configured to obtain the user input generated via a controller component, the user input indicating a selection of a feature in the environment for defining at least a part of the virtual space.
Optionally, the feature in the environment comprises a wall, a wall corner, an edge, or any combination of the foregoing.
Optionally, the user input indicates a cursor position in the screen.
Optionally, the user input indicates an orientation of the controller component, and wherein the selection of the feature in the environment is based on a direction of pointing by the controller component towards the feature in the environment.
Optionally, the apparatus further includes a camera, wherein the apparatus is configured to select a feature in the environment, for defining at least a part of the virtual space, based on a presence of an image of the feature in a camera image provided by the camera, wherein the camera image is the input obtained by the space definer.
Optionally, the apparatus is configured to select the feature in the environment automatically.
Optionally, the apparatus is configured to select the feature in response to the feature being presence in a sequence of camera images that includes the camera image within a duration exceeding a time threshold.
Optionally, the virtual content is also for interaction by an additional user.
Optionally, the apparatus is configured to connect the user and the additional user to the virtual space so that the user and the additional user can interact with the virtual content at the virtual space.
Optionally, the graphic generator is configured to provide the virtual content for interaction by the user and the additional user in different respective rooms.
Optionally, the graphic generator is configured to provide the virtual content for interaction by the user and the additional user in a same room.
Optionally, the apparatus is configured to register the user and the additional user to the environment so that the user and the additional user can interact with the virtual content while they are in the environment.
Optionally, the apparatus is configured to register the user and the additional user to the environment by determining one or more anchor points that are associated with a location of the user and a location of the additional user.
A method is performed by an apparatus that is configured to provide a virtual or augmented reality experience, the apparatus comprising a screen, the method comprising: obtaining, by a space definer, an input, wherein the space definer is configured to obtain the input while the screen is being worn by a user; defining a virtual space by the space definer based on the input; providing, by a graphic generator, a virtual content for display by a screen; and displaying the virtual content in the virtual space by the screen, wherein the screen is at least partially transparent for allowing the user of the apparatus to view an object in an environment surrounding the user.
Optionally, the act of defining the virtual space comprises defining a virtual wall for the virtual space.
Optionally, the virtual wall is offset from a real physical wall in the environment surrounding the user.
Optionally, the virtual wall is aligned with, or intersects, a real physical wall in the environment surrounding the user.
Optionally, the method further includes displaying, by the screen, a wall identifier at a location in the screen, such that when the user views the virtual wall, the wall identifier will be in a spatial relationship with respect to the virtual wall.
Optionally, the act of defining the virtual space comprises defining a plurality of virtual walls for the virtual space; and wherein the method further comprises displaying, by the screen, wall identifiers for the respective virtual walls.
Optionally, the act of defining the virtual space comprises defining a corner for the virtual space.
Optionally, the act of defining the virtual space comprises defining a wall edge for the virtual space.
Optionally, the method further includes obtaining the user input generated via a controller component, the user input indicating a selection of a feature in the environment for defining at least a part of the virtual space.
Optionally, the feature in the environment comprises a wall, a wall corner, an edge, or any combination of the foregoing.
Optionally, the user input indicates a cursor position in the screen.
Optionally, the user input indicates an orientation of the controller component, and wherein the selection of the feature in the environment is based on a direction of pointing by the controller component towards the feature in the environment.
Optionally, the method further includes selecting a feature in the environment, for defining at least a part of the virtual space, based on a presence of an image of the feature in a camera image provided by a camera, wherein the camera image is the input obtained by the space definer.
Optionally, the feature in the environment is selected automatically.
Optionally, the feature in the environment is selected in response to the feature being presence in a sequence of camera images that includes the camera image within a duration exceeding a time threshold.
Optionally, the virtual content is also for interaction by an additional user.
Optionally, the method further includes connecting the user and the additional user to the virtual space so that the user and the additional user can interact with the virtual content at the virtual space.
Optionally, the virtual content is provided for interaction by the user and the additional user in different respective rooms.
Optionally, the virtual content is provided for interaction by the user and the additional user in a same room.
Optionally, the method further includes registering the user and the additional user to the environment so that the user and the additional user can interact with the virtual content while they are in the environment.
Optionally, the act of registering the user and the additional user to the environment comprises determining one or more anchor points that are associated with a location of the user and a location of the additional user.
A processor-readable non-transitory medium stores a set of instructions, an execution of which by a processing unit will cause a method to be performed, the processing unit being a part of an apparatus that is configured to provide a virtual or augmented reality experience, the apparatus comprising a screen, the method comprising: obtaining, by a space definer, an input, wherein the space definer is configured to obtain the input while the screen is being worn by a user; defining a virtual space by the space definer based on the input; providing, by a graphic generator, a virtual content for display by a screen; and displaying the virtual content in the virtual space by the screen, wherein the screen is at least partially transparent for allowing the user of the apparatus to view an object in an environment surrounding the user.
Additional and other objects, features, and advantages of the disclosure are described in the detail description, figures and claims.
BRIEF DESCRIPTION OF THE DRAWINGS
The drawings illustrate the design and utility of various embodiments of the present disclosure. It should be noted that the figures are not drawn to scale and that elements of similar structures or functions are represented by like reference numerals throughout the figures. In order to better appreciate how to obtain the above-recited and other advantages and objects of various embodiments of the disclosure, a more detailed description of the present disclosures briefly described above will be rendered by reference to specific embodiments thereof, which are illustrated in the accompanying drawings. Understanding that these drawings depict only typical embodiments of the disclosure and are not therefore to be considered limiting of its scope, the disclosure will be described and explained with additional specificity and detail through the use of the accompanying drawings in which:
FIG. 1 A illustrates an image display system having an image display device in accordance with some embodiments.
FIG. 1 B illustrates an image display device displaying frames in multiple depth planes.
FIGS. 2 A- 2 B illustrate examples of information displayed by an image display device.
FIG. 3 illustrates another example of information displayed by an image display device.
FIGS. 4 A- 4 F illustrate examples of object identifications.
FIG. 5 A illustrates a method in accordance with some embodiments.
FIG. 5 B illustrates another method in accordance with some embodiments.
FIGS. 6 A- 6 C illustrate an example of a technique for inserting a virtual item.
FIG. 7 illustrates a method in accordance with some embodiments.
FIGS. 8 A- 8 E illustrate examples of techniques of presenting information in an augmented reality environment.
FIGS. 9 A- 9 E illustrate examples of features associated with a virtual pointer.
FIGS. 10 A- 10 E illustrate examples of virtual objects being interacted with by multiple users.
FIG. 11 A illustrates a method in accordance with some embodiments.
FIG. 11 B illustrates another method in accordance with some embodiments.
FIGS. 12 , 13 A, and 13 B illustrate a method in accordance with some embodiments.
FIGS. 14 A- 141 illustrate examples of techniques for defining a virtual space.
FIG. 15 illustrates a processing unit of an apparatus in accordance with some embodiments.
FIG. 16 A illustrates a method in accordance with some embodiments.
FIG. 16 B illustrates another method in accordance with some embodiments.
FIG. 17 illustrates a specialized processing system in accordance with some embodiments.
DETAILED DESCRIPTION
Various embodiments of the disclosure are directed to methods, apparatuses, and articles of manufacture for providing input for head-worn video image devices. Other objects, features, and advantages of the disclosure are described in the detailed description, figures, and claims.
Various embodiments are described hereinafter with reference to the figures. It should be noted that the figures are not drawn to scale and that elements of similar structures or functions are represented by like reference numerals throughout the figures. It should also be noted that the figures are only intended to facilitate the description of the embodiments. They are not intended as an exhaustive description of the invention or as a limitation on the scope of the invention. In addition, an illustrated embodiment needs not have all the aspects or advantages shown. An aspect or an advantage described in conjunction with a particular embodiment is not necessarily limited to that embodiment and can be practiced in any other embodiments even if not so illustrated, or if not so explicitly described.
The description that follows pertains to an illustrative VR, AR, and/or MR system with which embodiments described herein may be practiced. However, it is to be understood that the embodiments also lends themselves to applications in other types of display systems (including other types of VR, AR, and/or MR systems), and therefore the embodiments are not to be limited to only the illustrative examples disclosed herein.
Referring to FIG. 1 A , an augmented reality system 1 is illustrated featuring a head-worn viewing component (image display device) 2 , a hand-held controller component 4 , and an interconnected auxiliary computing or controller component 6 which may be configured to be worn as a belt pack or the like on the user. Each of these components may be operatively coupled ( 10 , 12 , 14 , 16 , 17 , 18 ) to each other and to other connected resources 8 such as cloud computing or cloud storage resources via wired or wireless communication configurations, such as those specified by IEEE 802.11, Bluetooth (RTM), and other connectivity standards and configurations. As described, for example, in U.S. patent application Ser. Nos. 14/555,585, 14/690,401, 14/331,218, 15/481,255, 62/627,155, 62/518,539, 16/229,532, 16/155,564, 15/413,284, 16/020,541, 62,702,322, 62/206,765, 15,597,694, 16/221,065, 15/968,673, 62/682,788, and 62/899,678 each of which is incorporated by reference herein in its entirety, various aspects of such components are described, such as various embodiments of the two depicted optical elements 20 through which the user may see the world around them along with visual components which may be produced by the associated system components, for an augmented reality experience. As illustrated in FIG. 1 A , such a system 1 may also comprise various sensors configured to provide information pertaining to the environment around the user, including but not limited to various camera type sensors (such as monochrome, color/RGB, and/or thermal imaging components) ( 22 , 24 , 26 ), depth camera sensors 28 , and/or sound sensors 30 such as microphones. There is a need for compact and persistently connected wearable computing systems and assemblies such as those described herein, which may be utilized to provide a user with the perception of rich augmented reality experiences.
The system 1 also includes an apparatus 7 for providing input for the image display device 2 . The apparatus 7 will be described in further detail below. The image display device 2 may be a VR device, an AR device, a MR device, or any of other types of display devices. As shown in the figure, the image display device 2 includes a frame structure worn by an end user, a display subsystem carried by the frame structure, such that the display subsystem is positioned in front of the eyes of the end user, and a speaker carried by the frame structure, such that the speaker is positioned adjacent the ear canal of the end user (optionally, another speaker (not shown) is positioned adjacent the other ear canal of the end user to provide for stereo/shapeable sound control). The display subsystem is designed to present the eyes of the end user with light patterns that can be comfortably perceived as augmentations to physical reality, with high-levels of image quality and three-dimensional perception, as well as being capable of presenting two-dimensional content. The display subsystem presents a sequence of frames at high frequency that provides the perception of a single coherent scene.
In the illustrated embodiments, the display subsystem employs âoptical see-throughâ display through which the user can directly view light from real objects via transparent (or semi-transparent) elements. The transparent element, often referred to as a âcombiner,â superimposes light from the display over the user's view of the real world. To this end, the display subsystem comprises a partially transparent display or a complete transparent display. The display is positioned in the end user's field of view between the eyes of the end user and an ambient environment, such that direct light from the ambient environment is transmitted through the display to the eyes of the end user.
In the illustrated embodiments, an image projection assembly provides light to the partially transparent display, thereby combining with the direct light from the ambient environment, and being transmitted from the display to the eyes of the user. The projection subsystem may be an optical fiber scan-based projection device, and the display may be a waveguide-based display into which the scanned light from the projection subsystem is injected to produce, e.g., images at a single optical viewing distance closer than infinity (e.g., arm's length), images at multiple, discrete optical viewing distances or focal planes, and/or image layers stacked at multiple viewing distances or focal planes to represent volumetric 3D objects. These layers in the light field may be stacked closely enough together to appear continuous to the human visual subsystem (i.e., one layer is within the cone of confusion of an adjacent layer).
Additionally or alternatively, picture elements may be blended across two or more layers to increase perceived continuity of transition between layers in the light field, even if those layers are more sparsely stacked (i.e., one layer is outside the cone of confusion of an adjacent layer). The display subsystem may be monocular or binocular.
The image display device 2 may also include one or more sensors mounted to the frame structure
CLAIMS
Claims ( 20 )
1 . An apparatus for providing an augmented reality experience, comprising:
a screen, wherein the screen is at least partially transparent for allowing a user of the apparatus to view an object in an environment surrounding the user; a space definer configured to obtain an input, and to define a virtual space based on the input, wherein the space definer is configured to obtain the input while the screen is being worn by the user; and a graphic generator configured to provide a virtual content for display by the screen, wherein the screen is configured to display the virtual content in the virtual space.
2 . The apparatus of claim 1 , wherein the space definer is configured to define a virtual wall for the virtual space.
3 . The apparatus of claim 2 , wherein the virtual wall is offset from a real physical wall in the environment surrounding the user.
4 . The apparatus of claim 3 , wherein the virtual wall is aligned with, or intersects, a real physical wall in the environment surrounding the user.
5 . The apparatus of claim 4 , wherein the screen is configured to display a wall identifier at a location in the screen, such that when the user views the virtual wall, the wall identifier will be in a spatial relationship with respect to the virtual wall.
6 . The apparatus of claim 1 , wherein the space definer is configured to define a plurality of virtual walls for the virtual space; and
wherein the screen is configured to display wall identifiers for the respective virtual walls.
7 . The apparatus of claim 1 , wherein the space definer is configured to define a corner for the virtual space.
8 . The apparatus of claim 1 , wherein the space definer is configured to define a wall edge for the virtual space.
9 . The apparatus of claim 1 , wherein the space definer is configured to obtain the user input generated via a controller component, the user input indicating a selection of a feature in the environment for defining at least a part of the virtual space.
10 . The apparatus of claim 9 , wherein the feature in the environment comprises a wall, a wall corner, an edge, or any combination of the foregoing.
11 . The apparatus of claim 9 , wherein the user input indicates a cursor position in the screen.
12 . The apparatus of claim 9 , wherein the user input indicates an orientation of the controller component, and wherein the selection of the feature in the environment is based on a direction of pointing by the controller component towards the feature in the environment.
13 . The apparatus of claim 1 , further comprising a camera, wherein the apparatus is configured to select a feature in the environment, for defining at least a part of the virtual space, based on a presence of an image of the feature in a camera image provided by the camera, wherein the camera image is the input obtained by the space definer.
14 . The apparatus of claim 13 , wherein the apparatus is configured to select the feature in the environment automatically.
15 . The apparatus of claim 13 , wherein the apparatus is configured to select the feature in response to the feature being presence in a sequence of camera images that includes the camera image within a duration exceeding a time threshold.
16 . The apparatus of claim 1 , wherein the virtual content is also for interaction by an additional user.
17 . The apparatus of claim 16 , wherein the apparatus is configured to connect the user and the additional user to the virtual space so that the user and the additional user can interact with the virtual content at the virtual space.
18 . The apparatus of claim 16 , wherein the graphic generator is configured to provide the virtual content for interaction by the user and the additional user in different respective rooms.
19 . A method performed by an apparatus that is configured to provide a virtual or augmented reality experience, the apparatus comprising a screen, the method comprising:
obtaining, by a space definer, an input, wherein the space definer is configured to obtain the input while the screen is being worn by a user; defining a virtual space by the space definer based on the input; providing, by a graphic generator, a virtual content for display by a screen; and displaying the virtual content in the virtual space by the screen, wherein the screen is at least partially transparent for allowing the user of the apparatus to view an object in an environment surrounding the user.
20 . A processor-readable non-transitory medium storing a set of instructions, an execution of which by a processing unit will cause a method to be performed, the processing unit being a part of an apparatus that is configured to provide a virtual or augmented reality experience, the apparatus comprising a screen, the method comprising:
obtaining, by a space definer, an input, wherein the space definer is configured to obtain the input while the screen is being worn by a user; defining a virtual space by the space definer based on the input; providing, by a graphic generator, a virtual content for display by a screen; and displaying the virtual content in the virtual space by the screen, wherein the screen is at least partially transparent for allowing the user of the apparatus to view an object in an environment surrounding the user.
US18/820,103
2020-03-24
2024-08-29
Systems and methods for virtual and augmented reality
Pending
US20240420508A1
( en )
Priority Applications (1)
Application Number
Priority Date
Filing Date
Title
US18/820,103
US20240420508A1
( en )
2020-03-24
2024-08-29
Systems and methods for virtual and augmented reality
Applications Claiming Priority (3)
Application Number
Priority Date
Filing Date
Title
US202062994159P
2020-03-24
2020-03-24
US17/211,502
US12112574B2
( en )
2020-03-24
2021-03-24
Systems and methods for virtual and augmented reality
US18/820,103
US20240420508A1
( en )
2020-03-24
2024-08-29
Systems and methods for virtual and augmented reality
Related Parent Applications (1)
Application Number
Title
Priority Date
Filing Date
US17/211,502
Continuation
US12112574B2
( en )
2020-03-24
2021-03-24
Systems and methods for virtual and augmented reality
Publications (1)
Publication Number
Publication Date
US20240420508A1
true
US20240420508A1 ( en )
2024-12-19
Family
ID=77857509
Family Applications (2)
Application Number
Title
Priority Date
Filing Date
US17/211,502
Active
2041-09-07
US12112574B2
( en )
2020-03-24
2021-03-24
Systems and methods for virtual and augmented reality
US18/820,103
Pending
US20240420508A1
( en )
2020-03-24
2024-08-29
Systems and methods for virtual and augmented reality
Family Applications Before (1)
Application Number
Title
Priority Date
Filing Date
US17/211,502
Active
2041-09-07
US12112574B2
( en )
2020-03-24
2021-03-24
Systems and methods for virtual and augmented reality
Country Status (5)
Country
Link
US
( 2 )
US12112574B2
( en )
EP
( 1 )
EP4128207A4
( en )
JP
( 1 )
JP7560568B2
( en )
CN
( 1 )
CN115335894A
( en )
WO
( 1 )
WO2021195266A1
( en )
Families Citing this family (24)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US11132827B2
( en )
2019-09-19
2021-09-28
Facebook Technologies, Llc
Artificial reality system architecture for concurrent application execution and collaborative 3D scene rendering
US11487080B2
( en )
*
2019-10-18
2022-11-01
Htc Corporation
Head-mounted display device and adjustment method
US11995774B2
( en )
*
2020-06-29
2024-05-28
Snap Inc.
Augmented reality experiences using speech and text captions
US12249092B2
( en )
2020-09-15
2025-03-11
Meta Platforms Technologies, Llc
Visual inertial odometry localization using sparse sensors
US20240025040A1
( en )
*
2020-12-01
2024-01-25
Omron Corporation
Apparatus and method for simulating a mobile robot at a work site
EP4259289A4
( en )
*
2020-12-10
2024-11-06
3M Innovative Properties Company
THERMAL IMAGING SYSTEM WITH IMAGE SHIFTING CAPABILITY
US12229977B2
( en )
*
2021-05-18
2025-02-18
Snap Inc.
Augmented reality guided depth estimation
US12096112B2
( en )
*
2021-07-29
2024-09-17
Meta Platforms Technologies, Llc
User interface to select field of view of a camera in a smart glass
US12008806B2
( en )
2021-08-19
2024-06-11
Meta Platforms Technologies, Llc
Methods and systems to allow three-dimensional map sharing between heterogeneous computing systems, cross-localization, and sharing content in three-dimensional space
US12277642B2
( en )
2021-10-26
2025-04-15
Meta Platforms Technologies, Llc
Localization failure handling on artificial reality systems
US20230186570A1
( en )
*
2021-12-10
2023-06-15
Meta Platforms Technologies, Llc
Methods and systems to allow three-dimensional maps sharing and updating
US12299918B2
( en )
2022-01-26
2025-05-13
Meta Platforms Technologies, Llc
Methods and systems to facilitate passive relocalization using three-dimensional maps
US12299835B1
( en )
2022-10-24
2025-05-13
Meta Platforms Technologies, Llc
Shared scene co-location for artificial reality devices
US12423828B2
( en )
*
2022-11-07
2025-09-23
PassiveLogic, Inc.
Door and window detection in an AR environment
US11768581B1
( en )
*
2022-12-13
2023-09-26
Illuscio, Inc.
Systems and methods for multi-modality interactions in a spatial computing environment
US12323713B2
( en )
*
2022-12-23
2025-06-03
Meta Platforms Technologies, Llc
Headsets having improved camera arrangements and depth sensors, and methods of use thereof
DE102023105342A1
( en )
*
2023-03-03
2024-09-05
Syncreality GmbH
Scanning a room for VR/AR experiences
CN116563740A
( en )
*
2023-05-15
2023-08-08
å京åè·³ç½ç»ææ¯æéå ¬å¸
Control method and device based on augmented reality, electronic equipment and storage medium
US12506800B2
( en )
*
2023-05-16
2025-12-23
Hexagon Technology Center Gmbh
Precision geometry service for thin client applications
US12554557B2
( en )
*
2023-05-16
2026-02-17
Hexagon Technology Center Gmbh
Precision geometry client for thin client applications
USD1068806S1
( en )
*
2023-06-04
2025-04-01
Apple Inc.
Display screen or portion thereof with graphical user interface
US12039431B1
( en )
*
2023-09-27
2024-07-16
OpenAI Opco, LLC
Systems and methods for interacting with a multimodal machine learning model
WO2025158548A1
( en )
*
2024-01-24
2025-07-31
ãã¯ã»ã«æ ªå¼ä¼ç¤¾
Information display device
US12422925B1
( en )
*
2024-04-26
2025-09-23
Huawei Technologies Co., Ltd.
Methods and systems for interacting with digital objects using a gaze tracking-enabled headset
Citations (7)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US20130196772A1
( en )
*
2012-01-31
2013-08-01
Stephen Latta
Matching physical locations for shared virtual experience
US20130257907A1
( en )
*
2012-03-30
2013-10-03
Sony Mobile Communications Inc.
Client device
US20140028712A1
( en )
*
2012-07-26
2014-01-30
Qualcomm Incorporated
Method and apparatus for controlling augmented reality
US20140333666A1
( en )
*
2013-05-13
2014-11-13
Adam G. Poulos
Interactions of virtual objects with surfaces
US20180197340A1
( en )
*
2016-06-10
2018-07-12
Dirtt Environmental Solutions, Ltd.
Mixed-reality and cad architectural design environment
US10937247B1
( en )
*
2019-03-11
2021-03-02
Amazon Technologies, Inc.
Three-dimensional room model generation using ring paths and photogrammetry
US11017611B1
( en )
*
2020-01-27
2021-05-25
Amazon Technologies, Inc.
Generation and modification of rooms in virtual reality environments
Family Cites Families (19)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US20130083003A1
( en )
*
2011-09-30
2013-04-04
Kathryn Stone Perez
Personal audio/visual system
US9041622B2
( en )
*
2012-06-12
2015-05-26
Microsoft Technology Licensing, Llc
Controlling a virtual object with a real controller device
EP2872967B1
( en )
*
2012-07-13
2018-11-21
Sony Depthsensing Solutions SA/NV
Method and system for detecting hand-related parameters for human-to-computer gesture-based interaction
WO2014108799A2
( en )
*
2013-01-13
2014-07-17
Quan Xiao
Apparatus and methods of real time presenting 3d visual effects with stereopsis more realistically and substract reality with external display(s)
US9563955B1
( en )
*
2013-05-15
2017-02-07
Amazon Technologies, Inc.
Object tracking techniques
US9870058B2
( en )
*
2014-04-23
2018-01-16
Sony Corporation
Control of a real world object user interface
EP3038061A1
( en )
*
2014-12-23
2016-06-29
Orange
Apparatus and method to display augmented reality data
US10317989B2
( en )
2016-03-13
2019-06-11
Logitech Europe S.A.
Transition between virtual and augmented reality
EP3465619A2
( en )
2016-05-27
2019-04-10
Holobuilder Inc,
Augmented and virtual reality
US10078377B2
( en )
2016-06-09
2018-09-18
Microsoft Technology Licensing, Llc
Six DOF mixed reality input by fusing inertial handheld controller with hand tracking
US10395428B2
( en )
*
2016-06-13
2019-08-27
Sony Interactive Entertainment Inc.
HMD transitions for focusing on specific content in virtual-reality environments
US20180095542A1
( en )
*
2016-09-30
2018-04-05
Sony Interactive Entertainment Inc.
Object Holder for Virtual Reality Interaction
CN110313021B
( en )
*
2017-03-06
2023-07-25
è¿æ ªå¼ä¼ç¤¾
Augmented reality providing method, apparatus, and computer-readable recording medium
US10747301B2
( en )
*
2017-03-28
2020-08-18
Magic Leap, Inc.
Augmented reality system with spatialized audio tied to user manipulated virtual object
WO2018222756A1
( en )
2017-05-30
2018-12-06
Ptc Inc.
Object initiated communication
US11176748B2
( en )
*
2017-12-19
2021-11-16
Sony Interactive Entertainment Inc.
Image processing apparatus, image processing method, and program
US10861242B2
( en )
*
2018-05-22
2020-12-08
Magic Leap, Inc.
Transmodal input fusion for a wearable system
US11733824B2
( en )
*
2018-06-22
2023-08-22
Apple Inc.
User interaction interpreter
US11030813B2
( en )
2018-08-30
2021-06-08
Snap Inc.
Video clip object tracking
2021
2021-03-24
JP
JP2022557750A
patent/JP7560568B2/en
active
Active
2021-03-24
WO
PCT/US2021/023961
patent/WO2021195266A1/en
not_active
Ceased
2021-03-24
US
US17/211,502
patent/US12112574B2/en
active
Active
2021-03-24
CN
CN202180024039.5A
patent/CN115335894A/en
active
Pending
2021-03-24
EP
EP21774273.3A
patent/EP4128207A4/en
active
Pending
2024
2024-08-29
US
US18/820,103
patent/US20240420508A1/en
active
Pending
Patent Citations (7)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US20130196772A1
( en )
*
2012-01-31
2013-08-01
Stephen Latta
Matching physical locations for shared virtual experience
US20130257907A1
( en )
*
2012-03-30
2013-10-03
Sony Mobile Communications Inc.
Client device
US20140028712A1
( en )
*
2012-07-26
2014-01-30
Qualcomm Incorporated
Method and apparatus for controlling augmented reality
US20140333666A1
( en )
*
2013-05-13
2014-11-13
Adam G. Poulos
Interactions of virtual objects with surfaces
US20180197340A1
( en )
*
2016-06-10
2018-07-12
Dirtt Environmental Solutions, Ltd.
Mixed-reality and cad architectural design environment
US10937247B1
( en )
*
2019-03-11
2021-03-02
Amazon Technologies, Inc.
Three-dimensional room model generation using ring paths and photogrammetry
US11017611B1
( en )
*
2020-01-27
2021-05-25
Amazon Technologies, Inc.
Generation and modification of rooms in virtual reality environments
Also Published As
Publication number
Publication date
US12112574B2
( en )
2024-10-08
JP7560568B2
( en )
2024-10-02
WO2021195266A1
( en )
2021-09-30
JP2023520765A
( en )
2023-05-19
EP4128207A1
( en )
2023-02-08
EP4128207A4
( en )
2023-12-27
CN115335894A
( en )
2022-11-11
US20210304509A1
( en )
2021-09-30
Similar Documents
Publication
Publication Date
Title
US12112574B2
( en )
2024-10-08
Systems and methods for virtual and augmented reality
TWI567659B
( en )
2017-01-21
Theme-based augmentation of photorepresentative view
US10373392B2
( en )
2019-08-06
Transitioning views of a virtual model
US12026351B2
( en )
2024-07-02
3D models for displayed 2D elements
JP7679395B2
( en )
2025-05-19
Systems and methods for multi-user virtual and augmented reality - Patents.com
US20160343166A1
( en )
2016-11-24
Image-capturing system for combining subject and three-dimensional virtual space in real time
US20150317831A1
( en )
2015-11-05
Transitions between body-locked and world-locked augmented reality
US20160004335A1
( en )
2016-01-07
Three Dimensional Display System and Use
TW201237691A
( en )
2012-09-16
User interface presentation and interactions
EP3066646A1
( en )
2016-09-14
Construction of synthetic augmented reality environment
US20240046903A1
( en )
2024-02-08
Systems and methods for virtual and augmented reality
JP2020181263A
( en )
2020-11-05
Image creation device and information presentation method
US20250044865A1
( en )
2025-02-06
Virtual, augmented, and mixed reality systems and methods
CN116863107A
( en )
2023-10-10
Augmented reality providing method, apparatus, and non-transitory computer readable medium
RU2695053C1
( en )
2019-07-18
Method and device for control of three-dimensional objects in virtual space
Hamadouche
2018
Augmented reality X-ray vision on optical see-through head mounted displays
Legal Events
Date
Code
Title
Description
2024-09-12
STPP
Information on status: patent application and granting procedure in general
Free format text : DOCKETED NEW CASE - READY FOR EXAMINATION
Free format text : NON FINAL ACTION COUNTED, NOT YET MAILED
2025-08-25
AS
Assignment
Owner name : MAGIC LEAP, INC., FLORIDA
Free format text : ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:BERKEBILE, ROBERT DAVID;REEL/FRAME:072113/0914
Effective date : 20200325
2025-10-29
AS
Assignment
Owner name : CITIBANK, N.A., AS COLLATERAL AGENT, NEW YORK
Free format text : SECURITY INTEREST;ASSIGNORS:MAGIC LEAP, INC.;MENTOR ACQUISITION ONE, LLC;MOLECULAR IMPRINTS, INC.;REEL/FRAME:073430/0225
Effective date : 20241121
2026-02-20
STPP
Information on status: patent application and granting procedure in general
Free format text : NON FINAL ACTION COUNTED, NOT YET MAILED
2026-02-27
STPP
Information on status: patent application and granting procedure in general
Free format text : NON FINAL ACTION MAILED