Google Patentsopen access

Technique for recording augmented reality data — Magic Leap, Inc. (US11380068B2)

ABSTRACT

Abstract

Disclosed is an improved approach for generated recordings from augmented reality systems from the perspective of a camera within the system. Instead of re-using rendered virtual content from the perspective of the user's eyes for AR recordings, additional virtual content is rendered from an additional perspective specifically for the AR recording. That additional virtual content is combined with image frames generated by a camera to form the AR recording.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of pending U.S. patent application Ser. No. 16/743,207, entitled “TECHNIQUE FOR RECORDING AUGMENTED REALITY DATA,” filed Jan. 15, 2020, which is a continuation of U.S. patent application Ser. No. 15/924,144, entitled “TECHNIQUE FOR RECORDING AUGMENTED REALITY DATA,” filed Mar. 16, 2018, which claims the benefit of priority to U.S. Provisional Application Ser. No. 62/472,923, filed on Mar. 17, 2017. The contents of these patent applications are hereby incorporated by reference in their entirety.

FIELD OF THE INVENTION

The present disclosure generally relates to augmented reality technologies.

BACKGROUND

Modern computing and display technologies have facilitated the development of systems for so-called “virtual reality” or “augmented reality”, where digitally reproduced images, or portions thereof, are presented to a user in a manner where they seem to be, or may be perceived as, real. A virtual reality (VR) scenario typically involves presentation of digital or virtual image information without transparency to other actual real-world visual input, whereas an augmented reality (AR) scenario typically involves presentation of digital or virtual image information as an augmentation to visualization of the actual real-world around a user.

VR and AR systems typically employ wearable display devices (e.g., head-worn displays, helmet-mounted displays, or smart glasses) that are at least loosely coupled to a user's head, and thus move when the end user's head moves. If the end user's head motions are detected by the display system, the data being displayed can be updated to take the change in head pose (e.g., the orientation and/or location of user's head) into account.

As an example, if a user wearing a head-worn display views a virtual representation of a three-dimensional (3D) object on the display and walks around the area where the 3D object appears, that 3D object can be re-rendered for each viewpoint, giving the end user the perception that he or she is walking around an object that occupies real space. If the head-worn display is used to present multiple objects within a virtual space (for instance, a rich virtual world), measurements of head pose can be used to re-render the scene to match the end user's dynamically changing head location and orientation and provide an increased sense of immersion in the virtual space.

Head-worn displays that enable AR provide concurrent viewing of both real and virtual objects. With an “optical see-through” display, the end user can see through transparent (or semi-transparent) elements in the display system to view directly the light from real objects in the environment. The transparent element, often referred to as a “combiner,” superimposes light from the display over the end user's view of the real world, where light produced by the display projects an image of the virtual content over the see-through view of the real world. A camera may be mounted onto the wearable display to capture images or videos of the scene being viewed by the user.

When operating the wearable display, the user may seek to preserve a recording of his or her experience in operating the AR device. For example, gamers that play electronic video games often choose to create a recording of the real-time action of the gameplay, where the recording can be played back at a later moment in time. In the context of AR systems, this means that the recording should include a recording of the combined real and virtual objects during operation of the display system.

The problem is that virtual objects in the AR system are rendered from the perspective of the user's eyes/head, whereas the camera that is actually recording the real objects is normally mounted at a different location and/or viewing angle from the user's eyes. This means that a recording which simply combines the rendered virtual objects with the recording of the real objects will show the virtual objects from a first perspective and the real objects from a second perspective—potentially creating a significant disconnect in the orientation and/or content of the virtual objects in the final image product for a given image frame of the recording.

To address this, “warping” may be applied to change the way that the virtual objects are presented in the video recording. This approach takes the virtual objects rendered from the first perspective, and shifts the way that the virtual objects are presented to attempt to display those virtual objects from the second perspective. One problem with this approach is that when a virtual object is originally rendered from the first perspective, the rendered virtual object may not include displayable information about the virtual object from the second perspective. For example, when a second virtual object is hidden behind a first virtual object from the perspective of the user's eyes (i.e., the first perspective), then a rendering of the virtual content may only show the first virtual object and not the second (hidden) virtual object which is viewable from the perspective of the camera (i.e., the second perspective). This means that even when warping is performed on the virtual content, the warped virtual content that only includes content for the first virtual object and will still be missing content for the second virtual object that is viewable from the perspective of the camera. This approach therefore may create video artifacts that affect the content veracity of the recorded image/video frames.

Another potential problem with this approach is that it introduces latency as the AR equipment and application attempts to perform the necessary processing to warp the virtual content for the recording. This may introduce perceptible delays in generating the real-time images for display to the user on the wearable display.

Therefore, there is a need for an improved approach to generate AR recordings having a combination of recorded real objects and rendered virtual objects.

SUMMARY

In accordance with some embodiments, instead of re-using rendered virtual content from a perspective of a user's eyes for augmented reality (AR) recordings, additional virtual content is rendered from an additional perspective specifically for the AR recording. That is, virtual content is rendered from a perspective of a camera for AR recordings. That additional virtual content is combined with image frames generated by the camera to form the AR recording. To reduce latency and computing overhead, gaps in normal GPU processing may be leveraged to perform the additional rendering of the virtual content.

Additional and other objects, features, and advantages of the disclosure are described in the detail description, figures and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings illustrate the design and utility of preferred embodiments of the present disclosure, in which similar elements are referred to by common reference numerals. In order to better appreciate how the above-recited and other advantages and objects of the present disclosures are obtained, a more particular description of the present disclosures briefly described above will be rendered by reference to specific embodiments thereof, which are illustrated in the accompanying drawings. Understanding that these drawings depict only typical embodiments of the disclosure and are not therefore to be considered limiting of its scope, the disclosure will be described and explained with additional specificity and detail through the use of the accompanying drawings in which:

FIGS. 1A-C are block diagrams of illustrative augmented reality systems constructed, according to some embodiments of the present disclosure.

FIG. 2 illustrates an example architecture for implementing the AR recording process according to some embodiments.

FIG. 3 shows a flowchart of an approach to generate AR recordings using warping, according to some embodiments.

FIGS. 4A-G provide an illustration of the warping approach to implement AR recording, according to some embodiments.

FIG. 5 illustrates an architecture for implementing this AR recording process, according to some embodiments.

FIG. 6 shows a flowchart of an approach to generate AR recordings, according to some embodiments.

FIGS. 7A-F illustrate an approach to generate AR recordings, according to some embodiments.

FIG. 8 illustrates an example GPU processing cycle, according to some embodiments.

FIG. 9A illustrates an example GPU processing cycle, according to some embodiments.

FIG. 9B shows a flow chart of an approach to implement a GPU processing cycle, according to some embodiments.

FIGS. 10A-B illustrate example alternative GPU processing cycle, according to some embodiments.

FIG. 10C shows a flow chart of an approach to implement an alternative GPU processing cycle, according to some embodiments.

FIG. 11 is a block diagram of an illustrative computing system, according to some embodiments.

DETAILED DESCRIPTION

The present disclosure is directed to an approach to implement augmented reality (AR) recordings using “third eye” virtual content rendering. According to some embodiments, instead of re-using rendered virtual content from a perspective of a user's eyes for AR recordings, additional virtual content is rendered from a perspective of the camera, and that additional virtual content is combined with the camera recording of a real world to form the AR recording. In some embodiments, gaps in normal GPU processing may be leveraged to perform the additional rendering to generate the additional virtual content rendering.

This disclosure will first provide a description of an illustrative AR system with which some embodiments of the disclosure may be practiced, followed by a description of one or more embodiments of an improved process and mechanism to generate AR content for recordings.

Illustrative Augmented Reality System

The description that follows pertains to an illustrative augmented reality (AR) system (which may be referred to herein as any of “augmented reality system”, “AR system”, or “mixed reality system) with which the disclosure may be practiced. However, it is to be understood that the disclosure also lends itself to applications in other types of augmented reality and virtual reality systems, and therefore the disclosure is not to be limited to only the illustrative system disclosed herein.

FIG. 1A is a block diagram illustrating an augmented reality (AR) system 100 a , according to one embodiment. The AR system 100 a may be operated in conjunction with an augmented reality control system 101 a , providing images of virtual objects intermixed with physical objects in a field of view of an end user 50 . This approach employs one or more at least partially transparent surfaces through which the ambient environment in the field of view of the end user 50 can be seen and on to which the AR system 100 a produces images of virtual objects.

For AR applications, it may be desirable to spatially position various virtual objects relative to physical objects in the field of view of the end user 50 . Virtual objects, also referred to herein as virtual tags or tag or call outs, may take any of a large variety of forms, having any variety of data, information, concept, or logical construct capable of being represented as an image. Non-limiting examples of virtual objects may include: a virtual text object, a virtual numeric object, a virtual alphanumeric object, a virtual tag object, a virtual field object, a virtual chart object, a virtual map object, a virtual instrumentation object, or a virtual visual representation of a physical object.

The AR system 100 a includes a frame structure 102 worn by the end user 50 , a display system 104 carried by the frame structure 102 , such that the display system 104 is positioned in front of eyes of the end user 50 , and a speaker 106 incorporated into or connected to the display system 104 . In the illustrated embodiment, the speaker 106 is carried by the frame structure 102 , such that the speaker 106 is positioned adjacent (in or around) an ear canal of the end user 50 (e.g., an earbud or headphone).

The display system 104 is designed to present the eyes of the end user 50 with photo-based radiation patterns that can be comfortably perceived as augmentations to physica

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of pending U.S. patent application Ser. No. 16/743,207, entitled “TECHNIQUE FOR RECORDING AUGMENTED REALITY DATA,” filed Jan. 15, 2020, which is a continuation of U.S. patent application Ser. No. 15/924,144, entitled “TECHNIQUE FOR RECORDING AUGMENTED REALITY DATA,” filed Mar. 16, 2018, which claims the benefit of priority to U.S. Provisional Application Ser. No. 62/472,923, filed on Mar. 17, 2017. The contents of these patent applications are hereby incorporated by reference in their entirety.

FIELD OF THE INVENTION

The present disclosure generally relates to augmented reality technologies.

BACKGROUND

Modern computing and display technologies have facilitated the development of systems for so-called “virtual reality” or “augmented reality”, where digitally reproduced images, or portions thereof, are presented to a user in a manner where they seem to be, or may be perceived as, real. A virtual reality (VR) scenario typically involves presentation of digital or virtual image information without transparency to other actual real-world visual input, whereas an augmented reality (AR) scenario typically involves presentation of digital or virtual image information as an augmentation to visualization of the actual real-world around a user.

VR and AR systems typically employ wearable display devices (e.g., head-worn displays, helmet-mounted displays, or smart glasses) that are at least loosely coupled to a user's head, and thus move when the end user's head moves. If the end user's head motions are detected by the display system, the data being displayed can be updated to take the change in head pose (e.g., the orientation and/or location of user's head) into account.

As an example, if a user wearing a head-worn display views a virtual representation of a three-dimensional (3D) object on the display and walks around the area where the 3D object appears, that 3D object can be re-rendered for each viewpoint, giving the end user the perception that he or she is walking around an object that occupies real space. If the head-worn display is used to present multiple objects within a virtual space (for instance, a rich virtual world), measurements of head pose can be used to re-render the scene to match the end user's dynamically changing head location and orientation and provide an increased sense of immersion in the virtual space.

Head-worn displays that enable AR provide concurrent viewing of both real and virtual objects. With an “optical see-through” display, the end user can see through transparent (or semi-transparent) elements in the display system to view directly the light from real objects in the environment. The transparent element, often referred to as a “combiner,” superimposes light from the display over the end user's view of the real world, where light produced by the display projects an image of the virtual content over the see-through view of the real world. A camera may be mounted onto the wearable display to capture images or videos of the scene being viewed by the user.

When operating the wearable display, the user may seek to preserve a recording of his or her experience in operating the AR device. For example, gamers that play electronic video games often choose to create a recording of the real-time action of the gameplay, where the recording can be played back at a later moment in time. In the context of AR systems, this means that the recording should include a recording of the combined real and virtual objects during operation of the display system.

The problem is that virtual objects in the AR system are rendered from the perspective of the user's eyes/head, whereas the camera that is actually recording the real objects is normally mounted at a different location and/or viewing angle from the user's eyes. This means that a recording which simply combines the rendered virtual objects with the recording of the real objects will show the virtual objects from a first perspective and the real objects from a second perspective—potentially creating a significant disconnect in the orientation and/or content of the virtual objects in the final image product for a given image frame of the recording.

To address this, “warping” may be applied to change the way that the virtual objects are presented in the video recording. This approach takes the virtual objects rendered from the first perspective, and shifts the way that the virtual objects are presented to attempt to display those virtual objects from the second perspective. One problem with this approach is that when a virtual object is originally rendered from the first perspective, the rendered virtual object may not include displayable information about the virtual object from the second perspective. For example, when a second virtual object is hidden behind a first virtual object from the perspective of the user's eyes (i.e., the first perspective), then a rendering of the virtual content may only show the first virtual object and not the second (hidden) virtual object which is viewable from the perspective of the camera (i.e., the second perspective). This means that even when warping is performed on the virtual content, the warped virtual content that only includes content for the first virtual object and will still be missing content for the second virtual object that is viewable from the perspective of the camera. This approach therefore may create video artifacts that affect the content veracity of the recorded image/video frames.

Another potential problem with this approach is that it introduces latency as the AR equipment and application attempts to perform the necessary processing to warp the virtual content for the recording. This may introduce perceptible delays in generating the real-time images for display to the user on the wearable display.

Therefore, there is a need for an improved approach to generate AR recordings having a combination of recorded real objects and rendered virtual objects.

SUMMARY

In accordance with some embodiments, instead of re-using rendered virtual content from a perspective of a user's eyes for augmented reality (AR) recordings, additional virtual content is rendered from an additional perspective specifically for the AR recording. That is, virtual content is rendered from a perspective of a camera for AR recordings. That additional virtual content is combined with image frames generated by the camera to form the AR recording. To reduce latency and computing overhead, gaps in normal GPU processing may be leveraged to perform the additional rendering of the virtual content.

Additional and other objects, features, and advantages of the disclosure are described in the detail description, figures and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings illustrate the design and utility of preferred embodiments of the present disclosure, in which similar elements are referred to by common reference numerals. In order to better appreciate how the above-recited and other advantages and objects of the present disclosures are obtained, a more particular description of the present disclosures briefly described above will be rendered by reference to specific embodiments thereof, which are illustrated in the accompanying drawings. Understanding that these drawings depict only typical embodiments of the disclosure and are not therefore to be considered limiting of its scope, the disclosure will be described and explained with additional specificity and detail through the use of the accompanying drawings in which:

FIGS. 1A-C are block diagrams of illustrative augmented reality systems constructed, according to some embodiments of the present disclosure.

FIG. 2 illustrates an example architecture for implementing the AR recording process according to some embodiments.

FIG. 3 shows a flowchart of an approach to generate AR recordings using warping, according to some embodiments.

FIGS. 4A-G provide an illustration of the warping approach to implement AR recording, according to some embodiments.

FIG. 5 illustrates an architecture for implementing this AR recording process, according to some embodiments.

FIG. 6 shows a flowchart of an approach to generate AR recordings, according to some embodiments.

FIGS. 7A-F illustrate an approach to generate AR recordings, according to some embodiments.

FIG. 8 illustrates an example GPU processing cycle, according to some embodiments.

FIG. 9A illustrates an example GPU processing cycle, according to some embodiments.

FIG. 9B shows a flow chart of an approach to implement a GPU processing cycle, according to some embodiments.

FIGS. 10A-B illustrate example alternative GPU processing cycle, according to some embodiments.

FIG. 10C shows a flow chart of an approach to implement an alternative GPU processing cycle, according to some embodiments.

FIG. 11 is a block diagram of an illustrative computing system, according to some embodiments.

DETAILED DESCRIPTION

The present disclosure is directed to an approach to implement augmented reality (AR) recordings using “third eye” virtual content rendering. According to some embodiments, instead of re-using rendered virtual content from a perspective of a user's eyes for AR recordings, additional virtual content is rendered from a perspective of the camera, and that additional virtual content is combined with the camera recording of a real world to form the AR recording. In some embodiments, gaps in normal GPU processing may be leveraged to perform the additional rendering to generate the additional virtual content rendering.

This disclosure will first provide a description of an illustrative AR system with which some embodiments of the disclosure may be practiced, followed by a description of one or more embodiments of an improved process and mechanism to generate AR content for recordings.

Illustrative Augmented Reality System

The description that follows pertains to an illustrative augmented reality (AR) system (which may be referred to herein as any of “augmented reality system”, “AR system”, or “mixed reality system) with which the disclosure may be practiced. However, it is to be understood that the disclosure also lends itself to applications in other types of augmented reality and virtual reality systems, and therefore the disclosure is not to be limited to only the illustrative system disclosed herein.

FIG. 1A is a block diagram illustrating an augmented reality (AR) system 100 a , according to one embodiment. The AR system 100 a may be operated in conjunction with an augmented reality control system 101 a , providing images of virtual objects intermixed with physical objects in a field of view of an end user 50 . This approach employs one or more at least partially transparent surfaces through which the ambient environment in the field of view of the end user 50 can be seen and on to which the AR system 100 a produces images of virtual objects.

For AR applications, it may be desirable to spatially position various virtual objects relative to physical objects in the field of view of the end user 50 . Virtual objects, also referred to herein as virtual tags or tag or call outs, may take any of a large variety of forms, having any variety of data, information, concept, or logical construct capable of being represented as an image. Non-limiting examples of virtual objects may include: a virtual text object, a virtual numeric object, a virtual alphanumeric object, a virtual tag object, a virtual field object, a virtual chart object, a virtual map object, a virtual instrumentation object, or a virtual visual representation of a physical object.

The AR system 100 a includes a frame structure 102 worn by the end user 50 , a display system 104 carried by the frame structure 102 , such that the display system 104 is positioned in front of eyes of the end user 50 , and a speaker 106 incorporated into or connected to the display system 104 . In the illustrated embodiment, the speaker 106 is carried by the frame structure 102 , such that the speaker 106 is positioned adjacent (in or around) an ear canal of the end user 50 (e.g., an earbud or headphone).

The display system 104 is designed to present the eyes of the end user 50 with photo-based radiation patterns that can be comfortably perceived as augmentations to physical reality with both two-dimensional and three-dimensional content. The display system 104 presents a sequence of frames at high frequency that provides the perception of a single coherent scene. To this end, the display subsystem 104 includes a projection subsystem 108 and a partially transparent display screen on which the projection subsystem 108 projects images. The display screen is positioned in the field of view of the end user 50 between the eyes of the end user 50 and an ambient (e.g., physical) environment.

In some embodiments, the projection subsystem 108 takes the form of a scan-based projection device and the display screen takes the form of a waveguide-based display into which the scanned light from the projection subsystem 108 is injected to produce images at single optical viewing distance closer than infinity (e.g., arm's length), images at multiple, discrete optical viewing distances or focal planes, and/or image layers stacked at multiple viewing distances or focal planes to represent volumetric 3D objects. These layers in the light field may be stacked closely enough together to appear continuous to the human visual subsystem (e.g., one layer is within a cone of confusion of an adjacent layer). Additionally or alternatively, picture elements may be blended across two or more layers to increase perceived continuity of transition between layers in the light field, even if those layers are more sparsely stacked (e.g., one layer is outside the cone of confusion of an adjacent layer). The display system 104 may be monocular or binocular. The scanning assembly includes one or more light sources that produce the light beam (e.g., emits light of different colors in defined patterns). The light source may take any of a large variety of forms, for instance, a set of RGB sources (e.g., laser diodes capable of outputting red, green, and blue light) operable to respectively produce red, green, and blue coherent collimated light according to defined pixel patterns specified in respective frames of pixel information or data. Laser light provides high color saturation and is highly energy efficient. The optical coupling subsystem includes an optical waveguide input apparatus, such as for instance, one or more reflective surfaces, diffraction gratings, mirrors, dichroic mirrors, or prisms to optically couple light into the end of the display screen. The optical coupling subsystem further includes a collimation element that collimates light from the optical fiber. Optionally, the optical coupling subsystem includes an optical modulation apparatus configured for converging the light from the collimation element towards a focal point in the center of the optical waveguide input apparatus, thereby allowing the size of the optical waveguide input apparatus to be minimized. Thus, the display system 104 generates a series of synthetic image frames of pixel information that present an undistorted image of one or more virtual objects to the user. Further details describing display systems are provided in U.S. Provisional Patent Application Ser. No. 61/801,219, filed on Mar. 15, 2013, entitled “Display Subsystem and Method”, which has been converted to U.S. Utility patent application Ser. No. 14/212,961, filed on Mar. 14, 2014, and U.S. patent application Ser. No. 14/331,218, entitled “Planar Waveguide Apparatus With Diffraction Element(s) and Subsystem Employing Same” filed on Jul. 14, 2014, all of which are expressly incorporated herein by reference in their entirety.

The AR system 100 a further includes one or more sensors mounted to the frame structure 102 for detecting position and movement of a head of the end user 50 and/or eye position and inter-ocular distance of the end user 50 . Such sensor(s) may include image capture devices (such as camera 144 ), microphones, inertial measurement units (IMUs), accelerometers, compasses, GPS units, radio devices, and/or gyros. For example, in one embodiment, the AR system 100 a includes a head worn transducer subsystem that includes one or more inertial transducers to capture inertial measures indicative of movement of the head of the end user 50 . Such devices may be used to sense, measure, or collect information about the head movements of the end user 50 . For instance, these devices may be used to detect measurement movements, speeds, acceleration, and/or positions of the head of the end user 50 .

The AR system 100 a includes one or more forward facing cameras 144 a (hereinafter referred to as “ camera 144 a ”). The camera 144 a may be employed for any number of purposes, such as recording of images/video from the forward direction of the AR system 100 a . In addition, the camera 144 a may be used to capture information about the environment in which the end user 50 is located, such as information indicative of distance, orientation, and/or angular position of the end user 50 with respect to that environment and specific objects in that environment.

In some embodiments, for example as illustrated in FIG. 1A , the camera 144 may be physically attached to the frame structure 102 , and in other embodiments, for example as illustrated in FIG. 1B , camera 144 b may be physically remote from the frame structure 102 . For example, the camera 144 b may be placed at wall or ceiling locations having a clear view of the room that the user is located in, where the camera 144 b is placed at known coordinate locations within the room. Any type of camera 144 ( camera 144 collectively referring to camera

144 a , 144 b , or any other suitable type of image acquisition device) may be used to perform AR recording according to the present embodiments.

The AR system 100 a may further include rearward facing cameras to track angular position (the direction in which the eye or eyes are pointing), blinking, and depth of focus (by detecting eye convergence) of the eyes of the end user 50 . Such eye tracking information may, for example, be discerned by projecting light at the end user's eyes, and detecting the return or reflection of at least some of that projected light.

The augmented reality system 100 a further include a control subsystem 101 a that may take any of a large variety of forms. The control subsystem 101 a includes a number of controllers, for instance one or more microcontrollers, microprocessors or central processing units (CPUs), digital signal processors, graphics processing units (GPUs), other integrated circuit controllers, such as application specific integrated circuits (ASICs), programmable gate arrays (PGAs), for instance field PGAs (FPGAs), and/or programmable logic controllers (PLUs). The control subsystem may include a digital signal processor (DSP), a central processing unit (CPU) 150 , a graphics processing unit (GPU) 152 , and one or more frame buffers 154 . Although FIG. 1A illustrates one frame buffer 154 , the control subsystem 101 a may include more than one frame buffer 154 . The control subsystem 101 may be communicatively coupled to the frame structure 102 , for example, via a wired or wireless connection.

The CPU 150 controls overall operation of the AR system 100 a . The CPU 150 may read from read only memory (ROM) and write into and/or reads out of random access memory (RAM).

The GPU 152 renders frames (e.g., translating a three-dimensional scene into a two-dimensional image) and stores these frames in the frame buffer(s) 154 . While not illustrated, one or more additional integrated circuits may control the writing into and/or reading out of frames from the frame buffer(s) 154 and operation of the scanning device of the display subsystem 104 . Reading into and/or out of the frame buffer 154 may employ dynamic addressing, for instance, where frames are over-rendered. The GPU 152 may accesses three dimensional (3D) data of one or more scenes for rendering frames, as well as synthetic sound data associated with virtual sound sources contained within the 3D scenes from a 3D database 160 .

The augmented reality system 100 a further includes a user orientation detection module 148 . The user orientation module 148 detects the instantaneous position of a head of the end user 50 and may predict the position of the head of the end user 50 based on position data received from the sensor(s). The user orientation module 148 also tracks eyes of the end user 50 , and in particular the direction and/or distance at which the end user 50 is focused based on the tracking data received from the sensor(s).

The various components of the AR system 100 a may be physically contained in a distributed subsystem. For example, the AR system 100 a includes a local processing and data module operatively coupled, such as by a wired lead or wireless connectivity, to a portion of the display subsystem 104 . The local processing and data module may be mounted in a variety of configurations, such as fixedly attached to the frame structure 102 , fixedly attached to a helmet or hat, embedded in headphones, removably attached to a torso of the end user 50 , or removably attached to a hip of the end user 50 in a belt-coupling style configuration. The AR system 100 a further includes a remote processing module and remote data repository operatively coupled, such as by a wired lead or wireless connectivity to the local processing and data module, such that these remote modules are operatively coupled to each other and available as resources to the local processing and data module. The local processing and data module may comprise a power-efficient processor or controller, as well as digital memory, such as flash memory, both of which may be utilized to assist in the processing, caching, and storage of data captured from the sensors and/or acquired and/or processed using the remote processing module and/or remote data repository, possibly for passage to the display subsystem 104 after such processing or retrieval. The remote processing module may comprise one or more relatively powerful processors or controllers configured to analyze and process data and/or image information. The remote data repository may comprise a relatively large-scale digital data storage facility, which may be available through the internet or other networking configuration in a “cloud” resource configuration. In one embodiment, all data is stored and all computation is performed in the local processing and data module, allowing fully autonomous use from any remote modules. The couplings between the various components described above may include one or more wired interfaces or ports for providing wires or optical communications, or one or more wireless interfaces or ports, such as via RF, microwave, and IR for providing wireless communications. In some implementations, all communications may be wired, while in other implementations all communications may be wireless, with the exception of the optical fiber(s).

FIG. 1C is a block diagram illustrating an augmented reality (AR) system 100 b , according to one embodiment. The AR system 100 b has a control subsystem 101 b . As was previously described above with respect to the control subsystem 101 a , the control system 101 b also includes a projection subsystem 108 , one or more frame buffers 154 , a GPU

152 , 3D database(s) 160 , image/ video data 171 , and CPU 150 . The control subsystem 101 b further includes one or more inertial measurement unit (IMU) 162 , a compositor 164 , a media encoder 166 , and recording database 168 .

The projection subsystem 108 provides images of virtual objects intermixed with physical objects in a field of view of the end user 50 , where the images may be stored in one or more frame buffers 154 .

The IMU 162 corresponds to a sensor/measurement apparatus that identifies pose data for the AR system, including for example, pose data corresponding to six degrees of freedom having three degrees pertaining to a gyroscope and three degrees pertaining to an accelerometer. The IMU 162 detects an instantaneous position of the head of the end user 50 . The IMU 162 may detect the instantaneous position of the head of the end user 50 based on a known relationship between the head of the end user 50 and the frame structure 102 on which the IMU 162 may reside. In some embodiments, the IMU 162 may predict a position of the head of the end user 50 based on previously detected positions of the head of the end user 50 . The IMU 162 may also track the eyes of the end user 50 , and in particular a direction and/or distance at which the end user 50 is focused on. The IMU 162 further detects instantaneous position of the camera 144 . In some embodiments, the IMU 162 extrapolates an instantaneous position of the camera 144 from the detected instantaneous position based on a known offset between the camera 144 and the frame structure 102 .

In some embodiments, instead of re-using rendered virtual content from a perspective of the user's eyes for AR recordings, virtual content is instead rendered from a perspective of the camera. That virtual content that is rendered is combined with the camera recording to form the AR recording. The CPU 150 controls the operation of the control subsystem 101 b such that the GPU 152 will render virtual content from the perspective of the camera 144 (using pose data for the camera 144 that was generated by the IMU 162 ).

The compositor 164 combines rendered virtual content with the camera recording content. The compositor 164 operates by superimposing the rendered virtual content from the GPU 152 with the recorded content from the camera 144 b , according to the correct relative positioning of the real content in the camera's image frame that is lined up properly in the appropriate coordinate space with the rendered virtual content from the GPU 152 .

The media encoder 166 takes the output data from the compositor 164 , and generates one or more image frame(s) into the desired media format type. The image frames may be generated as a single image screenshot and/or a stream/sequence of image frames to form a video file. The encoded media data is then stored within the recording database 168 .

AR Recording

The description that follows pertains to an approach for generating AR recordings, according to some embodiments. Instead of re-using rendered virtual content from a perspective of a user's eyes for AR recordings, additional virtual content is rendered from a perspective of a camera, and that additional virtual content is combined with a camera recording to form an AR recording.

Users of AR systems often seek to preserve a recording of their AR experience, where the recording includes both real world objects and virtual object (or content) produced by the AR system. A problem that arises is that image artifacts may be created due to the fact that the virtual objects are rendered from the perspective of the user's eyes, whereas the camera is recording the real objects from the perspective of the camera which is different from the perspective of the user's eyes.

FIG. 2 illustrates an example architecture for implementing an AR recording process using warping, according to some embodiments. In some embodiments, the warping process may result in image artifacts. At a given point in time, a request will be received to generate virtual image data for display on an AR device. At (1), the camera 144 captures a camera image frame and transmits the captured camera image frame to the compositor 164 . In some embodiments, the camera 144 may be attached to the frame structure 102 . In some embodiments, the camera 144 may be mounted at a determined location and orientation within the environment.

At (2), an IMU 162 captures/identifies pose data for the head and/or the eyes of the end user 50 and captures pose data for the camera 144 and transmits the captured pose data to the compositor 164 . In some embodiments, the IMU 162 captures/identifies pose data for the AR system, generally to determine the pose of the user's eyes or head, as well as the camera. As used herein, the term “pose” includes, for example, (X, Y, Z) coordinate locations as well as an angular direction/location/orientation. In some embodiments in which the camera 144 is physically attached to the frame structure 102 , the pose data for the camera 144 may be extrapolated based upon a known and fixed offset between the camera 144 and the frame structure 102 . In some embodiments in which the camera 144 is physically remote from the frame structure 102 , the pose data for the camera 144 can include fixed location/orientation data that is pre-designated, for example, within a system coordinate space.

At (3), the compositor 164 sends the pose data for the head and/or eyes of the end user 50 and the pose data for the camera 144 to the GPU 152 . The GPU 152 generates and renders virtual content based on the pose data for the head and/or the eyes of the end user 50 . In some embodiments, this rendering action generates the virtual content so that an image of the virtual content is generated from the perspective of the head and/or the eyes of the end user 50 . A 3D database 160 is employed to provide information about the virtual objects in the environme

CLAIMS

Claims ( 20 )

What is claimed is:

1. A method for generating an image recording, comprising

capturing a camera image frame with a camera;

obtaining pose data that corresponds to the camera image frame, the pose data comprising user pose data and camera pose data, wherein the user pose data is from a different perspective from the camera pose data;

rendering user pose virtual content for immediate display to a user in an augmented reality (AR) device, wherein the user pose virtual content corresponds to first virtual content from the user's perspective;

warping the user pose virtual content based on the camera pose data to generate camera pose virtual content, where the camera pose virtual content corresponds to second virtual content from the camera's perspective; and

composing the image recording by combining the camera image frame with the camera pose virtual content,

wherein a single graphics processing unit (GPU) renders the user pose virtual content and warps the user pose virtual content.

2. The method of claim 1 , further comprising:

receiving a request to display a display image frame; and

capturing the camera image frame with the camera in response to the request to display the display image frame.

3. The method of claim 1 , further comprising storing the image recording for later display.

4. The method of claim 1 , wherein a single GPU processing cycle includes processing to: render the user pose virtual content based on the user pose data, and warp the user pose virtual content based on the camera pose data to generate the camera pose virtual content.

5. The method of claim 4 , wherein the user pose virtual content based on the user pose data and the camera pose virtual content based on the camera pose data generated during the single GPU processing cycle is for different frames.

6. The method of claim 1 , wherein a compositor generates the image recording by combining the camera image frame with the camera pose virtual content.

7. The method of claim 1 , further comprising a media encoder encoding one or more image frames that combine the camera image frame with the camera pose virtual content.

8. The method of claim 7 , wherein the one or more image frames encoded by the media encoder comprises at least one of a single image frame or a stream of multiple image frames.

9. A computer program product embodied on a non-transitory computer readable medium, the non-transitory computer readable medium having stored thereon a sequence of instructions which, when executed by a processor causes the processor to execute a method for generating an image recording comprising:

capturing a camera image frame with a camera;

obtaining pose data that corresponds to the camera image frame, the pose data comprising user pose data and camera pose data, wherein the user pose data is from a different perspective from the camera pose data;

rendering user pose virtual content for immediate display to a user in an augmented reality (AR) device, wherein the user pose virtual content corresponds to first virtual content from the user's perspective;

warping the user pose virtual content based on the camera pose data to generate camera pose virtual content, where the camera pose virtual content corresponds to second virtual content from the camera's perspective; and

composing the image recording by combining the camera image frame with the camera pose virtual content,

wherein a single graphics processing unit (GPU) renders the user pose virtual content and warps the user pose virtual content.

10. The computer program product of claim 9 , the method further comprising:

receiving a request to display a display image frame; and

capturing the camera image frame with the camera in response to the request to display the display image frame.

11. The computer program product of claim 9 , the method further comprising storing the image recording for later display.

12. The computer program product of claim 9 , wherein a single GPU processing cycle includes processing to: render the user pose virtual content based on the user pose data, and warp the user pose virtual content based on the camera pose data to generate the camera pose virtual content.

13. The computer program product of claim 12 , wherein the user pose virtual content based on the user pose data and the camera pose virtual content based on the camera pose data generated during the single GPU processing cycle is for different frames.

14. The computer program product of claim 9 , wherein a compositor generates the image recording by combining the camera image frame with the camera pose virtual content.

15. The computer program product of claim 9 , wherein a media encoder encodes one or more image frames that combine the camera image frame with the camera pose virtual content.

16. The computer program product of claim 15 , wherein the one or more image frames encoded by the media encoder comprises at least one of a single image frame or a stream of multiple image frames.

17. A system for generating an image recording, comprising

an augmented reality display device that displays three dimensional content;

a camera that captures a camera image frame;

a single inertial measurement unit (IMU) that obtains pose data corresponding to the camera image frame, the pose data comprising user pose data and camera pose data, wherein the user pose data is from a different perspective from the camera pose data;

one or more graphics processing units (GPU) that:

renders user pose virtual content for immediate display to a user in the augmented reality (AR) device, wherein the user pose virtual content corresponds to first virtual content from the user's perspective, and

warps the user pose virtual content based on the camera pose data to generate camera pose virtual content, where the camera pose virtual content corresponds to second virtual content from the camera's perspective; and

a compositor that composes the image recording by combining the camera image frame with the camera pose virtual content.

18. The system of claim 17 , further comprising a media encoder that encodes one or more image frames that combine the camera image frame with the camera pose virtual content.

19. The system of claim 18 , wherein the one or more image frames encoded by the media encoder comprises at least one of a single image frame or a stream of multiple image frames.

20. The system of claim 17 , wherein a single GPU processing cycle includes processing to: render the user pose virtual content based on the user pose data, and warp the user pose virtual content based on the camera pose data to generate the camera pose virtual content.

US17/081,081

2017-03-17

2020-10-27

Technique for recording augmented reality data

Active

US11380068B2

( en )

Priority Applications (1)

Application Number

Priority Date

Filing Date

Title

US17/081,081

US11380068B2

( en )

2017-03-17

2020-10-27

Technique for recording augmented reality data

Applications Claiming Priority (4)

Application Number

Priority Date

Filing Date

Title

US201762472923P

2017-03-17

2017-03-17

US15/924,144

US10573078B2

( en )

2017-03-17

2018-03-16

Technique for recording augmented reality data

US16/743,207

US10861244B2

( en )

2017-03-17

2020-01-15

Technique for recording augmented reality data

US17/081,081

US11380068B2

( en )

2017-03-17

2020-10-27

Technique for recording augmented reality data

Related Parent Applications (1)

Application Number

Title

Priority Date

Filing Date

US16/743,207

Continuation

US10861244B2

( en )

2017-03-17

2020-01-15

Technique for recording augmented reality data

Publications (2)

Publication Number

Publication Date

US20210043008A1

US20210043008A1 ( en )

2021-02-11

US11380068B2

true

US11380068B2 ( en )

2022-07-05

Family

ID=63520263

Family Applications (3)

Application Number

Title

Priority Date

Filing Date

US15/924,144

Active

2038-03-19

US10573078B2

( en )

2017-03-17

2018-03-16

Technique for recording augmented reality data

US16/743,207

Active

US10861244B2

( en )

2017-03-17

2020-01-15

Technique for recording augmented reality data

US17/081,081

Active

US11380068B2

( en )

2017-03-17

2020-10-27

Technique for recording augmented reality data

Family Applications Before (2)

Application Number

Title

Priority Date

Filing Date

US15/924,144

Active

2038-03-19

US10573078B2

( en )

2017-03-17

2018-03-16

Technique for recording augmented reality data

US16/743,207

Active

US10861244B2

( en )

2017-03-17

2020-01-15

Technique for recording augmented reality data

Country Status (9)

Country

Link

US

( 3 )

US10573078B2

( en )

EP

( 2 )

EP4329305A3

( en )

JP

( 3 )

JP7007396B2

( en )

KR

( 3 )

KR102517876B1

( en )

CN

( 1 )

CN110402415A

( en )

AU

( 2 )

AU2018234929B2

( en )

CA

( 1 )

CA3055219A1

( en )

IL

( 3 )

IL290001B2

( en )

WO

( 1 )

WO2018170490A1

( en )

Cited By (1)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US12608884B2

( en )

2024-03-12

2026-04-21

International Business Machines Corporation

Selective volumetric boundary recording

Families Citing this family (38)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US10979685B1

( en )

2017-04-28

2021-04-13

Apple Inc.

Focusing for virtual and augmented reality systems

CN110574375B

( en )

2017-04-28

2023-06-02

苹果公司

Video pipeline

US10861142B2

( en )

2017-07-21

2020-12-08

Apple Inc.

Gaze direction-based adaptive pre-filtering of video data

KR102481884B1

( en )

*

2017-09-22

2022-12-28

삼성전자주식회사

Method and apparatus for displaying a virtual image

US11128783B2

( en )

*

2018-03-07

2021-09-21

Disney Enterprises, Inc.

Systems and methods for tracking objects in a field of view

EP3782010A1

( en )

*

2018-04-20

2021-02-24

PCMS Holdings, Inc.

Method and system for gaze-based control of mixed reality content

EP3837674A4

( en )

2018-08-13

2022-05-18

Magic Leap, Inc.

CROSS REALITY SYSTEM

US11227435B2

( en )

2018-08-13

2022-01-18

Magic Leap, Inc.

Cross reality system

JP7503542B2

( en )

2018-10-05

2024-06-20

マジック リープ， インコーポレイテッド

Rendering location-specific virtual content anywhere

CN114600064B

( en )

2019-10-15

2026-04-24

奇跃公司

Cross-reality system with location services

CN114586071B

( en )

2019-10-15

2026-03-03

奇跃公司

Cross-reality system supporting multiple device types

US11632679B2

( en )

2019-10-15

2023-04-18

Magic Leap, Inc.

Cross reality system with wireless fingerprints

CN114846434A

( en )

*

2019-10-25

2022-08-02

奇跃公司

Non-uniform stereoscopic rendering

CN114616509B

( en )

2019-10-31

2024-12-27

奇跃公司

Cross-reality system with quality information about persistent coordinate frames

US11386627B2

( en )

2019-11-12

2022-07-12

Magic Leap, Inc.

Cross reality system with localization service and shared location-based content

WO2021118962A1

( en )

*

2019-12-09

2021-06-17

Magic Leap, Inc.

Cross reality system with simplified programming of virtual content

WO2021163300A1

( en )

2020-02-13

2021-08-19

Magic Leap, Inc.

Cross reality system with map processing using multi-resolution frame descriptors

WO2021163295A1

( en )

2020-02-13

2021-08-19

Magic Leap, Inc.

Cross reality system with prioritization of geolocation information for localization

US11410395B2

( en )

2020-02-13

2022-08-09

Magic Leap, Inc.

Cross reality system with accurate shared maps

EP4111425B1

( en )

2020-02-26

2026-04-01

Magic Leap, Inc.

Cross reality system with fast localization

KR102650385B1

( en )

*

2020-04-09

2024-03-25

주식회사 피아몬드

Method and system for selecting content to expose through space of virtual world

US11328094B2

( en )

2020-04-09

2022-05-10

Piamond Corp.

Method and system for constructing virtual space

US11521359B2

( en )

2020-04-17

2022-12-06

Apple Inc.

Extended reality recorder

US11900547B2

( en )

2020-04-29

2024-02-13

Magic Leap, Inc.

Cross reality system for large scale environments

CN113961280B

( en )

*

2020-07-21

2023-11-24

抖音视界有限公司

View display method and device, electronic equipment and computer readable storage medium

US12051239B2

( en )

2020-08-11

2024-07-30

Disney Enterprises, Inc.

Item location tracking via image analysis and projection

CN112819967B

( en )

*

2021-01-14

2024-07-30

京东方科技集团股份有限公司

Display method, device and system, storage medium and display

US11651544B2

( en )

*

2021-04-30

2023-05-16

Varjo Technologies Oy

Systems and methods employing multiple graphics processing units for producing images

US20220351411A1

( en )

*

2021-04-30

2022-11-03

Varjo Technologies Oy

Display apparatus and method employing reprojection based on marker pose

DE102021117453B3

( en )

2021-07-06

2022-10-20

Holoride Gmbh

Method for operating data glasses in a motor vehicle while driving, correspondingly operable data glasses, processor circuit and motor vehicle

KR102625729B1

( en )

*

2021-10-05

2024-01-16

가온그룹 주식회사

method of rendering AR contents by use of clone object

US11694409B1

( en )

*

2021-12-08

2023-07-04

Google Llc

Augmented reality using a split architecture

US12062145B2

( en )

2022-02-01

2024-08-13

Samsung Electronics Co., Ltd.

System and method for three-dimensional scene reconstruction and understanding in extended reality (XR) applications

US20230342877A1

( en )

*

2022-04-20

2023-10-26

Snap Inc.

Cached cloud rendering

US12482198B2

( en )

2022-08-08

2025-11-25

Samsung Electronics Co., Ltd.

Real-time photorealistic view rendering on augmented reality (AR) device

KR20240026314A

( en )

*

2022-08-18

2024-02-28

한국전자기술연구원

Video-based augmented reality content providing system

US12626392B2

( en )

*

2023-07-10

2026-05-12

Htc Corporation

Pose calculating apparatus and method

CN117596373B

( en )

*

2024-01-17

2024-04-12

淘宝（中国）软件有限公司

Method and electronic device for displaying information based on dynamic digital human image

Citations (14)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

JP2002271693A

( en )

2001-03-13

2002-09-20

Canon Inc

Image processing apparatus, image processing method, and control program

US20060146138A1

( en )

2004-12-17

2006-07-06

Jun Xin

Method and system for synthesizing multiview videos

US20100033484A1

( en )

2006-12-05

2010-02-11

Nac-Woo Kim

Personal-oriented multimedia studio platform apparatus and method for authorization 3d content

US20130013664A1

( en )

2011-07-05

2013-01-10

Cisco Technology, Inc.

In-Network Middlebox Compositor For Distributed Virtualized Applications

US20130249900A1

( en )

2012-03-23

2013-09-26

Kyonggi University Industry & Academia Cooperation Foundation

Method and apparatus for processing media file for augmented reality service

US20130311548A1

( en )

2012-05-15

2013-11-21

Nvidia Corporation

Virtualized graphics processing for remote display

US20140354515A1

( en )

2013-05-30

2014-12-04

Oculus Vr, Llc

Perception based predictive tracking for head mounted displays

KR20150111999A

( en )

2013-01-30

2015-10-06

퀄컴 인코포레이티드

Real-time 3d reconstruction with power efficient depth sensor usage

US20160210783A1

( en )

2015-01-20

2016-07-21

Arthur Charles Tomlin

Holographic bird's eye view camera

US9417452B2

( en )

2013-03-15

2016-08-16

Magic Leap, Inc.

Display system and method

US20160246605A1

( en )

2012-11-14

2016-08-25

Facebook, Inc.

Systems and Methods for Optimizing Order of Image Transformations

KR20160112898A

( en )

2015-03-20

2016-09-28

한국과학기술원

Method and apparatus for providing dynamic service based augmented reality

US9671566B2

( en )

2012-06-11

2017-06-06

Magic Leap, Inc.

Planar waveguide apparatus with diffraction element(s) and system employing same

US20170200254A1

( en )

*

2016-01-07

2017-07-13

Microsoft Technology Licensing, Llc

View rendering from multiple server-side renderings

Family Cites Families (11)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US573078A

( en )

*

1896-12-15

Wattmeter

US20140176591A1

( en )

*

2012-12-26

2014-06-26

Georg Klein

Low-latency fusing of color image data

US9269003B2

( en )

*

2013-04-30

2016-02-23

Qualcomm Incorporated

Diminished and mediated reality effects from reconstruction

CN105408937B

( en )

*

2013-05-02

2018-02-13

高通股份有限公司

Methods for Facilitating Initialization of Computer Vision Applications

US9578016B2

( en )

*

2014-07-17

2017-02-21

Riverbed Technology, Inc.

Optimizing secure communications between a client authenticating server and a mobile client

US9240069B1

( en )

*

2015-06-30

2016-01-19

Ariadne's Thread (Usa), Inc.

Low-latency virtual reality display system

CN106228591B

( en )

*

2016-07-12

2020-06-26

江苏奥格视特信息科技有限公司

Virtual reality ultrahigh-speed real-time rendering method

CN106231205B

( en )

*

2016-08-10

2019-07-30

苏州黑盒子智能科技有限公司

Augmented reality mobile terminal

CN106131536A

( en )

*

2016-08-15

2016-11-16

万象三维视觉科技（北京）有限公司

A kind of bore hole 3D augmented reality interactive exhibition system and methods of exhibiting thereof

CN106327584B

( en )

*

2016-08-24

2020-08-07

深圳市瑞云科技有限公司

Image processing method and device for virtual reality equipment

CN106502427B

( en )

*

2016-12-15

2023-12-01

北京国承万通信息科技有限公司

Virtual reality system and scene presenting method thereof

2018

2018-03-16

CA

CA3055219A

patent/CA3055219A1/en

active

Pending

2018-03-16

EP

EP24151535.2A

patent/EP4329305A3/en

active

Pending

2018-03-16

JP

JP2019550199A

patent/JP7007396B2/en

active

Active

2018-03-16

IL

IL290001A

patent/IL290001B2/en

unknown

2018-03-16

EP

EP18768170.5A

patent/EP3596542B1/en

active

Active

2018-03-16

CN

CN201880018314.0A

patent/CN110402415A/en

active

Pending

2018-03-16

KR

KR1020227010984A

patent/KR102517876B1/en

active

Active

2018-03-16

KR

KR1020237010823A

patent/KR102598116B1/en

active

Active

2018-03-16

IL

IL297863A

patent/IL297863B2/en

unknown

2018-03-16

AU

AU2018234929A

patent/AU2018234929B2/en

active

Active

2018-03-16

KR

KR1020197030295A

patent/KR102384232B1/en

active

Active

2018-03-16

WO

PCT/US2018/023025

patent/WO2018170490A1/en

not_active

Ceased

2018-03-16

US

US15/924,144

patent/US10573078B2/en

active

Active

2019

2019-08-19

IL

IL268777A

patent/IL268777B/en

unknown

2020

2020-01-15

US

US16/743,207

patent/US10861244B2/en

active

Active

2020-10-27

US

US17/081,081

patent/US11380068B2/en

active

Active

2022

2022-01-06

JP

JP2022001002A

patent/JP7068562B2/en

active

Active

2022-04-28

JP

JP2022073937A

patent/JP7133115B2/en

active

Active

2022-09-26

AU

AU2022241459A

patent/AU2022241459A1/en

not_active

Abandoned

Patent Citations (15)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

JP2002271693A

( en )

2001-03-13

2002-09-20

Canon Inc

Image processing apparatus, image processing method, and control program

US20060146138A1

( en )

2004-12-17

2006-07-06

Jun Xin

Method and system for synthesizing multiview videos

US20100033484A1

( en )

2006-12-05

2010-02-11

Nac-Woo Kim

Personal-oriented multimedia studio platform apparatus and method for authorization 3d content

US20130013664A1

( en )

2011-07-05

2013-01-10

Cisco Technology, Inc.

In-Network Middlebox Compositor For Distributed Virtualized Applications

US20130249900A1

( en )

2012-03-23

2013-09-26

Kyonggi University Industry & Academia Cooperation Foundation

Method and apparatus for processing media file for augmented reality service

US20130311548A1

( en )

2012-05-15

2013-11-21

Nvidia Corporation

Virtualized graphics processing for remote display

US9671566B2

( en )

2012-06-11

2017-06-06

Magic Leap, Inc.

Planar waveguide apparatus with diffraction element(s) and system employing same

US20160246605A1

( en )

2012-11-14

2016-08-25

Facebook, Inc.

Systems and Methods for Optimizing Order of Image Transformations

KR20150111999A

( en )

2013-01-30

2015-10-06

퀄컴 인코포레이티드

Real-time 3d reconstruction with power efficient depth sensor usage

US9417452B2

( en )

2013-03-15

2016-08-16

Magic Leap, Inc.

Display system and method

US20140354515A1

( en )

2013-05-30

2014-12-04

Oculus Vr, Llc

Perception based predictive tracking for head mounted displays

WO2016118371A1

( en )

2015-01-20

2016-07-28

Microsoft Technology Licensing, Llc

Mixed reality system

US20160210783A1

( en )

2015-01-20

2016-07-21

Arthur Charles Tomlin

Holographic bird's eye view camera

KR20160112898A

( en )

2015-03-20

2016-09-28

한국과학기술원

Method and apparatus for providing dynamic service based augmented reality

US20170200254A1

( en )

*

2016-01-07

2017-07-13

Microsoft Technology Licensing, Llc

View rendering from multiple server-side renderings

Non-Patent Citations (26)

* Cited by examiner, † Cited by third party

Title

Bautin, M., et al., " Graphic Engine Resource Mangement, " Multimedia Computing and Networking, SPIE vol. 6818, dated 2008 (12 pages).

Extended European Search Report for EP Patent Appln. No. 18768170.5 dated Dec. 12, 2019.

Foreign Exam Report for AU Patent Appln. No. 2018234929 dated Jul. 8, 2021.

Foreign Exam Report for EP Patent Appln. No. 18768170.5 dated Jan. 14, 2022.

Foreign Exam Report for IN Patent Appln. No. 201947040460 dated Feb. 2, 2022.

Foreign Examination Report for EP Patent Appln. No. 18768170.5 dated Dec. 15, 2020.

Foreign FOA for KR Patent Appln. No. 10-2019-7030295 dated Nov. 8, 2021.

Foreign NOA for IL Patent Appln. No. 268777 dated Oct. 20, 2021.

Foreign NOA for JP Patent Appln. No. 2019-550199 dated Dec. 8, 2021.

Foreign NOA for JP Patent Appln. No. 2022-1002 dated Apr. 1, 2022.

Foreign NOA for KR Patent Appln. No. 10-2019-7030295 dated Jan. 3, 2022.

Foreign OA for CN Patent Application No. 201880018314.0 dated Nov. 9, 2020.

Foreign OA for CN Patent Appln. No. 201880018314.0 dated Mar. 4, 2021.

Foreign OA for JP Patent Appln. No. 2019-550199 dated Jul. 27, 2021.

Foreign OA for KR Patent Appln. No. 10-2019-7030295 dated Jul. 12, 2021.

Foreign Office Action for CN Patent Appln. No. 201880018314.0, dated Apr. 29, 2020.

Foreign Office Action Response for CN Patent Appln. No. 201880018314.0 dated Sep. 10, 2020.

Foreign Response for CN Patent Appln. No. 201880018314.0 dated Jun. 18, 2021.

Foreign Response for EP Patent Appln. No. 18768170.5 dated Apr. 21, 2021.

Foreign Response for EP Patent Appln. No. 18768170.5 dated May 24, 2022.

Foreign Response for JP Patent Appln. No. 2019-550199 dated Oct. 19, 2021.

Foreign Response for KR Patent Appln. No. 10-2019-7030295 dated Dec. 9, 2021.

Foreign Response for KR Patent Appln. No. 10-2019-7030295 dated Sep. 13, 2021.

Non-Final Office Action dated Jun. 3, 2019 for U.S. Appl. No. 15/924,144.

Notice of Allowance for U.S. Appl. No. 16/743,207 dated Aug. 26, 2020.

PCT International Search Report and Written Opinion for International Appln. No. PCT/US18/23025, Applicant Magic Leap, Inc., forms PCT/ISA/210, 220, and 237, dated Jun. 11, 2018 (10 pages).

Cited By (1)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US12608884B2

( en )

2024-03-12

2026-04-21

International Business Machines Corporation

Selective volumetric boundary recording

Also Published As

Publication number

Publication date

IL297863B2

( en )

2024-01-01

EP4329305A3

( en )

2024-05-22

AU2018234929A1

( en )

2019-09-05

IL290001A

( en )

2022-03-01

JP7007396B2

( en )

2022-01-24

WO2018170490A1

( en )

2018-09-20

IL297863A

( en )

2023-01-01

JP2020511718A

( en )

2020-04-16

CA3055219A1

( en )

2018-09-20

KR20190129093A

( en )

2019-11-19

EP3596542B1

( en )

2024-01-17

<a href="/patent/AU2018234929B2

Related documents

• Augmented reality system with spatialized audio tied to user manipulated … — Magic Leap, Inc. (US11231770B2)#653963
• Virtual reality system using multiple force arrays for a solver — DeepMotion, Inc. (US10657696B2)#653961
• High-efficiency fuel cell system with carbon dioxide capture assembly and … — Fuelcell Energy, Inc. (US10170782B2)#653965
• Integrated electrochemical cell power generation and electrolysis systems — Czero, Inc. (US20250087711A1)#653966
• Generating augmented reality experiences utilizing physical objects to … — Meta Platforms Technologies, Llc (US20260037214A1)#653960