ABSTRACT
Abstract
A two-dimensional augmentation image is rendered from a three-dimensional model from a first virtual perspective. A transformation is applied to the augmentation image to yield an updated two-dimensional augmentation image that approximates a second virtual perspective of the three-dimensional model without additional rendering from the three-dimensional model.
Description
BACKGROUND
Augmented-reality devices may be configured to display one or more augmentation images overlaid on a physical space from a perspective of a user in order to provide an augmented view of the physical space to the user. For example, an augmentation image may provide an illusion that a virtual object (e.g., a hologram) is present in the physical space.
SUMMARY
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all disadvantages noted in any part of this disclosure.
A two-dimensional augmentation image is rendered from a three-dimensional model from a first virtual perspective. A transformation is applied to the two-dimensional augmentation image to yield an updated two-dimensional augmentation image that approximates a second virtual perspective of the three-dimensional model without additional rendering from the three-dimensional model.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 shows an example physical space including a user wearing a head-mounted, at least partially see-through display device that is augmenting the physical space by visually presenting an augmentation image to the user via an at least partially see-through display.
FIG. 2 shows an example physical space including an augmented-reality device that is augmenting the physical space by visually presenting a mixed-reality image via a display.
FIGS. 3-5 show an example method for providing a mixed-reality experience.
FIG. 6 shows an example three-dimensional model of an object rendered as a two-dimensional augmentation image from a first perspective.
FIG. 7 shows a perspective transformation of the augmentation image of the object of FIG. 6 to an updated two-dimensional augmentation image from a second perspective.
FIG. 8 shows example virtual and real-world image streams that may be synchronized using corresponding timestamps.
FIG. 9 shows an example approach for handling a scenario where a perspective transformation of an augmentation image causes two virtual positions to be mapped to a same pixel.
FIG. 10 shows an example approach for handling a scenario where a perspective transformation of an augmentation image produces missing pixels in an updated augmentation.
FIG. 11 shows an example head-mounted, at least partially see-through display device.
FIG. 12 shows an example computing system.
DETAILED DESCRIPTION
The present disclosure is directed to an approach for controlling an augmented-reality device to output augmentation imagery in a performant manner by reducing a number of image rendering operations that are performed by the augmented-reality device. In particular, the augmented-reality device may eschew continuously performing image rendering operations to output augmentation imagery (e.g., for display or as a mixed-reality recording) by employing various post-rendering re-projection techniques to produce augmentation images that approximate different real-world perspectives. For example, an augmentation image may be rendered from a first real-world perspective of a display that visually presents the augmentation image, and a transformation may be applied to the augmentation image to yield an updated augmentation image that approximates a second real-world perspective of a point-of-view camera used to create a mixed-reality recording. By performing such post-rendering re-projection techniques to visually present augmentation imagery, computational expense may be reduced.
FIG. 1 shows an example physical space 100 in which a user 102 is wearing an augmented-reality device in the form of a head-mounted, at least partially see-through display device (referred to herein as a head-mounted display (HMD)) 104 . The HMD 104 provides the user 102 with an at least partially see-through field of view (FOV) 106 of the physical space 100 . Because the HMD 104 is mounted on the user's head, the FOV 106 of the physical space 100 may change as a pose of the user's head changes.
The physical space 100 may include a plurality of real-world objects 108 (e.g., 108 A, 108 B) that may be visible to the user 102 within the FOV 106 of the HMD 104 . Furthermore, the plurality of real-world objects 108 may be visible to other users in the physical space 100 that are not using an augmented-reality device to view the physical space 100 .
The HMD 104 may be configured to visually present augmented-reality images to the user 102 in the FOV 106 of the HMD 104 . These augmented-reality images may be displayed by display componentry of the HMD 104 . As such, display light of the HMD 104 may be directed to a user eye so that the user will see augmented-reality images that are not actually present in the physical space. In at least partially see-through display implementations, such display light may be directed to the user eye while light from the physical space 100 passes through the HMD 104 to the user eye. As such, the user eye simultaneously receives light from the physical environment (e.g., light reflecting from couch 108 A) and light from the at least partially see-through display.
The HMD 104 visually presents a plurality of augmented-reality objects 110 (e.g., 110 A, 110 B, 110 C) that collectively form an augmentation image 112 . In particular, a virtual fish 110 A appears to be swimming above a real- world couch 108 A, a virtual piece of seaweed 110 B and a virtual piece of coral 110 C appear to be located on a real-world end table 108 B.
Note that the plurality of augmented-reality objects 110 may only be seen by users of augmented-reality devices, such as the user 102 via the HMD 104 . In other words, the plurality of augmented-reality objects 110 may not be visible to other users in the physical space 100 that are not wearing HMDs because display light from the HMD does not reach the other users' eyes.
The HMD 104 may be configured to visually present augmented-reality images such that the displayed augmented-reality objects appear body-locked and/or world-locked. A body-locked augmented-reality object may appear to move with a perspective of the user 102 as a pose (e.g., 6 degrees of freedom (DOF): x, y, z, yaw, pitch, roll) of the HMD 104 changes. As such, a body-locked, augmented-reality object may appear to occupy the same portion of the FOV 106 and may appear to be at the same distance from the user 102 , even as the user 102 moves in the physical space 100 .
On the other hand, a world-locked, augmented-reality object may appear to remain in a fixed location in the physical space 100 , even as the pose of the HMD 104 and the perspective of the user 102 changes. For example, the plurality of augmented-reality objects 110 may be world-locked, augmented-reality objects that appear to be located at the same real-world locations regardless of a perspective from which the user 102 views the plurality of augmented-reality objects 110 . To support a world-locked, augmented-reality object, in one example, the HMD 104 may be configured to track a 6DOF pose of the HMD 104 and a geometric mapping/modeling of surface aspects of the physical space 100 . Such tracking and mapping will be discussed in further detail below with reference to FIG. 11 .
The HMD 104 may be configured to visually present augmented-reality images such that the displayed augmented-reality objects appear with hybrid body- and world-locking or without either body- or world-locking.
FIG. 2 shows another augmented-reality device in the form of a mobile computing device 202 including an outward-facing point-of- view camera 204 and a display 206 . Mobile computing device 202 provides a mixed reality view of physical space 200 . The point-of- view camera 204 images the physical space 200 within a field of view 208 , and the display 206 displays the images captured by the point-of- view camera 204 . In some implementations, the point-of- view camera 204 may be a visible-light camera.
The physical space 200 may include a plurality of real-world objects 210 (e.g., 210 A, 210 B) that may be visible to a user of the mobile computing device 202 . Furthermore, the plurality of real-world objects 210 may be visible to other users in the physical space 200 that are not using an augmented-reality device to view the physical space 200 .
The mobile computing device 202 may be configured to visually present a mixed- reality image 212 via the display 206 . The mixed- reality image 212 may include an augmentation image layer overlaid on a visible-light image layer. The augmentation image layer includes a plurality of augmented-reality objects 214 (e.g., 214 A, 214 B, 214 C). The visible-light image layer includes the plurality of real-world objects 210 located within the field of view 208 of the point-of-<figure-callout id="204" label="view camera" filenames="US10127725-20181113-D00000.png,US10127725-20181113-D00001.p
BACKGROUND
Augmented-reality devices may be configured to display one or more augmentation images overlaid on a physical space from a perspective of a user in order to provide an augmented view of the physical space to the user. For example, an augmentation image may provide an illusion that a virtual object (e.g., a hologram) is present in the physical space.
SUMMARY
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all disadvantages noted in any part of this disclosure.
A two-dimensional augmentation image is rendered from a three-dimensional model from a first virtual perspective. A transformation is applied to the two-dimensional augmentation image to yield an updated two-dimensional augmentation image that approximates a second virtual perspective of the three-dimensional model without additional rendering from the three-dimensional model.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 shows an example physical space including a user wearing a head-mounted, at least partially see-through display device that is augmenting the physical space by visually presenting an augmentation image to the user via an at least partially see-through display.
FIG. 2 shows an example physical space including an augmented-reality device that is augmenting the physical space by visually presenting a mixed-reality image via a display.
FIGS. 3-5 show an example method for providing a mixed-reality experience.
FIG. 6 shows an example three-dimensional model of an object rendered as a two-dimensional augmentation image from a first perspective.
FIG. 7 shows a perspective transformation of the augmentation image of the object of FIG. 6 to an updated two-dimensional augmentation image from a second perspective.
FIG. 8 shows example virtual and real-world image streams that may be synchronized using corresponding timestamps.
FIG. 9 shows an example approach for handling a scenario where a perspective transformation of an augmentation image causes two virtual positions to be mapped to a same pixel.
FIG. 10 shows an example approach for handling a scenario where a perspective transformation of an augmentation image produces missing pixels in an updated augmentation.
FIG. 11 shows an example head-mounted, at least partially see-through display device.
FIG. 12 shows an example computing system.
DETAILED DESCRIPTION
The present disclosure is directed to an approach for controlling an augmented-reality device to output augmentation imagery in a performant manner by reducing a number of image rendering operations that are performed by the augmented-reality device. In particular, the augmented-reality device may eschew continuously performing image rendering operations to output augmentation imagery (e.g., for display or as a mixed-reality recording) by employing various post-rendering re-projection techniques to produce augmentation images that approximate different real-world perspectives. For example, an augmentation image may be rendered from a first real-world perspective of a display that visually presents the augmentation image, and a transformation may be applied to the augmentation image to yield an updated augmentation image that approximates a second real-world perspective of a point-of-view camera used to create a mixed-reality recording. By performing such post-rendering re-projection techniques to visually present augmentation imagery, computational expense may be reduced.
FIG. 1 shows an example physical space 100 in which a user 102 is wearing an augmented-reality device in the form of a head-mounted, at least partially see-through display device (referred to herein as a head-mounted display (HMD)) 104 . The HMD 104 provides the user 102 with an at least partially see-through field of view (FOV) 106 of the physical space 100 . Because the HMD 104 is mounted on the user's head, the FOV 106 of the physical space 100 may change as a pose of the user's head changes.
The physical space 100 may include a plurality of real-world objects 108 (e.g., 108 A, 108 B) that may be visible to the user 102 within the FOV 106 of the HMD 104 . Furthermore, the plurality of real-world objects 108 may be visible to other users in the physical space 100 that are not using an augmented-reality device to view the physical space 100 .
The HMD 104 may be configured to visually present augmented-reality images to the user 102 in the FOV 106 of the HMD 104 . These augmented-reality images may be displayed by display componentry of the HMD 104 . As such, display light of the HMD 104 may be directed to a user eye so that the user will see augmented-reality images that are not actually present in the physical space. In at least partially see-through display implementations, such display light may be directed to the user eye while light from the physical space 100 passes through the HMD 104 to the user eye. As such, the user eye simultaneously receives light from the physical environment (e.g., light reflecting from couch 108 A) and light from the at least partially see-through display.
The HMD 104 visually presents a plurality of augmented-reality objects 110 (e.g., 110 A, 110 B, 110 C) that collectively form an augmentation image 112 . In particular, a virtual fish 110 A appears to be swimming above a real- world couch 108 A, a virtual piece of seaweed 110 B and a virtual piece of coral 110 C appear to be located on a real-world end table 108 B.
Note that the plurality of augmented-reality objects 110 may only be seen by users of augmented-reality devices, such as the user 102 via the HMD 104 . In other words, the plurality of augmented-reality objects 110 may not be visible to other users in the physical space 100 that are not wearing HMDs because display light from the HMD does not reach the other users' eyes.
The HMD 104 may be configured to visually present augmented-reality images such that the displayed augmented-reality objects appear body-locked and/or world-locked. A body-locked augmented-reality object may appear to move with a perspective of the user 102 as a pose (e.g., 6 degrees of freedom (DOF): x, y, z, yaw, pitch, roll) of the HMD 104 changes. As such, a body-locked, augmented-reality object may appear to occupy the same portion of the FOV 106 and may appear to be at the same distance from the user 102 , even as the user 102 moves in the physical space 100 .
On the other hand, a world-locked, augmented-reality object may appear to remain in a fixed location in the physical space 100 , even as the pose of the HMD 104 and the perspective of the user 102 changes. For example, the plurality of augmented-reality objects 110 may be world-locked, augmented-reality objects that appear to be located at the same real-world locations regardless of a perspective from which the user 102 views the plurality of augmented-reality objects 110 . To support a world-locked, augmented-reality object, in one example, the HMD 104 may be configured to track a 6DOF pose of the HMD 104 and a geometric mapping/modeling of surface aspects of the physical space 100 . Such tracking and mapping will be discussed in further detail below with reference to FIG. 11 .
The HMD 104 may be configured to visually present augmented-reality images such that the displayed augmented-reality objects appear with hybrid body- and world-locking or without either body- or world-locking.
FIG. 2 shows another augmented-reality device in the form of a mobile computing device 202 including an outward-facing point-of- view camera 204 and a display 206 . Mobile computing device 202 provides a mixed reality view of physical space 200 . The point-of- view camera 204 images the physical space 200 within a field of view 208 , and the display 206 displays the images captured by the point-of- view camera 204 . In some implementations, the point-of- view camera 204 may be a visible-light camera.
The physical space 200 may include a plurality of real-world objects 210 (e.g., 210 A, 210 B) that may be visible to a user of the mobile computing device 202 . Furthermore, the plurality of real-world objects 210 may be visible to other users in the physical space 200 that are not using an augmented-reality device to view the physical space 200 .
The mobile computing device 202 may be configured to visually present a mixed- reality image 212 via the display 206 . The mixed- reality image 212 may include an augmentation image layer overlaid on a visible-light image layer. The augmentation image layer includes a plurality of augmented-reality objects 214 (e.g., 214 A, 214 B, 214 C). The visible-light image layer includes the plurality of real-world objects 210 located within the field of view 208 of the point-of- view camera 204 . In particular, a virtual fish 214 A appears to be swimming above a real- world couch 210 A, a virtual piece of seaweed 214 B and a virtual piece of coral 214 C appear to be located on a real-world end table 210 B.
Note that the plurality of augmented-reality objects 214 only may be seen by one or more users viewing the display 206 . In other words, the plurality of augmented-reality objects 214 may not be visible to other users in the physical space 200 .
In the above described augmented-reality device implementations, real-time augmentation of a physical space may provide an immersive and realistic mixed-reality experience. In order to facilitate such real-time augmentation, the augmented reality devices may perform various post-rendering re-projection techniques to produce augmentation images in a performant manner. In other words, such techniques may be less computationally expensive relative to continuously rendering augmentation imagery from three-dimensional models.
FIGS. 3-5 show an example method 300 for controlling an augmented-reality device to provide a mixed-reality experience in a performant manner. In one example, the method 300 is performed by the HMD 104 shown in FIG. 1 . In another example the method 300 is performed by the mobile computing device 202 shown in FIG. 2 . In another example, the method 300 is performed by an HMD 1100 shown in FIG. 11 . In another example, the method 300 is performed by a computing system 1200 shown in FIG. 12 . In general, the method 300 may be performed by any suitable augmented-reality device.
FIGS. 6-10 shows various operations that may be performed by an augmented-reality device (e.g., the HMD 104 shown in FIG. 1 or the mobile computing device 202 shown in FIG. 2 ) in the course of performing the method 300 , and will be referenced throughout discussion of the method 300 .
At 302 , the method 300 may include rendering from a three-dimensional model a two-dimensional augmentation image from a first virtual perspective. The three-dimensional model may include any suitable virtual content (e.g., hologram) that may be produced by any suitable application of the augmented-reality device. For example, the three-dimensional model may include a virtual scene or virtual objects of a video game.
FIG. 6 shows an example virtual model 600 being rendered as an augmentation image 602 . The virtual model 600 and the augmentation image 602 are shown in simplified form. The virtual model 600 may be rendered as the augmentation image 602 from a first virtual perspective 604 (e.g., a virtual position of a virtual camera). In particular, a portion of the virtual model 600 that is viewable from the first virtual perspective 604 is projected onto a first image plane 606 as pixels 608 of the augmentation image 602 having screen coordinates defined in terms of the first virtual perspective 604 .
Continuing with FIG. 3 , at 304 , the method 300 may include outputting the augmentation image. The augmentation image may be output by the augmented-reality device in any suitable manner.
In some implementations, at 306 , the method 300 optionally may include outputting the augmentation image as a first layer of a mixed-reality recording. The mixed-reality recording may include a second layer including a visible-light image captured by the point-of-view camera of the augmented-reality device.
In some implementations, the mixed-reality recording may be stored in a storage machine (e.g., either local to the augmented-reality device or a remote storage machine, such as a network-connected storage machine) for visual presentation at a later time. In some implementations, the mixed-reality recording may be sent, via a network connection, to a remote or external display device for visual presentation by the remote display device. For example, the mixed-reality recording may be sent to an external display in order to provide a user that is not wearing an HMD a view that approximates the augmented perspective of the wearer of the HMD.
In some implementations, at 308 , the method 300 optionally may include outputting the augmentation image to a display of the augmented-reality device for visual presentation of the augmentation image by the display.
In some implementations where the augmented-reality device includes an at least partially see-through display (e.g., HMD 104 shown in FIG. 1 ) the augmentation image may be visually presented, via the at least partially see-through display of the augmented-reality device, while light from the physical space passes through the at least partially see-through display to a user eye.
In some implementations, the augmentation image may be visually presented, via the display of the augmented-reality device, as a first layer of a first mixed-reality image. The first mixed-reality image may include a second layer including the first visible-light image of the physical space. Note that the first mixed-reality image may be visually presented via an at least partially see-through display or a display that is not see-through.
At 310 , the method 300 may include obtaining, via a point-of-view camera of the augmented-reality device at a second real-world perspective, a visible-light image of the physical space. For example, the visible-light image may be obtained to generate a mixed-reality recording, determine extrinsic and intrinsic calibration data of the point-of-view camera/HMD, and/or to perform other optical analysis.
In some implementations, each visible-light image produced by the point-of-view camera may be tagged with metadata including extrinsic and intrinsic calibration data of the point-of-view camera. The extrinsic and intrinsic calibration data may be used to spatially register virtual content to the physical space.
The extrinsic calibration data defines a real-world position of the point-of-view camera. In one example, the extrinsic calibration data may include parameters that denote coordinate system transformations from three-dimensional world coordinates to three-dimensional camera coordinates. In another example, the extrinsic calibration data may define the position of the point-of-view camera's center and the point-of-view camera's heading in real-world world coordinates (e.g., the real-world pose). The extrinsic calibration data may be associated with each image frame as it represents a pose that changes based on frame-by-frame movement.
The intrinsic calibration data defines a configuration of the point-of-view camera. In one example, the intrinsic calibration data may include camera parameters that define a focal length, image sensor format, and principal point. The intrinsic calibration data may also be associated with each image frame as it may be dynamically modified on a frame-by-frame basis (e.g., video stabilization or cropping is enabled, or the interpupillary distance (IPD) changes).
Turning to FIG. 4 , at 312 , the method 300 may include applying a transformation to the augmentation image to yield the updated two-dimensional augmentation image that approximates the second virtual perspective of the three-dimensional model without additional rendering from the three-dimensional model. Any suitable transformation or series of transformations may be applied to the augmentation image having the first virtual perspective to yield the updated augmentation image having the second virtual perspective without departing from the scope of this disclosure.
In one example, the change in virtual perspective may correspond to a change in position between a point-of-view camera imaging the physical space and an at least partially see-through display that is viewed by a user eye. In another example, the change in virtual perspective may correspond to a change in position between different displays of a stereoscopic display (e.g., a left-eye display and a right-eye display). The change in virtual perspective may correspond to a change in position between any suitable different real-world perspectives.
In some implementations, each visible-light image produced by a point-of-view camera of the augmented-reality device may be tagged with a real-world timestamp. Further, the three-dimensional model may utilize a virtual camera that provides a virtual perspective from which the three-dimensional model may be viewed at any given time. The virtual perspective of the virtual camera at a designated point in time may be defined by a virtual image frame. Each virtual image frame may be tagged with a virtual timestamp.
Accordingly, in some such implementations, the transformation may be selected or determined based on a real-world timestamp of a visible-light image obtained by the point-of-view camera. In particular, a position and orientation of the augmented-reality device may be determined at each timestamp, and a delta between timestamps may be used in transforming an augmentation image to an updated augmentation image. For example, the virtual and/or real-world timestamps may be used to predict the pose or extrinsic position of the point-of-view camera at a particular time for which an updated augmentation image is desired. In this way, the updated augmentation image will better align with corresponding real-world images when generating a mixed reality recording Such an approach may be particularly applicable in cases where a frame rate of the virtual camera differs from a frame rate of the point-of-view camera.
FIG. 8 shows example video streams that may be produced by an augmented-reality device (e.g., the HMD 104 shown in FIG. 1 or the mobile computing device 202 shown in FIG. 2 ). An augmentation image stream 800 may include a plurality of augmentation image frames 802 (e.g., 802 A, 802 B, 802 C, 802 D, 802 E). Each of the plurality of augmentation image frames 802 may be rendered from a three-dimensional model. Each of the plurality of augmentation image frames 802 may include a virtual timestamp 804 . A real-world (e.g., visible-light) image stream 806 may include a plurality of real-world image frames 808 (e.g., 808 A, 808 B, 808 C, 808 D, 808 E, 808 F, 808 G, 808 H, 808 I, 808 J). Each of the plurality of real-world image frames 808 may be obtained from the point-of-view camera. Each of the plurality of real-world image frames 808 may include a real- world timestamp 810 . In the illustrated example, the augmentation image stream 800 and the real- world image stream 806 are depicted as having different frame rates. An updated augmentation image stream 812 may include a plurality of updated augmentation image frames 814 (e.g., 814 A, 814 B, 814 C, 814 D, 814 E, 814 F, 814 G, 814 H, 814 I). Each of the plurality of augmentation image frames 802 may be generated by applying a transformation to a corresponding augmentation image 802 .
In the illustrated example, the real- world image stream 806 has a higher frame rate than the augmentation image stream 800 . In order to generate an accurate mixed-reality recording in which each real-world image frame is layered with virtual content, updated augmentation image frames may be generated to accurately represent the virtual content in the mixed-reality recording in between successive augmentation image frames being rendered. In particular, a transformation may be selected to be applied to a given augmentation image frame to yield a corresponding updated augmentation image frame based on a real-world time stamp of a corresponding real-world image frame.
For example, real-world image frame 808 B and real-world image frame 808 C are obtained via the point-of-view camera after augmentation image frame 804 A has been rendered but prior to augmentation image 804 B being rendered. Instead of layering augmentation image 804 A on both real-world image frames 808 B and 808 C, different transformations may be applied to augmentation image frame 804 A to yield updated augmentation image frames 814 A and 814 B that correspond to real-world image frame 808 B and real-world image frame 808 C, respectively. In particular, the pose or extrinsic position data of timestamp 810 B may be used to select/apply the transformation that yields the updated augmentation image 814 A, and the extrinsic data of timestamp 810 C may be used to select/apply a different transformation that yields the updated augmentation image 814 B. The extrinsic data in each timestamp may be used to approximate the perspective of the augmented-reality device when the real-world image is obtained. In this way, the updated augmentation image frames may be generated to accurately represent the virtual content in between the augmentation images being rendered.
A real-world timestamp and associated data of a real-world image frame may be used to select/apply a transformation in any suitable manner without departing from the scope of this disclosure. In some implementations, the augmented image stream 800 and the real- world image stream 806 may have the same frame rate, and thus there may be a one-to-one correlation between virtual and real-world content.
In some implementations, the transformation (or re-projection) of the augmentation from the first virtual perspective to the updated augmentation image from the second virtual perspective may be a perspective transform (e.g., homography). Accordingly, in some implementations, at 314 , the method 300 optionally may include, for each pixel of the augmentation image, passing a screen coordinate of the pixel through an inverse view-projection matrix to yield a virtual position in a three-dimensional transformation model from the first virtual perspective.
In FIG. 7 , screen coordinates of each of the pixels 608 may be projected from the first image plane 606 of the first virtual perspective 604 to a virtual position 702 on a transformation image plane 704 . The virtual positions 702 projected on the transformation image plane 704 and defined in terms of the coordinate system of the first virtual perspective 604 may represent the three-dimensional transformation model. In one example, the transformation image plane 704 may have a designated depth in virtual space that may be predetermined. In another example, the transformation image plane 704 may have a depth that is provided by an application that is rendering the virtual model 600 .
Continuing with FIG. 4 , at 316 , the method 300 optionally may include, for each virtual position of the three-dimensional transformation model, mapping the virtual position to an updated virtual position in an updated three-dimensional transformation model to simulate the second virtual perspective.
Returning to FIG. 7 , the virtual positions 702 on the transformation image plane 704 may be mapped (e.g., rotation+translation) from the coordinate system of the first virtual perspective 604 to a coordinate system of the second virtual perspective 706 . The virtual positions 702 projected on the transformation image plane 704 and defined in terms of the coordinate system of the second virtual perspective 706 may represent the updated three-dimensional transformation model.
At 318 , the method 300 optionally may include, for each updated virtual position in the updated three-dimensional transformation model, passing the updated virtual position of the pixel through a view-projection matrix to yield an updated screen coordinate of the pixel.
Returning to FIG. 7 , the virtual positions 702 projected on the transformation image plane 704 and defined in terms of the coordinate system of the second virtual perspective 706 may be projected to a second image plane 708 of the second virtual perspective 706 to yield pixels 710 of the updated augmentation image 712 having screen coordinates defined in terms of the second virtual perspective 706 .
In one example, the above described homography is described in terms of a point X. The point X may be representative of any point in the virtual model that maps to a pixel of the augmentation image. The point X has three-dimensional coordinates that are relative to a camera-centered coordinate system in the three-dimensional real-world space. X is represented by a 4 dimensional vector:
X
=
(
x
y
z
1
)
X1 is the position of the point X that is projected onto the first image plane and defined in terms of the coordinate system of the first virtual perspective that is spatially registered to the first real-world perspective of the first visible-light image. X1 is represented by a 3 dimensional vector:
X
1
=
(
x
1
y
1
1
)
X
1
=
K
1
â¡
[
R
1
â
t
1
]
â¢
X
X
1
=
K
1
â¢
R
1
â¢
X
+
K
1
â¢
t
1
this is the transformation induced by the pose of the unit at the time of the first camera frame. [R 1 |t 1 ] is the rotation and translation matrix to move the point X to the coordinate system of the real-world perspective of the first visible-light image (e.g., the extrinsic calibration data). K 1 is the projection matrix associated with the first visible-light image (e.g., the camera a.k.a. the 3Ã3 camera intrinsic matrix).
X2 is the position of the point X that is projected onto the second image plane and defined in terms of the coordinate system of the second virtual perspective that is spatially registered to the second real-world perspective of the second visible-light image. X2 is represented by a 3 dimensional vector:
X
2
=
(
x
2
y
2
1
)
X
2
=
K
2
â¡
[
R
2
â
t
2
]
â¢
X
CLAIMS
Claims ( 19 )
The invention claimed is:
1. On an augmented-reality device, a method of providing a mixed-reality experience, the method comprising:
rendering from a three-dimensional model a two-dimensional augmentation image, the two-dimensional augmentation image providing a view of the three-dimensional model from a first virtual perspective of a virtual camera spatially registered to a first real-world location;
outputting the two-dimensional augmentation image;
applying a transformation to the two-dimensional augmentation image to yield an updated two-dimensional augmentation image, the updated two-dimensional augmentation image approximating a different view of the three-dimensional model from a second virtual perspective of the virtual camera spatially registered to a second real-world location different than the first real-world location such that the second virtual perspective is different than the first virtual perspective; and
outputting the updated two-dimensional augmentation image.
2. The method of claim 1 , wherein outputting the updated two-dimensional augmentation image includes outputting the updated two-dimensional augmentation image as a layer of a mixed-reality recording, the mixed-reality recording including a second layer including a visible-light image captured by a point-of-view camera.
3. The method of claim 2 , wherein the transformation is selected so that the updated two-dimensional augmentation image has a virtual perspective simulating a real-world perspective of the point-of-view camera at the second real-world location when the visible-light image is captured.
4. The method of claim 2 , further comprising:
cropping a larger of the updated two-dimensional augmentation image and the visible-light image to match a smaller of the updated two-dimensional augmentation image and the visible light image in the mixed-reality recording.
5. The method of claim 1 , wherein outputting the updated two-dimensional augmentation image includes outputting the updated two-dimensional augmentation image to a display of the augmented-reality device for visual presentation of the updated two-dimensional augmentation image by the display.
6. The method of claim 5 , wherein the display is an at least partially see-through display, wherein the updated two-dimensional augmentation image is visually presented on the at least partially see-through display while light from a physical space passes through the at least partially see-through display to a user eye.
7. The method of claim 5 , wherein the updated two-dimensional augmentation image has dimensions that are larger than dimensions of a field of view of the display, and the method further comprises, cropping the updated two-dimensional augmentation image.
8. The method of claim 1 , wherein applying a transformation to the augmentation image includes,
for each pixel of the augmentation image, passing a screen coordinate of the pixel through an inverse view-projection matrix to yield a virtual position in a three-dimensional transformation model from the first virtual perspective,
for each virtual position of the three-dimensional transformation model, mapping the virtual position to an updated virtual position in an updated three-dimensional transformation model to simulate the second virtual perspective, and
for each updated virtual position in the updated three-dimensional transformation model, passing the updated virtual position of the pixel through a view-projection matrix to yield an updated screen coordinate of the pixel.
9. The method of claim 8 , wherein the method further comprises:
if two or more updated virtual positions map to a same pixel, selecting an updated virtual position nearest to the second virtual perspective as the updated virtual position of the same pixel.
10. An augmented-reality device, comprising:
an at least partially see-through display;
a logic machine; and
a storage machine holding instructions executable by the logic machine to:
render from a three-dimensional model a two-dimensional augmentation image, the two-dimensional augmentation image providing a view of the three-dimensional model from a first virtual perspective of a virtual camera spatially registered to a first real-world location;
output the two-dimensional augmentation image;
applying a transformation to the two-dimensional augmentation image to yield an updated two-dimensional augmentation image, the updated two-dimensional augmentation image approximating a different view of the three-dimensional model from a second virtual perspective of the virtual camera spatially registered to a second real-world location different than the first real-world location such that the second virtual perspective is different than the first virtual perspective; and
output the updated two-dimensional augmentation image.
11. The augmented-reality device of claim 10 , wherein outputting the updated two-dimensional augmentation image includes outputting the updated two-dimensional augmentation image as a layer of a mixed-reality recording, the mixed-reality recording including a second layer including a visible-light image captured by a point-of-view camera.
12. The augmented-reality device of claim 11 , wherein the transformation is selected so that the updated two-dimensional augmentation image has a virtual perspective simulating a real-world perspective of the point-of-view camera at the second real-world location when the visible-light image is captured.
13. The augmented-reality device of claim 11 , wherein the storage machine further holds instructions executable by the logic machine to:
crop a larger of the updated two-dimensional augmentation image and the visible-light image to match a smaller of the updated two-dimensional augmentation image and the visible light image in the mixed-reality recording.
14. The augmented-reality device of claim 10 , wherein outputting the updated two-dimensional augmentation image includes outputting the updated two-dimensional augmentation image to the at least partially see-through display, and wherein the updated two-dimensional augmentation image is visually presented on the at least partially see-through display while light from a physical space passes through the at least partially see-through display to a user eye.
15. The augmented-reality device of claim 14 , wherein the updated two-dimensional augmentation image has dimensions that are larger than dimensions of a field of view of the display, and wherein the storage machine further holds instructions executable by the logic machine to:
crop the updated two-dimensional augmentation image.
16. The augmented-reality device of claim 10 , wherein applying a transformation to the augmentation image includes,
for each pixel of the augmentation image, pass a screen coordinate of the pixel through an inverse view-projection matrix to yield a virtual position in a three-dimensional transformation model from the first virtual perspective,
for each virtual position of the three-dimensional transformation model, map the virtual position to an updated virtual position in an updated three-dimensional transformation model to simulate the second virtual perspective, and
for each updated virtual position in the updated three-dimensional transformation model, pass the updated virtual position of the pixel through a view-projection matrix to yield an updated screen coordinate of the pixel.
17. The augmented-reality device of claim 16 , wherein the storage machine further holds instructions executable by the logic machine to:
if two or more updated virtual positions map to a same pixel, select an updated virtual position nearest to the second virtual perspective as the updated virtual position of the same pixel.
18. On an augmented-reality device, a method of providing a mixed-reality experience, the method comprising:
rendering from a three-dimensional model a two-dimensional augmentation image, the two-dimensional augmentation image providing a view of the three-dimensional model from a first virtual perspective of a virtual camera spatially registered to a real-world perspective of an at least partially see-through display of the augmented reality device;
outputting the two-dimensional augmentation image to the at least partially see-through display;
obtaining, via a point-of-view camera of the augmented-reality device, a visible-light image of a physical space;
applying a transformation to the two-dimensional augmentation image to yield an updated two-dimensional augmentation image, the updated two-dimensional augmentation image approximating a different view of the three-dimensional model from a second virtual perspective different than the first virtual perspective of the virtual camera and corresponding to a real-world perspective of the point-of-view camera when the visible-light image is captured, wherein applying the transformation to the augmentation image includes,
for each pixel of the augmentation image, passing a screen coordinate of the pixel through an inverse view-projection matrix to yield a virtual position in a three-dimensional transformation model from the first virtual perspective,
for each virtual position of the three-dimensional transformation model, mapping the virtual position to an updated virtual position in an updated three-dimensional transformation model to simulate the second virtual perspective, and
for each updated virtual position in the updated three-dimensional transformation model, passing the updated virtual position of the pixel through a view-projection matrix to yield an updated screen coordinate of the pixel;
combining the updated two-dimensional augmentation image and the visible light image in a mixed reality recording; and
outputting the mixed reality recording.
19. The method of claim 18 , wherein the method further comprises:
if two or more updated virtual positions map to a same pixel, selecting an updated virtual position nearest to the second virtual perspective as the updated virtual position of the same pixel.
US14/843,709
2015-09-02
2015-09-02
Augmented-reality imaging
Active
2035-09-16
US10127725B2
( en )
Priority Applications (1)
Application Number
Priority Date
Filing Date
Title
US14/843,709
US10127725B2
( en )
2015-09-02
2015-09-02
Augmented-reality imaging
Applications Claiming Priority (1)
Application Number
Priority Date
Filing Date
Title
US14/843,709
US10127725B2
( en )
2015-09-02
2015-09-02
Augmented-reality imaging
Publications (2)
Publication Number
Publication Date
US20170061693A1
US20170061693A1 ( en )
2017-03-02
US10127725B2
true
US10127725B2 ( en )
2018-11-13
Family
ID=58095604
Family Applications (1)
Application Number
Title
Priority Date
Filing Date
US14/843,709
Active
2035-09-16
US10127725B2
( en )
2015-09-02
2015-09-02
Augmented-reality imaging
Country Status (1)
Country
Link
US
( 1 )
US10127725B2
( en )
Cited By (14)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US20170115488A1
( en )
*
2015-10-26
2017-04-27
Microsoft Technology Licensing, Llc
Remote rendering for virtual images
US20200162713A1
( en )
*
2017-02-03
2020-05-21
Warner Bros. Entertainment Inc.
Rendering extended video in virtual reality
US10706459B2
( en )
2017-06-20
2020-07-07
Nike, Inc.
Augmented reality experience unlock via target image detection
US10726435B2
( en )
*
2017-09-11
2020-07-28
Nike, Inc.
Apparatus, system, and method for target search and using geocaching
US10922878B2
( en )
*
2017-10-04
2021-02-16
Google Llc
Lighting for inserted content
US11509653B2
( en )
2017-09-12
2022-11-22
Nike, Inc.
Multi-factor authentication and post-authentication processing system
US11586286B1
( en )
2022-05-18
2023-02-21
Bank Of America Corporation
System and method for navigating on an augmented reality display
US11720380B1
( en )
2022-05-18
2023-08-08
Bank Of America Corporation
System and method for updating augmented reality navigation instructions based on a detected error
TWI812369B
( en )
*
2021-07-28
2023-08-11
å®éåéé»åè¡ä»½æéå ¬å¸
Control method, tracking system and non-transitory computer-readable storage medium
US11961106B2
( en )
2017-09-12
2024-04-16
Nike, Inc.
Multi-factor authentication and post-authentication processing system
US20240137665A1
( en )
*
2019-02-07
2024-04-25
Magic Leap, Inc.
Lightweight cross reality device with passive depth extraction
US12201048B2
( en )
2021-08-11
2025-01-21
Deere & Company
Obtaining and augmenting agricultural data and generating an augmented display showing anomalies
US12373025B2
( en )
2019-02-07
2025-07-29
Magic Leap, Inc.
Lightweight and low power cross reality device with high temporal resolution
US12439840B2
( en )
2021-08-11
2025-10-14
Deere & Company
Obtaining and augmenting agricultural data and generating an augmented display
Families Citing this family (46)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US10176642B2
( en )
*
2015-07-17
2019-01-08
Bao Tran
Systems and methods for computer assisted operation
KR20240051334A
( en )
2016-02-18
2024-04-19
ì í ì¸í¬.
Head-mounted display for virtual and mixed reality with inside-out positional, user body and environment tracking
US9934758B1
( en )
*
2016-09-21
2018-04-03
Gopro, Inc.
Systems and methods for simulating adaptation of eyes to changes in lighting conditions
US10943396B1
( en )
*
2016-09-30
2021-03-09
Amazon Technologies, Inc.
Synchronizing transmitted video data and enhancements
US11295525B1
( en )
2016-09-30
2022-04-05
Amazon Technologies, Inc.
Augmenting transmitted video data
US10950049B1
( en )
2016-09-30
2021-03-16
Amazon Technologies, Inc.
Augmenting transmitted video data
US10979676B1
( en )
2017-02-27
2021-04-13
Amazon Technologies, Inc.
Adjusting the presented field of view in transmitted data
US11252391B2
( en )
*
2017-03-06
2022-02-15
Nevermind Capital Llc
Methods and apparatus for packing images into a frame and/or including additional content or graphics
US10567733B2
( en )
2017-03-06
2020-02-18
Nextvr Inc.
Methods and apparatus for communicating and/or using frames including a captured image and/or including additional image content
US10970930B1
( en )
2017-08-07
2021-04-06
Amazon Technologies, Inc.
Alignment and concurrent presentation of guide device video and enhancements
US10970545B1
( en )
2017-08-31
2021-04-06
Amazon Technologies, Inc.
Generating and surfacing augmented reality signals for associated physical items
US10755383B2
( en )
2017-09-29
2020-08-25
Apple Inc.
Multi-space rendering with configurable transformation parameters
US11472598B1
( en )
2017-11-16
2022-10-18
Amazon Technologies, Inc.
Systems and methods to encode sounds in association with containers
US10878838B1
( en )
2017-11-16
2020-12-29
Amazon Technologies, Inc.
Systems and methods to trigger actions based on encoded sounds associated with containers
US11014242B2
( en )
2018-01-26
2021-05-25
Microsoft Technology Licensing, Llc
Puppeteering in augmented reality
US10712920B1
( en )
*
2018-02-15
2020-07-14
Shutterstock, Inc.
In-browser image perspective manipulation
US11272081B2
( en )
*
2018-05-03
2022-03-08
Disney Enterprises, Inc.
Systems and methods for real-time compositing of video content
US11429086B1
( en )
2018-05-31
2022-08-30
Amazon Technologies, Inc.
Modifying functions of computing devices based on environment
GB2574882B
( en )
*
2018-06-22
2020-08-12
Sony Interactive Entertainment Inc
Method and system for displaying a virtual object
WO2020014706A1
( en )
*
2018-07-13
2020-01-16
Labrador Systems, Inc.
Visual navigation for mobile devices operable in differing environmental lighting conditions
CN110163942B
( en )
*
2018-07-18
2023-02-14
è ¾è®¯ç§æï¼æ·±å³ï¼æéå ¬å¸
Image data processing method and device
CN114615486B
( en )
2018-08-06
2024-05-07
è¹æå ¬å¸
Method, system and computer readable storage medium for generating a composite stream
CN110827411B
( en )
*
2018-08-09
2023-07-18
å京微æè§çç§ææéå ¬å¸
Method, device, equipment and storage medium for displaying augmented reality model of self-adaptive environment
US11809613B2
( en )
2018-11-12
2023-11-07
Magic Leap, Inc.
Event-based camera with high-resolution frame output
WO2020101892A1
( en )
2018-11-12
2020-05-22
Magic Leap, Inc.
Patch tracking image sensor
US11985440B2
( en )
2018-11-12
2024-05-14
Magic Leap, Inc.
Depth based dynamic vision sensor
US10854007B2
( en )
*
2018-12-03
2020-12-01
Microsoft Technology Licensing, Llc
Space models for mixed reality
CN109545003B
( en )
*
2018-12-24
2022-05-03
å京å¡è·¯éä¿¡æ¯ææ¯æéå ¬å¸
Display method, display device, terminal equipment and storage medium
WO2020139588A1
( en )
2018-12-24
2020-07-02
Dts, Inc.
Room acoustics simulation using deep learning image analysis
CN110139028B
( en )
*
2019-03-25
2020-07-07
å为ææ¯æéå ¬å¸
Image processing method and head-mounted display device
CN117590582A
( en )
*
2019-04-11
2024-02-23
ä¸æçµåæ ªå¼ä¼ç¤¾
Head-mounted display device and method of operating the same
US11846783B2
( en )
*
2019-05-17
2023-12-19
Sony Group Corporation
Information processing apparatus, information processing method, and program
US11265487B2
( en )
*
2019-06-05
2022-03-01
Mediatek Inc.
Camera view synthesis on head-mounted display for virtual reality and augmented reality
US10939034B2
( en )
*
2019-07-08
2021-03-02
Varjo Technologies Oy
Imaging system and method for producing images via gaze-based control
US11838486B1
( en )
*
2019-09-24
2023-12-05
Apple Inc.
Method and device for perspective correction using one or more keyframes
US11494992B2
( en )
2020-08-27
2022-11-08
Micron Technology, Inc.
Constructing an augmented reality image
JP7667859B2
( en )
*
2020-12-22
2025-04-23
ãã¸ã㯠ãªã¼ãï¼ ã¤ã³ã³ã¼ãã¬ã¤ããã
Calibration for Virtual or Augmented Reality Systems
US20230091386A1
( en )
*
2021-09-17
2023-03-23
Multics Co., Ltd.
Augmented reality hologram virtual aquarium system
US11914146B2
( en )
*
2022-05-27
2024-02-27
Sony Interactive Entertainment LLC
Methods and systems for adding real-world sounds to virtual reality scenes
CN115220576B
( en )
*
2022-06-21
2024-10-01
å京åè·³ç½ç»ææ¯æéå ¬å¸
Method, device, equipment and storage medium for controlling picture visual angle
US12505614B2
( en )
*
2022-07-21
2025-12-23
Apple Inc.
Editing mixed-reality recordings
US12505627B2
( en )
*
2022-08-31
2025-12-23
Samsung Electronics Co., Ltd.
System and method for disoccluded region completion for video rendering in video see-through (VST) augmented reality (AR)
JP2024047799A
( en )
*
2022-09-27
2024-04-08
ãã¤ãã³æ ªå¼ä¼ç¤¾
Image processing device, image processing method, and program
EP4354877A1
( en )
*
2022-10-12
2024-04-17
Holo-Light GmbH
Multi-viewer xr streaming method and xr streaming system
US12206830B2
( en )
*
2023-05-31
2025-01-21
Apple Inc.
Techniques for displaying and capturing images
CN119339041B
( en )
*
2024-12-20
2025-04-15
æµæ±å¤§å¦æµ·åç 究é¢
Three-dimensional video construction method, device and medium based on target monitoring area
Citations (32)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US20040119723A1
( en )
*
2002-12-18
2004-06-24
Renesas Technology Corp.
Apparatus manipulating two-dimensional image in a three-dimensional space
US20100182340A1
( en )
2009-01-19
2010-07-22
Bachelder Edward N
Systems and methods for combining virtual and real-time physical environments
US7804507B2
( en )
2006-07-27
2010-09-28
Electronics And Telecommunications Research Institute
Face-mounted display apparatus for mixed reality environment
US20110231162A1
( en )
*
2010-03-19
2011-09-22
Krishna Ramamurthi
System and Method for Generating Enhanced Density Distribution in a Three Dimenional Model of a Structure for Use in Skeletal Assessment Using a Limited Number of Two-Dimensional Views
US8040361B2
( en )
2005-04-11
2011-10-18
Systems Technology, Inc.
Systems and methods for combining virtual and real-time physical environments
US20110254860A1
( en )
2008-12-03
2011-10-20
Alcatel Lucent
Mobile device for augmented reality application
US20120077582A1
( en )
2010-09-24
2012-03-29
Hal Laboratory Inc.
Computer-Readable Storage Medium Having Program Stored Therein, Apparatus, System, and Method, for Performing Game Processing
US20120113223A1
( en )
2010-11-05
2012-05-10
Microsoft Corporation
User Interaction in Augmented Reality
US20120142415A1
( en )
2010-12-03
2012-06-07
Lindsay L Jon
Video Show Combining Real Reality and Virtual Reality
US20120188256A1
( en )
2009-06-25
2012-07-26
Samsung Electronics Co., Ltd.
Virtual world processing device and method
US8400548B2
( en )
2010-01-05
2013-03-19
Apple Inc.
Synchronized, interactive augmented reality displays for multifunction devices
US20130128364A1
( en )
2011-11-22
2013-05-23
Google Inc.
Method of Using Eye-Tracking to Center Image Content in a Display
US20130147838A1
( en )
2011-12-07
2013-06-13
Sheridan Martin Small
Updating printed content with personalized virtual data
US20130147687A1
( en )
*
2011-12-07
2013-06-13
Sheridan Martin Small
Displaying virtual data as printed content
US8477149B2
( en )
2009-04-01
2013-07-02
University Of Central Florida Research Foundation, Inc.
Real-time chromakey matting using image statistics
US20130169682A1
( en )
2011-08-24
2013-07-04
Christopher Michael Novak
Touch and social cues as inputs into a computer
US20130178257A1
( en )
2012-01-06
2013-07-11
Augaroo, Inc.
System and method for interacting with virtual objects in augmented realities
US8487838B2
( en )
2011-08-29
2013-07-16
John R. Lewis
Gaze detection in a see-through, near-eye, mixed reality display
US8547401B2
( en )
2004-08-19
2013-10-01
Sony Computer Entertainment Inc.
Portable augmented reality device and method
US8576276B2
( en )
2010-11-18
2013-11-05
Microsoft Corporation
Head-mounted display device which provides surround video
US20130326364A1
( en )
2012-05-31
2013-12-05
Stephen G. Latta
Position relative hologram interactions
US20140002491A1
( en )
2012-06-29
2014-01-02
Mathew J. Lamb
Deep augmented reality tags for head mounted displays
US20140002487A1
( en )
2012-07-02
2014-01-02
Microsoft Corporation
Animated visualization of alpha channel transparency
US8633946B2
( en )
2005-08-29
2014-01-21
Nant Holdings Ip, Llc
Interactivity with a mixed reality
US20140049559A1
( en )
2012-08-17
2014-02-20
Rod G. Fleck
Mixed reality holographic object development
US8678282B1
( en )
2010-11-29
2014-03-25
Lockheed Martin Corporation
Aim assist head-mounted display apparatus
US20140168264A1
( en )
2012-12-19
2014-06-19
Lockheed Martin Corporation
System, method and computer program product for real-time alignment of an augmented reality device
US20140218291A1
( en )
2013-02-07
2014-08-07
Microsoft Corporation
Aligning virtual camera with real camera
US20140368535A1
( en )
2013-06-18
2014-12-18
Tom G. Salter
Hybrid world/body locked hud on an hmd
US20140372957A1
( en )
2013-06-18
2014-12-18
Brian E. Keane
Multi-step virtual object selection
US20140368534A1
( en )
2013-06-18
2014-12-18
Tom G. Salter
Concurrent optimal viewing of virtual objects
US9240069B1
( en )
*
2015-06-30
2016-01-19
Ariadne's Thread (Usa), Inc.
Low-latency virtual reality display system
2015
2015-09-02
US
US14/843,709
patent/US10127725B2/en
active
Active
Patent Citations (32)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US20040119723A1
( en )
*
2002-12-18
2004-06-24
Renesas Technology Corp.
Apparatus manipulating two-dimensional image in a three-dimensional space
US8547401B2
( en )
2004-08-19
2013-10-01
Sony Computer Entertainment Inc.
Portable augmented reality device and method
US8040361B2
( en )
2005-04-11
2011-10-18
Systems Technology, Inc.
Systems and methods for combining virtual and real-time physical environments
US8633946B2
( en )
2005-08-29
2014-01-21
Nant Holdings Ip, Llc
Interactivity with a mixed reality
US7804507B2
( en )
2006-07-27
2010-09-28
Electronics And Telecommunications Research Institute
Face-mounted display apparatus for mixed reality environment
US20110254860A1
( en )
2008-12-03
2011-10-20
Alcatel Lucent
Mobile device for augmented reality application
US20100182340A1
( en )
2009-01-19
2010-07-22
Bachelder Edward N
Systems and methods for combining virtual and real-time physical environments
US8477149B2
( en )
2009-04-01
2013-07-02
University Of Central Florida Research Foundation, Inc.
Real-time chromakey matting using image statistics
US20120188256A1
( en )
2009-06-25
2012-07-26
Samsung Electronics Co., Ltd.
Virtual world processing device and method
US8400548B2
( en )
2010-01-05
2013-03-19
Apple Inc.
Synchronized, interactive augmented reality displays for multifunction devices
US20110231162A1
( en )
*
2010-03-19
2011-09-22
Krishna Ramamurthi
System and Method for Generating Enhanced Density Distribution in a Three Dimenional Model of a Structure for Use in Skeletal Assessment Using a Limited Number of Two-Dimensional Views
US20120077582A1
( en )
2010-09-24
2012-03-29
Hal Laboratory Inc.
Computer-Readable Storage Medium Having Program Stored Therein, Apparatus, System, and Method, for Performing Game Processing
US20120113223A1
( en )
2010-11-05
2012-05-10
Microsoft Corporation
User Interaction in Augmented Reality
US8576276B2
( en )
2010-11-18
2013-11-05
Microsoft Corporation
Head-mounted display device which provides surround video
US8678282B1
( en )
2010-11-29
2014-03-25
Lockheed Martin Corporation
Aim assist head-mounted display apparatus
US20120142415A1
( en )
2010-12-03
2012-06-07
Lindsay L Jon
Video Show Combining Real Reality and Virtual Reality
US20130169682A1
( en )
2011-08-24
2013-07-04
Christopher Michael Novak
Touch and social cues as inputs into a computer
US8487838B2
( en )
2011-08-29
2013-07-16
John R. Lewis
Gaze detection in a see-through, near-eye, mixed reality display
US20130128364A1
( en )
2011-11-22
2013-05-23
Google Inc.
Method of Using Eye-Tracking to Center Image Content in a Display
US20130147838A1
( en )
2011-12-07
2013-06-13
Sheridan Martin Small
Updating printed content with personalized virtual data
US20130147687A1
( en )
*
2011-12-07
2013-06-13
Sheridan Martin Small
Displaying virtual data as printed content
US20130178257A1
( en )
2012-01-06
2013-07-11
Augaroo, Inc.
System and method for interacting with virtual objects in augmented realities
US20130326364A1
( en )
2012-05-31
2013-12-05
Stephen G. Latta
Position relative hologram interactions
US20140002491A1
( en )
2012-06-29
2014-01-02
Mathew J. Lamb
Deep augmented reality tags for head mounted displays
US20140002487A1
( en )
2012-07-02
2014-01-02
Microsoft Corporation
Animated visualization of alpha channel transparency
US20140049559A1
( en )
2012-08-17
2014-02-20
Rod G. Fleck
Mixed reality holographic object development
US20140168264A1
( en )
2012-12-19
2014-06-19
Lockheed Martin Corporation
System, method and computer program product for real-time alignment of an augmented reality device
US20140218291A1
( en )
2013-02-07
2014-08-07
Microsoft Corporation
Aligning virtual camera with real camera
<tr itemprop="backwardReferences" itemscope re