ABSTRACT
Abstract
A virtual reality display system that renders images at different resolutions in different parts of a display. Reduces rendering latency by rendering at a lower resolution in selected regions, for example on the sides of a display where human vision has lower resolution than in the center. Pixels in low resolution regions are combined into grid elements, and rendering may generate grid element values rather than individual pixel values. Rendering may use ray casting, rasterization, or both. Variable resolution rendering may be combined with variable level of detail geometry models to further reduce rendering time. Selected objects may be designed as high resolution objects that are rendered at a high resolution even in low resolution display regions.
Description
This application is a continuation of U.S. Utility patent application Ser. No. 14/872,488 filed Oct. 1, 2015, issued as U.S. Pat. No. 9,607,428, which is a continuation in part of U.S. Utility patent application Ser. No. 14/788,633 filed Jun. 30, 2015, the specifications of which are hereby incorporated herein by reference.
BACKGROUND OF THE INVENTION
Field of the Invention
One or more embodiments of the invention are related to the field of virtual reality systems. More particularly, but not by way of limitation, one or more embodiments of the invention enable a virtual reality display system that renders different regions of a display at different resolutions as well as tracks movements of a user and quickly renders a virtual reality display based on those movements.
Description of the Related Art
Virtual reality systems are known in the art. Such systems generate a virtual world for a user that responds to the user's movements. Examples include various types of virtual reality headsets and goggles worn by a user, as well as specialized rooms with multiple displays. Virtual reality systems typically include sensors that track a user's head, eyes, or other body parts, and that modify the virtual world according to the user's movements. The virtual world consists of a three-dimensional model, computer-generated or captured from real-world scenes. Images of the three-dimensional model are generated based on the user's position and orientation. Generation of these images requires rendering of the three-dimensional model onto one or more two-dimensional displays. Rendering techniques are known in the art and are often used for example in 3D graphics systems or computer-based games, as well as in virtual reality systems.
A major challenge for existing virtual reality systems is combining realistic images with low-latency rendering, so that user's virtual reality experience matches the rapid feedback to movement observed in real environments. Existing systems often have long latency to measure changes in the user's position and orientation, and to rerender the virtual world based on these changes. 3D rendering is a complex and processor intensive operation that can take potentially hundreds of milliseconds. The result is that users perceive noticeable lag between their movements and the rendering of updated virtual environments on their displays. Three technology trends are compounding this challenge: (1) The complexity of 3D models is growing as more 3D data is captured and generated. (2) Resolution of virtual reality displays is increasing, requiring more computational power to render images. (3) Users are relying increasingly on mobile devices with limited processor capacity. As a result of these trends, high latency in rendering virtual reality displays has become a major factor limiting adoption and applications of virtual reality technology. There are no known systems that provide sufficiently low-latency rendering and display to generate highly responsive virtual reality environments given these technology constraints.
One factor contributing to rendering latency is the resolution of a virtual reality display. In general, displays with large numbers of pixels require more rendering computation and thus may experience greater latency. Displays known in the art typically consist of rectangular arrays of pixels with uniform pixel density throughout the display. However, human vision has high resolution only in the center of the field view. Therefore, rendering images at high resolution on the sides of a display may be unnecessary, and may contribute to higher latency without improving the user's experience.
For at least the limitations described above there is a need for a variable resolution virtual reality display system.
BRIEF SUMMARY OF THE INVENTION
One or more embodiments described in the specification are related to a low-latency virtual reality display system. Embodiments of the system use efficient approximations to rerender virtual reality displays quickly in response to changes in the position or orientation of a user. This efficient and rapid rerendering reduces latency and improves the user's virtual reality experience.
One or more embodiments of the system include one or more displays viewable by a user. For example, these displays may be embedded in virtual reality goggles or glasses. One or more embodiments also include one or more sensors that measure aspects of the user's position, orientation, or both. Aspects of the user's orientation and position are referred to as the user's âposeâ in this specification. Pose sensors may for example measure movements of the user's head, or of the user's eyes, or more generally of any body part or parts of the user. Embodiments of the system include a pose analyzer that receives sensor data and determines the user's pose from this data. The pose information is passed to a scene renderer, which generates the 3D virtual reality display viewed by the user. This display shows a portion of a 3D scene model that is visible to the user based on the user's current pose. The 3D scene model is the model of the virtual world that the user navigates through by changing pose.
The scene renderer generates one or more 2D projections from the 3D scene model. In one or more embodiments, these projections may be generated using well known 3D graphics techniques, for example using virtual cameras and perspective projection transformations onto the view planes of the virtual cameras. The 2D projections are then transmitted to the displays.
In addition, one or more embodiments of the system include an image warper. The image warper is the system component that provides for low-latency virtual reality display via efficient rerendering of scenes. The image warper may for example monitor the pose changes of the user and rerender displayed images based on these pose changes. The rerendering performed by the image warper may be a rerendering approximation, rather than a full perspective projection from the original 3D scene model. For example, some embodiments perform rerendering approximations by warping display images in relatively simple ways to partially reflect the changes in the user's pose. These rerendering approximations may offer lower latency display updates, although in some embodiments they may not be fully realistic compared to the full rendering process.
One or more embodiments of the system perform approximate rerendering by calculating a pixel translation vector, and then translating pixels of the display by this pixel translation vector. Effectively the image warper in these embodiments may shift pixels in a calculated direction and by a calculated amount to approximate the effect of the user's movements on the display. This approximation is not full 3D rendering, but it can be performed very quickly in some embodiments, greatly reducing latency between user's movements and display updates.
One or more embodiments of the system may use hardware acceleration to modify the pixels of a display to perform approximate rerendering. For example, display hardware or graphics processing unit hardware may support commands to directly shift pixels based on a pixel translation vector. Implementing pixel translations or other approximate rerendering transformations in hardware may further reduce latency in one or more embodiments.
In one or more embodiments, the rerendering approximations performed by the image warper may only be performed if the pose changes of a user are below a particular threshold value. For large changes in pose, the approximations used by the image warper may become inadequate, and it may be preferable to perform a full 3D rendering despite the high latency. For small changes in pose, the rerendering approximations may be sufficiently realistic.
In one or more embodiments, multiple pose changes for a user may be received while a full 3D rendering process is executed. By the time the 3D rendering process has completed, the initial user pose that was used for the rendering may be out of date, since newer pose data is by then available. One or more embodiments may perform a post-rendering correction on the rendered images, using the image warper to apply updates to the rendered images prior to displaying them. These post-rendering corrections may improve synchronization between the displayed images and the user's current pose.
One or more embodiments of the system may use pose prediction to calculate or estimate the pose of a user at a future time when the rendering and display processes are complete. Pose prediction may reduce the apparent latency between changes in user pose and corresponding display updates. One or more embodiments may use pose prediction for full rendering, for image warping, or for both. Embodiments may use any desired technique for pose prediction, including for example simple extrapolation of pose changes. With pose prediction, the predicted pose is provided to the rendering or approximate rerendering processes, rather than the measured pose. The rendering process calculates virtual camera poses from the predicted pose values, and renders a scene based on these virtual camera poses. The image warper calculates pose changes using the difference between the predicted future pose and the previously calculated virtual camera pose from full rendering of the scene.
One challenge faced by some embodiments is that the image warping process may leave holes in the display images with missing pixels. For example, if all pixels are shifted to the right, then the left edge of the display will have a hole without pixel data. Embodiments may employ various approaches to handle these holes. In one or more embodiments, the 3D renderer may render 2D projections that are larger than the display area. Pixels outside the display area may be cached in these embodiments in an off-screen cache, and retrieved when performing image warping to fill holes.
Another approach to filling holes employed by one or more embodiments is to estimate pixel values for the holes based on the pixel values of nearby pixels. For example, in one or more embodiments pixel values from the boundaries of regions may be propagated into the holes to fill them. Simple propagation of boundary pixels into holes may in some cases result in visual artifacts. In one or more embodiments, blur transformations may be applied to pixels in the holes or near the holes to reduce these artifacts.
One or more embodiments may employ various types of rerendering approximations for image warping. One technique used by some embodiments is to generate a simplified 3D model from the 2D projections received from the scene rendered, and to reproject these simplified 3D models onto the updated view planes that correspond to changes in the user's pose. For example, one or more embodiments may create a simplified 3D model by mapping a 2D projection from rendering onto another plane in the simplified 3D model, where the distance of this plane from the user reflects an average or typical depth of the objects in the complete 3D scene model. The depth of such an average plane may be fixed, or it may be supplied by the scene renderer with each 2D projection. One or more embodiments may use other simplified 3D models, such as spherical or cylindrical surfaces for example.
For small changes in pose, rerendering approximations based on reprojecting from a simplified 3D planar model may be approximately equivalent to using a pixel translation vector to shift pixels in display images in response to pose changes. For example, one or more embodiments may calculate a pixel translation vector for a rotation of a user around axis {circumflex over (Ï)} by a small angle Îθ as ({circumflex over (Ï)} y Îθ,â{circumflex over (Ï)} x Îθ), which is then scaled to the reflect the pixel dimensions of the display. This formula reflects that small angular rotations of a user's view approximately result in pixels shifting in response to the rotations, with the amount of shift proportional to the angle of rotation. Changes in user pose may also involve translations (linear motions of the user). For translations, the amount of shifting of pixels is also a function of the distance of objects from a user: the closer the object to the user, the more pixels shift in response to user translations. In one or more embodiments, a rerendering approximation may be estimated by a pixel translation vector using an average depth estimate z* for the distance between the user and the objects in the 2D projection. These embodiments may calculate a pixel translation vector for a user translation by small vector Îr as (âÎr x /z*,âÎr y /z*), which is then scaled to reflect the pixel dimensions of the display. This formula reflects that objects that are further away shift less than objects that are closer. It also reflects that pixels shift in the direction opposite to the movement of the user. One or more embodiments may user pixel translation vectors for rerendering approximations that combine the above effects of user rotation and user translation, such as for example ({circumflex over (Ï)} y ÎθâÎr x /z*,â{circumflex over (Ï)} x ÎθâÎr y /z*).
In summary, one or more embodiments of the invention enable a low-latency virtual reality display by using techniques to efficiently and approximately rerender images based on changes in the user's pose. Such techniques include, but are not limited to, shifting pixels by a pixel translation vector that is calculated from the user's movements. One or more embodiments may provide additional features such as filling of holes generated by image warping, and applying corrections prior to displaying rendered images to synchronize them with the user's current pose.
A major factor driving rendering time and latency is the number of pixels in a virtual reality display. Generally displays with more pixels require more rendering computation. Rendering latency may be reduced by using a low resolution display, but this approach may compromise the user experience. One or more embodiments of the invention may instead reduce rendering computation by rendering at lower resolution in selected portions of a display. The display hardware may have a high resolution, but not all regions of the display may be rendered at this resolution. Human vision has high resolution in the center of the field of view, but lower resolution on the sides of the field of view. One or more embodiments therefore may render a virtual environment at high resolution in the center of a display, and at lower resolution on the sides of the display. This approach may reduce latency without significantly compromising the user experience since the human mind may perceive items in an area where the focus is at a different resolution than other areas. In one or more embodiments, the center of the screen for each eye may be higher resolution that the outer edges of the screen (or screens). In other embodiments, or if programmatically altered via a user input, the center of the area that the eye is pointed at may be displayed at higher resolution, for example in embodiments that employ an eye tracker.
One or more embodiments incorporate a variable resolution virtual reality display system. The system may have one or more displays. The pixels of a display may be partitioned into regions, and rendering resolution may differ across regions. Each region may be portioned into a grid of grid elements, where each grid element contains one or more pixels. The ratio of pixels per grid element may vary across regions. For example, a high resolution center display region may have a ratio of 1 pixel per grid element, while relatively low resolution side display regions to the left and right of the center region may have a ratio of 4 or more pixels per grid element. These ratios are illustrative; one or more embodiments may have grids with any desired ratio of pixels per grid element. One or more embodiments may divide a display into any number and arrangement of display regions, with any desired pixel counts in the grid elements of each region.
One or more embodiments incorporate a 3D model of a scene, such as for example a virtual environment, and render this model onto the display or displays. The scene renderer that performs the rendering may project the 3D model onto the grid of each display region, and calculate a grid element value for each grid element from this projection. It may then assign pixel values for each pixel in the grid element based on the grid element value. Grid element values and pixel values may comprise for example, without limitation, any combination of color, intensity, hue, saturation, value, luminance, chrominance, lighting, mate
This application is a continuation of U.S. Utility patent application Ser. No. 14/872,488 filed Oct. 1, 2015, issued as U.S. Pat. No. 9,607,428, which is a continuation in part of U.S. Utility patent application Ser. No. 14/788,633 filed Jun. 30, 2015, the specifications of which are hereby incorporated herein by reference.
BACKGROUND OF THE INVENTION
Field of the Invention
One or more embodiments of the invention are related to the field of virtual reality systems. More particularly, but not by way of limitation, one or more embodiments of the invention enable a virtual reality display system that renders different regions of a display at different resolutions as well as tracks movements of a user and quickly renders a virtual reality display based on those movements.
Description of the Related Art
Virtual reality systems are known in the art. Such systems generate a virtual world for a user that responds to the user's movements. Examples include various types of virtual reality headsets and goggles worn by a user, as well as specialized rooms with multiple displays. Virtual reality systems typically include sensors that track a user's head, eyes, or other body parts, and that modify the virtual world according to the user's movements. The virtual world consists of a three-dimensional model, computer-generated or captured from real-world scenes. Images of the three-dimensional model are generated based on the user's position and orientation. Generation of these images requires rendering of the three-dimensional model onto one or more two-dimensional displays. Rendering techniques are known in the art and are often used for example in 3D graphics systems or computer-based games, as well as in virtual reality systems.
A major challenge for existing virtual reality systems is combining realistic images with low-latency rendering, so that user's virtual reality experience matches the rapid feedback to movement observed in real environments. Existing systems often have long latency to measure changes in the user's position and orientation, and to rerender the virtual world based on these changes. 3D rendering is a complex and processor intensive operation that can take potentially hundreds of milliseconds. The result is that users perceive noticeable lag between their movements and the rendering of updated virtual environments on their displays. Three technology trends are compounding this challenge: (1) The complexity of 3D models is growing as more 3D data is captured and generated. (2) Resolution of virtual reality displays is increasing, requiring more computational power to render images. (3) Users are relying increasingly on mobile devices with limited processor capacity. As a result of these trends, high latency in rendering virtual reality displays has become a major factor limiting adoption and applications of virtual reality technology. There are no known systems that provide sufficiently low-latency rendering and display to generate highly responsive virtual reality environments given these technology constraints.
One factor contributing to rendering latency is the resolution of a virtual reality display. In general, displays with large numbers of pixels require more rendering computation and thus may experience greater latency. Displays known in the art typically consist of rectangular arrays of pixels with uniform pixel density throughout the display. However, human vision has high resolution only in the center of the field view. Therefore, rendering images at high resolution on the sides of a display may be unnecessary, and may contribute to higher latency without improving the user's experience.
For at least the limitations described above there is a need for a variable resolution virtual reality display system.
BRIEF SUMMARY OF THE INVENTION
One or more embodiments described in the specification are related to a low-latency virtual reality display system. Embodiments of the system use efficient approximations to rerender virtual reality displays quickly in response to changes in the position or orientation of a user. This efficient and rapid rerendering reduces latency and improves the user's virtual reality experience.
One or more embodiments of the system include one or more displays viewable by a user. For example, these displays may be embedded in virtual reality goggles or glasses. One or more embodiments also include one or more sensors that measure aspects of the user's position, orientation, or both. Aspects of the user's orientation and position are referred to as the user's âposeâ in this specification. Pose sensors may for example measure movements of the user's head, or of the user's eyes, or more generally of any body part or parts of the user. Embodiments of the system include a pose analyzer that receives sensor data and determines the user's pose from this data. The pose information is passed to a scene renderer, which generates the 3D virtual reality display viewed by the user. This display shows a portion of a 3D scene model that is visible to the user based on the user's current pose. The 3D scene model is the model of the virtual world that the user navigates through by changing pose.
The scene renderer generates one or more 2D projections from the 3D scene model. In one or more embodiments, these projections may be generated using well known 3D graphics techniques, for example using virtual cameras and perspective projection transformations onto the view planes of the virtual cameras. The 2D projections are then transmitted to the displays.
In addition, one or more embodiments of the system include an image warper. The image warper is the system component that provides for low-latency virtual reality display via efficient rerendering of scenes. The image warper may for example monitor the pose changes of the user and rerender displayed images based on these pose changes. The rerendering performed by the image warper may be a rerendering approximation, rather than a full perspective projection from the original 3D scene model. For example, some embodiments perform rerendering approximations by warping display images in relatively simple ways to partially reflect the changes in the user's pose. These rerendering approximations may offer lower latency display updates, although in some embodiments they may not be fully realistic compared to the full rendering process.
One or more embodiments of the system perform approximate rerendering by calculating a pixel translation vector, and then translating pixels of the display by this pixel translation vector. Effectively the image warper in these embodiments may shift pixels in a calculated direction and by a calculated amount to approximate the effect of the user's movements on the display. This approximation is not full 3D rendering, but it can be performed very quickly in some embodiments, greatly reducing latency between user's movements and display updates.
One or more embodiments of the system may use hardware acceleration to modify the pixels of a display to perform approximate rerendering. For example, display hardware or graphics processing unit hardware may support commands to directly shift pixels based on a pixel translation vector. Implementing pixel translations or other approximate rerendering transformations in hardware may further reduce latency in one or more embodiments.
In one or more embodiments, the rerendering approximations performed by the image warper may only be performed if the pose changes of a user are below a particular threshold value. For large changes in pose, the approximations used by the image warper may become inadequate, and it may be preferable to perform a full 3D rendering despite the high latency. For small changes in pose, the rerendering approximations may be sufficiently realistic.
In one or more embodiments, multiple pose changes for a user may be received while a full 3D rendering process is executed. By the time the 3D rendering process has completed, the initial user pose that was used for the rendering may be out of date, since newer pose data is by then available. One or more embodiments may perform a post-rendering correction on the rendered images, using the image warper to apply updates to the rendered images prior to displaying them. These post-rendering corrections may improve synchronization between the displayed images and the user's current pose.
One or more embodiments of the system may use pose prediction to calculate or estimate the pose of a user at a future time when the rendering and display processes are complete. Pose prediction may reduce the apparent latency between changes in user pose and corresponding display updates. One or more embodiments may use pose prediction for full rendering, for image warping, or for both. Embodiments may use any desired technique for pose prediction, including for example simple extrapolation of pose changes. With pose prediction, the predicted pose is provided to the rendering or approximate rerendering processes, rather than the measured pose. The rendering process calculates virtual camera poses from the predicted pose values, and renders a scene based on these virtual camera poses. The image warper calculates pose changes using the difference between the predicted future pose and the previously calculated virtual camera pose from full rendering of the scene.
One challenge faced by some embodiments is that the image warping process may leave holes in the display images with missing pixels. For example, if all pixels are shifted to the right, then the left edge of the display will have a hole without pixel data. Embodiments may employ various approaches to handle these holes. In one or more embodiments, the 3D renderer may render 2D projections that are larger than the display area. Pixels outside the display area may be cached in these embodiments in an off-screen cache, and retrieved when performing image warping to fill holes.
Another approach to filling holes employed by one or more embodiments is to estimate pixel values for the holes based on the pixel values of nearby pixels. For example, in one or more embodiments pixel values from the boundaries of regions may be propagated into the holes to fill them. Simple propagation of boundary pixels into holes may in some cases result in visual artifacts. In one or more embodiments, blur transformations may be applied to pixels in the holes or near the holes to reduce these artifacts.
One or more embodiments may employ various types of rerendering approximations for image warping. One technique used by some embodiments is to generate a simplified 3D model from the 2D projections received from the scene rendered, and to reproject these simplified 3D models onto the updated view planes that correspond to changes in the user's pose. For example, one or more embodiments may create a simplified 3D model by mapping a 2D projection from rendering onto another plane in the simplified 3D model, where the distance of this plane from the user reflects an average or typical depth of the objects in the complete 3D scene model. The depth of such an average plane may be fixed, or it may be supplied by the scene renderer with each 2D projection. One or more embodiments may use other simplified 3D models, such as spherical or cylindrical surfaces for example.
For small changes in pose, rerendering approximations based on reprojecting from a simplified 3D planar model may be approximately equivalent to using a pixel translation vector to shift pixels in display images in response to pose changes. For example, one or more embodiments may calculate a pixel translation vector for a rotation of a user around axis {circumflex over (Ï)} by a small angle Îθ as ({circumflex over (Ï)} y Îθ,â{circumflex over (Ï)} x Îθ), which is then scaled to the reflect the pixel dimensions of the display. This formula reflects that small angular rotations of a user's view approximately result in pixels shifting in response to the rotations, with the amount of shift proportional to the angle of rotation. Changes in user pose may also involve translations (linear motions of the user). For translations, the amount of shifting of pixels is also a function of the distance of objects from a user: the closer the object to the user, the more pixels shift in response to user translations. In one or more embodiments, a rerendering approximation may be estimated by a pixel translation vector using an average depth estimate z* for the distance between the user and the objects in the 2D projection. These embodiments may calculate a pixel translation vector for a user translation by small vector Îr as (âÎr x /z*,âÎr y /z*), which is then scaled to reflect the pixel dimensions of the display. This formula reflects that objects that are further away shift less than objects that are closer. It also reflects that pixels shift in the direction opposite to the movement of the user. One or more embodiments may user pixel translation vectors for rerendering approximations that combine the above effects of user rotation and user translation, such as for example ({circumflex over (Ï)} y ÎθâÎr x /z*,â{circumflex over (Ï)} x ÎθâÎr y /z*).
In summary, one or more embodiments of the invention enable a low-latency virtual reality display by using techniques to efficiently and approximately rerender images based on changes in the user's pose. Such techniques include, but are not limited to, shifting pixels by a pixel translation vector that is calculated from the user's movements. One or more embodiments may provide additional features such as filling of holes generated by image warping, and applying corrections prior to displaying rendered images to synchronize them with the user's current pose.
A major factor driving rendering time and latency is the number of pixels in a virtual reality display. Generally displays with more pixels require more rendering computation. Rendering latency may be reduced by using a low resolution display, but this approach may compromise the user experience. One or more embodiments of the invention may instead reduce rendering computation by rendering at lower resolution in selected portions of a display. The display hardware may have a high resolution, but not all regions of the display may be rendered at this resolution. Human vision has high resolution in the center of the field of view, but lower resolution on the sides of the field of view. One or more embodiments therefore may render a virtual environment at high resolution in the center of a display, and at lower resolution on the sides of the display. This approach may reduce latency without significantly compromising the user experience since the human mind may perceive items in an area where the focus is at a different resolution than other areas. In one or more embodiments, the center of the screen for each eye may be higher resolution that the outer edges of the screen (or screens). In other embodiments, or if programmatically altered via a user input, the center of the area that the eye is pointed at may be displayed at higher resolution, for example in embodiments that employ an eye tracker.
One or more embodiments incorporate a variable resolution virtual reality display system. The system may have one or more displays. The pixels of a display may be partitioned into regions, and rendering resolution may differ across regions. Each region may be portioned into a grid of grid elements, where each grid element contains one or more pixels. The ratio of pixels per grid element may vary across regions. For example, a high resolution center display region may have a ratio of 1 pixel per grid element, while relatively low resolution side display regions to the left and right of the center region may have a ratio of 4 or more pixels per grid element. These ratios are illustrative; one or more embodiments may have grids with any desired ratio of pixels per grid element. One or more embodiments may divide a display into any number and arrangement of display regions, with any desired pixel counts in the grid elements of each region.
One or more embodiments incorporate a 3D model of a scene, such as for example a virtual environment, and render this model onto the display or displays. The scene renderer that performs the rendering may project the 3D model onto the grid of each display region, and calculate a grid element value for each grid element from this projection. It may then assign pixel values for each pixel in the grid element based on the grid element value. Grid element values and pixel values may comprise for example, without limitation, any combination of color, intensity, hue, saturation, value, luminance, chrominance, lighting, material, texture, normal vector direction, transparency, size, shape, or pattern. Assigning pixel values based on grid element values may in one or more embodiments be a direct copying of the grid element value to the pixel values of each pixel within the grid element. One or more embodiments may perform any desired transformations to map grid element values into pixel values.
One or more embodiments may further optimize rendering using multiple geometry models at different levels of detail for one or more objects. For example, a scene renderer may select a level of detail for an object based entirely or in part on the resolution of the display region in which the object is rendered. A low level of detail model may be used for a low resolution display region, with for example multiple pixels per grid element; a high level of detail model may be used for a high resolution display region, such as for example a center region with one pixel per grid element.
One or more embodiments may use any rendering techniques known in the art. For example, one or more embodiments may use ray casting to render objects from a 3D model to a grid associated with a display region. Ray casting for a grid may for example project rays through a grid element towards objects in a 3D model, instead of through individual pixels as is typical in the art. Because low resolution display regions may have relatively small numbers of grid elements compared to pixels, efficiency of ray casting may be improved relative to a per-pixel ray casting approach.
One or more embodiments may use rasterization rendering techniques. For example, one or more embodiments may project geometric primitives from a 3D model onto an image plane associated with a display region, and then rasterize these projected primitives onto the grid for the display region. Rasterization may generate grid element fragments, which are then blended to form a final rasterized image on the display. Because the number of grid elements may be smaller than the number of pixels, rasterization to the grid may be more efficient that typical pixel-based rasterization.
Grid element fragments may comprise any information associated with a grid element, such as for example color, intensity, hue, saturation, value, depth, texture, normal direction, lighting, or material. In one or more embodiments a grid element fragment may also include a list of pixels associated with the fragment. In some cases rasterization may generate grid element fragments associated with all pixels contained within the grid element. However, one or more embodiments may rasterize projected primitives to a sub-grid-element level.
In one or more embodiments a 3D model may designate selected objects as high resolution objects, which for example may be rendered at a high resolution even in a low resolution display region. Other objects may be designated as low resolution objects. Rasterization of a high resolution object may for example generate grid element fragments with a single pixel per fragment, instead of fragments that comprise all pixels within a grid element. Low resolution objects may be rasterized to grid element fragments that contain all pixels within the grid element. One or more embodiments may select the resolution for rasterization of an object (such as single pixel or a complete grid element, for example) based on multiple factors instead of or in addition to the resolution of the display region in which the object is rasterized.
BRIEF DESCRIPTION OF THE DRAWINGS
The above and other aspects, features and advantages of the invention will be more apparent from the following more particular description thereof, presented in conjunction with the following drawings wherein:
FIG. 1 illustrates the key components of at least one embodiment of low-latency virtual reality display system, configured for illustration with displays, sensors, and some processing modules embedded in virtual reality goggles, and rendering performed by a wirelessly connected mobile device.
FIG. 2 shows a high-level architectural view of the embodiment shown in FIG. 1 .
FIG. 3 shows a swimlane diagram for the major rendering activities of the embodiment shown in FIG. 2 .
FIG. 4 illustrates an embodiment of the system that implements low-latency rerendering using a pixel translation.
FIG. 4A illustrates an embodiment of the system that uses hardware accelerated rerendering using offset registers for reading frame buffer memory.
FIG. 5 illustrates an embodiment of the system that executes low-latency rerendering if the changes in a user's pose are below a threshold value.
FIG. 6 shows a swimlane diagram for the major rendering activities of the embodiment shown in FIG. 5 .
FIG. 7 illustrates an embodiment of the system that performs a post-rendering modification to rendered scenes using a low-latency correction for recent changes in the user's pose.
FIG. 8 shows a swimlane diagram for the major rendering activities of the embodiment shown in FIG. 7 .
FIG. 8A shows a swimlane diagram for an embodiment of the system that use pose prediction to reduce apparent latency between pose changes and display updates.
FIG. 9 illustrates an embodiment of the system that renders a scene larger than the display into an offscreen buffer, in order to fill holes generated by low-latency rerendering transformations.
FIG. 10 illustrates an embodiment of the system that fills holes generated by low-latency rerendering transformations by extending pixels from the image boundary.
FIG. 11 illustrates an embodiment of the system that fills holes generated by low-latency rerendering transformations by blurring pixels near the image boundary.
FIG. 12 illustrates an embodiment of the system that generates a low-latency rerendering transformation by projecting the rendered image onto a plane, and then rerendering onto a modified image plane corresponding the user's modified pose.
FIG. 13 shows a 2D model of an approximate rerendering calculation that generates a pixel translation vector from small angular rotations of a user's orientation.
FIG. 14 shows a 2D model of an approximate rerendering calculation that generates a pixel translation vector from translations of a user's position.
FIG. 15 illustrates an embodiment of the system that partitions a display into a high resolution center region and two low resolution side regions; rendering onto the low resolution regions uses coarser grid elements that comprise multiple pixels.
FIG. 16 illustrates another embodiment of a display partitioned into regions with different rendering resolutions.
FIG. 17 illustrates the rendering process for an embodiment with variable resolution display regions.
FIG. 18 illustrates an embodiment of the system that renders objects to a grid, which may be coarser than the pixel grid of a display, and then maps grid element values into pixel values.
FIG. 19 illustrates an embodiment of the system that renders different objects at different levels of detail, in addition to rendering different display regions at different resolutions.
FIG. 20 illustrates an embodiment that uses ray casting to render objects to a grid of grid elements, where a grid element may comprise multiple pixels.
FIG. 21 illustrates an embodiment that uses rasterization to generate grid element fragments from projected geometric primitives associated with objects.
FIG. 22 illustrates an embodiment that incorporates a list of pixel addresses into a grid element fragment.
FIG. 23 illustrates an embodiment that identifies selected objects as high resolution objects, and that renders these high resolution objects to individual pixels even in low resolution display regions.
DETAILED DESCRIPTION OF THE INVENTION
A low-latency virtual reality display system will now be described. In the following exemplary description numerous specific details are set forth in order to provide a more thorough understanding of embodiments of the invention. It will be apparent, however, to an artisan of ordinary skill that the present invention may be practiced without incorporating all aspects of the specific details described herein. In other instances, specific features, quantities, or measurements well known to those of ordinary skill in the art have not been described in detail so as not to obscure the invention. Readers should note that although examples of the invention are set forth herein, the claims, and the full scope of any equivalents, are what define the metes and bounds of the invention.
FIG. 1 shows a high-level schematic diagram of an embodiment of the invention that embeds elements of the system into virtual reality goggles. Other embodiments may embed elements of the system into any other devices wearable by or viewable by one or more users. For example, without limitation, one or more embodiments may embed elements of the system into goggles, glasses, sunglasses, monocles, helmets, visors, binoculars, contact lenses, or ocular implants. Some embodiments may not be worn by users, but may be placed on walls, in televisions, in mirrors, on ceilings or floors, inside flight simulators or other simulators, in windshields, in windows, or in or on any other location where a virtual reality experience is desired.
In FIG. 1 , user 101 wears a head-mounted device 120 that incorporates several elements of the embodiment shown. Displays
110 and 111 are in front of the user's left and right eyes, respectively. These displays are shown offset from user 101 for exposition; in reality many embodiments may position displays of head-mounted devices directly in front of the user's eyes. While the embodiment shown has two displaysâone for each eyeâembodiments may use any number of displays, including for example only a single display, or two displays as shown in FIG. 1 , or more than two displays. In FIG. 1 , the images shown on displays
110 and 111 are different; this may be useful in one or more embodiment for example to provide a stereoscopic 3D display. One or more embodiments may use the same image for multiple displays.
Device 120 includes a sensor (or multiple sensors 121 ). Sensor 121 measures some aspect of the position or orientation of user 101 , or of changes thereto. The position and orientation of an object in three-dimensional space is referred to in the art as the âposeâ of that object. Hence sensor 121 is a type of pose sensor. One or more embodiments may measure any desired aspects of the pose of any body parts of user 101 . For example, in some embodiments sensor 121 may measure the pose of the user's head. In some embodiments sensor 121 may measure the pose of one or more of the user's eyes. Combinations of pose measurements for different body parts may also be used in one or more embodiments. Examples of sensors that may be used in one or more embodiments include, without limitation, accelerometers, gyroscopes, GPS trackers, ultrasonic rangefinders, pressure sensors, video cameras, altimeters, radars, sonars, magnetometers, flow meters, Doppler shift meters, or tilt sensors. Embodiments of the system may use only a single sensor, or multiple sensors. Some embodiments may use one or more sensors that directly measure some aspect of the pose of a body part of the user; for example, a magnetometer may provide partial orientation information directly. Some embodiments may use one or more sensors that indirectly measure pose; for example, a gyroscope may measure angular velocity, which must be integrated to yield orientation. The schematic of FIG. 1 shows sensor 121 located near the back of the head of user 101 ; this location is arbitrary and may vary in different embodiments of the invention. For example, an embodiment that uses a video camera eye tracker to measure the orientation of a user's eye may be mounted near the user's eyes. One or more embodiments may use multiple sensors at different locations of a user's body. One or more embodiments may use sensors that are not mounted on the user's body at all, but that measure some aspect of the pose of a user or one or more of the user's body parts. For example, one or more embodiments may use video cameras located near the user, and may analyze images from these cameras to determine the user's pose.
In FIG. 1 , device 120 also includes pose analyzer 122 . This element receives sensor data from the sensor or sensors 121 , and uses this data to calculate the pose of one or more body parts of user 101 . The calculations made by pose analyzer 122 will in general depend on the type of sensor or sensors 121 . For example, one or more embodiments may use inertial sensors for the sensors 121 , in which case the pose analyzer 122 may execute an inertial tracking algorithm to estimate the position and orientation of the user. Such inertial tracking algorithms are well known in the art. Embodiments may use any methodology to translate the raw sensor data into pose information. One or more embodiments may use more than one pose analyzer; for example, an embodiment with eye tracking sensors may use a separate pose analyzer for each eye. While FIG. 1 illustrates an embodiment with pose analyzer 122 mounted on device 120 that is attached to the user, embodiments may use pose analyzers that are not attached to the user, or may use a combination of pose analyzers on a user-mounted device and pose analyzers remote from the user.
In general a virtual reality device generates virtual reality display images based on the user's pose. For example, as a user moves or turns, different images are displayed to simulate the real experience of viewing different parts of a scene. This functionality requires a 3D model of one or more scenes, and a rendering system that renders views of the scene based on the user's pose. In the embodiment shown in FIG. 1 , the 3D scene model 141 and the scene renderer 142 are located in mobile device 140 . This mobile device 140 communicates with the head-mounted device 120 over a wireless network 130 . This separation of functionality between a head-mounted device and a remote device is only illustrative; embodiments may use any desired architecture to organize elements of the system into devices. For example, in one or more embodiments, all elements of the system may be incorporated into a device such as head-mounted device 120 that is worn by a user. In one or more embodiments, all of the elements of the system may be remote from the user: for example, the user's orientation may be detected by video cameras in a room, the pose analyzer and scene renderer may execute on computers in the room, and the rendered images may be displayed on monitors mounted on the walls of the room. In one or more embodiments, the system may be a distributed system with elements distributed over multiple nodes that communicate over a network; for example a 3D scene model may be hosted on a remote server, rendering may be done on a device that is local to the user but not attached to the user, and the sensors and displays may be on a user-mounted device. Embodiments may use any type of network communication between elements of the system, including wired or wireless networks, or combinations thereof. Any network media and network protocols may be used to communicate between elements of the system.
3D scene model 141 contains a 3D representation of the objects that may be displayed to the user; it is a model of the 3D âvirtual world.â This scene model may be static, or it may change over time. Dynamic 3D scene models may also change in response to user actions or to changes in user pose. The 3D scene model may include computer-generated elements, real scene data captured by cameras or 3D scanners, or combinations of computer-generated and real data. Embodiments may use any desired type of 3D scene model, and any desired data representation for the scene model such as for example, without limitation, VRML, X3D, OBJ, COLLADA, Blender, 3DS, or any other proprietary or open format for 3D information.
Scene renderer 142 generates one or more rendered 2D images from scene model 141 . In one or more embodiments of the system, the scene render generates one or more âvirtual camerasâ based on the pose data received from pose analyzer 122 . These virtual cameras have a location and orientation in the 3D space defined by the 3D scene model. In the embodiment shown in FIG. 1 , scene renderer 142 generates two virtual cameras
150 and 151 , each of which corresponds to one of the two displays
110 and 111 . Embodiments may use any number of virtual cameras and associate these virtual cameras in any desired manner with displays. Rendering generates a 2D projection for each of the virtual cameras. Techniques for rendering 2D projections from 3D scenes are well known in the art, and these techniques are implemented in many readily available software libraries and graphics processing units. Embodiments may use any of the well known techniques, software packages, or devices for 3D rendering to generate 2D projections. In the embodiment illustrated in FIG. 1 , virtual camera 150 generates 2D projection 160 , and virtual camera 151 generates 2D projection 161 . 2D projections
160 and 161 are transmitted back to device 120 over network 130 . These projections may be displayed directly on displays
110 and 111 .
In the embodiment shown in FIG. 1 , device 120 includes image warper 123 . The image warper provides a low-latency ârerenderingâ of the projections
160 and 161 for certain types of changes in the user's pose. Specifically, the image warper receives data on the virtual camera poses 150 and 151 that were used to generate projections
160 and 161 . It also receives updates to the user's pose from pose analyzer 122 . By comparing the user's new pose to the virtual camera poses used to render the 2D projections, the image warper calculates a change in pose. When a user's pose changes, the full rendering path to generate new 2D projections would require another iteration of the original rendering path: pose data would be sent to device 140 , and converted to virtual camera poses 150 and 151 ; then scene renderer 142 would generate new 2D projections from 3D scene model 141 , and transmit these new 2D projections back to device 120 . This full rendering path may be relatively slow, leading to observable latency for the user. The function of the image warper is to reduce this latency by performing a rapid ârerendering approximationâ that provides a relatively quick and efficient update to the images
110 and 111 based on changes to the pose. This rerendering approximation is not a complete rendering as would be performed by the scene renderer 142 ; instead it uses approximations to reduce the calculations and communications required to update the display, thereby reducing latency. Illustrative details of how various embodiments may perform image warping are provided below.
FIG. 2 shows a conceptual block diagram of the embodiment of FIG. 1 , illustrating the main data paths. Sensor (or sensors) 121 generate sensor data 221 . This sensor data may include, for example, angular velocity data, acceleration data, velocity data, or any other data generated by any of the types of sensors discussed above or any sensor that may measure any aspect of the pose of a user's body part. The sensor data 221 is sent to pose analyzer 122 , which generates body pose 222 from the sensor data. Body pose 222 may include multiple poses, depending on the embodiment; for example in one or more embodiments with eye trackers, body pose 222 may have a separate pose for each of the user's eyes. Body pose 222 is sent to scene renderer 142 , which takes 3D scene model 141 , and renders one or more 2D projections such as 161 . 2D projections 161 are sent to displays 111 . The scene renderer 142 also generates virtual camera poses 242 for the virtual camera or virtual cameras used to generate the 2D projections. For some subsequent changes in pose, the new body pose 222 and the virtual camera pose 242 may be sent to image warper 123 . Embodiments may use various techniques to determine when, whether, and how to use rerendering via the image warper vs. full rendering iterations via the scene renderer. Image warper 123 calculates a change in pose 250 . The change in pose 250 and the original 2D projections 161 are sent to the rerendering approximation 260 , which performs the image warper to transform 2D projection 161 into modified 2D projection 261 , which is then sent to display 111 . In some embodiments the rerendering approximation process may be repeated multiple times before another full rendering of the scene. Embodiments may employ various techniques for repeated rerendering approximations. In some embodiments for example the repeated rerendering may be âiterativeâ: warped projection 261 may be sent back to the rendering approximation 260 on path 271 , for another iteration of warping when a new body pose 222 is available. In these iterative embodiments of repeated rerendering, the pose of the last warped image may also be provided on path 272 to the pose change calculation 250 so that pose changes represent only the change from the last warped image. In other embodiments the repeated rerendering may instead by âcumulativeâ: original 2D projection 111 may be saved, and repeated rerendering approximations may be performed on the original projection rather than on the last warped image. Some embodiments may employ combinations of these iterative and cumulative rerendering approaches.
FIG. 3 shows an illustrative âswimlaneâ process timing diagram of some of the key steps described above. This diagram presumes that a 3D scene has been previously rendered and is currently displayed on the displays. Initially the Pose Analyzer calculates a pose at 303 , and sends this pose to the Scene Renderer. The Scene Renderer launches a Render process 301 which is time-consuming. If the system waited for the Render process 301 to complete, the display would not be updated until the new display 302 is available. To provide a lower latency display that is responsive to user's movements, the Pose Analyzer sends the pose 303 to the Image Warper as well. The Image Warper executes a rapid Rerender process at 304 to modify the current display based on the change in pose. This Rerender process finishes qu
CLAIMS
Claims ( 6 )
What is claimed is:
1. A variable resolution virtual reality display system comprising:
at least one display viewable by a user and partitioned into at least two display regions having different resolutions;
a pose analyzer that calculates a pose of one or more body parts of said user;
a scene renderer coupled to said at least one display and to said pose analyzer, wherein said scene renderer
receives said pose from said pose analyzer; and,
based on said pose, calculates a rendered image for each of said at least two display regions; and,
an image warper coupled to said at least one display, said scene renderer, and said pose analyzer, wherein said image warper
receives said pose from said pose analyzer;
calculates a change in pose from said pose; and,
when said change in pose is below a threshold value,
generates a rerendering approximation of said rendered image for each of said at least two display regions based on said change in pose; and,
modifies one or more pixels of said at least one display based on said rerendering approximation.
2. The system of claim 1 , wherein
said at least two display regions comprise
a high resolution center region located in a center of a field of view of said user;
a low resolution left region located in a left portion of said field of view; and,
a low resolution right region located in a right portion of said field of view.
3. The system of claim 1 , further comprising a 3D model of a scene, wherein
at least one object in said 3D model comprises at least two geometry models at different levels of detail;
for each display region of said at least two display regions, said scene renderer
selects a level of detail for said at least one object; and,
calculates said rendered image using a geometry model for said level of detail associated with said at least one object.
4. The system of claim 1 , further comprising
at least one sensor that generates sensor data that measures one or more aspects of said pose of said one or more body parts of said user; and,
wherein said pose analyzer calculates said pose of said one or more body parts of said user based on said sensor data generated by said at least one sensor.
5. The system of claim 1 , wherein said rerendering approximation comprises
calculating a pixel translation vector; and,
translating one or more pixels of said rendered image for each of said at least two display regions by said pixel translation vector.
6. The system of claim 5 , wherein calculating said pixel translation vector comprises
approximating said change in pose as a rotation around a unit vector {circumflex over (Ï)} comprising {circumflex over (Ï)} y and {circumflex over (Ï)} x by an angle Îθ;
calculating a spatial translation vector ({circumflex over (Ï)} y Îθ,â{circumflex over (Ï)} x Îθ);
calculating a scaling factor to convert spatial distances to pixels based on pixel dimensions and fields of view of said rendered image for each of said at least two display regions; and,
calculating said pixel translation vector by scaling said spatial translation vector by said scaling factor.
US15/472,243
2015-06-30
2017-03-28
Variable resolution virtual reality display system
Active - Reinstated
US10083538B2
( en )
Priority Applications (1)
Application Number
Priority Date
Filing Date
Title
US15/472,243
US10083538B2
( en )
2015-06-30
2017-03-28
Variable resolution virtual reality display system
Applications Claiming Priority (3)
Application Number
Priority Date
Filing Date
Title
US14/788,633
US9240069B1
( en )
2015-06-30
2015-06-30
Low-latency virtual reality display system
US14/872,488
US9607428B2
( en )
2015-06-30
2015-10-01
Variable resolution virtual reality display system
US15/472,243
US10083538B2
( en )
2015-06-30
2017-03-28
Variable resolution virtual reality display system
Related Parent Applications (1)
Application Number
Title
Priority Date
Filing Date
US14/872,488
Continuation
US9607428B2
( en )
2015-06-30
2015-10-01
Variable resolution virtual reality display system
Publications (2)
Publication Number
Publication Date
US20170200304A1
US20170200304A1 ( en )
2017-07-13
US10083538B2
true
US10083538B2 ( en )
2018-09-25
Family
ID=57684380
Family Applications (2)
Application Number
Title
Priority Date
Filing Date
US14/872,488
Active - Reinstated
US9607428B2
( en )
2015-06-30
2015-10-01
Variable resolution virtual reality display system
US15/472,243
Active - Reinstated
US10083538B2
( en )
2015-06-30
2017-03-28
Variable resolution virtual reality display system
Family Applications Before (1)
Application Number
Title
Priority Date
Filing Date
US14/872,488
Active - Reinstated
US9607428B2
( en )
2015-06-30
2015-10-01
Variable resolution virtual reality display system
Country Status (1)
Country
Link
US
( 2 )
US9607428B2
( en )
Cited By (5)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US20170115488A1
( en )
*
2015-10-26
2017-04-27
Microsoft Technology Licensing, Llc
Remote rendering for virtual images
US10373334B2
( en )
*
2016-10-19
2019-08-06
Seiko Epson Corporation
Computer program, object tracking method, and object tracking device
US10878527B2
( en )
*
2018-05-09
2020-12-29
Arm Limited
Variable resolution graphics processing
US11113785B2
( en )
2019-11-11
2021-09-07
International Business Machines Corporation
Mitigating latency-based display delays
US12558618B2
( en )
2020-07-20
2026-02-24
Telefonaktiebolaget Lm Ericsson (Publ)
5G optimized game rendering
Families Citing this family (96)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
JP6268303B2
( en )
2014-02-04
2018-01-24
ãã©ã¦ã³ãã¼ãã¡ã¼âã²ã¼ã«ã·ã£ããã»ãã¼ã«ã»ãã§ã«ãã«ã³ã°ã»ãã«ã»ã¢ã³ã²ã´ã¡ã³ãã³ã»ãã©ã«ã·ã¥ã³ã°ã»ã¢ã¤ã³ã²ãã©ã¼ã²ãã«ã»ãã§ã©ã¤ã³
2D image analyzer
US9847079B2
( en )
*
2016-05-10
2017-12-19
Google Llc
Methods and apparatus to use predicted actions in virtual reality environments
US10802711B2
( en )
2016-05-10
2020-10-13
Google Llc
Volumetric virtual reality keyboard methods, user interface, and interactions
US9681096B1
( en )
*
2016-07-18
2017-06-13
Apple Inc.
Light field capture
US10410328B1
( en )
2016-08-29
2019-09-10
Perceptin Shenzhen Limited
Visual-inertial positional awareness for autonomous and non-autonomous device
US11314262B2
( en )
2016-08-29
2022-04-26
Trifo, Inc.
Autonomous platform guidance systems with task planning and obstacle avoidance
US10032276B1
( en )
2016-08-29
2018-07-24
PerceptIn, Inc.
Visual-inertial positional awareness for autonomous and non-autonomous device
US10366508B1
( en )
2016-08-29
2019-07-30
Perceptin Shenzhen Limited
Visual-inertial positional awareness for autonomous and non-autonomous device
US10395117B1
( en )
2016-08-29
2019-08-27
Trifo, Inc.
Visual-inertial positional awareness for autonomous and non-autonomous tracking
US10571925B1
( en )
2016-08-29
2020-02-25
Trifo, Inc.
Autonomous platform guidance systems with auxiliary sensors and task planning
US10390003B1
( en )
2016-08-29
2019-08-20
Perceptln Shenzhen Limited
Visual-inertial positional awareness for autonomous and non-autonomous device
US10043076B1
( en )
2016-08-29
2018-08-07
PerceptIn, Inc.
Visual-inertial positional awareness for autonomous and non-autonomous tracking
US10402663B1
( en )
2016-08-29
2019-09-03
Trifo, Inc.
Visual-inertial positional awareness for autonomous and non-autonomous mapping
US10162362B2
( en )
2016-08-29
2018-12-25
PerceptIn, Inc.
Fault tolerance to provide robust tracking for autonomous positional awareness
US10571926B1
( en )
2016-08-29
2020-02-25
Trifo, Inc.
Autonomous platform guidance systems with auxiliary sensors and obstacle avoidance
US10453213B2
( en )
*
2016-08-29
2019-10-22
Trifo, Inc.
Mapping optimization in autonomous and non-autonomous platforms
GB2553353B
( en )
*
2016-09-05
2021-11-24
Advanced Risc Mach Ltd
Graphics processing systems and graphics processors
TWI625687B
( en )
*
2016-11-01
2018-06-01
ç·¯åµè³éè¡ä»½æéå ¬å¸
Interactive clothing try-on method and display system thereof and computer readable recording medium
US10713923B1
( en )
2016-11-10
2020-07-14
FetchFind LLC
System and methods for using Bluetooth and GPS technologies to assist user to avoid losing eyeglasses
KR102734508B1
( en )
*
2016-11-21
2024-11-25
ìì§ëì¤íë ì´ ì£¼ìíì¬
Gate driving circuit and display panel using the same
US10580167B1
( en )
2017-01-24
2020-03-03
Amazon Technologies, Inc.
Excluding masked regions of virtual reality (VR) frames from encoder processing
US10593122B1
( en )
*
2017-01-24
2020-03-17
Amazon Technologies, Inc.
Mirroring edge pixels
WO2018144890A1
( en )
2017-02-03
2018-08-09
Warner Bros. Entertainment, Inc.
Rendering extended video in virtual reality
CN106597673B
( en )
*
2017-02-28
2020-04-03
京ä¸æ¹ç§æéå¢è¡ä»½æéå ¬å¸
Virtual reality display device and its driving method and driving module
US10277943B2
( en )
*
2017-03-27
2019-04-30
Microsoft Technology Licensing, Llc
Selective rendering of sparse peripheral displays based on user movements
US10216260B2
( en )
*
2017-03-27
2019-02-26
Microsoft Technology Licensing, Llc
Selective rendering of sparse peripheral displays based on element saliency
WO2018226676A1
( en )
*
2017-06-05
2018-12-13
Google Llc
Smoothly varying foveated rendering
US10444761B2
( en )
2017-06-14
2019-10-15
Trifo, Inc.
Monocular modes for autonomous platform guidance systems with auxiliary sensors
US9998664B1
( en )
2017-06-20
2018-06-12
Sliver VR Technologies, Inc.
Methods and systems for non-concentric spherical projection for multi-resolution view
EP3422708A1
( en )
*
2017-06-29
2019-01-02
Koninklijke Philips N.V.
Apparatus and method for generating an image
EP3422711A1
( en )
*
2017-06-29
2019-01-02
Koninklijke Philips N.V.
Apparatus and method for generating an image
US10496104B1
( en )
2017-07-05
2019-12-03
Perceptin Shenzhen Limited
Positional awareness with quadocular sensor in autonomous platforms
GB2565302B
( en )
*
2017-08-08
2022-04-13
Sony Interactive Entertainment Inc
Head-mountable apparatus and methods
US10395418B2
( en )
2017-08-18
2019-08-27
Microsoft Technology Licensing, Llc
Techniques for predictive prioritization of image portions in processing graphics
US10311548B2
( en )
2017-09-05
2019-06-04
Microsoft Technology Licensing, Llc
Scaling render targets to a higher rendering resolution to display higher quality video frames
US10755383B2
( en )
2017-09-29
2020-08-25
Apple Inc.
Multi-space rendering with configurable transformation parameters
US10777006B2
( en )
2017-10-23
2020-09-15
Sony Interactive Entertainment Inc.
VR body tracking without external sensors
US10580117B2
( en )
2017-11-20
2020-03-03
Comcast Cable Communications, Llc
Method to generate additional level of detail when zooming in on an image
GB2569176B
( en )
*
2017-12-08
2022-04-13
Displaylink Uk Ltd
Processing visual information for display on a screen
WO2019117569A1
( en )
2017-12-14
2019-06-20
Samsung Electronics Co., Ltd.
Method and apparatus for managing immersive data
KR102604738B1
( en )
*
2017-12-22
2023-11-22
ì¼ì±ì ì주ìíì¬
Graphic processor performing warping process, Rendering system having the same and Operating method of graphic processor
CN115842907A
( en )
*
2018-03-27
2023-03-24
京ä¸æ¹ç§æéå¢è¡ä»½æéå ¬å¸
Rendering method, computer product and display device
US11067805B2
( en )
2018-04-19
2021-07-20
Magic Leap, Inc.
Systems and methods for operating a display system based on user perceptibility
US10740957B1
( en )
*
2018-06-14
2020-08-11
Kilburn Live, Llc
Dynamic split screen
US10573060B1
( en )
*
2018-06-14
2020-02-25
Kilburn Live, Llc
Controller binding in virtual domes
WO2020013567A1
( en )
*
2018-07-10
2020-01-16
ì¼ì±ì ì 주ìíì¬
Method and device for processing content
CN111047676B
( en )
*
2018-10-12
2023-04-25
ä¸å½ç§»å¨éä¿¡éå¢å¹¿è¥¿æéå ¬å¸
Image rendering method, device and storage medium
US11774983B1
( en )
2019-01-02
2023-10-03
Trifo, Inc.
Autonomous platform guidance systems with unknown environment mapping
US10466489B1
( en )
2019-03-29
2019-11-05
Razmik Ghazaryan
Methods and apparatus for a variable-resolution screen
US11284053B2
( en )
2019-03-29
2022-03-22
Razmik Ghazaryan
Head-mounted display and projection screen
US10554940B1
( en )
2019-03-29
2020-02-04
Razmik Ghazaryan
Method and apparatus for a variable-resolution screen
BR112022001434A2
( en )
2019-07-28
2022-06-07
Google Llc
Methods, systems and media for rendering immersive video content with optimized meshes
US10943388B1
( en )
*
2019-09-06
2021-03-09
Zspace, Inc.
Intelligent stylus beam and assisted probabilistic input to element mapping in 2D and 3D graphical user interfaces
US10984758B1
( en )
*
2019-11-26
2021-04-20
Apical Limited and Arm Limited
Image enhancement
US11544903B2
( en )
*
2019-12-13
2023-01-03
Sony Group Corporation
Reducing volumetric data while retaining visual fidelity
CN111290581B
( en )
*
2020-02-21
2024-04-16
京ä¸æ¹ç§æéå¢è¡ä»½æéå ¬å¸
Virtual reality display method, display device and computer readable medium
TWI906293B
( en )
*
2020-05-08
2025-12-01
ç¾åé«éå ¬å¸
Apparatus and methods for multi-layer reprojection techniques for augmented reality
GB2595872B
( en )
*
2020-06-09
2023-09-20
Sony Interactive Entertainment Inc
Gaze tracking apparatus and systems
US20220011431A1
( en )
*
2020-07-10
2022-01-13
Xin Jin
Camera sensor for lidar with doppler-sensing pixels
KR102354132B1
( en )
*
2020-10-14
2022-01-24
íêµì ì기ì ì°êµ¬ì
Apparatus for providing divided-rendered virtual reality images and method therefor
US11070768B1
( en )
2020-10-20
2021-07-20
Katmai Tech Holdings LLC
Volume areas in a three-dimensional virtual conference space, and applications thereof
US11457178B2
( en )
2020-10-20
2022-09-27
Katmai Tech Inc.
Three-dimensional modeling inside a virtual video conferencing environment with a navigable avatar, and applications thereof
US10952006B1
( en )
2020-10-20
2021-03-16
Katmai Tech Holdings LLC
Adjusting relative left-right sound to provide sense of an avatar's position in a virtual space, and applications thereof
US11076128B1
( en )
2020-10-20
2021-07-27
Katmai Tech Holdings LLC
Determining video stream quality based on relative position in a virtual space, and applications thereof
US10979672B1
( en )
2020-10-20
2021-04-13
Katmai Tech Holdings LLC
Web-based videoconference virtual environment with navigable avatars, and applications thereof
US11095857B1
( en )
2020-10-20
2021-08-17
Katmai Tech Holdings LLC
Presenter mode in a three-dimensional virtual conference space, and applications thereof
FR3117632A1
( en )
*
2020-12-14
2022-06-17
Orange
Methods and devices for immersing a user in an immersive scene and for processing 3D objects.
EP4672737A3
( en )
*
2021-01-22
2026-03-25
Tmrw Spain S.L.U.
System for broadcasting volumetric videoconferences in 3d animated virtual environment with audio information, and method for operating said system
KR20240004337A
( en )
2021-04-29
2024-01-11
ëë¹ ì¸í°ë¤ì ë ìì´ë¹
Method, apparatus and system for modeling audio objects with range
US11184362B1
( en )
2021-05-06
2021-11-23
Katmai Tech Holdings LLC
Securing private audio in a virtual conference, and applications thereof
US11743430B2
( en )
2021-05-06
2023-08-29
Katmai Tech Inc.
Providing awareness of who can hear audio in a virtual conference, and applications thereof
CN113706696B
( en )
*
2021-09-02
2023-09-19
é¿æ¥ç工大å¦
Adaptive selection method of level of detail for 3D geometric models based on object visual saliency
US20240087238A1
( en )
*
2022-06-21
2024-03-14
T-Mobile Innovations Llc
System and method for extended reality processing in a local wireless environment with reduced latency
US12022235B2
( en )
2022-07-20
2024-06-25
Katmai Tech Inc.
Using zones in a three-dimensional virtual environment for limiting audio and video
US12009938B2
( en )
2022-07-20
2024-06-11
Katmai Tech Inc.
Access control in zones
US11651108B1
( en )
2022-07-20
2023-05-16
Katmai Tech Inc.
Time access control in virtual environment application
US11928774B2
( en )
2022-07-20
2024-03-12
Katmai Tech Inc.
Multi-screen presentation in a virtual videoconferencing environment
US11876630B1
( en )
2022-07-20
2024-01-16
Katmai Tech Inc.
Architecture to control zones
US11741664B1
( en )
2022-07-21
2023-08-29
Katmai Tech Inc.
Resituating virtual cameras and avatars in a virtual environment
US11700354B1
( en )
2022-07-21
2023-07-11
Katmai Tech Inc.
Resituating avatars in a virtual environment
US11704864B1
( en )
2022-07-28
2023-07-18
Katmai Tech Inc.
Static rendering for a combination of background and foreground objects
US12368821B2
( en )
2022-07-28
2025-07-22
Katmai Tech Inc.
Optimizing physics for static objects in a three-dimensional virtual environment
US11711494B1
( en )
2022-07-28
2023-07-25
Katmai Tech Inc.
Automatic instancing for efficient rendering of three-dimensional virtual environment
US11562531B1
( en )
2022-07-28
2023-01-24
Katmai Tech Inc.
Cascading shadow maps in areas of a three-dimensional environment
US11593989B1
( en )
2022-07-28
2023-02-28
Katmai Tech Inc.
Efficient shadows for alpha-mapped models
US11956571B2
( en )
2022-07-28
2024-04-09
Katmai Tech Inc.
Scene freezing and unfreezing
US11682164B1
( en )
2022-07-28
2023-06-20
Katmai Tech Inc.
Sampling shadow maps at an offset
US11776203B1
( en )
2022-07-28
2023-10-03
Katmai Tech Inc.
Volumetric scattering effect in a three-dimensional virtual environment with navigable video avatars
US11748939B1
( en )
2022-09-13
2023-09-05
Katmai Tech Inc.
Selecting a point to navigate video avatars in a three-dimensional environment
WO2024064089A1
( en )
*
2022-09-20
2024-03-28
Apple Inc.
Image generation with resolution constraints
CN118217624A
( en )
*
2022-12-19
2024-06-21
è ¾è®¯ç§æï¼æ·±å³ï¼æéå ¬å¸
Image rendering method and related device
CN116681755B
( en )
*
2022-12-29
2024-02-09
广ä¸ç¾çç½è²å®¶çµææ¯åæ°ä¸å¿æéå ¬å¸
Position prediction method and device
GB2627210A
( en )
*
2023-02-15
2024-08-21
Sony Interactive Entertainment Inc
Graphics rendering apparatus and method
CN115883792B
( en )
*
2023-02-15
2023-05-05
æ·±å³å¸å®ç¾æ¾ç¤ºç§ææéå ¬å¸
Cross-space live-action user experience system utilizing 5G and 8K technologies
CN120359450A
( en )
*
2023-11-20
2025-07-22
京ä¸æ¹ç§æéå¢è¡ä»½æéå ¬å¸
Virtual reality display device and control method for virtual reality display device
US20250191128A1
( en )
*
2023-12-07
2025-06-12
Karl Storz Imaging, Inc
Mixed Latency Video System for Generating Images
Citations (44)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US5841439A
( en )
1994-07-22
1998-11-24
Monash University
Updating graphical objects based on object validity periods
US6445815B1
( en )
1998-05-08
2002-09-03
Canon Kabushiki Kaisha
Measurement of depth image considering time delay
US20070183649A1
( en )
2004-03-15
2007-08-09
Koninklijke Philips Electronic, N.V.
Image visualization
US20080143729A1
( en )
2006-12-15
2008-06-19
Nvidia Corporation
System, method and computer program product for adjusting a refresh rate of a display for power savings
US7469166B2
( en )
2001-06-29
2008-12-23
Honda Motor Co., Ltd.
System and method of predicting novel motion in a serial chain system
US20100111370A1
( en )
2008-08-15
2010-05-06
Black Michael J
Method and apparatus for estimating body shape
US20100232727A1
( en )
2007-05-22
2010-09-16
Metaio Gmbh
Camera pose estimation apparatus and method for augmented reality imaging
US7860676B2
( en )
2007-06-28
2010-12-28
Hillcrest Laboratories, Inc.
Real-time dynamic tracking of bias
US20110018811A1
( en )
2009-07-21
2011-01-27
Jerzy Miernik
Gradual proximity touch screen
US20110018903A1
( en )
2004-08-03
2011-01-27
Silverbrook Research Pty Ltd
Augmented reality device for presenting virtual imagery registered to a viewed surface
US20110157028A1
( en )
2009-12-31
2011-06-30
Verizon Patent And Licensing, Inc.
Text entry for a touch screen
US20110302293A1
( en )
2010-06-02
2011-12-08
Microsoft Corporation
Recognition system for sharing information
US8113991B2
( en )
2008-06-02
2012-02-14
Omek Interactive, Ltd.
Method and system for interactive fitness training program
US20120188179A1
( en )
2010-12-10
2012-07-26
Sony Ericsson Mobile Communications Ab
Touch sensitive display
US20120235902A1
( en )
2009-10-13
2012-09-20
Recon Instruments Inc.
Control systems and methods for head-mounted information systems
US20120256840A1
( en )
2011-04-10
2012-10-11
Mahmoud Razzaghi
Virtual keyboard
US20130018679A1
( en )
2011-07-14
2013-01-17
Qiuhang Richard Qian
3D version of self-changing a hotel, motel, resort room , a theater, stadium or an arena seat & business solution for their businesses
US20130235696A1
( en )
2012-03-06
2013-09-12
Schlumberger Technology Corporation
Interpretation and feature detection in a seismic volume using bump mapping and lighting
US20130238295A1
( en )
2012-03-06
2013-09-12
Samsung Electronics Co., Ltd.
Method and apparatus for pose recognition
US20130241947A1
( en )
2012-03-15
2013-09-19
Sony Corporation
Display device, image processing device, image processing method, and computer program
US8629836B2
( en )
2004-04-30
2014-01-14
Hillcrest Laboratories, Inc.
3D pointing devices with orientation compensation and improved usability
US20140049983A1
( en )
2010-11-18
2014-02-20
Anthony John Nichol
Light emitting device comprising a lightguide film and aligned coupling lightguides
US20140125471A1
( en )
2012-11-05
2014-05-08
Advanced Input Devices, Inc.
Haptic feedback systems and methods
US20140146394A1
( en )
2012-11-28
2014-05-29
Nigel David Tout
Peripheral display for a near-eye display device
US20140201690A1
( en )
2013-01-15
2014-07-17
Leap Motion, Inc.
Dynamic user interactions for display control and scaling responsiveness of display objects
US8866742B2
( en )
2004-05-28
2014-10-21
Ultimatepointer, Llc
Easily deployable interactive direct-pointing system and presentation control system and calibration method therefor
US20150029218A1
( en )
2013-07-25
2015-01-29
Oliver Michael Christian Williams
Late stage reprojection
US20150049004A1
( en )
2008-01-23
2015-02-19
Michael Frank Deering
Eye Mounted Displays and Systems Using Eye Mounted Displays
US20150097858A1
( en )
2013-10-07
2015-04-09
Sony Corporation