ABSTRACT
Abstract
A split hierarchy graphics processor system including a master node executing a virtual reality (VR) application responsive to input from a client device received over a network to generate primitives for in a VR environment. The graphics processor system including render nodes performing rendering based on the primitives for views into the VR environment taken from a location in the VR environment, the views corresponding to a grid map of the VR environment. Each of the render nodes renders, encodes and streams a corresponding sequence of frames of a corresponding view to the client device. The processor system including an asset library storing input geometries for the objects used for building the VR environment, wherein the objects in the asset library are accessible by the master node and the render nodes.
Description
TECHNICAL FIELD
The present disclosure is related to computer generated images, and more specifically to real-time rendering of computer generated graphics.
BACKGROUND OF THE DISCLOSURE
Computer rendering of virtual reality (VR) scenes in a rendering pipeline requires central processing unit (CPU) and graphic processing unit (GPU) resources. A VR scene may be rendered over a wide viewing range, though only a smaller portion of that viewing range is displayed. In addition, VR scenes may be more complex than traditional scenes, and also may require a higher frame rate for image processing to avoid motion sickness by the user.
A VR scene may be generated for example as a cube map which is used to project the VR scene from a given location onto the sides of a cube. The cube surrounds the location from which the views are generated. Specifically, each side of the cube map is generated by rendering the VR scene for the corresponding view for that location in the VR environment. A cube map viewer will blend one or more views for a given viewpoint into the VR environment.
Typically, a single computing resource including both the CPU and the GPU are used to perform the rendering of views for the cube map. However, the computing resource may be incapable of processing VR scenes without incurring motion sickness in the user without implementing some cost saving measures in terms of processing power. For example, views of a cube map that are not used to generate a user's viewpoint into the VR environment may be generated at lower resolution than those views that are used to generate the viewpoint. That is, if the user is viewing between the ranges of 280-40 degrees (forward) into a VR environment (e.g., corresponding to a location of a character), then the cube map views corresponding to 180 degrees (backwards) are rendered at low resolution. However, when the user physically turns his head to view other areas of the VR environment (e.g., rotate backwards towards 180 degrees), the views being shown on the head mounted display (HMD) are generated from low resolution images. In that case, the user may be highly susceptible to motion sickness when going from high resolution to lower resolution views.
It is in this context that embodiments of the disclosure arise.
SUMMARY
Embodiments of the present disclosure relate to a split hierarchy graphics processor system implemented in a cloud system for performing multi-server cloud VR streaming of VR content. For example, the split hierarchy graphics process system is configured to perform streaming of VR content as implemented through the generation of each side of a grid map on a separate render node/server, wherein the render nodes are controlled by a master node performing application logic (e.g., executing a gaming application). Several inventive embodiments of the present disclosure are described below.
In one embodiment, a split hierarchy graphics processor system implemented in a cloud system includes a master node executing a virtual reality (VR) application responsive to input from a client device of a user received over a network to generate a plurality of primitives for a plurality of objects in a VR environment. The graphics processor system includes a plurality of render nodes performing rendering based on the plurality of primitives for a plurality of views into the VR environment taken from a location in the VR environment, the plurality of views corresponding to a grid map of the VR environment, wherein each of the render nodes renders, encodes and streams a corresponding sequence of frames of a corresponding view to the client device. The graphics processor system includes an asset library storing input geometries for the plurality of objects used for building the VR environment, wherein the plurality of objects in the asset library are accessible by the master node and the plurality of render nodes
In one embodiment, a method for performing graphics processing using a split hierarchy graphics processor system of a cloud system is disclosed. The method including executing at a master node of a cloud system a VR application responsive to input from a client device of a user to generate a plurality of primitives for a plurality of objects in a VR environment of the VR application. The method including rendering a plurality of sequences of frames at a plurality of render nodes based on the plurality of primitives for a plurality of views into the VR environment taken from a location in the VR environment, the plurality of views corresponding to a grid map of the VR environment, wherein a corresponding sequence of frames is associated with a corresponding view. The method including storing input geometries for the plurality of objects in an asset library, the plurality of objects used for building the VR environment, wherein the plurality of objects in the asset library are accessible by the master node and the plurality of render nodes. The method includes encoding at each render node a corresponding sequence of frames. The method including streaming from each render node a corresponding sequence of frames to the client device.
In another embodiment, a non-transitory computer-readable medium storing a computer program for performing graphics processing using a split hierarchy graphics processor system of a cloud system. The computer-readable medium including program instructions for executing at a master node of a cloud system a VR application responsive to input from a client device of a user to generate a plurality of primitives for a plurality of objects in a VR environment of the VR application. The computer-readable medium including program instructions for rendering a plurality of sequences of frames at a plurality of render nodes based on the plurality of primitives for a plurality of views into the VR environment taken from a location in the VR environment, the plurality of views corresponding to a grid map of the VR environment, wherein a corresponding sequence of frames is associated with a corresponding view. The computer-readable medium including program instructions for storing input geometries for the plurality of objects in an asset library, the plurality of objects used for building the VR environment, wherein the plurality of objects in the asset library are accessible by the master node and the plurality of render nodes. The computer-readable medium including program instructions for encoding at each render node a corresponding sequence of frames. The computer-readable medium including program instructions for streaming from each render node a corresponding sequence of frames.
Other aspects of the disclosure will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the disclosure.
BRIEF DESCRIPTION OF THE DRAWINGS
The disclosure may best be understood by reference to the following description taken in conjunction with the accompanying drawings in which:
FIG. 1A illustrates a system configured for providing an interactive experience with VR content that is generated using multiple servers and/or compute nodes of a cloud system, wherein some input control may be provided through a handheld controller, and some input control may be managed through tracking of body parts as implemented through a camera, in accordance with one embodiment of the present disclosure.
FIG. 1B illustrates a system configured for providing an interactive experience with VR content that is generated using multiple servers and/or compute nodes of a cloud system, wherein some input control for editing may be provided through a handheld controller, and some input control may be managed through tracking of body parts as implemented through a camera, wherein the camera also tracks movement of the HMD for purposes of beam tracking of an RF emitter transmitting data to the HMD, in accordance with one embodiment of the present disclosure.
FIG. 1C illustrates a system configured for providing an interactive experience with VR content that is generated using multiple servers and/or compute nodes of a cloud system, wherein some input control for editing may be provided through a handheld controller, and some input control may be managed through the magnetic tracking of body parts as implemented in part through a magnetic source, in accordance with one embodiment of the present disclosure.
FIG. 2 conceptually illustrates the function of a HMD in conjunction with executing video game and for displaying VR content that is generated using multiple servers and/or compute nodes of a cloud system, in accordance with an embodiment of the disclosure.
FIG. 3 is a data flow diagram illustrating the generation of VR content at a cloud system using multiple servers and/or compute nodes of a cloud system to implement a split hierarchy graphics processor system, in accordance with one embodiment of the present disclosure.
FIG. 4 is a data flow diagram illustrating the generation of VR content at a cloud system using multiple servers and/or compute nodes of a cloud system, including six render nodes to generate views for the six sides of cube map and a master node for executing the game logic of the gaming application and managing the synchronization of the render nodes, in accordance with one embodiment of the present disclosure.
FIG. 5 illustrates a split hierarchy processor system configured to implement a graphics pipeline, the split hierarchy implemented across multiple servers and/or compute nodes of a cloud system to perform functions of a central processing unit (CPU) and/or graphics processing unit (GPU) to execute game logic and for independently rendering image frames for each of the sides of a grid map, in accordance with one embodiment of the present disclosure.
FIG. 6 is a flow diagram illustrating steps in a method for implementing a graphics pipeline using multiple servers and/or compute nodes of a cloud system, in accordance with one embodiment of the present disclosure.
FIG. 7 illustrates a computer system implementing a graphics pipeline using multiple servers and/or compute nodes of a cloud system, in accordance with one embodiment of the present disclosure. hypervisor system
FIG. 8 illustrates components of an example device that can be used to perform aspects of the various embodiments of the present disclosure, in accordance with an embodiment of the disclosure.
FIG. 9 is a block diagram of a Game System, according to various embodiments of the disclosure.
DETAILED DESCRIPTION
Although the following detailed description contains many specific details for the purposes of illustration, anyone of ordinary skill in the art will appreciate that many variations and alterations to the following details are within the scope of the present disclosure. Accordingly, the aspects of the present disclosure described below are set forth without any loss of generality to, and without imposing limitations upon, the claims that follow this description.
Generally speaking, the various embodiments of the present disclosure describe systems and methods for a split hierarchy graphics processor system implemented in a cloud system for performing multi-server cloud VR streaming of VR content. For example, the split hierarchy graphics process system is configured to perform streaming of VR content as implemented through the generation of each side of a grid map on a separate render node/server. In particular, the split hierarchy graphics processor system includes multiple servers/compute nodes on the cloud system. For example, a master node is configured to run game logic. The master node also may synchronize the operations of the plurality of render nodes (e.g., through synchronization packets), each of which renders video frames for a corresponding view of the grid map. In one implementation, the master node and the render nodes are placed into a server rack for communication purposes. In addition, synchronization is inherent between the render nodes because the multi-server system is implemented in one rack. Traditional VR systems need to sacrifice resolution quality in order to process all angles of a VR scene (e.g., rendering unviewed sides of a grid map with lower resolution) because the processing power is limited to a single hardware system performing all of the graphics processing and/or rendering. That is, as the user rotates an HMD in space using a traditional system the display is showing views that switch between low and high resolutions thereby inducing motion sickness in the user. On the other hand, embodiments of the present disclosure provide for the generation of views of each side of a grid map at the highest resolution no matter which views are used to generate a current viewpoint into the VR environment of a user. That is, embodiments of the present disclosure may grant additional CPU and/or GPU resources to an execution engine (e.g., executing a gaming and/or interactive VR application) at a cloud system due to the split architecture implementing the graphics pipeline. With the increased CPU and/or GPU resources providing at least for rendering at the highest resolution of all the views corresponding to sides of the grid map, one advantage includes reducing motion sickness of the user. For example, the images presented to the user remain at high resolution, especially as the user rotates the HMD in space to view different views of the VR environment, such that there is no switching between views of low resolution and high resolution. Several inventive embodiments of the present disclosure are described below.
With the above general understanding of the various embodiments, example details of the embodiments will now be described with reference to the various drawings.
Throughout the specification, the reference to âgaming applicationâ is meant to represent any type of interactive application that is directed through execution of input commands. For illustration purposes only, an interactive application includes applications for gaming, word processing, video processing, video game processing, etc. Further, the terms video game and gaming application are interchangeable.
FIG. 1A illustrates a system for providing user interaction with a VR scene or environment (e.g., interactive gameplay of a gaming application), in accordance with an embodiment of the disclosure. A user 100 is shown wearing an HMD 102 , wherein the HMD 102 is worn in a manner similar to glasses, goggles, or a helmet, and is configured to display a video game from an interactive gaming application or other content from interactive application, to the user 100 . The HMD 102 provides a very immersive experience to the user by virtue of its provision of display mechanisms in close proximity to the user's eyes. Thus, the HMD 102 can provide display regions to each of the user's eyes which occupy large portions or even the entirety of the field of view of the user.
In one embodiment, the <figure-callout id="102" label="HMD" filenames="US11232532-20220125-D00001.png,US11232532-20220125-D00002.png"
TECHNICAL FIELD
The present disclosure is related to computer generated images, and more specifically to real-time rendering of computer generated graphics.
BACKGROUND OF THE DISCLOSURE
Computer rendering of virtual reality (VR) scenes in a rendering pipeline requires central processing unit (CPU) and graphic processing unit (GPU) resources. A VR scene may be rendered over a wide viewing range, though only a smaller portion of that viewing range is displayed. In addition, VR scenes may be more complex than traditional scenes, and also may require a higher frame rate for image processing to avoid motion sickness by the user.
A VR scene may be generated for example as a cube map which is used to project the VR scene from a given location onto the sides of a cube. The cube surrounds the location from which the views are generated. Specifically, each side of the cube map is generated by rendering the VR scene for the corresponding view for that location in the VR environment. A cube map viewer will blend one or more views for a given viewpoint into the VR environment.
Typically, a single computing resource including both the CPU and the GPU are used to perform the rendering of views for the cube map. However, the computing resource may be incapable of processing VR scenes without incurring motion sickness in the user without implementing some cost saving measures in terms of processing power. For example, views of a cube map that are not used to generate a user's viewpoint into the VR environment may be generated at lower resolution than those views that are used to generate the viewpoint. That is, if the user is viewing between the ranges of 280-40 degrees (forward) into a VR environment (e.g., corresponding to a location of a character), then the cube map views corresponding to 180 degrees (backwards) are rendered at low resolution. However, when the user physically turns his head to view other areas of the VR environment (e.g., rotate backwards towards 180 degrees), the views being shown on the head mounted display (HMD) are generated from low resolution images. In that case, the user may be highly susceptible to motion sickness when going from high resolution to lower resolution views.
It is in this context that embodiments of the disclosure arise.
SUMMARY
Embodiments of the present disclosure relate to a split hierarchy graphics processor system implemented in a cloud system for performing multi-server cloud VR streaming of VR content. For example, the split hierarchy graphics process system is configured to perform streaming of VR content as implemented through the generation of each side of a grid map on a separate render node/server, wherein the render nodes are controlled by a master node performing application logic (e.g., executing a gaming application). Several inventive embodiments of the present disclosure are described below.
In one embodiment, a split hierarchy graphics processor system implemented in a cloud system includes a master node executing a virtual reality (VR) application responsive to input from a client device of a user received over a network to generate a plurality of primitives for a plurality of objects in a VR environment. The graphics processor system includes a plurality of render nodes performing rendering based on the plurality of primitives for a plurality of views into the VR environment taken from a location in the VR environment, the plurality of views corresponding to a grid map of the VR environment, wherein each of the render nodes renders, encodes and streams a corresponding sequence of frames of a corresponding view to the client device. The graphics processor system includes an asset library storing input geometries for the plurality of objects used for building the VR environment, wherein the plurality of objects in the asset library are accessible by the master node and the plurality of render nodes
In one embodiment, a method for performing graphics processing using a split hierarchy graphics processor system of a cloud system is disclosed. The method including executing at a master node of a cloud system a VR application responsive to input from a client device of a user to generate a plurality of primitives for a plurality of objects in a VR environment of the VR application. The method including rendering a plurality of sequences of frames at a plurality of render nodes based on the plurality of primitives for a plurality of views into the VR environment taken from a location in the VR environment, the plurality of views corresponding to a grid map of the VR environment, wherein a corresponding sequence of frames is associated with a corresponding view. The method including storing input geometries for the plurality of objects in an asset library, the plurality of objects used for building the VR environment, wherein the plurality of objects in the asset library are accessible by the master node and the plurality of render nodes. The method includes encoding at each render node a corresponding sequence of frames. The method including streaming from each render node a corresponding sequence of frames to the client device.
In another embodiment, a non-transitory computer-readable medium storing a computer program for performing graphics processing using a split hierarchy graphics processor system of a cloud system. The computer-readable medium including program instructions for executing at a master node of a cloud system a VR application responsive to input from a client device of a user to generate a plurality of primitives for a plurality of objects in a VR environment of the VR application. The computer-readable medium including program instructions for rendering a plurality of sequences of frames at a plurality of render nodes based on the plurality of primitives for a plurality of views into the VR environment taken from a location in the VR environment, the plurality of views corresponding to a grid map of the VR environment, wherein a corresponding sequence of frames is associated with a corresponding view. The computer-readable medium including program instructions for storing input geometries for the plurality of objects in an asset library, the plurality of objects used for building the VR environment, wherein the plurality of objects in the asset library are accessible by the master node and the plurality of render nodes. The computer-readable medium including program instructions for encoding at each render node a corresponding sequence of frames. The computer-readable medium including program instructions for streaming from each render node a corresponding sequence of frames.
Other aspects of the disclosure will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the disclosure.
BRIEF DESCRIPTION OF THE DRAWINGS
The disclosure may best be understood by reference to the following description taken in conjunction with the accompanying drawings in which:
FIG. 1A illustrates a system configured for providing an interactive experience with VR content that is generated using multiple servers and/or compute nodes of a cloud system, wherein some input control may be provided through a handheld controller, and some input control may be managed through tracking of body parts as implemented through a camera, in accordance with one embodiment of the present disclosure.
FIG. 1B illustrates a system configured for providing an interactive experience with VR content that is generated using multiple servers and/or compute nodes of a cloud system, wherein some input control for editing may be provided through a handheld controller, and some input control may be managed through tracking of body parts as implemented through a camera, wherein the camera also tracks movement of the HMD for purposes of beam tracking of an RF emitter transmitting data to the HMD, in accordance with one embodiment of the present disclosure.
FIG. 1C illustrates a system configured for providing an interactive experience with VR content that is generated using multiple servers and/or compute nodes of a cloud system, wherein some input control for editing may be provided through a handheld controller, and some input control may be managed through the magnetic tracking of body parts as implemented in part through a magnetic source, in accordance with one embodiment of the present disclosure.
FIG. 2 conceptually illustrates the function of a HMD in conjunction with executing video game and for displaying VR content that is generated using multiple servers and/or compute nodes of a cloud system, in accordance with an embodiment of the disclosure.
FIG. 3 is a data flow diagram illustrating the generation of VR content at a cloud system using multiple servers and/or compute nodes of a cloud system to implement a split hierarchy graphics processor system, in accordance with one embodiment of the present disclosure.
FIG. 4 is a data flow diagram illustrating the generation of VR content at a cloud system using multiple servers and/or compute nodes of a cloud system, including six render nodes to generate views for the six sides of cube map and a master node for executing the game logic of the gaming application and managing the synchronization of the render nodes, in accordance with one embodiment of the present disclosure.
FIG. 5 illustrates a split hierarchy processor system configured to implement a graphics pipeline, the split hierarchy implemented across multiple servers and/or compute nodes of a cloud system to perform functions of a central processing unit (CPU) and/or graphics processing unit (GPU) to execute game logic and for independently rendering image frames for each of the sides of a grid map, in accordance with one embodiment of the present disclosure.
FIG. 6 is a flow diagram illustrating steps in a method for implementing a graphics pipeline using multiple servers and/or compute nodes of a cloud system, in accordance with one embodiment of the present disclosure.
FIG. 7 illustrates a computer system implementing a graphics pipeline using multiple servers and/or compute nodes of a cloud system, in accordance with one embodiment of the present disclosure. hypervisor system
FIG. 8 illustrates components of an example device that can be used to perform aspects of the various embodiments of the present disclosure, in accordance with an embodiment of the disclosure.
FIG. 9 is a block diagram of a Game System, according to various embodiments of the disclosure.
DETAILED DESCRIPTION
Although the following detailed description contains many specific details for the purposes of illustration, anyone of ordinary skill in the art will appreciate that many variations and alterations to the following details are within the scope of the present disclosure. Accordingly, the aspects of the present disclosure described below are set forth without any loss of generality to, and without imposing limitations upon, the claims that follow this description.
Generally speaking, the various embodiments of the present disclosure describe systems and methods for a split hierarchy graphics processor system implemented in a cloud system for performing multi-server cloud VR streaming of VR content. For example, the split hierarchy graphics process system is configured to perform streaming of VR content as implemented through the generation of each side of a grid map on a separate render node/server. In particular, the split hierarchy graphics processor system includes multiple servers/compute nodes on the cloud system. For example, a master node is configured to run game logic. The master node also may synchronize the operations of the plurality of render nodes (e.g., through synchronization packets), each of which renders video frames for a corresponding view of the grid map. In one implementation, the master node and the render nodes are placed into a server rack for communication purposes. In addition, synchronization is inherent between the render nodes because the multi-server system is implemented in one rack. Traditional VR systems need to sacrifice resolution quality in order to process all angles of a VR scene (e.g., rendering unviewed sides of a grid map with lower resolution) because the processing power is limited to a single hardware system performing all of the graphics processing and/or rendering. That is, as the user rotates an HMD in space using a traditional system the display is showing views that switch between low and high resolutions thereby inducing motion sickness in the user. On the other hand, embodiments of the present disclosure provide for the generation of views of each side of a grid map at the highest resolution no matter which views are used to generate a current viewpoint into the VR environment of a user. That is, embodiments of the present disclosure may grant additional CPU and/or GPU resources to an execution engine (e.g., executing a gaming and/or interactive VR application) at a cloud system due to the split architecture implementing the graphics pipeline. With the increased CPU and/or GPU resources providing at least for rendering at the highest resolution of all the views corresponding to sides of the grid map, one advantage includes reducing motion sickness of the user. For example, the images presented to the user remain at high resolution, especially as the user rotates the HMD in space to view different views of the VR environment, such that there is no switching between views of low resolution and high resolution. Several inventive embodiments of the present disclosure are described below.
With the above general understanding of the various embodiments, example details of the embodiments will now be described with reference to the various drawings.
Throughout the specification, the reference to âgaming applicationâ is meant to represent any type of interactive application that is directed through execution of input commands. For illustration purposes only, an interactive application includes applications for gaming, word processing, video processing, video game processing, etc. Further, the terms video game and gaming application are interchangeable.
FIG. 1A illustrates a system for providing user interaction with a VR scene or environment (e.g., interactive gameplay of a gaming application), in accordance with an embodiment of the disclosure. A user 100 is shown wearing an HMD 102 , wherein the HMD 102 is worn in a manner similar to glasses, goggles, or a helmet, and is configured to display a video game from an interactive gaming application or other content from interactive application, to the user 100 . The HMD 102 provides a very immersive experience to the user by virtue of its provision of display mechanisms in close proximity to the user's eyes. Thus, the HMD 102 can provide display regions to each of the user's eyes which occupy large portions or even the entirety of the field of view of the user.
In one embodiment, the HMD 102 can be connected to a computer or gaming console 106 . The connection to computer 106 can be wired or wireless. In some implementations, the HMD 102 may also communicate with the computer through alternative mechanisms or channels, such as via network 112 to which both HMD 102 and the computer 106 are connected. The computer 106 can be any general or special purpose computer known in the art, including but not limited to, a gaming console, personal computer, laptop, tablet computer, mobile device, cellular phone, tablet, thin client, set-top box, media streaming device, etc. In one embodiment, the computer 106 can be configured to execute a gaming application and/or interactive VR application, and output the video and audio from the gaming or VR application for rendering by the HMD 102 . The computer 106 is not restricted to executing a gaming application but may also be configured to execute an interactive application, which outputs VR content 191 for rendering by the HMD 102 .
In another embodiment, the computer 106 functions as a thin client in communication over a network with a cloud system 114 , or back-end server system. In that manner, the cloud system 114 maintains and executes the gaming application and/or interactive VR application being played by the user 102 . The local computer 106 transmits inputs from the HMD 102 , the controller 104 and the camera 108 , to the cloud system 114 , which processes the inputs to affect the game state of the executing gaming and/or VR application. The output from the executing application, such as video data, audio data, and haptic feedback data, is transmitted to the computer 106 . The computer 106 may further process the data before transmission or may directly transmit the data to the relevant devices. For example, video and audio streams are provided to the HMD 102 , whereas the haptic feedback data is used to generate a vibration feedback command, which is provided to the controller 104 .
In one embodiment, the cloud system is configured to implement a split hierarchy graphics processor system for performing multi-server cloud VR streaming of VR content. For example, the cloud system 114 includes an executing engine 300 (e.g., gaming engine) that includes a master node 310 and a plurality of render nodes 320 . The master node 310 is configured to perform CPU functions when executing the gaming and/or interactive VR application, such as performing simulation, running scripts, and providing the necessary input (e.g., primitives) to the graphics engine for performing rendering. The master node 310 may also perform synchronization of the render nodes. The plurality of render nodes 320 is configured to perform rendering of frames for each side of a grid map. For example, for a cube map with six sides, six render nodes are used to generate views for each of the sides of the cube map, wherein each render node generates image and/or video frames for a corresponding view of the cube map.
The user 100 may operate a controller 104 to provide input for the gaming and/or interactive VR application. The connection to computer 106 can be wired or wireless. Additionally, a camera 108 can be configured to capture one or more images of the interactive environment in which the user 100 is located. These captured images can be analyzed to determine the location and movements of the user 100 , parts of the user (e.g., tracking hand gestures for input commands), the HMD 102 , and the controller 104 . In one embodiment, the controller 104 includes a light or other marker elements which can be tracked to determine its location and orientation. Additionally, HMD 102 may include one or more lights which can be tracked to determine the location and orientation of the HMD 102 . The tracking functionality as implemented in part by camera 108 provides for input commands generated through movement of the controller 104 and/or body parts (e.g., hand) of the user 100 . The camera 108 can include one or more microphones to capture sound from the interactive environment. Sound captured by a microphone array may be processed to identify the location of a sound source. Sound from an identified location can be selectively utilized or processed to the exclusion of other sounds not from the identified location. Furthermore, the camera 108 can be defined to include multiple image capture devices (e.g. stereoscopic pair of cameras), an IR camera, a depth camera, and combinations thereof.
In one embodiment, the HMD 102 , controller 104 , and camera 108 , may themselves be networked devices that connect to the network 150 to communicate with the cloud system 114 . For example, the computer 106 may be a local network device, such as a router, that does not otherwise perform video game processing, but facilitates passage network traffic. The connections to the network by the HMD 102 , controller 104 , and camera (i.e., image capture device) 108 may be wired or wireless.
In yet another embodiment, the computer 106 may execute a portion of the gaming application, while the remaining portion of the gaming application may be executed on a cloud system 114 . In other embodiments, portions of the gaming application may also be executed on HMD 102 . For example, a request for downloading the gaming and/or interactive VR application from the computer 106 may be serviced by the cloud system 114 . While the request is being serviced, the cloud system 114 may execute a portion of the gaming and or interactive VR application and provide game content to the computer 106 for rendering on the HMD 102 . The computer 106 may communicate with the cloud system 114 over a network 150 . Inputs received from the HMD 102 , the controller 104 and the camera 108 , are transmitted to the cloud system 114 , while the gaming application is downloading on to the computer 106 . The cloud system 114 processes the inputs to affect the game state of the executing gaming and/or interactive VR application. The output from the executing gaming and/or interactive VR application, such as video data, audio data, and haptic feedback data, is transmitted to the computer 106 for onward transmission to the respective devices.
Once the gaming and/or interactive VR application has been completely downloaded to the computer 106 , the computer 106 may execute the gaming and/or interactive VR application and resume game play of the gaming and/or interactive VR application from where it was left off on the cloud system 114 . The inputs from the HMD 102 , the controller 104 , and the camera 108 are processed by the computer 106 , and the game state of the gaming application is adjusted, in response to the inputs received from the HMD 102 , the controller 104 , and the camera 108 . In such embodiments, a game state of the gaming and/or interactive VR application at the computer 106 is synchronized with the game state at the cloud system 114 . The synchronization may be done periodically to keep the state of the gaming and/or interactive VR application current at both the computer 106 and the cloud system 114 . The computer 106 may directly transmit the output data to the relevant devices. For example, video and audio streams are provided to the HMD 102 , whereas the haptic feedback data is used to generate a vibration feedback command, which is provided to the controller 104 .
FIG. 1B illustrates a system configured for providing an interactive experience with VR content, wherein the VR content is generated using a split hierarchy graphics processor system for performing multi-server cloud VR streaming of the VR content, in accordance with one embodiment of the present disclosure. In particular, the system (e.g., HMD 102 , computer 106 , and/or cloud 114 ) is configured for performing multi-server cloud VR streaming of VR content. As previously described, the cloud system 114 includes an executing engine 300 (e.g., gaming engine) that includes a master node 310 (e.g., performing simulation, game logic, running scripts, generating primitives, performing synchronization, etc.) and a plurality of render nodes 320 , each of which is configured to render frames (to generate views) for a corresponding side of a grid map. FIG. 1B is similar to the system described in FIG. 1A , with the addition of the transmitter/receiver (transceiver) 110 that is configured for data delivery to the HMD 102 via RF signals, for example. The transceiver 110 is configured to transmit (by wired connection or wireless connection) the video and audio from the gaming application to the HMD 102 for rendering thereon. In one embodiment, the transceiver 110 is configured for beam forming or beam steering for purposes of optimizing data throughput (e.g., efficient transmission of information). In addition, the transceiver 110 is configured to transmit images, video, and audio of 3D digital content. In this implementation, optional camera 108 may be configured to track movement of the HMD 102 , such that the transceiver 110 may beam steer the majority of its RF power (as delivered through an RF radiation pattern) to the HMD 102 (e.g., for purpose of delivering data), in accordance with one embodiment of the present disclosure. That is, once the position of the HMD is known in space and in relation to the transceiver 110 , that information is fed back to the transceiver such that the transceiver is able to direct the majority of its transmission power (e.g., beam steer) in the direction of the HMD 102 . HMD 102 is also configured for communicating with the computer 106 through a side or secondary channel 191 , either through wired or wireless (e.g., Bluetooth, Wi-Fi, etc.) communication paths to pass information to and from the HMD 102 .
FIG. 1C illustrates a system configured for providing an interactive experience with VR content, wherein the VR content is generated using a split hierarchy graphics processor system for performing multi-server cloud VR streaming of the VR content, in accordance with one embodiment of the present disclosure. In particular, the system (e.g., HMD 102 , computer 106 , and/or cloud 114 ) is configured for performing multi-server cloud VR streaming of VR content. As previously described, the cloud system 114 includes an executing engine 300 (e.g., gaming engine) that includes a master node 310 (e.g., performing simulation, game logic, running scripts, generating primitives, performing synchronization, etc.) and a plurality of render nodes 320 , each of which is configured to render frames (to generate views) for a corresponding side of a grid map. FIG. 1C is similar to the system described in FIG. 1A , with the addition of the magnetic source 116 configured to emit a magnetic field to enable magnetic tracking of the HMD 102 , controller 104 (e.g., configured as an interface controller), or any object configured with magnetic sensors/antennas (e.g., gloves, strips located on body partsâsuch as fingers, etc.). For example, the magnetic sensors could be inductive elements. In particular, the magnetic sensors can be configured to detect the magnetic field (e.g., strength, orientation) as emitted by the magnetic source 116 . The information gathered from the magnetic sensors can be used to determine and track the location and/or orientation of the HMD 102 , controller 104 , and other interface objects, etc., such as in order to provide input commands. In embodiments, the magnetic tracking is combined with tracking performed through the camera 108 and/or inertial sensors within the HMD 102 , controller 104 and/or other interface objects.
In some implementations, the interface object (e.g., controller 104 ) is tracked relative to the HMD 102 . For example, the HMD 102 may include an externally facing camera that captured images including the interface object. In other embodiments, HMD 102 may include an IR emitter used for tracking external objects, such as the interface object. That is, HMD 102 is configured to independently track controller 104 through signals 192 (e.g., IR emitter, magnetic sensors, etc.) to determine relative position of the interface object (e.g., controller 104 ) to the HMD 102 . For example, the captured images may be analyzed to determine the location/orientation of the interface object relate to the HMD 102 , and using a known location/orientation of the HMD 102 , so as to determine the location/orientation and/or movement of the interface object in the local environment.
The way the user 100 interfaces with the virtual reality scene of a gaming application, or of the interactive VR environment, displayed in the HMD 102 can vary, and other interface devices in addition to the interface object (e.g., controller 104 ), can be used. For instance, various kinds of single-handed, as well as two-handed controllers 104 can be used. In some implementations, the controllers 104 themselves can be tracked by tracking lights included in the controllers, or tracking of shapes, sensors, and inertial data associated with the controllers 104 . Using these various types of controllers 104 , or even simply hand gestures that are made and captured by one or more cameras, and magnetic sensors, it is possible to interface, control, maneuver, interact with, and participate in the virtual reality gaming environment presented on the HMD 102 .
FIG. 2 conceptually illustrates the function of a HMD 102 in conjunction with the display of VR content 291 , wherein the VR content is generated at a back-end cloud system delivering the VR content over a network. The VR content may be generated from a gaming application and/or an interactive VR application, for example. In particular, the cloud system may include a split hierarchy graphics processor system for performing multi-server cloud VR streaming of the VR content to the HMD 102 and/or a local computer 106 for display on the HMD 102 . For example, the cloud system may include an executing engine (e.g., gaming engine) that includes a master node (e.g., performing simulation, game logic, running scripts, generating primitives, performing synchronization, etc.) and a plurality of render nodes, each of which is configured to render frames (to generate views) for a corresponding side of a grid map.
In some embodiments, the HMD 102 can be configured to independently generate VR content. In other embodiments, the VR content engine 220 is being executed on a computer 106 (not shown) that is communicatively coupled to the HMD 102 , and/or in combination with the HMD 102 . The computer may be local to the HMD (e.g., part of local area network) or may be remotely located (e.g., part of a wide area network, a cloud network, etc.) and accessed via a network. The communication between the HMD 102 and the computer 106 may follow a wired or a wireless connection protocol.
In an example, the VR content engine 220 executing an application may be a video game engine executing a gaming application and/or an interactive VR application, and is configured to receive inputs to update a game state of the gaming application. The following description of FIG. 2 is described within the context of the VR content engine 220 executing a gaming and/or interactive VR application, for purposes of brevity and clarity, and is intended to represent the execution of any application capable of generating VR content 291 . As previously described, the VR content engine can be implemented at the back-end cloud system, on the local gaming console, on the HMD 102 itself, or any combination thereof. The game state of the gaming application can be defined, at least in part, by values of various parameters of the video game which define various aspects of the current gameplay, such as the presence and location of objects, the conditions of a virtual environment, the triggering of events, user profiles, view perspectives, etc.
In the illustrated embodiment, the VR content engine 220 receives, by way of example, controller input 261 , audio input 262 and motion input 263 . The controller input 261 may be defined from the operation of a gaming controller separate from the <figure-callout id="102" label="HMD" filenames="US11232532-20220125-D00001.png,US11232532-20220125
CLAIMS
Claims ( 25 )
What is claimed is:
1. A split hierarchy graphics processor system in a cloud system, comprising:
a master node at a server of the cloud system executing a virtual reality (VR) application responsive to input from a client device of a user received over a network to generate a plurality of primitives for a plurality of objects in a VR environment;
a plurality of render nodes at the server performing rendering in parallel based on the plurality of primitives for a plurality of views into the VR environment taken from a location in the VR environment, the plurality of views corresponding to a grid map of the VR environment, wherein each of the render nodes renders, encodes and streams a corresponding sequence of frames of a corresponding view of the plurality of views to the client device, one or more of the plurality of views being selected for presentation of the VR environment based on a detected point-of-view requested by the client device; and
an asset library storing input geometries for the plurality of objects used for building the VR environment, wherein the plurality of objects in the asset library are accessible by the master node and the plurality of render nodes,
wherein the grid map provides a three dimensional representation of the VR environment centered to the location in the VR environment,
wherein each of a plurality of sides of the grid map corresponds to one of the plurality of views into the VR environment.
2. The split hierarchy graphics processor system of claim 1 , wherein frames in the corresponding sequence of frames that is streamed are each associated with an identifier and configured to be decoded and assembled as a 3D view of the VR environment based on one or more sequences of frames and the detected point-of-view into the VR environment.
3. The split hierarchy graphics processor system of claim 1 , wherein the master node is configured to broadcast the plurality of primitives simultaneously to each of the plurality of render nodes.
4. The split hierarchy graphics processor system of claim 1 , wherein the grid map includes a cube map including six sides corresponding to the plurality of views.
5. The split hierarchy graphics processor system of claim 1 , wherein each render node is dedicated to rendering a corresponding view of the grid map in a one-to-one relationship.
6. The split hierarchy graphics processor system of claim 1 , wherein the master node includes a physics simulation modeler for applying a simulation of natural or defined laws of physics to the plurality of objects.
7. The split hierarchy graphics processor system of claim 1 , wherein each render node is configured to perform rasterization, fragment shading, output merging, and frame buffering.
8. The split hierarchy graphics processor system of claim 1 ,
wherein the client device includes one or more decoders configured for decoding and buffering encoded frames streamed from the plurality of render nodes, and
wherein the client device is configured for generating the detected point-of-view at the location in the VR environment based on one or more decoded sequences of encoded frames of views into the VR environment.
9. The split hierarchy graphics processor system of claim 1 , wherein each of the render nodes is configured to attach timestamps or frame numbers to frames of the corresponding sequence of frames for synchronization of frames from different sequences of frames at the client device.
10. The split hierarchy graphics processor system of claim 1 , wherein each of the render nodes streams the corresponding sequence of frames of the corresponding view to the client device in a UDP broadcast.
11. The split hierarchy graphics processor system of claim 1 , wherein each of the render nodes generates the corresponding sequence of frames of the corresponding view in full resolution.
12. A method for performing graphics processing using a split hierarchy graphics processor system of a cloud system, comprising:
executing at a master node at a server of the cloud system a virtual reality (VR) application responsive to input from a client device of a user received over a network to generate a plurality of primitives for a plurality of objects in a VR environment of the VR application;
rendering in parallel a plurality of sequences of frames at a plurality of render nodes at the server based on the plurality of primitives for a plurality of views into the VR environment taken from a location in the VR environment, the plurality of views corresponding to a grid map of the VR environment;
storing input geometries for the plurality of objects in an asset library, the plurality of objects used for building the VR environment, wherein the plurality of objects in the asset library are accessible by the master node and the plurality of render nodes; and
encoding at each of the plurality of render nodes a corresponding sequence of frames that is associated with a corresponding view of the plurality of views; and
streaming from each of the plurality of render nodes the corresponding sequence of frames to the client device,
wherein one or more of the plurality of views being selected for presentation of the VR environment based on a detected point-of-view requested by the client device,
wherein the grid map provides a three dimensional representation of the VR environment centered to the location in the VR environment,
wherein each of a plurality of sides of the grid map corresponds to one of the plurality of views into the VR environment.
13. The method of claim 12 , further comprising:
simultaneously broadcasting from the master node the plurality of primitives to each of the plurality of render nodes.
14. The method of claim 12 , wherein the grid map includes a cube map including six sides corresponding to the plurality of views, and each render node is dedicated to rendering a corresponding view of the grid map in a one-to-one relationship.
15. The method of claim 12 , further comprising:
performing physics simulation at the master node to apply a simulation of natural or defined laws of physics to the plurality of objects.
16. The method of claim 12 , further comprising:
performing at each render node for a frame in the corresponding sequence of frames that is associated with the corresponding view rasterization, fragment shading, output merging, and frame buffering.
17. The method of claim 12 , wherein the client device is configured to:
decode each of the plurality of sequences of encoded frames; and
store decoded sequences of encoded frames in a plurality of buffers for display when called; and
generate the detected point-of-view at the location based on one or more decoded sequences of encoded frames of views into the VR environment.
18. The method of claim 12 , further comprising:
attaching at each of the render nodes timestamps or frame numbers to frames of the corresponding sequences of frames for synchronization of frames from different sequences of frames at the client device.
19. A non-transitory computer-readable medium storing a computer program for performing graphics processing using a split hierarchy graphics processor system of a cloud system, the computer-readable medium comprising:
program instructions for executing at a master node at a server of the cloud system a virtual reality (VR) application responsive to input from a client device of a user received over a network to generate a plurality of primitives for a plurality of objects in a VR environment of the VR application;
program instructions for rendering in parallel a plurality of sequences of frames at a plurality of render nodes at the server based on the plurality of primitives for a plurality of views into the VR environment taken from a location in the VR environment, the plurality of views corresponding to a grid map of the VR environment;
program instructions for storing input geometries for the plurality of objects in an asset library, the plurality of objects used for building the VR environment, wherein the plurality of objects in the asset library are accessible by the master node and the plurality of render nodes;
program instructions for encoding at each of the plurality of render nodes a corresponding sequence of frames that is associated with a corresponding view of the plurality of views; and
program instructions for streaming from each of the plurality of render nodes the corresponding sequence of frames,
wherein one or more of the plurality of views being selected for presentation of the VR environment based on a detected point-of-view requested by the client device,
wherein the grid map provides a three dimensional representation of the VR environment centered to the location in the VR environment,
wherein each of a plurality of sides of the grid map corresponds to one of the plurality of views into the VR environment.
20. The computer-readable medium of claim 19 , further comprising:
program instructions for simultaneously broadcasting from the master node the plurality of primitives to each of the plurality of render nodes.
21. The computer-readable medium of claim 19 , wherein the grid map includes a cube map including six sides corresponding to the plurality of views, and each render node is dedicated to rendering a corresponding view of the grid map in a one-to-one relationship.
22. The computer-readable medium of claim 19 , further comprising:
program instructions for performing physics simulation at the master node to apply a simulation of natural or defined laws of physics to the plurality of objects.
23. The computer-readable medium of claim 19 , further comprising:
program instructions for performing at each render node for a frame in the corresponding sequence of frames that is associated with the corresponding view rasterization, fragment shading, output merging, and frame buffering.
24. The computer-readable medium of claim 19 , wherein the client device is configured to:
decode each of the plurality of sequences of encoded frames; and
store decoded sequences of encoded frames in a plurality of buffers for display when called; and
generate the detected point-of-view at the location based on one or more decoded sequences of encoded frames of views into the VR environment.
25. The computer-readable medium of claim 19 , further comprising:
program instruction for attaching at each of the render nodes timestamps or frame numbers to frames of the corresponding sequences of frames for synchronization of frames from different sequences of frames at the client device.
US15/993,064
2018-05-30
2018-05-30
Multi-server cloud virtual reality (VR) streaming
Active
2038-07-07
US11232532B2
( en )
Priority Applications (10)
Application Number
Priority Date
Filing Date
Title
US15/993,064
US11232532B2
( en )
2018-05-30
2018-05-30
Multi-server cloud virtual reality (VR) streaming
JP2020566269A
JP7273068B2
( en )
2018-05-30
2019-03-22
Multi-server cloud virtual reality (VR) streaming
CN201980048535.7A
CN112470483B
( en )
2018-05-30
2019-03-22
Multi-server cloud Virtual Reality (VR) streaming
KR1020207037983A
KR102472152B1
( en )
2018-05-30
2019-03-22
Multi-Server Cloud Virtual Reality (VR) Streaming
KR1020227041340A
KR102606469B1
( en )
2018-05-30
2019-03-22
Multi-server Cloud Virtual Reality (VR) Streaming
EP19715751.4A
EP3804340A1
( en )
2018-05-30
2019-03-22
Multi-server cloud virtual reality (vr) streaming
PCT/US2019/023621
WO2019231525A1
( en )
2018-05-30
2019-03-22
Multi-server cloud virtual reality (vr) streaming
US17/581,697
US11636568B2
( en )
2018-05-30
2022-01-21
Split hierarchy graphics processing system
US18/306,922
US12169877B2
( en )
2018-05-30
2023-04-25
Client side processing of streams of video frames generated by a split hierarchy graphics processing system
JP2023073069A
JP7538284B2
( en )
2018-05-30
2023-04-27
Multi-server cloud virtual reality (VR) streaming
Applications Claiming Priority (1)
Application Number
Priority Date
Filing Date
Title
US15/993,064
US11232532B2
( en )
2018-05-30
2018-05-30
Multi-server cloud virtual reality (VR) streaming
Related Child Applications (1)
Application Number
Title
Priority Date
Filing Date
US17/581,697
Continuation
US11636568B2
( en )
2018-05-30
2022-01-21
Split hierarchy graphics processing system
Publications (2)
Publication Number
Publication Date
US20190370926A1
US20190370926A1 ( en )
2019-12-05
US11232532B2
true
US11232532B2 ( en )
2022-01-25
Family
ID=66041774
Family Applications (3)
Application Number
Title
Priority Date
Filing Date
US15/993,064
Active
2038-07-07
US11232532B2
( en )
2018-05-30
2018-05-30
Multi-server cloud virtual reality (VR) streaming
US17/581,697
Active
US11636568B2
( en )
2018-05-30
2022-01-21
Split hierarchy graphics processing system
US18/306,922
Active
US12169877B2
( en )
2018-05-30
2023-04-25
Client side processing of streams of video frames generated by a split hierarchy graphics processing system
Family Applications After (2)
Application Number
Title
Priority Date
Filing Date
US17/581,697
Active
US11636568B2
( en )
2018-05-30
2022-01-21
Split hierarchy graphics processing system
US18/306,922
Active
US12169877B2
( en )
2018-05-30
2023-04-25
Client side processing of streams of video frames generated by a split hierarchy graphics processing system
Country Status (6)
Country
Link
US
( 3 )
US11232532B2
( en )
EP
( 1 )
EP3804340A1
( en )
JP
( 2 )
JP7273068B2
( en )
KR
( 2 )
KR102472152B1
( en )
CN
( 1 )
CN112470483B
( en )
WO
( 1 )
WO2019231525A1
( en )
Families Citing this family (37)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
GB2555378B
( en )
*
2016-10-18
2021-06-09
Virtually Live Switzerland Gmbh
HMD delivery system and method
JP7159057B2
( en )
*
2017-02-10
2022-10-24
ããã½ãã㯠ã¤ã³ãã¬ã¯ãã¥ã¢ã« ãããã㣠ã³ã¼ãã¬ã¼ã·ã§ã³ ãªã ã¢ã¡ãªã«
Free-viewpoint video generation method and free-viewpoint video generation system
KR102138977B1
( en )
*
2018-10-10
2020-07-28
민ì½ë·ì£¼ìíì¬
System of Providing Gaming Video Using Cloud Computer
US11625806B2
( en )
*
2019-01-23
2023-04-11
Qualcomm Incorporated
Methods and apparatus for standardized APIs for split rendering
US11328471B2
( en )
*
2019-05-24
2022-05-10
Nvidia Corporation
Fine grained interleaved rendering applications in path tracing for cloud computing environments
US11159776B2
( en )
*
2019-08-16
2021-10-26
At&T Intellectual Property I, L.P.
Method for streaming ultra high definition panoramic videos
CN111179437B
( en )
*
2019-12-30
2023-08-08
ä¸æµ·æ¼ææ°åææ¯è¡ä»½æéå ¬å¸
Cloud VR connectionless streaming system and connection method
GB2594714B
( en )
*
2020-05-04
2022-12-07
Createc Robotics Ltd
Virtual view generation
US11350138B2
( en )
*
2020-05-20
2022-05-31
Sony Corporation
Managing a multi-view event comprising several streams, stream buffers, and rendering onto a single canvas
US11637913B2
( en )
*
2020-07-15
2023-04-25
Stryker Corporation
Automatic correspondence of encoders and decoders to video ports
EP3958574A1
( en )
*
2020-08-19
2022-02-23
XRSpace CO., LTD.
Method and system for rendering virtual environment
CN114640670B
( en )
*
2020-11-30
2023-07-21
ä¸ç§»(æé½)ä¿¡æ¯éä¿¡ç§ææéå ¬å¸
Method, device and equipment for virtual reality VR information sharing
CN112835621B
( en )
*
2021-01-13
2024-04-02
西å®é£è¶èæç°å®ç§ææéå ¬å¸
Cross-platform virtual reality resource processing method and processing system
WO2022159628A1
( en )
2021-01-22
2022-07-28
Zinn Labs, Inc.
Headset integrated into healthcare platform
US11625095B2
( en )
*
2021-01-22
2023-04-11
Zinn Labs, Inc.
Gaze sensors and display elements for detection of gaze vectors and user control at headset
US11599338B2
( en )
*
2021-06-18
2023-03-07
Qingdao Pico Technology Co., Ltd.
Model loading method and apparatus for head-mounted display device, and head-mounted display device
US12132967B2
( en )
*
2021-06-30
2024-10-29
Tencent America LLC
Bidirectional presentation datastream using control and data plane channels
CN113476830A
( en )
*
2021-07-26
2021-10-08
西å®ä¸åçµåç§ææéå ¬å¸
Cloud game data processing method and device, game terminal and processor
US11693491B1
( en )
*
2021-08-23
2023-07-04
Waleed Abdulla
Tracking a paired peripheral input device based on a contact criterion
KR102669261B1
( en )
*
2021-10-15
2024-05-28
(주)ì´ë¨¸ìë¸ìºì¤í¸
Cloud xr-based program virtualizing method
CN113645484B
( en )
*
2021-10-16
2021-12-21
æé½ä¸ç§åè¿ ç§ææéå ¬å¸
Data visualization accelerated rendering method based on graphic processor
US20230117444A1
( en )
*
2021-10-19
2023-04-20
Microsoft Technology Licensing, Llc
Ultra-low latency streaming of real-time media
US11962867B2
( en )
*
2021-11-04
2024-04-16
Tencent America LLC
Asset reusability for lightfield/holographic media
CN114170364B
( en )
*
2021-12-10
2025-10-21
å京åè·³ç½ç»ææ¯æéå ¬å¸
Method, device and electronic device for determining rendering process
US20230308846A1
( en )
*
2022-02-08
2023-09-28
Meta Platforms Technologies, Llc
Systems and methods for multicast communication for ar/vr system
CN114442814B
( en )
*
2022-03-31
2022-09-16
ç¯å½±ç§ææéå ¬å¸
A cloud desktop display method, device, device and storage medium
TWI807860B
( en )
*
2022-06-14
2023-07-01
åé²ç§æè¡ä»½æéå ¬å¸
Virtual Reality Network Performer System and Computer Program Products
CN115190156B
( en )
*
2022-07-05
2023-03-21
æ·±å³å¸å¼åç§é信设å¤æéå ¬å¸
Dual-network data transmission method and virtual reality system
US20240108984A1
( en )
*
2022-09-29
2024-04-04
Sony Interactive Entertainment Inc.
Game asset optimization over network at optimizer server
EP4351157A1
( en )
*
2022-10-06
2024-04-10
Holo-Light GmbH
Xr streaming method and xr streaming system
US12327315B2
( en )
2022-10-25
2025-06-10
Bank Of America Corporation
System and method for auto-generating and sharing customized virtual environments
WO2024237964A1
( en )
*
2023-05-17
2024-11-21
Qualcomm Incorporated
Mixed reality split-chip power loop control flow
CN116563085B
( en )
*
2023-07-06
2023-09-01
æ¹å马æ å±±è§é¢å è¿ææ¯ç 究é¢æéå ¬å¸
Large-scale parallel processing method and system for offline rendering
CN117472188B
( en )
*
2023-12-07
2024-04-19
èéæ²é³ä¹æåæéå ¬å¸
VR gesture information control device and method
US12519850B2
( en )
2024-02-29
2026-01-06
Microsoft Technology Licensing, Llc
Peer-to-peer ultra-low latency streaming of real-time media
KR102859925B1
( en )
2024-04-30
2025-09-16
í½ì ë¦¬í° ì£¼ìíì¬
Virtual reality implementation system with level streaming method for graphics optimization
CN120849135B
( en )
*
2025-09-22
2025-12-02
èéå¨çº¿ä¿¡æ¯ç§ææéå ¬å¸
Cloud rendering method and system for units application and electronic equipment
Citations (25)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US20060059024A1
( en )
*
2002-11-07
2006-03-16
Flytecomm, Inc.
Advanced travel management system
US20070024594A1
( en )
*
2005-08-01
2007-02-01
Junichiro Sakata
Information processing apparatus and method, and program
US20090241037A1
( en )
*
2008-03-18
2009-09-24
Nortel Networks Limited
Inclusion of Web Content in a Virtual Environment
US20090265661A1
( en )
*
2008-04-14
2009-10-22
Gary Stephen Shuster
Multi-resolution three-dimensional environment display
US20120154527A1
( en )
*
2010-12-21
2012-06-21
Priya Dwivedula
Method and System for Frame Rate Conversion of 3D Frames
US20120191476A1
( en )
*
2011-01-20
2012-07-26
Reid C Shane
Systems and methods for collection, organization and display of ems information
US20120307096A1
( en )
*
2011-06-05
2012-12-06
Apple Inc.
Metadata-Assisted Image Filters
US20130070051A1
( en )
*
2011-09-20
2013-03-21
Cheng-Tsai Ho
Video encoding method and apparatus for encoding video data inputs including at least one three-dimensional anaglyph video, and related video decoding method and apparatus
GB2506727A
( en )
2012-07-30
2014-04-09
Nvidia Corp
Server-rendering of graphics for remote client
US20140210819A1
( en )
*
2013-01-30
2014-07-31
Qualcomm Incorporated
Output ordering of domain coordinates for tessellation
US20160188936A1
( en )
*
2012-02-06
2016-06-30
Cognex Corporation
System and method for expansion of field of view in a vision system
US20170104819A1
( en )
*
2015-10-11
2017-04-13
Microsoft Technology Licensing, Llc
Synchronization scheme for physics simulations
US9659400B2
( en )
2007-10-29
2017-05-23
Julian Michael Urbach
Efficiently implementing and displaying independent 3-dimensional interactive viewports of a virtual world on multiple client devices
US20170227765A1
( en )
*
2016-02-10
2017-08-10
Advanced Micro Devices, Inc.
Method and system for streaming information in wireless virtual reality
US20170280126A1
( en )
*
2016-03-23
2017-09-28
Qualcomm Incorporated
Truncated square pyramid geometry and frame packing structure for representing virtual reality video content
US20170323469A1
( en )
2016-05-05
2017-11-09
Nvidia Corporation
Stereo multi-projection implemented using a graphics processing pipeline
US20180007395A1
( en )
*
2015-01-12
2018-01-04
Nokia Technologies Oy
A Method, An Apparatus and a Computer Readable Storage Medium for Video Streaming
WO2018011054A1
( en )
2016-07-15
2018-01-18
Koninklijke Kpn N.V.
Streaming virtual reality video
US20180034583A1
( en )
*
2016-08-01
2018-02-01
Sony Interactive Entertainment America Llc
Forward error correction for streaming data
US20180081429A1
( en )
2016-09-16
2018-03-22
Tomas G. Akenine-Moller
Virtual reality/augmented reality apparatus and method
US20180189980A1
( en )
*
2017-01-03
2018-07-05
Black Sails Technology Inc.
Method and System for Providing Virtual Reality (VR) Video Transcoding and Broadcasting
US20190011566A1
( en )
*
2017-07-04
2019-01-10
Baidu Online Network Technology (Beijing) Co., Ltd.
Method and apparatus for identifying laser point cloud data of autonomous vehicle
US10410376B1
( en )
*
2016-09-26
2019-09-10
Amazon Technologies, Inc.
Virtual reality media content decoding of portions of image frames
US10412412B1
( en )
*
2016-09-30
2019-09-10
Amazon Technologies, Inc.
Using reference-only decoding of non-viewed sections of a projected video
US20200228524A1
( en )
*
2017-08-23
2020-07-16
Visa International Service Association
Secure authorization for access to private data in virtual reality
Family Cites Families (17)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US7120205B2
( en )
*
2002-01-15
2006-10-10
Jawad Ahmed Salehi
Method and system for multiple-shift code acquisition of optical orthogonal codes in optical CDMA systems
US7068722B2
( en )
*
2002-09-25
2006-06-27
Lsi Logic Corporation
Content adaptive video processor using motion compensation
CA2569524A1
( en )
2004-06-01
2005-12-15
Supun Samarasekera
Method and system for performing video flashlight
CN101197647A
( en )
*
2006-12-13
2008-06-11
åå·å·å¤§æºè软件è¡ä»½æéå ¬å¸
Multi-channel real-time 3D scene rendering display method
US9454847B2
( en )
*
2009-02-24
2016-09-27
Google Inc.
System and method of indicating transition between street level images
US9294335B2
( en )
*
2012-04-09
2016-03-22
Empire Technology Development Llc
Processing load distribution
EP2711670B1
( en )
*
2012-09-21
2019-01-30
NavVis GmbH
Visual localisation
CA2885501C
( en )
*
2014-03-20
2023-12-19
Nanjing Yuyan Information Technology Ltd.
Efficient software for transcoding to hevc on multi-core processors
US20150301708A1
( en )
*
2014-04-21
2015-10-22
VMIX Media, Inc.
Video Editing Graphical User Interface
EP4503602A3
( en )
2014-09-03
2025-04-16
Nevermind Capital LLC
Methods and apparatus for capturing, streaming and/or playing back content
JP2018501532A
( en )
2014-10-02
2018-01-18
ã·ã¼ã¡ã³ã¹ ã¢ã¯ãã¨ã³ã²ã¼ã«ã·ã¤ããï¼³ï½ï½ ï½ï½ ï½ï½ Ａï½ï½ï½ï½ ï½ï½ï½ ï½ï½ ï½ï½ï½ï½ï½ï½ï½ï½
Automation programming in 3D graphic editors using tightly coupled logic and physical simulation
JP6620079B2
( en )
2016-09-08
2019-12-11
æ ªå¼ä¼ç¤¾ã½ãã¼ã»ã¤ã³ã¿ã©ã¯ãã£ãã¨ã³ã¿ãã¤ã³ã¡ã³ã
Image processing system, image processing method, and computer program
US10356417B2
( en )
*
2016-09-30
2019-07-16
Intel Corporation
Method and system of video coding using projected motion vectors
US10917564B2
( en )
2016-10-12
2021-02-09
Qualcomm Incorporated
Systems and methods of generating and processing files for partial decoding and most interested regions
CN106780769B
( en )
*
2016-12-23
2020-11-13
太ç«ç§æ(å京)æéå ¬å¸
Three-dimensional model drawing system and method for reducing shielding of close-distance object
US20180242017A1
( en )
*
2017-02-22
2018-08-23
Twitter, Inc.
Transcoding video
US10559126B2
( en )
*
2017-10-13
2020-02-11
Samsung Electronics Co., Ltd.
6DoF media consumption architecture using 2D video decoder
2018
2018-05-30
US
US15/993,064
patent/US11232532B2/en
active
Active
2019
2019-03-22
EP
EP19715751.4A
patent/EP3804340A1/en
active
Pending
2019-03-22
JP
JP2020566269A
patent/JP7273068B2/en
active
Active
2019-03-22
WO
PCT/US2019/023621
patent/WO2019231525A1/en
not_active
Ceased
2019-03-22
CN
CN201980048535.7A
patent/CN112470483B/en
active
Active
2019-03-22
KR
KR1020207037983A
patent/KR102472152B1/en
active
Active
2019-03-22
KR
KR1020227041340A
patent/KR102606469B1/en
active
Active
2022
2022-01-21
US
US17/581,697
patent/US11636568B2/en
active
Active
2023
2023-04-25
US
US18/306,922
patent/US12169877B2/en
active
Active
2023-04-27
JP
JP2023073069A
patent/JP7538284B2/en
active
Active
Patent Citations (25)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US20060059024A1
( en )
*
2002-11-07
2006-03-16
Flytecomm, Inc.
Advanced travel management system
US20070024594A1
( en )
*
2005-08-01
2007-02-01
Junichiro Sakata
Information processing apparatus and method, and program
US9659400B2
( en )
2007-10-29
2017-05-23
Julian Michael Urbach
Efficiently implementing and displaying independent 3-dimensional interactive viewports of a virtual world on multiple client devices
US20090241037A1
( en )
*
2008-03-18
2009-09-24
Nortel Networks Limited
Inclusion of Web Content in a Virtual Environment
US20090265661A1
( en )
*
2008-04-14
2009-10-22
Gary Stephen Shuster
Multi-resolution three-dimensional environment display
US20120154527A1
( en )
*
2010-12-21
2012-06-21
Priya Dwivedula
Method and System for Frame Rate Conversion of 3D Frames
US20120191476A1
( en )
*
2011-01-20
2012-07-26
Reid C Shane
Systems and methods for collection, organization and display of ems information
US20120307096A1
( en )
*
2011-06-05
2012-12-06
Apple Inc.
Metadata-Assisted Image Filters
US20130070051A1
( en )
*
2011-09-20
2013-03-21
Cheng-Tsai Ho
Video encoding method and apparatus for encoding video data inputs including at least one three-dimensional anaglyph video, and related video decoding method and apparatus
US20160188936A1
( en )
*</s