Google Patentsopen access

Virtual reality session capture and replay systems and methods — Vreal Inc (US9616338B1)

ABSTRACT

Abstract

Provided herein are systems and methods for dynamically capturing the graphics engine function calls and other virtual reality object data as it is created during a virtual reality session by a client virtual reality device and creating a modified virtual reality session data stream based on the captured virtual reality session for rendering a virtual reality replay session of the captured virtual reality session by the same or another client virtual reality device (or by multiple client replay devices either independently and/or in a synchronized, interactive replay), where the modified virtual reality session data stream allows the virtual reality replay session to mimic the virtual reality scenes and environment created during the captured virtual reality session, while permitting a “third person” spectator perspective to be selected, and dynamically re-selected, during the course of the virtual reality replay session.

Description

FIELD

The present disclosure relates to computing techniques for generating virtual reality sessions and more particularly, to systems and methods for capturing and replaying such virtual reality sessions from dynamically selectable viewing perspectives.

BACKGROUND

Virtual reality or virtual realities (VR) (also sometimes interchangeably referred to as immersive multimedia or computer-simulated reality) describes a simulated environment designed to provide a user with an interactive sensory experience that seeks to replicate the sensory experience of the user's physical presence in an artificial environment, such as a reality-based environment or a non-reality-based environment, such as a video game. A virtual reality may include audio and haptic components, in addition to a visual component.

The visual component of a virtual reality may be displayed either on a computer screen or with a stereoscopic head-mounted display (HMD), such as the Rift, a virtual reality head-mounted display headset developed by Oculus VR of Seattle, Wash. Some conventional HMDs simply project an image or symbology on a wearer's visor or reticle. The projected image is not slaved to the real world (i.e., the image does not change based on the wearer's head position). Other HMDs incorporate a positioning system that tracks the wearer's head position and angle, so that the picture or symbology projected by the display is congruent with the outside world using see-through imagery. Head-mounted displays may also be used with tracking sensors that allow changes of angle and orientation of the wearer to be recorded. When such data is available to the system providing the virtual reality environment, it can be used to generate a display that corresponds to the wearer's the angle-of-look at the particular time. This allows the wearer to “look around” a virtual reality environment simply by moving the head without the need for a separate controller to change the angle of the imagery. Wireless-based systems allow the wearer to move about within the tracking limits of the system. Appropriately placed sensors may also allow the virtual reality system to track the HMD wearer's hand movements to allow natural interaction with content and a convenient game-play mechanism

However, a condition referred to as virtual reality sickness (also known as cybersickness) may occur when a user is exposed to a virtual reality environment. Virtual reality sickness can cause symptoms that are similar to the symptoms caused by motion sickness, such as general discomfort, headache, stomach awareness, nausea, vomiting, pallor, sweating, fatigue, drowsiness, disorientation, and apathy. Other symptoms may include postural instability and retching. Virtual reality sickness is distinguishable from motion sickness because it can be caused by the visually-induced perception of self-motion; real self-motion is not needed.

A significant aggravating factor in virtual reality sickness is whether the visual component of the virtual reality being experienced by the user represents is tied to the user's self-motion. In other words, if a user is “seeing” a virtual reality where the viewing-user's perspective is controlled by another user (or a computer), the viewing-user is more likely to develop symptoms of virtual reality sickness (this may be analogous to the anecdotally common real-world scenario of a passenger in a car developing symptoms of motion sickness while the driver of the car does not). For example, if the user controlling the perspective may unexpectedly swivel his or her head from side to side and the viewing-user is not expecting that sudden shift in perspective, virtual reality sickness may result.

Some conventional software applications, particularly video games where multiple players interact with a video game program within the same video game session, allow non-participating, “spectator,” users to passively interact with the game session, e.g. by selectively navigating around the game session's environment, while the game session is ongoing.

Some conventional video games (and game engines) may include a static replay creation feature that records game-state data corresponding to a game session and allows a user to view a replay video of the game session within the video game. Third party tools that capture the display data generated during a game session and stores it in a video format for later playback are also known. Video-based replays created by the third party tools support sharing via online playback and commonly available video players but require much more storage space than replays created with in the context of the video game itself.

These exiting tools illustrate a recognition that video game users may desire to record a game session in order to replay at a later time or to distribute for others replay, e.g. for entertainment purposes, similar to viewing a recorded sporting event. Game replays may also be used in the prevention and detection of cheating or to develop skill at a video game, e.g. by watching better players and observing their techniques. Video game replays may also be used to raise stature in a particular game's user community, e.g. to establish rankings among players or to settle disputes. Video game replays are also used to produce replay reviews, i.e. a review of a game replay by an experienced player to determine what can be done to improve and expand upon the original player's skill.

Because many conventional video game replay techniques are designed to recreate the player's perspective, they cannot be used in a virtual reality implementation because a user experiencing a replay of another user's virtual reality perspective may be at a heightened risk of developing virtual reality sickness. Such concerns over virtual reality sickness may be a major barrier to applying conventional video game replay techniques to virtual reality video games.

Additionally, many virtual reality-based video games create expansive and detailed virtual environments which may not be sufficiently explorable using conventional player-perspective video replay techniques, such as static video capture.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates exemplary network topology of a virtual reality session capture/replay system in accordance with at least one embodiment.

FIG. 2 illustrates several components of an exemplary client virtual reality capture device in accordance with at least one embodiment.

FIG. 3 illustrates several components of an exemplary client virtual reality replay device in accordance with at least one embodiment.

FIG. 4 illustrates several components of an exemplary virtual reality session capture/replay server in accordance with at least one embodiment.

FIG. 5 illustrates a virtual reality session capture data flow between various hardware and software components of an exemplary client virtual reality capture device in accordance with at least one embodiment.

FIG. 6 illustrates a virtual reality session replay data flow between various hardware and software components of an exemplary client virtual reality replay device in accordance with at least one embodiment.

FIG. 7 illustrates a first exemplary series of data communications between execution threads instantiated in memory of an exemplary client virtual reality capture device in accordance with at least one embodiment.

FIGS. 8A-8B illustrate a second exemplary series of data communications between execution threads instantiated in memory of an exemplary client virtual reality capture device in accordance with at least one embodiment.

FIG. 9 illustrates a third exemplary series of data communications between execution threads instantiated in memory of an exemplary client virtual reality capture device in accordance with at least one embodiment.

FIG. 10 illustrates a fourth exemplary series of data communications between an exemplary client virtual reality capture device, an exemplary virtual reality session capture/replay server, and an exemplary client virtual reality replay device in accordance with at least one embodiment.

FIG. 11 illustrates a fifth exemplary series of data communications between execution threads instantiated in memory of an exemplary client virtual reality replay device in accordance with at least one embodiment.

FIG. 12 illustrates a flow diagram of an exemplary virtual reality session capture routine in accordance with at least one embodiment.

FIG. 13 illustrates a flow diagram of an exemplary graphics engine function call capture sub-routine in accordance with at least one embodiment.

FIG. 14 illustrates a flow diagram of an exemplary positional data capture sub-routine in accordance with at least one embodiment.

FIG. 15 illustrates a flow diagram of an exemplary virtual reality replay session routine in accordance with at least one embodiment.

FIG. 16 illustrates a flow diagram of an exemplary virtual reality replay session frame update sub-routine in accordance with at least one embodiment.

DESCRIPTION

The detailed description that follows is represented largely in terms of processes and symbolic representations of operations by various computer components, including a processor, memory storage devices for the processor, connected display devices and input devices. Furthermore, these processes and operations may utilize conventional computer components in a heterogeneous distributed computing environment, including remote file servers, computer servers, and/or memory storage devices. Each of these conventional distributed computing components is accessible by the processor via a communication network, which may include, but is not limited to, the Internet.

The phrases “in one embodiment,” “in various embodiments,” “in some embodiments,” and the like are used repeatedly. Such phrases do not necessarily refer to the same embodiment. The terms “comprising,” “having,” and “including” are synonymous, unless the context dictates otherwise.

Reference is now made in detail to the description of the embodiments as illustrated in the drawings. While embodiments are described in connection with the drawings and related descriptions, there is no intent to limit the scope to the embodiments disclosed herein. On the contrary, the intent is to cover all alternatives, modifications, and equivalents. In alternate embodiments, additional devices, or combinations of illustrated devices, may be added to, or combined, without limiting the scope to the embodiments disclosed herein. For example, the embodiments set forth below are primarily described in the context of capturing and replaying visual/video data. However, these embodiments are exemplary and are in no way limited to visual/video data, and should be construed to include capturing and replaying audio data, haptic data, and the like.

In various embodiments described herein, the present methods and systems may be employed to capture and store a game-state data set corresponding to a VR game session and use the game-state data set to provide a virtual reality experience corresponding to a replay of the VR game session with a dynamically selectable spectator-perspective of the game session's virtual environment.

In an exemplary embodiment, a VR gaming replay service provider may provide systems and methods for: obtaining a game state data set as it is created during a virtual reality gaming session by a VR gaming device; creating a modified game-state data set based on the obtained game-state data set; and generating a data stream corresponding to the modified game-state data set for rendering a virtual reality replay session of the virtual reality gaming session by a client replay device (or by multiple client replay devices either independently and/or in a synchronized, interactive replay), where the modified game-state data set allows the virtual reality replay game session to mimic the virtual environment created during the VR game session, while permitting the client replay device to select (and dynamically re-select during the course of the replay game session) a “third person” spectator perspective. That is, a user/spectator of the replay game session may select his/her view point in the virtual environment created during the original game session, independent of the viewpoint of the original player.

In accordance with various aspects of the present embodiments, game event data for a game session may be obtained via at least two game-state data capture techniques operating in parallel on a remote server as the virtual reality game session is being created on a VR gaming device. In a first exemplary technique, the remote server may passively capture game function calls that are redirected from the VR gaming device to the remote server during the normal course of the virtual reality game session, e.g. via replay redirect instructions embedded into specific functions of the video game itself. In a second exemplary technique, the remote server may actively query the VR gaming device for game event data during the virtual reality game session, such as event data relating to the rotation, position and scale of three dimensional objects in the virtual environment (hereafter collectively referred to as positional event data and positional events, respectively).

In accordance with various aspects of some embodiments, the remote server may capture data corresponding to some game events data at a relatively high frame rate (e.g. the same frame rate as the original game session) and may capture data corresponding to other game events at a relatively low frame rate (e.g. 25%, 33%, 50%, 75%, etc. of the original game session frame rate). When generating a replay game session, in order for the replay game session to be capable of being rendered by a client replay device at the same frame rate as the original game session and with comparable visual quality, the remote server may apply one or more interpolation techniques to the game event data that was captured at the lower frame rate in order to match the corresponding gaps in the game event data that was captured at the high frame rate.

For example, the passive captured function call capture technique described above may capture data at the same frame rate as the game session is proceeding, while the active positional event querying technique described above may capture game event data at a relatively low frame rate, e.g. at 50% of the frame rate of the original game session. During generation of a replay game session at the same frame rate as the original game session, interpolation techniques may be used to provide estimated positional event data for those temporal interval, i.e. the display time of a single frame of a game session, where no positional event data was obtained directly.

An Exemplary Network Topology of a Virtual Reality Session Capture/Replay System

FIG. 1 illustrates an exemplary network topology of a virtual reality session capture/ replay system 100 in accordance with various embodiments. Client <figure-callout id="200" label="VR capture device" filenam

FIELD

The present disclosure relates to computing techniques for generating virtual reality sessions and more particularly, to systems and methods for capturing and replaying such virtual reality sessions from dynamically selectable viewing perspectives.

BACKGROUND

Virtual reality or virtual realities (VR) (also sometimes interchangeably referred to as immersive multimedia or computer-simulated reality) describes a simulated environment designed to provide a user with an interactive sensory experience that seeks to replicate the sensory experience of the user&#39;s physical presence in an artificial environment, such as a reality-based environment or a non-reality-based environment, such as a video game. A virtual reality may include audio and haptic components, in addition to a visual component.

The visual component of a virtual reality may be displayed either on a computer screen or with a stereoscopic head-mounted display (HMD), such as the Rift, a virtual reality head-mounted display headset developed by Oculus VR of Seattle, Wash. Some conventional HMDs simply project an image or symbology on a wearer&#39;s visor or reticle. The projected image is not slaved to the real world (i.e., the image does not change based on the wearer&#39;s head position). Other HMDs incorporate a positioning system that tracks the wearer&#39;s head position and angle, so that the picture or symbology projected by the display is congruent with the outside world using see-through imagery. Head-mounted displays may also be used with tracking sensors that allow changes of angle and orientation of the wearer to be recorded. When such data is available to the system providing the virtual reality environment, it can be used to generate a display that corresponds to the wearer&#39;s the angle-of-look at the particular time. This allows the wearer to “look around” a virtual reality environment simply by moving the head without the need for a separate controller to change the angle of the imagery. Wireless-based systems allow the wearer to move about within the tracking limits of the system. Appropriately placed sensors may also allow the virtual reality system to track the HMD wearer&#39;s hand movements to allow natural interaction with content and a convenient game-play mechanism

However, a condition referred to as virtual reality sickness (also known as cybersickness) may occur when a user is exposed to a virtual reality environment. Virtual reality sickness can cause symptoms that are similar to the symptoms caused by motion sickness, such as general discomfort, headache, stomach awareness, nausea, vomiting, pallor, sweating, fatigue, drowsiness, disorientation, and apathy. Other symptoms may include postural instability and retching. Virtual reality sickness is distinguishable from motion sickness because it can be caused by the visually-induced perception of self-motion; real self-motion is not needed.

A significant aggravating factor in virtual reality sickness is whether the visual component of the virtual reality being experienced by the user represents is tied to the user&#39;s self-motion. In other words, if a user is “seeing” a virtual reality where the viewing-user&#39;s perspective is controlled by another user (or a computer), the viewing-user is more likely to develop symptoms of virtual reality sickness (this may be analogous to the anecdotally common real-world scenario of a passenger in a car developing symptoms of motion sickness while the driver of the car does not). For example, if the user controlling the perspective may unexpectedly swivel his or her head from side to side and the viewing-user is not expecting that sudden shift in perspective, virtual reality sickness may result.

Some conventional software applications, particularly video games where multiple players interact with a video game program within the same video game session, allow non-participating, “spectator,” users to passively interact with the game session, e.g. by selectively navigating around the game session&#39;s environment, while the game session is ongoing.

Some conventional video games (and game engines) may include a static replay creation feature that records game-state data corresponding to a game session and allows a user to view a replay video of the game session within the video game. Third party tools that capture the display data generated during a game session and stores it in a video format for later playback are also known. Video-based replays created by the third party tools support sharing via online playback and commonly available video players but require much more storage space than replays created with in the context of the video game itself.

These exiting tools illustrate a recognition that video game users may desire to record a game session in order to replay at a later time or to distribute for others replay, e.g. for entertainment purposes, similar to viewing a recorded sporting event. Game replays may also be used in the prevention and detection of cheating or to develop skill at a video game, e.g. by watching better players and observing their techniques. Video game replays may also be used to raise stature in a particular game&#39;s user community, e.g. to establish rankings among players or to settle disputes. Video game replays are also used to produce replay reviews, i.e. a review of a game replay by an experienced player to determine what can be done to improve and expand upon the original player&#39;s skill.

Because many conventional video game replay techniques are designed to recreate the player&#39;s perspective, they cannot be used in a virtual reality implementation because a user experiencing a replay of another user&#39;s virtual reality perspective may be at a heightened risk of developing virtual reality sickness. Such concerns over virtual reality sickness may be a major barrier to applying conventional video game replay techniques to virtual reality video games.

Additionally, many virtual reality-based video games create expansive and detailed virtual environments which may not be sufficiently explorable using conventional player-perspective video replay techniques, such as static video capture.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates exemplary network topology of a virtual reality session capture/replay system in accordance with at least one embodiment.

FIG. 2 illustrates several components of an exemplary client virtual reality capture device in accordance with at least one embodiment.

FIG. 3 illustrates several components of an exemplary client virtual reality replay device in accordance with at least one embodiment.

FIG. 4 illustrates several components of an exemplary virtual reality session capture/replay server in accordance with at least one embodiment.

FIG. 5 illustrates a virtual reality session capture data flow between various hardware and software components of an exemplary client virtual reality capture device in accordance with at least one embodiment.

FIG. 6 illustrates a virtual reality session replay data flow between various hardware and software components of an exemplary client virtual reality replay device in accordance with at least one embodiment.

FIG. 7 illustrates a first exemplary series of data communications between execution threads instantiated in memory of an exemplary client virtual reality capture device in accordance with at least one embodiment.

FIGS. 8A-8B illustrate a second exemplary series of data communications between execution threads instantiated in memory of an exemplary client virtual reality capture device in accordance with at least one embodiment.

FIG. 9 illustrates a third exemplary series of data communications between execution threads instantiated in memory of an exemplary client virtual reality capture device in accordance with at least one embodiment.

FIG. 10 illustrates a fourth exemplary series of data communications between an exemplary client virtual reality capture device, an exemplary virtual reality session capture/replay server, and an exemplary client virtual reality replay device in accordance with at least one embodiment.

FIG. 11 illustrates a fifth exemplary series of data communications between execution threads instantiated in memory of an exemplary client virtual reality replay device in accordance with at least one embodiment.

FIG. 12 illustrates a flow diagram of an exemplary virtual reality session capture routine in accordance with at least one embodiment.

FIG. 13 illustrates a flow diagram of an exemplary graphics engine function call capture sub-routine in accordance with at least one embodiment.

FIG. 14 illustrates a flow diagram of an exemplary positional data capture sub-routine in accordance with at least one embodiment.

FIG. 15 illustrates a flow diagram of an exemplary virtual reality replay session routine in accordance with at least one embodiment.

FIG. 16 illustrates a flow diagram of an exemplary virtual reality replay session frame update sub-routine in accordance with at least one embodiment.

DESCRIPTION

The detailed description that follows is represented largely in terms of processes and symbolic representations of operations by various computer components, including a processor, memory storage devices for the processor, connected display devices and input devices. Furthermore, these processes and operations may utilize conventional computer components in a heterogeneous distributed computing environment, including remote file servers, computer servers, and/or memory storage devices. Each of these conventional distributed computing components is accessible by the processor via a communication network, which may include, but is not limited to, the Internet.

The phrases “in one embodiment,” “in various embodiments,” “in some embodiments,” and the like are used repeatedly. Such phrases do not necessarily refer to the same embodiment. The terms “comprising,” “having,” and “including” are synonymous, unless the context dictates otherwise.

Reference is now made in detail to the description of the embodiments as illustrated in the drawings. While embodiments are described in connection with the drawings and related descriptions, there is no intent to limit the scope to the embodiments disclosed herein. On the contrary, the intent is to cover all alternatives, modifications, and equivalents. In alternate embodiments, additional devices, or combinations of illustrated devices, may be added to, or combined, without limiting the scope to the embodiments disclosed herein. For example, the embodiments set forth below are primarily described in the context of capturing and replaying visual/video data. However, these embodiments are exemplary and are in no way limited to visual/video data, and should be construed to include capturing and replaying audio data, haptic data, and the like.

In various embodiments described herein, the present methods and systems may be employed to capture and store a game-state data set corresponding to a VR game session and use the game-state data set to provide a virtual reality experience corresponding to a replay of the VR game session with a dynamically selectable spectator-perspective of the game session&#39;s virtual environment.

In an exemplary embodiment, a VR gaming replay service provider may provide systems and methods for: obtaining a game state data set as it is created during a virtual reality gaming session by a VR gaming device; creating a modified game-state data set based on the obtained game-state data set; and generating a data stream corresponding to the modified game-state data set for rendering a virtual reality replay session of the virtual reality gaming session by a client replay device (or by multiple client replay devices either independently and/or in a synchronized, interactive replay), where the modified game-state data set allows the virtual reality replay game session to mimic the virtual environment created during the VR game session, while permitting the client replay device to select (and dynamically re-select during the course of the replay game session) a “third person” spectator perspective. That is, a user/spectator of the replay game session may select his/her view point in the virtual environment created during the original game session, independent of the viewpoint of the original player.

In accordance with various aspects of the present embodiments, game event data for a game session may be obtained via at least two game-state data capture techniques operating in parallel on a remote server as the virtual reality game session is being created on a VR gaming device. In a first exemplary technique, the remote server may passively capture game function calls that are redirected from the VR gaming device to the remote server during the normal course of the virtual reality game session, e.g. via replay redirect instructions embedded into specific functions of the video game itself. In a second exemplary technique, the remote server may actively query the VR gaming device for game event data during the virtual reality game session, such as event data relating to the rotation, position and scale of three dimensional objects in the virtual environment (hereafter collectively referred to as positional event data and positional events, respectively).

In accordance with various aspects of some embodiments, the remote server may capture data corresponding to some game events data at a relatively high frame rate (e.g. the same frame rate as the original game session) and may capture data corresponding to other game events at a relatively low frame rate (e.g. 25%, 33%, 50%, 75%, etc. of the original game session frame rate). When generating a replay game session, in order for the replay game session to be capable of being rendered by a client replay device at the same frame rate as the original game session and with comparable visual quality, the remote server may apply one or more interpolation techniques to the game event data that was captured at the lower frame rate in order to match the corresponding gaps in the game event data that was captured at the high frame rate.

For example, the passive captured function call capture technique described above may capture data at the same frame rate as the game session is proceeding, while the active positional event querying technique described above may capture game event data at a relatively low frame rate, e.g. at 50% of the frame rate of the original game session. During generation of a replay game session at the same frame rate as the original game session, interpolation techniques may be used to provide estimated positional event data for those temporal interval, i.e. the display time of a single frame of a game session, where no positional event data was obtained directly.

An Exemplary Network Topology of a Virtual Reality Session Capture/Replay System

FIG. 1 illustrates an exemplary network topology of a virtual reality session capture/ replay system 100 in accordance with various embodiments. Client VR capture device 200 , client VR replay device 300 , and virtual reality session capture/ replay server 400 are in data communication with a network 102 . In various embodiments, network 102 may include the Internet, one or more local area networks (“LANs”), one or more wide area networks (“WANs”), cellular data networks, and/or other data networks. Network 114 may, at various points, be a wired and/or wireless network. Virtual reality session capture/ replay server 400 may be in data communication with a data store 104 .

In various embodiments, a client VR capture device 200 and a client VR replay device 300 may be networked client virtual reality devices. In the example shown in FIG. 1 , client VR capture device 200 and client VR replay device 300 are depicted as having a laptop/notebook computer form factor. Depending on the embodiment, the hardware embodying client VR capture device 200 and/or client VR replay device 300 may have a form factor comparable to a general purpose computer (including “desktop,” “laptop,” “notebook,” “tablet” computers, or the like); a mobile phone; “smart glasses,” or other wearable client virtual reality device; a video game console; a specialized hardware device specifically designed for the implementation of the present systems and methods; or the like. However, as will be apparent from the following description, not all devices having such form factors will be suitable for implementing the present systems and methods. In various embodiments there may be many more instantiations of client VR capture device 200 and client VR replay device 300 than are shown in FIG. 1 . The functional components of an exemplary, form-factor independent, client VR capture device 200 are described below in reference to FIG. 2 . The functional components of an exemplary, form-factor independent, client VR replay device 300 are described below in reference to FIG. 3 . An exemplary virtual reality input device(s) 106 and an exemplary VR output display device(s) 108 are connected to client VR capture device 200 for obtaining user input and providing visual output, respectively. An exemplary perspective selection input device 110 and an exemplary vr output display device 112 are connected to client VR replay device 300 . In many embodiments, there may be additional input devices, such as a positional tracker (not shown), and output devices, such as headphones (not shown), and haptic feedback devices (not shown) connected to client VR capture device 200 and/or client VR replay device 300 .

In various embodiments, virtual reality session capture/ replay server 400 may be a networked client virtual reality device generally capable of accepting requests over network 102 , e.g. from client VR capture device 200 , client VR replay device 300 , and/or other networked computing devices (not shown), and providing responses accordingly. The functional components of an exemplary virtual reality session capture/ replay server 400 are described below in reference to FIG. 4 .

An Exemplary Client Virtual Reality Capture Device

Referring to FIG. 2 , several components of an exemplary client VR capture device 200 are illustrated. In some embodiments, a client gaming device may include many more components than those shown in FIG. 2 . However, it is not necessary that all of these generally conventional components be shown in order to disclose an illustrative embodiment. As shown in FIG. 2 , exemplary client VR capture device 200 includes a central processing unit 202 in data communication with memory 204 via an item 206 .

Central processing unit 202 is an electronic circuit designed to carry out instructions of a computer program, e.g. obtained from memory 204 , by performing the basic arithmetic, logical, control and input/output (I/O) operations specified by the program&#39;s instructions. Central processing unit 202 may have multiple processor cores (not shown). A multi-core processor is a single computing component with two or more independent data processing units (called “cores”), which are the units that read and execute program instructions. Multiple cores can process multiple instructions at the same time, increasing overall speed for applications amenable to parallel computing, such as virtual reality applications. An exemplary central processing unit 202 may be an Intel® Core™ i5-4590 Processor, manufactured by Intel Corporation of Santa Clara, Calif.

Memory 204 generally comprises some or all of random access memory (RAM), read-only memory (ROM), and/or a permanent mass storage device, such as a disk drive, flash memory, or the like.

Client VR capture device 200 may also include a network interface 210 for connecting to a network such as network 102 , a virtual reality input device(s) (optional) 212 (or a user input port for connecting an external user input device, such as virtual reality input device(s) 106 ), a speaker (optional) 216 (or an output port (not shown) for connecting to an external audio device (not shown)), and the like.

In certain embodiments, client VR capture device 200 may include graphics processing unit 208 . Graphics processing unit 208 is an electronic circuit designed to rapidly manipulate and alter electronic data to accelerate the creation of images in a frame buffer intended for output to a display. Graphics processing unit 208 may be in data communication with a VR output display device (optional) 214 (or an output port (not shown) for connecting to an external virtual reality display device such as VR output display device(s) 108 ). In some embodiments, speaker (optional) 216 and VR output display device (optional) 214 may be integrated into a single hardware device.

Memory 204 of exemplary client VR capture device 200 may store program code, executable by central processing unit 202 , corresponding to an operating system 218 , as well as program code corresponding to a virtual reality application 220 and other software applications (not shown).

Virtual reality application 220 , and other software applications (not shown), as well as various data files (not shown) may be loaded into memory 204 via network interface 210 or via a computer readable storage medium 222 , such as a hard-disk drive, a solid-state drive, an optical disc, a removable memory card, and/or the like.

In operation, operating system 218 manages the hardware and software resources of the client VR capture device 200 and provides common services and memory allocation for various software applications, such as virtual reality application 220 . To enable software applications, such as virtual reality application 220 to interact with various hardware components of client VR capture device 200 , such as network communications via network interface 210 , obtaining input data via virtual reality input device(s) (optional) 212 , rendering output data via VR output display device (optional) 214 and/or speaker (optional) 216 , operating system 218 may provide a variety application program interfaces (APIs), such graphics API 224 , such as the OpenGL (managed by Khronos Group of Beaverton, Oreg.), Direct3D (developed by Microsoft Corporation of Redmond, Wash.), proprietary APIs, and the like; an audio API 226 ; a network API 228 ; an input API 230 , a storage API 232 ; and the like.

As is described in more detail below, virtual reality application 220 may include a VR program 234 , such as a VR video game, a graphics engine 236 , such as the Unity Engine (developed by Unity Technologies of San Francisco, Calif.), a VR session capture component 238 , and (optionally) a vr session replay component 240 . Instructions of VR program 234 may be executed by central processing unit 202 and may call various functions of graphics engine 236 in order to render visual data generated by the VR program. The functions of graphics engine 236 may be executed by central processing unit 202 and/or graphics processing unit 208 .

Although an exemplary client VR capture device 200 has been described, a client gaming device may be any of a great number of computing devices capable of executing program code, such as the program code corresponding to operating system 218 and virtual reality application 220 and rendering the visual data generated by virtual reality application 220 .

An Exemplary Client Virtual Reality Replay Device

Referring to FIG. 3 , several components of an exemplary client VR replay device 300 are illustrated. Client VR replay device 300 may include many of the same hardware and software components as client VR capture device 200 . In some embodiments, a client VR replay device may include many more components than those shown in FIG. 3 . However, it is not necessary that all of these generally conventional components be shown in order to disclose an illustrative embodiment.

As shown in FIG. 3 , exemplary client VR replay device 300 includes a central processing unit 302 in data communication with memory 304 via a item 306 . Central processing unit 302 is an electronic circuit designed to carry out instructions of a computer program, e.g. obtained from memory 304 , by performing the basic arithmetic, logical, control and input/output (I/O) operations specified by the program&#39;s instructions. Memory 304 generally comprises some or all of random access memory (RAM), read-only memory (ROM), and/or a permanent mass storage device, such as a disk drive, solid-state device, flash memory, and/or the like.

Client VR replay device 300 may also include a network interface 310 for connecting to a network such as network 102 , a virtual reality input device(s) (optional) 312 , or a user input port for connecting an external user input device, such as perspective selection input device 110 , a speaker (optional) 316 , or an output port (not shown) for connecting to an external audio device (not shown)), and the like.

In certain embodiments, client VR replay device 300 may include graphics processing unit 308 . Graphics processing unit 308 is an electronic circuit designed to rapidly manipulate and alter electronic data to accelerate the creation of images in a frame buffer intended for output to a display. Graphics processing unit 308 may be in data communication with a VR output display device (optional) 314 , or an output port (not shown) for connecting to an external virtual reality display device such as vr output display device 112 . In some embodiments, speaker (optional) 316 and VR output display device (optional) 314 may be integrated into a single hardware device.

Memory 304 of exemplary client VR replay device 300 may store program code, executable by central processing unit 302 , corresponding to an operating system 318 , as well as program code corresponding to a virtual reality application 320 and other software applications (not shown).

virtual reality application 320 and other software applications (not shown), as well as various data files (not shown) may be loaded into memory 304 via network interface 310 or via a computer readable storage medium 322 , such as a hard-disk drive, a solid-state drive, an optical disc, a removable memory card, and/or the like.

In operation, operating system 318 manages the hardware and software resources of the client VR replay device 300 and provides common services and memory allocation for various software applications, such as virtual reality application 320 . To enable software applications, such as virtual reality application 320 to interact with various hardware components of client VR replay device 300 , such as network communications via network interface 310 , obtaining input data via virtual reality input device(s) (optional) 312 , rendering output data via VR output display device (optional) 314 and/or speaker (optional) 316 , operating system 318 may provide a variety application program interfaces (APIs), such a graphics API 324 , such as the OpenGL (managed by Khronos Group of Beaverton, Oreg.), Direct3D (developed by Microsoft Corporation of Redmond, Wash.), proprietary graphics APIs, and the like; an audio API 326 ; a network API 328 ; an input API 330 , a storage API 332 , and the like.

As is described in more detail below, virtual reality application 320 a graphics engine 336 , such as the Unity Engine (developed by Unity Technologies of San Francisco, Calif.) and a VR session replay component 340 . Instructions of virtual reality application 320 may be executed by central processing unit 302 and graphics processing unit 308 in order to render visual data generated by a VR session replay data stream (not shown).

Although an exemplary client VR replay device 300 has been described, a client VR replay device may be any of a great number of computing devices capable of executing program code, such as the program code corresponding to operating system 318 and rendering the visual data generated by virtual reality application 320 .

An Exemplary Virtual Reality Session Capture/Replay Server

Referring to FIG. 4 , several components of an exemplary virtual reality session capture/ replay server 400 are illustrated. In some embodiments, a virtual reality session capture/replay server may include many more components than those shown in FIG. 4 . However, it is not necessary that all of these generally conventional components be shown in order to disclose an illustrative embodiment. As shown in FIG. 4 , exemplary virtual reality session capture/ replay server 400 includes a processing unit 402 in data communication with memory 404 via a bus 406 . Memory 404 generally comprises some or all of random access memory (RAM), read-only memory (ROM), and/or a permanent mass storage device, such as a disk drive, flash memory, or the like.

Virtual reality session capture/ replay server 400 may also include a network interface 408 for connecting to a network such as network 102 and a user input (optional) 410 (or a user input port for connecting an external user input device (not shown)) and/or a display (optional) 412 (or a display port for connecting to an external display device (not shown)).

Memory 404 of virtual reality session capture/ replay server 400 may store program code, executable by processing unit 402 , corresponding to an operating system 416 , as well as program code corresponding to a front end service 418 , a virtual reality session capture service 420 , and a virtual reality session replay service 422 .

These and other software components, as well as various data files (not shown) may be loaded into memory 404 via network interface 408 or via a selectively removable computer readable storage medium 424 , such as an optical disc, memory card, or the like.

In operation, operating system 416 manages the hardware and software resources of virtual reality session capture/ replay server 400 . For hardware functions such as network communications via network interface 408 , obtaining data via user input (optional) 410 , and/or rendering data via display (optional) 412 , and allocation of memory 404 to various resources, operating system 416 may act as an intermediary between software executing on virtual reality session capture/ replay server 400 and the server&#39;s hardware.

Although an exemplary virtual reality session capture/ replay server 400 has been described, a virtual reality session capture/replay server may be any of a great number of computing devices capable executing program code, such as the program code corresp

CLAIMS

Claims ( 20 )

The invention claimed is:

1. A client virtual reality device useful in generating virtual reality replay streams, virtual reality replay streams corresponding to virtual reality sessions generated by the client virtual reality device, the client virtual reality device comprising:

at least one computing processing unit;

a network interface;

a computer readable storage medium; and

memory in data communication with said at least one computer processing unit and containing executable instructions for causing said at least one computing processing unit to perform a method comprising:

(a) instantiating, on the client virtual reality device, a virtual reality session capture thread associated with a virtual reality session being generated by the client virtual reality device;

(b) providing, by said virtual reality session capture thread, a plurality of positional update requests to a graphics engine rendering thread instantiated on the client virtual reality device and associated with said virtual reality session;

(c) obtaining, by said virtual reality session capture thread, positional update data corresponding to a plurality of virtual reality objects associated with said virtual reality session, said positional update data corresponding to a current position and orientation of said plurality of virtual reality objects within said virtual reality session;

(d) obtaining, by said virtual reality session capture thread, a virtual reality session termination communication, said virtual reality session termination communication indicating an end of said virtual reality session;

(e) creating a virtual reality replay stream corresponding to said plurality of virtual reality game session data records and being associated with a virtual reality replay session identifier corresponding to the virtual reality session; and

(f) saving said virtual reality replay stream to said computer readable storage medium; and

wherein, prior to step (d), steps (b) and (c) are repeated in accordance with a positional update sample rate associated with said virtual reality session capture thread, step (a) includes determining a frame rate associated with said virtual reality session and said positional update sample rate associated with said virtual reality session capture thread and step (b) includes monitoring a current frame count of said virtual reality session and determining when to provide a positional update request of said plurality of positional update requests based at least in part on said positional update sample rate and said current frame count.

2. The client virtual reality device of claim 1 , wherein said method further comprises providing said virtual reality replay stream to a remote virtual reality session capture and replay server via said network interface.

3. The client virtual reality device of claim 1 , wherein said at least one computing processing unit comprises a central processing unit and a graphics processing unit.

4. The client virtual reality device of claim 1 , further comprising at least one virtual reality input device and at least one virtual reality display device.

5. The client virtual reality device of claim 1 , further comprising capturing, by said virtual reality session capture thread, graphics engine function call data for each of a plurality of graphics engine function calls made during said virtual reality session.

6. The client virtual reality device of claim 5 , wherein said virtual reality replay stream includes graphics engine function call data corresponding to at least 90% of graphics engine function calls made during said virtual reality session.

7. The client virtual reality device of claim 5 , wherein capturing, by said virtual reality session capture thread, graphics engine function call data for each of a plurality of graphics engine function calls made during said virtual reality session includes obtaining, by said virtual reality session capture thread, a graphics engine function call capture request, said graphics engine function call capture request including graphics engine function call data corresponding to at least one graphics engine function call of said virtual reality session.

8. The client virtual reality device of claim 7 , wherein capturing, by said virtual reality session capture thread, graphics engine function call data for each of a plurality of graphics engine function calls made during said virtual reality session includes providing said at least one graphics engine function call to a graphics engine rendering thread instantiated on the client virtual reality device.

9. The client virtual reality device of claim 5 , wherein capturing, by said virtual reality session capture thread, graphics engine function call data for each of a plurality of graphics engine function calls made during said virtual reality session includes obtaining a sequence of graphics engine function call capture requests, each of said sequence of graphics engine function call capture requests including graphics engine function call data corresponding to a graphics engine function call provided to a graphics engine rendering thread instantiated on the client virtual reality device, and, for each of said sequence of graphics engine function call capture requests, pushing said graphics engine function call data onto a virtual reality session bit stream stack.

10. A client virtual reality device useful in generating virtual reality replay streams, virtual reality replay streams corresponding to virtual reality sessions generated by the client virtual reality device, the client virtual reality device comprising:

at least one computing processing unit;

a network interface;

a computer readable storage medium; and

memory in data communication with said at least one computer processing unit and containing executable instructions for causing said at least one computing processing unit to perform a method comprising:

(a) instantiating, on the client virtual reality device, a virtual reality session capture thread associated with a virtual reality session being generated by the client virtual reality device;

(b) obtaining, by said virtual reality session capture thread, a graphics engine function call capture request, said graphics engine function call capture request including graphics engine function call data corresponding to at least one graphics engine function call of said virtual reality session;

(c) providing, by said virtual reality session capture thread, a plurality of positional update requests to a graphics engine rendering thread instantiated on the client virtual reality device and associated with said virtual reality session;

(d) obtaining, by said virtual reality session capture thread, positional update data corresponding to a plurality of virtual reality objects associated with said virtual reality session, said positional update data corresponding to a current position and orientation of said plurality of virtual reality objects within said virtual reality session;

(e) obtaining, by said virtual reality session capture thread, a virtual reality session termination communication, said virtual reality session termination communication indicating an end of said virtual reality session;

(f) creating a virtual reality replay stream corresponding to said plurality of virtual reality game session data records and being associated with a virtual reality replay session identifier corresponding to the virtual reality session; and

(g) saving said virtual reality replay stream to said computer readable storage medium; and

wherein, prior to step (e), (c), and (d) are repeated in accordance with a positional update sample rate associated with said virtual reality session capture thread, step (a) includes determining a frame rate associated with said virtual reality session and said positional update sample rate associated with said virtual reality session capture thread and step (c) includes monitoring a current frame count of said virtual reality session and determining when to provide a positional update request of said plurality of positional update requests based at least in part on said positional update sample rate and said current frame count.

11. The client virtual reality device of claim 10 , wherein step (b) comprises obtaining a sequence of graphics engine function call capture requests, each of said sequence of graphics engine function call capture requests including graphics engine function call data corresponding to a graphics engine function call provided to said graphics engine rendering thread, and, for each of said sequence of graphics engine function call capture requests, pushing said graphics engine function call data onto a virtual reality session bit stream stack.

12. The client virtual reality device of claim 10 , wherein step (b) further comprises providing said at least one graphics engine function call to said graphics engine rendering thread.

13. The client virtual reality device of claim 10 , wherein said method further comprises providing said virtual reality replay stream to a remote virtual reality session capture and replay server via said network interface.

14. The client virtual reality device of claim 10 , wherein said at least one computing processing unit comprises a central processing unit and a graphics processing unit.

15. The client virtual reality device of claim 10 , further comprising at least one virtual reality input device and at least one virtual reality display device.

16. The client virtual reality device of claim 10 , further comprising capturing, by said virtual reality session capture thread, graphics engine function call data for each of a plurality of graphics engine function calls made during said virtual reality session.

17. The client virtual reality device of claim 16 , wherein said virtual reality replay stream includes graphics engine function call data corresponding to at least 90% of graphics engine function calls made during said virtual reality session.

18. The client virtual reality device of claim 16 , wherein capturing, by said virtual reality session capture thread, graphics engine function call data for each of a plurality of graphics engine function calls made during said virtual reality session includes obtaining, by said virtual reality session capture thread, a graphics engine function call capture request, said graphics engine function call capture request including graphics engine function call data corresponding to at least one graphics engine function call of said virtual reality session.

19. The client virtual reality device of claim 18 , wherein capturing, by said virtual reality session capture thread, graphics engine function call data for each of a plurality of graphics engine function calls made during said virtual reality session includes providing said at least one graphics engine function call to a graphics engine rendering thread instantiated on the client virtual reality device.

20. The client virtual reality device of claim 16 , wherein capturing, by said virtual reality session capture thread, graphics engine function call data for each of a plurality of graphics engine function calls made during said virtual reality session includes obtaining a sequence of graphics engine function call capture requests, each of said sequence of graphics engine function call capture requests including graphics engine function call data corresponding to a graphics engine function call provided to a graphics engine rendering thread instantiated on the client virtual reality device, and, for each of said sequence of graphics engine function call capture requests, pushing said graphics engine function call data onto a virtual reality session bit stream stack.

US15/239,740

2016-02-29

2016-08-17

Virtual reality session capture and replay systems and methods

Expired - Fee Related

US9616338B1

( en )

Priority Applications (1)

Application Number

Priority Date

Filing Date

Title

US15/239,740

US9616338B1

( en )

2016-02-29

2016-08-17

Virtual reality session capture and replay systems and methods

Applications Claiming Priority (2)

Application Number

Priority Date

Filing Date

Title

US15/057,003

US20170249785A1

( en )

2016-02-29

2016-02-29

Virtual reality session capture and replay systems and methods

US15/239,740

US9616338B1

( en )

2016-02-29

2016-08-17

Virtual reality session capture and replay systems and methods

Related Parent Applications (1)

Application Number

Title

Priority Date

Filing Date

US15/057,003

Continuation

US20170249785A1

( en )

2016-02-29

2016-02-29

Virtual reality session capture and replay systems and methods

Publications (1)

Publication Number

Publication Date

US9616338B1

true

US9616338B1 ( en )

2017-04-11

Family

ID=58461007

Family Applications (2)

Application Number

Title

Priority Date

Filing Date

US15/057,003

Abandoned

US20170249785A1

( en )

2016-02-29

2016-02-29

Virtual reality session capture and replay systems and methods

US15/239,740

Expired - Fee Related

US9616338B1

( en )

2016-02-29

2016-08-17

Virtual reality session capture and replay systems and methods

Family Applications Before (1)

Application Number

Title

Priority Date

Filing Date

US15/057,003

Abandoned

US20170249785A1

( en )

2016-02-29

2016-02-29

Virtual reality session capture and replay systems and methods

Country Status (1)

Country

Link

US

( 2 )

US20170249785A1

( en )

Cited By (34)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

CN107612947A

( en )

*

2017-11-03

2018-01-19

北京微视酷科技有限责任公司

Synchronous communication means and system between a kind of virtual reality device

WO2018208481A1

( en )

*

2017-05-12

2018-11-15

Microsoft Technology Licensing, Llc

Multiple listener cloud render with enhanced instant replay

CN108874267A

( en )

*

2017-05-09

2018-11-23

腾讯科技（深圳）有限公司

Virtual reality applications data processing method, computer equipment and storage medium

WO2018231279A1

( en )

*

2017-06-12

2018-12-20

Google Llc

Intelligent command batching in an augmented and/or virtual reality environment

CN109246286A

( en )

*

2018-07-13

2019-01-18

深圳超多维科技有限公司

Control method, system, equipment and the storage medium of intelligent terminal application operating

US20190056799A1

( en )

*

2017-08-15

2019-02-21

TinMoon Studios, LLC

Method and system for resolving spatial boundary conflicts of virtual objects using simulated spring constraints in 3d virtual space

WO2019067480A1

( en )

*

2017-09-29

2019-04-04

Sony Interactive Entertainment America Llc

Venue mapping for virtual reality spectating of electronic sports

US20190155465A1

( en )

*

2016-04-29

2019-05-23

String Limited

Augmented media

CN109952757A

( en )

*

2017-08-24

2019-06-28

腾讯科技（深圳）有限公司

Method, terminal device and storage medium based on virtual reality applications recorded video

US10346295B2

( en )

*

2017-04-14

2019-07-09

Microsoft Technology Licensing, Llc

Traffic replay to detect interface changes

CN110678827A

( en )

*

2017-06-08

2020-01-10

霍尼韦尔国际公司

Apparatus and method for recording and playing back interactive content in augmented/virtual reality in industrial automation systems and other systems

US10572017B2

( en )

*

2018-04-20

2020-02-25

Immersion Corporation

Systems and methods for providing dynamic haptic playback for an augmented or virtual reality environments

CN112130660A

( en )

*

2020-08-14

2020-12-25

青岛小鸟看看科技有限公司

Interaction method and system based on virtual reality all-in-one machine

US11163588B2

( en )

2018-03-01

2021-11-02

Vreal Inc.

Source code independent virtual reality capture and replay systems and methods

US20210394046A1

( en )

*

2020-06-17

2021-12-23

Delta Electronics, Inc.

Method for producing and replaying courses based on virtual reality and system thereof

US11395972B2

( en )

*

2018-05-01

2022-07-26

Cygames, Inc.

System, server, program, and method for detecting cheating by user in game

US20220345789A1

( en )

*

2021-04-27

2022-10-27

Digital Seat Media, Inc.

Systems and methods for delivering augmented reality content

US11870852B1

( en )

*

2023-03-31

2024-01-09

Meta Platforms Technologies, Llc

Systems and methods for local data transmission

US20240013487A1

( en )

*

2018-01-22

2024-01-11

Apple Inc.

Method and device for generating a synthesized reality reconstruction of flat video content

US11972396B2

( en )

2017-06-07

2024-04-30

Digital Seat Media, Inc.

Method and system for digital record verification

US12035020B2

( en )

2021-05-12

2024-07-09

Qualcomm Incorporated

Split rendering of extended reality data over 5G networks

US12061997B2

( en )

2020-04-27

2024-08-13

Digital Seat Media, Inc.

System and method for exchanging tickets via a machine-readable code

US12075018B2

( en )

2021-04-21

2024-08-27

Microsoft Technology Licensing, Llc

Distributed virtual reality

US12238059B2

( en )

2021-12-01

2025-02-25

Meta Platforms Technologies, Llc

Generating a summary of a conversation between users for an additional user in response to determining the additional user is joining the conversation

US12333662B2

( en )

2022-08-11

2025-06-17

Apple Inc.

Presentations in multi-user communication sessions

US12340350B2

( en )

2020-04-27

2025-06-24

Digital Seat Media, Inc.

Interactive and dynamic digital event program

US12373906B2

( en )

2020-04-27

2025-07-29

Digital Seat Media, Inc.

System and platform for engaging educational institutions and stakeholders

US12450589B2

( en )

2019-03-06

2025-10-21

Digital Seat Media, Inc.

System and method for location-based individualized content and mobile wallet offers

US12482177B2

( en )

2023-05-16

2025-11-25

International Business Machines Corporation

Implementing multiple thread scenes in virtualized metaverse

US12489958B2

( en )

2021-05-12

2025-12-02

Qualcomm Incorporated

Split rendering of extended reality data over 5G networks

US12566498B2

( en )

2023-06-04

2026-03-03

Apple Inc.

Devices, methods, and graphical user interfaces for displaying presentation environments for a presentation application

US12586326B2

( en )

2022-09-29

2026-03-24

Meta Platforms Technologies, Llc

Mapping a real-world room for a shared artificial reality environment

US12608884B2

( en )

2024-03-12

2026-04-21

International Business Machines Corporation

Selective volumetric boundary recording

US12614355B2

( en )

2022-09-29

2026-04-28

Meta Platforms Technologies, Llc

Mapping a real-world room for a shared artificial reality environment

Families Citing this family (12)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

TWI606418B

( en )

*

2012-09-28

2017-11-21

輝達公司

Computer system and method for gpu driver-generated interpolated frames

US11163358B2

( en )

*

2016-03-17

2021-11-02

Sony Interactive Entertainment Inc.

Spectating virtual (VR) environments associated with VR user interactivity

US11181990B2

( en )

*

2016-03-18

2021-11-23

Sony Interactive Entertainment Inc.

Spectator view tracking of virtual reality (VR) user in VR environments

US10645178B2

( en )

*

2016-11-29

2020-05-05

Ncr Corporation

Omni-channel virtual reality (VR) collaboration

US11269580B2

( en )

*

2017-02-13

2022-03-08

Comcast Cable Communications, Llc

Guided collaborative viewing of navigable image content

US11836285B2

( en )

*

2017-03-10

2023-12-05

Kenneth Perlin

System for a spectator to view a virtual reality of a performer and method

JP6277503B1

( en )

*

2017-03-15

2018-02-14

株式会社コナミデジタルエンタテインメント

Server apparatus and computer program used therefor

US20180357922A1

( en )

2017-06-08

2018-12-13

Honeywell International Inc.

Apparatus and method for assessing and tracking user competency in augmented/virtual reality-based training in industrial automation systems and other systems

CN109819240A

( en )

*

2017-11-21

2019-05-28

天津三星电子有限公司

A kind of method, display and the display system of display virtual real VR signal

CN111400184B

( en )

*

2020-03-17

2021-10-29

腾讯科技（深圳）有限公司

Game testing method, device, system, device and cloud platform

JP7339198B2

( en )

*

2020-03-26

2023-09-05

株式会社ソニー・インタラクティブエンタテインメント

Information processing device and game image distribution method

CN112230776B

( en )

*

2020-10-29

2024-07-02

北京京东方光电科技有限公司

Virtual reality display method, device and storage medium

Citations (6)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20070060343A1

( en )

*

2005-09-09

2007-03-15

Microsoft Corporation

Role play system

US20090199089A1

( en )

*

2004-01-28

2009-08-06

Sankhya Technologies Private Limited

Converting a Heterogeneous Document

US20120093214A1

( en )

*

2010-10-19

2012-04-19

Julian Michael Urbach

Composite video streaming using stateless compression

US20130335509A1

( en )

*

2012-06-18

2013-12-19

Mobile Video Date, Inc.

Methods, systems, and articles of manufacture for online video dating

US20140168229A1

( en )

*

2012-12-14

2014-06-19

Microsoft

Cpu-gpu parallelization

US20140194211A1

( en )

*

2013-01-09

2014-07-10

Blizzard Entertainment, Inc.

Restoring gameplay by replaying past inputs

Family Cites Families (1)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20020154214A1

( en )

*

2000-11-02

2002-10-24

Laurent Scallie

Virtual reality game system using pseudo 3D display driver

2016

2016-02-29

US

US15/057,003

patent/US20170249785A1/en

not_active

Abandoned

2016-08-17

US

US15/239,740

patent/US9616338B1/en

not_active

Expired - Fee Related

Patent Citations (6)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20090199089A1

( en )

*

2004-01-28

2009-08-06

Sankhya Technologies Private Limited

Converting a Heterogeneous Document

US20070060343A1

( en )

*

2005-09-09

2007-03-15

Microsoft Corporation

Role play system

US20120093214A1

( en )

*

2010-10-19

2012-04-19

Julian Michael Urbach

Composite video streaming using stateless compression

US20130335509A1

( en )

*

2012-06-18

2013-12-19

Mobile Video Date, Inc.

Methods, systems, and articles of manufacture for online video dating

US20140168229A1

( en )

*

2012-12-14

2014-06-19

Microsoft

Cpu-gpu parallelization

US20140194211A1

( en )

*

2013-01-09

2014-07-10

Blizzard Entertainment, Inc.

Restoring gameplay by replaying past inputs

Cited By (57)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20190155465A1

( en )

*

2016-04-29

2019-05-23

String Limited

Augmented media

US10346295B2

( en )

*

2017-04-14

2019-07-09

Microsoft Technology Licensing, Llc

Traffic replay to detect interface changes

CN108874267A

( en )

*

2017-05-09

2018-11-23

腾讯科技（深圳）有限公司

Virtual reality applications data processing method, computer equipment and storage medium

US11307891B2

( en )

2017-05-09

2022-04-19

Tencent Technology (Shenzhen) Company Ltd

Virtual reality application data processing method, computer device and storage medium

CN108874267B

( en )

*

2017-05-09

2022-12-09

腾讯科技（深圳）有限公司

Virtual reality application data processing method, computer equipment and storage medium

EP3611610A4

( en )

*

2017-05-09

2021-01-27

Tencent Technology (Shenzhen) Company Limited

VIRTUAL REALITY APPLICATION DATA PROCESSING PROCESS, COMPUTER DEVICE AND STORAGE MEDIA

WO2018208481A1

( en )

*

2017-05-12

2018-11-15

Microsoft Technology Licensing, Llc

Multiple listener cloud render with enhanced instant replay

US10278001B2

( en )

2017-05-12

2019-04-30

Microsoft Technology Licensing, Llc

Multiple listener cloud render with enhanced instant replay

US12373793B2

( en )

2017-06-07

2025-07-29

Digital Seat Media, Inc.

Method and system for digital record verification

US11972396B2

( en )

2017-06-07

2024-04-30

Digital Seat Media, Inc.

Method and system for digital record verification

CN110678827A

( en )

*

2017-06-08

2020-01-10

霍尼韦尔国际公司

Apparatus and method for recording and playing back interactive content in augmented/virtual reality in industrial automation systems and other systems

CN110678827B

( en )

*

2017-06-08

2023-11-10

霍尼韦尔国际公司

Apparatus and method for recording and replaying interactive content in augmented/virtual reality in industrial automation systems and other systems

US10976890B2

( en )

2017-06-12

2021-04-13

Google Llc

Intelligent command batching in an augmented and/or virtual reality environment

CN110520826A

( en )

*

2017-06-12

2019-11-29

谷歌有限责任公司

Intelligence order in enhancing and/or reality environment, which is closed, to be criticized

CN110520826B

( en )

*

2017-06-12

2023-03-31

谷歌有限责任公司

Smart command pooling in augmented and/or virtual reality environments

WO2018231279A1

( en )

*

2017-06-12

2018-12-20

Google Llc

Intelligent command batching in an augmented and/or virtual reality environment

US10698561B2

( en )

2017-06-12

2020-06-30

Google Llc

Intelligent command batching in an augmented and/or virtual reality environment

US20190056799A1

( en )

*

2017-08-15

2019-02-21

TinMoon Studios, LLC

Method and system for resolving spatial boundary conflicts of virtual objects using simulated spring constraints in 3d virtual space

CN109952757B

( en )

*

2017-08-24

2020-06-05

腾讯科技（深圳）有限公司

Method for recording video based on virtual reality application, terminal equipment and storage medium

CN109952757A

( en )

*

2017-08-24

2019-06-28

腾讯科技（深圳）有限公司

Method, terminal device and storage medium based on virtual reality applications recorded video

US11000766B2

( en )

2017-08-24

2021-05-11

Tencent Technology (Shenzhen) Company Limited

Video recording method based on virtual reality application, terminal device, and storage medium

US11436803B2

( en )

2017-09-29

2022-09-06

Sony Interactive Entertainment LLC

Insertion of VR spectator in live video of a live event

CN112135673B

( en )

*

2017-09-29

2025-03-11

索尼互动娱乐有限责任公司

Field Mapping for Virtual Reality Viewing of Esports

CN112135673A

( en )

*

2017-09-29

2020-12-25

索尼互动娱乐有限责任公司

Site mapping for virtual reality viewing for electronic athletics

US10621784B2

( en )

2017-09-29

2020-04-14

Sony Interactive Entertainment America Llc

Venue mapping for virtual reality spectating of live events

WO2019067480A1

( en )

*

2017-09-29

2019-04-04

Sony Interactive Entertainment America Llc

Venue mapping for virtual reality spectating of electronic sports

CN107612947A

( en )

*

2017-11-03

2018-01-19

北京微视酷科技有限责任公司

Synchronous communication means and system between a kind of virtual reality device

US20240013487A1

( en )

*

2018-01-22

2024-01-11

Apple Inc.

Method and device for generating a synthesized reality reconstruction of flat video content

US12620177B2

( en )

*

2018-01-22

2026-05-05

Apple Inc.

Method and device for generating a synthesized reality reconstruction of flat video content

US11169824B2

( en )

2018-03-01

2021-11-09

Vreal Inc

Virtual reality replay shadow clients systems and methods

US11163588B2

( en )

2018-03-01

2021-11-02

Vreal Inc.

Source code independent virtual reality capture and replay systems and methods

US10572017B2

( en )

*

2018-04-20

2020-02-25

Immersion Corporation

Systems and methods for providing dynamic haptic playback for an augmented or virtual reality environments

US11395972B2

( en )

*

2018-05-01

2022-07-26

Cygames, Inc.

System, server, program, and method for detecting cheating by user in game

CN109246286A

( en )

*

2018-07-13

2019-01-18

深圳超多维科技有限公司

Control method, system, equipment and the storage medium of intelligent terminal application operating

US12450589B2

( en )

2019-03-06

2025-10-21

Digital Seat Media, Inc.

System and method for location-based individualized content and mobile wallet offers

US12373906B2

( en )

2020-04-27

2025-07-29

Digital Seat Media, Inc.

System and platform for engaging educational institutions and stakeholders

US12061997B2

( en )

2020-04-27

2024-08-13

Digital Seat Media, Inc.

System and method for exchanging tickets via a machine-readable code

US12340350B2

( en )

2020-04-27

2025-06-24

Digital Seat Media, Inc.

Interactive and dynamic digital event program

US11887365B2

( en )

*

2020-06-17

2024-01-30

Delta Electronics, Inc.

Method for producing and replaying courses based on virtual reality and system thereof

US20210394046A1

( en )

*

2020-06-17

2021-12-23

Delta Electronics, Inc.

Method for producing and replaying courses based on virtual reality and system thereof

CN112130660B

( en )

*

2020-08-14

2024-03-15

青岛小鸟看看科技有限公司

An interactive method and system based on virtual reality all-in-one machine

CN112130660A

( en )

*

2020-08-14

2020-12-25

青岛小鸟看看科技有限公司

Interaction method and system based on virtual reality all-in-one machine

US12075018B2

( en )

2021-04-21

2024-08-27

Microsoft Technology Licensing, Llc

Distributed virtual reality

US12008672B2

( en )

*

2021-04-27

2024-06-11

Digital Seat Media, Inc.

Systems and methods for delivering augmented reality content

US20220345789A1

( en )

*

2021-04-27

2022-10-27

Digital Seat Media, Inc.

Systems and methods for delivering augmented reality content

US12062108B2

( en )

2021-04-27

2024-08-13

Digital Seat Media, Inc.

Wagering platforms and access derived from machine-readable codes

US12035020B2

( en )

2021-05-12

2024-07-09

Qualcomm Incorporated

Split rendering of extended reality data over 5G networks

US12489958B2

( en )

2021-05-12

2025-12-02

Qualcomm Incorporated

Split rendering of extended reality data over 5G networks

US12238059B2

( en )

2021-12-01

2025-02-25

Meta Platforms Technologies, Llc

Generating a summary of a conversation between users for an additional user in response to determining the additional user is joining the conversation

US12333662B2

( en )

2022-08-11

2025-06-17

Apple Inc.

Presentations in multi-user communication sessions

US12462494B2

( en )

2022-08-11

2025-11-04

Apple Inc.

Interactive element in a replay

US12586326B2

( en )

2022-09-29

<td itemprop="publicationDat

Related documents

• Generating augmented reality experiences utilizing physical objects to … — Meta Platforms Technologies, Llc (US20260037214A1)#653960
• Virtual reality system using multiple force arrays for a solver — DeepMotion, Inc. (US10657696B2)#653961
• Tracking position of device inside-out for virtual reality interactivity — Sony Interactive Entertainment America Llc (US10535153B2)#653962
• Augmented reality system with spatialized audio tied to user manipulated … — Magic Leap, Inc. (US11231770B2)#653963
• Big data-based autonomous flight drone system and autonomous flight method … — Rgblab Co., Ltd. (US20210390867A1)#653964