ABSTRACT
Abstract
A method may include rendering a first view of a three-dimensional (3-D) virtual scene comprising a view of first content being displayed on a virtual display device from a location in the 3-D virtual scene. The method may also include rendering a second view comprising one or more content objects. The second view may be rendered from the location in the 3-D virtual scene, and the second scene may include a view of the display device as would be seen through a pair of augmented-reality glasses that display the one or more content objects. The method may additionally include generating a composite view by combing the first view and the second view. The method may further include causing the composite view to be displayed on a virtual-reality headset.
Description
CROSS REFERENCES
This application claims the benefit of the following U.S. Provisional Application:
U.S. Provisional Application No. 62/239,134, filed on Oct. 8, 2015, entitled âAUGMENTED REALITY SIMULATION,â by Koperwas et al, which is incorporated herein by reference.
This application is also a continuation-in-part of U.S. patent application Ser. No. 15/194,094 filed on Jun. 27, 2016, entitled âCONTENT PRESENTATION AND LAYERING ACROSS MULTIPLE DEVICES,â by Gaeta et al., which is incorporated herein by reference. U.S. patent application Ser. No. 15/194,094 claims the benefit of the following U.S. Provisional Applications:
U.S. Provisional Application No. 62/185,422, filed on Jun. 26, 2015, entitled âINTERACTING WITH AN IMMERSIVE ENVIRONMENT,â by Johnston et al, which is incorporated herein by reference. U.S. Provisional Application No. 62/232,985, filed on Sep. 25, 2015, entitled âCONTENT PRESENTATION AND LAYERING ACROSS MULTIPLE DEVICES,â by Koperwas et al, which is incorporated herein by reference. U.S. Provisional Application No. 62/291,648, filed on Feb. 5, 2016, entitled âCONTENT PRESENTATION AND LAYERING ACROSS MULTIPLE DEVICES,â by Koperwas et al, which is incorporated herein by reference.
BACKGROUND
Augmented reality includes a live view of a real-world environment that is augmented by computer generated sensory input(s), such as GPS graphics, video, sound, data statistics, and so forth. In contrast to virtual reality, which replaces the real-world environment with a simulated one, augmented reality elements are often displayed in real time in semantic context with elements of the real-world environment. For example, sports scores can be displayed on a television during a basketball game on a same screen. Headmounted displays can also be used to place the virtual images over a view of the physical world such that both are in the user's field of view.
BRIEF SUMMARY
In some embodiments, a computer-implemented method may include rendering a first view of a three-dimensional (3-D) virtual scene including a view of first content being displayed on a virtual display device from a location in the 3-D virtual scene. The method may also include rendering a second view comprising one or more content objects. The second view may be rendered from the location in the 3-D virtual scene. The second scene may include a view of the display device as would be seen through a pair of augmented-reality glasses that display the one or more content objects. The method may additionally include generating a composite view by combing the first view and the second view, and causing the composite view to be displayed on a virtual-reality headset.
In some embodiments, a non-transitory, computer-readable medium may include instructions that, when executed by one or more processors, cause the one or more processors to perform operations including rendering a first view of a three-dimensional (3-D) virtual scene including a view of first content being displayed on a virtual display device from a location in the 3-D virtual scene. The operations may also include rendering a second view comprising one or more content objects. The second view may be rendered from the location in the 3-D virtual scene. The second scene may include a view of the display device as would be seen through a pair of augmented-reality glasses that display the one or more content objects. The operations may also include generating a composite view by combing the first view and the second view, and causing the composite view to be displayed on a virtual-reality headset.
In some embodiments a system may include one or more processors and one or more memory devices comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including rendering a first view of a three-dimensional (3-D) virtual scene including a view of first content being displayed on a virtual display device from a location in the 3-D virtual scene. The operations may also include rendering a second view comprising one or more content objects. The second view may be rendered from the location in the 3-D virtual scene. The second scene may include a view of the display device as would be seen through a pair of augmented-reality glasses that display the one or more content objects. The operations may also include generating a composite view by combing the first view and the second view, and causing the composite view to be displayed on a virtual-reality headset.
In any of the embodiments described herein, one more of the following features may be included in any combination and without limitation. The 3-D virtual scene may include a virtual movie theater, the first content may include a movie generated at least in part from a rendered scene, and the virtual display device may include a virtual movie screen in the virtual movie theater displaying the movie. The virtual-reality headset may be located in a physical environment, and the 3-D virtual scene may be modeled after the physical environment. The physical environment may include a physical display device, and the virtual display device may be modeled after the physical display device. The method/operations may also include determining the movement and orientation of the virtual-reality headset, and altering the first view and the second view based on the movement and orientation of the virtual-reality headset. The second view may include a view of a virtual pair of augmented reality glasses. The one or more content objects may interact with the first content being displayed on the virtual display device in the composite view.
BRIEF DESCRIPTION OF THE DRAWINGS
A further understanding of the nature and advantages of the present invention may be realized by reference to the remaining portions of the specification and the drawings, wherein like reference numerals are used throughout the several drawings to refer to similar components. In some instances, a sub-label is associated with a reference numeral to denote one of multiple similar components. When reference is made to a reference numeral without specification to an existing sub-label, it is intended to refer to all such multiple similar components.
FIG. 1A illustrates an environment for presenting layered content across multiple devices, according to some embodiments.
FIG. 1B illustrates a custom hardware/software system for generating and displaying layered content on multiple devices, according to some embodiments.
FIG. 2 illustrates examples of the first mobile device, according to some embodiments.
FIG. 3A illustrates a scene displayed by the display system in the absence of the first mobile device, according to some embodiments.
FIG. 3B illustrates the same scene displayed by the mobile device in the absence of display system, according to some embodiments.
FIG. 3C illustrates the same scene displayed by the mobile device with the display system, according to some embodiments.
FIG. 3D illustrates the same scene after a change of position or orientation of the mobile device 178 , according to some embodiments.
FIG. 4A illustrates the relative sizing of an image rendered for display on the mobile device as it appears layered against the screen, according to some embodiments.
FIG. 4B illustrates the scene of FIG. 4A at a new perspective, according to some embodiments.
FIG. 5A illustrates an image displayed on the mobile device that is attached to the screen and consistent across viewing locations, according to some embodiments.
FIG. 5B illustrates an image displayed on the mobile device when viewed from a position to the left of the screen, according to some embodiments.
FIG. 6A illustrates an image displayed on the mobile device that is attached to a screen location with a consistent screen projection, according to some embodiments.
FIG. 6B illustrates the image displayed on the mobile device when viewed from a rightward viewing location, according to some embodiments.
FIG. 7A illustrates an example of how a motion path traveled by an object projected on the mobile device can change according to viewing location, according to some embodiments.
FIG. 7B illustrates a motion path that has been altered according to a different viewing location, according to some embodiments.
FIG. 8 illustrates a blur effect added to background landscape elements displayed on the screen, according to some embodiments.
FIG. 9A illustrates individualized motion paths for interactive content at two different locations, according to some embodiments.
FIG. 9B illustrates how the timing of various character interactions with the audience can be individualized, according to some embodiments.
FIG. 10A illustrates how content displayed on the mobile device can be compressed in depth based on a distance to the screen 152 , according to some embodiments.
FIG. 10B illustrates an effect for viewers that are far from the screen, according to some embodiments.
FIG. 11A illustrates how a dynamic compositing operation can be used to have objects displayed by the mobile device interact with objects displayed on the screen, according to some embodiments.
FIG. 11B illustrates how the character passes behind the first tree while moving along the motion path, according to some embodiments.
FIG. 11C illustrates how the character is viewed differently at the same point in time from a different location in the viewing environment, according to some embodiments.
FIG. 12 illustrates a flowchart of a computer-implemented method for displaying layered content across multiple devices, according to some embodiments.
FIG. 13A illustrates an example user interface, according to some embodiments.
FIG. 13B illustrates an example user interface similar to that shown in FIG. 13A , according to some embodiments.
FIG. 14 illustrates a user wearing a pair of virtual reality goggles to view a virtual environment.
FIG. 15 illustrates a block diagram of a system for simulating an augmented reality environment using a virtual-reality device, according to some embodiments.
FIG. 16A illustrates an environment in which the augmented reality simulation system may be used.
FIG. 16B illustrates how the view can change for the user when they move relative to a starting position, according to some embodiments.
FIG. 17A illustrates a first view inside the virtual-reality goggles of the scene in FIGS. 16A-16B , according to some embodiments.
FIG. 17B illustrates a second view inside the virtual-reality goggles of the scene in FIGS. 16A-16B , according to some embodiments.
FIG. 17C illustrates a third view inside the virtual-reality goggles of the scene in FIGS. 16A-16B , according to some embodiments.
FIGS. 18A-18B illustrates how objects can be synchronized between the first layer presented on a simulated screen and the second layer presented on the simulated augmented reality glasses, according to some embodiments.
FIGS. 19A-B illustrate how content objects and be adjusted that are displayed on the first layer corresponding to the virtual screen, according to some embodiments.
FIG. 20 illustrates a flowchart of a method for simulating augmented reality content, according to some embodiments.
FIG. 21 illustrates an exemplary computer system, in which parts of various embodiments of the present invention may be implemented.
DETAILED DESCRIPTION
Embodiments described herein may be directed at the integration and synchronization of content shown over a screen with additional content displayed via, for example, a pair of augmented-reality (AR) glasses. For example, a feature film may be presented over a typical movie screen. A pair of AR glasses worn by a viewer may simultaneously present additional content that augments the content presented over the screen. In this way, various visual effects and/or additional information may be conveyed to the viewer to enhance the viewer's experience. Illustratively, a pitcher may be shown as throwing a baseball in a movie presented over a screen. At some point, the baseball may âpop-outâ of the screen by being transitioned for display over AR glasses worn by a viewer.
In some embodiments, a display system manages the integration and synchronization of content presented over a screen with additional content presented via a pair of AR glasses. In one aspect, the presentation system automatically determines the location of every pair of AR glasses within an viewing environment. Specifically, the AR glasses may include one or more sensors and/or communication devices, such as GPS sensors, WiFi, bluetooth, cameras, visual markers, etc. The sensors/communication devices may be used to determine the locations of each viewer within the auditorium. The presentation system and/or each pair of AR glasses may further automatically determine the real and/or viewer relative dimensions of the screen in the auditorium.
Based on the locations of each pair of AR glasses and the screen dimensions, the presentation system determines how the additional content for each individual
CROSS REFERENCES
This application claims the benefit of the following U.S. Provisional Application:
U.S. Provisional Application No. 62/239,134, filed on Oct. 8, 2015, entitled âAUGMENTED REALITY SIMULATION,â by Koperwas et al, which is incorporated herein by reference.
This application is also a continuation-in-part of U.S. patent application Ser. No. 15/194,094 filed on Jun. 27, 2016, entitled âCONTENT PRESENTATION AND LAYERING ACROSS MULTIPLE DEVICES,â by Gaeta et al., which is incorporated herein by reference. U.S. patent application Ser. No. 15/194,094 claims the benefit of the following U.S. Provisional Applications:
U.S. Provisional Application No. 62/185,422, filed on Jun. 26, 2015, entitled âINTERACTING WITH AN IMMERSIVE ENVIRONMENT,â by Johnston et al, which is incorporated herein by reference. U.S. Provisional Application No. 62/232,985, filed on Sep. 25, 2015, entitled âCONTENT PRESENTATION AND LAYERING ACROSS MULTIPLE DEVICES,â by Koperwas et al, which is incorporated herein by reference. U.S. Provisional Application No. 62/291,648, filed on Feb. 5, 2016, entitled âCONTENT PRESENTATION AND LAYERING ACROSS MULTIPLE DEVICES,â by Koperwas et al, which is incorporated herein by reference.
BACKGROUND
Augmented reality includes a live view of a real-world environment that is augmented by computer generated sensory input(s), such as GPS graphics, video, sound, data statistics, and so forth. In contrast to virtual reality, which replaces the real-world environment with a simulated one, augmented reality elements are often displayed in real time in semantic context with elements of the real-world environment. For example, sports scores can be displayed on a television during a basketball game on a same screen. Headmounted displays can also be used to place the virtual images over a view of the physical world such that both are in the user's field of view.
BRIEF SUMMARY
In some embodiments, a computer-implemented method may include rendering a first view of a three-dimensional (3-D) virtual scene including a view of first content being displayed on a virtual display device from a location in the 3-D virtual scene. The method may also include rendering a second view comprising one or more content objects. The second view may be rendered from the location in the 3-D virtual scene. The second scene may include a view of the display device as would be seen through a pair of augmented-reality glasses that display the one or more content objects. The method may additionally include generating a composite view by combing the first view and the second view, and causing the composite view to be displayed on a virtual-reality headset.
In some embodiments, a non-transitory, computer-readable medium may include instructions that, when executed by one or more processors, cause the one or more processors to perform operations including rendering a first view of a three-dimensional (3-D) virtual scene including a view of first content being displayed on a virtual display device from a location in the 3-D virtual scene. The operations may also include rendering a second view comprising one or more content objects. The second view may be rendered from the location in the 3-D virtual scene. The second scene may include a view of the display device as would be seen through a pair of augmented-reality glasses that display the one or more content objects. The operations may also include generating a composite view by combing the first view and the second view, and causing the composite view to be displayed on a virtual-reality headset.
In some embodiments a system may include one or more processors and one or more memory devices comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including rendering a first view of a three-dimensional (3-D) virtual scene including a view of first content being displayed on a virtual display device from a location in the 3-D virtual scene. The operations may also include rendering a second view comprising one or more content objects. The second view may be rendered from the location in the 3-D virtual scene. The second scene may include a view of the display device as would be seen through a pair of augmented-reality glasses that display the one or more content objects. The operations may also include generating a composite view by combing the first view and the second view, and causing the composite view to be displayed on a virtual-reality headset.
In any of the embodiments described herein, one more of the following features may be included in any combination and without limitation. The 3-D virtual scene may include a virtual movie theater, the first content may include a movie generated at least in part from a rendered scene, and the virtual display device may include a virtual movie screen in the virtual movie theater displaying the movie. The virtual-reality headset may be located in a physical environment, and the 3-D virtual scene may be modeled after the physical environment. The physical environment may include a physical display device, and the virtual display device may be modeled after the physical display device. The method/operations may also include determining the movement and orientation of the virtual-reality headset, and altering the first view and the second view based on the movement and orientation of the virtual-reality headset. The second view may include a view of a virtual pair of augmented reality glasses. The one or more content objects may interact with the first content being displayed on the virtual display device in the composite view.
BRIEF DESCRIPTION OF THE DRAWINGS
A further understanding of the nature and advantages of the present invention may be realized by reference to the remaining portions of the specification and the drawings, wherein like reference numerals are used throughout the several drawings to refer to similar components. In some instances, a sub-label is associated with a reference numeral to denote one of multiple similar components. When reference is made to a reference numeral without specification to an existing sub-label, it is intended to refer to all such multiple similar components.
FIG. 1A illustrates an environment for presenting layered content across multiple devices, according to some embodiments.
FIG. 1B illustrates a custom hardware/software system for generating and displaying layered content on multiple devices, according to some embodiments.
FIG. 2 illustrates examples of the first mobile device, according to some embodiments.
FIG. 3A illustrates a scene displayed by the display system in the absence of the first mobile device, according to some embodiments.
FIG. 3B illustrates the same scene displayed by the mobile device in the absence of display system, according to some embodiments.
FIG. 3C illustrates the same scene displayed by the mobile device with the display system, according to some embodiments.
FIG. 3D illustrates the same scene after a change of position or orientation of the mobile device 178 , according to some embodiments.
FIG. 4A illustrates the relative sizing of an image rendered for display on the mobile device as it appears layered against the screen, according to some embodiments.
FIG. 4B illustrates the scene of FIG. 4A at a new perspective, according to some embodiments.
FIG. 5A illustrates an image displayed on the mobile device that is attached to the screen and consistent across viewing locations, according to some embodiments.
FIG. 5B illustrates an image displayed on the mobile device when viewed from a position to the left of the screen, according to some embodiments.
FIG. 6A illustrates an image displayed on the mobile device that is attached to a screen location with a consistent screen projection, according to some embodiments.
FIG. 6B illustrates the image displayed on the mobile device when viewed from a rightward viewing location, according to some embodiments.
FIG. 7A illustrates an example of how a motion path traveled by an object projected on the mobile device can change according to viewing location, according to some embodiments.
FIG. 7B illustrates a motion path that has been altered according to a different viewing location, according to some embodiments.
FIG. 8 illustrates a blur effect added to background landscape elements displayed on the screen, according to some embodiments.
FIG. 9A illustrates individualized motion paths for interactive content at two different locations, according to some embodiments.
FIG. 9B illustrates how the timing of various character interactions with the audience can be individualized, according to some embodiments.
FIG. 10A illustrates how content displayed on the mobile device can be compressed in depth based on a distance to the screen 152 , according to some embodiments.
FIG. 10B illustrates an effect for viewers that are far from the screen, according to some embodiments.
FIG. 11A illustrates how a dynamic compositing operation can be used to have objects displayed by the mobile device interact with objects displayed on the screen, according to some embodiments.
FIG. 11B illustrates how the character passes behind the first tree while moving along the motion path, according to some embodiments.
FIG. 11C illustrates how the character is viewed differently at the same point in time from a different location in the viewing environment, according to some embodiments.
FIG. 12 illustrates a flowchart of a computer-implemented method for displaying layered content across multiple devices, according to some embodiments.
FIG. 13A illustrates an example user interface, according to some embodiments.
FIG. 13B illustrates an example user interface similar to that shown in FIG. 13A , according to some embodiments.
FIG. 14 illustrates a user wearing a pair of virtual reality goggles to view a virtual environment.
FIG. 15 illustrates a block diagram of a system for simulating an augmented reality environment using a virtual-reality device, according to some embodiments.
FIG. 16A illustrates an environment in which the augmented reality simulation system may be used.
FIG. 16B illustrates how the view can change for the user when they move relative to a starting position, according to some embodiments.
FIG. 17A illustrates a first view inside the virtual-reality goggles of the scene in FIGS. 16A-16B , according to some embodiments.
FIG. 17B illustrates a second view inside the virtual-reality goggles of the scene in FIGS. 16A-16B , according to some embodiments.
FIG. 17C illustrates a third view inside the virtual-reality goggles of the scene in FIGS. 16A-16B , according to some embodiments.
FIGS. 18A-18B illustrates how objects can be synchronized between the first layer presented on a simulated screen and the second layer presented on the simulated augmented reality glasses, according to some embodiments.
FIGS. 19A-B illustrate how content objects and be adjusted that are displayed on the first layer corresponding to the virtual screen, according to some embodiments.
FIG. 20 illustrates a flowchart of a method for simulating augmented reality content, according to some embodiments.
FIG. 21 illustrates an exemplary computer system, in which parts of various embodiments of the present invention may be implemented.
DETAILED DESCRIPTION
Embodiments described herein may be directed at the integration and synchronization of content shown over a screen with additional content displayed via, for example, a pair of augmented-reality (AR) glasses. For example, a feature film may be presented over a typical movie screen. A pair of AR glasses worn by a viewer may simultaneously present additional content that augments the content presented over the screen. In this way, various visual effects and/or additional information may be conveyed to the viewer to enhance the viewer's experience. Illustratively, a pitcher may be shown as throwing a baseball in a movie presented over a screen. At some point, the baseball may âpop-outâ of the screen by being transitioned for display over AR glasses worn by a viewer.
In some embodiments, a display system manages the integration and synchronization of content presented over a screen with additional content presented via a pair of AR glasses. In one aspect, the presentation system automatically determines the location of every pair of AR glasses within an viewing environment. Specifically, the AR glasses may include one or more sensors and/or communication devices, such as GPS sensors, WiFi, bluetooth, cameras, visual markers, etc. The sensors/communication devices may be used to determine the locations of each viewer within the auditorium. The presentation system and/or each pair of AR glasses may further automatically determine the real and/or viewer relative dimensions of the screen in the auditorium.
Based on the locations of each pair of AR glasses and the screen dimensions, the presentation system determines how the additional content for each individual pair of AR glasses is to be presented to a corresponding viewer. Specifically, the positioning, orientation, size, and other features of the additional content may be automatically adjusted so that the additional content can be correctly integrated with the content presented over the screen for each viewer. Other viewers wearing other AR glasses may have the additional content be adjusted differently since they may be sitting at different locations within the auditorium.
As an example, an auditorium may be divided into multiple logical zones. The presentation system may indicate to a pair of AR glasses to present the additional content based on the zone in which the AR glasses are located. For instance, if a viewer were located at a logical zone on the right side of the auditorium, the presentation system may have the additional content presented to the viewer in a manner that compensates for the fact that the viewer is at the right side of the auditorium. Illustratively, the additional content may be placed in a particular position based on the fact that the viewer is located in the logical zone on the right side of the auditorium. The additional content may be placed at a different position for viewers located in other logical zones.
As another example, the AR glasses may automatically use a built-in camera or some other sensor to determine the dimensions of the screen in an auditorium relative to the viewer wearing the AR glasses. Based on this determination, the AR glasses may present the additional content such that it is scaled based on the determined relative dimensions of the screen for the viewer. For instance, a first user in an auditorium may be presented the additional content at a scale of 90%. A second user in the auditorium who is further back (and thus the relative dimensions of the screen may be smaller) may be presented the additional content at a scale of 50%.
As still another example, the presentation system may automatically use information regarding the location of the AR glasses and the predefined real dimensions of the screen to scale the additional content, which it then provides to the AR glasses for presentation to the user. For example, based on the real dimensions of the screen and the location of the AR glasses, the presentation system can determine the dimensions of the screen relative to the viewer wearing the AR glasses. Based on this determination, the presentation system can scale the additional content so that it appears âcorrectâ relative to the content presented over the screen.
In some aspects, the additional content may be automatically synchronized with the content presented over the screen. In some aspects, a camera or microphone built into the AR glasses may be used to trigger or signal when certain events are occurring in the feature film and cause certain portions of the additional content to be accordingly presented. In one aspect, the aforementioned triggering may be based on the matching of audio or image-based fingerprints. In other aspects, the presentation system may keep track of the run-time of the feature film. Based on the run-time, the presentation system may signal to the AR glasses when to present portions of the additional content.
Throughout the rest of this disclosure, a particular viewing environment will be used as an example. This viewing environment may be described as a traditional movie theater auditorium with a plurality of seats and a large screen at the front of the viewing area onto which a motion picture is projected. Each of the viewers seated in the auditorium may be equipped with a mobile device, such as the AR glasses described above. The mobile device can provide additional layered content that appears in an immersive and fluid fashion with the content projected on the screen. However, it will be understood that this particular viewing environment is merely used by way of example, and is not meant to be limiting. Other environments may also include a home with a television screen replacing the movie theater screen described below, and a tablet device, laptop device, mobile phone, and/or the like replacing the AR glasses described above. Therefore, throughout the remainder of this disclosure, any of the specific devices, arrangements, systems, or architectures that specifically refer to the theater environment may also be readily applied to a home viewing environment, or any other similar viewing environment, in any combination and without limitation.
FIG. 1A illustrates an environment 100 for presenting layered content across multiple devices, according to some embodiments. The environment 100 may include a display system and a mobile device. It should be appreciated that there may be additional components to the representative system. Additionally, certain components may not necessarily be required by the system. Components may also be combined. In one embodiment, each of the components in the system may be connected over a suitable connection. In certain embodiments, the components may be connected over the Internet or a local Intranet.
The display system may include devices suitable for projecting or displaying an item of content, such as a movie. The movie may include a digital animated film, a live-action film, a videogame, and/or any combination thereof. In some embodiments, the display system may include a projection device 150 and/or a screen 152 . The projection device 150 may project an item of content onto the screen 152 . In other embodiments, the screen 152 may include an active screen display, such as a television, an LCD television, a wall of LED screens, and/or the like. In still other embodiments, the display system may include a computer with a monitor, a tablet device, a smartphone, a watch, a portable DVD player, etc. For example, some embodiments may omit the projection device 150 , and the screen 152 can actively display the item of content.
The mobile device may be a set of virtual reality or augmented reality glasses or goggles. The mobile device may also include portable screens, such as the smart phone, a tablet device, a watch, a laptop computer, and/or the like. In some embodiments, the mobile device may include a display or a mechanism for projecting images that are visible to a user. The display of the mobile device may be transparent or semi-transparent such that a user can also see through the display. The projection of images over the display of the mobile device may only obscure a portion of the field of view of the user. In this way, a user can view both the image displayed by the mobile device and at least a portion of the physical environment behind the display of the mobile device.
In some embodiments, the physical environment behind the display of the mobile device may include the screen 152 . In these embodiments, the user can simultaneously view both an image displayed by the first mobile device and at least a portion of the display of the screen 152 . The embodiments described herein layer content between the screen 152 and the mobile device in a seamless and interactive way, such that the user is provided with an immersive content experience that is unique based on their location in the environment 100 .
In certain embodiments, the display system may load an item of content, such as a movie, and begin presenting the item of content to one or more users. Although not shown explicitly due to size constraints in FIG. 1A , each viewer sitting in the seats depicted in the environment 100 may be equipped with a mobile device. When referring to a specific viewer in a specific location in the environment 100 , the mobile device used by that specific viewer may be referred to as the âfirst mobile deviceâ or simply the âmobile device.â For example, the environment 100 includes
locations
120 , 122 , 124 , and 126 , each of which may include a user equipped with a mobile device. When referring specifically to the experience of a user sitting in location 120 , the mobile device of the user in location 120 may be referred to as the âmobile device.â When comparing the viewing experience of users in different locations, the respective mobile devices may be referred to as the âfirst mobile deviceâ and the âsecond mobile device.â Therefore, it will be understood that the terms âfirstâ and âsecondâ do not imply ordering, precedence, and/or preference. Instead, these terms are used merely to distinguish one mobile device from another mobile device.
FIG. 1B illustrates a custom hardware/software system for generating and displaying layered content on multiple devices, according to some embodiments. The system may include a content repository 156 that includes one or more different types of content. First, the content repository 156 may include a content data store 162 that includes content to be displayed by the display system 164 . In some embodiments, the content data store 162 may include full-length movies or complete media productions that can be displayed on the screen 152 . The content data store 162 may include a plurality of two-dimensional (2-D) frames that can be sequentially displayed on the screen 152 .
The system may also include a mobile device 178 , such as a pair of augmented- reality glasses 152 . The system may be configured to display additional content on the mobile device 178 during playback of the content in the content data store 162 on the screen 152 . In some embodiments, a position and/or orientation of the mobile device 178 may be known and static, in that the position and orientation of the mobile device 178 is constant and unchanging during playback of the content from the content data store 162 . In this case, the additional content to be displayed on the mobile device 178 can also be stored as a plurality of 2-D frames that are displayed on the mobile device 178 . In these embodiments, no additional rendering or compositing may be necessary in order to display the additional content on the mobile device 178 .
In some embodiments, the position and/or orientation of the mobile device 178 may be unknown and/or changing during playback of the content from the content data store 162 . In these embodiments, the system can generate a sequence of 2-D frames to be displayed on the mobile device 178 that account for the position and/or orientation of the mobile device 178 . In some cases, the approximate position of the mobile device 178 may be known, but the orientation of the mobile device 178 may be allowed to change dynamically. The content repository 156 may also include a rendered scene elements data store 162 that includes scene elements rendered from a particular approximate position of the mobile device 178 . These rendered scene elements can be retrieved by a mobile content processor 166 and processed into a sequence of 2-D frames for display on a mobile device. The rendered scene elements can be inserted into a compositing engine 172 to be edited before being sent to the mobile device 178 for display. In some scenes, the rendered scene elements may interact with elements of the content from the content repository 162 currently being displayed on the screen 152 . For example, a character or object being displayed on the mobile device 178 may pass behind an object being displayed on the screen 152 . The compositing engine 172 can generate a matte or cut out that removes a portion of the rendered scene elements comprising the character or object as it passes behind the object displayed on screen 152 . An example of this situation is described below in greater detail.
In the situation described above where a series of rendered, 2-D frames have been stored corresponding to a specific location in front of the screen 152 , that set of 2-D frames can be used for all viewer locations position at that orientation from the screen 152 , regardless of the distance from the screen (z-depth). However, while the same images may be reused, the distance from the screen can be used to scale, skew, and/or stretch each of these images based on distance from the screen. This may be done to compensate for the fact that an object displayed on the mobile device close to the screen will look very small, while the same object displayed at the same size on the mobile device far away from the screen will look very large in comparison to objects on the screen. Examples of this type of scaling are shown in greater detail in the examples given below.
The mobile content processor 166 may be co-implemented with the content repository 156 , and may transmit the sequence of 2-D frames to the mobile device 178 after the sequence is generated. In other embodiments, the mobile content processor 166 may be integrated with the mobile device 178 such that each mobile device 178 in the environment 100 is able to process its own individual sequence of 2-D frames for display. In a theater with hundreds of viewers, pushing the mobile content processor 166 out to each of the individual mobile devices 178 can alleviate decentralized transmission and/or generation of 3-D images.
In some embodiments, the content to be displayed on the mobile device 178 may be generated in real time as the content from the content repository 162 is displayed on the screen 152 . A 3-D scene content repository 158 can store animated sequences for a 3-D scene that can be rendered in real time for display on the mobile device 178 . The mobile content processor 166 can accept an animation sequence comprising 3-D scene elements. Turning back briefly to FIG. 1A , a location where the viewer is sitting can be used as a position of a virtual camera in the 3-D virtual scene. The image plane for the rendered scene can be positioned directly in front of the 3-D virtual camera to correspond to the position of the screen/lens of the mobile device 178 in front of the viewer's eye. The content in the content repository 162 being displayed on the screen 152 may have been rendered from the same scene with the image plane positioned at the location of the screen 152 relative to the position of the mobile device 178 and the viewer. The position and/or orientation of the mobile device 178 can be determined in real time and fed into the rendering engine 170 such that the position/orientation of the virtual camera and image plane can be updated in real-time. The rendering engine 170 can use the determined position and the 3-D scene content in order to generate a rendered set of 2-D images for display a mobile device.
When the position of the mobile device 178 is directly in front of the screen 152 , the rendered scene elements can be displayed without alteration on the mobile device 178 . However, when the location of the mobile device 178 is to the left or right of the center of the screen 152 , the rendered content may need to be skewed in order to match the apparent skew of the screen 152 . For example, when the viewer is sitting in location 124 of the environment 100 of FIG. 1A , the rendered 3-D elements can be displayed on the mobile device 178 without alteration. However, when a viewer is sitting in location 126 , the objects appearing on the screen 152 will appear skewed because of the viewing angle relative to the screen 152 . To solve this problem, a skew operation 168 can be added to the mobile content processor 162 that uses the location of the mobile device 178 to skew the geometry of the 3-D elements before they are rendered by the rendering engine 170 . Alternatively or additionally, stored 2-D images can be stretched or compressed in the direction of the required skew without performing another rendering operation. An example of this situation is described in greater detail later in this disclosure.
As the viewer is viewing the item of content while wearing the mobile device 178 , the mobile device 178 may display additional content to augment the item of content displayed by the display system. This may require a synchronization process 174 to coordinate the timing of images displayed on the screen 152 and images displayed on the mobile device 178 . For example, the mobile device 178 may be configured to detect when certain scenes or events are occurring within the content displayed on screen 152 . In one embodiment, the mobile device 178 may detect such events by using a built in camera to match a currently displayed frame by the screen 152 with a predefined frame representative of the scene or event. In another embodiment, the mobile device 178 may detect such events by using a built in microphone to determine whether the audio currently being presented by the display system matches with a predefined audio representative of the scene or event. In yet another embodiment, a communication link between the display system and the mobile device 178 may be established. The display system may send a signal to the mobile device 178 via the communication link to indicate triggering or occurrence of a scene or event. If a match/event/scene is detected, the mobile device 178 may automatically present additional content associated with the scene or event. In some embodiments, the additional content may only be presented on a portion of the display of the mobile device 178 . The remaining portion may be kept transparent so that a user can still view the content being displayed over the screen 152 . In this way, the additional content presented by the mobile device 178 can appear to be layered over, overlaid, or composited with the content being displayed by the screen 152 . As an illustrative example, a user may be viewing a movie showing a starship battle scene. When the particular scene is detected by the mobile device 178 , the mobile device 178 may display a flying saucer that appears to interact with the content being displayed over the projection device 150 . This may only be perceived by the user wearing the mobile device 178 . Users who are not wearing mobile devices may not perceive the flying saucer or any additional content.
The mobile device 178 may include one or more position determining sensors 180 . In some embodiments, the sensors may include accelerometers, gyroscopes, GPS receivers, gravitation sensors, and/or digital compasses. The sensors can be used to determine a position and/or orientation of the mobile device 178 by measuring displacements from an initial, known position. For example, each mobile device in FIG. 1A could be assigned to a specific seat location in the environment 100 . At the beginning of the movie, the viewer may be instructed to look directly at a symbol in the middle of the screen 152 to establish a known, initial orientation. From that point, the accelerometers and gyroscopes could be sampled to measure displacements from this initial position. These sensors, along with a gravitational sensor can be used to measure a change in orientation from the initial orientation.
In some embodiments, the determining sensors 180 may include a radio transmitter an
CLAIMS
Claims ( 20 )
What is claimed is:
1. A computer-implemented method comprising:
rendering a first view of a three-dimensional (3-D) virtual scene from a point-of-view of a viewer in the 3-D virtual scene, the first view comprising a view of first content being displayed on a virtual display device;
rendering a second view comprising one or more content objects, wherein:
the second view is rendered from the same point-of-view of the viewer in the 3-D virtual scene; and
the second view comprises a view of the virtual display device as would be seen through a pair of augmented-reality glasses worn by the viewer that display the one or more content objects;
generating a composite view by combining the first view and the second view; and
causing the composite view to be displayed on a virtual-reality headset in a real-world environment.
2. The computer-implemented method of claim 1 , wherein:
the 3-D virtual scene comprises a virtual movie theater;
the first content comprises a movie generated at least in part from a rendered scene; and
the virtual display device comprises a virtual movie screen in the virtual movie theater displaying the movie.
3. The computer-implemented method of claim 1 , wherein:
the virtual-reality headset is located in a physical environment; and
the 3-D virtual scene is modeled after the physical environment.
4. The computer-implemented method of claim 3 , wherein:
the physical environment comprises a physical display device; and
the virtual display device is modeled after the physical display device.
5. The computer-implemented method of claim 1 , further comprising:
determining a movement and orientation of the virtual-reality headset; and
altering the first view and the second view based on the movement and orientation of the virtual-reality headset.
6. The computer-implemented method of claim 1 , wherein the second view comprises a view of a virtual pair of augmented reality glasses.
7. The computer-implemented method of claim 1 , wherein the one or more content objects interact with the first content being displayed on the virtual display device in the composite view.
8. A non-transitory, computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
rendering a first view of a three-dimensional (3-D) virtual scene from a point-of-view of a viewer in the 3-D virtual scene, the first view comprising a view of first content being displayed on a virtual display device;
rendering a second view comprising one or more content objects, wherein:
the second view is rendered from the same point-of-view of the viewer in the 3-D virtual scene; and
the second view comprises a view of the virtual display device as would be seen through a pair of augmented-reality glasses worn by the viewer that display the one or more content objects;
generating a composite view by combining the first view and the second view; and
causing the composite view to be displayed on a virtual-reality headset in a real-world environment.
9. The non-transitory, computer-readable medium of claim 8 , wherein:
the 3-D virtual scene comprises a virtual movie theater;
the first content comprises a movie generated at least in part from a rendered scene; and
the virtual display device comprises a virtual movie screen in the virtual movie theater displaying the movie.
10. The non-transitory, computer-readable medium of claim 8 , wherein:
the virtual-reality headset is located in a physical environment; and
the 3-D virtual scene is modeled after the physical environment.
11. The non-transitory, computer-readable medium of claim 10 , wherein:
the physical environment comprises a physical display device; and
the virtual display device is modeled after the physical display device.
12. The non-transitory, computer-readable medium of claim 8 , wherein the instructions cause the one or more processors to perform further operations comprising:
determining a movement and orientation of the virtual-reality headset; and
altering the first view and the second view based on the movement and orientation of the virtual-reality headset.
13. The non-transitory, computer-readable medium of claim 8 , wherein the second view comprises a view of a virtual pair of augmented reality glasses.
14. The non-transitory, computer-readable medium of claim 8 , wherein the one or more content objects interact with the first content being displayed on the virtual display device in the composite view.
15. A system comprising:
one or more processors;
one or more memory devices comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
rendering a first view of a three-dimensional (3-D) virtual scene from a point-of-view of a viewer in the 3-D virtual scene, the first view comprising a view of first content being displayed on a virtual display device;
rendering a second view comprising one or more content objects, wherein:
the second view is rendered from the same point-of-view of the viewer in the 3-D virtual scene; and
the second view comprises a view of the virtual display device as would be seen through a pair of augmented-reality glasses worn by the viewer that display the one or more content objects;
generating a composite view by combining the first view and the second view; and
causing the composite view to be displayed on a virtual-reality headset in a real-world environment.
16. The system of claim 15 , wherein:
the 3-D virtual scene comprises a virtual movie theater;
the first content comprises a movie generated at least in part from a rendered scene; and
the virtual display device comprises a virtual movie screen in the virtual movie theater displaying the movie.
17. The system of claim 15 , wherein:
the virtual-reality headset is located in a physical environment; and
the 3-D virtual scene is modeled after the physical environment.
18. The system of claim 15 , wherein the instructions cause the one or more processors to perform further operations comprising:
determining a movement and orientation of the virtual-reality headset; and
altering the first view and the second view based on the movement and orientation of the virtual-reality headset.
19. The system of claim 15 , wherein the second view comprises a view of a virtual pair of augmented reality glasses.
20. The system of claim 15 , wherein the one or more content objects interact with the first content being displayed on the virtual display device in the composite view.
US15/289,858
2015-06-26
2016-10-10
Augmented reality simulation
Active
US10078917B1
( en )
Priority Applications (1)
Application Number
Priority Date
Filing Date
Title
US15/289,858
US10078917B1
( en )
2015-06-26
2016-10-10
Augmented reality simulation
Applications Claiming Priority (6)
Application Number
Priority Date
Filing Date
Title
US201562185422P
2015-06-26
2015-06-26
US201562232985P
2015-09-25
2015-09-25
US201562239134P
2015-10-08
2015-10-08
US201662291648P
2016-02-05
2016-02-05
US15/194,094
US10484824B1
( en )
2015-06-26
2016-06-27
Content presentation and layering across multiple devices
US15/289,858
US10078917B1
( en )
2015-06-26
2016-10-10
Augmented reality simulation
Related Parent Applications (1)
Application Number
Title
Priority Date
Filing Date
US15/194,094
Continuation-In-Part
US10484824B1
( en )
2015-06-26
2016-06-27
Content presentation and layering across multiple devices
Publications (1)
Publication Number
Publication Date
US10078917B1
true
US10078917B1 ( en )
2018-09-18
Family
ID=63491152
Family Applications (1)
Application Number
Title
Priority Date
Filing Date
US15/289,858
Active
US10078917B1
( en )
2015-06-26
2016-10-10
Augmented reality simulation
Country Status (1)
Country
Link
US
( 1 )
US10078917B1
( en )
Cited By (66)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US20170105052A1
( en )
*
2015-10-09
2017-04-13
Warner Bros. Entertainment Inc.
Cinematic mastering for virtual reality and augmented reality
US20180067316A1
( en )
*
2016-09-08
2018-03-08
Lg Electronics Inc.
Head mounted display and method for controlling the same
US20180182168A1
( en )
*
2015-09-02
2018-06-28
Thomson Licensing
Method, apparatus and system for facilitating navigation in an extended scene
US20180196523A1
( en )
*
2017-01-10
2018-07-12
Disney Enterprises, Inc.
Simulation experience with physical objects
US20180255285A1
( en )
*
2017-03-06
2018-09-06
Universal City Studios Llc
Systems and methods for layered virtual features in an amusement park environment
US20180256974A1
( en )
*
2016-07-12
2018-09-13
Sony Corporation
Information processing apparatus, information processing method, and program
US10306180B2
( en )
*
2016-10-21
2019-05-28
Liquidsky Software, Inc.
Predictive virtual reality content streaming techniques
US10466775B2
( en )
*
2015-09-16
2019-11-05
Colopl, Inc.
Method and apparatus for changing a field of view without synchronization with movement of a head-mounted display
US10475247B1
( en )
*
2018-05-24
2019-11-12
Disney Enterprises, Inc.
Configuration for resuming/supplementing an augmented reality experience
US20190348169A1
( en )
*
2018-05-14
2019-11-14
Novarad Corporation
Aligning image data of a patient with actual views of the patient using an optical code affixed to the patient
US20200053253A1
( en )
*
2016-10-25
2020-02-13
Sony Interactive Entertainment Inc.
Video content synchronisation method and apparatus
US10565158B2
( en )
*
2017-07-31
2020-02-18
Amazon Technologies, Inc.
Multi-device synchronization for immersive experiences
US10602117B1
( en )
*
2017-09-11
2020-03-24
Bentley Systems, Incorporated
Tool for onsite augmentation of past events
US10602200B2
( en )
2014-05-28
2020-03-24
Lucasfilm Entertainment Company Ltd.
Switching modes of a media content item
US10621784B2
( en )
*
2017-09-29
2020-04-14
Sony Interactive Entertainment America Llc
Venue mapping for virtual reality spectating of live events
JP2020512723A
( en )
*
2017-04-11
2020-04-23
ãã«ãã¼ ã©ãã©ããªã¼ãº ã©ã¤ã»ã³ã·ã³ã° ã³ã¼ãã¬ã¤ã·ã§ã³
Layered enhanced entertainment experience
US10719987B1
( en )
*
2017-06-28
2020-07-21
Kilburn Live, Llc
Augmented reality in a virtual reality environment
US20200241733A1
( en )
*
2019-01-24
2020-07-30
Disney Enterprises, Inc.
Extended on-screen gameplay via augmented reality
US10764556B2
( en )
2018-09-26
2020-09-01
Snap Inc.
Depth sculpturing of three-dimensional depth images utilizing two-dimensional input selection
US10796489B1
( en )
2017-09-13
2020-10-06
Lucasfilm Entertainment Company Ltd.
Game engine responsive to motion-capture data for mixed-reality environments
EP3734418A1
( en )
*
2019-05-03
2020-11-04
XRSpace CO., LTD.
Head mounted display system capable of displaying a virtual scene and a map of a real environment in a picture-in-picture mode, related method and related non-transitory computer readable storage medium
GB2586148A
( en )
*
2019-08-07
2021-02-10
Sony Interactive Entertainment Inc
Content generation system and method
US20210142058A1
( en )
*
2019-11-08
2021-05-13
Msg Entertainment Group, Llc
Providing visual guidance for presenting visual content in a venue
CN112945222A
( en )
*
2021-01-27
2021-06-11
æå·é±èªè¹è¶ä¿®é æéå ¬å¸
Ship driving-assistant glasses image fusion method and system based on field direction
US11113884B2
( en )
2018-07-30
2021-09-07
Disney Enterprises, Inc.
Techniques for immersive virtual reality experiences
US11128984B1
( en )
2015-06-26
2021-09-21
Lucasfilm Eniertainment Company Ltd.
Content presentation and layering across multiple devices
US20210360155A1
( en )
*
2017-04-24
2021-11-18
Intel Corporation
Object pre-encoding for 360-degree view for optimal quality and latency
US11206459B2
( en )
*
2017-06-28
2021-12-21
At&T Intellectual Property I, L.P.
Method and apparatus for augmented reality presentation associated with a media program
US20210409677A1
( en )
*
2018-10-02
2021-12-30
Within Unlimited, Inc.
Methods, systems and devices supporting real-time shared virtual reality environment
US20220021863A1
( en )
*
2020-07-14
2022-01-20
Chad Lee
Methods and systems for facilitating population of a virtual space around a 2d content
CN114119817A
( en )
*
2020-08-26
2022-03-01
æé½ä¸ç»´è§æç§ææéå ¬å¸
System for automatically outputting image based on 3dsmax script automatic matching size correction scene
WO2022040729A1
( en )
*
2020-08-26
2022-03-03
Clapshaw Charles Leon
Volumetric immersion system & method
US20220076300A1
( en )
*
2015-12-31
2022-03-10
Manolo Fabio Rivera
Crowdfunding Using IPTV Systems and Fundraising Methods
US20220092332A1
( en )
*
2017-04-16
2022-03-24
Facebook, Inc.
Systems and methods for provisioning content
US11322117B2
( en )
*
2016-11-17
2022-05-03
Google Llc
Media rendering with orientation metadata
US11373402B2
( en )
*
2018-12-20
2022-06-28
Google Llc
Systems, devices, and methods for assisting human-to-human interactions
US11436806B1
( en )
2021-04-07
2022-09-06
Penumbra, Inc.
Dual perspective rendering in virtual reality
US11455549B2
( en )
2016-12-08
2022-09-27
Disney Enterprises, Inc.
Modeling characters that interact with users as part of a character-as-a-service implementation
US20220309542A1
( en )
*
2019-05-08
2022-09-29
Data Vault Holdings, Inc.
System and method for funding a virtual location
US11480787B2
( en )
*
2018-03-26
2022-10-25
Sony Corporation
Information processing apparatus and information processing method
US20220375026A1
( en )
*
2021-05-18
2022-11-24
Snap Inc.
Late warping to minimize latency of moving objects
US11561611B2
( en )
2020-10-29
2023-01-24
Micron Technology, Inc.
Displaying augmented reality responsive to an input
US20230025585A1
( en )
*
2021-07-21
2023-01-26
Sony Interactive Entertainment Inc.
Augmented reality artificial intelligence enhance ways user perceive themselves
US11580734B1
( en )
*
2021-07-26
2023-02-14
At&T Intellectual Property I, L.P.
Distinguishing real from virtual objects in immersive reality
WO2023049293A1
( en )
2021-09-24
2023-03-30
Dolby Laboratories Licensing Corporation
Augmented reality and screen image rendering coordination
US20230237760A1
( en )
*
2017-12-05
2023-07-27
Google Llc
Method for Converting Landscape Video to Portrait Mobile Layout Using a Selection Interface
US11715270B2
( en )
*
2021-09-14
2023-08-01
Verizon Patent And Licensing Inc.
Methods and systems for customizing augmentation of a presentation of primary content
EP4224289A1
( en )
*
2022-02-08
2023-08-09
Nokia Technologies Oy
Mediated reality
US20230256332A1
( en )
*
2022-02-16
2023-08-17
Sony Interactive Entertainment Inc.
Massively multiplayer local co-op and competitive gaming
US11798236B2
( en )
2020-02-28
2023-10-24
Mark Strachan
Augmented reality system and method
EP4274238A1
( en )
*
2022-05-07
2023-11-08
Nokia Technologies Oy
Display of video content
CN117321472A
( en )
*
2021-05-18
2023-12-29
æ¯çº³æ®å ¬å¸
Perform post-warping to minimize latency on moving objects
US20240054690A1
( en )
*
2022-08-15
2024-02-15
Disney Enterprises, Inc.
User Responsive Augmented Reality Enhancement of Moving Images
US12003821B2
( en )
2020-04-20
2024-06-04
Disney Enterprises, Inc.
Techniques for enhanced media experience
US20240212301A1
( en )
*
2022-12-23
2024-06-27
Beijing Zitiao Network Technology Co., Ltd.
Methods, apparatuses, device and storage medium for displaying content in virtual scene
US12086301B2
( en )
2022-06-01
2024-09-10
Sphere Entertainment Group, Llc
System for multi-user collaboration within a virtual reality environment
EP4521414A1
( en )
2023-09-11
2025-03-12
OP-Hygiene IP GmbH
Hygiene compliance system using augmented reality
US12322006B2
( en )
2022-08-15
2025-06-03
Disney Enterprises, Inc.
Augmented reality enhancement of moving images
RU2846181C1
( en )
*
2024-04-15
2025-09-01
Ðоо "ЯндекÑ"
Method and system for processing video content
US12423883B2
( en )
2022-08-15
2025-09-23
Disney Enterprises, Inc.
Dynamic scale augmented reality enhancement of images
US12433486B2
( en )
2022-04-08
2025-10-07
Quasistatics Inc.
Distributed body area network computing system, method, and computer program products
US20250341937A1
( en )
*
2022-05-25
2025-11-06
Beijing Zitiao Network Technology Co., Ltd.
Method, apparatus, device and medium for determining a virtual cursor in a virtual reality scene
US12481514B2
( en )
2023-06-04
2025-11-25
Apple Inc.
Volumetric user interface layers for electronic devices
WO2026008486A1
( en )
*
2024-07-05
2026-01-08
Human Reality Limited
A contextualized augmented reality system
US12592045B2
( en )
2020-02-28
2026-03-31
Mark Strachan
Augmented reality system and method
US12619308B2
( en )
2023-12-10
2026-05-05
Quasistatics Inc.
Wearable device for electro-quasistatic human body communication and a method thereof
Citations (14)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US20080246694A1
( en )
*
2007-04-06
2008-10-09
Ronald Fischer
Personal theater display
US20100142928A1
( en )
*
2005-08-06
2010-06-10
Quantum Signal, Llc
Overlaying virtual content onto video stream of people within venue based on analysis of the people within the video stream
US20110007069A1
( en )
*
2009-07-09
2011-01-13
Eun Young Lee
Method and system for presenting content
US20110187706A1
( en )
*
2010-01-29
2011-08-04
Vesely Michael A
Presenting a View within a Three Dimensional Scene
US20120127284A1
( en )
2010-11-18
2012-05-24
Avi Bar-Zeev
Head-mounted display device which provides surround video
US20120326948A1
( en )
2011-06-22
2012-12-27
Microsoft Corporation
Environmental-light filter for see-through head-mounted display device
US20130328762A1
( en )
2012-06-12
2013-12-12
Daniel J. McCulloch
Controlling a virtual object with a real controller device
US20140198096A1
( en )
*
2013-01-11
2014-07-17
Disney Enterprises, Inc.
Mobile tele-immersive gameplay
US20140364212A1
( en )
*
2013-06-08
2014-12-11
Sony Computer Entertainment Inc.
Systems and methods for transitioning between transparent mode and non-transparent mode in a head mounted dipslay
US20150070391A1
( en )
*
2012-06-29
2015-03-12
Sony Computer Entertainment Inc.
Image processing device, image processing method, and image processing system
US20150350628A1
( en )
*
2014-05-28
2015-12-03
Lucasfilm Entertainment CO. LTD.
Real-time content immersion system
US20160035140A1
( en )
*
2014-07-29
2016-02-04
Sony Computer Entertainment Europe Limited
Image processing
US20160182877A1
( en )
2013-07-28
2016-06-23
Michael J. DeLuca
Augmented reality based user interfacing
US20160379417A1
( en )
*
2011-12-06
2016-12-29
Microsoft Technology Licensing, Llc
Augmented reality virtual monitor
2016
2016-10-10
US
US15/289,858
patent/US10078917B1/en
active
Active
Patent Citations (14)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US20100142928A1
( en )
*
2005-08-06
2010-06-10
Quantum Signal, Llc
Overlaying virtual content onto video stream of people within venue based on analysis of the people within the video stream
US20080246694A1
( en )
*
2007-04-06
2008-10-09
Ronald Fischer
Personal theater display
US20110007069A1
( en )
*
2009-07-09
2011-01-13
Eun Young Lee
Method and system for presenting content
US20110187706A1
( en )
*
2010-01-29
2011-08-04
Vesely Michael A
Presenting a View within a Three Dimensional Scene
US20120127284A1
( en )
2010-11-18
2012-05-24
Avi Bar-Zeev
Head-mounted display device which provides surround video
US20120326948A1
( en )
2011-06-22
2012-12-27
Microsoft Corporation
Environmental-light filter for see-through head-mounted display device
US20160379417A1
( en )
*
2011-12-06
2016-12-29
Microsoft Technology Licensing, Llc
Augmented reality virtual monitor
US20130328762A1
( en )
2012-06-12
2013-12-12
Daniel J. McCulloch
Controlling a virtual object with a real controller device
US20150070391A1
( en )
*
2012-06-29
2015-03-12
Sony Computer Entertainment Inc.
Image processing device, image processing method, and image processing system
US20140198096A1
( en )
*
2013-01-11
2014-07-17
Disney Enterprises, Inc.
Mobile tele-immersive gameplay
US20140364212A1
( en )
*
2013-06-08
2014-12-11
Sony Computer Entertainment Inc.
Systems and methods for transitioning between transparent mode and non-transparent mode in a head mounted dipslay
US20160182877A1
( en )
2013-07-28
2016-06-23
Michael J. DeLuca
Augmented reality based user interfacing
US20150350628A1
( en )
*
2014-05-28
2015-12-03
Lucasfilm Entertainment CO. LTD.
Real-time content immersion system
US20160035140A1
( en )
*
2014-07-29
2016-02-04
Sony Computer Entertainment Europe Limited
Image processing
Non-Patent Citations (1)
* Cited by examiner, â Cited by third party
Title
U.S. Appl. No. 15/194,094, " Non-Final Office Action ", dated Aug. 24, 2017, 23 pages.
Cited By (122)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US10602200B2
( en )
2014-05-28
2020-03-24
Lucasfilm Entertainment Company Ltd.
Switching modes of a media content item
US11508125B1
( en )
2014-05-28
2022-11-22
Lucasfilm Entertainment Company Ltd.
Navigating a virtual environment of a media content item
US10600245B1
( en )
*
2014-05-28
2020-03-24
Lucasfilm Entertainment Company Ltd.
Navigating a virtual environment of a media content item
US11128984B1
( en )
2015-06-26
2021-09-21
Lucasfilm Eniertainment Company Ltd.
Content presentation and layering across multiple devices
US20180182168A1
( en )
*
2015-09-02
2018-06-28
Thomson Licensing
Method, apparatus and system for facilitating navigation in an extended scene
US11699266B2
( en )
*
2015-09-02
2023-07-11
Interdigital Ce Patent Holdings, Sas
Method, apparatus and system for facilitating navigation in an extended scene
US12293470B2
( en )
2015-09-02
2025-05-06
Interdigital Ce Patent Holdings, Sas
Method, apparatus and system for facilitating navigation in an extended scene
US10466775B2
( en )
*
2015-09-16
2019-11-05
Colopl, Inc.
Method and apparatus for changing a field of view without synchronization with movement of a head-mounted display
US20200228881A1
( en )
*
2015-10-09
2020-07-16
Warner Bros. Entertainment Inc.
Cinematic mastering for virtual reality and augmented reality
US20250030925A1
( en )
*
2015-10-09
2025-01-23
Warner Bros. Entertainment Inc.
Cinematic mastering for virtual reality and augmented reality
US10511895B2
( en )
*
2015-10-09
2019-12-17
Warner Bros. Entertainment Inc.
Cinematic mastering for virtual reality and augmented reality
US20230336840A1
( en )
*
2015-10-09
2023-10-19
Warner Bros. Entertainment Inc.
Cinematic mastering for virtual reality and augmented reality
US20170105052A1
( en )
*
2015-10-09
2017-04-13
Warner Bros. Entertainment Inc.
Cinematic mastering for virtual reality and augmented reality
US12143690B2
( en )
*
2015-10-09
2024-11-12
Warner Bros. Entertainment Inc.
Cinematic mastering for virtual reality and augmented reality
US12581170B2
( en )
*
2015-10-09
2026-03-17
Warner Bros. Entertainment Inc.
Cinematic mastering for virtual reality and augmented reality
US10924817B2
( en )
*
2015-10-09
2021-02-16
Warner Bros. Entertainment Inc.
Cinematic mastering for virtual reality and augmented reality
US11451882B2
( en )
*
2015-10-09
2022-09-20
Warner Bros. Entertainment Inc.
Cinematic mastering for virtual reality and augmented reality
US11922468B2
( en )
*
2015-12-31
2024-03-05
Manolo Fabio Rivera
Crowdfunding using IPTV systems and fundraising methods
US20220076300A1
( en )
*
2015-12-31
2022-03-10
Manolo Fabio Rivera
Crowdfunding Using IPTV Systems and Fundraising Methods
US20180256974A1
( en )
*
2016-07-12
2018-09-13
Sony Corporation
Information processing apparatus, information processing method, and program
US10668372B2
( en )
*
2016-07-12
2020-06-02
Sony Corporation
Information processing apparatus, information processing method, and program
US10746996B2
( en )
*
2016-09-08
2020-08-18
Lg Electronics Inc.
Head mounted display and method for controlling the same
US20180067316A1
( en )
*
2016-09-08
2018-03-08
Lg Electronics Inc.
Head mounted display and method for controlling the same
US10306180B2
( en )
*
2016-10-21
2019-05-28
Liquidsky Software, Inc.
Predictive virtual reality content streaming techniques
US11330150B2
( en )
*
2016-10-25
2022-05-10
<span itemprop="assignee