ABSTRACT
Abstract
A system and method for displaying computer-based content in a virtual or augmented environment includes receiving a selection of content for display on a window of a computing system, rendering the content as a virtual screen independently orientable from other virtual screens corresponding to other windows of the computing system, compositing the virtual screen into a virtual or augmented reality environment, detecting a selection of the virtual screen using a virtual controller, based on a gaze of a user within the virtual or augmented reality environment, or both, and changing properties of the virtual screen based on manipulation of the virtual controller by the user. In some embodiments, the system and method further include one or more of rendering a pointing ray and changing the properties based on movement of the virtual controller as if the virtual controller and the virtual screen are connected by a rigid link.
Description
RELATED APPLICATIONS
The present application claims priority to U.S. Provisional Patent Application No. 62/366,860, entitled âSystem and Method for Displaying Computer-Based Content in a Virtual or Augmented Environment,â filed Jul. 26, 2016, the entire disclosure of which is herein incorporated by reference.
TECHNICAL FIELD
The present disclosure relates generally to computing systems and the presentation of virtual and/or augmented environments and more specifically to displaying of computer-based content in a virtual and/or augmented environment.
BACKGROUND
The popularity of virtual and/or augmented reality systems is increasing. However, users of virtual and/or augmented reality systems, especially those using a head-mounted display, often have limited ability to interact with objects and the environment around them. This is especially true when users want to interact with windows and other non-virtual content available on their desktops, laptops, and/or tablets that is not part of the virtual and/or augmented environment which they are exploring. Accordingly, it would be advantageous to have a virtual and/or augmented reality system that allows users to select, display, and/or manipulate non-virtual content within their virtual and/or augmented environments.
SUMMARY
According to some embodiments, a method performed by one or more processors includes receiving a selection of content for display on a window of a computing system, rendering the content as a virtual screen independently orientable from one or more other virtual screens corresponding to other windows of the computing system, compositing the virtual screen into a virtual or augmented reality environment, detecting a selection of the virtual screen using a virtual controller, based on a gaze of a user within the virtual or augmented reality environment, or both, and changing one or more properties of the virtual screen based on manipulation of the virtual controller by the user.
According to some embodiments, a system includes a head-mounted display, a hand-held controller, and one or more processors. The one or more processors are coupled to the head-mounted display and the hand-held controller. The one or more processors are configured to receive a selection of content for display on a window of a computing system, render the content as a virtual screen independently orientable from one or more other virtual screens corresponding to other windows of the computing system, render the hand-held controller as a virtual controller within the virtual or augmented reality environment, composite the virtual screen and the virtual controller into a virtual or augmented reality environment displayed on the head-mounted display, detect a selection of the virtual screen or other virtual object using the virtual controller, based on a gaze of a user within the virtual or augmented reality environment, or both, and change one or more properties of the virtual screen or the other virtual object based on manipulation of the hand-held controller by the user.
According to some embodiments, a non-transitory computer-readable medium includes a plurality of machine-readable instructions which when executed by one or more processors associated with a virtual or augmented reality system are adapted to cause the one or more processors to perform a method. The method includes receiving a selection of content for display on a window of a computing system, rendering the content as a virtual screen as a virtual screen independently orientable from one or more other virtual screens corresponding to other windows of the computing system, rendering a hand-held controller as a virtual controller within the virtual or augmented reality environment, compositing the virtual screen into a virtual or augmented reality environment, detecting a selection of the virtual screen using the virtual controller, based on a gaze of a user within the virtual or augmented reality environment, or both, and changing one or more properties of the virtual screen based on manipulation of the virtual controller by a user.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 is a simplified diagram of a computing environment according to some embodiments.
FIG. 2 is a simplified diagram of a computing system according to some embodiments.
FIG. 3 is a simplified diagram of a virtual or augmented reality environment incorporating non-virtual content according to some embodiments.
FIG. 4 is a simplified diagram of a method of incorporating non-virtual content in a virtual or augmented reality environment according to some embodiments.
FIG. 5 is a simplified diagram of virtual screen manipulation according to some embodiments.
In the figures, elements having the same designations have the same or similar functions.
DETAILED DESCRIPTION
In the following description, specific details are set forth describing some embodiments consistent with the present disclosure. It will be apparent, however, to one skilled in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are meant to be illustrative but not limiting. One skilled in the art may realize other elements that, although not specifically described here, are within the scope and the spirit of this disclosure. In addition, to avoid unnecessary repetition, one or more features shown and described in association with one embodiment may be incorporated into other embodiments unless specifically described otherwise or if the one or more features would make an embodiment non-functional.
FIG. 1 is a simplified diagram of a computing environment 100 according to some embodiments. As shown in FIG. 1 , the elements of computing environment 100 are grouped into input devices, content sources, displayed objects, and a virtual reality (VR) system. The elements of computing environment 100 cooperate to generate a virtual and/or augmented environment so that both virtual reality-based content, such as backgrounds, avatars, virtual objects and/or the like, may be displayed along with non-virtual reality-based content such as computer desktops, application screens and windows, steaming content received over a network, and/or the like. The combined content is then displayed to a user using, for example, a head-mounted display (HMD) 150 .
The input devices of computing environment 100 include many different types of devices that are often used by computers, such as desktops, laptops, tablets, and/or the like, and/or virtual reality systems to receive user and/or user-related input. As shown, computing environment 100 includes one or more virtual reality input devices 111 , one or more head-mounted (HM) cameras 112 , one or more desktop cameras 113 , and one or more computer input devices 114 . The one or more virtual reality input devices 111 include input devices that may be used by a user in a virtual and/or augmented environment to manipulate objects and/or elements in the virtual and/or augmented environment. In some embodiments, the one or more virtual reality input devices 111 may include one or more positioning systems, tracking systems, accelerators, gyroscopes, cameras, and/or the like to track motion of the user's head, arms, hands, legs, feet, and/or the like. In some examples, the tracked motion may be evaluated to determine whether it corresponds to one or more inputs or commands to computing environment 100 . Examples of systems and methods capable of interpreting tracked motions and movements as commands to a computing system are described further in U.S. Provisional Patent Application No. 62/349,612 (entitled âSystem And Method For A Blended Reality User Interface And Gesture Control Systemâ) filed Jun. 24, 2016, which is hereby incorporated by reference in its entirety. In some examples, the one or more virtual reality input devices 111 may be co-located with one or more of the one or more head-mounted cameras 112 and/or head-mounted display 150 in order to capture head movement of the user so as to be able to, for example, track the direction of the user's gaze so that as the user looks around the virtual and/or augmented environment, the portions of the virtual and/or augmented environment displayed on head-mounted display 150 is updated accordingly. In some examples, the one or more virtual reality input devices 111 may be associated with hand-held and/or worn devices, such as virtual reality controllers, game controllers, hand-held controllers, remote controls, data gloves, and/or the like. The hand-held and/or worn devices may then be monitored and/or tracked to allow the user to select elements from the virtual and/or augmented environment and manipulate them by moving them around, rotating them, and/or the like.
The one or more head-mounted cameras 112 may be used to capture the physical environment around the user. The images and/or video feeds captured by the one or more head-mounted cameras 112 may be used to provide content for the virtual and/or augmented environment, such as when the virtual and physical environments are blended. Examples of systems and methods capable of doing this are described further in U.S. Provisional Patent Application No. 62/349,612 (entitled âSystem And Method For A Blended Reality User Interface And Gesture Control Systemâ) filed Jun. 24, 2016, which is hereby incorporated by reference in its entirety. The images and/or video feeds may further be used to determine the location and/or orientation of the user's head to further support the one or more virtual reality input devices 111 .
The one or more desktop cameras 113 may include one or more integrated and/or mounted cameras that may, for example, be used to capture the physical environment around the user and/or to capture images of the user. The images and/or video feeds captured by the one or more head-mounted cameras 112 may also be used to determine the location and/or orientation of the user's head, arms, hands, legs, feet, and/or the like to further support the one or more virtual reality input devices 111 .
The one or more computer input devices 114 include one or more keyboards, keypads, mice, trackballs, touch pads, joy sticks, game controllers, and/or the like typically used to interact with desktop, laptop, tablet, and/or similar computing devices. Using the one or more computer input devices 114 , the user may interact with the computer hosting the virtual reality system, select content, and/or direct the content for inclusion in the virtual and/or augmented environment as is discussed in further detail below.
The content sources of computing environment 100 include many different types of sources that may be used to provide visual and/or other content for inclusion in the virtual and/or augmented environment. As shown, computing environment 100 includes one or more user sensors 121 , one or more widgets 122 , one or more network streams 123 , one or more desktops and/or applications 124 , third party content 125 , and/or one or more render surfaces 126 . The one or more user sensors 121 provide content that is derived from the one or more virtual reality input devices 111 , the one or more head-mounted cameras 112 , and/or the
RELATED APPLICATIONS
The present application claims priority to U.S. Provisional Patent Application No. 62/366,860, entitled âSystem and Method for Displaying Computer-Based Content in a Virtual or Augmented Environment,â filed Jul. 26, 2016, the entire disclosure of which is herein incorporated by reference.
TECHNICAL FIELD
The present disclosure relates generally to computing systems and the presentation of virtual and/or augmented environments and more specifically to displaying of computer-based content in a virtual and/or augmented environment.
BACKGROUND
The popularity of virtual and/or augmented reality systems is increasing. However, users of virtual and/or augmented reality systems, especially those using a head-mounted display, often have limited ability to interact with objects and the environment around them. This is especially true when users want to interact with windows and other non-virtual content available on their desktops, laptops, and/or tablets that is not part of the virtual and/or augmented environment which they are exploring. Accordingly, it would be advantageous to have a virtual and/or augmented reality system that allows users to select, display, and/or manipulate non-virtual content within their virtual and/or augmented environments.
SUMMARY
According to some embodiments, a method performed by one or more processors includes receiving a selection of content for display on a window of a computing system, rendering the content as a virtual screen independently orientable from one or more other virtual screens corresponding to other windows of the computing system, compositing the virtual screen into a virtual or augmented reality environment, detecting a selection of the virtual screen using a virtual controller, based on a gaze of a user within the virtual or augmented reality environment, or both, and changing one or more properties of the virtual screen based on manipulation of the virtual controller by the user.
According to some embodiments, a system includes a head-mounted display, a hand-held controller, and one or more processors. The one or more processors are coupled to the head-mounted display and the hand-held controller. The one or more processors are configured to receive a selection of content for display on a window of a computing system, render the content as a virtual screen independently orientable from one or more other virtual screens corresponding to other windows of the computing system, render the hand-held controller as a virtual controller within the virtual or augmented reality environment, composite the virtual screen and the virtual controller into a virtual or augmented reality environment displayed on the head-mounted display, detect a selection of the virtual screen or other virtual object using the virtual controller, based on a gaze of a user within the virtual or augmented reality environment, or both, and change one or more properties of the virtual screen or the other virtual object based on manipulation of the hand-held controller by the user.
According to some embodiments, a non-transitory computer-readable medium includes a plurality of machine-readable instructions which when executed by one or more processors associated with a virtual or augmented reality system are adapted to cause the one or more processors to perform a method. The method includes receiving a selection of content for display on a window of a computing system, rendering the content as a virtual screen as a virtual screen independently orientable from one or more other virtual screens corresponding to other windows of the computing system, rendering a hand-held controller as a virtual controller within the virtual or augmented reality environment, compositing the virtual screen into a virtual or augmented reality environment, detecting a selection of the virtual screen using the virtual controller, based on a gaze of a user within the virtual or augmented reality environment, or both, and changing one or more properties of the virtual screen based on manipulation of the virtual controller by a user.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 is a simplified diagram of a computing environment according to some embodiments.
FIG. 2 is a simplified diagram of a computing system according to some embodiments.
FIG. 3 is a simplified diagram of a virtual or augmented reality environment incorporating non-virtual content according to some embodiments.
FIG. 4 is a simplified diagram of a method of incorporating non-virtual content in a virtual or augmented reality environment according to some embodiments.
FIG. 5 is a simplified diagram of virtual screen manipulation according to some embodiments.
In the figures, elements having the same designations have the same or similar functions.
DETAILED DESCRIPTION
In the following description, specific details are set forth describing some embodiments consistent with the present disclosure. It will be apparent, however, to one skilled in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are meant to be illustrative but not limiting. One skilled in the art may realize other elements that, although not specifically described here, are within the scope and the spirit of this disclosure. In addition, to avoid unnecessary repetition, one or more features shown and described in association with one embodiment may be incorporated into other embodiments unless specifically described otherwise or if the one or more features would make an embodiment non-functional.
FIG. 1 is a simplified diagram of a computing environment 100 according to some embodiments. As shown in FIG. 1 , the elements of computing environment 100 are grouped into input devices, content sources, displayed objects, and a virtual reality (VR) system. The elements of computing environment 100 cooperate to generate a virtual and/or augmented environment so that both virtual reality-based content, such as backgrounds, avatars, virtual objects and/or the like, may be displayed along with non-virtual reality-based content such as computer desktops, application screens and windows, steaming content received over a network, and/or the like. The combined content is then displayed to a user using, for example, a head-mounted display (HMD) 150 .
The input devices of computing environment 100 include many different types of devices that are often used by computers, such as desktops, laptops, tablets, and/or the like, and/or virtual reality systems to receive user and/or user-related input. As shown, computing environment 100 includes one or more virtual reality input devices 111 , one or more head-mounted (HM) cameras 112 , one or more desktop cameras 113 , and one or more computer input devices 114 . The one or more virtual reality input devices 111 include input devices that may be used by a user in a virtual and/or augmented environment to manipulate objects and/or elements in the virtual and/or augmented environment. In some embodiments, the one or more virtual reality input devices 111 may include one or more positioning systems, tracking systems, accelerators, gyroscopes, cameras, and/or the like to track motion of the user's head, arms, hands, legs, feet, and/or the like. In some examples, the tracked motion may be evaluated to determine whether it corresponds to one or more inputs or commands to computing environment 100 . Examples of systems and methods capable of interpreting tracked motions and movements as commands to a computing system are described further in U.S. Provisional Patent Application No. 62/349,612 (entitled âSystem And Method For A Blended Reality User Interface And Gesture Control Systemâ) filed Jun. 24, 2016, which is hereby incorporated by reference in its entirety. In some examples, the one or more virtual reality input devices 111 may be co-located with one or more of the one or more head-mounted cameras 112 and/or head-mounted display 150 in order to capture head movement of the user so as to be able to, for example, track the direction of the user's gaze so that as the user looks around the virtual and/or augmented environment, the portions of the virtual and/or augmented environment displayed on head-mounted display 150 is updated accordingly. In some examples, the one or more virtual reality input devices 111 may be associated with hand-held and/or worn devices, such as virtual reality controllers, game controllers, hand-held controllers, remote controls, data gloves, and/or the like. The hand-held and/or worn devices may then be monitored and/or tracked to allow the user to select elements from the virtual and/or augmented environment and manipulate them by moving them around, rotating them, and/or the like.
The one or more head-mounted cameras 112 may be used to capture the physical environment around the user. The images and/or video feeds captured by the one or more head-mounted cameras 112 may be used to provide content for the virtual and/or augmented environment, such as when the virtual and physical environments are blended. Examples of systems and methods capable of doing this are described further in U.S. Provisional Patent Application No. 62/349,612 (entitled âSystem And Method For A Blended Reality User Interface And Gesture Control Systemâ) filed Jun. 24, 2016, which is hereby incorporated by reference in its entirety. The images and/or video feeds may further be used to determine the location and/or orientation of the user's head to further support the one or more virtual reality input devices 111 .
The one or more desktop cameras 113 may include one or more integrated and/or mounted cameras that may, for example, be used to capture the physical environment around the user and/or to capture images of the user. The images and/or video feeds captured by the one or more head-mounted cameras 112 may also be used to determine the location and/or orientation of the user's head, arms, hands, legs, feet, and/or the like to further support the one or more virtual reality input devices 111 .
The one or more computer input devices 114 include one or more keyboards, keypads, mice, trackballs, touch pads, joy sticks, game controllers, and/or the like typically used to interact with desktop, laptop, tablet, and/or similar computing devices. Using the one or more computer input devices 114 , the user may interact with the computer hosting the virtual reality system, select content, and/or direct the content for inclusion in the virtual and/or augmented environment as is discussed in further detail below.
The content sources of computing environment 100 include many different types of sources that may be used to provide visual and/or other content for inclusion in the virtual and/or augmented environment. As shown, computing environment 100 includes one or more user sensors 121 , one or more widgets 122 , one or more network streams 123 , one or more desktops and/or applications 124 , third party content 125 , and/or one or more render surfaces 126 . The one or more user sensors 121 provide content that is derived from the one or more virtual reality input devices 111 , the one or more head-mounted cameras 112 , and/or the one or more desktop cameras 113 . In some examples, the one or more user sensors 121 may be used to capture information about the location, orientation, activity, and/or the like of the user that may be incorporated into the virtual and/or augmented environment. For example, input from the one or more virtual reality input devices 111 and/or one or more desktop cameras 113 may be used to capture the location, orientation, and/or activity of the user's hands so that virtual hands mimicking the behavior the user's hands may be included in the virtual and/or augmented environment. In some examples, the virtual hands may correspond to one or more of the one or more virtual controllers 132 so that movements and gestures made by the user's hands may be interpreted as inputs and/or other commands with the virtual and/or augmented environment. In some examples, the one or more user sensors 121 may further be used to track the one or more virtual reality input devices 111 so that virtual controllers, such as one or more of the one or more virtual controllers 132 and/or the one or more virtual objects 133 , corresponding to the one or more virtual reality input devices 111 may be included in the virtual and/or augmented environment.
The one or more widgets 122 may include any virtual widgets corresponding to information received from any of the one or more virtual reality input devices 111 , the one or more head-mounted cameras 112 , the one or more desktop cameras 113 , the one or more computer input devices 114 , and/or the virtual reality system in general. In some examples, the widgets 122 may provide status information regarding the virtual reality system, the underlying computing system, and/or the like. In some examples, the one or more widgets 122 may include a monitor of central processing unit (CPU) temperature, CPU utilization, displayed frame rate, and/or the like. In some examples, the one or more widgets 122 used for monitoring may provide numeric information, an analog-style gauge, and/or the like. In some examples, the one or more widgets 122 may provide virtual analogs to input device information, such as providing a virtual video screen showing images and/or video feeds captured by the one or more head-mounted cameras 112 and/or the one or more desktop cameras 113 . In some examples, the one or more widgets 122 may include tooltips and/or other help mechanisms to aid the user in using and/or navigating through the virtual and/or augmented environment. In some examples, the one or more widgets 122 providing tip and/or help information may be disabled by the user and/or be automatically phased out as the user becomes more familiar with the virtual and/or augmented environment. In some examples, the one or more widgets 122 may be displayed as one or more of the one or more virtual objects 133 within the virtual and/or augmented environment.
The one or more networks streams 123 correspond to content that may be included in the virtual and/or augmented environment over a network coupled to computing environment 100 . In some examples, the one or more network streams 123 may include one or more video feeds, such as streaming television, one or more audio feeds, such as streaming radio, audio books, and/or the like, and/or streaming virtual reality content from other computing system, other virtual reality systems, other virtual reality users sharing their content, and/or the like. Each of the one or more network streams 123 may be displayed as one or more of the virtual screens 134 in the virtual and/or augmented environment.
The one or more desktops and/or applications 124 provide content that is currently being displayed on a desktop, laptop, tablet, and/or the like. In some examples, the content may include captures of an entire desktop, laptop, or tablet screen and/or may be limited to content displayed within a single application and/or window. In some examples, the user may use the one or more virtual reality input devices 111 and/or the one or more computer input devices 114 to select the desktops, applications, and/or windows to be displayed within the virtual and/or augmented environment. In some examples, the content may be selected using one or more specialized commands, application programming interface (API) calls, menu selections, hot keys, and/or the like. In some examples, the graphic content of the desktops, applications, and/or windows may be obtained by computing environment 100 through one or more specialized API calls of an underlying operating system, directly accessing display memory, and/or the like. Each of the one or more desktops and/or applications 124 may be displayed as one or more of the virtual screens 134 in the virtual and/or augmented environment. In some examples, when the underlying computing system includes more than one display, each display may be included in the virtual and/or augmented environment as a separate one of the one or more virtual screens 134 . In some examples, the one or more desktops and/or applications 124 may also provide one or more of the one or more virtual objects 133 being displayed in the virtual and/or augmented environment. For example, an e-commerce application and/or an e-commerce web site may provide 3D models of virtual analogs to products for sale that may be displayed in the virtual and/or augmented environment and/or manipulated by the user using the one or more virtual reality input devices 111 .
The third party content 125 may include content from third party sources including content stored locally as well as content accessed via a network. The third party content 125 may be relatively static and/or dynamic. In some examples, the third party content 125 may be streamed over the network in a similar fashion to the one or more network streams 123 . In some examples, the third party content 125 may be obtained from one or more content stores, content repositories, social networks, crowd sources (e.g., content from the Steam Workshop), feeds, portals, other virtual and/or augmented reality systems, and/or the like. Depending upon the nature and/or content of the third party content 125 , the third party content may be rendered as part of background and environment 131 and/or as one or more of the one or more virtual objects 133 . In some examples, the third party content 125 may correspond to one or more avatars of other users sharing the virtual or augmented environment.
The one or more render surfaces 126 may correspond to one or more virtual surfaces within the virtual and/or augmented environment and/or one or more surfaces mapped to the desktops, applications, and/or windows being captured. Each of the one or more render surfaces 126 may be displayed as one or more of the virtual screens 134 in the virtual and/or augmented environment.
The displayed objects of computing environment 100 include many different types of elements that may be displayed in the virtual and/or augmented environment. As shown, computing environment 100 includes displayed objects that include background and environment 131 , one or more virtual controllers 132 , one or more virtual objects, one or more virtual screens 134 , and/or the like. The background and environment 131 include displayed content elements that provide a backdrop for the virtual and/or augmented environment. The background and environment 131 may include any graphic content including images and/or artistic renditions of locations including famous locations, mundane locations, fictitious environments and landscapes, space scenes, and/or the like. In some examples, background and environment 131 may be different in different regions of the virtual and/or augmented environment, thus creating separate areas, rooms, zones, and/or the like. In some examples, choice of the background and environment 131 may be customized for each of the different regions to reflect a theme, purpose (e.g., a virtual office, a virtual recreation room, and/or the like), and/or the like of the respective region. The background and environment 131 may further be user customizable including the ability to recolor, reorient, scale, and/or the like, available content and/or the combine and/or otherwise blend content from multiple sources. In some examples, the background and environment may be received as part of the third party content 125 . In some examples, the background and environment 131 may be modeled using one or more of 2D models, 3D models, tiling, wire frames, solid models, particle systems, and/or the like.
The one or more virtual controllers 132 include any of the virtual analogs to the one or more virtual reality input devices 111 , the one or more head-mounted cameras 112 , the one or more desktop cameras, the user's hands, and/or the one or more computer input devices 114 so that these virtual analogs may be interacted with and/or used by the user to manipulate the virtual and/or augmented environment. In some examples, the one or more virtual controllers 132 may include one or more virtual hands that may be moved around as the user's hands are tracked so that gestures made by the user's hand and thus the virtual hands are considered as possible commands and/or inputs within the virtual and/or augmented environment. In some examples, the one or virtual controllers 132 may include a virtual rendition of a hand-held virtual reality input device 111 that may be moved around and/or repositioned in the virtual and/or augmented environment so as to track the movement and operation of a corresponding virtual reality input device 111 as is described in further detail below. In some examples, the one or more virtual controllers 132 may include one or more virtual extensions, such as a pointing ray that may aid the user in manipulating the one or more virtual controllers 132 in the virtual and/or augmented environment. In some embodiments, the one or more virtual controllers 132 may be used to activate menus, use menus, and/or the like to select, manipulate, transform, etc. the one or more virtual objects 133 and/or the one or more virtual screens 134 in order to help build and/or manipulate the virtual and/or augmented environment.
The one or more virtual objects 133 include any of one or more widgets 122 and/or part of the third party content 125 . In some examples, the one or more virtual objects 133 may be placed and/or oriented subject to input received from the one or more user sensors 121 . For example, input from the one or more user sensors 121 may be used to detect user operations that correspond to a user grasping, moving, rotating, and/or otherwise manipulating one of the one or more virtual objects 133 . In some examples, one or more of the one or more virtual objects 133 may be supplied as a corresponding 2D and/or 3D model by one or more of the one or more desktops and applications 124 , such as to support an e-commerce application and/or web site, a third party content source, and/or the like. In some examples, the one or more virtual objects 133 may be modeled using one or more of tiling, wire frames, solid models, particle systems, and/or the like.
The one or more virtual screens 134 correspond to the locations within the virtual and/or augmented environment where the virtualized content from any of the one or more network streams 123 , the one or more desktops or applications 124 , and/or the one or more render surfaces 126 are displayed. Each of the one or more virtual screens 134 may be manipulated independently of each other. In some examples, each of the one or more virtual screens 134 may appear within the virtual and/or augmented environment as it they are a screen, window, sign, and/or the like physically suspended within the virtual and/or augmented environment. Depending upon their configurations, each of the one or more virtual screens 134 may be rendered as a flat or a curved surface. In some examples, visual content on the one or more virtual screens 134 may be rendered as monoscopic and/or stereoscopic content based on whether the source content is 2D or 3D. In some examples, content having a 3D encoding (e.g., content coded in the Side-by-Side (SBS) 3D encoding) may be rendered stereoscopically so that appears to be three-dimensional when rendered in the virtual and/or augmented environment.
In some embodiments, the content displayed on the one or more virtual screens 134 may include active content, user selectable and/or manipulatable elements and/or the like that are consistent with the content being displayed on the respective virtual screen 134 . In some examples, the active content may include user interface elements, such as buttons and/or other controls, active regions, hyperlinks, and/or the like that may be activated on the respective virtual screen 134 within the virtual and/or augmented environment in much the same way corresponding user interfaces, web pages, and/or the like may be activated on a computer display. In some examples, activation of the controls, active regions, and/or hyperlinks may result in a change in content on the respective virtual screen 134 , creation of a new virtual screen 134 , cause the download and/or creation of new virtual objects 133 , new elements in the background and environment 131 , and/or the like. In some examples, an ecommerce website and/or application displayed in the virtual and/or augmented environment using one or more virtual screens 134 may be interacted with to view one or more virtual representations of objects for sale, view or more samples of content, and purchase one or more physical items, one or more virtual objects 133 , content, and/or the like.
According to some embodiments, placement of a newly created virtual screen 134 , virtual object 133 , and/or element in the background and environment 131 may be constrained by virtual geometries of the virtual and/or augmented environment. In some examples, a placement location may be dependent upon the identification of a suitably sized flat area, such as a textured surface, to provide one or more attachment points for the newly created virtual screen 134 , virtual object 133 , and/or element in the background and environment 131 . In some examples, the placement may further depend on the existence of a suitably sized volumetric space about the one or more attachment points (e.g., a ceiling fan may preferable be associated with a minimal clearance volume to allow the fan blades to spin without encountering other objects, elements, avatars, and/or the like). In some examples, the placement may further depend on an orientation of the one or more attachment points (e.g., a ceiling fan may be associated one or more attachment points on a flat surface oriented downward or mostly downward, a virtual screen 134 may be associated with one or more attachment points to a vertical or substantially vertical surface). In some examples, the placement may further depend on a type of texturing of the one or more attachment points. In some examples, when no suitable placement may be found for the new virtual screen 134 , virtual object 133 , and/or element in the background and environment 131 , a suitable replacement virtual screen 134 , virtual object 133 , and/or element in the background and environment 131 may be recommended that is similar to and/or consistent with the virtual screen 134 , virtual object 133 , and/or element in the background and environment 131 that is not placeable. In some examples, the virtual screen 134 , virtual object 133 , and/or element in the background and environment 131 may be scaled, either automatically or based on user input, to fit the one or more attachment points and/or the available volumetric space.
In some embodiments, the one or more virtual screens 134 may also act as one or more artificial light sources that may be used to illuminate any of the background and environment 131 , the one or more virtual controllers 132 , and/or the one or more virtual objects also being displayed in the virtual and/or augmented environment. In this way, the one or more virtual screens 134 may be included within the virtual and/or augmented environment in a way that allows them to illuminate areas around them in much the same way a physical computer screen would light up a nearby physical environment. In some examples, each of the one or more virtual screens 134 may be modeled as separate point, directional, and/or area light sources.
In some embodiments, when the one or more virtual screens 134 include content with an audio component, such as a video sequence with sound, the one or more virtual screens 134 may further act as a sound source, which may be modeled either as a background sound source and/or a sound source with position and/or direction. In some example, the sound source may be modeled with a predetermined range so that the sound source may not be heard when the user is farther away than the predetermined range from the sound source. In some examples, known APIs such as OpenAL and/or DirectSound may be used to model the sound within the virtual and/or augmented environment so that a user with stereo and/or surround sound speakers may be able to perceive the sound as if the sound originated at the location and/or orientation of the corresponding virtual screen 134 .
As shown, the virtual reality interface system of computing environment 100 includes a compositor 141 , an object repositioner 142 , and a process injector 143 . Compositor 141 receives the content associated with background and environment 131 , the one or more virtual controllers 132 , the one or more virtual objects, and/or the one or more virtual screens 134 and generates the virtual and/or augmented environment for display to the user using head-mounted display 150 . Compositor 141 includes support for graphical tools and/or the like useful for rendering immersive 3D virtual and/or augmented environments including support for depth or z-buffering to correctly obscure content hidden by content located closer to the user, lighting, blending, clipping, providing audio, and/or the like. In some examples, compositor 141 may use one or more of 2D modeling, 3D modeling, tiling, wire framing, solid modeling, particle modeling, ray tracing, shading, shadowing, and/or the like to composite any of the content in the virtual and/or augmented environment.
In some examples, when a render surface 126 is created for display as one of the one or more virtual screens 134 , compositor 141 may grab the content for display in the one or more virtual screens 134 from the display buffer of the underlying computing system. In some examples, when compositor 141 grabs the content from display from the display buffer, compositor may have to generate one or more intermediate frames to convert between the frame rate of the display buffer and the frame rate of head-mounted display 150 (e.g., between the 60 Hz or similar frame rate at which display images are refreshed in the display buffer and the 90 Hz or similar frame rate used to refresh head-mounted display 150 ). In some examples, compositor 141 may alternatively use inter-process communication to start a thread in the remote process responsible for generating the content of the render surface 126 and the virtual screen 134 . The thread may be used to capture the screen directly, detect data and/or metadata changes, and/or the like and relay that information back to the compositor 141 so that compositor 141 may more effectively and efficiently render and composite the content into the virtual screen 134 . In some examples, use of the thread may also simplify synchronization of the content to be displayed within the virtual and/or augmented environment with the frame rate of head-mounted
CLAIMS
Claims ( 20 )
What is claimed is:
1. A method comprising:
receiving, by one or more processors, a selection of content for display on a window of a computing system;
rendering, by the one or more processors, the content as a virtual screen, the virtual screen being independently rotationally orientable about any combination of x, y, or z axes in a virtual or augmented reality environment from one or more other virtual screens corresponding to other windows of the computing system;
compositing, by the one or more processors, the virtual screen into the virtual or augmented reality environment;
rendering a hand-held input device as a virtual controller spaced from the virtual screens, the virtual controller being manipulatable as a virtual object within the virtual or augmented reality environment, the virtual controller having one or more rendered input controls;
rendering a pointing ray extending from the virtual controller;
detecting, by the one or more processors, a selection of the virtual screen using a virtual controller, based on a gaze of a user within the virtual or augmented reality environment, or both, and representing said selection in the virtual or augmented reality environment as selection by the virtual controller aiming at the virtual screen from a distance as aided by the pointing ray;
changing, by the one or more processors, an orientation of the virtual screen about any combination of the x, y, or z axes in the virtual or augmented reality environment independent of an orientation of the one or more other virtual screens based on manipulation of the virtual controller by the user so that rotations of the hand-held input device result in corresponding rotations of the virtual screen as if the virtual controller and the virtual screen are connected via a rigid link between a representative point on the virtual controller and a point on the virtual screen, the rigid link extending along the pointing ray, the pointing ray reaching said point on the virtual screen, wherein the manipulation of the virtual controller corresponds to a filtered version of a manipulation of the hand-held input device by a user;
automatically reorienting the virtual screen, after the virtual screen is deselected, to align a screen up direction of the virtual screen with a view up direction of virtual or augmented reality environment when the screen up direction of the virtual screen is within a first angular separation from the view up direction of the virtual or augmented reality environment;
automatically reorienting the virtual screen, after the virtual screen is deselected, to align an orientation of the virtual screen with a direction of view of the user when the orientation of the virtual screen is within a second angular separation from the direction of view of the user;
relocating, after the virtual screen is deselected, a sound source associated with the virtual screen based on a change in location or the orientation of the virtual screen due to the manipulation of the virtual controller; and
relocating, after the virtual screen is deselected, an illumination source associated with the virtual screen based on a change in location or the orientation of the virtual screen due to the manipulation of the virtual controller, the illumination source contributing to illumination of the virtual or augmented reality environment as a computer screen displaying a same video content as being displayed on the virtual screen would illuminate a real environment.
2. The method of claim 1 , wherein each of the one or more rendered input controls being selected from a group consisting of a button, a touchpad, a slider bar, a scroll wheel, and a rocker switch.
3. The method of claim 2 , further comprising:
changing one or more properties of the virtual screen, the one or more properties including one or more of a position of the virtual screen, a size of the virtual screen, or the orientation of the virtual screen;
wherein:
the changing of the position of the virtual screen is based on changes in a position of the hand-held input device as if the virtual controller and the virtual screen are connected by rigid link; and
the changing of the size of the virtual screen is based on up and down swipes on a touchpad, movement of a slider bar, rotation of a scroll wheel, or activation of a rocker switch.
4. The method of claim 1 , further comprising using the virtual screen to obtain a virtual object corresponding to an element displayed on the virtual screen.
5. The method of claim 1 , further comprising saving a state of the virtual or the augmented reality environment.
6. The method of claim 1 , further comprising adding a sound source to the virtual or augmented environment when the content includes an audio component.
7. The method of claim 1 , further comprising using the virtual screen as an illumination source for one or more other virtual objects within the virtual or augmented environment.
8. The method of claim 1 , further comprising aligning the virtual screen with an up direction in the virtual or augmented reality environment.
9. The method of claim 1 , further comprising aligning the virtual screen so it is directed toward the user in the virtual or augmented environment.
10. The method of claim 1 , wherein changing the orientation of the virtual screen comprises temporarily changing a transparency of the virtual screen so that it becomes semi-transparent when the virtual screen is selected and an opacity of the virtual screen is restored after the orientation is changed.
11. The method of claim 1 , wherein the user may navigate past or move behind the virtual screen within the virtual or augmented environment.
12. The method of claim 1 , further comprising rendering a respective menu for each of the virtual screen and the other virtual screens, each respective menu being usable by the user to manipulate a corresponding virtual screen.
13. The method of claim 1 , wherein a level of the filtering is adjusted based on a translational or a rotational velocity of the hand-held input device.
14. A system comprising:
a head-mounted display;
a hand-held controller; and
one or more processors coupled to the head-mounted display and the hand-held controller, the one or more processors being configured to:
receive a selection of content for display on a window of a computing system;
render the content as a virtual screen, the virtual screen being independently rotationally orientable about any combination of x, y, or z axes in a virtual or augmented reality environment from one or more other virtual screens corresponding to other windows of the computing system;
render the hand-held controller as a virtual controller, the virtual controller being manipulatable as a virtual object within the virtual or augmented reality environment, the virtual controller having one or more rendered input controls;
composite the virtual screen and the virtual controller into the virtual or augmented reality environment displayed on the head-mounted display;
detect a selection of the virtual screen by a user;
represent a selection of the virtual screen as selected by the virtual controller in the virtual or augmented reality environment when the virtual controller is at a distance from the virtual screen, wherein representing the selection of the virtual screen comprises:
rendering the virtual controller to aim at the virtual screen from a distance;
rendering a pointing ray extending from the virtual controller to the virtual screen; and
rendering a manipulation of the one or more rendered input controls of the virtual controller to select the virtual screen by the virtual controller aiming at the virtual screen from a distance;
change an orientation of the virtual screen about any combination of the x, y, or z axes in the virtual or augmented reality environment independent of an orientation of the one or more other virtual screens based on manipulation of the virtual controller by the user so that rotations of the hand-held controller result in corresponding rotations of the virtual screen as if the pointing ray is at least part of a rigid link connecting a representative point on the virtual controller to a point where the pointing ray meets the virtual screen, wherein the manipulation of the virtual controller corresponds to a filtered version of a manipulation of the hand-held controller by a user;
automatically reorienting the virtual screen, after the virtual screen is deselected, to align a screen up direction of the virtual screen with a view up direction of virtual or augmented reality environment when the screen up direction of the virtual screen is within a first angular separation from the view up direction of the virtual or augmented reality environment;
automatically reorienting the virtual screen, after the virtual screen is deselected, to align an orientation of the virtual screen with a direction of view of the user when the orientation of the virtual screen is within a second angular separation from the direction of view of the user;
relocating, after the virtual screen is deselected, a sound source associated with the virtual screen based on a change in location or the orientation of the virtual screen due to the manipulation of the virtual controller; and
relocating, after the virtual screen is deselected, an illumination source associated with the virtual screen based on a change in location or the orientation of the virtual screen due to the manipulation of the virtual controller, the illumination source contributing to illumination of the virtual or augmented reality environment as a computer screen displaying a same video content as being displayed on the virtual screen would illuminate a real environment.
15. The system of claim 14 , wherein:
the one or more processors are further configured to change one or more of a position of the virtual screen or, a size of the virtual screen, or the orientation of the virtual screen;
the change in the position of the virtual screen is based on changes in position of the hand-held controller as if the virtual controller and the virtual screen are connected by the rigid link; and
the change of the size of the virtual screen is based on up and down swipes on a touchpad, movement of a slider bar, rotation of a scroll wheel, or activation of a rocker switch.
16. The system of claim 14 , wherein the one or more processors are further configured to add a sound source to the virtual or augmented reality environment when the content includes an audio component, use the virtual screen as an illumination source for one or more other virtual objects within the virtual or augmented reality environment, or both.
17. The system of claim 14 , wherein the one or more processors are further configured to align the virtual screen with an up direction in the virtual or augmented reality environment, so it is directed toward the user in the virtual or augmented environment, or both.
18. A non-transitory computer-readable medium comprising a plurality of machine-readable instructions which when executed by one or more processors associated with a virtual or augmented reality system are adapted to cause the one or more processors to perform a method comprising:
receiving a selection of content for display on a window of a computing system;
rendering the content as a virtual screen, the virtual screen being independently rotationally orientable about any combination of x, y, or z axes in the virtual or augmented reality environment from one or more other virtual screens corresponding to other windows of the computing system;
rendering a hand-held controller as a virtual controller, the virtual controller being manipulatable as a virtual object within the virtual or augmented reality environment, the virtual controller having one or more rendered input controls, each of the one or more rendered input controls being selected from a group consisting of a button, a touchpad, a slider bar, a scroll wheel, and a rocker switch;
compositing the virtual screen into the virtual or augmented reality environment;
rendering a pointing ray extending from the virtual controller;
detecting a selection of the virtual screen by detecting that the virtual controller is aiming toward the virtual screen from a distance as indicated by the pointing ray;
changing an orientation of the virtual screen about any combination of the x, y, or z axes in the virtual or augmented reality environment independent of an orientation of the one or more other virtual screens based on manipulation of the virtual controller by a user so that rotations of the hand-held controller result in corresponding rotations of the virtual screen as if the virtual controller and the virtual screen are connected via a rigid link between a representative point on the virtual controller and a point on the virtual screen, the rigid link extending along the pointing ray, the pointing ray reaching said point on the virtual screen, wherein the manipulation of the virtual controller corresponds to a filtered version of a manipulation of the hand-held controller by a user;
automatically reorienting the virtual screen, after the virtual screen is deselected, to align a screen up direction of the virtual screen with a view up direction of virtual or augmented reality environment when the screen up direction of the virtual screen is within a first angular separation from the view up direction of the virtual or augmented reality environment;
automatically reorienting the virtual screen, after the virtual screen is deselected, to align an orientation of the virtual screen with a direction of view of the user when the orientation of the virtual screen is within a second angular separation from the direction of view of the user;
relocating, after the virtual screen is deselected, a sound source associated with the virtual screen based on a change in location or the orientation of the virtual screen due to the manipulation of the virtual controller; and
relocating, after the virtual screen is deselected, an illumination source associated with the virtual screen based on a change in location or the orientation of the virtual screen due to the manipulation of the virtual controller, the illumination source contributing to illumination of the virtual or augmented reality environment as a computer screen displaying a same video content as being displayed on the virtual screen would illuminate a real environment.
19. The non-transitory computer-readable medium of claim 18 , wherein the method further comprises:
changing one or more of a curvature, a position, and an orientation of the virtual screen, the changing of the position of the virtual screen being based on movement of the hand-held controller as if the virtual controller and the virtual screen are connected by the rigid link.
20. The non-transitory computer-readable medium of claim 18 , wherein the method further comprises adding a sound source to the virtual or augmented reality environment when the content includes an audio component, using the virtual screen as an illumination source for one or more other virtual objects within the virtual or augmented reality environment, or both.
US15/341,812
2016-07-26
2016-11-02
System and method for displaying computer-based content in a virtual or augmented environment
Active
US10489978B2
( en )
Priority Applications (1)
Application Number
Priority Date
Filing Date
Title
US15/341,812
US10489978B2
( en )
2016-07-26
2016-11-02
System and method for displaying computer-based content in a virtual or augmented environment
Applications Claiming Priority (2)
Application Number
Priority Date
Filing Date
Title
US201662366860P
2016-07-26
2016-07-26
US15/341,812
US10489978B2
( en )
2016-07-26
2016-11-02
System and method for displaying computer-based content in a virtual or augmented environment
Publications (2)
Publication Number
Publication Date
US20180033204A1
US20180033204A1 ( en )
2018-02-01
US10489978B2
true
US10489978B2 ( en )
2019-11-26
Family
ID=61009796
Family Applications (1)
Application Number
Title
Priority Date
Filing Date
US15/341,812
Active
US10489978B2
( en )
2016-07-26
2016-11-02
System and method for displaying computer-based content in a virtual or augmented environment
Country Status (1)
Country
Link
US
( 1 )
US10489978B2
( en )
Cited By (6)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US10930075B2
( en )
*
2017-10-16
2021-02-23
Microsoft Technology Licensing, Llc
User interface discovery and interaction for three-dimensional virtual environments
US11422669B1
( en )
*
2019-06-07
2022-08-23
Facebook Technologies, Llc
Detecting input using a stylus in artificial reality systems based on a stylus movement after a stylus selection action
US20220375171A1
( en )
*
2020-07-27
2022-11-24
Meta Platforms Technologies, Llc
Systems and Methods for User Interaction with Artificial Reality Environments
US12099693B2
( en )
2019-06-07
2024-09-24
Meta Platforms Technologies, Llc
Detecting input in artificial reality systems based on a pinch and pull gesture
US12572213B2
( en )
2022-02-11
2026-03-10
Meta Platforms Technologies, Llc
Systems and methods for navigating within an artificial reality environment based on a distance between points of an input gesture
US12619316B1
( en )
2022-04-08
2026-05-05
Meta Platforms Technologies, Llc
Ultra low friction gestural interface for artificial reality
Families Citing this family (37)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
EP3239811B1
( en )
*
2016-04-29
2021-11-03
Nokia Technologies Oy
A method, apparatus or computer program for user control of access to displayed content
US10198874B2
( en )
*
2016-05-13
2019-02-05
Google Llc
Methods and apparatus to align components in virtual reality environments
WO2018026893A1
( en )
*
2016-08-03
2018-02-08
Google Llc
Methods and systems for determining positional data for three-dimensional interactions inside virtual reality environments
US11507216B2
( en )
*
2016-12-23
2022-11-22
Realwear, Inc.
Customizing user interfaces of binary applications
US10620910B2
( en )
2016-12-23
2020-04-14
Realwear, Inc.
Hands-free navigation of touch-based operating systems
US11099716B2
( en )
2016-12-23
2021-08-24
Realwear, Inc.
Context based content navigation for wearable display
US10564800B2
( en )
2017-02-23
2020-02-18
Spatialand Inc.
Method and apparatus for tool selection and operation in a computer-generated environment
US11086396B2
( en )
*
2017-03-31
2021-08-10
Sony Interactive Entertainment LLC
Depth-keying of web content
JP7141410B2
( en )
2017-05-01
2022-09-22
ãã¸ã㯠ãªã¼ãï¼ ã¤ã³ã³ã¼ãã¬ã¤ããã
Matching Content to Spatial 3D Environments
JP6257826B1
( en )
*
2017-05-31
2018-01-10
æ ªå¼ä¼ç¤¾ã³ããã©
Method, program, and information processing apparatus executed by computer to provide virtual space
US10216333B2
( en )
*
2017-06-30
2019-02-26
Microsoft Technology Licensing, Llc
Phase error compensation in single correlator systems
US10748340B1
( en )
*
2017-07-31
2020-08-18
Apple Inc.
Electronic device with coordinated camera and display operation
JP7196179B2
( en )
2017-12-22
2022-12-26
ãã¸ã㯠ãªã¼ãï¼ ã¤ã³ã³ã¼ãã¬ã¤ããã
Method and system for managing and displaying virtual content in a mixed reality system
US11188144B2
( en )
*
2018-01-05
2021-11-30
Samsung Electronics Co., Ltd.
Method and apparatus to navigate a virtual content displayed by a virtual reality (VR) device
JP7042644B2
( en )
*
2018-02-15
2022-03-28
æ ªå¼ä¼ç¤¾ã½ãã¼ã»ã¤ã³ã¿ã©ã¯ãã£ãã¨ã³ã¿ãã¤ã³ã¡ã³ã
Information processing equipment, image generation method and computer program
IL276621B2
( en )
2018-02-22
2024-05-01
Magic Leap Inc
A browser for mixed reality systems
AU2019225931A1
( en )
2018-02-22
2020-08-20
Magic Leap, Inc.
Object creation with physical manipulation
JP7155242B2
( en )
*
2018-03-14
2022-10-18
ãã¯ã»ã«æ ªå¼ä¼ç¤¾
Personal digital assistant
US10943402B2
( en )
*
2018-03-20
2021-03-09
Logitech Europe S.A.
Method and system for mixed reality interaction with peripheral device
CN110554784B
( en )
*
2018-05-31
2023-07-14
广ä¸èæç°å®ç§ææéå ¬å¸
Input method, device, display device and storage medium
WO2020060569A1
( en )
*
2018-09-21
2020-03-26
Practicum Virtual Reality Media, Inc.
System and method for importing a software application into a virtual reality setting
US11386623B2
( en )
2019-04-03
2022-07-12
Magic Leap, Inc.
Methods, systems, and computer program product for managing and displaying webpages in a virtual three-dimensional space with a mixed reality system
CN118981250A
( en )
*
2019-05-29
2024-11-19
è¹æå ¬å¸
Generating content for physical elements
US10890983B2
( en )
2019-06-07
2021-01-12
Facebook Technologies, Llc
Artificial reality system having a sliding menu
CN111459266A
( en )
*
2020-03-02
2020-07-28
éåºç±å¥èºæºè½ç§ææéå ¬å¸
A method and apparatus for operating a 2D application in a 3D scene of virtual reality
CN114461057B
( en )
*
2020-11-09
2025-07-01
å为ææ¯æéå ¬å¸
VR display control method, electronic device and computer-readable storage medium
EP4258085A1
( en )
*
2022-04-04
2023-10-11
Deutsche Telekom AG
Control of a cursor when using virtual screens
US12284698B2
( en )
2022-07-20
2025-04-22
Snap Inc.
Secure peer-to-peer connections between mobile devices
US12472435B2
( en )
*
2022-08-12
2025-11-18
Snap Inc.
External controller for an eyewear device
US12236512B2
( en )
2022-08-23
2025-02-25
Snap Inc.
Avatar call on an eyewear device
US12229901B2
( en )
2022-10-05
2025-02-18
Snap Inc.
External screen streaming for an eyewear device
US12400414B2
( en )
2023-02-08
2025-08-26
Meta Platforms Technologies, Llc
Facilitating system user interface (UI) interactions in an artificial reality (XR) environment
US12387449B1
( en )
2023-02-08
2025-08-12
Meta Platforms Technologies, Llc
Facilitating system user interface (UI) interactions in an artificial reality (XR) environment
US20240281070A1
( en )
*
2023-02-16
2024-08-22
Meta Platforms Technologies, Llc
Simultaneous Controller and Touch Interactions
CN116560533A
( en )
*
2023-04-23
2023-08-08
å京åè·³ç½ç»ææ¯æéå ¬å¸
Application window adjustment method, device, device and medium
US12249038B1
( en )
2024-09-17
2025-03-11
Monsarrat, Inc.
Navigating real and virtual worlds with disparate terrains in augmented reality
US12263407B1
( en )
*
2024-09-17
2025-04-01
Monsarrat, Inc.
Real world walking to control superhuman virtual world movement
Citations (38)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US6031519A
( en )
*
1997-12-30
2000-02-29
O'brien; Wayne P.
Holographic direct manipulation interface
US20040046736A1
( en )
*
1997-08-22
2004-03-11
Pryor Timothy R.
Novel man machine interfaces and applications
US20060044265A1
( en )
*
2004-08-27
2006-03-02
Samsung Electronics Co., Ltd.
HMD information apparatus and method of operation thereof
US20070176908A1
( en )
*
2004-04-01
2007-08-02
Power 2B, Inc.
Control apparatus
US20090189974A1
( en )
*
2008-01-23
2009-07-30
Deering Michael F
Systems Using Eye Mounted Displays
US20090189830A1
( en )
*
2008-01-23
2009-07-30
Deering Michael F
Eye Mounted Displays
US20090289956A1
( en )
*
2008-05-22
2009-11-26
Yahoo! Inc.
Virtual billboards
US20100007602A1
( en )
*
2006-10-02
2010-01-14
Pioneer Corporation
Image display device
US20100013767A1
( en )
*
2008-07-18
2010-01-21
Wei Gu
Methods for Controlling Computers and Devices
US20110164029A1
( en )
*
2010-01-05
2011-07-07
Apple Inc.
Working with 3D Objects
US20110285704A1
( en )
*
2010-02-03
2011-11-24
Genyo Takeda
Spatially-correlated multi-display human-machine interface
US20120019645A1
( en )
*
2010-07-23
2012-01-26
Maltz Gregory A
Unitized, Vision-Controlled, Wireless Eyeglasses Transceiver
US20120105312A1
( en )
*
2010-10-29
2012-05-03
Microsoft Corporation
User Input Device
US20120206485A1
( en )
*
2010-02-28
2012-08-16
Osterhout Group, Inc.
Ar glasses with event and sensor triggered user movement control of ar eyepiece facilities
US20120249741A1
( en )
*
2011-03-29
2012-10-04
Giuliano Maciocci
Anchoring virtual images to real world surfaces in augmented reality systems
US20120304059A1
( en )
*
2011-05-24
2012-11-29
Microsoft Corporation
Interactive Build Instructions
US20120314934A1
( en )
*
2011-06-09
2012-12-13
Yusuke Kudo
Information processing device, information processing method and program
US20130093756A1
( en )
*
2008-11-25
2013-04-18
Perceptive Pixel Inc.
Volumetric Data Exploration Using Multi-Point Input Controls
US20130154913A1
( en )
*
2010-12-16
2013-06-20
Siemens Corporation
Systems and methods for a gaze and gesture interface
US20130258207A1
( en )
*
2012-03-29
2013-10-03
Geonwoo KIM
Multimedia device connected to at least one electronic device and controling method thereof
US20130314320A1
( en )
*
2012-05-23
2013-11-28
Jae In HWANG
Method of controlling three-dimensional virtual cursor by using portable electronic device
US20130326364A1
( en )
*
2012-05-31
2013-12-05
Stephen G. Latta
Position relative hologram interactions
US20130328925A1
( en )
*
2012-06-12
2013-12-12
Stephen G. Latta
Object focus in a mixed reality environment
US20140002351A1
( en )
*
2012-07-02
2014-01-02
Sony Computer Entertainment Inc.
Methods and systems for interaction with an expanded information space
US20140015831A1
( en )
*
2012-07-16
2014-01-16
Electronics And Telecommunications Research Institude
Apparatus and method for processing manipulation of 3d virtual object
US20140104156A1
( en )
*
2004-11-02
2014-04-17
Pierre A. Touma
Hand-held wireless electronic device with accelerometer for interacting with a display
US20140347391A1
( en )
*
2013-05-23
2014-11-27
Brian E. Keane
Hologram anchoring and dynamic positioning
US20150143297A1
( en )
*
2011-11-22
2015-05-21
Google Inc.
Input detection for a head mounted device
US20150243082A1
( en )
*
2012-09-27
2015-08-27
Kyocera Corporation
Display device, control system, and control program
US20150363980A1
( en )
*
2014-06-17
2015-12-17
Valorisation-Recherche, Limited Partnership
3d virtual environment interaction system
US20160048204A1
( en )
*
2014-01-23
2016-02-18
Jason Scott
Gaze swipe selection
US20160063762A1
( en )
*
2014-09-03
2016-03-03
Joseph Van Den Heuvel
Management of content in a 3d holographic environment
US9299183B2
( en )
*
2010-07-02
2016-03-29
Zspace, Inc.
Detection of partially obscured objects in three dimensional stereoscopic scenes
US20160225192A1
( en )
*
2015-02-03
2016-08-04
Thales USA, Inc.
Surgeon head-mounted display apparatuses
US20170185261A1
( en )
*
2015-12-28
2017-06-29
Htc Corporation
Virtual reality device, method for virtual reality
US20170329419A1
( en )
*
2016-05-11
2017-11-16
Google Inc.
Combining gyromouse input and touch input for navigation in an augmented and/or virtual reality environment
US20170364198A1
( en )
*
2016-06-21
2017-12-21
Samsung Electronics Co., Ltd.
Remote hover touch system and method
US20180005438A1
( en )
*
2016-06-30
2018-01-04
Cheyne Rory Quin Mathey-Owens
Adaptive smoothing based on user focus on a target object
2016
2016-11-02
US
US15/341,812
patent/US10489978B2/en
active
Active
Patent Citations (38)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US20040046736A1
( en )
*
1997-08-22
2004-03-11
Pryor Timothy R.
Novel man machine interfaces and applications
US6031519A
( en )
*
1997-12-30
2000-02-29
O'brien; Wayne P.
Holographic direct manipulation interface
US20070176908A1
( en )
*
2004-04-01
2007-08-02
Power 2B, Inc.
Control apparatus
US20060044265A1
( en )
*
2004-08-27
2006-03-02
Samsung Electronics Co., Ltd.
HMD information apparatus and method of operation thereof
US20140104156A1
( en )
*
2004-11-02
2014-04-17
Pierre A. Touma
Hand-held wireless electronic device with accelerometer for interacting with a display
US20100007602A1
( en )
*
2006-10-02
2010-01-14
Pioneer Corporation
Image display device
US20090189974A1
( en )
*
2008-01-23
2009-07-30
Deering Michael F
Systems Using Eye Mounted Displays
US20090189830A1
( en )
*
2008-01-23
2009-07-30
Deering Michael F
Eye Mounted Displays
US20090289956A1
( en )
*
2008-05-22
2009-11-26
Yahoo! Inc.
Virtual billboards
US20100013767A1
( en )
*
2008-07-18
2010-01-21
Wei Gu
Methods for Controlling Computers and Devices
US20130093756A1
( en )
*
2008-11-25
2013-04-18
Perceptive Pixel Inc.
Volumetric Data Exploration Using Multi-Point Input Controls
US20110164029A1
( en )
*
2010-01-05
2011-07-07
Apple Inc.
Working with 3D Objects
US20110285704A1
( en )
*
2010-02-03
2011-11-24
Genyo Takeda
Spatially-correlated multi-display human-machine interface
US20120206485A1
( en )
*
2010-02-28
2012-08-16
Osterhout Group, Inc.
Ar glasses with event and sensor triggered user movement control of ar eyepiece facilities
US9299183B2
( en )
*
2010-07-02
2016-03-29
Zspace, Inc.
Detection of partially obscured objects in three dimensional stereoscopic scenes
US20120019645A1
( en )
*
2010-07-23
2012-01-26
Maltz Gregory A
Unitized, Vision-Controlled, Wireless Eyeglasses Transceiver
US20120105312A1
( en )
*
2010-10-29
2012-05-03
Microsoft Corporation
User Input Device
US20130154913A1
( en )
*
2010-12-16
2013-06-20
Siemens Corporation
Systems and methods for a gaze and gesture interface
US20120249741A1
( en )
*
2011-03-29
2012-10-04
Giuliano Maciocci
Anchoring virtual images to real world surfaces in augmented reality systems
US20120304059A1
( en )
*
2011-05-24
2012-11-29
Microsoft Corporation
Interactive Build Instructions
US20120314934A1
( en )
*
2011-06-09
2012-12-13
Yusuke Kudo
Information processing device, information processing method and program
US20150143297A1
( en )
*
2011-11-22
2015-05-21
Google Inc.
Input detection for a head mounted device
US20130258207A1
( en )
*
2012-03-29
2013-10-03
Geonwoo KIM
Multimedia device connected to at least one electronic device and controling method thereof
US20130314320A1
( en )
*
2012-05-23
2013-11-28
Jae In HWANG
Method of controlling three-dimensional virtual cursor by using portable electronic device
US20130326364A1
( en )
*
2012-05-31
2013-12-05
Stephen G. Latta
Position relative hologram interactions
US20130328925A1
( en )
*
2012-06-12
2013-12-12
Stephen G. Latta
Object focus in a mixed reality environment
US20140002351A1
( en )
*