ABSTRACT
Abstract
Examples are disclosed herein that relate to receiving virtual reality input. An example provides a head-mounted display device comprising a sensor system, a display, a logic machine, and a storage machine holding instructions executable by the logic machine. The instructions are executable to execute a 3D virtual reality experience on the head-mounted display device, track, via the sensor system, a touch-sensitive input device, render, on the display, in a 3D location in the 3D virtual reality experience based on the tracking of the touch-sensitive input device, a user interface, receive, via a touch sensor of the touch-sensitive input device, a user input, and, in response to receiving the user input, control the 3D virtual reality experience to thereby vary visual content being rendered on the display.
Description
BACKGROUND
In a virtual reality experience, virtual imagery is displayed in a user's field-of-view to provide a life-like, convincing feeling of occupying the experience. The virtual reality experience may allow a user to interact with virtual objects in the experience. Various input devices for facilitating such interaction exist, such as hand-held controllers that provide translational and rotational input.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 shows an example use environment.
FIG. 2 illustrates an example rendering of a user interface including a magnified representation of visual content.
FIG. 3 illustrates an example rendering of multiple user interfaces.
FIG. 4 shows an example user interface configured to facilitate rendering and/or editing of visual content.
FIG. 5 shows an example user interface in which visual content in the form of a 3D model is rendered.
FIG. 6 shows an example representation of a virtual reality experience.
FIG. 7 shows a flowchart illustrating an example method of receiving virtual reality input on an HMD device.
FIG. 8 shows aspects of an example virtual-reality computing system.
FIG. 9 shows a block diagram of an example computing system.
DETAILED DESCRIPTION
In a virtual reality experience, virtual imagery is displayed in a user's field-of-view to provide a life-like, convincing feeling of occupying the experience in an immersive way. The virtual reality experience may allow a user to interact with virtual objects in the experience. Various input devices for facilitating such interaction exist, such as hand-held controllers that provide translational and rotational input.
Hand-held controllers can present various drawbacks when used to provide interaction in a virtual reality experience. For example, some controllers may be unable to provide other types of input beyond translation and rotation. Further, such controllers are often designed specifically for virtual reality experiences and thus tend to be unfamiliar to users as they are becoming acquainted with virtual reality technology. Gaining virtual reality experience may also require acclimation to the tactile feel of a controller for a user to be aware of its actuation, as visual feedback indicating such actuation may be unavailable due to the controller's lack of display in the virtual reality experience. In other virtual reality experiences, hand gestures may facilitate interaction. Such experiences, however, may require gesture detection by a dedicated sensor system, and may suffer from gesture ambiguity and a limited variety of possible gestures.
Accordingly, implementations are disclosed herein that relate to receiving user input from a touch-sensitive input device in a virtual reality experience. In some examples, the touch-sensitive input device may enable the reception of touch input, which may be an intuitive and familiar input paradigm to a user (e.g., as a result of repeated, daily use of a smart-phone). Touch and/or other types of input may extend the range of inputs that can be supplied beyond translation and rotation enabled by the hand-held controllers described above. The input device itself may also provide a familiar paradigm of interaction, as in some examples the input device may be, as referenced above, a smart/phone or other mobile device of the user. Moreover, the virtual reality experience may complement and/or extend input device functionality. As described in further detail below, the virtual reality experience may render a representation of the input device that visually indicates the device and also provides a user interface for interacting with the device and virtual reality experience. In some examples, the user interface may provide additional information and points of interaction beyond a user interface rendered by the input device itself.
FIG. 1 shows an example use environment 100 in which a user 102 is engaged in a three-dimensional (3D) virtual reality experience executed on a head-mounted display (HMD) device 104 . As part of the virtual reality experience, virtual content 108 is rendered on a display of HMD device 104 in a region 108 , which may be greater than, less than, or coextensive with the user's field-of-view. Aspects of user 102 may be tracked also as part of the virtual reality experience. For example, head motion and/or a gaze direction of user 102 may be tracked so that virtual content 106 is rendered responsive to changes in user position and/or orientation to provide an immersive and convincing virtual reality experience that reflects changes to the user's perspective.
A touch- sensitive input device 110 may facilitate interaction with the virtual reality experience. Input device 110 includes a touch sensor configured to receive user input, which may be provided to HMD device 104 and used to control the 3D virtual reality experience by varying visual content 106 rendered on the HMD display. User input received by the touch sensor may include touch, hover, and/or hand gesture input, for example, and may be sensed via any suitable touch sensing technology (e.g., capacitive, resistive, acoustic, optical). The user input may influence the rendering of visual content 106 in various ways, examples of which are depicted below with reference to FIGS. 2-6 . Briefly, and without limitation, input device 110 may enable one or more of (1) drawing of visual content, which may be displayed on HMD device 104 as two-dimensional (2D) or 3D imagery, (2) editing and/or manipulation of visual content rendered on the HMD device, and (3) display and selection of user interface controls that affect visual content, among other potential functions.
HMD device 104 may track touch- sensitive input device 110 and render a user interface 112 based on the tracking of the input device. User interface 112 may include a representation of input device 110 (e.g., an image of a user's smartphone). In the example depicted in FIG. 1 , user interface 112 includes a display portion 113 and a body portion 114 that are rendered in 3D locations in the virtual reality experience that respectively correspond to the real-world locations of a display and a body of input device 110 . Were user 102 to view input device 110 in the real, physical environment held at the location represented in FIG. 1 , the display and body would appear substantially coextensive with display portion 113 and body portion 114 in the virtual reality experience. As such, and by potentially rendering display portion 113 and body portion 114 with substantially the same apparent size and angular orientation as their real-world counterparts, the portions may mimic or simulate the real-world appearance of input device 110 . Thus, user input device 110 may be represented in a convincing and familiar manner, appearing to positionally occupy the virtual reality experience with functionality that the user is familiar with. Despite the occlusion of the real-world appearance of input device 110 by the virtual reality experience, user 102 may retain the ability to visually locate the input device via its representation, where the user may view information presented by user interface 112 and/or provide input to the input device. However, HMD device 104 may render any suitable user interface and/or representation of input device 110 , further examples of which are described below. Other representations may be rendered as well, such as a representation of the hands of user 102 .
HMD device 104 includes a sensor system 116 for tracking touch- sensitive input device 110 . In some examples, a communications subsystem may at least partially implement sensor system 116 by receiving signals from input device 110 that enable tracking of the input device. The signals may be received via a wired or wireless connection to input device 110 , which may communicatively couple to HMD device 104 via a direct connection or indirectly through a suitable network. The signals may include any suitable information enabling tracking of input device 110 , such as output from one or more inertial measurement units (e.g., accelerometer, gyroscope, magnetometer) in the input device, a global positioning system (GPS) sensor in the input device, etc. In some examples, the signals may indicate the position (e.g., in the form of a three-dimensional coordinate) and/or orientation (e.g., in the form of a three-dimensional rotational coordinate) of input device 110 . Alternatively or in addition to receiving tracking signals from input device 110 , sensor system 116 may include one or more optical sensors for tracking the input device. For example, sensor system 116 may employ a visible light and/or depth camera to locate and segment image data corresponding to <figure-callout id="110" label="input device" filenames
BACKGROUND
In a virtual reality experience, virtual imagery is displayed in a user's field-of-view to provide a life-like, convincing feeling of occupying the experience. The virtual reality experience may allow a user to interact with virtual objects in the experience. Various input devices for facilitating such interaction exist, such as hand-held controllers that provide translational and rotational input.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 shows an example use environment.
FIG. 2 illustrates an example rendering of a user interface including a magnified representation of visual content.
FIG. 3 illustrates an example rendering of multiple user interfaces.
FIG. 4 shows an example user interface configured to facilitate rendering and/or editing of visual content.
FIG. 5 shows an example user interface in which visual content in the form of a 3D model is rendered.
FIG. 6 shows an example representation of a virtual reality experience.
FIG. 7 shows a flowchart illustrating an example method of receiving virtual reality input on an HMD device.
FIG. 8 shows aspects of an example virtual-reality computing system.
FIG. 9 shows a block diagram of an example computing system.
DETAILED DESCRIPTION
In a virtual reality experience, virtual imagery is displayed in a user's field-of-view to provide a life-like, convincing feeling of occupying the experience in an immersive way. The virtual reality experience may allow a user to interact with virtual objects in the experience. Various input devices for facilitating such interaction exist, such as hand-held controllers that provide translational and rotational input.
Hand-held controllers can present various drawbacks when used to provide interaction in a virtual reality experience. For example, some controllers may be unable to provide other types of input beyond translation and rotation. Further, such controllers are often designed specifically for virtual reality experiences and thus tend to be unfamiliar to users as they are becoming acquainted with virtual reality technology. Gaining virtual reality experience may also require acclimation to the tactile feel of a controller for a user to be aware of its actuation, as visual feedback indicating such actuation may be unavailable due to the controller's lack of display in the virtual reality experience. In other virtual reality experiences, hand gestures may facilitate interaction. Such experiences, however, may require gesture detection by a dedicated sensor system, and may suffer from gesture ambiguity and a limited variety of possible gestures.
Accordingly, implementations are disclosed herein that relate to receiving user input from a touch-sensitive input device in a virtual reality experience. In some examples, the touch-sensitive input device may enable the reception of touch input, which may be an intuitive and familiar input paradigm to a user (e.g., as a result of repeated, daily use of a smart-phone). Touch and/or other types of input may extend the range of inputs that can be supplied beyond translation and rotation enabled by the hand-held controllers described above. The input device itself may also provide a familiar paradigm of interaction, as in some examples the input device may be, as referenced above, a smart/phone or other mobile device of the user. Moreover, the virtual reality experience may complement and/or extend input device functionality. As described in further detail below, the virtual reality experience may render a representation of the input device that visually indicates the device and also provides a user interface for interacting with the device and virtual reality experience. In some examples, the user interface may provide additional information and points of interaction beyond a user interface rendered by the input device itself.
FIG. 1 shows an example use environment 100 in which a user 102 is engaged in a three-dimensional (3D) virtual reality experience executed on a head-mounted display (HMD) device 104 . As part of the virtual reality experience, virtual content 108 is rendered on a display of HMD device 104 in a region 108 , which may be greater than, less than, or coextensive with the user's field-of-view. Aspects of user 102 may be tracked also as part of the virtual reality experience. For example, head motion and/or a gaze direction of user 102 may be tracked so that virtual content 106 is rendered responsive to changes in user position and/or orientation to provide an immersive and convincing virtual reality experience that reflects changes to the user's perspective.
A touch- sensitive input device 110 may facilitate interaction with the virtual reality experience. Input device 110 includes a touch sensor configured to receive user input, which may be provided to HMD device 104 and used to control the 3D virtual reality experience by varying visual content 106 rendered on the HMD display. User input received by the touch sensor may include touch, hover, and/or hand gesture input, for example, and may be sensed via any suitable touch sensing technology (e.g., capacitive, resistive, acoustic, optical). The user input may influence the rendering of visual content 106 in various ways, examples of which are depicted below with reference to FIGS. 2-6 . Briefly, and without limitation, input device 110 may enable one or more of (1) drawing of visual content, which may be displayed on HMD device 104 as two-dimensional (2D) or 3D imagery, (2) editing and/or manipulation of visual content rendered on the HMD device, and (3) display and selection of user interface controls that affect visual content, among other potential functions.
HMD device 104 may track touch- sensitive input device 110 and render a user interface 112 based on the tracking of the input device. User interface 112 may include a representation of input device 110 (e.g., an image of a user's smartphone). In the example depicted in FIG. 1 , user interface 112 includes a display portion 113 and a body portion 114 that are rendered in 3D locations in the virtual reality experience that respectively correspond to the real-world locations of a display and a body of input device 110 . Were user 102 to view input device 110 in the real, physical environment held at the location represented in FIG. 1 , the display and body would appear substantially coextensive with display portion 113 and body portion 114 in the virtual reality experience. As such, and by potentially rendering display portion 113 and body portion 114 with substantially the same apparent size and angular orientation as their real-world counterparts, the portions may mimic or simulate the real-world appearance of input device 110 . Thus, user input device 110 may be represented in a convincing and familiar manner, appearing to positionally occupy the virtual reality experience with functionality that the user is familiar with. Despite the occlusion of the real-world appearance of input device 110 by the virtual reality experience, user 102 may retain the ability to visually locate the input device via its representation, where the user may view information presented by user interface 112 and/or provide input to the input device. However, HMD device 104 may render any suitable user interface and/or representation of input device 110 , further examples of which are described below. Other representations may be rendered as well, such as a representation of the hands of user 102 .
HMD device 104 includes a sensor system 116 for tracking touch- sensitive input device 110 . In some examples, a communications subsystem may at least partially implement sensor system 116 by receiving signals from input device 110 that enable tracking of the input device. The signals may be received via a wired or wireless connection to input device 110 , which may communicatively couple to HMD device 104 via a direct connection or indirectly through a suitable network. The signals may include any suitable information enabling tracking of input device 110 , such as output from one or more inertial measurement units (e.g., accelerometer, gyroscope, magnetometer) in the input device, a global positioning system (GPS) sensor in the input device, etc. In some examples, the signals may indicate the position (e.g., in the form of a three-dimensional coordinate) and/or orientation (e.g., in the form of a three-dimensional rotational coordinate) of input device 110 . Alternatively or in addition to receiving tracking signals from input device 110 , sensor system 116 may include one or more optical sensors for tracking the input device. For example, sensor system 116 may employ a visible light and/or depth camera to locate and segment image data corresponding to input device 110 from other image data. Additional detail regarding example hardware configurations of HMD device 104 is described below with reference to FIGS. 8 and 9 .
Various modifications to the example configuration depicted in FIG. 1 are contemplated. For example, HMD device 104 may alternatively take the form of a mixed reality display device including an at least partially transparent display with which virtual, rendered imagery is overlaid on real, background imagery corresponding to the surrounding physical environment. As such, âvirtual realityâ and âvirtual reality experienceâ as used herein may also refer to mixed reality and mixed reality experiences, and one or more of the examples described herein that relate to the use of touch- sensitive input device 110 for interacting with a virtual reality experience may be adapted to a mixed reality experience. Further, the approaches described herein may be implemented on display device types other than a head-mounted display device, such as a non-head-mounted head-up display. Similarly, while shown in the form of a mobile device such as a smartphone, tablet, etc., input device 110 may assume other forms, including but not limited to that of a game console, controller, appliance, etc.
In some examples, user interface 112 may include a representation of visual content rendered on touch- sensitive input device 110 . For example, display portion 113 may substantially reproduce the visual content rendered on the display of input device 110 , enabling user 102 to bring input device functionality into the virtual reality experience, such as phone conversations and software applications. To this end, input device 110 may transmit the contents of a display buffer (e.g., in compressed or uncompressed form), and/or other data enabling HMD device 104 to render the visual content, to the HMD device. In these examples, the display of input device 110 may be turned off while its visual content is rendered on HMD device 104 . In other examples, HMD device 104 may capture, via one or more optical sensors of the sensor system, the visual content rendered on the display of input device 110 and make use of it to influence the experience being provided by the HMD. Further, HMD device 104 may process visual content rendered on input device 110 in representing the visual content to reduce issues associated with a potential mismatch between the display resolution of the HMD device and that of the input device. Generally, HMD device 104 may generate a representation of any suitable output from input device 110 . As another example, HMD device 104 may reproduce audio received, or output recorded, from input device 110 for implementations in which the HMD device includes an audio output device (e.g., speaker, headphones).
In some examples, a representation of visual content rendered on touch- sensitive input device 110 may be magnified. As an example of such magnification, FIG. 2 illustrates the rendering of a user interface 202 in region 108 including a magnified representation 204 of visual content rendered on input device 110 . Representation 204 magnifies a portion 206 of the visual content rendered on input device 110 , whose real-world appearance (were the display active) is illustrated in FIG. 2 . In other examples, representation 204 may magnify the entirety of the visual content. Further, representation 204 is rendered in a 3D location proximate to input device 110 where the representation appears to float over the input device, while preserving the visibility of display portion 113 and body portion 114 . In this way, the visibility of the visual content rendered on input device 110 is increased while continuing to visually apprise a user of the location of the input device via its virtual representation, which may allow the user to continue interacting with the input device via touch input without disorientation from occlusion by representation 204 . As with user interface 112 , rendering representation 204 in a 3D location that corresponds to the real-world location of input device 110 may facilitate intuitive interaction with the input device and provide a visual cue with which input can be applied to the input device.
User input may control aspects of the display of representation 204 in various manners. For example, representation 204 may be scrolled to change the visual content shown therein based on a gaze direction of a user. To this end, the sensor system of HMD device 104 may include one or more gaze sensors configured to detect the gaze direction of the user. As another example, the sensor system of HMD device 104 may detect (e.g., via an outward-facing image sensor) a hand gesture that effects scrolling of representation 204 . Similarly, gaze, gestural, and/or other inputs may effect or cease the display itself of representation 204 .
HMD device 104 may render multiple user interfaces based on tracking touch- sensitive input device 110 . As an example, FIG. 3 illustrates the rendering of a user interface 302 in combination with an additional user interface 304 in region 108 . In contrast to user interface 302 , which represents input device 110 via display portion 113 and body portion 114 in 3D locations respectively corresponding to the real-world locations of the display and body of the input device, user interface 304 includes two panels
306 A and 306 B rendered away from the real-world location of the input device display and body. As shown in FIG. 3 , panels
306 A and 306 B are rendered laterally proximate to the real-world location of the input device display, though the panels may be rendered at other locations spaced away from the real-world location. As such, rendering a user interface âaway fromâ the real-world location of the display of input device 110 as used herein may refer to a variety of locations that do not correspond directly to the real-world locationâfor example, any 3D location in region 108 that does not correspond to the real-world location. FIG. 3 also illustrates how panels
306 A and 306 B may be rendered in a manner that they appear locked to the real-world location of the display of input device 110 ; as this real-world location changes, so too may the 3D locations of the panels change accordingly to appear spatially locked to the input device display. Considered from another perspective, HMD device 104 may render a user interface with multiple different portions/sections. As such, an âadditionalâ user interface as used herein may also refer to a user interface including multiple portions/sections.
In some examples, additional user interface 304 may extend/augment the functionality provided by user interface 302 , whose real-world appearance as rendered on touch- sensitive input device 110 is shown (were the display active). For example, additional user interface 304 may include selectable controls not shown in user interface 302 that may relate to the content shown in user interface 302 . As one specific example, user interface 302 may display information regarding a purchasable productâe.g., as part of a storefront application executed on HMD device 104 . User interface 304 may enhance the shopping experience by presenting, in panel 306 A, a listing of related items relevant to the product displayed in user interface 302 , and in panel 306 E, a comparison of specifications of the product to those of another related item. As another example, user interface 304 may present a settings menu including controls selectable to change settings of an application displayed in user interface 302 .
As yet another example of extending/augmenting the functionality provided by user interface 302 , user interface 304 may present information regarding a contact with whom a phone or video call is conducted in user interface 302 . As still another example, user interface 304 may present information regarding participants in an online game conducted in user interface 302 . As further examples, user interface 304 may present instructions (e.g., regarding an application displayed in user interface 302 ), and/or page(s) of a book (e.g., immediately previous and subsequent pages in panels 308 A and 306 B, respectively, relative to a current page displayed in user interface 302 ). Generally, user interface 304 may present any suitable information relating to an interaction carried out through user interface 302 , and/or information unrelated to such an interaction.
As HMD device 104 may render user interface 304 away from touch- sensitive input device 110 , the input device may be unable to detect user input applied to the additional interface via its touch sensor. As such, HMD device 104 may defect such user inputâe.g., by detecting the intersection of a user gaze direction with a selectable portion of user interface 304 , and/or by defecting a hand gesture applied to the user additional interface. In some examples, the detection of user input applied to user interface 304 by HMD device 104 may be less accurate than the detection of user input applied to user interface 302 by input device 110 . As such, user interface controls for which interaction benefits from higher detection accuracy may be allocated to user interface 302 so that these interactions are detected via the touch sensor of input device 110 .
In some examples, a user interface may facilitate the rendering of new, and/or editing of existing, visual content on HMD device 104 . To this end, FIG. 4 shows a user interface 400 configured to facilitate such rendering and/or editing of visual content. Specifically, FIG. 4 shows the real-world appearance of user interface 400 as rendered on the display of touch-sensitive input device 110 (were the display active) as well as its appearance in the virtual reality experience when represented therein by display portion 113 .
User interface 400 includes one or more controls for rendering new visual content. For example, user interface 400 may include a canvas portion 402 in which a user can draw 2D or 3D shapes and images via the application of touch input. FIG. 4 shows one such possible shape 404 drawn in canvas portion 402 , which is rendered in region 108 on HMD device 104 . Other controls may affect the rendering of visual content. As examples, FIG. 4 shows the inclusion in user interface 400 of brush controls 406 , for changing a stroke or pattern that is rendered in response to input applied to canvas portion 402 ; thickness controls 408 , for changing the thickness of the selected stroke/pattern; and a color control 410 in the form of a color wheel for selecting the color of the selected stroke/pattern. As shown, HMD device 104 may render an icon 412 in region 108 representing the selected stroke/pattern at 3D location corresponding to the last location where input was applied in canvas portion 402 .
It will thus be understood that user interface 400 may function as a palette of visual content rendering controls. The controls shown in FIG. 4 are provided as examples, and user interface 400 may include alternative and/or additional controls not shown. Further, user interface 400 may be deployed in any suitable context. In addition to its use in a 2D or 3D drawing/rendering application illustrated in FIG. 4 , user interface 400 may provide input to other applications and contexts, such as enabling (e.g., by receiving input at canvas portion 402 ) a user to draw a signature, which may be used to sign a document or illustration, authorize a transaction, or verify an identity, for example.
As described above, shape 404 may represent a 2D or 3D input applied to canvas portion 402 . Various approaches may enable the reception of 3D input at canvas portion 402 . When touch- sensitive input device 110 is operable to detect a z-coordinate of touch input into/out of the input deviceâe.g., as a force or pressureâthe user may control such force/pressure to vary the z-coordinate of input applied to canvas portion 402 . As a particular example, the z-coordinate may be varied as shape 404 is traced along canvas portion 402 so that the shape is rendered with a corresponding variation in apparent depth in region 108 . In other examples, gaze and/or hand gesture input may control the z-coordinates of shapes rendered in region 108 .
Alternatively or in addition to the use of canvas portion 402 for drawing visual content that is rendered in region 108 , movement of touch- sensitive input device 110 may effect the rendering of visual content. For example, HMD device 104 may render shapes in correspondence with the translation of input device 110 throughout real, physical space. Such an approach may enable an intuitive mechanism with which 2D and 3D shapes are formed. For these examples in which movement of input device 110 is used to draw shapes, the representation of input device 110 in region 108 shown in FIG. 4 âe.g., portions
113 and 114 âmay instead be rendered at the 3D location corresponding to the last location in physical space where drawing was performedâe.g., at the 3D location occupied by icon 412 , and in lieu of the icon. User interface 400 may continue to provide controls as described herein for varying visual content rendered in response to movement of input device 110 , though in some examples HMD device 104 may render the user interface away from the input device (e.g., at a fixed location in region 108 ).
HMD device 104 may render representations of visual content received from touch- sensitive input device 110 in other ways. As an example, FIG. 5 shows a user interface 500 in which visual content in the form of a 3D model 502 is rendered. Specifically, FIG. 5 shows the real-world appearance of user interface 500 as rendered on the display of touch-sensitive input device 110 (were the display active) as well as its appearance in the virtual reality experience when represented therein by display portion 113 .
In the depicted example, 3D model 502 is initially accessed on touch- sensitive input device 110 . A user may then transfer 3D model 502 (e.g., if stored on input device 110 ) to HMD device 104 , or cause access to the 3D model by the HMD device (e.g., if the model is stored elsewhere such as a remote network location), via any suitable input, such as a hand gesture in which the user swipes from a location proximate to the input device into the virtual reality experience. HMD device 104 then renders a representation of 3D model 502 in region 108 at a 3D location that, in some examples, may be a function of the location of input device 110 âe.g., a virtual location of the input device in the virtual reality experience. Further, HMD device 104 may render a coordinate axes control 504 representing the 3D location of 3D model 502 . Control 504 may be manipulated (e.g., via a pointing or grasping hand gesture) to vary the 3D location of 3D model 502 in region 108 .
HMD device 104 may receive user input causing an edit to the representation of 3D model 502 , and in response render the edited representation in region 108 . HMD device 104 may then transfer the edited representation to <figure-callout id="110" label="input device"
CLAIMS
Claims ( 13 )
The invention claimed is:
1. A head-mounted display device, comprising:
a sensor system;
a display;
a logic machine; and
a storage machine holding instructions executable by the logic machine to:
execute a 3D virtual reality experience on the head-mounted display device;
track, via the sensor system, a touch-sensitive input device;
render a user interface on the display, the user interface being rendered in a 3D location in the 3D virtual reality experience based on the tracking of the touch-sensitive input device;
receive, via a touch sensor of the touch-sensitive input device, a user input;
in response to receiving the user input, control the 3D virtual reality experience to thereby vary visual content being rendered on the display by rendering a representation of visual content received from the touch-sensitive input device;
receive an additional user input causing an edit to the representation;
render the edited representation on the display; and
transmit the edited representation to the touch-sensitive input device.
2. The head-mounted display device of claim 1 , where the user interface includes a representation of the touch-sensitive input device.
3. The head-mounted display device of claim 1 , where the instructions are further executable to map a location of the user input on the touch-sensitive input device to a location in the user interface.
4. The head-mounted display device of claim 1 , where the 3D location in which the user interface is rendered corresponds to a real-world location of a display of the touch-sensitive input device.
5. The head-mounted display device of claim 4 , where the user interface includes one or more controls, and where the instructions executable to control the 3D virtual reality experience to thereby vary visual content being rendered on the display are executable to render new visual content based on a selected one of the one or more controls.
6. The head-mounted display device of claim 4 , where the instructions are further executable to:
render an additional user interface away from the real-world location of the display of the touch-sensitive input device; and
receive, via the sensor system, a user input applied to the additional user interface.
7. The head-mounted display device of claim 1 , where the instructions are further executable to adjust one or more 3D coordinates of the visual content in the 3D virtual reality experience in response to movement of the touch-sensitive input device.
8. The head-mounted display device of claim 1 , where the instructions executable to control the 3D virtual reality experience to thereby vary visual content being rendered on the display are executable to magnify the representation.
9. The head-mounted display device of claim 1 , where the instructions executable to control the 3D virtual reality experience to thereby vary visual content being rendered on the display are executable to render a representation of a real-world environment at a 3D location in the 3D virtual reality experience corresponding to a real-world location of the touch-sensitive input device.
10. On a head-mounted display device, a method of receiving virtual reality input, comprising:
executing a 3D virtual reality experience on the head-mounted display device;
tracking, via a sensor system of the head-mounted display device, a touch-sensitive input device;
rendering, on a display of the head-mounted display device, in a 3D location in the 3D virtual reality experience based on the tracking of the touch-sensitive input device, a user interface;
receiving, via a touch sensor of the touch-sensitive input device, a user input;
in response to receiving the user input, controlling the 3D virtual reality experience to thereby vary visual content being rendered on the display by rendering a representation of visual content received from the touch-sensitive input device;
receiving an additional user input causing an edit to the representation;
rendering the edited representation on the display; and
transmitting the edited representation to the touch-sensitive input device.
11. The method of claim 10 , where the user interface includes a representation of the touch-sensitive input device.
12. The method of claim 10 , where the user interface includes one or more controls, and where controlling the 3D virtual reality experience to thereby vary visual content being rendered on the display includes rendering new visual content based on a selected one of the one or more controls.
13. A head-mounted display device, comprising:
a sensor system;
a display;
a logic machine; and
a storage machine holding instructions executable by the logic machine to:
execute a 3D virtual reality experience on the head-mounted display device;
track, via the sensor system, a touch-sensitive input device;
render, on the display, in a 3D location in the 3D virtual reality experience based on the tracking of the touch-sensitive input device, a user interface including a representation of the touch-sensitive input device;
receive, via a touch sensor of the touch-sensitive input device, a user input;
in response to receiving the user input, control the 3D virtual reality experience to thereby vary visual content being rendered on the display by rendering a representation of visual content received from the touch-sensitive input device;
receive an additional user input causing an edit to the representation of the visual content;
render the edited representation on the display; and
transmit the edited representation to the touch-sensitive input device.
US15/685,870
2017-08-24
2017-08-24
Virtual reality input
Active
2038-01-06
US10754496B2
( en )
Priority Applications (2)
Application Number
Priority Date
Filing Date
Title
US15/685,870
US10754496B2
( en )
2017-08-24
2017-08-24
Virtual reality input
PCT/US2018/038656
WO2019040166A1
( en )
2017-08-24
2018-06-21
Virtual reality input
Applications Claiming Priority (1)
Application Number
Priority Date
Filing Date
Title
US15/685,870
US10754496B2
( en )
2017-08-24
2017-08-24
Virtual reality input
Publications (2)
Publication Number
Publication Date
US20190065026A1
US20190065026A1 ( en )
2019-02-28
US10754496B2
true
US10754496B2 ( en )
2020-08-25
Family
ID=62916768
Family Applications (1)
Application Number
Title
Priority Date
Filing Date
US15/685,870
Active
2038-01-06
US10754496B2
( en )
2017-08-24
2017-08-24
Virtual reality input
Country Status (2)
Country
Link
US
( 1 )
US10754496B2
( en )
WO
( 1 )
WO2019040166A1
( en )
Cited By (8)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US20230368475A1
( en )
*
2022-05-10
2023-11-16
Apple Inc.
Multi-Device Content Handoff Based on Source Device Position
US11870852B1
( en )
*
2023-03-31
2024-01-09
Meta Platforms Technologies, Llc
Systems and methods for local data transmission
US20240211090A1
( en )
*
2022-12-23
2024-06-27
Rovi Guides, Inc.
Methods and systems for displaying virtual elements in an xr environment
US12058301B2
( en )
2022-06-03
2024-08-06
Apple Inc.
Electronic device that displays virtual objects
US12178641B2
( en )
2022-07-21
2024-12-31
Shanghai United Imaging Intelligence Co., Ltd.
Systems and methods for fetus monitoring
US12353627B2
( en )
2023-03-28
2025-07-08
Samsung Electronics Co., Ltd.
Head-wearable electronic, method, and non-transitory computer readable storage medium for executing function based on identification of contact point and gaze point
US12436607B1
( en )
2024-05-29
2025-10-07
Adeia Guides Inc.
Systems and methods for providing enhanced extended reality interactions using personal electronic devices as input mechanisms
US12619296B2
( en )
2020-09-25
2026-05-05
Apple Inc.
Sensor emulation
Families Citing this family (51)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
CN109511004B
( en )
*
2017-09-14
2023-09-01
ä¸å ´é讯è¡ä»½æéå ¬å¸
A video processing method and device
US11861136B1
( en )
*
2017-09-29
2024-01-02
Apple Inc.
Systems, methods, and graphical user interfaces for interacting with virtual reality environments
JP6582205B2
( en )
*
2018-02-28
2019-10-02
æ ªå¼ä¼ç¤¾ã³ãããã¸ã¿ã«ã¨ã³ã¿ãã¤ã³ã¡ã³ã
Information processing apparatus, information processing apparatus program, head mounted display, and information processing system
US10540015B2
( en )
*
2018-03-26
2020-01-21
Chian Chiu Li
Presenting location related information and implementing a task based on gaze and voice detection
CN112585564B
( en )
2018-06-21
2024-11-29
å¥è·å ¬å¸
Method and apparatus for providing input for head-mounted image display device
US10841174B1
( en )
*
2018-08-06
2020-11-17
Apple Inc.
Electronic device with intuitive control interface
US11032532B2
( en )
*
2018-10-04
2021-06-08
Samsung Electronics Co., Ltd.
Electronic device and method for providing virtual device via at least portion of content
US11294534B2
( en )
*
2018-11-13
2022-04-05
Unbnd Group Pty Ltd
Technology adapted to provide a user interface via presentation of two-dimensional content via three-dimensional display objects rendered in a navigablé virtual space
JP2020129263A
( en )
2019-02-08
2020-08-27
ã»ã¤ã³ã¼ã¨ãã½ã³æ ªå¼ä¼ç¤¾
Display system, program for controlling information processing device, and method of controlling information processing device
US11422669B1
( en )
*
2019-06-07
2022-08-23
Facebook Technologies, Llc
Detecting input using a stylus in artificial reality systems based on a stylus movement after a stylus selection action
US11334212B2
( en )
2019-06-07
2022-05-17
Facebook Technologies, Llc
Detecting input in artificial reality systems based on a pinch and pull gesture
US11816800B2
( en )
2019-07-03
2023-11-14
Apple Inc.
Guided consumer experience
US11775130B2
( en )
*
2019-07-03
2023-10-03
Apple Inc.
Guided retail experience
US20210011556A1
( en )
*
2019-07-09
2021-01-14
Facebook Technologies, Llc
Virtual user interface using a peripheral device in artificial reality environments
US11023036B1
( en )
2019-07-09
2021-06-01
Facebook Technologies, Llc
Virtual drawing surface interaction using a peripheral device in artificial reality environments
US11023035B1
( en )
2019-07-09
2021-06-01
Facebook Technologies, Llc
Virtual pinboard interaction using a peripheral device in artificial reality environments
US11231827B2
( en )
*
2019-08-03
2022-01-25
Qualcomm Incorporated
Computing device and extended reality integration
IL292407B2
( en )
2019-10-25
2024-04-01
Sentiar Inc
Electrogram annotation system
CN112835445B
( en )
*
2019-11-25
2024-04-26
å为ææ¯æéå ¬å¸
Interaction method, device and system in virtual reality scene
US11127380B2
( en )
*
2019-12-13
2021-09-21
Qualcomm Incorporated
Content stabilization for head-mounted displays
WO2021236170A1
( en )
*
2020-05-18
2021-11-25
Google Llc
Low-power semi-passive relative six-degree-of-freedom tracking
US20230222738A1
( en )
*
2020-05-25
2023-07-13
Sony Group Corporation
Information processing apparatus, information processing method, and program
US12360747B1
( en )
2020-07-15
2025-07-15
Apple Inc.
Devices and methods for visual programming
US12436619B1
( en )
*
2020-07-15
2025-10-07
Apple Inc.
Devices and methods for digital assistant
CN111970456B
( en )
*
2020-09-14
2022-01-11
Oppo广ä¸ç§»å¨éä¿¡æéå ¬å¸
Shooting control method, device, equipment and storage medium
US12511037B1
( en )
*
2020-10-26
2025-12-30
Apple Inc.
Method and device for typing visualization
US12482145B2
( en )
2021-07-15
2025-11-25
Qualcomm Incorporated
Remote landmark rendering for extended reality interfaces
CN113703704B
( en )
*
2021-08-26
2024-01-02
æå·çµä¼´ç§ææéå ¬å¸
Interface display method, head-mounted display device and computer-readable medium
CN118043724A
( en )
2021-09-29
2024-05-14
ä¸æçµåæ ªå¼ä¼ç¤¾
Electronic device for providing augmented reality service and operation method thereof
US20240420436A1
( en )
*
2021-10-06
2024-12-19
Korea Advanced Institute Of Science And Technology
Electronic system for performing control over virtual reality space, electronic device, and method for operating same
US11726578B1
( en )
2022-02-11
2023-08-15
Meta Platforms Technologies, Llc
Scrolling and navigation in virtual reality
US12619316B1
( en )
2022-04-08
2026-05-05
Meta Platforms Technologies, Llc
Ultra low friction gestural interface for artificial reality
US12242666B2
( en )
2022-04-08
2025-03-04
Meta Platforms Technologies, Llc
Artificial reality input using multiple modalities
US12093462B2
( en )
2022-04-11
2024-09-17
Meta Platforms Technologies, Llc
Virtual keyboard selections using multiple input modalities
US12299252B2
( en )
*
2022-06-10
2025-05-13
Meta Platforms Technologies, Llc
Systems and methods for coordinating operation of a head-wearable device and an electronic device to assist a user in interacting with the electronic device
US12260145B1
( en )
*
2022-09-09
2025-03-25
Apple Inc.
Transfer of the display of content between devices with animation
US12307600B2
( en )
2022-11-15
2025-05-20
Adobe Inc.
Modifying two-dimensional images utilizing iterative three-dimensional meshes of the two-dimensional images
US12367626B2
( en )
2022-11-15
2025-07-22
Adobe Inc.
Modifying two-dimensional images utilizing three-dimensional meshes of the two-dimensional images
US12469194B2
( en )
2022-10-03
2025-11-11
Adobe Inc.
Generating shadows for placed objects in depth estimated scenes of two-dimensional images
US12482172B2
( en )
2022-10-03
2025-11-25
Adobe Inc.
Generating shadows for objects in two-dimensional images utilizing a plurality of shadow maps
US12277652B2
( en )
2022-11-15
2025-04-15
Adobe Inc.
Modifying two-dimensional images utilizing segmented three-dimensional object meshes of the two-dimensional images
US12347124B2
( en )
2022-11-15
2025-07-01
Adobe Inc.
Generating adaptive three-dimensional meshes of two-dimensional images
US12499574B2
( en )
2022-10-06
2025-12-16
Adobe Inc.
Generating three-dimensional human models representing two-dimensional humans in two-dimensional images
US12394166B2
( en )
2022-10-06
2025-08-19
Adobe Inc.
Modifying poses of two-dimensional humans in two-dimensional images by reposing three-dimensional human models representing the two-dimensional humans
US12530851B2
( en )
*
2022-11-14
2026-01-20
Qualcomm Incorporated
Augmented reality enhanced media
WO2024107554A1
( en )
*
2022-11-14
2024-05-23
Qualcomm Incorporated
Augmented reality enhanced media
US12175619B2
( en )
*
2023-05-11
2024-12-24
Adobe Inc.
Generating and visualizing planar surfaces within a three-dimensional space for modifying objects in a two-dimensional editing interface
US12423855B2
( en )
2023-04-20
2025-09-23
Adobe Inc.
Generating modified two-dimensional images by customizing focal points via three-dimensional representations of the two-dimensional images
WO2024248856A1
( en )
*
2023-05-31
2024-12-05
Google Llc
Multi-modal localization of streaming content providers in a virtual environment
US20240422302A1
( en )
*
2023-06-15
2024-12-19
Apple Inc.
Systems and methods for image capture for extended reality applications using peripheral object
JP2025082903A
( en )
*
2023-11-20
2025-05-30
ãã¤ãã³æ ªå¼ä¼ç¤¾
Information processing device
Citations (22)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US20080074560A1
( en )
*
2006-09-21
2008-03-27
Seiko Epson Corporation
Image display device, image display system, and network connection method
US20100287485A1
( en )
2009-05-06
2010-11-11
Joseph Bertolami
Systems and Methods for Unifying Coordinate Systems in Augmented Reality Applications
US20100328204A1
( en )
*
2009-06-25
2010-12-30
The Boeing Company
Virtual Control Station
US20130141419A1
( en )
2011-12-01
2013-06-06
Brian Mount
Augmented reality with realistic occlusion
US20130265220A1
( en )
*
2012-04-09
2013-10-10
Omek Interactive, Ltd.
System and method for combining three-dimensional tracking with a three-dimensional display for a user interface
US20140248950A1
( en )
*
2013-03-01
2014-09-04
Martin Tosas Bautista
System and method of interaction for mobile devices
US20140267234A1
( en )
2013-03-15
2014-09-18
Anselm Hook
Generation and Sharing Coordinate System Between Users on Mobile
US20140361977A1
( en )
*
2013-06-07
2014-12-11
Sony Computer Entertainment Inc.
Image rendering responsive to user actions in head mounted display
US9066200B1
( en )
2012-05-10
2015-06-23
Longsand Limited
User-generated content in a virtual reality environment
US9218212B2
( en )
2011-11-11
2015-12-22
International Business Machines Corporation
Pairing physical devices to virtual devices to create an immersive environment
US20160027215A1
( en )
2014-07-25
2016-01-28
Aaron Burns
Virtual reality environment with real world objects
US20160154476A1
( en )
2013-05-22
2016-06-02
Nokia Technologies Oy
Apparatuses, methods and computer programs for remote control
US20160188021A1
( en )
*
2014-12-24
2016-06-30
Samsung Electronics Co., Ltd.
Method of receiving user input by detecting movement of user and apparatus therefor
WO2016118606A1
( en )
2015-01-21
2016-07-28
Google Inc.
Phone control and presence in virtual reality
WO2016123035A1
( en )
2015-01-30
2016-08-04
The Directv Group, Inc.
Method and system for viewing set top box content in a virtual reality device
WO2016144560A1
( en )
2015-03-09
2016-09-15
Google Inc.
Virtual reality headset connected to a mobile computing device
US20160292922A1
( en )
2013-05-21
2016-10-06
Sony Corporation
Display control device, display control method, and recording medium
US9542011B2
( en )
2014-04-08
2017-01-10
Eon Reality, Inc.
Interactive virtual reality systems and methods
WO2017007637A1
( en )
2015-07-07
2017-01-12
Google Inc.
System for tracking a handheld device in virtual reality
US20180197347A1
( en )
2017-01-11
2018-07-12
Taqtile
Managing virtual reality objects
US20190043264A1
( en )
2017-08-03
2019-02-07
Taqtile, Inc.
Authoring virtual and augmented reality environments via an xr collaboration application
US20190102922A1
( en )
*
2017-09-29
2019-04-04
Qualcomm Incorporated
Display of a live scene and auxiliary object
2017
2017-08-24
US
US15/685,870
patent/US10754496B2/en
active
Active
2018
2018-06-21
WO
PCT/US2018/038656
patent/WO2019040166A1/en
not_active
Ceased
Patent Citations (22)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US20080074560A1
( en )
*
2006-09-21
2008-03-27
Seiko Epson Corporation
Image display device, image display system, and network connection method
US20100287485A1
( en )
2009-05-06
2010-11-11
Joseph Bertolami
Systems and Methods for Unifying Coordinate Systems in Augmented Reality Applications
US20100328204A1
( en )
*
2009-06-25
2010-12-30
The Boeing Company
Virtual Control Station
US9218212B2
( en )
2011-11-11
2015-12-22
International Business Machines Corporation
Pairing physical devices to virtual devices to create an immersive environment
US20130141419A1
( en )
2011-12-01
2013-06-06
Brian Mount
Augmented reality with realistic occlusion
US20130265220A1
( en )
*
2012-04-09
2013-10-10
Omek Interactive, Ltd.
System and method for combining three-dimensional tracking with a three-dimensional display for a user interface
US9066200B1
( en )
2012-05-10
2015-06-23
Longsand Limited
User-generated content in a virtual reality environment
US20140248950A1
( en )
*
2013-03-01
2014-09-04
Martin Tosas Bautista
System and method of interaction for mobile devices
US20140267234A1
( en )
2013-03-15
2014-09-18
Anselm Hook
Generation and Sharing Coordinate System Between Users on Mobile
US20160292922A1
( en )
2013-05-21
2016-10-06
Sony Corporation
Display control device, display control method, and recording medium
US20160154476A1
( en )
2013-05-22
2016-06-02
Nokia Technologies Oy
Apparatuses, methods and computer programs for remote control
US20140361977A1
( en )
*
2013-06-07
2014-12-11
Sony Computer Entertainment Inc.
Image rendering responsive to user actions in head mounted display
US9542011B2
( en )
2014-04-08
2017-01-10
Eon Reality, Inc.
Interactive virtual reality systems and methods
US20160027215A1
( en )
2014-07-25
2016-01-28
Aaron Burns
Virtual reality environment with real world objects
US20160188021A1
( en )
*
2014-12-24
2016-06-30
Samsung Electronics Co., Ltd.
Method of receiving user input by detecting movement of user and apparatus therefor
WO2016118606A1
( en )
2015-01-21
2016-07-28
Google Inc.
Phone control and presence in virtual reality
WO2016123035A1
( en )
2015-01-30
2016-08-04
The Directv Group, Inc.
Method and system for viewing set top box content in a virtual reality device
WO2016144560A1
( en )
2015-03-09
2016-09-15
Google Inc.
Virtual reality headset connected to a mobile computing device
WO2017007637A1
( en )
2015-07-07
2017-01-12
Google Inc.
System for tracking a handheld device in virtual reality
US20180197347A1
( en )
2017-01-11
2018-07-12
Taqtile
Managing virtual reality objects
US20190043264A1
( en )
2017-08-03
2019-02-07
Taqtile, Inc.
Authoring virtual and augmented reality environments via an xr collaboration application
US20190102922A1
( en )
*
2017-09-29
2019-04-04
Qualcomm Incorporated
Display of a live scene and auxiliary object
Non-Patent Citations (3)
* Cited by examiner, â Cited by third party
Title
" International Search Report and Written Opinion Issued in PCT Application No. PCT/US2018/038656 ", dated Sep. 12, 2018, 15 Pages.
Lee, Max, " HTC Vive VR Unboxing ", Retrieved from https://www.youtube.com/watch?v=_Tj6DcMcptY&t=822s, May 4, 2016, 1 Page.
Taqtile. Screenshot of video entitled " HoloMaps " at 0:17 [online]. May 1, 2017 [retrieved on Jul. 24, 2019]. Retrieved form the Internet: <URL: https://vimeo.com/215573038>, 2 pages.
Cited By (9)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US12619296B2
( en )
2020-09-25
2026-05-05
Apple Inc.
Sensor emulation
US20230368475A1
( en )
*
2022-05-10
2023-11-16
Apple Inc.
Multi-Device Content Handoff Based on Source Device Position
US12058301B2
( en )
2022-06-03
2024-08-06
Apple Inc.
Electronic device that displays virtual objects
US12309344B2
( en )
2022-06-03
2025-05-20
Apple Inc.
Electronic device that displays virtual objects at different apparent depths
US12178641B2
( en )
2022-07-21
2024-12-31
Shanghai United Imaging Intelligence Co., Ltd.
Systems and methods for fetus monitoring
US20240211090A1
( en )
*
2022-12-23
2024-06-27
Rovi Guides, Inc.
Methods and systems for displaying virtual elements in an xr environment
US12353627B2
( en )
2023-03-28
2025-07-08
Samsung Electronics Co., Ltd.
Head-wearable electronic, method, and non-transitory computer readable storage medium for executing function based on identification of contact point and gaze point
US11870852B1
( en )
*
2023-03-31
2024-01-09
Meta Platforms Technologies, Llc
Systems and methods for local data transmission
US12436607B1
( en )
2024-05-29
2025-10-07
Adeia Guides Inc.
Systems and methods for providing enhanced extended reality interactions using personal electronic devices as input mechanisms
Also Published As
Publication number
Publication date
WO2019040166A1
( en )
2019-02-28
US20190065026A1
( en )
2019-02-28
Similar Documents
Publication
Publication Date
Title
US20190065026A1
( en )
2019-02-28
Virtual reality input
US10553031B2
( en )
2020-02-04
Digital project file presentation
US9824499B2
( en )
2017-11-21
Mixed-reality image capture
US10186086B2
( en )
2019-01-22
Augmented reality control of computing device
US10248192B2
( en )
2019-04-02
Gaze target application launcher
US9934614B2
( en )
2018-04-03
Fixed size augmented reality objects
EP3619599B1
( en )
2022-06-08
Virtual content displayed with shared anchor
EP3433706B1
( en )
2020-08-12
Virtual-reality navigation
EP3532177B1
( en )
2022-07-13
Virtual object movement