ABSTRACT
Abstract
Methods, systems, and computer programs are provided for generating an interactive space. One method includes operations for associating a first device to a reference point in 3D space, and for calculating by the first device a position of the first device in the 3D space based on inertial information captured by the first device and utilizing dead reckoning. Further, the method includes operations for capturing images with a camera of the first device, and for identifying locations of one or more static features in the images. The position of the first device is corrected based on the identified locations of the one or more static features, and a view of an interactive scene is presented in a display of the first device, where the interactive scene is tied to the reference point and includes virtual objects.
Description
CLAIM OF PRIORITY
This application is a Continuation Application under 35 USC § 120 and claims priority from U.S. application Ser. No. 14/260,208, filed Apr. 23, 2014 entitled âCalibration of Portable Devices in a Shared Virtual Spaceâ, which claims priority from U.S. application Ser. No. 14/017,208, filed Sep. 3, 2013, entitled âCalibration of Portable devices in a Shared Virtual Space,â which is a Continuation Application under 35 USC § 120 and claims priority from U.S. application Ser. No. 12/973,827 entitled âCalibration of Portable devices in a Shared Virtual Space,â filed on Dec. 20, 2010, which claims priority from U.S. Provisional Patent Application No. 61/311,251, filed Mar. 5, 2010, entitled âMAINTAINING MULTIPLE VIEWS ON A SHARED STABLE VIRTUAL SPACEâ; and U.S. Provisional Patent Application No. 61/323,762, filed Apr. 13, 2010, and entitled âCALIBRATION OF PORTABLE DEVICES IN A SHARED VIRTUAL SPACEâ, all of which are incorporated herein by reference.
CROSS REFERENCE TO RELATED APPLICATIONS
This application is related to U.S. application Ser. No. 12/647,291, filed Dec. 24, 2009, and entitled âWIRELESS DEVICE PAIRING METHODSâ; U.S. application Ser. No. 12/647,296, filed Dec. 24, 2009, and entitled âWIRELESS DEVICE PAIRING AND GROUPING METHODSâ; U.S. application Ser. No. 12/647,299, filed Dec. 24, 2009, and entitled âWIRELESS DEVICE MULTIMEDIA FEED SWITCHINGâ; and U.S. Provisional Application 61/319,021, filed Mar. 30, 2010, and entitled âMETHOD FOR AN AUGMENTED REALITY CHARACTER TO MAINTAIN AND EXHIBIT AWARENESS OF AN OBSERVERâ, which are incorporated herein by reference.
BACKGROUND
1. Field of the Invention
The present invention relates to methods, devices, and computer programs for controlling a view of a virtual scene with a portable device, and more particularly, methods, devices, and computer programs for enabling multiplayer interaction in a virtual or augmented reality.
2. Description of the Related Art
Virtual reality (VR) is a computer-simulated environment, whether that environment is a simulation of the real world or an imaginary world, where users can interact with a virtual environment or a virtual artifact either through the use of standard input devices or specialized multidirectional input devices. The simulated environment can be similar to the real world, for example, simulations for pilot or combat training, or it can differ significantly from reality, as in VR games. Virtual Reality is often used to describe a wide variety of applications, commonly associated with its immersive, highly visual, three-dimensional (3D) environments. The development of Computer Aided Design (CAD) software, graphics hardware acceleration, head-mounted displays, database gloves, and miniaturization have helped popularize the notion. Augmented Reality (AR) provides a live view of a physical real-world environment whose elements are merged with (or augmented by) virtual computer-generated imagery to create a mixed reality. The augmentation is conventionally in real-time and in semantic context with environmental elements, such as sports scores on television during a match. With the help of advanced AR technology (e.g. adding computer vision and object recognition) the information about the surrounding real world of the user becomes interactive and digitally usable.
The term Augmented Virtuality (AV) is also used in the virtual reality world and is similar to AR. Augmented Virtuality also refers to the merging of real world objects into virtual worlds. As an intermediate case in the Virtuality Continuum, AV refers to predominantly virtual spaces, where physical elements, e.g. physical objects or people, are dynamically integrated into, and can interact with the virtual world in real-time. The term VR is used in this application as a generic term that also encompasses AR and AV, unless otherwise specified.
VR games typically required a large amount of computer resources. Implementation in handheld devices of VR games is rare and the existing games are rather simplistic with rudimentary VR effects. Additionally, multiplayer AR games allow for the interaction of players in a virtual world, but the interactions are limited to objects manipulated by the player in the virtual world (e.g., cars, rackets, balls, etc.) The virtual world is computer generated and independent of the location of the players and the portable devices. The relative location of the players with respect to each other and with respect to their surroundings is not taken into account when creating a ârealisticâ virtual reality experience.
It is in this context that embodiments of the invention arise.
SUMMARY
Embodiments of the present invention provide methods, systems, and computer programs for generating an interactive space viewable through at least a first and a second handheld devices are presented. It should be appreciated that the present invention can be implemented in numerous ways, such as a process, an apparatus, a system, a device or a method on a computer readable medium. Several inventive embodiments of the present invention are described below.
In one embodiment, a method includes an operation for taking an image with a camera in the first device. In addition, the method includes an operation for determining a relative position of the second device with reference to the first device, based on image analysis of the taken image to identify a geometry of the second device. Furthermore, the method includes operations for identifying a reference point in a three-dimensional (3D) space based on the relative position, and for generating views of an interactive scene in corresponding displays of the first device and the second device. The interactive scene is tied to the reference point and includes virtual objects, and each view shows all or part of the interactive scene as observed from a current location of the corresponding device.
In another embodiment, a method, for generating an interactive space viewable through at least a first device and a second device, includes an operation for taking an image with a camera in the first device, where the first device and the second device are handheld devices. In addition, the method includes an operation for determining a relative position of the second device with reference to the first device based on image analysis of the image to identify a light source in the second device. Further yet, the method includes operations for identifying a reference point in a three-dimensional (3D) space based on the relative position, and for generating views of an interactive scene in corresponding displays of the first device and the second device. The interactive scene is tied to the reference point and includes virtual objects, and each view shows all or part of the interactive scene as observed from a current location of the corresponding device.
In yet another embodiment, a method, for generating an interactive space viewable through at least a first device and a second device, includes an operation for taking an image with a camera in the first device, where the first device and the second device are handheld devices. In addition, the method includes an operation for determining a relative position of the second device with reference to the first device, based on image analysis of the image to identify a graphic shown on a second display of the second device. Additionally, the method includes operations for identifying a reference point in a three-dimensional (3D) space based on the relative position, and for generating views of an interactive scene in corresponding displays of the first device and the second device. The interactive scene is tied to the reference point and includes virtual objects, and each view shows all or part of the interactive scene as observed from a current location of the corresponding device.
Other aspects of the invention will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the invention.
BRIEF DESCRIPTION OF THE DRAWINGS
The invention may best be understood by reference to the following description taken in conjunction with the accompanying drawings in which:
FIG. 1 depicts a user before synchronization a portable device to a reference point in space, according to one embodiment.
FIG. 2 illustrates a virtual reality scene observed with the portable device.
FIG. 3 illustrates how movement of the portable device has a similar effect on the display as when moving a camera in the virtual space, according to one embodiment.
FIG. 4 shows a two-dimensional representation of the change in the image shown in the display when turning the portable device, according to one embodiment.
FIG. 5 illustrates how to play an interactive game over a network connection, according to one embodiment.
FIG. 6 illustrates the process of tapping two portable devices to synchronize their positions, according to one embodiment.
FIG. 7 shows two portable devices after tapping, according to one embodiment.
FIG. 8 illustrates the creation of a virtual scene around the reference point, in accordance with one embodiment of the invention.
FIG. 9 depicts two players in the same space where a virtual reality has been created around a reference point on a table, according to one embodiment.
FIGS. 10A-10B illustrate the process of detecting a location of another portable device using image recognition, according to one embodiment.
FIG. 11 shows how to detect the second device by finding a light source in the second device, according to one embodiment.
FIG. 12 illustrates detecting the second device by finding the display of the second device, according to one embodiment.
FIG. 13 shows an embodiment for tracking the portable device via dead reckoning.
FIG. 14 illustrates how to adjust dead reckoning using static features in the background, according to one embodiment.
FIG. 15 illustrates one embodiment of a calibration method for a multi-player environment.
FIG. 16 depicts a multi-player virtual reality game, according to one embodiment.
FIG. 17 shows the flow of an algorithm for generating an interactive space viewable through at least a first and a second device, in accordance with one embodiment of the invention.
FIG. 18 illustrates the architecture of a device that may be used to implement embodiments of the invention.
FIG. 19 is an exemplary illustration of scene A through scene E with respective user A through user E interacting with game clients 1102 that are connected to server processing via the internet, in accordance with one embodiment of the present invention.
FIG. 20 illustrates an embodiment of an Information Service Provider architecture.
DETAILED DESCRIPTION
The following embodiments describe methods, apparatus, and computer programs for generating an interactive space viewable through at least a first and a second device. It will be obvious, however, to one skilled in the art, that the present invention may be practiced without some or all of these specific details. In other instances, well known process operations have not been described in detail in order not to unnecessarily obscure the present invention.
FIG. 1 depicts a user before synchronizing a portable device to a reference point in space, according to one embodiment. Portable device 104 is standing on a table in preparation for synchronizing the portable device to a reference point. User 102 has placed the portable device in a point that will serve as a reference point or anchor to build a virtual reality around the point. In the case shown in FIG. 1 , the portable device is sitting in the approximate center of a table, and a virtual world is built around the center of the table once the portable device is synchronized. The portable device can be synchronized in a variety of ways, such as pushing a button on portable device 104 , touching the touch-sensitive screen in the portable device, letting the device stand still for a period of time (e.g., five seconds), entering a voice command, etc.
Once the portable device receives the input to be synchronized, position tracking modules in the portable device are reset. The portable device can include a variety of position tracking modules, as discussed below in reference to FIG. 18 , such as an accelerometer, a magnetometer, a Global Positioning System (GPS) device, a camera, a depth camera, a compass, a gyroscope, etc.
The portable device can be one of many types, such as a handheld portable gaming device, a cell phone, a tablet, a notebook, a netbook, a Personal Digital Assistant (PDA), etc. Embodiments of the invention are described in reference to a portable gaming device, but the principles can be applied to any portable electronic device with a display. Principles of the invention can also be applied to game controllers or other input devices connected to a computing device with a display.
FIG. 2 illustrates a virtual reality scene observed with the portable device. After synchronizing device 104 with respect to reference points 106 , the portable device will start displaying a view of the virtual reality 108 . The view in the display is created by simulating that a camera in the back of the portable device moves within the 3D space around reference point 106 . FIG. 2 depicts a virtual reality that includes a chess board. Portable device 104 is capable of detecting motion and determining its relative
CLAIM OF PRIORITY
This application is a Continuation Application under 35 USC § 120 and claims priority from U.S. application Ser. No. 14/260,208, filed Apr. 23, 2014 entitled âCalibration of Portable Devices in a Shared Virtual Spaceâ, which claims priority from U.S. application Ser. No. 14/017,208, filed Sep. 3, 2013, entitled âCalibration of Portable devices in a Shared Virtual Space,â which is a Continuation Application under 35 USC § 120 and claims priority from U.S. application Ser. No. 12/973,827 entitled âCalibration of Portable devices in a Shared Virtual Space,â filed on Dec. 20, 2010, which claims priority from U.S. Provisional Patent Application No. 61/311,251, filed Mar. 5, 2010, entitled âMAINTAINING MULTIPLE VIEWS ON A SHARED STABLE VIRTUAL SPACEâ; and U.S. Provisional Patent Application No. 61/323,762, filed Apr. 13, 2010, and entitled âCALIBRATION OF PORTABLE DEVICES IN A SHARED VIRTUAL SPACEâ, all of which are incorporated herein by reference.
CROSS REFERENCE TO RELATED APPLICATIONS
This application is related to U.S. application Ser. No. 12/647,291, filed Dec. 24, 2009, and entitled âWIRELESS DEVICE PAIRING METHODSâ; U.S. application Ser. No. 12/647,296, filed Dec. 24, 2009, and entitled âWIRELESS DEVICE PAIRING AND GROUPING METHODSâ; U.S. application Ser. No. 12/647,299, filed Dec. 24, 2009, and entitled âWIRELESS DEVICE MULTIMEDIA FEED SWITCHINGâ; and U.S. Provisional Application 61/319,021, filed Mar. 30, 2010, and entitled âMETHOD FOR AN AUGMENTED REALITY CHARACTER TO MAINTAIN AND EXHIBIT AWARENESS OF AN OBSERVERâ, which are incorporated herein by reference.
BACKGROUND
1. Field of the Invention
The present invention relates to methods, devices, and computer programs for controlling a view of a virtual scene with a portable device, and more particularly, methods, devices, and computer programs for enabling multiplayer interaction in a virtual or augmented reality.
2. Description of the Related Art
Virtual reality (VR) is a computer-simulated environment, whether that environment is a simulation of the real world or an imaginary world, where users can interact with a virtual environment or a virtual artifact either through the use of standard input devices or specialized multidirectional input devices. The simulated environment can be similar to the real world, for example, simulations for pilot or combat training, or it can differ significantly from reality, as in VR games. Virtual Reality is often used to describe a wide variety of applications, commonly associated with its immersive, highly visual, three-dimensional (3D) environments. The development of Computer Aided Design (CAD) software, graphics hardware acceleration, head-mounted displays, database gloves, and miniaturization have helped popularize the notion. Augmented Reality (AR) provides a live view of a physical real-world environment whose elements are merged with (or augmented by) virtual computer-generated imagery to create a mixed reality. The augmentation is conventionally in real-time and in semantic context with environmental elements, such as sports scores on television during a match. With the help of advanced AR technology (e.g. adding computer vision and object recognition) the information about the surrounding real world of the user becomes interactive and digitally usable.
The term Augmented Virtuality (AV) is also used in the virtual reality world and is similar to AR. Augmented Virtuality also refers to the merging of real world objects into virtual worlds. As an intermediate case in the Virtuality Continuum, AV refers to predominantly virtual spaces, where physical elements, e.g. physical objects or people, are dynamically integrated into, and can interact with the virtual world in real-time. The term VR is used in this application as a generic term that also encompasses AR and AV, unless otherwise specified.
VR games typically required a large amount of computer resources. Implementation in handheld devices of VR games is rare and the existing games are rather simplistic with rudimentary VR effects. Additionally, multiplayer AR games allow for the interaction of players in a virtual world, but the interactions are limited to objects manipulated by the player in the virtual world (e.g., cars, rackets, balls, etc.) The virtual world is computer generated and independent of the location of the players and the portable devices. The relative location of the players with respect to each other and with respect to their surroundings is not taken into account when creating a ârealisticâ virtual reality experience.
It is in this context that embodiments of the invention arise.
SUMMARY
Embodiments of the present invention provide methods, systems, and computer programs for generating an interactive space viewable through at least a first and a second handheld devices are presented. It should be appreciated that the present invention can be implemented in numerous ways, such as a process, an apparatus, a system, a device or a method on a computer readable medium. Several inventive embodiments of the present invention are described below.
In one embodiment, a method includes an operation for taking an image with a camera in the first device. In addition, the method includes an operation for determining a relative position of the second device with reference to the first device, based on image analysis of the taken image to identify a geometry of the second device. Furthermore, the method includes operations for identifying a reference point in a three-dimensional (3D) space based on the relative position, and for generating views of an interactive scene in corresponding displays of the first device and the second device. The interactive scene is tied to the reference point and includes virtual objects, and each view shows all or part of the interactive scene as observed from a current location of the corresponding device.
In another embodiment, a method, for generating an interactive space viewable through at least a first device and a second device, includes an operation for taking an image with a camera in the first device, where the first device and the second device are handheld devices. In addition, the method includes an operation for determining a relative position of the second device with reference to the first device based on image analysis of the image to identify a light source in the second device. Further yet, the method includes operations for identifying a reference point in a three-dimensional (3D) space based on the relative position, and for generating views of an interactive scene in corresponding displays of the first device and the second device. The interactive scene is tied to the reference point and includes virtual objects, and each view shows all or part of the interactive scene as observed from a current location of the corresponding device.
In yet another embodiment, a method, for generating an interactive space viewable through at least a first device and a second device, includes an operation for taking an image with a camera in the first device, where the first device and the second device are handheld devices. In addition, the method includes an operation for determining a relative position of the second device with reference to the first device, based on image analysis of the image to identify a graphic shown on a second display of the second device. Additionally, the method includes operations for identifying a reference point in a three-dimensional (3D) space based on the relative position, and for generating views of an interactive scene in corresponding displays of the first device and the second device. The interactive scene is tied to the reference point and includes virtual objects, and each view shows all or part of the interactive scene as observed from a current location of the corresponding device.
Other aspects of the invention will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the invention.
BRIEF DESCRIPTION OF THE DRAWINGS
The invention may best be understood by reference to the following description taken in conjunction with the accompanying drawings in which:
FIG. 1 depicts a user before synchronization a portable device to a reference point in space, according to one embodiment.
FIG. 2 illustrates a virtual reality scene observed with the portable device.
FIG. 3 illustrates how movement of the portable device has a similar effect on the display as when moving a camera in the virtual space, according to one embodiment.
FIG. 4 shows a two-dimensional representation of the change in the image shown in the display when turning the portable device, according to one embodiment.
FIG. 5 illustrates how to play an interactive game over a network connection, according to one embodiment.
FIG. 6 illustrates the process of tapping two portable devices to synchronize their positions, according to one embodiment.
FIG. 7 shows two portable devices after tapping, according to one embodiment.
FIG. 8 illustrates the creation of a virtual scene around the reference point, in accordance with one embodiment of the invention.
FIG. 9 depicts two players in the same space where a virtual reality has been created around a reference point on a table, according to one embodiment.
FIGS. 10A-10B illustrate the process of detecting a location of another portable device using image recognition, according to one embodiment.
FIG. 11 shows how to detect the second device by finding a light source in the second device, according to one embodiment.
FIG. 12 illustrates detecting the second device by finding the display of the second device, according to one embodiment.
FIG. 13 shows an embodiment for tracking the portable device via dead reckoning.
FIG. 14 illustrates how to adjust dead reckoning using static features in the background, according to one embodiment.
FIG. 15 illustrates one embodiment of a calibration method for a multi-player environment.
FIG. 16 depicts a multi-player virtual reality game, according to one embodiment.
FIG. 17 shows the flow of an algorithm for generating an interactive space viewable through at least a first and a second device, in accordance with one embodiment of the invention.
FIG. 18 illustrates the architecture of a device that may be used to implement embodiments of the invention.
FIG. 19 is an exemplary illustration of scene A through scene E with respective user A through user E interacting with game clients 1102 that are connected to server processing via the internet, in accordance with one embodiment of the present invention.
FIG. 20 illustrates an embodiment of an Information Service Provider architecture.
DETAILED DESCRIPTION
The following embodiments describe methods, apparatus, and computer programs for generating an interactive space viewable through at least a first and a second device. It will be obvious, however, to one skilled in the art, that the present invention may be practiced without some or all of these specific details. In other instances, well known process operations have not been described in detail in order not to unnecessarily obscure the present invention.
FIG. 1 depicts a user before synchronizing a portable device to a reference point in space, according to one embodiment. Portable device 104 is standing on a table in preparation for synchronizing the portable device to a reference point. User 102 has placed the portable device in a point that will serve as a reference point or anchor to build a virtual reality around the point. In the case shown in FIG. 1 , the portable device is sitting in the approximate center of a table, and a virtual world is built around the center of the table once the portable device is synchronized. The portable device can be synchronized in a variety of ways, such as pushing a button on portable device 104 , touching the touch-sensitive screen in the portable device, letting the device stand still for a period of time (e.g., five seconds), entering a voice command, etc.
Once the portable device receives the input to be synchronized, position tracking modules in the portable device are reset. The portable device can include a variety of position tracking modules, as discussed below in reference to FIG. 18 , such as an accelerometer, a magnetometer, a Global Positioning System (GPS) device, a camera, a depth camera, a compass, a gyroscope, etc.
The portable device can be one of many types, such as a handheld portable gaming device, a cell phone, a tablet, a notebook, a netbook, a Personal Digital Assistant (PDA), etc. Embodiments of the invention are described in reference to a portable gaming device, but the principles can be applied to any portable electronic device with a display. Principles of the invention can also be applied to game controllers or other input devices connected to a computing device with a display.
FIG. 2 illustrates a virtual reality scene observed with the portable device. After synchronizing device 104 with respect to reference points 106 , the portable device will start displaying a view of the virtual reality 108 . The view in the display is created by simulating that a camera in the back of the portable device moves within the 3D space around reference point 106 . FIG. 2 depicts a virtual reality that includes a chess board. Portable device 104 is capable of detecting motion and determining its relative position with respect to reference point 106 as the device moves around. Location and position determination can be done with different methods and different levels of accuracy. For example, location can be detected by analyzing images captured with a camera, or data obtained from inertial systems, GPS, ultrasonic triangulation, WiFi communications, dead reckoning (DR), etc., or a combination thereof.
When the user enters a command to set the reference points, all motion sensing devices are zeroed out or calibrated to that position in space. For example, the user can place the device on the table and press a button to calibrate all motion sensing data (accelerometers, gyroscope, GPS, etc.) From this point on, all captured positional information for the device is recorded and processed relative to the initial calibrated position via inertial navigation tracking. All subsequent captured positional information is considered relative to the device calibrated position (reference point.)
In one embodiment, the device keeps track of the location in space of the portable device with respect to reference point 106 , as well as the position in space of the portable device. The position is used to determine the viewing angle of the camera, that is, the portable device acts as a camera into the virtual scene. If the portable device is aimed towards the right, then the view will turn to the right, etc. In other words, the viewing angle is defined as a vector with origin in the center of the display (or other part of the device), and with a direction perpendicular to and away from the display. In another embodiment, only the position in space is tracked, and the view in the display is calculated as if the camera is aiming from the location in space where the portable device is located and towards the reference point.
In some existing implementations, an AR tag is placed on a table, and utilized as a fiduciary marker for generating the augmented reality. The AR tag may be an object or figure that is recognized when present in the captured image stream of the real environment. The AR tag serves as a fiduciary marker which enables determination of a location within the real environment. Embodiments to the invention eliminate the need for AR Tags, because of the calibration within the 3D space and the location tracking of the portable device. Additionally, the location information allows games in the portable device to deliver a realistic 3D virtual experience. Further, an array of networked portable devices can be used to create a shared virtual world, as described below in reference to FIGS. 15 and 16 .
FIG. 3 illustrates how movement of the portable device has a similar effect on the display as when moving a camera in the virtual space, according to one embodiment. FIG. 3 shows car 302 inside a virtual sphere. Assuming that a portable device is aimed from a point in the sphere towards car 302 , multiple views of the car can be obtained as the portable device moves within the sphere. For example, a view from the ânorth poleâ will show the roof of the car, and a view from the âsouth poleâ will show the bottom of the car. Also shown in FIG. 3 are views for the side, front, and rear of the car.
In one embodiment, the player can enter a command to change or flip the view of the virtual world. For example, in the case of the car, a player goes from seeing the front of the car to seeing the back of the car, as if the scene had rotated 180° around and axis running vertically through the reference point. This way, the player does not have to move around the room to get different viewing angles. Other inputs may produce different effects, such as a 90° turn, a scaling of the view (to make the virtual world seem smaller or greater), rotation with respect to the x, y, or z axis, etc. In another embodiment, a flip of the portable device, i.e., a 180° spin on the player's hand will cause view of the virtual world to flip upside down.
FIG. 4 shows a two-dimensional representation of the change in the image shown in the display when turning the portable device, according to one embodiment. Portable device 402 is aimed towards a wall with a viewing angle α, resulting in a projection 410 on the wall. Thus, the view on portable device 402 will correspond to projection 410 . When device 402 is turned an angle β, the portable device ends in position 404 . The view also turns an angle β while maintaining a camera viewing angle α. As a result, the view on the portable device corresponds to projection 412 . It should be noted that the view on the screen is independent of the eye position, such as positions
408 and 406 , and the view is independent from where the player is. The image on the display depends on the position of the portable device, which is acting as a virtual camera.
FIG. 5 illustrates how to play an interactive game over a network connection, according to one embodiment. Many types of games are possible within a shared space. For example, the portable device can be used as a paddle to play a game of ping-pong. The device is moved around as if it where a paddle that can hit a virtual ball. Players see the ball float between the screen and the opponent's screen. In an embodiment of a war game, the player looks through the portable device and aims the catapult at the enemies' ramparts. The player pulls the device backwards to load the catapult, and then press a button to fire the catapult toward the enemies' castle.
Shared spaces can also be created when players are in different locations, as shown in FIG. 5 . The players have established a network connection to play the game. Each player synchronizes his device to a reference point in the player's space, and a virtual reality is created, such as a ping-pong table. The opponent is shown behind his side of the table, where the movement of an opponent's device is matched to the motions of the opponent's paddle. The game may also add an avatar to hold the paddle, for an even more realistic game experience. During play, each device keeps track of the motion and position of the device. This information is shared with the other device to enable the other device to place a virtual paddle that matches the motion of the device. Other game information is also shared, such as the location and movement of the ball.
FIG. 6 illustrates the process of tapping two portable devices to synchronize their positions, according to one embodiment. One method for setting a common virtual or augmented space includes tapping the two portable devices. Tapping means striking lightly one device against the other. In the scenario shown in FIG. 6 , two players are holding portable devices
604 and 606 , respectively. In order to calibrate both devices to the same reference point, the players tap the devices by placing both devices together. In FIG. 6 , both devices are placed back to back, but any position is possible for calibration, such as front to back. The key to detect the tapping is that one or two of the devices notices an abrupt change in motion, such as a sudden deceleration of the device.
For example, both devices can be moving towards each other, and when they tap, both devices come to a stop. The inertial modules in the devices, such as gyroscope 612 and accelerometer 614 , notice the change of momentum and then the tap can be established. In another scenario, one portable device is stationary while the other portable device moves towards the stationary device. When the devices tap, the moving device will notice a sudden change in momentum while the stationary device may or may not detect the tapping as a small change in momentum can be attributed to the natural motion of the player's hand. To detect the tapping, it is enough that one device detects the tapping, and it is not a requirement that both devices detect the tapping simultaneously. In one embodiment, both devices detect the tapping and if the detection is substantially simultaneously, then it is determined that the tapping has occurred. To determine that the detection if simultaneous, the devices exchange timing information regarding the event.
In another embodiment, once portable devices
602 and 604 are synchronized to the same virtual space, their movement is tracked with accelerometer 614 , enabling the creation of a stable and persistent augmented reality environment regardless of how the portable device is moved by the user. In yet another embodiment, the inertial movement information can be complemented with image data captured with camera
606 or 610 . The image data can be used to detect the other device and estimate the distance between the devices, as discussed in more detail below in reference to FIGS. 10A-12 .
FIG. 7 shows two portable devices after tapping, according to one embodiment. Tapping can be done by having portable devices touch, but tapping can also be done without having both devices actually come in contact. All that is required is that their change in motion occurs at about the same time. For example, if the fingers of a player gets is behind a first device, the tapping will be detected when the second device touches the fingers of the player, causing the change in motion of the devices.
Once the tapping is detected, by either device or by both devices, the portable devices exchange data to confirm that the tapping has occurred. For example, the devices can communicate via WiFi or ultrasonic communications. A reference point is created, as previously discussed in FIGS. 1-2 . The reference point can be situated somewhere in the back of the devices, such as the center of each device when the devices tap. In another embodiment, the exchange of data can be performed using a TransferJet interface. TransferJet is an interface that enables communication when close proximity is detected between two devices.
It should be noted, that the actual reference point for each device may not be the exact same point in space for both devices. In other words, each device may have a different reference point, although in most cases, the reference points will be proximate to each other. The important thing is that both devices set up a reference point and then start tracking movement around the virtual or augmented space. The result is a common virtual space. The reference point can be set in the center of the back of the portable device, in the center of the display, where the camera is located, where the accelerometer is located, etc.
Once the reference point is set, the motion tracking modules are reset to zero to set an origin point to measure position in space. This operation is referred to herein as calibrating the portable devices, and the calculation of the three-dimensional position of the portable device is computed in reference to this origin.
FIG. 8 illustrates the creation of a virtual scene around the reference point, in accordance with one embodiment of the invention. As the two portable devices of FIG. 7 are pulled apart, a virtual or augmented reality play area is created. The virtual objects, such as fighters
804 and 806 , are assigned corresponding positions in the 3D space where the portable devices are located. The coordinate origin in the 3D space is reference point 808 . Because the virtual objects are positioned relative to a point in space, there is no need for SLAM or ARTAG for maintaining augmented realities.
The virtual camera associated with the view from the display is controlled is by physically moving the portable device around the game world, which has been placed in a fixed position in reference to the real world. It should be noted that the virtual world is not confined to the space between the portable devices, but can expand and cover areas above, below, and behind any of the portable devices.
FIG. 9 depicts two players 906 a - 906 b in the same space where a virtual reality has been created around reference point 902 on table 904 , according to one embodiment. Players
906 a and 906 b have synchronized their devices
908 a and 908 b to common reference point 902 , which is located on top of table 904 . Since point P
0 902 is the reference point, P 0 is also the coordinate origin and it has coordinates (X 0 =0, Y 0 =0, Z 0 =0). The players are inside a room but the virtual reality, also referred to herein as virtual scene, extends beyond the physical boundaries of the room.
In one example embodiment, the virtual scene is tied to the reference point because the geometry of the virtual scene (as seen through a screen of the device) is based, at least in part, on the reference point. For example, the coordinates of the virtual objects in the virtual scene may be determined with respect to the reference point. In one embodiment, the reference point is the coordinate origin, thus the reference point has coordinates (0, 0, 0).
The coordinates can be measured using any standard of measure. However, to provide a visual example, and without limitation on actual coordinates used, if the coordinates of the virtual scene are measured in meters, an object with coordinates (1, 0, 0) would be situated one meter to the right of the reference point. Of course, the coordinates of objects, real or virtual, may be dynamically updated as the scene changes, such as when a virtual object moves within the scene. And, the changes can be defined by actions set by the computer (e.g., interactive program), driven by actions of the user, or combinations of both. Additionally, for sake of clarity, the interactive program can be any type of program, such as a video game, a business program, an internet interface, or simply a graphical user interface that provides access to data, to other users, to programs, or to objects that may or may not be displayed or projected by a speaker.
Still further, other embodiments may have different coordinates systems or use scaling. For example, the coordinate system, instead of being a Cartesian system, can be polar, spherical, parabolic, etc. Additionally, the reference point does not have to be the origin of the coordinate system, and can be positioned at a different place. For sake of providing an example, the reference point can be located at coordinates (5, 5, 5) to enable a buffer of 5 meters in each direction before having to use negative coordinate values in points beyond the 5 meters. In another scenario, the virtual objects are built to scale and the coordinates are also measured in a scale. For example, the virtual objects may be built on a scale of 1:10, and the geometric axis can also have a scale of 1:10, such that an object with coordinates (1, 0, 0) is 1 meter away in the ârealâ world and 10 meters away in the virtual world.
In FIG. 9 , virtual objects include helicopters 914 a - 914 c , clouds, birds, sun 916 , etc. As players
906 a and 906 b move their devices, the view of the virtual scene changes as if the players were holding a camera into the virtual world. It should be noted that the view shown in devices
908 a and 908 b may include or may not include the reference point. For example, device 908 a held by player 906 a is aiming away from reference point 902 , therefore reference point 902 is not viewable in device 908 a , although the view in device 908 a is calculated based on the locations of device 908 a and reference point 902 . Further, the actual reference point may be seen in a player's display as some kind of marker (such as an âXâ) to let the player know where the reference point. In other embodiments, the reference point is not viewable and it exists only as a geographical location without any particular markers.
The room includes other static objects besides table 904 , such as television 912 and window 910 . Sun 918 is visible through window 910 . It should be noted that virtual sun 916 does not have to correspond to actual sun 918 . In one embodiment, virtual sun 916 can be placed where the actual sun 918 is located, or where a source of light is located in the room. This way, lighting and shadows in the virtual or augmented world will create realistic effects matching the lighting and shadows in the room.
As seen in FIG. 9 , just because the portable devices are synchronized to a point on a table, the virtual objects do not have to be on the table or near the reference point. The virtual objects can be located anywhere in space. When the portable devices include a camera, the static features in the room can be used by the portable devices to maintain an accurate measurement of the current position by adjusting their inertial measurements with the views from their cameras. Image analysis in the portable device can detect the edges of a window, a light source, the edges of the table, a painting on the wall, a television, etc. More details are described below in reference to FIGS. 13-14 .
In one embodiment, the players can do a rapid recalibration of the portable device, by placing the device on the reference point again, and then entering a command to reset the motion detection modules in the device.
FIGS. 10A-10B illustrate the process of detecting a location of another portable device using image recognition, according to one embodiment. If both portable devices are the same model and include a camera 1006 , it is possible to use camera-captured images to synchronize both devices, as seen in FIG. 10A . For example, portable device 1002 has taken an image with camera 1006 . Since portable device 1002 has received a command to synchronize, portable device 1002 is scanning the image for the back of the other portable device 1004 . An schematic of the image taken is shown in FIG. 10B .
Portable device 1022 has detected a rectangle 1010 that matches the characteristics of the portable device being searched. Rectangle 1010 has horizontal axis 1012 and vertical axis 1014 . Horizontal axis 1012 is tilted and angle α with respect to horizon 1018 . Since portable device 1002 knows the dimensions of the other portable device 1004 , portable device 1002 makes the proper mathematical calculations comparing the size of the portable device in the image with the real measurements to determine the distance, location, and orientation of portable device 1004 .
Once portable device knows the relative position of portable device 1004 , portable devices
1002 and 1004 exchange location information to set the common reference point and build the common virtual or augmented reality around the reference point.
FIG. 11 shows how to detect the second device by finding a light source in the second device, according to one embodiment. The synchronization process of FIG. 11 is similar to the process described above in reference to FIG. 10 , except that portable device 1102 has a light source that is detected by the other portable device. When the portable devices are in the process of synchronizing, the light source is turned on to facilitate the detection of the light within the image taken with the camera. The light source can be a Light-Emitting Diode (LED), an infrared light, a camera flash, etc.
In another embodiment, a depth camera in one portable device is used to measure the distance to the other device. Once the distance is known, the reference point can be set based on the position of either device. Additional data, such as image data, can also be used to complement the calculations of the relative positions of the devices.
It should be noted that it is possible that both devices detect each other at about the same time. In this case, the calculation of the relative position of the portable devices can be done by one portable device or by the other, or it can be a combination of the measurements taken by both portable devices.
FIG. 12 illustrates detecting the second device by finding the display of the second device, according to one embodiment. When calibrating portable devices
1206 and 1208 to a common space, one of the devices, such as portable device 1208 is âturned aroundâ such that display 1202 is facing the camera in portable device 1206 . For example, portable device 1208 has shown a message asking the user to âturn device around in order have display facing other portable device.â
Display 1202 can then be brightly lit, such as for example showing a white screen. The brightness of the display facilitates the detection by portable device 1206 of the display. A brightly lit display shows a high contrast against most backgrounds, including in many cases the front of the player holding the device. Other patterns or colors can be added to the display to improve its detection. For example, FIG. 12 shows a circle with a square pattern inside for easier recognition of the display using geometry.
Other methods for calibrating both devices, also referred to as pairing, can be found in U.S. application Ser. No. 12/647,291, filed Dec. 24, 2009, and entitled âWIRELESS DEVICE PAIRING METHODSâ; U.S. application Ser. No. 12/647,296, filed Dec. 24, 2009, and entitled âWIRELESS DEVICE PAIRING AND GROUPING METHODSâ; and U.S. application Ser. No. 12/647,299, filed Dec. 24, 2009, and entitled âWIRELESS DEVICE MULTIMEDIA FEED SWITCHINGâ, which are incorporated herein by reference.
In another embodiment, portable device 1208 has two displays, one in the front and one in the back. In this case, it is not necessary to turn around portable device 1208 in order to have the display face the other player. Portable device 1206 will be able to detect either the display in the front or the display in the back from portable device 1208 to perform the calibration.
FIG. 13 shows an embodiment for tracking the portable device via dead reckoning. If the portable device is equipped with a camera facing away from the users face, then it is possible to know the position of the device via dead reckoning and inertial navigation. Additionally, the cameras real-world views can be blended with computer generated graphics.
Dead reckoning is the process of estimating a current position based upon a previously determined position, or fix, and advancing that position based upon known or estimated speeds over elapsed time and course. A disadvantage of dead reckoning is that since new positions are calculated solely from previous positions, the errors of the process are cumulative, so the error in the position fix grows with time.
FIG. 13 shows two portable devices that have been calibrated to a point on table 1310 . Over time, device 1302 A has traveled around the room, as observed in dead-reckoning measured trajectory 1306 . The points in trajectory 1306 show the times where the position was estimated using the previous position and the movement of the portable device 1302 A since the last measurement. On the other hand, real trajectory 1308 shows the actual trajectory of the portable device in space. Over time, the measured trajectory tends to diverge farther and farther apart from the real trajectory due to the nature of dead reckoning and error accumulation. However, dead reckoning tracking can be adjusted to correct the deviation from the actual trajectory, as described below in reference to FIG. 14 .
FIG. 14 illustrates how to adjust dead reckoning using static features in the background, according to one embodiment. The dead reckoning drift is usually small, but the accumulation of error over time can create larger deviations from actual course. In one embodiment, the DR errors are corrected by video tracking static patterns in the room, such as an edge of table 1301 , a window, a television, etc., as previously described in reference to FIG. 9 .
Error correction can be performed at every DR position measurement, or at certain intervals, or every certain number of measurements. To perform a correction, DR calculates the position of the device and the camera takes an image in front of the device. Portable device keeps track of one or more static features and compares the actual location of the static feature with the expected location. Since static features do not move, the difference is attributed to DR reckoning error. The position is recalculated in order to have the expected static feature and the measured static feature be in the same place.
In another embodiment, the portable devices communicate with each other and coordinate the motion tracking of both devices by exchanging DR and feature information. This way a realistic 3D model of the static objects in the room can be obtained.
FIG. 15 illustrates one embodiment of a calibration method for a multi-player environment. The positional information gained from the devices sensors (accelerometers, GPS, compass, depth-cam, etc.) is transmitted to other linked devices to enhance the collaboratively maintained data in the virtual space. In one embodiment for creating a common shared space synchronized to a common reference point 1502 , a first player 1504 A synchronizes her device into the 3D space with respect to reference point 1502 . Other players in the shared space establish a communication link with the first player to exchange position and game information. The relative position can be obtain in different ways, such as u
CLAIMS
Claims ( 16 )
What is claimed is:
1. A method comprising:
synchronizing a device to a reference point in a three-dimensional (3D) space, wherein the 3D space is associated with a first coordinate system and the reference point is viewable via a camera of the device at synchronization and one or more other times after synchronization to relocate and use the reference point, where synchronization includes calibrating to an origin point of the device to the reference point;
determining by the device a first position and a first orientation of the device within the 3D space based on inertial information captured by the device;
generating by the device a virtual reality (VR) space based on the reference point, wherein the VR space includes virtual objects, wherein a second coordinate system of the VR space is based on the first coordinate system of the 3D space such that the first position within the 3D space is translated to a second position in the VR space; and
presenting a first view of the VR space based on the first position of the device and the first orientation of the device relative to the reference point, such that the first view and changes in the first view are independent of a character in the VR space.
2. The method of claim 1 , wherein the first position and first orientation are based on at least one image captured from the camera of the device.
3. The method of claim 1 , further comprising:
capturing one or more images of the 3D space with the camera of the device;
identifying locations of one or more static features in the one or more images; and
correcting the first position and first orientation of the device based on identified locations of the one or more static features.
4. The method of claim 1 , wherein the 3D space comprises a real-world environment.
5. The method of claim 1 , further comprising:
determining a second position and a second orientation of the device in the 3D space based on the inertial information; and
presenting a second view of the VR space based on the second position and the second orientation of the device.
6. The method of claim 1 , wherein the device is taken from a group consisting essentially of:
a portable gaming device;
a handheld portable gaming device;
a game controller;
a cell phone;
a head mounted display;
a notebook;
a tablet;
a netbook; and
a PDA.
7. The method of claim 1 , wherein the synchronizing comprises at least one of the following:
tapping the device;
placing the first device at the reference point;
providing an input to a user interface; and
engaging an actuator.
8. A method comprising:
synchronizing a device to a reference point in a three-dimensional (3D) space, wherein the 3D space is associated with a first coordinate system, and the reference point is viewable via a camera of the device at synchronization and one or more other times after synchronization to relocate and use the reference point, where synchronization includes calibrating to an origin point of the device to the reference point;
determining by the device a first position and a first orientation of the device within the 3D space based on inertial information captured by the device;
generating by the device an augmented reality (AR) space based on the reference point, wherein the AR space includes virtual objects and real objects, wherein a second coordinate system of the AR space is based on the first coordinate system of the 3D space such that a first position within the 3D space is translated to a second position in the AR space; and
presenting a first view of the AR space based on the first position of the device and the first orientation of the device relative to the reference point, such that the first view and changes in the first view are independent of a character in the VR space.
9. The method of claim 8 , further comprising:
capturing one or more images of the 3D space with the camera of the device;
identifying locations of one or more static features in the one or more images; and
correcting the first position and first orientation of the device based on identified locations of the one or more static features.
10. The method of claim 8 , wherein the 3D space comprises a real-world environment.
11. The method of claim 8 , wherein the device is taken from a group consisting essentially of:
a portable gaming device;
a handheld portable gaming device;
a game controller;
a cell phone;
a head mounted display;
a notebook;
a tablet;
a netbook; and
a PDA.
12. A portable handheld device for creating virtual reality, the portable handheld device comprising:
a camera and inertial system for synchronizing the device to a reference point in a three-dimensional (3D) space, and the reference point is viewable via the camera of the device at synchronization and one or more other times after synchronization to relocate and use the reference point, where synchronization includes calibrating to an origin point of the device to the reference point, wherein the 3D space is associated with a first coordinate system, wherein the camera and inertial system is further configured for determining a first position of the device and a first orientation of the device in the 3D space based on inertial information captured by the inertial system of the device;
a processor of the device for generating a virtual reality (VR) space based on the reference point, wherein the VR space includes virtual objects, wherein a second coordinate system of the VR space is based on the first coordinate system of the 3D space such that a first position within the 3D space is translated to a second position in the VR space, wherein the processor is further configured for generating a first view of the VR space based on the first position of the device and the first orientation of the device relative to the reference point, such that the first view and changes in the first view are independent of a character in the VR space; and
a display for presenting the first view of the VR space.
13. The portable handheld device of claim 12 ,
wherein the camera and inertial system is further configured for tracking the device within the 3D space, and for determining a second position and a second orientation of the device in the 3D space based on the inertial information, and
wherein the processor is further configured for presenting a second view of the VR space based on the second position of the device and the second orientation of the device.
14. The portable handheld device of claim 12 , wherein the camera and inertial system includes at least one location tracking module for capturing the inertial information, wherein the at least one location tracking module is taken from a group consisting of:
a 3-axis accelerometer;
a 3 axis gyroscope;
a camera;
a GPS module; and
a depth camera.
15. The portable handheld device of claim 12 , further comprising:
the camera being used for capturing one or more images of the 3D space,
wherein the camera and inertial system is further configured for identifying locations of one or more static features in the one or more images, and correcting the first position and first orientation of the device based on identified locations of the one or more static features.
16. The portable handheld device of claim 12 , wherein the portable handheld device is taken from a group consisting essentially of:
a portable gaming device;
a handheld portable gaming device;
a game controller;
a cell phone;
a head mounted display;
a notebook;
a tablet;
a netbook; and
a PDA.
US15/371,151
2009-12-24
2016-12-06
Tracking position of device inside-out for virtual reality interactivity
Active
2031-01-17
US10535153B2
( en )
Priority Applications (2)
Application Number
Priority Date
Filing Date
Title
US15/371,151
US10535153B2
( en )
2009-12-24
2016-12-06
Tracking position of device inside-out for virtual reality interactivity
US16/742,831
US11244469B2
( en )
2010-03-05
2020-01-14
Tracking position of device inside-out for augmented reality interactivity
Applications Claiming Priority (10)
Application Number
Priority Date
Filing Date
Title
US12/647,291
US8620213B2
( en )
2009-12-24
2009-12-24
Wireless device pairing methods
US12/647,296
US8463182B2
( en )
2009-12-24
2009-12-24
Wireless device pairing and grouping methods
US12/647,299
US9167071B2
( en )
2008-06-24
2009-12-24
Wireless device multimedia feed switching
US31125110P
2010-03-05
2010-03-05
US31902110P
2010-03-30
2010-03-30
US32376210P
2010-04-13
2010-04-13
US12/973,827
US8537113B2
( en )
2010-03-05
2010-12-20
Calibration of portable devices in a shared virtual space
US14/017,208
US8717294B2
( en )
2010-03-05
2013-09-03
Calibration of portable devices in a shared virtual space
US14/260,208
US9513700B2
( en )
2009-12-24
2014-04-23
Calibration of portable devices in a shared virtual space
US15/371,151
US10535153B2
( en )
2009-12-24
2016-12-06
Tracking position of device inside-out for virtual reality interactivity
Related Parent Applications (1)
Application Number
Title
Priority Date
Filing Date
US14/260,208
Continuation
US9513700B2
( en )
2009-12-24
2014-04-23
Calibration of portable devices in a shared virtual space
Related Child Applications (1)
Application Number
Title
Priority Date
Filing Date
US16/742,831
Continuation
US11244469B2
( en )
2010-03-05
2020-01-14
Tracking position of device inside-out for augmented reality interactivity
Publications (2)
Publication Number
Publication Date
US20170084051A1
US20170084051A1 ( en )
2017-03-23
US10535153B2
true
US10535153B2 ( en )
2020-01-14
Family
ID=44530901
Family Applications (8)
Application Number
Title
Priority Date
Filing Date
US12/947,290
Active
2031-11-16
US8730156B2
( en )
2006-03-06
2010-11-16
Maintaining multiple views on a shared stable virtual space
US12/973,827
Active
2031-10-30
US8537113B2
( en )
2008-10-27
2010-12-20
Calibration of portable devices in a shared virtual space
US14/017,208
Active
US8717294B2
( en )
2009-12-24
2013-09-03
Calibration of portable devices in a shared virtual space
US14/260,216
Active
US9310883B2
( en )
2010-03-05
2014-04-23
Maintaining multiple views on a shared stable virtual space
US14/260,208
Active
2031-08-16
US9513700B2
( en )
2009-12-24
2014-04-23
Calibration of portable devices in a shared virtual space
US15/089,360
Active
2030-12-15
US10424077B2
( en )
2010-03-05
2016-04-01
Maintaining multiple views on a shared stable virtual space
US15/371,151
Active
2031-01-17
US10535153B2
( en )
2009-12-24
2016-12-06
Tracking position of device inside-out for virtual reality interactivity
US16/742,831
Active
US11244469B2
( en )
2010-03-05
2020-01-14
Tracking position of device inside-out for augmented reality interactivity
Family Applications Before (6)
Application Number
Title
Priority Date
Filing Date
US12/947,290
Active
2031-11-16
US8730156B2
( en )
2006-03-06
2010-11-16
Maintaining multiple views on a shared stable virtual space
US12/973,827
Active
2031-10-30
US8537113B2
( en )
2008-10-27
2010-12-20
Calibration of portable devices in a shared virtual space
US14/017,208
Active
US8717294B2
( en )
2009-12-24
2013-09-03
Calibration of portable devices in a shared virtual space
US14/260,216
Active
US9310883B2
( en )
2010-03-05
2014-04-23
Maintaining multiple views on a shared stable virtual space
US14/260,208
Active
2031-08-16
US9513700B2
( en )
2009-12-24
2014-04-23
Calibration of portable devices in a shared virtual space
US15/089,360
Active
2030-12-15
US10424077B2
( en )
2010-03-05
2016-04-01
Maintaining multiple views on a shared stable virtual space
Family Applications After (1)
Application Number
Title
Priority Date
Filing Date
US16/742,831
Active
US11244469B2
( en )
2010-03-05
2020-01-14
Tracking position of device inside-out for augmented reality interactivity
Country Status (1)
Country
Link
US
( 8 )
US8730156B2
( en )
Cited By (4)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US11504608B2
( en )
*
2017-01-18
2022-11-22
Xvisio Technology Corp.
6DoF inside-out tracking game controller
US11525883B2
( en )
2017-10-23
2022-12-13
Beijing Boe Technology Development Co., Ltd.
Acquisition equipment, sound acquisition method, and sound source tracking system and method
US20230069452A1
( en )
*
2013-09-16
2023-03-02
Rovi Guides, Inc.
Methods and systems for presenting direction-specific media assets
US20250208695A1
( en )
*
2022-03-14
2025-06-26
Daniel BAIRAMIAN
An augmented reality point of view synchronisation system
Families Citing this family (455)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US8730156B2
( en )
*
2010-03-05
2014-05-20
Sony Computer Entertainment America Llc
Maintaining multiple views on a shared stable virtual space
US9250703B2
( en )
2006-03-06
2016-02-02
Sony Computer Entertainment Inc.
Interface with gaze detection and voice input
EP2023812B1
( en )
2006-05-19
2016-01-27
The Queen's Medical Center
Motion tracking system for real time adaptive imaging and spectroscopy
US8506404B2
( en )
*
2007-05-07
2013-08-13
Samsung Electronics Co., Ltd.
Wireless gaming method and wireless gaming-enabled mobile terminal
US10063848B2
( en )
*
2007-08-24
2018-08-28
John G. Posa
Perspective altering display system
US9703369B1
( en )
*
2007-10-11
2017-07-11
Jeffrey David Mullen
Augmented reality video game systems
US9037468B2
( en )
*
2008-10-27
2015-05-19
Sony Computer Entertainment Inc.
Sound localization for user in motion
EP2193825B1
( en )
*
2008-12-03
2017-03-22
Alcatel Lucent
Mobile device for augmented reality applications
US9258523B2
( en )
*
2009-11-12
2016-02-09
Arun Sobti & Associates Llc
Apparatus and method for integrating computing devices
JP2011197777A
( en )
*
2010-03-17
2011-10-06
Sony Corp
Information processing device, information processing method and program
KR20110116525A
( en )
*
2010-04-19
2011-10-26
ìì§ì ì 주ìíì¬
Image display apparatus for providing 3D object, system thereof, and operation control method thereof
US9132352B1
( en )
*
2010-06-24
2015-09-15
Gregory S. Rabin
Interactive system and method for rendering an object
US9159165B2
( en )
2010-07-13
2015-10-13
Sony Computer Entertainment Inc.
Position-dependent gaming, 3-D controller, and handheld as a remote
US20120026088A1
( en )
*
2010-08-01
2012-02-02
T-Mobile Usa, Inc.
Handheld device with projected user interface and interactive image
US20120032877A1
( en )
*
2010-08-09
2012-02-09
XMG Studio
Motion Driven Gestures For Customization In Augmented Reality Applications
WO2012048252A1
( en )
2010-10-07
2012-04-12
Aria Glassworks, Inc.
System and method for transitioning between interface modes in virtual and augmented reality applications
US10359545B2
( en )
2010-10-21
2019-07-23
Lockheed Martin Corporation
Fresnel lens with reduced draft facet visibility
US9632315B2
( en )
2010-10-21
2017-04-25
Lockheed Martin Corporation
Head-mounted display apparatus employing one or more fresnel lenses
US8253684B1
( en )
*
2010-11-02
2012-08-28
Google Inc.
Position and orientation determination for a mobile computing device
JP2012103746A
( en )
*
2010-11-05
2012-05-31
Avix Inc
Method for display control for fascinating viewer, digital signage system, and computer program
US9041743B2
( en )
*
2010-11-24
2015-05-26
Aria Glassworks, Inc.
System and method for presenting virtual and augmented reality scenes to a user
US9070219B2
( en )
2010-11-24
2015-06-30
Aria Glassworks, Inc.
System and method for presenting virtual and augmented reality scenes to a user
WO2012071466A2
( en )
2010-11-24
2012-05-31
Aria Glassworks, Inc.
System and method for acquiring virtual and augmented reality scenes by a user
US8953022B2
( en )
2011-01-10
2015-02-10
Aria Glassworks, Inc.
System and method for sharing virtual and augmented reality scenes between users and viewers
US20120182313A1
( en )
*
2011-01-13
2012-07-19
Pantech Co., Ltd.
Apparatus and method for providing augmented reality in window form
US9480913B2
( en )
2011-01-26
2016-11-01
WhitewaterWest Industries Ltd.
Interactive entertainment using a mobile device with object tagging and/or hyperlinking
US20120196684A1
( en )
*
2011-02-01
2012-08-02
David Richardson
Combining motion capture and timing to create a virtual gaming experience
JP5704962B2
( en )
*
2011-02-25
2015-04-22
任天å æ ªå¼ä¼ç¤¾
Information processing system, information processing method, information processing apparatus, and information processing program
US9118970B2
( en )
2011-03-02
2015-08-25
Aria Glassworks, Inc.
System and method for embedding and viewing media files within a virtual and augmented reality scene
US8810598B2
( en )
2011-04-08
2014-08-19
Nant Holdings Ip, Llc
Interference based augmented reality hosting platforms
US9160931B2
( en )
*
2011-04-08
2015-10-13
Hewlett-Packard Development Company, L.P.
Modifying captured image based on user viewpoint
EP3462286B1
( en )
2011-05-06
2026-02-11
Magic Leap, Inc.
Massive simultaneous remote digital presence world
US10120438B2
( en )
2011-05-25
2018-11-06
Sony Interactive Entertainment Inc.
Eye gaze to alter device behavior
US20120314899A1
( en )
2011-06-13
2012-12-13
Microsoft Corporation
Natural user interfaces for mobile image viewing
FR2976681B1
( en )
*
2011-06-17
2013-07-12
Inst Nat Rech Inf Automat
SYSTEM FOR COLOCATING A TOUCH SCREEN AND A VIRTUAL OBJECT AND DEVICE FOR HANDLING VIRTUAL OBJECTS USING SUCH A SYSTEM
US9886552B2
( en )
2011-08-12
2018-02-06
Help Lighting, Inc.
System and method for image registration of multiple video streams
EP2747641A4
( en )
2011-08-26
2015-04-01
Kineticor Inc
METHODS, SYSTEMS, AND DEVICES FOR CORRECTING INTRA-SCAN MOVEMENTS
US9155964B2
( en )
*
2011-09-14
2015-10-13
Steelseries Aps
Apparatus for adapting virtual gaming with real world information
US9606992B2
( en )
*
2011-09-30
2017-03-28
Microsoft Technology Licensing, Llc
Personal audio/visual apparatus providing resource management
EP2764421A4
( en )
*
2011-10-03
2015-05-13
Blackberry Ltd
Methods and devices to allow common user interface mode based on orientation
US20130111369A1
( en )
*
2011-10-03
2013-05-02
Research In Motion Limited
Methods and devices to provide common user interface mode based on images
EP2579128B1
( en )
*
2011-10-05
2017-11-22
Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V.
Portable device, virtual reality system and method
CA3048647C
( en )
2011-10-28
2022-08-16
Magic Leap, Inc.
System and method for augmented and virtual reality
US9785254B2
( en )
2011-11-01
2017-10-10
Qualcomm Incorporated
System and method for improving orientation data
JP5948429B2
( en )
2011-11-09
2016-07-06
ã¨ã³ãã¤ã¢ ãã¯ããã¸ã¼ ãã£ããããã¡ã³ã ã¨ã«ã¨ã«ã·ã¼
Virtual and augmented reality
JP6121647B2
( en )
*
2011-11-11
2017-04-26
ã½ãã¼æ ªå¼ä¼ç¤¾
Information processing apparatus, information processing method, and program
JP2015501984A
( en )
2011-11-21
2015-01-19
ãã³ã ãã¼ã«ãã£ã³ã°ã¹ ã¢ã¤ãã¼ï¼ã¨ã«ã¨ã«ã·ã¼
Subscription bill service, system and method
US20130141419A1
( en )
*
2011-12-01
2013-06-06
Brian Mount
Augmented reality with realistic occlusion
US9497501B2
( en )
2011-12-06
2016-11-15
Microsoft Technology Licensing, Llc
Augmented reality virtual monitor
US9230367B2
( en )
*
2011-12-13
2016-01-05
Here Global B.V.
Augmented reality personalization
WO2013095393A1
( en )
*
2011-12-20
2013-06-27
Intel Corporation
Augmented reality representations across multiple devices
WO2013100900A1
( en )
*
2011-12-27
2013-07-04
Intel Corporation
Full 3d interaction on mobile devices
US9041739B2
( en )
*
2012-01-31
2015-05-26
Microsoft Technology Licensing, Llc
Matching physical locations for shared virtual experience
US8610663B2
( en )
*
2012-02-06
2013-12-17
Lg Electronics Inc.
Portable device and method for controlling the same
EP2813069A4
( en )
*
2012-02-08
2016-12-07
Intel Corp
CREATING INCREASED REALITY FROM A REAL SCENE
JP5942456B2
( en )
*
2012-02-10
2016-06-29
ã½ãã¼æ ªå¼ä¼ç¤¾
Image processing apparatus, image processing method, and program
JPWO2013145614A1
( en )
*
2012-03-27
2015-12-10
ããã½ããã¯ï¼©ï½ããã¸ã¡ã³ãæ ªå¼ä¼ç¤¾
Information processing apparatus, server apparatus, information processing method, and program
JP5966510B2
( en )
*
2012-03-29
2016-08-10
ã½ãã¼æ ªå¼ä¼ç¤¾
Information processing system
US20130271355A1
( en )
2012-04-13
2013-10-17
Nokia Corporation
Multi-segment wearable accessory
GB2501925B
( en )
*
2012-05-11
2015-04-29
Sony Comp Entertainment Europe
Method and system for augmented reality
HK1207918A1
( en )
*
2012-05-16
2016-02-12
æ ¼çè´æ ¼ 丹尼å°
A system worn by a moving user for fully augmenting reality by anchoring virtual objects
US9035955B2
( en )
2012-05-16
2015-05-19
Microsoft Technology Licensing, Llc
Synchronizing virtual actor's performances to a speaker's voice
US9020203B2
( en )
2012-05-21
2015-04-28
Vipaar, Llc
System and method for managing spatiotemporal uncertainty
JP5966624B2
( en )
*
2012-05-29
2016-08-10
æ ªå¼ä¼ç¤¾ãªã³ã¼
Information processing apparatus and information display system
US8532675B1
( en )
2012-06-27
2013-09-10
Blackberry Limited
Mobile communication device user interface for manipulation of data items in a physical space
US9105210B2
( en )
2012-06-29
2015-08-11
Microsoft Technology Licensing, Llc
Multi-node poster location
US9035970B2
( en )
2012-06-29
2015-05-19
Microsoft Technology Licensing, Llc
Constraint based information inference
US9384737B2
( en )
2012-06-29
2016-07-05
Microsoft Technology Licensing, Llc
Method and device for adjusting sound levels of sources based on sound source priority
US9317971B2
( en )
2012-06-29
2016-04-19
Microsoft Technology Licensing, Llc
Mechanism to give holographic objects saliency in multiple spaces
KR20140010616A
( en )
*
2012-07-16
2014-01-27
íêµì ìíµì ì°êµ¬ì
Apparatus and method for processing manipulation of 3d virtual object
US10620902B2
( en )
*
2012-09-28
2020-04-14
Nokia Technologies Oy
Method and apparatus for providing an indication regarding content presented to another user
US9626799B2
( en )
2012-10-02
2017-04-18
Aria Glassworks, Inc.
System and method for dynamically displaying multiple virtual and augmented reality scenes on a single display
US10269179B2
( en )
2012-10-05
2019-04-23
Elwha Llc
Displaying second augmentations that are based on registered first augmentations
US10180715B2
( en )
2012-10-05
2019-01-15
Elwha Llc
Correlating user reaction with at least an aspect associated with an augmentation of an augmented view
US10713846B2
( en )
2012-10-05
2020-07-14
Elwha Llc
Systems and methods for sharing augmentation data
WO2015048890A1
( en )
*
2013-10-03
2015-04-09
Sulon Technologies Inc.
System and method for augmented reality and virtual reality applications
US9132342B2
( en )
2012-10-31
2015-09-15
Sulon Technologies Inc.
Dynamic environment and location based augmented reality (AR) systems
US9524585B2
( en )
2012-11-05
2016-12-20
Microsoft Technology Licensing, Llc
Constructing augmented reality environment with pre-computed lighting
US9658695B2
( en )
*
2012-11-08
2017-05-23
Cuesta Technology Holdings, Llc
Systems and methods for alternative control of touch-based devices
US9671874B2
( en )
*
2012-11-08
2017-06-06
Cuesta Technology Holdings, Llc
Systems and methods for extensions to alternative control of touch-based devices
US11372536B2
( en )
2012-11-20
2022-06-28
Samsung Electronics Company, Ltd.
Transition and interaction model for wearable electronic device
US11157436B2
( en )
2012-11-20
2021-10-26
Samsung Electronics Company, Ltd.
Services associated with wearable electronic device
US10423214B2
( en )
2012-11-20
2019-09-24
Samsung Electronics Company, Ltd
Delegating processing from wearable electronic device
US8994827B2
( en )
2012-11-20
2015-03-31
Samsung Electronics Co., Ltd
Wearable electronic device
US9477313B2
( en )
2012-11-20
2016-10-25
Samsung Electronics Co., Ltd.
User gesture input to wearable electronic device involving outward-facing sensor of device
US10551928B2
( en )
2012-11-20
2020-02-04
Samsung Electronics Company, Ltd.
GUI transitions on wearable electronic device
US10185416B2
( en )
2012-11-20
2019-01-22
Samsung Electronics Co., Ltd.
User gesture input to wearable electronic device involving movement of device
US11237719B2
( en )
2012-11-20
2022-02-01
Samsung Electronics Company, Ltd.
Controlling remote electronic device with wearable electronic device
US9030446B2
( en )
*
2012-11-20
2015-05-12
Samsung Electronics Co., Ltd.
Placement of optical sensor on wearable electronic device
US20140160162A1
( en )
*
2012-12-12
2014-06-12
Dhanushan Balachandreswaran
Surface projection device for augmented reality
US9264749B2
( en )
*
2012-12-13
2016-02-16
Microsoft Technology Licensing, Llc
Server GPU assistance for mobile GPU applications
US9710968B2
( en )
*
2012-12-26
2017-07-18
Help Lightning, Inc.
System and method for role-switching in multi-reality environments
US9076257B2
( en )
2013-01-03
2015-07-07
Qualcomm Incorporated
Rendering augmented reality based on foreground object
US20140201205A1
( en )
*
2013-01-14
2014-07-17
Disney Enterprises, Inc.
Customized Content from User Data
US9717461B2
( en )
2013-01-24
2017-08-01
Kineticor, Inc.
Systems, devices, and methods f