ABSTRACT
Abstract
The present disclosure relates to a method and device for augmented reality display of real physical model. By adopting the method and/or device of the present disclosure, the real physical model may be combined with its virtual physical model, and the display of the physical model may become more vivid with the help of AR technology.
Description
CROSS REFERENCE TO RELATED APPLICATIONS
This application claims priority to Chinese Patent Application Number 201710108466.7 filed on Feb. 27, 2017, the entire contents of which are incorporated herein by reference.
TECHNICAL FIELD
The present disclosure relates to a method and device for augmented reality display of real physical model.
BACKGROUND ART
Virtual Reality and Augmented Reality (VR/AR)
Virtual Reality (VR) is a type of human-computer interaction using computer and sensor technology. VR generates a virtual environment with realistic images, sounds and other sensations that replicate a real environment (or create an imaginary setting) to simulate a user's physical presence in this environment, and to interact with the environment.
Augmented Reality (AR) is a live view direct or indirect of a physical, real-world environment whose elements are augmented (or supplemented) by computer-generated sensory input such as sound, video, graphics or GPS data. AR computes the real-time position and angle of a camera, and adds corresponding images, videos, etc., to combine a virtual environment with the real-world environment and provide interaction between users and the environment. For example, in monitor-based AR, the computing device combines real world images captured by a camera with the virtual environment generated by the computing device, and outputs the combined view on the display for users.
Hardware components for AR usually include processor, display device, sensors and input devices. Suitable display devices may include head-mounted display (HMD), eyeglasses or monitors of computing devices such as smartphones and tablet computers. For example, HMDs use sensors to allow the system to align virtual information with the physical world and adjust accordingly with the user's head movements. Eyeglasses often use cameras to intercept the real world view and re-display its augmented view through the eye pieces.
With continuous improvement of the computation ability of portable electronic products, augmented reality is now widely used in various fields.
Simultaneous Localization and Mapping (SLAM)
In order to allow a user to âwanderâ in a virtual or augmented environment, one needs to consider the moving of a user wearing an AR/VR device. In this situation, the AR/VR device can be regarded as a robot, which leads to the problem of vision recognition. Simultaneous Localization and Mapping (SLAM) solves the problem. SLAM helps a robot locate itself while moving based on its estimated location and the map, and build an incremental map of the environment to achieve self-location and navigation. In AR/VR, location information may be provided by SLAM, and an AR/VR device may build a map and generate an environment using the map.
With continuous improvement of AR/VR technologies, it is proposed to apply AR/VR to real physical model display, for example, to sand table display.
Sand table display is needed in various fields, such as military affairs, real estate, education, etc., as an important way to convey information. However, traditional sand table display has its drawbacks.
First, the cost of traditional sand tables is relatively high, and it is difficult to be modified in real-time. Second, traditional sand tables need to be presented in the same physical space with the audience and therefore is not suitable for remote presentation. Third, traditional sand tables have limited interaction methods with audiences or operators.
On the other hand, virtual sand table display based on VR/AR may be employed. Usually, a virtual sand table is overlapped on a plane surface, and the audience need to wear special devices (for example, special helmet or glasses) to watch the virtual or augmented effect. This kind of display has high demands on devices, and does not support simultaneous watch of a plurality of people.
Therefore, the present disclosure proposes to combine virtual information into real physical model (such as, a sand table) using AR technology. In this way, the virtual physical model may be combined with the real physical model to provide a more vivid display of the real physical model.
SUMMARY
According to one aspect of the present disclosure, a method for augmented reality display of real physical model is proposed. The method comprises: an initialization step, comprising providing a real physical model, obtaining a virtual model from the real physical model using 3D reconstruction, combining the virtual model and the real physical model; and an augmented reality display step, for displaying an augmented reality scene of the real physical model by at least one display device.
According to another aspect of the present disclosure, a system for augmented reality display of real physical model is proposed. The system comprises: a virtual model construction device, for constructing a virtual model from a real physical model using 3D reconstruction; a combination device, for combining the virtual model and the real physical model; and an augmented reality display device, for displaying an augmented reality scene of the real physical model.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only, and are not restrictive of the invention.
BRIEF DESCRIPTION OF THE DRAWINGS
In order to illustrate the solutions of embodiments of the present disclosure more clearly, the drawings used in connection with the embodiments will be briefly described below. It should be understood that the following drawings illustrate only certain embodiments of the present disclosure, and the scope of the present disclosure is not limited thereto. In the drawings:
FIG. 1 shows the specific steps in the initialization phase;
FIG. 2 shows an exemplary real sand table;
FIG. 3 shows the virtual sand table obtained from the real sand table of FIG. 2 using 3D reconstruction;
FIG. 4 shows the specific steps in the virtual-real sand tables combination step;
FIG. 5 shows the real sand table disposed with an identification graph;
FIG. 6 shows the 3D point cloud computed with reference to the real sand table of FIG. 5 ;
FIG. 7 shows the combined real and virtual sand tables;
FIG. 8 shows the specific steps in the augmented reality display phase;
FIG. 9A and FIG. 9B show exemplary interactive scenarios of the combined sand table.
FIG. 10 shows an interactive control example based on an ocean scenario of the sand table;
FIG. 11 shows an interactive control example based on an atmosphere scenario of the sand table;
FIG. 12 shows an interactive control example based on a chemical substance scenario of the sand table;
FIG. 13 shows an AR glasses device applied to the device of the present disclosure.
EMBODIMENTS OF THE INVENTION
The present disclosure will now be described more comprehensively hereinafter with reference to the accompanying drawings, in which various exemplary embodiments of the invention are shown. This invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.
With reference to the accompanying drawings, a method and device for displaying a real physical model based on AR technology will now be described in detail in relation to a sand table, which serves as a non-limiting example of the present disclosure.
According to one embodiment of the present disclosure, the method comprises in general an initialization phase and an AR display phase. Now, each phase will be described in detail.
Initialization Phase
In this phase, it rebuilds a virtual sand table based on the real sand table using 3D reconstruction technology, and then combine the virtual sand table with the real sand table.
FIG. 1 shows the specific steps in this phase, and each step will now be described in detail.
Step 1100 , providing a real sand table.
In this step, a user may design a real sand table according to actual needs.
FIG. 2 shows an exemplary sand table, which shows the geographic information of North China Plain.
Take the sand table shown in FIG. 2 as an example, the sand table may be made using frame, stool, sand, mud, landmarks, etc. The steps for manufacturing the sand table may comprise: Setting a sand table by disposing sands within the frame and flattening and compressing the sand to create a plane with the lowest contour line; Building landform by drawing locations of mountain tops, mountain pass, rivers and the like, and stacking sands to a certain level to represent these locations; and Providing landmarks by disposing markers indicating the name of the region, river, mountain and the like on corresponding positions.
Step 1200 , reconstructing a virtual sand table using 3D reconstruction technology.
In this step, it uses a camera to scan the real sand table provided in Step 1100 , and then uses 3D reconstruction technology to obtain a corresponding virtual sand table.
In one example, it reconstructs a 3D model using ToF (Time of Flight) camera. ToF camera resolves distance by measuring the time-of-flight of a light signal between the camera and the subject for each point of the image. More specifically, it continuously transmits light pulses to an object, and then receives lights returned from the object using sensors. The distance to the object can be obtained by calculating the time of flight of the light pulses. In this embodiment, it uses a ToF camera to scan the real sand table and captures the 3D data of the whole scene. Then, it rebuilds a 3D model under a unitary coordinate based on the 3D data.
According to another example, it reconstructs a 3D model based on images. 3D Reconstruction based on images generally comprises of following steps:
Obtaining images: it uses a camera to obtain 2D images of a 3D object; Calibrating the camera: by calibrating the camera, it builds an imaging model and determines the position and parameters of the camera so as to determine the relationship between the space coordinate of the 3D object and its pixel; Extracting the features: based on feature extraction algorithms, it extracts features such as point features, line features or regional features of the 3D object; Stereo matching: based on the extracted features, it builds corresponding relations between 2D images; in other words, it matches the different image points of the same physical spatial point in two or more 2D images; 3D reconstruction: based on a relatively accurate matching result and with reference to the calibration parameters of the camera, it reconstructs the 3D model of the object.
It should be understood that other 3D reconstruction technologies may be used and are also included in the scope of the present disclosure.
FIG. 3 shows the virtual sand table obtained by 3D reconstruction of the real sand table of FIG. 2 .
Step 1300 , combining the real sand table and the virtual sand model.
FIG. 4 shows the specific steps of combining the real sand table with the virtual sand table. Each step will now be described in detail.
Step 1310 , building a world coordinate system.
In this step, it disposes a 2D identification graph on the real sand table. For example, it disposes an identification graph in the same plane as the main plane of the real sand table, and then fixes the relative position between the identification graph and the real sand table.
Then, it builds a world coordinate system based on the identification graph. For example, the origin of the world coordinate system may be located on the center of the identification graph, and the x-y plane of the world coordinate system may be within the plane where the identification graph lies.
FIG. 5 shows the real sand table with an identification graph disposed thereon, wherein the grey image in the bottom right corner is the identification graph.
Step 1320 , establishing the 3D point cloud of the feature points.
In this step, it uses a camera to scan the real sand table, and calculates the 3D coordinate of the feature points extracted from the real sand table under the world coordinate system obtained in step 1310 . Then, it stores the calculated point cloud.
Here, feature points refer to the significant points in the image. For example, for real sand table, its feature points may include mountain peaks or valleys that change significantly, or cities or sights that need to be displayed during presentation.
In this embodiment, it uses ORB (Oriented FAST and Rotated BRIEF) to extract feature points. The ORB is proposed in an article named âORB: An Efficient Alternative to SIFT or SURFâ by Ethan Rublee, Vincent Rabaud, Kurt Konolige and Gary R. Bradski in 2011.
ORB is a fusion of FAST keypoint (i.e. feature point) detector and BRIEF (Binary Robust Independent Elementary Features) descriptor with modifications to enhance the
CROSS REFERENCE TO RELATED APPLICATIONS
This application claims priority to Chinese Patent Application Number 201710108466.7 filed on Feb. 27, 2017, the entire contents of which are incorporated herein by reference.
TECHNICAL FIELD
The present disclosure relates to a method and device for augmented reality display of real physical model.
BACKGROUND ART
Virtual Reality and Augmented Reality (VR/AR)
Virtual Reality (VR) is a type of human-computer interaction using computer and sensor technology. VR generates a virtual environment with realistic images, sounds and other sensations that replicate a real environment (or create an imaginary setting) to simulate a user's physical presence in this environment, and to interact with the environment.
Augmented Reality (AR) is a live view direct or indirect of a physical, real-world environment whose elements are augmented (or supplemented) by computer-generated sensory input such as sound, video, graphics or GPS data. AR computes the real-time position and angle of a camera, and adds corresponding images, videos, etc., to combine a virtual environment with the real-world environment and provide interaction between users and the environment. For example, in monitor-based AR, the computing device combines real world images captured by a camera with the virtual environment generated by the computing device, and outputs the combined view on the display for users.
Hardware components for AR usually include processor, display device, sensors and input devices. Suitable display devices may include head-mounted display (HMD), eyeglasses or monitors of computing devices such as smartphones and tablet computers. For example, HMDs use sensors to allow the system to align virtual information with the physical world and adjust accordingly with the user's head movements. Eyeglasses often use cameras to intercept the real world view and re-display its augmented view through the eye pieces.
With continuous improvement of the computation ability of portable electronic products, augmented reality is now widely used in various fields.
Simultaneous Localization and Mapping (SLAM)
In order to allow a user to âwanderâ in a virtual or augmented environment, one needs to consider the moving of a user wearing an AR/VR device. In this situation, the AR/VR device can be regarded as a robot, which leads to the problem of vision recognition. Simultaneous Localization and Mapping (SLAM) solves the problem. SLAM helps a robot locate itself while moving based on its estimated location and the map, and build an incremental map of the environment to achieve self-location and navigation. In AR/VR, location information may be provided by SLAM, and an AR/VR device may build a map and generate an environment using the map.
With continuous improvement of AR/VR technologies, it is proposed to apply AR/VR to real physical model display, for example, to sand table display.
Sand table display is needed in various fields, such as military affairs, real estate, education, etc., as an important way to convey information. However, traditional sand table display has its drawbacks.
First, the cost of traditional sand tables is relatively high, and it is difficult to be modified in real-time. Second, traditional sand tables need to be presented in the same physical space with the audience and therefore is not suitable for remote presentation. Third, traditional sand tables have limited interaction methods with audiences or operators.
On the other hand, virtual sand table display based on VR/AR may be employed. Usually, a virtual sand table is overlapped on a plane surface, and the audience need to wear special devices (for example, special helmet or glasses) to watch the virtual or augmented effect. This kind of display has high demands on devices, and does not support simultaneous watch of a plurality of people.
Therefore, the present disclosure proposes to combine virtual information into real physical model (such as, a sand table) using AR technology. In this way, the virtual physical model may be combined with the real physical model to provide a more vivid display of the real physical model.
SUMMARY
According to one aspect of the present disclosure, a method for augmented reality display of real physical model is proposed. The method comprises: an initialization step, comprising providing a real physical model, obtaining a virtual model from the real physical model using 3D reconstruction, combining the virtual model and the real physical model; and an augmented reality display step, for displaying an augmented reality scene of the real physical model by at least one display device.
According to another aspect of the present disclosure, a system for augmented reality display of real physical model is proposed. The system comprises: a virtual model construction device, for constructing a virtual model from a real physical model using 3D reconstruction; a combination device, for combining the virtual model and the real physical model; and an augmented reality display device, for displaying an augmented reality scene of the real physical model.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only, and are not restrictive of the invention.
BRIEF DESCRIPTION OF THE DRAWINGS
In order to illustrate the solutions of embodiments of the present disclosure more clearly, the drawings used in connection with the embodiments will be briefly described below. It should be understood that the following drawings illustrate only certain embodiments of the present disclosure, and the scope of the present disclosure is not limited thereto. In the drawings:
FIG. 1 shows the specific steps in the initialization phase;
FIG. 2 shows an exemplary real sand table;
FIG. 3 shows the virtual sand table obtained from the real sand table of FIG. 2 using 3D reconstruction;
FIG. 4 shows the specific steps in the virtual-real sand tables combination step;
FIG. 5 shows the real sand table disposed with an identification graph;
FIG. 6 shows the 3D point cloud computed with reference to the real sand table of FIG. 5 ;
FIG. 7 shows the combined real and virtual sand tables;
FIG. 8 shows the specific steps in the augmented reality display phase;
FIG. 9A and FIG. 9B show exemplary interactive scenarios of the combined sand table.
FIG. 10 shows an interactive control example based on an ocean scenario of the sand table;
FIG. 11 shows an interactive control example based on an atmosphere scenario of the sand table;
FIG. 12 shows an interactive control example based on a chemical substance scenario of the sand table;
FIG. 13 shows an AR glasses device applied to the device of the present disclosure.
EMBODIMENTS OF THE INVENTION
The present disclosure will now be described more comprehensively hereinafter with reference to the accompanying drawings, in which various exemplary embodiments of the invention are shown. This invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.
With reference to the accompanying drawings, a method and device for displaying a real physical model based on AR technology will now be described in detail in relation to a sand table, which serves as a non-limiting example of the present disclosure.
According to one embodiment of the present disclosure, the method comprises in general an initialization phase and an AR display phase. Now, each phase will be described in detail.
Initialization Phase
In this phase, it rebuilds a virtual sand table based on the real sand table using 3D reconstruction technology, and then combine the virtual sand table with the real sand table.
FIG. 1 shows the specific steps in this phase, and each step will now be described in detail.
Step 1100 , providing a real sand table.
In this step, a user may design a real sand table according to actual needs.
FIG. 2 shows an exemplary sand table, which shows the geographic information of North China Plain.
Take the sand table shown in FIG. 2 as an example, the sand table may be made using frame, stool, sand, mud, landmarks, etc. The steps for manufacturing the sand table may comprise: Setting a sand table by disposing sands within the frame and flattening and compressing the sand to create a plane with the lowest contour line; Building landform by drawing locations of mountain tops, mountain pass, rivers and the like, and stacking sands to a certain level to represent these locations; and Providing landmarks by disposing markers indicating the name of the region, river, mountain and the like on corresponding positions.
Step 1200 , reconstructing a virtual sand table using 3D reconstruction technology.
In this step, it uses a camera to scan the real sand table provided in Step 1100 , and then uses 3D reconstruction technology to obtain a corresponding virtual sand table.
In one example, it reconstructs a 3D model using ToF (Time of Flight) camera. ToF camera resolves distance by measuring the time-of-flight of a light signal between the camera and the subject for each point of the image. More specifically, it continuously transmits light pulses to an object, and then receives lights returned from the object using sensors. The distance to the object can be obtained by calculating the time of flight of the light pulses. In this embodiment, it uses a ToF camera to scan the real sand table and captures the 3D data of the whole scene. Then, it rebuilds a 3D model under a unitary coordinate based on the 3D data.
According to another example, it reconstructs a 3D model based on images. 3D Reconstruction based on images generally comprises of following steps:
Obtaining images: it uses a camera to obtain 2D images of a 3D object; Calibrating the camera: by calibrating the camera, it builds an imaging model and determines the position and parameters of the camera so as to determine the relationship between the space coordinate of the 3D object and its pixel; Extracting the features: based on feature extraction algorithms, it extracts features such as point features, line features or regional features of the 3D object; Stereo matching: based on the extracted features, it builds corresponding relations between 2D images; in other words, it matches the different image points of the same physical spatial point in two or more 2D images; 3D reconstruction: based on a relatively accurate matching result and with reference to the calibration parameters of the camera, it reconstructs the 3D model of the object.
It should be understood that other 3D reconstruction technologies may be used and are also included in the scope of the present disclosure.
FIG. 3 shows the virtual sand table obtained by 3D reconstruction of the real sand table of FIG. 2 .
Step 1300 , combining the real sand table and the virtual sand model.
FIG. 4 shows the specific steps of combining the real sand table with the virtual sand table. Each step will now be described in detail.
Step 1310 , building a world coordinate system.
In this step, it disposes a 2D identification graph on the real sand table. For example, it disposes an identification graph in the same plane as the main plane of the real sand table, and then fixes the relative position between the identification graph and the real sand table.
Then, it builds a world coordinate system based on the identification graph. For example, the origin of the world coordinate system may be located on the center of the identification graph, and the x-y plane of the world coordinate system may be within the plane where the identification graph lies.
FIG. 5 shows the real sand table with an identification graph disposed thereon, wherein the grey image in the bottom right corner is the identification graph.
Step 1320 , establishing the 3D point cloud of the feature points.
In this step, it uses a camera to scan the real sand table, and calculates the 3D coordinate of the feature points extracted from the real sand table under the world coordinate system obtained in step 1310 . Then, it stores the calculated point cloud.
Here, feature points refer to the significant points in the image. For example, for real sand table, its feature points may include mountain peaks or valleys that change significantly, or cities or sights that need to be displayed during presentation.
In this embodiment, it uses ORB (Oriented FAST and Rotated BRIEF) to extract feature points. The ORB is proposed in an article named âORB: An Efficient Alternative to SIFT or SURFâ by Ethan Rublee, Vincent Rabaud, Kurt Konolige and Gary R. Bradski in 2011.
ORB is a fusion of FAST keypoint (i.e. feature point) detector and BRIEF (Binary Robust Independent Elementary Features) descriptor with modifications to enhance the performance.
For detectors, ORB uses FAST to find keypoints, then applies Harris corner measure to find top N points among them. It also uses pyramid to produce multiscale-features. To allow FAST to compute the orientation, it computes the intensity weighted centroid of the patch with located corner at center. The direction of the vector from this corner point to centroid gives the orientation. To improve the rotation invariance, moments are computed with x and y which should be in a circular region of radius r, where r is the size of the patch.
For descriptors, ORB uses BRIEF descriptors. ORB âsteersâ BRIEF according to the orientation of keypoints. For any feature set of n binary tests at location (x i , y i ), it defines a 2Ãn matrix S which contains the coordinates of these pixels. Then, using the orientation of patch θ, it find its rotation matrix and rotates the S to get the steered (rotated) version S θ .
It should be understood that other algorithms for extracting feature points may also be used, and are also included in the present disclosure.
Then, in order to match the feature points and establish a corresponding relation between two consecutive images, the Hamming distances between each feature point in a second image and all of the feature points in a first image are calculated. The calculated Hamming distance may be referred to as D(V p , V q ), wherein V p is the feature vector of a certain feature point p in the second image, and V q is the feature vector of the closest feature point q in the first image. The smaller the D(V p , V q ) is, the more similar the two feature points are. The two points with the minimum Hamming distance form a pair.
After matching the feature points, the movement of the camera may be estimated. Then, based on the estimated moving information, it computes the spatial position of each feature point. In other words, it computes the 3D coordinate of each feature point under the world coordinate system so as to establish the 3D point cloud of the feature points.
FIG. 6 shows the 3D point cloud computed with reference to the real sand table of FIG. 5 .
Step 1330 , computing the world coordinate of the virtual sand table.
In this step, based on the relative position of the identification graph and the real sand table, it computes the coordinate of the virtual sand table under the world coordinate system obtained in step 1310 .
More specifically, first, it computes the conversion coefficient between the real physical world size and the algorithm's pixel size; then, it measures the relative position between the center of the identification graph and the center of the real physical model; lastly, it converts the relative position to corresponding pixel size and obtains the world coordinate of the virtual sand table.
Then, it disposes the virtual sand table into the corresponding world coordinate system and aligns the virtual sand table with the real sand table, so that their origins, directions and sizes coincide. In this way, the virtual sand table and the real sand table are combined.
FIG. 7 shows the combined sand table.
It should be noted that although in the present embodiment, it computes the world coordinate of the virtual sand table after establishing the 3D point cloud, in other embodiments the world coordinate of the virtual sand table may be computed before establishing the 3D point cloud. For example, Step 1320 and Step 1330 may be conducted consecutively, reversely or independently.
AR Display Phase
In this phase, the combined sand table is displayed on at least one terminal device to show related information.
FIG. 8 shows the specific steps in the AR display phase.
Step
8100 , 3D point cloud sharing.
In this step, it sends the 3D point cloud obtained in step 1320 to at least one terminal device. Here, the terminal device may be a PC, a mobile phone or a VR/AR glasses, etc.
Step 8200 , localization and 3D rendering.
According to one example, in this step, each terminal device uses SLAM algorithm to locate itself while moving, and conducts corresponding 3D rendering. More specifically, the camera of the terminal device may be regarded as the robot in SLAM algorithm. Each device computes a location estimation and a map of the environment based on the 3D point cloud determined in step 1320 . Then, based on the computation result, it completes the 3D rendering of the virtual sand table.
This step ensures that the virtual sand table and the real sand table match in real time. In this way, each device running the SLAM algorithm independently may display the same virtual scene.
According to one embodiment of the present disclosure, two or more terminal devices (for example, PC, mobile phone and AR/VR glasses) may be connected to a common network and display the virtual scene simultaneously.
According to another embodiment of the present disclosure, one of the plurality of terminal devices may be selected as the master device, which controls and interacts with the AR scene. Other devices served as slave devices may synchronize with the master device and display the same AR scene controlled by the master device. For example, the operation of the master device may be converted into a specific format and sent to other slave devices. In this way, the plurality of terminal devices may perform the same operation.
According to yet another embodiment of the present disclosure, at least one terminal device may be placed close to the location where the real physical model lies, or be placed distant from the location where the real physical model lies.
Moreover, according to one embodiment of the present disclosure, interactive AR display may be performed according to actual needs. Interactive AR display may include visual interaction, voice interaction or gesture interaction, etc.
FIG. 9A and FIG. 9B show exemplary interactive scenarios of the combined sand table.
As shown in FIG. 9A , after combining the real and virtual sand tables, the sea displayed on the terminal device may be moving, with boats sailing thereon and planes flying in the sky.
As shown in FIG. 9B , markers are displayed above main cities and sights. By clicking the markers or saying the name of the city or sight, said city or sight may be selected, and a window showing pictures and information of the city or sight will be displayed on the terminal device.
Moreover, take the virtual geographic sand table in this embodiment for example, it may display geographic information including ocean, atmosphere and chemical substances. A user may interact with the virtual sand table by saying âoceanâ, âatmosphereâ, âchemical substanceâ, for example, and the AR scene displayed on the virtual sand table will change accordingly.
FIG. 10 shows an interactive control example based on an ocean scenario of the sand table.
It is known that the sea level will raise 4-5 meters when the global average temperature increases by 1° C. The ocean scenario may show how global warming influences the sea level. During presentation, when the operator waves up his/her hand or says âupâ, for example, it represents that the global average temperature increases by 1° C. Consequently, the sea level of the AR scene displayed on the terminal device will go up and submerge part of the land. Similarly, when the operator waves down his/her hand or says âdownâ, for example, it represents that the global average temperature decreases by 1° C. Consequently, the sea level of the AR scene displayed on the terminal device will go down and reveal part of the land. In this way, the operator may control the sand table via voice or gesture and the audiences may see the change of landform on their terminal devices.
FIG. 11 shows an interactive control example based on an atmosphere scenario of the sand table.
The atmosphere scenario may show the general atmospheric circulation at different heights from ground surface to high-speed air flow layer (e.g. 1000 hpa, 750 hpa, 500 hpa and 250 hpa), as well as atmospheric information including airflow, temperature, humidity, etc.
As shown in FIG. 11 , the operator may select the air flow layer and atmospheric information to be displayed via voice instructions (e.g., voice command âupâ) or gesture command, and the audience may see the general atmospheric circulation and atmospheric information in this layer on their terminal devices.
FIG. 12 shows an interactive control example based on a chemical substances scenario of the sand table.
Chemical substances scenario may show the distribution of chemical substances at different heights from ground surface to high-speed air flow layer (e.g. 1000 hpa, 750 hpa, 500 hpa and 250 hpa). The distribution of CO, CO 2 , SO 2 , for example, may be displayed.
As shown in FIG. 12 , the operator may select the air flow layer and chemical substance to be displayed via voice instructions (e.g., voice command âupâ) or gesture command, and the audiences may see the distribution of chemical substances in this layer on their terminal devices.
In one embodiment of the present disclosure, the terminal device may be AR glasses.
FIG. 13 shows the AR glasses applied to the system of the present disclosure. The AR glasses may be used to watch and control the AR scene based on the real physical model.
As shown in FIG. 13 , the AR glasses may include a glasses frame 1 , a pair of glasses legs 2 , a pair of clamping members 3 and soft supporting layers 4 . With the help of these components, the AR device may be stably held on the user's head. Each of the glasses legs 2 is mounted on one side of the glasses frame 1 respectively, and the end of each of the glasses legs 2 bends inwardly to form a first arc portion. The clamping member comprises an elastic clamping element 3 , which is disposed on an inner side of the glasses legs 2 . The end of the elastic clamping element 3 bends inwardly to form a second arc portion. Moreover, the elastic clamping element 3 may be made of stainless steel shrapnel, which may reinforce the clamping force of the glasses legs 2 . The soft support layers 4 may be placed on the inner side of the arc portion, and may be made of rubber layers and/or foam layers. With the soft supporting layers 4 , user comfortability can be improved. For example, the pressure on the user's head may be reduced, and the friction between the glasses legs and the user's head may be increased. In this way, the AR glasses will be more stable and comfortable to wear.
In addition, the AR glasses may further include light- shading lenses 11 disposed on the glasses frame 1 , a depth camera module 12 disposed at the center portion of the glasses frame 1 , and a light emitting module 13 disposed at each lateral portion of the glasses frame 1 . The depth camera module 12 and the light emitting module 13 are coupled to a control circuit, and the light emitting module 13 is in optical connection with the imaging lenses disposed behind the light- shading lenses 11 . For a user who wears the AR glasses, the imaging lenses function as a display screen. From said imaging lenses, display content of the AR glasses, such as a graphic user interface, may be observed. The imaging lenses may be, for example, semi-transparent and semi-reflective lenses/films, or holographic diffraction waveguide gratings, or other suitable imaging components.
Wearing the AR glasses, a user may watch the virtual physical model displayed on the imaging lens (screen), and the real physical model via imaging lens and light- shading lenses 11 . In other words, AR glasses may overlay the AR scene onto the real scene. In addition, the AR glasses may detect the shape or color of the real physical model via the depth camera module 12 and/or other camera modules, as well as detect gestures or other operations of the user.
It should be understood that the AR device of the present disclosure is not limit to AR glasses, and other AR devices, such AR helmets are also included in the scope of the present disclosure.
It should also be understood that in other embodiments, the AR device may also include other modules for detecting the user's operations, such as microphones, inertia sensors, etc. These modules may detect gesture signals, voice signals, head movement signals respectively, as so to determine the user's operation.
It should also be understood that the present disclosure is not limited to sand table AR interaction/display. The technical solution disclosed in the application may be applied to any physical models/objects, suitable to entertainment needs of personal consumers or to industrial needs of manufacturing, transporting, etc. For example, it may be applied in in-door environment or out-door public area, in education, industry, tourism, everyday-life, etc.
Technical Effects
The method and device for augmented reality display of real physical model has the following beneficial effects:
By adopting the method and/or device of the present disclosure, the real physical model may be combined with its virtual physical model, and the display of the physical model may become more vivid with the help of AR technology. Moreover, it allows the users to watch the same AR scene at different places and control the AR scene by various forms of interactions (e.g. via voice, gestures, etc.).
In addition, the method and/or device of the present disclosure allows a plurality of users to interact with the virtual scene via a plurality terminal devices. It also allows different terminal devices to share the same AR scene, making it suitable for remote display.
Claims ( 24 )
What is claimed is:
1. A method for augmented reality display of a real physical model, comprising:
an initialization step, comprising:
providing a real physical model, wherein the real physical model is a model that occupies a real physical space;
obtaining a virtual model from the real physical model using 3D reconstruction;
combining the virtual model and the real physical model, wherein combining the virtual model and the real physical model further comprises:
a world coordinate system establishing step, for establishing a world coordinate system based on an identification graph, said identification graph being disposed adjacent to the real physical model, wherein a relative position between the identification graph and the real physical model is fixed, wherein the identification graph is a 2D graph and a direction information of the identification graph corresponds to a direction information of the real physical model; and
a virtual model world coordinate computing step, for computing the world coordinate of the virtual model under the established world coordinate system; and
an augmented reality display step, for displaying an augmented reality scene of the real physical model by at least one display device.
2. The method of claim 1 , the initialization step further comprises:
a 3D point cloud constructing step, comprising: extracting a plurality of feature points of the real physical model, and computing the 3D coordinates of the plurality of feature points under the world coordinate system so as to form the 3D point cloud.
3. The method of claim 2 , wherein the augmented reality display step further comprises:
a 3D point cloud sharing step, for sending the 3D point cloud to the at least one display device;
a locating step, for locating movement of the at least one display device; and
a 3D rendering step, for rendering the virtual model displayed by the at least one display device based on the located movement thereof .
4. The method of claim 1 , wherein a center of the identification graph is used as an origin of the world coordinate system, wherein the virtual model world coordinate computing step further comprises:
computing a conversion coefficient between the real physical model's size and the algorithm's pixel size;
measuring the relative position between the center of the identification graph and the center of the real physical model; and
computing the world coordinate of the virtual model by converting the relative position to corresponding pixel size.
5. The method of claim 1 , wherein obtaining a virtual model from the real physical model using 3D reconstruction further comprises:
an image obtaining step, for obtaining 2D images of the real physical model;
a calibrating step, for determining corresponding relations between the real physical model's size and the pixel size of 2D images;
a feature extraction step, for extracting features of the real physical model;
a stereo matching step, for establishing matching results between 2D images based on said extracted features;
a 3D reconstruction step, for reconstructing a virtual model of the real physical model based on the matching results and the corresponding relations.
6. The method of claim 1 , further comprising:
an augmented reality scene controlling step, for allowing a user to control the augmented reality scene.
7. The method in claim 6 , wherein said control comprises:
at least one of gesture, voice, or head movement of a user that is detectable by the at least one display device.
8. The method in claim 7 , wherein the at least one display device comprises:
a master device and at least one slave device, wherein the master device controls the augmented reality scene, wherein each of the master device and at least one slave device is capable of displaying the augmented reality scene of the real physical model.
9. The method in claim 8 , further comprising:
receiving data from the master device, by the at least one slave device, and
displaying the augmented reality scene based on the data received from the master device, by the at least one slave device.
10. The method in claim 1 , wherein the real physical model is chosen from a geographic model, an architectural model, or a weather model.
11. The method in claim 1 , wherein the at least one display device is chosen from an AR device, a VR device, a mobile phone or a computer.
12. The method of claim 1 , wherein the at least one display device comprises at least one of:
a local display device placed close to the real physical model,
a remote display device placed distant from the real physical model.
13. The method of claim 1 , wherein the identification graph is coplanar with at least one plane of the real physical model, and the direction information of the identification graph corresponds to a direction information of the at least one plane of the real physical model.
14. A system for augmented reality display of real physical model, comprising:
a virtual model construction device, for constructing a virtual model from a real physical model using 3D reconstruction, wherein the real physical model is a model that occupies a real physical space;
a combination device, for combining the virtual model and the real physical model, wherein the combination device further comprises:
a world coordinate system establishing module, for establishing a world coordinate system based on an identification graph, said identification graph being disposed adjacent to the real physical model, wherein a relative position between the identification graph and the real physical model is fixed, wherein the identification graph is a 2D graph and a direction information of the identification graph corresponds to a direction information of the real physical model; and
a virtual model world coordinate computing module, for computing the world coordinate of the virtual model under the established world coordinate system; and
an augmented reality display device, for displaying an augmented reality scene of the real physical model.
15. The system of claim 14 , the combination device further comprises:
a 3D point cloud constructing module, for extracting a plurality of feature points of the real physical model, and computing 3D coordinates of the plurality of feature points under the world coordinate system so as to form the 3D point cloud.
16. The system of claim 15 , wherein the augmented reality display device further comprises:
a 3D point cloud sharing module, for sending the 3D point cloud to at least one display device;
a locating module, for locating movement of the at least one display device;
a 3D rendering module, for rendering the virtual model displayed by the at least one display device based on the located movement thereof.
17. The system of claim 14 , wherein a center of the identification graph is used as an origin of the world coordinate system, wherein the virtual model world coordinate computing module further comprises:
a conversion unit, for computing a conversion coefficient between the real physical model's size and the algorithm's pixel size;
a measuring unit, for measuring the relative position between the center of the identification graph and the center of the real physical model; and
a computing unit, for computing world coordinates of the virtual model by converting the relative position to corresponding pixel size.
18. The system of claim 14 , the virtual model constructing device further comprises:
an image obtaining module, for obtaining 2D images of the real physical model;
a calibrating module, for determining corresponding relations between the real physical model's size and the pixel size of 2D images;
a feature extraction module, for extracting features of the real physical model;
a stereo matching module, for establishing matching results between the 2D images based on said extracted features;
a 3D reconstruction module, for reconstructing a the virtual model of the real physical model based on the matching results and the corresponding relations.
19. The system of claim 14 , wherein the augmented reality display device further comprises an AR device, a VR device, a mobile phone or a computer.
20. The system of claim 14 , wherein the augmented reality display device further comprises:
an augmented reality scene control module, for allowing a user to control the augmented reality scene by interacting with the at least one display device.
21. The system of claim 20 , wherein the augmented reality scene control module further comprises:
a user control input unit, for receiving a gesture, voice, or head movement of the user.
22. The system of claim 20 , wherein the at least one display device comprises:
a master device and at least one slave device, wherein the master device controls the augmented reality scene, wherein each of the master device and at least one slave device is capable of displaying the augmented reality scene of the real physical model.
23. The system of claim 14 , wherein the at least one display device comprises at least one of:
a local display device placed close to the real physical model,
a remote display device placed distant from the real physical model.
24. The system of claim 14 , wherein the identification graph is coplanar with at least one plane of the real physical model, and the direction information of the identification graph corresponds to a direction information of the at least one plane of the real physical model.
US15/487,449
2017-02-27
2017-04-14
Method and device for augmented reality display of real physical model
Active
2037-07-14
US10181222B2
( en )
Applications Claiming Priority (3)
Application Number
Priority Date
Filing Date
Title
CN201710108466.7A
CN108510592B
( en )
2017-02-27
2017-02-27
Augmented reality display method of real physical model
CN201710108466.7
2017-02-27
CN201710108466
2017-02-27
Publications (2)
Publication Number
Publication Date
US20180247456A1
US20180247456A1 ( en )
2018-08-30
US10181222B2
true
US10181222B2 ( en )
2019-01-15
Family
ID=63246415
Family Applications (1)
Application Number
Title
Priority Date
Filing Date
US15/487,449
Active
2037-07-14
US10181222B2
( en )
2017-02-27
2017-04-14
Method and device for augmented reality display of real physical model
CLAIMS
Claims ( 24 )
What is claimed is:
1. A method for augmented reality display of a real physical model, comprising:
an initialization step, comprising:
providing a real physical model, wherein the real physical model is a model that occupies a real physical space;
obtaining a virtual model from the real physical model using 3D reconstruction;
combining the virtual model and the real physical model, wherein combining the virtual model and the real physical model further comprises:
a world coordinate system establishing step, for establishing a world coordinate system based on an identification graph, said identification graph being disposed adjacent to the real physical model, wherein a relative position between the identification graph and the real physical model is fixed, wherein the identification graph is a 2D graph and a direction information of the identification graph corresponds to a direction information of the real physical model; and
a virtual model world coordinate computing step, for computing the world coordinate of the virtual model under the established world coordinate system; and
an augmented reality display step, for displaying an augmented reality scene of the real physical model by at least one display device.
2. The method of claim 1 , the initialization step further comprises:
a 3D point cloud constructing step, comprising: extracting a plurality of feature points of the real physical model, and computing the 3D coordinates of the plurality of feature points under the world coordinate system so as to form the 3D point cloud.
3. The method of claim 2 , wherein the augmented reality display step further comprises:
a 3D point cloud sharing step, for sending the 3D point cloud to the at least one display device;
a locating step, for locating movement of the at least one display device; and
a 3D rendering step, for rendering the virtual model displayed by the at least one display device based on the located movement thereof .
4. The method of claim 1 , wherein a center of the identification graph is used as an origin of the world coordinate system, wherein the virtual model world coordinate computing step further comprises:
computing a conversion coefficient between the real physical model's size and the algorithm's pixel size;
measuring the relative position between the center of the identification graph and the center of the real physical model; and
computing the world coordinate of the virtual model by converting the relative position to corresponding pixel size.
5. The method of claim 1 , wherein obtaining a virtual model from the real physical model using 3D reconstruction further comprises:
an image obtaining step, for obtaining 2D images of the real physical model;
a calibrating step, for determining corresponding relations between the real physical model's size and the pixel size of 2D images;
a feature extraction step, for extracting features of the real physical model;
a stereo matching step, for establishing matching results between 2D images based on said extracted features;
a 3D reconstruction step, for reconstructing a virtual model of the real physical model based on the matching results and the corresponding relations.
6. The method of claim 1 , further comprising:
an augmented reality scene controlling step, for allowing a user to control the augmented reality scene.
7. The method in claim 6 , wherein said control comprises:
at least one of gesture, voice, or head movement of a user that is detectable by the at least one display device.
8. The method in claim 7 , wherein the at least one display device comprises:
a master device and at least one slave device, wherein the master device controls the augmented reality scene, wherein each of the master device and at least one slave device is capable of displaying the augmented reality scene of the real physical model.
9. The method in claim 8 , further comprising:
receiving data from the master device, by the at least one slave device, and
displaying the augmented reality scene based on the data received from the master device, by the at least one slave device.
10. The method in claim 1 , wherein the real physical model is chosen from a geographic model, an architectural model, or a weather model.
11. The method in claim 1 , wherein the at least one display device is chosen from an AR device, a VR device, a mobile phone or a computer.
12. The method of claim 1 , wherein the at least one display device comprises at least one of:
a local display device placed close to the real physical model,
a remote display device placed distant from the real physical model.
13. The method of claim 1 , wherein the identification graph is coplanar with at least one plane of the real physical model, and the direction information of the identification graph corresponds to a direction information of the at least one plane of the real physical model.
14. A system for augmented reality display of real physical model, comprising:
a virtual model construction device, for constructing a virtual model from a real physical model using 3D reconstruction, wherein the real physical model is a model that occupies a real physical space;
a combination device, for combining the virtual model and the real physical model, wherein the combination device further comprises:
a world coordinate system establishing module, for establishing a world coordinate system based on an identification graph, said identification graph being disposed adjacent to the real physical model, wherein a relative position between the identification graph and the real physical model is fixed, wherein the identification graph is a 2D graph and a direction information of the identification graph corresponds to a direction information of the real physical model; and
a virtual model world coordinate computing module, for computing the world coordinate of the virtual model under the established world coordinate system; and
an augmented reality display device, for displaying an augmented reality scene of the real physical model.
15. The system of claim 14 , the combination device further comprises:
a 3D point cloud constructing module, for extracting a plurality of feature points of the real physical model, and computing 3D coordinates of the plurality of feature points under the world coordinate system so as to form the 3D point cloud.
16. The system of claim 15 , wherein the augmented reality display device further comprises:
a 3D point cloud sharing module, for sending the 3D point cloud to at least one display device;
a locating module, for locating movement of the at least one display device;
a 3D rendering module, for rendering the virtual model displayed by the at least one display device based on the located movement thereof.
17. The system of claim 14 , wherein a center of the identification graph is used as an origin of the world coordinate system, wherein the virtual model world coordinate computing module further comprises:
a conversion unit, for computing a conversion coefficient between the real physical model's size and the algorithm's pixel size;
a measuring unit, for measuring the relative position between the center of the identification graph and the center of the real physical model; and
a computing unit, for computing world coordinates of the virtual model by converting the relative position to corresponding pixel size.
18. The system of claim 14 , the virtual model constructing device further comprises:
an image obtaining module, for obtaining 2D images of the real physical model;
a calibrating module, for determining corresponding relations between the real physical model's size and the pixel size of 2D images;
a feature extraction module, for extracting features of the real physical model;
a stereo matching module, for establishing matching results between the 2D images based on said extracted features;
a 3D reconstruction module, for reconstructing a the virtual model of the real physical model based on the matching results and the corresponding relations.
19. The system of claim 14 , wherein the augmented reality display device further comprises an AR device, a VR device, a mobile phone or a computer.
20. The system of claim 14 , wherein the augmented reality display device further comprises:
an augmented reality scene control module, for allowing a user to control the augmented reality scene by interacting with the at least one display device.
21. The system of claim 20 , wherein the augmented reality scene control module further comprises:
a user control input unit, for receiving a gesture, voice, or head movement of the user.
22. The system of claim 20 , wherein the at least one display device comprises:
a master device and at least one slave device, wherein the master device controls the augmented reality scene, wherein each of the master device and at least one slave device is capable of displaying the augmented reality scene of the real physical model.
23. The system of claim 14 , wherein the at least one display device comprises at least one of:
a local display device placed close to the real physical model,
a remote display device placed distant from the real physical model.
24. The system of claim 14 , wherein the identification graph is coplanar with at least one plane of the real physical model, and the direction information of the identification graph corresponds to a direction information of the at least one plane of the real physical model.
US15/487,449
2017-02-27
2017-04-14
Method and device for augmented reality display of real physical model
Active
2037-07-14
US10181222B2
( en )
Applications Claiming Priority (3)
Application Number
Priority Date
Filing Date
Title
CN201710108466.7A
CN108510592B
( en )
2017-02-27
2017-02-27
Augmented reality display method of real physical model
CN201710108466.7
2017-02-27
CN201710108466
2017-02-27
Publications (2)
Publication Number
Publication Date
US20180247456A1
US20180247456A1 ( en )
2018-08-30
US10181222B2
true
US10181222B2 ( en )
2019-01-15
Family
ID=63246415
Family Applications (1)
Application Number
Title
Priority Date
Filing Date
US15/487,449
Active
2037-07-14
US10181222B2
( en )
2017-02-27
2017-04-14
Method and device for augmented reality display of real physical model
Country Status (2)
Country
Link
US
( 1 )
US10181222B2
( en )
CN
( 1 )
CN108510592B
( en )
Families Citing this family (48)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US10902680B2
( en )
*
2018-04-03
2021-01-26
Saeed Eslami
Augmented reality application system and method
CN109377560A
( en )
*
2018-10-26
2019-02-22
å京ç工大å¦
A method for outdoor augmented reality military simulation training
CN109345634B
( en )
*
2018-10-26
2021-01-19
éåºå¤§å¦
Magnetic field demonstration method and device based on magnetic liquid and mixed reality
CN109636917B
( en )
*
2018-11-02
2023-07-18
å京微æè§çç§ææéå ¬å¸
Three-dimensional model generation method, device and hardware device
EP3877957A4
( en )
*
2018-11-07
2022-08-17
Services Pétroliers Schlumberger
Data structures for augmented reality planning of geographic locations
CN109579745A
( en )
*
2018-11-26
2019-04-05
æ±èç§æ大å¦
Novel house Area computing method based on augmented reality and cell phone software
CN109727319A
( en )
*
2018-11-26
2019-05-07
广ä¸çæ°è±¡æ¢æµæ°æ®ä¸å¿(广ä¸çæ°è±¡ææ¯è£ å¤ä¸å¿ã广ä¸çæ°è±¡ç§æå¹è®ä¸å¿)
A three-dimensional holographic scene construction method for meteorological detection equipment
CN109828658B
( en )
*
2018-12-17
2022-03-08
å½æä¸
Man-machine co-fusion remote situation intelligent sensing system
CN109545003B
( en )
*
2018-12-24
2022-05-03
å京å¡è·¯éä¿¡æ¯ææ¯æéå ¬å¸
Display method, display device, terminal equipment and storage medium
CN110058684B
( en )
*
2019-03-21
2022-07-01
æµ·åè¯ºäº¦è ¾æµ·æ´ç§æç 究é¢æéå ¬å¸
Geographic information interaction method, system and storage medium based on VR technology
CN110009985B
( en )
*
2019-03-27
2020-12-25
æ·±å³å¸é®åºä¿¡æ¯ææ¯æéå ¬å¸
Sand table making devices based on machine learning
CN109920315B
( en )
*
2019-03-29
2019-11-08
æµå大å¦
A Virtual α Particle Scattering Experimental System
CN110223556A
( en )
*
2019-04-16
2019-09-10
æå·ç¾åå°ç§ææéå ¬å¸
Immersion intelligence chemical industry analog simulation equipment
CN110262283B
( en )
*
2019-06-11
2022-08-23
è¿å½¢æ¶ç©ºç§æï¼å京ï¼æéå ¬å¸
Multi-scene visual robot simulation platform and method
CN110288716B
( en )
*
2019-06-14
2023-08-08
å京达佳äºèä¿¡æ¯ææ¯æéå ¬å¸
Image processing method, device, electronic equipment and storage medium
CN110310328B
( en )
*
2019-07-22
2021-04-30
é 客æºæ §ï¼å京ï¼ç§ææéå ¬å¸
Mixed reality operation registration method and device
CN111667393B
( en )
*
2019-08-26
2023-07-07
ç¦å»ºæ°å讯信æ¯ç§ææéå ¬å¸
Method and terminal for simulating raining in virtual scene
CN110879979B
( en )
*
2019-11-13
2024-01-02
æ³å·å¸èå¦é¢
An augmented reality system based on mobile terminals
CN112308954A
( en )
*
2019-11-26
2021-02-02
æµ·ååæ§æºæ §ç¯å¢å»ºè®¾éå¢æéå ¬å¸
Building model informatization and real scene virtual simulation method thereof
CN111061374B
( en )
*
2019-12-20
2024-04-09
京ä¸æ¹ç§æéå¢è¡ä»½æéå ¬å¸
A method and device for supporting multi-person mode augmented reality application
CN111241681B
( en )
*
2020-01-13
2023-03-24
é西å¿åä¿¡æ¯ç§ææéå ¬å¸
Display range simulation method and device of signal machine
CN111524240A
( en )
*
2020-05-11
2020-08-11
ç»´æ²ç§»å¨éä¿¡æéå ¬å¸
Scene switching method and device and augmented reality equipment
CN111651055A
( en )
*
2020-06-09
2020-09-11
æµæ±å汤ç§æå¼åæéå ¬å¸
City virtual sand table display method and device, computer equipment and storage medium
CN111653175B
( en )
*
2020-06-09
2022-08-16
æµæ±å汤ç§æå¼åæéå ¬å¸
Virtual sand table display method and device
CN111651052A
( en )
*
2020-06-10
2020-09-11
æµæ±å汤ç§æå¼åæéå ¬å¸
Display method, device, electronic device and storage medium of virtual sand table
CN111857341B
( en )
*
2020-06-10
2023-06-13
æµæ±å汤ç§æå¼åæéå ¬å¸
Display control method and device
CN111651069A
( en )
*
2020-06-11
2020-09-11
æµæ±å汤ç§æå¼åæéå ¬å¸
Virtual sand table display method and device, electronic equipment and storage medium
CN111599223A
( en )
*
2020-06-12
2020-08-28
æµæ±å汤ç§æå¼åæéå ¬å¸
Sand table display system and sand table display method
CN111724481A
( en )
*
2020-06-24
2020-09-29
ååºå¦é¢
Method, device, device and storage medium for three-dimensional reconstruction of two-dimensional images
CN111956240A
( en )
*
2020-07-29
2020-11-20
å京ææç§ææéå ¬å¸
Psychological analysis device and method of 3D real-scene simulation sand table based on wireless sensing technology
CN111882675A
( en )
*
2020-07-31
2020-11-03
å京å¸å汤ç§æå¼åæéå ¬å¸
Model presentation method and device, electronic equipment and computer storage medium
CN112132940A
( en )
*
2020-09-16
2020-12-25
å京å¸å汤ç§æå¼åæéå ¬å¸
Display method, display device and storage medium
CN112435558A
( en )
*
2020-11-10
2021-03-02
æ·±å³å¸çç«è§å¤åªä½ç§ææéå ¬å¸
Holographic 3D intelligent interactive digital virtual sand table and interactive method thereof
CN112733366B
( en )
*
2021-01-12
2022-09-16
ä¸å½äººæ°è§£æ¾åéååäºäº¤éå¦é¢åäºäº¤éè¿è¾ç 究æ
Real-time high-explosive-bomb fragment damage simulation method
CN112817547B
( en )
*
2021-01-22
2024-12-13
å京å°ç±³ç§»å¨è½¯ä»¶æéå ¬å¸
Display method, device, and storage medium
CN113034668B
( en )
*
2021-03-01
2023-04-07
ä¸ç§æ°æ®(éå²)ç§æä¿¡æ¯æéå ¬å¸
AR-assisted mechanical simulation operation method and system
US20230260240A1
( en )
*
2021-03-11
2023-08-17
Quintar, Inc.
Alignment of 3d graphics extending beyond frame in augmented reality system with remote presentation
CN113516776B
( en )
*
2021-04-19
2024-09-06
æ·±å³å¸æ¨åç§ææéå ¬å¸
Augmented reality synchronization method, device, terminal equipment and storage medium
CN113436559B
( en )
*
2021-05-19
2023-04-14
åæ大å¦
A sand table dynamic landscape real-time display system and display method
CN115878191B
( en )
*
2021-08-06
2025-11-04
å为ææ¯æéå ¬å¸
A method for hot-plugging a device and a terminal device
CN113888709A
( en )
*
2021-09-30
2022-01-04
å京åå¸ç½é»ä¿¡æ¯ææ¯æéå ¬å¸
Electronic sand table generation method and device and non-transient storage medium
TWI779922B
( en )
*
2021-11-10
2022-10-01
財åæ³äººè³è¨å·¥æ¥çé²æ
Augmented reality processing device and method
CN116778115A
( en )
*
2022-03-18
2023-09-19
å®è¾¾å½é çµåè¡ä»½æéå ¬å¸
Wearable devices and control methods
CN114663635B
( en )
*
2022-03-31
2025-09-09
ååç工大å¦
Virtual sand table system and application thereof in psychological assessment
CN114460760B
( en )
*
2022-04-13
2022-06-10
æ¦æ±ç»å ¸æ¨¡åææ¯æéå ¬å¸
Dynamic display method and system for electronic sand table object
CN117351797B
( en )
*
2023-10-23
2024-10-08
æ±èå®æ¿ç§ææéå ¬å¸
Position real-time linkage system
CN117827012B
( en )
*
2024-03-04
2024-05-07
å京å½æåå¾ç§ææéå ¬å¸
Real-time visual angle tracking system of 3D sand table
CN119272528B
( en )
*
2024-11-15
2025-09-26
åå°æ»¨å·¥ä¸å¤§å¦
Design Method of Virtual Simulation System for Spacecraft Deployment Mechanism
Citations (7)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US20090129630A1
( en )
*
2007-11-16
2009-05-21
Sportvision, Inc.
3d textured objects for virtual viewpoint animations
US20130286004A1
( en )
*
2012-04-27
2013-10-31
Daniel J. McCulloch
Displaying a collision between real and virtual objects
US20150130790A1
( en )
*
2013-11-14
2015-05-14
Nintendo Of America Inc.
Visually Convincing Depiction of Object Interactions in Augmented Reality Images
US20150301599A1
( en )
*
2014-04-18
2015-10-22
Magic Leap, Inc.
Eye tracking systems and method for augmented or virtual reality
US20170053447A1
( en )
*
2015-08-20
2017-02-23
Microsoft Technology Licensing, Llc
Augmented Reality
US20180089899A1
( en )
*
2016-09-23
2018-03-29
Apple Inc.
Adaptive vehicle augmented reality display using stereographic imagery
CN108136257A
( en )
*
2015-08-17
2018-06-08
ä¹é«å ¬å¸
The method for creating virtual game environment and the interaction game system using this method
Family Cites Families (9)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US20140240351A1
( en )
*
2013-02-27
2014-08-28
Michael Scavezze
Mixed reality augmentation
CN103839277B
( en )
*
2014-02-21
2017-06-06
å京ç工大å¦
A kind of mobile augmented reality register method of outdoor largescale natural scene
US10068373B2
( en )
*
2014-07-01
2018-09-04
Samsung Electronics Co., Ltd.
Electronic device for providing map information
US9858720B2
( en )
*
2014-07-25
2018-01-02
Microsoft Technology Licensing, Llc
Three-dimensional mixed-reality viewport
CN104134235B
( en )
*
2014-07-25
2017-10-10
æ·±å³è¶ å¤ç»´å çµåæéå ¬å¸
Real space and the fusion method and emerging system of Virtual Space
CN104656893B
( en )
*
2015-02-06
2017-10-13
西åå·¥ä¸å¤§å¦
The long-distance interactive control system and method in a kind of information physical space
CN105912121A
( en )
*
2016-04-14
2016-08-31
å京è¶æ³è±¡å½é ç§è´¸åå±æéå ¬å¸
Method and system enhancing reality
CN106340064B
( en )
*
2016-08-25
2019-02-01
å京大è§æ¯ç§ææéå ¬å¸
A kind of mixed reality sand table device and method
CN106373198A
( en )
*
2016-09-18
2017-02-01
ç¦å·å¤§å¦
Method for realizing augmented reality
2017
2017-02-27
CN
CN201710108466.7A
patent/CN108510592B/en
active
Active
2017-04-14
US
US15/487,449
patent/US10181222B2/en
active
Active
Patent Citations (7)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US20090129630A1
( en )
*
2007-11-16
2009-05-21
Sportvision, Inc.
3d textured objects for virtual viewpoint animations
US20130286004A1
( en )
*
2012-04-27
2013-10-31
Daniel J. McCulloch
Displaying a collision between real and virtual objects
US20150130790A1
( en )
*
2013-11-14
2015-05-14
Nintendo Of America Inc.
Visually Convincing Depiction of Object Interactions in Augmented Reality Images
US20150301599A1
( en )
*
2014-04-18
2015-10-22
Magic Leap, Inc.
Eye tracking systems and method for augmented or virtual reality
CN108136257A
( en )
*
2015-08-17
2018-06-08
ä¹é«å ¬å¸
The method for creating virtual game environment and the interaction game system using this method
US20170053447A1
( en )
*
2015-08-20
2017-02-23
Microsoft Technology Licensing, Llc
Augmented Reality
US20180089899A1
( en )
*
2016-09-23
2018-03-29
Apple Inc.
Adaptive vehicle augmented reality display using stereographic imagery
Non-Patent Citations (4)
* Cited by examiner, â Cited by third party
Title
" An Optical See-Through Augmented Reality System with Gesture-Based Interaction ", by Zhen et al., 2016 International Conference on Virtual Reality and Visualization, pp. 447-452, Sep. 24-26, 2016. (Year: 2016).
*
" Digital Sand Model Using Virtual Reality Workbench ", by Syed et al., 2012 International Conference on Computer Communication and Informatics, pp. 1-6, Jan. 10-12, 2012. (Year: 2012).
*
" Virtual reality realization technology and its application based on augmented reality ", by Ning Li et al., 2017 IEEE Global Conference on Signal and Information Processing (GlobalSIP), Nov. 14-16, 2017, pp. 1383-1387 (Year: 2017).
*
" Virtual Tactical Map with Tangible Augmented Reality Interface ", by Jung et al., 2008 International Conference on Computer Science and Software Engineering, pp. 1170-1173, Dec. 12-14, 2008. (Year: 2008).
*
Also Published As
Publication number
Publication date
US20180247456A1
( en )
2018-08-30
CN108510592A
( en )
2018-09-07
CN108510592B
( en )
2021-08-31
Similar Documents
Publication
Publication Date
Title
US20180247456A1
( en )
2018-08-30
Method and device for augmented reality display of real physical model
US12007564B2
( en )
2024-06-11
Methods and apparatuses for corner detection using neural network and corner detector
US10977554B2
( en )
2021-04-13
Fully convolutional interest point detection and description via homographic adaptation
US20200410713A1
( en )
2020-12-31
Generating pose information for a person in a physical environment
US11232636B2
( en )
2022-01-25
Methods, devices, and systems for producing augmented reality
US20140368539A1
( en )
2014-12-18
Head wearable electronic device for augmented reality and method for generating augmented reality using the same
JP7623487B2
( en )
2025-01-28
Three-dimensional (3D) facial feature tracking for an automated stereoscopic telepresence system
US20100287500A1
( en )
2010-11-11
Method and system for displaying conformal symbology on a see-through display
CN105212418A
( en )
2016-01-06
Augmented reality intelligent helmet based on infrared night viewing function is developed
US11682138B2
( en )
2023-06-20
Localization and mapping using images from multiple devices
US20190259198A1
( en )
2019-08-22
Systems and methods for generating visual representations of a virtual object for display by user devices
WO2022023142A1
( en )
2022-02-03
Virtual window
CN107977082A
( en )
2018-05-01
A kind of method and system for being used to AR information be presented
EP4050400B1
( en )
2023-09-06
Display apparatuses and methods incorporating image masking
US11727658B2
( en )
2023-08-15
Using camera feed to improve quality of reconstructed images
Selvam et al.
2016
Augmented reality for information retrieval aimed at museum exhibitions using smartphones
US20170300121A1
( en )
2017-10-19
Input/output device, input/output program, and input/output method
Kaur et al.
2015
Computer vision and sensor fusion for efficient hybrid tracking in augmented reality systems
WO2019044123A1
( en )
2019-03-07
Information processing device, information processing method, and recording medium
Yan
2023
Application of VR virtual reality technology in 3D image interior design system
CA3142001C
( en )
2023-06-20
Spherical image based registration and self-localization for onsite and offsite viewing
Behringer
1998
Improving the registration precision by visual horizon silhouette matching
Ravi et al.
2024
A study of object recognition and tracking techniques for augmented reality applications
US20240404099A1
( en )
2024-12-05
Estimating Device Pose Using Plane Normal Vectors
US20250085103A1
( en )
2025-03-13
Feature correlation
Legal Events
Date
Code
Title
Description
2018-10-31
AS
Assignment
Owner name : HISCENE INFORMATION TECHNOLOGY CO., LTD, CHINA
Free format text : ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:TUO, YUXIANG;JIANG, JIAYI;ZHAO, XUEBIN;AND OTHERS;REEL/FRAME:047362/0732
Effective date : 20181024
2018-12-22
STCF
Information on status: patent grant
Free format text : PATENTED CASE
2022-06-29
MAFP
Maintenance fee payment
Free format text : PAYMENT OF MAINTENANCE FEE, 4TH YR, SMALL ENTITY (ORIGINAL EVENT CODE: M2551); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY
Year of fee payment : 4