Conceptio › Archive › Google Patents

Google Patentsopen access

Method and system for compositing an augmented reality scene — Apple Inc. (US10565796B2)

Apple Inc. · Google Patents

Google Patents · Patents · License: Open Access

Open Source ↗

appleinc.

patent, google patents, intellectual property, US10565796B2, Apple Inc., Netanel Hagbi, en, 2020

ABSTRACT

Abstract

Disclosed are systems and methods for compositing an augmented reality scene, the methods including the steps of extracting, by an extraction component into a memory of a data-processing machine, at least one object from a real-world image detected by a sensing device; geometrically reconstructing at least one virtual model from at least one object; and compositing AR content from at least one virtual model in order to augment the AR content on the real-world image, thereby creating AR scene. Preferably, the method further includes; extracting at least one annotation from the real-world image into the memory of the data-processing machine for modifying at least one virtual model according to at least one annotation. Preferably, the method further includes: interacting with AR scene by modifying AR content based on modification of at least one object and/or at least one annotation in the real-world image.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent application is a continuation of U.S. patent application Ser. No. 13/063,724, filed Dec. 9, 2011, which is a National Stage Entry of PCT Application No. PCT/IL2009/000890, which claims priority to U.S. Provisional Patent Application No. 61/095,989, filed Sep. 11, 2008, all of which are hereby incorporated by reference in their entireties.

FIELD OF THE INVENTION

The present invention relates to the Augmented Reality (AR) field. More particularly, the present invention relates to a method and system for compositing an AR scene from AR content which is embedded within an image (e.g., a print, a hand sketch), according to one or more rules, thereby substantially eliminating the need for downloading the AR content from a local or remote library/database, and allowing content modifications by sketching on the image.

Definitions, Acronyms and Abbreviations

Throughout this specification, the following definitions are employed:

Augmented Reality (AR): refers to the merging of virtual worlds into of the real world to produce new environments and visualizations, where physical and digital objects coexist and interact in real time. An augmented reality system maintains an AR scene and renders it on images of the real world in real time. Generally, the AR scene is designed to enhance the user's perception of the real-world, who can see it or interact with its virtual content. The AR Scene is registered to the real-world scene in a 3-dimensional (3D) manner.

AR Scene: refers to digital information that represents visual content to be registered and rendered on top of images of the real world in real time. An AR scene can include of different kinds of AR content. The AR scene may include dynamic information as well, such as animation and simulation properties, which may affect the appearance of the AR content when rendered.

AR content: refers to digital information customized to be presented on a view of the real world. This information may be virtual models, text, sound, etc.

Virtual model: A digital geometric description of a spatial entity. The most common examples of virtual models are those created in 3D space for the purpose of visualization. A virtual model fundamentally includes of geometric structure. It usually also includes additional properties such as texture, color, material, position, and rotation.

Connected Components Analysis/Labeling: is an operation for scantling an image and grouping its pixels into components, based on some form of connectivity.

Morphological Operations: are related to mathematical morphology, which is a theoretical approach to multi-dimensional digital signal analysis or image analysis based on shape.

Projective Transformation: is a linear transformation in homogeneous coordinates, which may be used to describe the relation between the poses of elements in the world and their poses from the observer's point of view.

Registration: In augmented reality, (vision-based) registration involves determining the position and orientation of a sensing device (e.g., a camera) with respect to the real-world or to a specific object within it in order to align AR content to the coordinate system of the real world or to a specific object within it.

Template Matching: is a technique in Digital Image Processing (DIP) for determining portions of an image that match a predefined template image.

BACKGROUND OF THE INVENTION

The past decade has been characterized by significant development and research in AR, AR content may, include for example, of 3D virtual models, which can be created using modeling software (application). According to prior art, the virtual content (such as AR content) is usually stored within a database (library) or model file, and is fetched by the AR system for rendering. Hence, according to the prior art, the AR content is digitally stored, and any interaction with it, such as editing, requires using dedicated computer software.

Content libraries/databases in AR systems are commonly maintained either locally on the augmenting device, or remotely on a database server. Updating local libraries on user devices is a relatively complicated task (especially if the devices are distributed between a large number of users). On the other hand, updating remote libraries (databases) is a relatively simple and centralized task, yet it requires connecting to a data network, such as the Internet or a cellular network. For example, it is supposed that a user points a camera of his cellular phone, on which an advertising AR application is installed, to an image printed in a newspaper. Then, the user observes a virtual scene containing one or more AR models, which are fetched from a remote database, superimposed over the printed image. For this purpose, according to the prior art, in order to download the AR content the user can point his cellular phone camera to a pattern or a barcode in the newspaper that encodes an index to the corresponding AR content to be downloaded. After that, the advertising AR application (installed on his cellular phone) connects to a remote database for downloading the AR, content according to the index. Therefore, according to the above, a user takes an active role in retrieving the AR contents. In addition the user may not be able to modify the content since the models compositing the AR scene are hard-coded in the database. The interaction with the models is also hard-coded in the database or even in the application itself.

Further, according to the prior art, the AR content can be digitally created by dedicated 3D modeling software. Such virtual content creation process usually requires relatively highly skilled content creators. Also, the 3D models have to be designed, as do their behaviors (for example, their time-dependent animation).

Also, according to the prior art, barcodes are widely used as a vision-based mechanism for identification, and can contain relatively complicated information. For example, the GameboyÂ® product of the NintendoÂ® company (located in United States), uses electronic cards with barcodes, wherein a complete game can be encoded within a barcode. Each individual electronic card has enough memory space to digitally encode a complete mini-game. Barcodes can also be used to identify a remote source of content (for example, barcodes can encode web page addresses). When the barcode is scanned by means of a cellular phone, the remote source of content is accessed over the cellular network. Although barcodes usually appear as a collection of black dots or lines, images can be incorporated into the barcodes in order to make them visually pleasant, as shown on FIG. 1A (incorporating photo 105 of a man's face). The barcode can be, for example, a dynamic 4D barcode, allowing the transmission of data to a mobile device (e.g., cellular phone 120 ) by providing a stream of barcode images, as illustrated in FIG. 1B .

Moreover, according to the prior art, in computer vision-based AR systems, visual markers are commonly used. The markers are usually used for: (1) identifying the AR content that should be augmented in a real-world scene; and (2) performing registration, thereby determining camera location relative to real objects within the real-world scene. For example, ARToolKitÂ®, a software library for building AR applications developed and maintained by Human Interface Technology (HIT) Laboratory, University of Washington, USA, uses square fiducial markers (as illustrated on FIG. 1C ). In addition, in the AR PlayStationÂ® game titled âThe Eye of Judgmentâ (http://www.eyeofjudgment.com/), cards are used to identify models and actions that can be performed: the cards are placed on a tracked board and their appearance reflects the content they identify.

It should also be noted that according to the prior art, several methods exist for interpreting relatively rich content from symbols, and more generally, from visual languages. Some visual languages are understandable to people without any computer system. For example, Pictography is the expression of words and ideas through standard languages that omit unnecessary details, e.g., icons, road signs, and Chinese characters. Visual languages are also common in cartography: non-spatial features are commonly described by legends, and spatial information, such as terrain topography or temperature is commonly described using color scales. It should be noted that a sketched 3D drawing (e.g., a perspective or isometric sketch of a cube) is also based on a visual language that is understandable to people; people who see such a perspective sketch of the cube, can relatively easily interpret it as the corresponding 3D object.

In addition, it should be noted that different approaches have been described in the literature for analyzing graphical annotations. For example, mathematical and physical laws can be automatically interpreted.

Therefore, there is a need in the prior art to provide AR method and system, wherein AR content is embedded within an image (e.g., a sketch) by eliminating the need for downloading the AR content from a local or remote library/database, and allowing interaction with the the embedded content by hand sketching on the image; i.e., there is a need to provide a convenient and efficient way to generate AR content without maintaining a data content library/database, thereby extracting virtual model geometry, texture and behavior substantially directly from the image. For a better understanding of, and interaction with the embedded content there is a need for the content to employ dual perception. Dual perception means that the the embedded AR content is visually understandable also without using any AR system. Furthermore, there is a need to provide a way to interact with AR content to be augmented by enabling editing the AR content (e.g., by sketching), while displaying it to a user, in substantially real-time.

SUMMARY OF THE INVENTION

The present invention relates to a method and system for compositing an AR scene, wherein AR content is embedded within an image (e.g., a sketch), according to one or more rules, thereby substantially eliminating the need for downloading the AR content from a local or remote library/database.

Therefore, according to the present invention, there is provided for the first time a system for compositing an AR scene, the system including: (a) an extraction component, operationally connected to a sensing device for capturing a real-world image, configured for: (i) extracting at least one object from the real-world image into a memory of a data-processing machine; (ii) geometrically reconstructing at least one virtual model from at least one object; and (iii) compositing AR content from at least one virtual model in order to augment the AR content on the real-world image, thereby creating the AR scene.

Preferably, the extraction component is further configured for: (iv) extracting at least one annotation from the real-world image into the memory of the data-processing machine for modifying at least one virtual model according to at least one annotation.

Preferably, the extracting is performed independently of a predefined content repository.

Preferably, the real-world image includes at least one image type selected from the group consisting of a printed image, a hand-sketched image, a hand-drawn image, a sprayed image, an image displayed on a screen, an image projected on a surface, a weaved image, a carved-surface image, and an engraved-surface image.

Preferably, the real-world image includes at least two correlated annotations for modifying an effect of other annotations.

Preferably, the real-world image includes a legend having at least one item selected from the group consisting of: an object referring to at least one image object, an annotation referring to at least one image annotation, an annotation referring to at least one object, and at least two correlated annotations referring to at least one object.

Preferably, at least one object includes the entirety of the real-world image.

Preferably, the extraction component includes at least one functional component selected from the group consisting of an object-extractor component, a geometry-analysis component, a modeling-cues-analysis component, an animation-analysis component, a space-warp-analysis component, a physical-property-analysis component, a user-interaction-analysis component, an inter-object-connection-analysis component, a mesh-generation component, a texture-generation component, and an object-factory component.

Preferably, the extraction component is further configured for: (iv) interacting with the AR scene by modifying the AR content based on modification of at least one object and/or at least one annotation in the real-world image.

According to the present invention, there is provided for the first time a method for compositing an AR scene, the method including the steps of (a) extracting, by an extraction component into a memory of a data-processing machine, at least one object from a real-world image detected by a sensing device; (b) geometrically reconstructing at least one virtual model from at least one object; and (c) compositing AR content from at least one virtual model in order to augment the AR content on the real-world image, thereby creating the AR scene.

Preferably, the method further includes the step of (d) extracting at least one annotation from the real-world image into the memory of the data-processing machine for modifying at least one virtual model according to at least one annotation.

Preferably, the step of extracting is performed independently of a predefined content repository.

Preferably, the real-world image includes a legend having at least one item selected from the group consisting of an object referring to at least one image object, an annotation referring to at least one image annotation, an annotation referring to at least one object, and at least two correlated annotations referring to at least one object.

Preferably, the method further includes the step of: (d) interacting with the AR scene by modifying the AR content based on modification of at least one object and/or at least one annotation in the real-world image.

According to the present invention, there is provided for the first time a computer-readable storage medium having computer-readable code embodied on the computer-readable storage medium, the computer-readable cede including: (a) program code for extracting, by an extraction component into a memory of a data-processing machine, at least one object from a real-world image detected by a sensing device; (b) program code for geometrically reconstructing at least one virtual model from at least one object; and (c) program code for compositing AR content from at least one virtual model in order to augment the AR content on the real-world image, thereby creating the AR scene.

These and further embodiments will be apparent from the detailed description and examples that follow

BRIEF DESCRIPTION OF THE DRAWINGS

In order to understand the invention and to see how it may be carried out in practice, various embodiments of the invention will now be described by way of non-limiting examples only, with reference to the accompanying drawings, in which:

FIG. 1A is a sample illustration of a barcode, containing predefined information, according to the

CROSS-REFERENCE TO RELATED APPLICATIONS

FIELD OF THE INVENTION

Definitions, Acronyms and Abbreviations

Throughout this specification, the following definitions are employed:

AR content: refers to digital information customized to be presented on a view of the real world. This information may be virtual models, text, sound, etc.

Connected Components Analysis/Labeling: is an operation for scantling an image and grouping its pixels into components, based on some form of connectivity.

Morphological Operations: are related to mathematical morphology, which is a theoretical approach to multi-dimensional digital signal analysis or image analysis based on shape.

Template Matching: is a technique in Digital Image Processing (DIP) for determining portions of an image that match a predefined template image.

BACKGROUND OF THE INVENTION

SUMMARY OF THE INVENTION

Preferably, the extracting is performed independently of a predefined content repository.

Preferably, the real-world image includes at least two correlated annotations for modifying an effect of other annotations.

Preferably, at least one object includes the entirety of the real-world image.

Preferably, the step of extracting is performed independently of a predefined content repository.

These and further embodiments will be apparent from the detailed description and examples that follow

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a sample illustration of a barcode, containing predefined information, according to the prior art;

FIG. 1B is a sample illustration of a dynamic 4D barcode, allowing transmitting data to a mobile device by providing a stream of barcode images, according to the prior art;

FIG. 1C is a sample illustration of a square fiducial marker for determining/calculating camera position in space relative to the marker, while the pattern inside the square is used to specify which virtual model should be retrieved from the library, according to the prior art;

FIG. 2A is a schematic block diagram of an augmented reality system based on the presented In-Place Augmented Reality (IPAR) approach, according to an embodiment of the present invention;

FIG. 2B is a schematic block diagram of an AR system based on the presented In-Place Augmented Reality approach, according to another embodiment of the present invention;

FIG. 2C is a schematic block diagram of an AR system based on the presented In-Place Augmented Reality approach, according to still another embodiment of the present invention;

FIG. 3 is a block diagram of the Extraction sub-stage for determining and providing AR virtual models (represented by objects within the image) and their corresponding behavior (represented by annotations within the image) into the AR application for compositing an AR scene, and in turn, displaying it to a user at the Application Usage stage, according to an embodiment of the present invention;

FIG. 4A is a sample map of a ski site, such as Saint Moritz ski site in Switzerland, comprising embedded AR content based on the presented In-Place Augmented Reality approach, according to an embodiment of the present invention;

FIG. 4B is a corresponding AR scene composited based on the sample ski site map, according to an embodiment of the present invention;

FIG. 5A is a sample image of an interactive ocean artwork, based on the presented In-Place Augmented Reality approach, according to an embodiment of the present invention;

FIG. 5B is a corresponding AR scene composited based on the interactive ocean artwork, according to an embodiment of the present invention;

FIG. 6A demonstrates viewing geographic information, employing two or more layers, based on the presented In-Place Augmented Reality approach, according to an embodiment of the present invention;

FIG. 6B is a corresponding AR scene composited by combining geographic layers, according to an embodiment of the present invention;

FIG. 7A is a sample contour map sketch to be rendered and displayed as a 3D AR scene, based on the presented In-Place Augmented Reality approach, according to an embodiment of the present invention;

FIG. 7B is another sample sketch to be rendered and displayed as a 3D AR scene, based on the presented In-Place Augmented Reality approach, according to an embodiment of the present invention;

FIG. 8 illustrates other sample sketches to be composited, for example, in a single 3D AR scene based on the presented In-Place Augmented Reality approach, according to an embodiment of the present invention;

FIG. 9A is a sample illustration of a sketch of a physical system, from which a corresponding AR scene is to be composited, based on the presented In-Place Augmented Reality approach, according to an embodiment of the present invention;

FIG. 9B is another sample illustration of another sketch of a physical system, from which a corresponding AR scene is to be composited, based on the presented In-Place Augmented Reality approach, according to an embodiment of the present invention;

FIG. 10 is still another sample image representing a car racing game, based on the presented In-Place Augmented Reality approach, according to an embodiment of the present invention;

FIG. 11 is an additional sample image comprising embedded AR content of assembly instructions to be displayed in a 3D AR scene based on the presented In-Place Augmented Reality approach, according to an embodiment of the present invention; and

FIGS. 12A to 12C are sample sketches of multiple views of an architectural building plan, comprising embedded AR content to be displayed in a 3D AR scene, based on the presented In-Place Augmented Reality approach, according to an embodiment of the present invention.

It will be appreciated that for simplicity and clarity of illustration, elements shown in the figures have not necessarily been drawn to scale. For example, the dimensions of some of the elements may be exaggerated relative to other elements for clarity. Further, where considered appropriate, reference numerals may be repeated among the figures to indicate corresponding or analogous elements.

DETAILED DESCRIPTION OF THE EMBODIMENTS

In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the invention. However, it will be understood by those skilled in the art that the present invention may be practiced without these specific details. In other instances, well-known methods, systems, procedures, units, components, and the like have not been described in detail so as not to obscure the present invention.

According to the prior art, in conventional augmented reality applications, virtual models are created by artists and stored in a local library (within digital files) or in a remote database. At runtime, these models are retrieved from the library, registered, and rendered onto the scene, appearing to overlay the real-world content. The trigger in the real-world content that causes this virtual content retrieval to commence is commonly an image or marker that has to be printed by the user. This not only limits the usability of AR applications, but also turns the experience they provide from instant and straightforward into one that requires preparation by the user. According to an embodiment of the present invention, based on the presented In-Place Augmented Reality approach, the AR content, which includes of virtual models and their behaviors, is embedded in a real-world image (e.g., a sketch), captured by the augmented reality system, and then extracted from the captured image in order to be rendered back as AR content over the real-world image.

According to an embodiment of the present invention, the AR content is embedded substantially on the same place (e.g., on real printed paper), where it is augmented and, in turn, viewed by a user. This gives rise to the above term of âIn-Place Augmented Realityâ (IPAR), according to an embodiment of the present invention. According to another embodiment of the present invention, the AR content can be augmented in a predefined location within the image, or externally to the image, according to one or more predefined rules. Bach rule can be either indicative of at least one object provided within the image and/or can be related to an annotation describing a behavior or an action (e.g., animation, simulation) to be applied to at least one object. An IPAR system enables presenting AR content to a plurality of users independently of a content library/database. Virtual models and their behaviors are embedded into an image as objects and their annotations, and substantially no library/database is required to store them. Also, according to an embodiment of the present invention, the embedded AR content exhibits dual perception, meaning the real-world content is intuitive and understandable to users also without using the IPAR system. Furthermore, interaction is possible with the given AR content, e.g., by editing objects and annotations, and new AR content may be created by a user, for example, by sketching while the AR content is displayed to the user, substantially in real-time. According to an embodiment of the present invention, to make the embedded AR content identifiable in the image, the sketching has to follow one or more predefined rules of a visual language. One of these rules can be, for example, sketching an object to be augmented, in a specific color/grey-level. The IPAR system can be designed and maintained as a general system for processing and displaying any amount of AR content as long the content consistently follows a visual language. Furthermore, since the content does not have to be retrieved by any communication means, the number of concurrent users substantially does not affect the operation of the presented IPAR system. This is in contrast to the case where AR content is prepared in advance and stored on a remote database/library. Hence, the number of concurrent users is unlimited. According to a further embodiment of the present invention, the IPAR system can also function when no data network (e.g., the Internet, cellular network, etc.) can be used. According to a further embodiment of the present invention, the IPAR system can also waive the need to print or use pre-printed material since content can be created substantially in real time, e.g., by hand-sketching it from scratch.

FIG. 2A is a schematic block diagram of an augmented reality system 200 based on the presented In-Place Augmented Reality approach, according to an embodiment of the present invention. System 200 comprises three main stages: a) Authoring stage 205 having Embedding/ Encoding sub-stage 206 for embedding AR content into a real-world image, according to a set of (visual language) predefined rules; b) Content Retrieval stage 210 having Acquisition sub-stage 211 for acquiring the image with the embedded AR content, and Extraction sub-stage 212 for extracting the embedded AR content from the acquired image; and c) Application Usage stage 215 having Registration sub-stage 216 for registering the extracted AR content, and Rendering sub-stage 217 for rendering the extracted AR content over the image. It should be noted that the image can be, for example, printed on real paper (or on any other substance, such as a shirt, canvas, poster, etc.), or displayed on a screen (such as a computer/mobile device or television screen).

According to an embodiment of the present invention, in Authoring stage 205 the AR content (being one or more virtual models and their behaviors, such as changes in the model texture, geometry, location) is embedded into an image by embedding corresponding objects, and optionally their annotations (cues), which represent the AR content within the image. The set of predefined rules enables IPAR system 200 to define how to extract the AR content from the image. According to an embodiment of the present invention, determining one or more objects within the image, can be applied by marking an object with an identifying feature, such as a predefined background color (e.g., green color), or by following other rules that determine what is considered to be an object (e.g., according to size, position, connectivity). Also, an object may be superimposed on area that can be used as texture, according to a predefined rule. In this case, the texture that is covered by the object can be in-painted during AR presentation to the user if the object is moved from its original location during the AR presentation. It should be noted that a composited AR scene may contain a background image, which in turn can also be an object. In this case, the embedding of the background image may not be marked by any identifying feature.

At Content Retrieval stage 210 , the image with the embedded AR content is acquired (at Acquisition sub-step 211 ) by means of a conventional camera (e.g., a web camera, mobile device camera), and then the embedded AR content is extracted (at Extraction sub-step 212 ) from the image by means of a dedicated AR application that receives the acquired image from the camera. According to an embodiment of the present invention, the extraction is performed according to a set of predefined rules (e.g., objects are provided with a green background, animation paths are red color curves). According to an embodiment of the present invention, the rules, which derive the appearance of the image (e.g., by defining the cues/annotations that can be used) is the syntax of a visual language. The rule interpretation is a semantic element of this syntax; generally, each rule defines how to embed certain AR content into the image and therefore how to extract it and geometrically reconstruct it, thereby enabling compositing an AR scene. The image containing the embedded AR content is a spatial grouping of the syntax, which corresponds to the AR scene. It should be noted that each AR scene can be composited from more than one image. Also, the rules can be grouped into families, such as geometric rules, texture rules, animation rules, and so on.

At Application Usage stage 215 , the extracted AR content (i.e., the extracted and geometrically reconstructed virtual model(s) and their behavior(s)) is registered and rendered over the real-world image by means of an AR application. The AR application can be installed, for example, on a user's mobile device (e.g., a cellular phone, PDA (Personal Digital Assistant)). In such case, the user can also acquire the image with the embedded AR content at Acquisition sub-stage 211 by means of his mobile device camera, eliminating the need to have additional equipment. Then, a user is presented with a composited AR scene, wherein the AR content is displayed over the real-world image. Such composited AR scene can be viewed on a screen of the mobile device, on a computer/television screen, or on a head-mounted display, for example.

According to an embodiment of the present invention, before (or during) rendering an AR scene, various visual effects can be added to it (such as lighting, shininess, blending, smoothing, and so on), which are embedded within the image by using the appropriate annotations.

According to an embodiment of the present invention, the IPAR approach enables extracting AR content (virtual models and their behaviors) from an image that contains objects and annotations that represent them, after capturing the image by means of IPAR system 200 at Acquisition sub-stage 211 . It should be noted that the behavior of a virtual model refers to the way the virtual model acts substantially in real-time, and to the way it is displayed to the user. The object is a representation of the virtual model, while annotations (e.g., an animation path) of the object represent virtual model behaviors. For example, the animation path for an object can be represented by a continuous curve drawn on the image, which is printed on real paper. Further, an order for presenting virtual models to a user in a composited AR scene can be predefined according to their desired appearance, and therefore objects (that correspond to the virtual models) are arranged according to this order. According to an embodiment of the present invention, the set of rules comprises, for example, the following rules: objects within the image are drawn in a predefined color/grey-level or are surrounded by a predefined (color/grayscale) background; and annotations (cues) are predefined symbols, enabling them to be relatively easily identifiable and extractable by means of IPAR system 200 . To associate an annotation with an object, the annotation can be placed, for example, in proximity to the object.

According to another embodiment of the present invention, before the image with the embedded AR content is analyzed (e.g., after being acquired by means of an imaging device, such as a mobile device camera), the image can be corrected by applying to it a projective transformation. For example, if the image is surrounded by means of a black frame, then the projective transformation matrix can be calculated according to the vertices of the black frame by using conventional techniques. In addition, to overcome lighting differences within the image and reduce the effect of relatively low camera quality, image enhancement procedures (e.g., color correction, super sampling) can be further applied to the captured image. Also, in order to deal with color distortion, predefined reference color shapes (e.g., a red square, green square) can be added to the image. Such reference color shapes can be added to a predefined location in the image (e.g., into an image âlegendâ). Then, through the image acquisition, each pixel color is interpolated according to these reference colors. It should be noted that acquisition of images by means of a camera can be performed by taking one or more still pictures of the same real-world environment, or by shooting a video clip. In the case where the real-world environment is captured more than once (obtaining more than one still image), further processing of the real-world environment is performed, according to conventional techniques in the field (e.g., techniques for creating panoramic images, or restoring satellite images).

According to an embodiment of the present invention, performing the above preprocessing of the captured image before executing Extraction sub-stage 212 is used to prepare the image for extraction of the embedded AR content, which is represented within the image by means of one or more objects with corresponding cues/annotations. For this, an image that is shot, for example by video, can be further brought to a planar state, making the objects relatively easy to locate within the image. If required, new images with enhanced features can be generated from the captured image. For example, a new image can be generated in which green color is made greener to make it easily identifiable, or in which green pixels are set as binary â1'sâ and other color pixels are set as binary â0'sâ. Further, morphological operations may be applied to annotations (which may define the behavior of an object, such as moving the object or morphing the object geometry or texture) to make the cues more prominent.

According to another embodiment of the present invention, when extracting the embedded AR content from the image, Embedding 206 process ( FIG. 2A ) may be logically reversed. First, the acquired (and optionally enhanced) image is analyzed, and its objects, along with their corresponding annotations, are interpreted by means of IPAR system 200 in order to yield a collection of virtual models with their corresponding behaviors. Extraction stage 212 involves the following operations: a) locating objects and geometrically reconstructing corresponding virtual models (including generating virtual model geometry and texture) from the objects; b) identifying annotations/cues related to predefined behaviors of the virtual models; and c) assigning the predefined behaviors to the virtual models, accordingly.

According to another embodiment of the present invention, annotations (cues) also have an identifying feature (e.g., red color) that makes them relatively easy to extract from the image. So, for identifying and preserving annotations, binarization of the image can also be performed. For annotations whose shape is fixed and predefined by means of one or more rules of IPAR system 200 , then such shape is defined as a template, and normalized cross-correlation template matching can then be performed between a set of predefined annotations templates and the binary image for determining the location of each annotation within the acquired (captured) image. Then, each identified annotation is assigned to its nearest object by using, for example, conventional Euclidean distance transform of the object and annotation. Once the annotations are extracted from the image, the predefined behaviors they represent are assigned to the corresponding extracted objects (and in turn, to the corresponding virtual models). For example, when a hand annotation that is associated to an object is identified, IPAR system 200 associates an operation of pressing a keyboard arrow to a movement of the specific virtual model, so that the object can be interactively manipulated by the user.

It should be noted that according to an embodiment of the present invention, animation paths can be continuous curves/lines, having a predefined color/grey-level (e.g., a red color). Therefore, animation paths can be represented by pixels that remain unidentified after previous steps (e.g., after the template matching). To extract animation paths (or any other line/curve), conventional morphological operations can be performed.

According to an embodiment of the present invention, at Application Usage stage 215 , after the AR content (virtual models and their corresponding behaviors) has been reconstructed from the acquired real-world image, then substantially real-time augmentation and AR viewing can start. For each captured video frame/still image, virtual models are registered into the real-world image and rendered at sub-stages

216 and 217 , respectively, according to conventional prior art techniques. For example, the registration can be performed by using the ARToolKitÂ® marker-based registration package, with relatively slight modifications. Then, in Rendering sub-stage 217 , the extracted virtual models are rendered into the image, and updated/modified according to their assigned behaviors. For example, the new position of a model is calculated at each frame, and then the object is transformed to the new position accordingly.

According to a further embodiment of the present invention, at least a portion of the predefined rules and/or at least a portion of the AR content (according to which a corresponding AR scene is composited and presented to a user) is stored within one or more of the following: a file that can be provided locally, for example, within a user's computer/mobile device (e.g., a cellular phone), on which the AR application that enables viewing the composited AR scene is installed; or a library/database, provided within a (remote) server over a data network (e.g., the Internet or a cellular network, to which the user's device connects).

FIG. 2B is a schematic block diagram of an augmented reality system 200 based on the presented In-Place Augmented Reality approach, according to another embodiment of the present invention. According to this embodiment, an image with AR content is sketched by hand on real paper, based on a set of predefined rules. It should be noted that the sketching can be done in real-time, without a need to perform embedding/encoding 206 ( FIG. 2A ) of the AR content in advance. This can be especially useful as an interaction technique in games or in educational applications, such as school/university educational applications allowing a pupil/student to sketch an image based on a set of predefined rules, and in parallel, or afterwards, see the desired AR content presented over the sketched image as a composited AR scene. The sketch can be observed by a user in 3D by using, for example, a mobile device.

It should be noted that according to an embodiment of the present invention, the user sketches a 2D projection of a 3D object on real paper (e.g. a cube), and then, at Application Usage stage 215 ; he observes his sketch in 3D on a screen, the sketch (image) being interpreted and presented in 3D environment on the same place it is sketched. According to this embodiment of the present invention, the user's sketch can be presented to him in 3D environment substantially in real-time, even if the user has not yet finished sketching.

It should be noted that according to another embodiment of the present invention, after performing Content Retrieval stage 210 and augmenting the extracted AR content on the image, the AR content can be either augmented substantially on the same place it was sketched/embedded within the image, or it may be augmented in a predefined location within the image or externally to the image, for compositing a corresponding AR scene. Furthermore, an already running AR scene may be introduced with new AR content by capturing, by means of an imaging device, additional AR content (represented by objects and/or annotations), and in turn augmenting the AR content and modifying the AR scene or compositing a new one. It should be noted that the AR scene can be composited by initially retrieving AR content from a local/remote database, and then introducing new AR content by means of IPAR system 200 . For example, a game may contain a slider that is sliding on a terrain, according to one or more rules that are predefined and fetched from a database. Then, IPAR system 200 enables the user to add a hill to the terrain, according to one or more predefined rules, by placing a card with an embedded hill sketched on it within the real-world view of the camera. After that, the hill is captured and extracted, and a new hill virtual model is generated, for compositing a corresponding AR scene and displaying it on a screen. As a result, the slider can interact with the hill (e.g., by climbing on it), while the user is holding the card of the 2D hill in front of his camera. On the other hand, when the user removes the card from the camera field of view, the hill virtual model is detached from the card (the card displayed on the screen with the hill sketched on) and remains in the running AR scene.

According to an embodiment of the present invention, at the Content Retrieval stage 210 a hand-drawn or printed sketch is analyzed. Such analysis can comprise analyzing sketch lines, corners, curves, and other types of elements, considering the visual language used for the sketching, and considering errors in the sketch that may be unintentionally caused by the user during the sketching. Instead of sketching by hand, a sketch can be provided by projecting a 3D model by using computer software, or sketched in 2D by using computer software, such as MicrosoftÂ® Visio, which reduces sketching errors. It should be noted that according to another embodiment of the present invention, a sketch can be drawn such that the colors, styles (e.g., dotted style), thicknesses of its lines have specific meanings, as predefined by the set of rules. For example, in a wireframe model of a cube, rear lines that are occluded may be drawn using a different color. Furthermore, additional cues can be added that help interpreting the image. For example, the 3D coordinate system (X, Y, and Z axes) can be depicted in the sketch using three mows. Further, it should be noted that according to still another embodiment of the present invention, the sketch can comprise a 3D object that is composed of different parts. In such a case, the object may be decomposed to make the sketch interpretation easier. Decomposition rules may be added as cues to the sketch (e.g., as arrows, or using different line thickness, color).

FIG. 2C is a schematic block diagram of AR system 200 based on the presented In-Place Augmented Reality approach, according to still another embodiment of the present invention. According to this embodiment, the image 230 is an existing conventional image, and Embedding/Encoding sub-stage 206 ( FIG. 2A ) is skipped. It should be noted that the conventional image (e.g., sketch) may already comprise 3D cues (e.g., depth/height cues) that enable extraction and interpretation of a 3D AR scene (as, for example, shown in FIG. 7A ).

According to another embodiment of the present invention, Embedding sub-stage 206 may involve the use of a dedicated compiler. In such case, an existing 3D image (e.g., generated by using 3D software, such as 3D StudioÂ®), comprising objects (representing, for example, 3D models) and cues (annotations) that represent 3D models behavior, can be compiled to a corresponding 2D image that represents the 3D scene by embedding additional information (geometry, behavior and etc), according to set of predefined rules. Such rules can be related to a perspective projection of the 3D model from a view point that emphasizes cues for reconstructing the corresponding 3D geometry. For example, one rule can state that curved edges are drawn in a different identifying color from straight ones. According to another rule, an animation path near an object can indicate object behavior in time. Thus, the compiler can place (e.g., in proximity to the object) one or more annotations to describe the desired animation and/or to describe an object modifier (an operation to be applied on the object), etc. The compiler also âknowsâ how to correlate the object with its annotation. The compiler-generated image may be further edited in 2D on a computer screen. For example, the user may move an annotation to a different location within the image after it has been placed there by the compiler. The resulting 2D image can then be printed on real paper or displayed on a computer/mobile device screen to be further acquired by an imaging device (e.g., a web/cellular phone camera).

It should be noted that according to an embodiment of the present invention, a virtual model has geometry. Also, an annotation (cue) can be used to specify model behavior, providing additional information about the model, such as animation, user-interaction to be applied to the model, physical properties (e.g., an elastic force) to be used in a physical system simulation, etc.

In addition, it should be noted that according to an embodiment of the present invention, IPAR system 200 enables providing image dual perception to a user, i.e., AR content (virtual models and their behaviors) is encoded into an image in such a way that their embedded representations visually reflect the virtual models and their behaviors for a user also without using the IPAR system 200 . Further, it should be noted that since AR content is embedded into a printed image (or into an image displayed on a screen), there is substantially no need to ask from a user (or from an AR application installed on a user's device) to send an identification code to the AR content provider for downloading the AR content. At the same time, content providers do not have to be concerned with securely distributing their copyrighted content electronically over a data network.

According to still another embodiment of the present invention, AR content library/database is not provided for retrieving AR content, since the AR content is extracted from the image at Extraction sub-stage 212 . In turn, a communication infrastructure (such as a communication network like the Internet, a cellular network, etc.) is also not required.

It should be noted that, according to a further embodiment of the present invention the AR application (installed on a user's device, such as a cellular phone) may require using a local or remote database/library (or file). Consequently a data network connection (e.g., a cellular network connection) may be required for compositing AR scenes. For example, a portion of the AR content may be retrieved from a remote database/library or file. The index to such AR content may then appear as a part of a set of predefined rules to be used for extraction of the AR content. For another example, the predefined rules (rather than the AR content) are retrieved from a local or remote database/library.

Since an IPAR image with the embedded AR content (a printed image, or an image displayed on a computer/mobile device screen) has dual perception and visually reflects the embedded augmented contents to a user (i.e., the IPAR image is understandable also without using IPAR system 200 ), then a user's interaction with the IPAR image is natural and relatively easy, even without using an Augmented Reality application. For example, a user who sees a perspective sketch of a cube understands that it is a 3D cube without using any AR application.

According to the present invention, the presented In-Place Augmented Reality approach can be useful in a wide range of different domains. In education, for example, physics books can include figures of physical scenarios with embedded AR content, demonstrating the material discussed in les

CLAIMS

Claims ( 21 )

The invention claimed is:

1. A method for compositing an augmented reality (AR) scene, the method comprising:

obtaining an image comprising an object;

extracting the object from the image, wherein the object comprises instructions to geometrically construct a virtual model of the object according to a predefined rule set;

geometrically constructing the virtual model of the object based on the instructions by generating a geometry and a texture of the virtual model based on the extracted object according to the predefined rule set; and

generating an AR scene by overlaying AR content from the constructed virtual model on the image.

2. The method of claim 1 , wherein the predefined rule set comprises rules assigned to one or more of object color, background color, texture, styles of lines, thicknesses of lines, and spatial arrangement.

3. The method of claim 1 , wherein the predefined rule set is stored on a same device that geometrically constructs the virtual model.

4. The method of claim 1 , wherein extracting the object from the image comprises extracting the object as the image is created.

5. The method of claim 1 , wherein the image comprising the object is hand-drawn.

6. The method of claim 1 , wherein geometrically constructing the virtual model of the object further comprises considering unintentional errors.

7. The method of claim 1 , wherein geometrically constructing the virtual model further comprises:

identifying, in the object, one or more annotations related to predefined behaviors of the virtual model; and

assigning the predefined behaviors to the virtual model.

8. A non-transitory machine-readable medium comprising instructions executable by one or more processors to:

obtain an image comprising an object;

extract the object from the image, wherein the object comprises instructions to geometrically construct a virtual model of the object according to a predefined rule set;

geometrically construct the virtual model of the object based on the instructions by generating a geometry and a texture of the virtual model based on the extracted object according to the predefined rule set; and

generate an augmented reality (AR) scene by overlaying AR content from the constructed virtual model on the image.

9. The non-transitory machine-readable medium of claim 8 , wherein the predefined rule set comprises rules assigned to one or more of object color, background color, texture, styles of lines, thicknesses of lines, and spatial arrangement.

10. The non-transitory machine-readable medium of claim 8 , wherein the predefined rule set is stored on a same device that geometrically constructs the virtual model.

11. The non-transitory machine-readable medium of claim 8 , wherein the instructions to extract the object from the image further comprise instructions to extract the object as the image is created.

12. The non-transitory machine-readable medium of claim 8 , wherein the image comprising the object is hand-drawn.

13. The non-transitory machine-readable medium of claim 8 , wherein the instructions to geometrically construct the virtual model of the object further comprise instructions to consider unintentional errors.

14. The non-transitory machine-readable medium of claim 8 , wherein the instructions to geometrically construct the virtual model further comprise instructions to:

identify, in the object, one or more annotations related to predefined behaviors of the virtual model; and

assign the predefined behaviors to the virtual model.

15. A system, comprising:

one or more processors; and

a memory coupled to the one or more processors and comprising instructions executable by the one or more processors to cause the system to:

obtain an image comprising an object;

extract the object from the image, wherein the object comprises instructions to geometrically construct a virtual model of the object according to a predefined rule set;

generate an augmented reality (AR) scene by overlaying AR content from the constructed virtual model on the image.

16. The system of claim 15 , wherein the predefined rule set comprises rules assigned to one or more of object color, background color, texture, styles of lines, thicknesses of lines, and spatial arrangement.

17. The system of claim 15 , wherein the predefined rule set is stored on a same device that geometrically constructs the virtual model.

18. The system of claim 15 , wherein the instructions to extract the object from the image further comprise instructions to extract the object as the image is created.

19. The system of claim 15 , wherein the image comprising the object is hand-drawn.

20. The system of claim 15 , wherein the instructions to geometrically construct the virtual model of the object further comprise instructions to consider unintentional errors.

21. The system of claim 15 , wherein the instructions to geometrically construct the virtual model further comprise instructions to:

identify, in the object, one or more annotations related to predefined behaviors of the virtual model; and

assign the predefined behaviors to the virtual model.

US15/818,299

2008-09-11

2017-11-20

Method and system for compositing an augmented reality scene

Active

2029-11-30

US10565796B2

( en )

Priority Applications (1)

Application Number

Priority Date

Filing Date

Title

US15/818,299

US10565796B2

( en )

2008-09-11

2017-11-20

Method and system for compositing an augmented reality scene

Applications Claiming Priority (4)

Application Number

Priority Date

Filing Date

Title

US9598908P

2008-09-11

PCT/IL2009/000890

WO2010029553A1

( en )

2008-09-11

2009-09-13

Method and system for compositing an augmented reality scene

US201113063724A

2011-12-09

US15/818,299

US10565796B2

( en )

2008-09-11

2017-11-20

Method and system for compositing an augmented reality scene

Related Parent Applications (2)

Application Number

Title

Priority Date

Filing Date

US13/063,724

Continuation

US9824495B2

( en )

2008-09-11

2009-09-13

Method and system for compositing an augmented reality scene

PCT/IL2009/000890

Continuation

WO2010029553A1

( en )

2008-09-11

2009-09-13

Method and system for compositing an augmented reality scene

Publications (2)

Publication Number

Publication Date

US20180082486A1

US20180082486A1 ( en )

2018-03-22

US10565796B2

true

US10565796B2 ( en )

2020-02-18

Family

ID=42004851

Family Applications (2)

Application Number

Title

Priority Date

Filing Date

US13/063,724

Active

2030-12-18

US9824495B2

( en )

2008-09-11

2009-09-13

Method and system for compositing an augmented reality scene

US15/818,299

Active

2029-11-30

US10565796B2

( en )

2008-09-11

2017-11-20

Method and system for compositing an augmented reality scene

Family Applications Before (1)

Application Number

Title

Priority Date

Filing Date

US13/063,724

Active

2030-12-18

US9824495B2

( en )

2008-09-11

2009-09-13

Method and system for compositing an augmented reality scene

Country Status (2)

Country

Link

( 2 )

US9824495B2

( en )

( 1 )

WO2010029553A1

( en )

Cited By (5)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US11295531B1

( en )

2021-02-24

2022-04-05

Allison Cecile Westley

System and method for generating interactive virtual image frames in an augmented reality presentation

TWI815782B

( en )

2022-11-27

2023-09-11

äºåç´°èè¡ä»½æéå¬å¸

Method for recognizing object assemblies in augmented reality images

US11798236B2

( en )

2020-02-28

2023-10-24

Mark Strachan

Augmented reality system and method

US12001750B2

( en )

2022-04-20

2024-06-04

Snap Inc.

Location-based shared augmented reality experience system

US12592045B2

( en )

2020-02-28

2026-03-31

Mark Strachan

Augmented reality system and method

Families Citing this family (175)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US8730156B2

( en )

2010-03-05

2014-05-20

Sony Computer Entertainment America Llc

Maintaining multiple views on a shared stable virtual space

US8718363B2

( en )

2008-01-16

2014-05-06

The Charles Stark Draper Laboratory, Inc.

Systems and methods for analyzing image data using adaptive neighborhooding

US8737703B2

( en )

2008-01-16

2014-05-27

The Charles Stark Draper Laboratory, Inc.

Systems and methods for detecting retinal abnormalities

US8463053B1

( en )

2008-08-08

2013-06-11

The Research Foundation Of State University Of New York

Enhanced max margin learning on multimodal data mining in a multimedia database

KR101646669B1

( en )

2009-06-24

2016-08-08

ì¼ì±ì ìì£¼ìíì¬

Method and apparatus for updating a composition database using user pattern, and digital photographing apparatus

US20110065496A1

( en )

2009-09-11

2011-03-17

Wms Gaming, Inc.

Augmented reality mechanism for wagering game systems

US20110109617A1

( en )

2009-11-12

2011-05-12

Microsoft Corporation

Visualizing Depth

WO2011106520A1

( en )

2010-02-24

2011-09-01

Ipplex Holdings Corporation

Augmented reality panorama supporting visually impaired individuals

US8866845B2

( en )

2010-03-10

2014-10-21

Empire Technology Development Llc

Robust object recognition by dynamic modeling in augmented reality

JP2012038239A

( en )

2010-08-11

2012-02-23

Sony Corp

Information processing equipment, information processing method and program

KR101669119B1

( en )

2010-12-14

2016-10-25

ì¼ì±ì ìì£¼ìíì¬

System and method for multi-layered augmented reality

KR20120086810A

( en )

2011-01-27

2012-08-06

ì¼ì±ì ìì£¼ìíì¬

Terminal and method for processing image thereof

US8848970B2

( en )

2011-04-26

2014-09-30

Digimarc Corporation

Salient point-based arrangements

CN103702726B

( en )

2011-05-23

2016-01-13

ä¹é«å¬å¸

Toy is built system, is produced the method and data handling system that build instruction

EP3042704B1

( en )

2011-05-23

2019-03-06

Lego A/S

A toy construction system

US10242456B2

( en )

2011-06-23

2019-03-26

Limitless Computing, Inc.

Digitally encoded marker-based augmented reality (AR)

US8681179B2

( en )

2011-12-20

2014-03-25

Xerox Corporation

Method and system for coordinating collisions between augmented reality and real reality

US20130278633A1

( en )

2012-04-20

2013-10-24

Samsung Electronics Co., Ltd.

Method and system for generating augmented reality scene

GB2501929B

( en )

2012-05-11

2015-06-24

Sony Comp Entertainment Europe

Apparatus and method for augmented reality

US9035970B2

( en )

2012-06-29

2015-05-19

Microsoft Technology Licensing, Llc

Constraint based information inference

US9105210B2

( en )

2012-06-29

2015-08-11

Microsoft Technology Licensing, Llc

Multi-node poster location

EP2875491A4

( en )

2012-07-19

2016-06-15

Gaurav Vats

USER-CONTROLLED 3D SIMULATION PROVIDING REALISTIC AND ENHANCED VISUALIZATION AND INTERACTION EXPERIENCE WITH DIGITAL OBJECT

US9224322B2

( en )

2012-08-03

2015-12-29

Apx Labs Inc.

Visually passing data through video

US9530232B2

( en )

2012-09-04

2016-12-27

Qualcomm Incorporated

Augmented reality surface segmentation

US9401121B2

( en )

2012-09-27

2016-07-26

Futurewei Technologies, Inc.

Network visualization through augmented reality and modeling

US9158389B1

( en )

2012-10-15

2015-10-13

Tangible Play, Inc.

Virtualization of tangible interface objects

US10657694B2

( en )

2012-10-15

2020-05-19

Tangible Play, Inc.

Activity surface detection, display and enhancement of a virtual scene

US10033943B1

( en )

2012-10-15

2018-07-24

Tangible Play, Inc.

Activity surface detection, display and enhancement

US8878846B1

( en )

2012-10-29

2014-11-04

Google Inc.

Superimposing virtual views of 3D objects with live images

US10462499B2

( en )

2012-10-31

2019-10-29

Outward, Inc.

Rendering a modeled scene

US10013804B2

( en )

2012-10-31

2018-07-03

Outward, Inc.

Delivering virtualized content

US20140313287A1

( en )

2012-11-20

2014-10-23

Linzhi Qi

Information processing method and information processing device

CN110084878B

( en )

2012-12-21

2023-07-07

è¹æå¬å¸

Method for representing virtual information in a real environment

US9110572B2

( en )

2013-02-04

2015-08-18

Visible Spectrum, Inc.

Network based video creation

US9208583B2

( en )

2013-02-13

2015-12-08

Blackberry Limited

Device with enhanced augmented reality functionality

US10133342B2

( en )

2013-02-14

2018-11-20

Qualcomm Incorporated

Human-body-gesture-based region and volume selection for HMD

US20140340423A1

( en )

2013-03-15

2014-11-20

Nexref Technologies, Llc

Marker-based augmented reality (AR) display with inventory management

US10509533B2

( en )

2013-05-14

2019-12-17

Qualcomm Incorporated

Systems and methods of generating augmented reality (AR) objects

US10262462B2

( en )

2014-04-18

2019-04-16

Magic Leap, Inc.

Systems and methods for augmented and virtual reality

US9413784B2

( en )

2013-09-06

2016-08-09

Microsoft Technology Licensing, Llc

World-driven access control

US9424239B2

( en )

2013-09-06

2016-08-23

Microsoft Technology Licensing, Llc

Managing shared state information produced by applications

US9355268B2

( en )

2013-09-06

2016-05-31

Microsoft Technology Licensing, Llc

Managing access by applications to perceptual information

US9697365B2

( en )

2013-09-06

2017-07-04

Microsoft Technology Licensing, Llc

World-driven access control using trusted certificates

US10217284B2

( en )

2013-09-30

2019-02-26

Qualcomm Incorporated

Augmented virtuality

US9626801B2

( en )

2013-12-31

2017-04-18

Daqri, Llc

Visualization of physical characteristics in augmented reality

US9558592B2

( en )

2013-12-31

2017-01-31

Daqri, Llc

Visualization of physical interactions in augmented reality

IN2014DE00332A

( en )

2014-02-05

2015-08-07

Nitin Vats

US9311639B2

( en )

2014-02-11

2016-04-12

Digimarc Corporation

Methods, apparatus and arrangements for device to device communication

US9477689B2

( en )

2014-02-15

2016-10-25

Barry Crutchfield

Embedding digital content within a digital photograph during capture of the digital photograph

JP6039594B2

( en )

2014-02-20

2016-12-07

æ ªå¼ä¼ç¤¾ã½ãã¼ã»ã¤ã³ã¿ã©ã¯ãã£ãã¨ã³ã¿ãã¤ã³ã¡ã³ã

Information processing apparatus and information processing method

US9615177B2

( en )

2014-03-06

2017-04-04

Sphere Optics Company, Llc

Wireless immersive experience capture and viewing

US9177239B1

( en )

2014-04-10

2015-11-03

Anki, Inc.

Generating machine-readable optical codes with aesthetic component

US9964409B1

( en )

2014-05-27

2018-05-08

Apple Inc.

Localized map generation

US10277280B2

( en )

2014-05-29

2019-04-30

Sony Interactive Entertainment LLC

Configuration of data and power transfer in near field communications

US10965159B2

( en )

2014-05-29

2021-03-30

Sony Corporation

Scalable antenna system

US9843360B2

( en )

2014-08-14

2017-12-12

Sony Corporation

Method and system for use in configuring multiple near field antenna systems

US9577463B2

( en )

2014-05-29

2017-02-21

Sony Corporation

Portable device to portable device wireless power transfer methods and systems

JP5762600B1

( en )

2014-06-06

2015-08-12

ãã¤ãã³æ ªå¼ä¼ç¤¾

Information processing apparatus and information processing method

FR3023020A1

( en )

2014-06-30

2016-01-01

Orange

METHOD FOR MANAGING AN INTERFACE

US9426610B2

( en )

2014-07-16

2016-08-23

Sony Corporation

Applying mesh network to luggage

US10127601B2

( en )

2014-07-16

2018-11-13

Sony Corporation

Mesh network applied to fixed establishment with movable items therein

US9361802B2

( en )

2014-07-16

2016-06-07

Sony Corporation

Vehicle ad hoc network (VANET)

US9516461B2

( en )

2014-07-16

2016-12-06

Sony Corporation

Mesh network applied to arena events

US9906897B2

( en )

2014-07-16

2018-02-27

Sony Corporation

Applying mesh network to pet carriers

US9900748B2

( en )

2014-07-16

2018-02-20

Sony Corporation

Consumer electronics (CE) device and related method for providing stadium services

US10922851B2

( en )

2014-09-25

2021-02-16

The Boeing Company

Virtual reality environment color and contour processing system

WO2016048658A1

( en )

2014-09-25

2016-03-31

Pcms Holdings, Inc.

System and method for automated visual content creation

US9659381B2

( en )

2015-01-26

2017-05-23

Daqri, Llc

Real time texture mapping for augmented reality system

US20170061700A1

( en )

2015-02-13

2017-03-02

Julian Michael Urbach

Intercommunication between a head mounted display and a real world object

US10360729B2

( en )

2015-04-06

2019-07-23

Scope Technologies Us Inc.

Methods and apparatus for augmented reality applications

US10083551B1

( en )

2015-04-13

2018-09-25

Allstate Insurance Company

Automatic crash detection

US9767625B1

( en )

2015-04-13

2017-09-19

Allstate Insurance Company

Automatic crash detection

KR20160142772A

( en )

2015-06-03

2016-12-13

ëìì¤ê¸ë¡ë² (ì£¼)

Method for providing augmented reality service for coloring and system of the same

US9975040B2

( en )

2015-06-07

2018-05-22

Apple Inc.

Procedural navigation graphs for identifying a route for a virtual object

US10101457B1

( en )

2015-07-15

2018-10-16

Apple Inc.

Mirror tilt actuator

US10235808B2

( en )

2015-08-20

2019-03-19

Microsoft Technology Licensing, Llc

Communication system

US10169917B2

( en )

2015-08-20

2019-01-01

Microsoft Technology Licensing, Llc

Augmented reality

EP3165979B1

( en )

2015-11-05

2020-12-30

Rohde & Schwarz GmbH & Co. KG

Providing mounting information

CN105913495A

( en )

2016-03-31

2016-08-31

èæ³(åäº¬)æéå¬å¸

Information processing method and electronic equipment

CN106373187B

( en )

2016-06-28

2019-01-11

ä¸æµ·äº¤éå¤§å¦

Two dimensional image based on AR is converted to the implementation method of three-dimensional scenic

EP3481519A1

( en )

2016-07-05

2019-05-15

Lego A/S

Method for creating a virtual object

US11433310B2

( en )

2016-07-05

2022-09-06

Lego A/S

Method for creating a virtual object

US11361380B2

( en )

2016-09-21

2022-06-14

Allstate Insurance Company

Enhanced image capture and analysis of damaged tangible objects

US10902525B2

( en )

2016-09-21

2021-01-26

Allstate Insurance Company

Enhanced image capture and analysis of damaged tangible objects

US10332317B2

( en )

2016-10-25

2019-06-25

Microsoft Technology Licensing, Llc

Virtual reality and cross-device experiences

US10620540B2

( en )

2016-11-07

2020-04-14

Mohammad Reza Mollazadeh Sardroudi

Encoded illustrations

JP2018077644A

( en )

2016-11-08

2018-05-17

å¯å£«ã¼ããã¯ã¹æ ªå¼ä¼ç¤¾

Information processing system and program

US11042161B2

( en )

2016-11-16

2021-06-22

Symbol Technologies, Llc

Navigation control method and apparatus in a mobile automation system

WO2018204342A1

( en )

2017-05-01

2018-11-08

Symbol Technologies, Llc

Product status detection system

US10726273B2

( en )

2017-05-01

2020-07-28

Symbol Technologies, Llc

Method and apparatus for shelf feature and object placement detection from shelf images

US11449059B2

( en )

2017-05-01

2022-09-20

Symbol Technologies, Llc

Obstacle detection for a mobile automation apparatus

CN110603533A

( en )

2017-05-01

2019-12-20

è®¯å®ç§ææéè´£ä»»å¬å¸

Method and apparatus for object state detection

US10949798B2

( en )

2017-05-01

2021-03-16

Symbol Technologies, Llc

Multimodal localization and mapping for a mobile automation apparatus

US11367092B2

( en )

2017-05-01

2022-06-21

Symbol Technologies, Llc

Method and apparatus for extracting and processing price text from an image set

WO2018201423A1

( en )

2017-05-05

2018-11-08

Symbol Technologies, Llc

Method and apparatus for detecting and interpreting price label text

CN110709895B

( en )

2017-05-31

2024-03-29

äº¤äºæ°åVcæ§è¡å¬å¸

Apparatus and method for dynamic white point compensation to improve perceived color of synthesized content

US10311630B2

( en )

2017-05-31

2019-06-04

Verizon Patent And Licensing Inc.

Methods and systems for rendering frames of a virtual scene from different vantage points based on a virtual entity description frame of the virtual scene

US10586377B2

( en )

2017-05-31

2020-03-10

Verizon Patent And Licensing Inc.

Methods and systems for generating virtual reality data that accounts for level of detail

US10347037B2

( en )

2017-05-31

2019-07-09

Verizon Patent And Licensing Inc.

Methods and systems for generating and providing virtual reality data that accounts for level of detail

US9940535B1

( en )

2017-09-07

2018-04-10

Symbol Technologies, Llc

Imaging-based sensor calibration

US10740911B2

( en )

2018-04-05

2020-08-11

Symbol Technologies, Llc

Method, system and apparatus for correcting translucency artifacts in data representing a support structure

US10832436B2

( en )

2018-04-05

2020-11-10

Symbol Technologies, Llc

Method, system and apparatus for recovering label positions

US11327504B2

( en )

2018-04-05

2022-05-10

Symbol Technologies, Llc

Method, system and apparatus for mobile automation apparatus localization

US10823572B2

( en )

2018-04-05

2020-11-03

Symbol Technologies, Llc

Method, system and apparatus for generating navigational data

US10809078B2

( en )

2018-04-05

2020-10-20

Symbol Technologies, Llc

Method, system and apparatus for dynamic path generation

US10861238B2

( en )

2018-05-14

2020-12-08

Microsoft Technology Licensing, Llc

Experiential representation of data in mixed reality

US11074717B2

( en )

2018-05-17

2021-07-27

Nvidia Corporation

Detecting and estimating the pose of an object using a neural network model

US10825223B2

( en )

2018-05-31

2020-11-03

Microsoft Technology Licensing, Llc

Mixed reality animation

US10380804B1

( en )

2018-06-01

2019-08-13

Imajion Corporation

Seamless injection of augmented three-dimensional imagery using a positionally encoded video stream

EP3579127A1

( en )

2018-06-07

2019-12-11

Hexagon Technology Center GmbH

Method of generation of an enhanced plant model

CN109102565B

( en )

2018-07-05

2022-10-14

Method for automatically generating virtual terrain

CN110852143B

( en )

2018-08-21

2024-04-09

åå¹³å°å¬å¸

Interactive text effects in an augmented reality environment

US11022863B2

( en )

2018-09-17

2021-06-01

Tangible Play, Inc

Display positioning system

US11506483B2

( en )

2018-10-05

2022-11-22

Zebra Technologies Corporation

Method, system and apparatus for support structure depth determination

US11010920B2

( en )

2018-10-05

2021-05-18

Zebra Technologies Corporation

Method, system and apparatus for object detection in point clouds

US11003188B2

( en )

2018-11-13

2021-05-11

Zebra Technologies Corporation

Method, system and apparatus for obstacle handling in navigational path generation

US11090811B2

( en )

2018-11-13

2021-08-17

Zebra Technologies Corporation

Method and apparatus for labeling of support structures

US11416000B2

( en )

2018-12-07

2022-08-16

Zebra Technologies Corporation

Method and apparatus for navigational ray tracing

<tr itemprop="forw

Method and system for compositing an augmented reality scene — Apple Inc. (US10565796B2)

Related documents