Conceptio › Archive › Google Patents

Google Patentsopen access

Model retrieval for objects in images using field descriptors — Qualcomm Technologies, Inc. (US11361505B2)

Qualcomm Technologies, Inc. · Google Patents

Google Patents · Patents · License: Open Access

patent, google patents, intellectual property, US11361505B2, Qualcomm Technologies, Inc., Alexander Grabner, en, 2022

ABSTRACT

Abstract

Techniques are provided for one or more three-dimensional models representing one or more objects. For example, an input image including one or more objects can be obtained. From the input image, a location field can be generated for each object of the one or more objects. A location field descriptor can be determined for each object of the one or more objects, and a location field descriptor for an object of the one or more objects can be compared to a plurality of location field descriptors for a plurality of three-dimensional models. A three-dimensional model can be selected from the plurality of three-dimensional models for each object of the one or more objects. A three-dimensional model can be selected for the object based on comparing a location field descriptor for the object to the plurality of location field descriptors for the plurality of three-dimensional models.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No. 62/858,268, filed Jun. 6, 2019, which is hereby incorporated by reference, in its entirety and for all purposes.

FIELD

The present disclosures generally relate to model retrieval for objects in images, and more specifically to determining three-dimensional models for representing objects using field descriptors.

BACKGROUND

Determining objects that are present in real images and attributes of those objects is useful for many applications. For instance, a model can be determined for representing an object in an image, and can be used to facilitate effective operation of various systems. Examples of such applications and systems include augmented reality (AR), robotics, automotive and aviation, three-dimensional scene understanding, object grasping, object tracking, in addition to many other applications and systems.

In AR environments, for example, a user may view images that include an integration of artificial or virtual graphics with the user's natural surroundings. AR applications allow real images to be processed to add virtual objects to the images and to align the virtual objects to the image in multiple dimensions. For instance, a real-world object that exists in reality can be represented using a model that resembles or is an exact match of the real-world object. In one example, a model of a virtual airplane representing a real airplane sitting on a runway may be presented in the view of an AR device (e.g., glasses, goggles, or other device) while the user continues to view his or her natural surroundings in the AR environment. The viewer may be able to manipulate the model while viewing the real-world scene. In another example, an actual object sitting on a table may be identified and rendered with a model that has a different color or different physical attributes in the AR environment. In some cases, artificial virtual objects that do not exist in reality or computer-generated copies of actual objects or structures of the user's natural surroundings can also be added to the AR environment.

SUMMARY

In some embodiments, techniques and systems are described for performing three-dimensional (3D) model retrieval (e.g., in the wild) using location fields and location field descriptors. The techniques and systems can be used to select a 3D model for an object in an image (e.g., a red-green-blue (RGB) image) based on a location field descriptor generated for the object and location field descriptors generated for a plurality of 3D models. In some cases, the 3D model retrieval can be performed using a single image as input.

The techniques described herein establish a common low-level representation in the form of location fields for 3D models and for one or more objects detected in an image. A location field is an image-like representation that encodes a 3D surface coordinate for each object pixel, providing correspondences between 2D pixels and 3D surface coordinates. The location fields for the 3D models can be rendered directly from the 3D models. The location field for an object in an image can be predicted from the image (e.g., using a first convolutional neural network (CNN) or other type of machine learning system).

3D shape descriptors (referred to as âlocation field descriptorsâ) can then be computed from the location fields. A location field descriptor is a 3D shape descriptor that includes information defining the shape of the object (e.g., an object detected in an image or an object represented by a 3D model). Instead of exhaustively comparing location fields from different viewpoints, pose-invariant 3D location field descriptors can be computed (from the location fields) in an embedding space optimized for retrieval from the location fields. The pose-invariant location field descriptors can be computed using a second CNN or other type of machine learning system.

In one illustrative example, an input image can be obtained, and one or more objects can be detected in the image (e.g., using the first CNN). A location field can be generated for each object of the one or more objects. A location field descriptor can then be generated for each object of the one or more objects (e.g., using the second CNN). A location field descriptor can also be generated for each 3D model of a plurality of 3D models (e.g., using the second CNN). The location field descriptor generated for the object can be compared to the location field descriptors generated for the plurality of 3D models. The 3D model having the location field descriptor that is closest (e.g., based on a distance, such as Euclidean distance or Cosine distance) to the location field descriptor of the object can be selected. An output image can then be generated that includes the selected 3D model rendered with the input image (e.g., the 3D model can replace the two dimensional object in the image).

The 3D model selected for representing an object in an image can be provided for use by any suitable application that can utilize a 3D model (e.g., 3D mesh) for performing one or more operations. In one illustrative example, the selected 3D model can be used by an AR application to represent the object in an AR environment. In other examples, the 3D mesh of the 3D model can be used for 3D scene understanding, object grasping (e.g., in robotics, surgical applications, and/or other suitable applications), object tracking, scene navigation, and/or other suitable applications.

According to at least one example, a method of determining one or more three-dimensional models is provided. The method includes determining a location field descriptor for at least one object of one or more objects in an input image, and comparing the location field descriptor for the at least one object to a plurality of location field descriptors for a plurality of three-dimensional models. The method further includes selecting, from the plurality of three-dimensional models, a three-dimensional model for the at least one object. The three-dimensional model is selected for the at least one object based on comparing the location field descriptor for the at least one object to the plurality of location field descriptors for the plurality of three-dimensional models.

In another example, an apparatus for determining one or more three-dimensional models is provided. The apparatus includes a memory configured to store one or more images and a processor implemented in circuitry and coupled to the memory. The processor is configured to and can determine a location field descriptor for at least one object of one or more objects in an input image, and compare the location field descriptor for the at least one object to a plurality of location field descriptors for a plurality of three-dimensional models. The processor is further configured to and can select, from the plurality of three-dimensional models, a three-dimensional model for the at least one object of the one or more objects. The three-dimensional model is selected for the at least one object based on comparing the location field descriptor for the at least one object to the plurality of location field descriptors for the plurality of three-dimensional models.

In another example, a non-transitory computer readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processor to: determine a location field descriptor for at least one object of one or more objects in an input image; compare the location field descriptor for the at least one object to a plurality of location field descriptors for a plurality of three-dimensional models; and select, from the plurality of three-dimensional models, a three-dimensional model for the at least one object, wherein the three-dimensional model is selected for the at least one object based on comparing the location field descriptor for the object to the plurality of location field descriptors for the plurality of three-dimensional models.

In another example, an apparatus for determining one or more three-dimensional models is provided. The apparatus includes means for determining a location field descriptor for at least one object of one or more objects in an input image. The apparatus further includes means for comparing the location field descriptor for the at least one object to a plurality of location field descriptors for a plurality of three-dimensional models. The apparatus further includes means for selecting, from the plurality of three-dimensional models, a three-dimensional model for the at least one object. The three-dimensional model is selected for the at least one object based on comparing the location field descriptor for the at least one object to the plurality of location field descriptors for the plurality of three-dimensional models.

The location field descriptor for the at least one object can be determined from the location field for the at least one object.

In some examples, the three-dimensional model is selected for the at least one object based on the location field descriptor of the at least one object having a closest match with a location field descriptor of the three-dimensional model.

In some examples, comparing the location field descriptor for the at least one object to the plurality of location field descriptors for the plurality of three-dimensional models includes: determining distances between the location field descriptor for the at least one object and the plurality of location field descriptors for the plurality of three-dimensional models.

In some examples, the three-dimensional model is selected for the at least one object based on the location field descriptor of the at least one object having a closest distance with a location field descriptor of the three-dimensional model.

In some examples, the distances include Euclidean distances or Cosine distances.

In some examples, the location field descriptor for the at least one object is based on three-dimensional surface coordinate information for a plurality of pixels associated with the at least one object in the input image.

In some examples, the location field descriptor for the at least one object includes a feature vector with values defining a shape of the at least one object.

In some examples, each three-dimensional model of the plurality of three-dimensional models includes a three-dimensional mesh representing an object.

In some examples, the methods, apparatuses, and computer readable medium described above further comprise: obtaining the plurality of three-dimensional models; and determining the plurality of location field descriptors for the plurality of three-dimensional models, wherein a location field descriptor is determined for each three-dimensional model of the plurality of three-dimensional models.

In some cases, the methods, apparatuses, and computer readable medium described above further comprise generating a location field for a three-dimensional model of the plurality of three-dimensional models by generating a rendering of the three-dimensional model. In some examples, the location field is generated using a first convolutional neural network, where the first convolutional neural network can use the input image as input. In some examples, the methods, apparatuses, and computer readable medium described above further comprise detecting, using the first convolutional neural network, the one or more objects from the input image. In some examples, the location field descriptor for the at least one object is determined using a second convolutional neural network, where the second convolutional neural network can use the location field as input.

In some cases, generating the rendering of the three-dimensional model includes: rasterizing a three-dimensional mesh of the three-dimensional model to determine a three-dimensional surface coordinate for each vertex of the three-dimensional mesh; and interpolating three-dimensional surface coordinates for points between vertices of the three-dimensional mesh.

In some examples, the methods, apparatuses, and computer readable medium described above further comprise generating a plurality of location fields for a three-dimensional model of the plurality of three-dimensional models. A first location field of the plurality of location fields can be generated for a first pose of the three-dimensional model, and a second location field of the plurality of location fields can be generated for a second pose of the three-dimensional model.

In some examples, the methods, apparatuses, and computer readable medium described above further comprise storing the plurality of location field descriptors for the plurality of three-dimensional models in a database.

In some examples, the methods, apparatuses, and computer readable medium described above further comprise determining, for each three-dimensional model of the plurality of three-dimensional models, a pose-invariant center descriptor. In such examples, the plurality of location field descriptors that are compared to the location field descriptor for the at least one object include a plurality of pose-invariant center descriptors. In some cases, the plurality of pose-invariant center descriptors are determined using a convolutional neural network.

In some examples, the methods, apparatuses, and computer readable medium described above further comprise generating an output image based on the selected three-dimensional model and the input image.

In some examples, the methods, apparatuses, and computer readable medium described above further comprise: receiving a user input to manipulate the selected three-dimensional model; and adjusting one or more of a pose, a location, or a property of the selected three-dimensional model in an output image based on the user input.

In some examples, the methods, apparatuses, and computer readable medium described above further comprise: obtaining an additional input image, the additional input image including the at least one object in one or more of a different pose or a different location than a pose or location of the at least one object in the input image; and adjusting one or more of a pose or a location of the selected three-dimensional model in an output image based on a difference between the pose or location of the at least one object in the additional input image and the pose or location of the at least one object in the input image.

In some examples, the apparatus is a mobile device. In some examples, the apparatus includes a camera for capturing one or more images. In some examples, the apparatus includes a display for displaying one or more images.

This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

The foregoing, together with other features and embodiments, will become more apparent upon referring to the following specification, claims, and accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

Illustrative embodiments of the present application are described in detail below with reference to the below figures:

FIG. 1 is diagram illustrating an example of an input image and models retrieved for objects in the input image, in accordance with some examples;

FIG. 2 is a diagram illustrat

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No. 62/858,268, filed Jun. 6, 2019, which is hereby incorporated by reference, in its entirety and for all purposes.

FIELD

The present disclosures generally relate to model retrieval for objects in images, and more specifically to determining three-dimensional models for representing objects using field descriptors.

BACKGROUND

SUMMARY

The location field descriptor for the at least one object can be determined from the location field for the at least one object.

In some examples, the distances include Euclidean distances or Cosine distances.

In some examples, the location field descriptor for the at least one object includes a feature vector with values defining a shape of the at least one object.

In some examples, each three-dimensional model of the plurality of three-dimensional models includes a three-dimensional mesh representing an object.

In some examples, the methods, apparatuses, and computer readable medium described above further comprise generating an output image based on the selected three-dimensional model and the input image.

The foregoing, together with other features and embodiments, will become more apparent upon referring to the following specification, claims, and accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

Illustrative embodiments of the present application are described in detail below with reference to the below figures:

FIG. 1 is diagram illustrating an example of an input image and models retrieved for objects in the input image, in accordance with some examples;

FIG. 2 is a diagram illustrating an example of a projection from a three-dimensional (3D) scene to a two-dimensional (2D) image, in accordance with some examples;

FIG. 3 is a block diagram illustrating an example of a model retrieval system, in accordance with some examples;

FIG. 4A is a diagram illustrating an example of a location field generated for an object in an input image, in accordance with some examples;

FIG. 4B and FIG. 4C are diagrams illustrating an example of a location field generated for a 3D model, in accordance with some examples;

FIG. 5A and FIG. 5B are diagrams illustrating examples of location field descriptors generated from location fields of different objects, in accordance with some examples;

FIG. 6 is a block diagram illustrating an example implementation of the model retrieval system, in accordance with some examples;

FIG. 7 is a diagram illustrating an example of results generated using the model retrieval system, in accordance with some examples;

FIG. 8 is a diagram illustrating another example of results generated using the model retrieval system, in accordance with some examples;

FIG. 9A and FIG. 9B are diagrams illustrating other examples of results generated using the model retrieval system, in accordance with some examples;

FIG. 10A - FIG. 10E are diagrams illustrating other examples of results generated using the model retrieval system, in accordance with some examples;

FIG. 11 is a flowchart illustrating an example of a process of determining one or more 3D models, in accordance with some examples;

FIG. 12 is a flowchart illustrating an example of another process of determining one or more 3D models, in accordance with some examples; and

FIG. 13 illustrates an example of a computing system in which one or more embodiments may be implemented.

DETAILED DESCRIPTION

Certain aspects and embodiments of this disclosure are provided below. Some of these aspects and embodiments may be applied independently and some of them may be applied in combination as would be apparent to those of skill in the art. In the following description, for the purposes of explanation, specific details are set forth in order to provide a thorough understanding of embodiments of the application. However, it will be apparent that various embodiments may be practiced without these specific details. The figures and description are not intended to be restrictive.

The ensuing description provides exemplary embodiments only, and is not intended to limit the scope, applicability, or configuration of the disclosure. Rather, the ensuing description of the exemplary embodiments will provide those skilled in the art with an enabling description for implementing an exemplary embodiment. It should be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the application as set forth in the appended claims.

As described herein, methods and systems are described for performing three-dimensional (3D) model retrieval (e.g., in the wild) using location fields and location field descriptors. For example, as shown in FIG. 1 , given a single image 102 (e.g., an RGB image) with one or more objects, the methods and systems can retrieve

3D models

104 , 106 , and 108 with accurate geometry for each object. The retrieved

3D models

104 , 106 , and 108 can then be provided in an output image 103 .

As described in more detail below, the methods and systems can map two-dimensional (2D) images and 3D models to a common descriptor space that is optimized for 3D model retrieval. For example, a descriptor can be computed for each available 3D model. For a given image of an object, a descriptor can be computed from the image. Based on a comparison of the descriptor computed from the image with the descriptors of the 3D models, the 3D model with the most similar descriptor can be selected for the object in the image. It can be a challenge to compute descriptors (for a 3D model and for an image) that are similar when the image contains an object that is similar to an object represented by the 3D model, and dissimilar when the image contains an object that is different than the object represented by the 3D model. The methods and systems described herein resolve this challenge, in part, by computing the descriptors based on location fields, including location fields generated for the 3D models and a location field generated for each object in an image. Such descriptors are referred to herein as location field descriptors.

Retrieving the three-dimensional (3D) shapes of objects from images (especially a single image) can be useful for many applications, such as augmented reality (AR) applications, 3D scene modeling, 3D scene understanding, object grasping, object tracking, robotics, 3D printing, among others. Compared to model reconstruction, 3D model retrieval provides 3D models designed by humans which are rich in detail. Due to the growing number of large- scale 3D model databases (e.g., ShapeNet, 3D Warehouse, among others), efficient image-based model retrieval approaches have become commonplace.

However, inferring a 3D model from 2D observations can be highly difficult. For example, 3D models and color images have very different natures. As shown in FIG. 2 , inherent information loss occurs due to projection of a 3D scene 203 to a two-dimensional (2D) image 202 . 3D model retrieval can also be difficult due to unknown camera parameters and unknown object poses. Furthermore, training data for training 3D model retrieval machine learning systems is scarce. For instance, there are not many training images with 3D model annotations.

Various techniques can be used for retrieving the 3D shapes of objects. For example, the retrieval task can be addressed by directly mapping 3D models and images (e.g., RGB images) to a common embedding space. However, such a technique can have a number of limitations in practice. For example, the learned mapping is highly prone to overfitting, because training data in the form of RGB images with 3D model annotations is scarce. Further, systems purely trained on synthetic data do not generalize to real data due to the domain gap between RGB images and RGB renderings. Even further, the black box characteristic of these systems makes it hard to understand, why the approaches fail in certain scenarios.

Various techniques can be used in the fields of 3D coordinate regression and 3D model retrieval from a single RGB image. Regressing 3D coordinates from 2D observations is a common problem in computer vision. While some approaches generate 3D point clouds from multi-view RGB images, other approaches predict unstructured 3D point clouds from a single RGB image using deep learning. In some cases, such unstructured 3D point clouds can be used to address various 3D vision tasks with deep learning. As described in more detail below, the techniques described herein predict structured 3D point clouds in the form of location fields. A location field encodes a 3D surface coordinate for each object pixel, and it can be important to know which pixels belong to an object and which pixels belong to the background or another object. Deep learning techniques for instance segmentation can be used to increase the accuracy of location field generation.

Regarding 3D model retrieval, some techniques perform retrieval given a query 3D model. Such techniques either directly operate on 3D data (e.g., in the form of voxel grids, spherical maps, point clouds, or other 3D data), or process multi-view renderings of the query 3D model to compute a shape descriptor. However, as described below, the techniques described herein can be used to perform the much more challenging task of 3D model retrieval from a single image (e.g., an RGB image or other type of image). One approach to retrieve 3D model from a single image is to train a classifier that provides a 3D model for each fine-grained class on top of handcrafted or learned features extracted from the image. Such an approach restricts the retrieval to 3D models seen during training. One way to overcome this limitation is to map 3D models and images to a common embedding space, where model retrieval is performed using distance-based matching (as described below). In this case, the mapping, the embedding space, and the distance measure can be designed in a variety of ways.

In some cases, features extracted from an image can be matched against features extracted from multi-view image renderings to predict both shape and viewpoint. In this context, one approach is to use a convolutional neural network (CNN) trained for ImageNet classification to extract features. In some cases, in addition to using such a CNN, nonlinear feature adaption can be additionally performed to overcome the domain gap between real and rendered images. Another approach is to use a CNN trained for object detection as a feature extractor. However, such a CNN is not optimized for 3D model retrieval.

In some cases, techniques can train mappings to predefined embedding spaces. One approach is to train CNNs to map 3D models, RGB images, depth maps, and sketches to an embedding space based on text for cross-modal retrieval. Another approach is to construct a low-dimensional embedding space by performing principal component analysis (PCA) on 3D key points and map 3D key points predicted using a CNN to that space for retrieval. Another approach is to train a CNN to map RGB images to an embedding space computed from pairwise similarities between 3D models.

Instead of handcrafting an embedding space, an embedding space capturing 3D shape properties can be learned. One approach is to reconstruct voxel grids from RGB images of objects using CNNs. The low-dimensional bottle-neck shape descriptor can also be used for retrieval. Another approach is to combine a 3D voxel encoder and an RGB image encoder with a shared 3D voxel decoder to perform reconstruction from a joint embedding. 3D model retrieval can then be performed by matching embeddings of voxel grids against those of RGB images.

In some cases, an embedding space can be explicitly learned that is optimized for 3D model retrieval. One approach is to use a single CNN to map RGB images and RGB renderings to an embedding space that is optimized using a Euclidean distance-based lifted structure loss. At test time, the distances between an embedding of an RGB image and embeddings of multi-view RGB renderings can be averaged to compensate for the unknown object pose. Another approach is to use two CNNs to map RGB images and gray-scale renderings to an embedding space and optimize a Euclidean distance-based Triplet loss. In some cases, cross-view convolutions can be employed to aggregate a sequence of multi-view renderings into a single descriptor to reduce the matching complexity. Another approach is to also train two CNNs, but map RGB images and depth maps to a common space. In contrast to other approaches, the 3D pose of the object in the RGB image is explicitly estimated and used in the 3D model retrieval.

As noted above, methods and systems are described for performing 3D model retrieval using location fields and location field descriptors. The techniques described herein learn an embedding space that is optimized for 3D model retrieval, but first predict location fields from images (e.g., RGB images) and 3D models. 3D shape descriptors (e.g., pose-invariant 3D shape descriptors), referred to as location field descriptors, can then be computed from predicted and rendered location fields in an end-to-end trainable way. For example, a 3D model can be selected for an object in an image by matching a location field descriptor generated for the object with the location field descriptors generated for the 3D models to find the best matching 3D model. Given a single image showing one or more objects, the 3D model retrieval techniques described herein can retrieve a 3D model with accurate geometry for each object in an image.

FIG. 3 is a block diagram illustrating an example of a model retrieval system 304 . The model retrieval system 304 includes various components, including a location field generation engine 308 , a location field rendering engine 310 , a location field descriptor generation engine 312 , and a descriptor matching engine 314 . The components (e.g., the various engines) of the model retrieval system 304 can include and/or can be implemented using electronic circuits or other electronic hardware, which can include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and/or other suitable electronic circuits), and/or can include and/or be implemented using computer software, firmware, or any combination thereof, to perform the various operations described herein.

While the model retrieval system 304 is shown to include certain components, one of ordinary skill will appreciate that the model retrieval system 304 can include more or fewer components than those shown in FIG. 3 . For example, the model retrieval system 304 can also include an input device and an output device (not shown). The model retrieval system 304 may also include, in some instances, one or more memory devices (e.g., one or more random access memory (RAM) components, read-only memory (ROM) components, cache memory components, buffer components, database components, and/or other memory devices), one or more processing devices (e.g., one or more CPUs, GPUs, and/or other processing devices implemented in circuitry) in communication with and/or electrically connected to the one or more memory devices, one or more wireless interfaces (e.g., including one or more transceivers and a baseband processor for each wireless interface) for performing wireless communications, one or more wired interfaces (e.g., a serial interface such as a universal serial bus (USB) input, a lightening connector, and/or other wired interface) for performing communications over one or more hardwired connections, and/or other components that are not shown in FIG. 3 .

The model retrieval system 304 can obtain the input images 302 from an image source (not shown). The model retrieval system 304 can process the obtained input images 302 to determine one or more output 3D models 316 for representing one or more objects detected in the input images 302 . The input images 302 can include color images, such as red-green-blue (RGB) images, images having luma and chroma (or chroma-difference) color components (e.g., YCbCr, Yâ²CbCr, YUV, or the like), or images in any other suitable color format. RGB images will be used in various examples provided herein, however one of ordinary skill will appreciate that the techniques described herein can be performed using any type of image. The input images 302 can be one or more stand-alone images, or can be part of a sequence of images, such as a video, a burst of images, or other sequence of images. The image source can include an image capture device (e.g., a camera, a camera phone, a video camera, a tablet device with a built-in camera, or other suitable image capture device), an image storage device, an image archive containing stored images, an image server or content provider providing image data, a media feed interface receiving images or video from a server or content provider, a computer graphics system for generating computer graphics image data, a combination of such sources, and/or other source of image content.

In some examples, the model retrieval system 304 and the image source can be part of the same computing device. For example, the computing device can include an electronic device, such as a camera (e.g., a digital camera, a camera phone, a video phone, a tablet device with a built-in camera or other suitable capture device), a mobile or stationary telephone handset (e.g., smartphone, cellular telephone, or the like), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a display device, a digital media player, a video gaming console, a video streaming device, a head-mounted display (HMD) or virtual reality headset, a heads-up display (HUD), a vehicle (e.g., an autonomous vehicle or a human-driven vehicle), or any other suitable electronic device. In some cases, the computing device (or devices) can include one or more wireless transceivers for wireless communications. In some examples, the model retrieval system 304 and the image source can be part of separate computing devices.

In contrast to previous methods that directly map 3D models and images to an embedding space, the model retrieval system 304 establishes a common low-level representation in the form of location fields for 3D models and for one or more objects detected in an image. The location field for an object in an image is predicted from the image. For example, the location field generation engine 308 can generate or predict a location filed for each object in an input image from the input images 302 (e.g., a first location field for a first object, a second location field for a second object, and so on). In some examples, the location field generation engine 308 can detect objects in an image, and can then compute a location field for each detected object.

In some cases, as described below, the location field generation engine 308 can include a convolutional neural network (CNN) (referred to as a location field CNN or LF-CNN) or other type of machine learning system that can be used to generate location fields for objects detected in an image. The LF-CNN can be trained to predict location fields for images of objects. In some cases, during a testing or inference stage of the LF-CNN, the LF-CNN can be used to detect objects in 2D in the image and compute a LF for each detected object. For example, LF-CNN can be a custom network that is trained to detect and localize one or multiple objects in an image (e.g., an RGB image) and to predict a LF for each detected object. In one illustrative example described in more detail below, the LF-CNN can be a Mask-RCNN with custom prediction branches.

The location fields for the 3D models 306 can be rendered directly from the 3D models 306 . For example, the location field rendering engine 310 can render a location field rendering from each 3D model from the 3D models 306 (e.g., a first location field for a first model, a second location field for a second model, and so on). In one illustrative example described in more detail below, location fields can be generated directly from the 3D models 306 via custom shaders, such as using OpenGL and custom shaders.

A location field is an image-like representation that encodes a 3D surface coordinate in the canonical object coordinate system for each surface (or visible) pixel of an object, providing correspondences between 2D pixels and 3D surface coordinates. FIG. 4A is a diagram illustrating an example of a location field 420 generated for a chair object in an input image 402 . Each pixel of the location field 420 includes an x-coordinate, a y-coordinate, and a z-coordinate relative to a center point of the object, indicating the 3D surface coordinate for that pixel. The three channels of the location field 420 are shown in FIG. 4A , which correspond to the x-, y-, and z-values of the 3D coordinates in separate images. For example, as shown in FIG. 4A , a horizontal x-component location field (LF(X)) includes the x-coordinate of each surface pixel of the chair, a vertical y-component location field (LF(Y)) includes the y-coordinate of each surface pixel of the chair, and a z-component location field (LF(Z)) in the depth direction includes the z-coordinate of each surface pixel of the chair. As shown in FIG. 4B and FIG. 4C (discussed below), a âz value corresponds to the front direction (coming out of the image), a +x value corresponds to the right direction, and a +y value corresponds to an up direction. The location field representation captures 3D shape and 3D pose information efficiently in an image-like structure, which is well suited for processing with machine learning systems, such as CNNs. Location fields implicitly handle occlusions and truncations and are invariant to texture and lighting. Because a location field encodes correspondences between 2D pixels and 3D surface coordinates, it explicitly captures 3D shape and 3D pose information without appearance variations that are irrelevant to the 3D model retrieval task.

FIG. 4B and FIG. 4C include a location field 430 rendered for a 3D model of a chair. For example, the 3D models 306 can be pre-processed (e.g., by the location field rendering engine 310 or by another component of the model retrieval system that is not shown in FIG. 3 ) by aligning the models in a consistent manner before generating the location fields for the 3D models 306 . For example, the models can be scaled and/or translated to fit inside of a unit cube, and can be rotated to have a consistent front-facing direction in their canonical coordinate space. Referring to FIG. 4B , as shown in the LF(X) component of the location field 430 , the chair model is scaled to fit inside a unit cube. A unit cube includes side lengths equal to one, and sits in the center of the coordinate system, as indicated by the (x,y,z) value of (0,0,0) in the LF(X) component of the location field 430 . As shown in FIG. 4C , the chair model can also be rotated to a front facing direction in the canonical coordinate space. In some cases, a location field generated from an input image (e.g., predicted by the location field generation engine 308 , which as described in more detail below can include a location field convolutional neural network (LF-CNN) can already have the consistent alignment described above, because the LF-CNN can be trained on location fields rendered from 3D models that were pre-processed using consistent alignment.

Regarding 3D model retrieval, location fields have several advantages compared to other rendered representations, such as RGB renderings, texture-less gray-scale renderings, silhouettes, depth renderings, normal renderings, or other renderings. For example, RGB renderings are subject to appearance variations, which are irrelevant for the task caused by material, texture, and lighting. Texture-less gray-scale renderings are affected by scene lighting. Silhouettes are not affected by such appearance variations, but discard valuable 3D shape information. Depth and normal renderings capture 3D geometry but lose the relation to the 3D pose in the object's canonical coordinate system. In contrast, location fields explicitly present 3D shape and 3D pose information, as they establish correspondences between 2D object pixels and 3D coordinates on the object surface. With respect to the 3D shape, the dense 3D coordinates provide a partial reconstruction of the object geometry. With respect to the 3D pose, the object rotation and translation can be geometrically recovered (if needed) from the 2D-3D correspondences using a PnP algorithm.

The location field descriptor generation engine 312 can compute location field descriptors from the location fields of the 3D models 306 and from the location fields of the objects from the input images 302 . A location field descriptor is a 3D shape descriptor that includes information defining the shape of the object (e.g., an object detected in an image or an object represented by a 3D model). A location field descriptor can be in the form of an M-dimensional vector, where M is an integer greater than 1. In some cases, instead of exhaustively comparing location fields from different viewpoints, pose-invariant 3D location field descriptors (referred to herein as pose-invariant center descriptors, or center descriptors) can be computed from the location fields in an embedding space optimized for retrieval from the location fields. In some cases, a database of location field descriptors can be built for the 3D models 306 . In some implementations, the location field descriptors can be generated for the 3D models 306 offline before 3D model retrieval is performed for the input images 302 (before the input images 302 are analyzed for selecting 3D models for objects in the input images), in which case the database of location field descriptors can be built and then used to perform 3D model retrieval for input images.

In some cases, the location field descriptor generation engine 312 can include a CNN (referred to as a location field descriptor CNN or LFD-CNN) or other type of machine learning system that can be used to compute the location field descriptors. The LFD-CNN can be a custom network that maps location fields to a descriptor space that is optimized for 3D model retrieval. The LFD-CNN can be trained to predict location field descriptors from both location fields predicted from images and location fields rendered from 3D models. During training, a pose-invariant center descriptor can be learned for each 3D model. In one illustrative example described in more detail below, the LFD-CNN can include a DenseNet-like CNN and can optimize a Triplet-Center-Loss. During the training, the LFD-CNN can be optimized so that the distances (e.g., Euclidean distance, Cosine distance, or other suitable distance) between location field descriptors of location fields showing the same 3D model under different poses are small, but the distances between location field descriptors of location fields showing different 3D models are large. For example, referring to FIG. 5A and FIG. 5B , the dots with the different patterns represent different location field descriptors generated using the location field descriptor generation engine 312 . The location field descriptors shown with a dot having the same pattern are associated with locations fields showing the same 3D model. During a testing or inference stage of the LFD-CNN when analyzing input images, the LFD-CNN can compute a location field descriptor for each location field generated for an object in an image (and in some cases location fields for the models 306 ).

The descriptor matching engine 314 can compare the location field descriptor generated for an object detected in an input image to the location field descriptors (e.g., stored in the database of location field descriptors) generated for the 3D models 306 . When a pose-invariant center descriptor is generated for each location field, only a single center descriptor needs to be evaluated during inference for each 3D model in the database, allowing the 3D model retrieval fast and scalable. The 3D model having the location field descriptor that is closest (e.g., based on a distance, such as Euclidean distance or Cosine distance) to the location field descriptor of the object can be selected as an output 3D model 316 for representing the object. In some cases, a ranked list of 3D models for the object can be generated, and a 3D model can be selected from the list (e.g., a top ranged 3D model). The ranked list of 3D models can include the best matching 3D model having a highest rank, a second- best matching 3D model having a second-to-highest rank, and so on.

The 3D model selected for representing an object in an image can be provided for use by any suitable application that can utilize a 3D model (e.g., 3D mesh) for performing one or more operations. In one illustrative example, an output image (e.g., image 103 shown in FIG. 3 ) can be generated that includes the selected 3D model 316 rendered with the input image. For example, the selected 3D model can be used by an augmented reality (AR), virtual reality (VR), and/or mixed reality (MR) application to represent the object in an AR, VR, and/or MR environment. In one example, referring to FIG. 6 , the top-ranked 3D chair model from the ranked 3D models 616 can be selected to represent the chair object in the input image 602 . The 3D chair model can replace the chair object in

CLAIMS

Claims ( 30 )

What is claimed is:

1. A method of determining one or more three-dimensional models, the method comprising:

obtaining an input image including at least one object;

generating, from the input image, three-dimensional surface coordinate information for a plurality of pixels associated with the at least one object in the input image;

generating, based on the three-dimensional surface coordinate information, at least one location field descriptor for the at least one object in the input image, the at least one location field descriptor including information defining a shape of the at least one object;

comparing the at least one location field descriptor for the at least one object to a plurality of location field descriptors including information defining one or more shapes of a plurality of three-dimensional models; and

selecting, from the plurality of three-dimensional models, a three-dimensional model for the at least one object, wherein the three-dimensional model is selected for the at least one object based on comparing the at least one location field descriptor for the at least one object to the plurality of location field descriptors for the plurality of three-dimensional models.

2. The method of claim 1 , wherein the three-dimensional model is selected for the at least one object based on the at least one location field descriptor of the at least one object having a closest match with a location field descriptor of the three-dimensional model.

3. The method of claim 1 , wherein comparing the at least one location field descriptor for the at least one object to the plurality of location field descriptors for the plurality of three-dimensional models includes:

determining distances between the at least one location field descriptor for the at least one object and the plurality of location field descriptors for the plurality of three-dimensional models.

4. The method of claim 3 , wherein the three-dimensional model is selected for the at least one object based on the at least one location field descriptor of the at least one object having a closest distance with a location field descriptor of the three-dimensional model.

5. The method of claim 3 , wherein the distances include Euclidean distances or Cosine distances.

6. The method of claim 1 , wherein the information defining the shape of the at least one object in the input image includes a feature vector with values defining the shape of the at least one object.

7. The method of claim 1 , wherein each three-dimensional model of the plurality of three-dimensional models includes a three-dimensional mesh representing an object.

8. The method of claim 1 , further comprising:

obtaining the plurality of three-dimensional models; and

determining the plurality of location field descriptors for the plurality of three-dimensional models, wherein a location field descriptor is determined for each three-dimensional model of the plurality of three-dimensional models.

9. The method of claim 1 , further comprising generating a location field for a three-dimensional model of the plurality of three-dimensional models.

10. The method of claim 9 , wherein generating the location field for the three-dimensional model is based on a rendering of the three-dimensional model, wherein the rendering of the three-dimensional model is generated at least in part by:

rasterizing a three-dimensional mesh of the three-dimensional model to determine a three-dimensional surface coordinate for each vertex of the three-dimensional mesh; and

interpolating three-dimensional surface coordinates for points between vertices of the three-dimensional mesh.

11. The method of claim 1 , wherein generating the three-dimensional surface coordinate information for the plurality of pixels includes generating a location field for the at least one object, the location field including the three-dimensional surface coordinate information.

12. The method of claim 11 , wherein the location field is generated using a first convolutional neural network, the first convolutional neural network using the input image as input.

13. The method of claim 12 , wherein the at least one location field descriptor for the at least one object is determined using a second convolutional neural network, the second convolutional neural network using the location field as input.

14. The method of claim 12 , further comprising:

detecting, using the first convolutional neural network, the at least one object from the input image.

15. The method of claim 1 , further comprising:

generating a plurality of location fields for a three-dimensional model of the plurality of three-dimensional models, a first location field of the plurality of location fields being generated for a first pose of the three-dimensional model, and a second location field of the plurality of location fields being generated for a second pose of the three-dimensional model.

16. The method of claim 1 , further comprising:

storing the plurality of location field descriptors for the plurality of three-dimensional models in a database.

17. The method of claim 1 , further comprising:

determining, for each three-dimensional model of the plurality of three-dimensional models, a pose-invariant center descriptor, wherein the plurality of location field descriptors that are compared to the at least one location field descriptor for the at least one object include a plurality of pose-invariant center descriptors.

18. The method of claim 17 , wherein the plurality of pose-invariant center descriptors are determined using a convolutional neural network.

19. The method of claim 1 , further comprising:

generating an output image based on the selected three-dimensional model and the input image.

20. The method of claim 1 , further comprising:

receiving a user input to manipulate the selected three-dimensional model; and

adjusting one or more of a pose, a location, or a property of the selected three-dimensional model in an output image based on the user input.

21. The method of claim 20 , wherein one or more of the pose, the location, or the property of the selected three-dimensional model is adjusted based on semantic information defined for the selected three-dimensional model.

22. The method of claim 1 , further comprising:

obtaining an additional input image, the additional input image including the at least one object in one or more of a different pose or a different location than a pose or location of the at least one object in the input image; and

adjusting one or more of a pose or a location of the selected three-dimensional model in an output image based on a difference between the pose or location of the at least one object in the additional input image and the pose or location of the at least one object in the input image.

23. An apparatus for determining one or more three-dimensional models, comprising:

a memory configured to store one or more images; and

a processor implemented in circuitry and configured to:

obtain an input image including at least one object;

generate, from the input image, three-dimensional surface coordinate information for a plurality of pixels associated with the at least one object in the input image;

generate, based on the three-dimensional surface coordinate information, at least one location field descriptor for the at least one object in the input image, the at least one location field descriptor including information defining a shape of the at least one object;

compare the at least one location field descriptor for the at least one object to a plurality of location field descriptors including information defining one or more shapes of a plurality of three-dimensional models; and

select, from the plurality of three-dimensional models, a three-dimensional model for the at least one object, wherein the three-dimensional model is selected for the at least one object based on comparing the at least one location field descriptor for the at least one object to the plurality of location field descriptors for the plurality of three-dimensional models.

24. The apparatus of claim 23 , wherein the three-dimensional model is selected for the at least one object based on the at least one location field descriptor of the at least one object having a closest match with a location field descriptor of the three-dimensional model.

25. The apparatus of claim 23 , wherein the information defining the shape of the at least one object in the input image includes a feature vector with values defining the shape of the at least one object.

26. The apparatus of claim 23 , wherein the processor is further configured to:

generate an output image based on the selected three-dimensional model and the input image.

27. The apparatus of claim 23 , wherein the processor is further configured to:

receive a user input to manipulate the selected three-dimensional model; and

adjust one or more of a pose, a location, or a property of the selected three-dimensional model in an output image based on the user input.

28. The apparatus of claim 23 , wherein the apparatus comprises a mobile device with a camera for capturing the one or more images.

29. The apparatus of claim 23 , further comprising a display for displaying the one or more images.

30. The apparatus of claim 23 , wherein, to generate the three-dimensional surface coordinate information for the plurality of pixels, the processor is configured to generate a location field for the at least one object, the location field including the three-dimensional surface coordinate information.

US16/655,174

2019-06-06

2019-10-16

Model retrieval for objects in images using field descriptors

Active

2040-02-12

US11361505B2

( en )

Priority Applications (3)

Application Number

Priority Date

Filing Date

Title

US16/655,174

US11361505B2

( en )

2019-06-06

2019-10-16

Model retrieval for objects in images using field descriptors

PCT/US2020/025415

WO2020247048A1

( en )

2019-06-06

2020-03-27

Model retrieval for objects in images using field descriptors

CN202080040032.8A

CN114041170A

( en )

2019-06-06

2020-03-27

Model retrieval for objects in an image using field descriptors

Applications Claiming Priority (2)

Application Number

Priority Date

Filing Date

Title

US201962858268P

2019-06-06

US16/655,174

US11361505B2

( en )

2019-06-06

2019-10-16

Model retrieval for objects in images using field descriptors

Publications (2)

Publication Number

Publication Date

US20200388071A1

US20200388071A1 ( en )

2020-12-10

US11361505B2

true

US11361505B2 ( en )

2022-06-14

Family

ID=73650708

Family Applications (1)

Application Number

Title

Priority Date

Filing Date

US16/655,174

Active

2040-02-12

US11361505B2

( en )

2019-06-06

2019-10-16

Model retrieval for objects in images using field descriptors

Country Status (3)

Country

Link

( 1 )

US11361505B2

( en )

( 1 )

CN114041170A

( en )

( 1 )

WO2020247048A1

( en )

Cited By (3)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20220189170A1

( en )

2019-02-22

2022-06-16

Google Llc

Memory-Guided Video Object Detection

US20230100305A1

( en )

2021-09-27

2023-03-30

Baidu Usa Llc

System and process for repainting of planar objects in video

US20230342368A1

( en )

2022-04-24

2023-10-26

Jingdong City (Beijing) Digits Technology Co., Ltd.

Method and apparatus for spatial data processing

Families Citing this family (42)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US10192129B2

( en )

2015-11-18

2019-01-29

Adobe Systems Incorporated

Utilizing interactive deep learning to select objects in digital visual media

US11568627B2

( en )

2015-11-18

2023-01-31

Adobe Inc.

Utilizing interactive deep learning to select objects in digital visual media

WO2019205069A1

( en )

2018-04-27

2019-10-31

Beijing Didi Infinity Technology And Development Co., Ltd.

Systems and methods for updating 3d model of building

US11244195B2

( en )

2018-05-01

2022-02-08

Adobe Inc.

Iteratively applying neural networks to automatically identify pixels of salient objects portrayed in digital images

US11133059B2

( en )

2018-12-06

2021-09-28

Western Digital Technologies, Inc.

Non-volatile memory die with deep learning neural network

US12393845B2

( en )

2018-12-06

2025-08-19

Western Digital Technologies, Inc.

Non-volatile memory die with deep learning neural network

US11282208B2

( en )

2018-12-24

2022-03-22

Adobe Inc.

Identifying target objects using scale-diverse segmentation neural networks

EP3675062A1

( en )

2018-12-29

2020-07-01

Dassault SystÃ¨mes

Learning a neural network for inference of solid cad features

EP3675063B1

( en )

2018-12-29

2026-02-11

Dassault SystÃ¨mes

Forming a dataset for inference of solid cad features

US11501109B2

( en )

2019-06-20

2022-11-15

Western Digital Technologies, Inc.

Non-volatile memory die with on-chip data augmentation components for use with machine learning

US11520521B2

( en )

2019-06-20

2022-12-06

Western Digital Technologies, Inc.

Storage controller having data augmentation components for use with non-volatile memory die

US11494953B2

( en )

2019-07-01

2022-11-08

Microsoft Technology Licensing, Llc

Adaptive user interface palette for augmented reality

US11908149B2

( en )

2019-10-10

2024-02-20

Andrew Thomas Busey

Pattern-triggered object modification in augmented reality system

FR3104786B1

( en )

2019-12-12

2022-01-21

Retail Vr

METHOD AND SYSTEM FOR GENERATING 3D DIGITAL MODELS

CN113362382B

( en )

2020-03-04

2024-09-20

åä¸ºææ¯æéå¬å¸

Three-dimensional reconstruction method and three-dimensional reconstruction device

CN115605862B

( en )

2020-03-04

2025-04-25

è¥¿é¨åå·¥ä¸è½¯ä»¶æéå¬å¸

Training a differentiable renderer and neural network for querying 3D model databases

US20210279967A1

( en )

2020-03-06

2021-09-09

Apple Inc.

Object centric scanning

US11238650B2

( en )

2020-03-13

2022-02-01

Nvidia Corporation

Self-supervised single-view 3D reconstruction via semantic consistency

US11475589B2

( en )

2020-04-03

2022-10-18

Fanuc Corporation

3D pose estimation by a 2D camera

US11830253B2

( en )

2020-04-14

2023-11-28

Toyota Research Institute, Inc.

Semantically aware keypoint matching

US11367263B1

( en )

2020-06-24

2022-06-21

Amazon Technologies, Inc.

Image-guided three dimensional modeling

US11335004B2

( en )

2020-08-07

2022-05-17

Adobe Inc.

Generating refined segmentation masks based on uncertain pixels

US12055944B2

( en )

2020-12-09

2024-08-06

Continental Autonomous Mobility US, LLC

Method for determining a tow hitch position

US11676279B2

( en )

2020-12-18

2023-06-13

Adobe Inc.

Utilizing a segmentation neural network to process initial object segmentations and object user indicators within a digital image to generate improved object segmentations

CN116745814A

( en )

2020-12-18

2023-09-12

è¥¿é¨åè¡ä»½å¬å¸

Multidimensional object pose estimation and refinement

CN112732956B

( en )

2020-12-24

2024-11-26

æ±èæºæ°´æºè½ç§ææéè´£ä»»å¬å¸

Efficient query method based on perceptual multimodal big data

US11823415B2

( en )

2021-03-03

2023-11-21

Nvidia Corporation

3D pose estimation in robotics

US20230260240A1

( en )

2021-03-11

2023-08-17

Quintar, Inc.

Alignment of 3d graphics extending beyond frame in augmented reality system with remote presentation

US11875510B2

( en )

2021-03-12

2024-01-16

Adobe Inc.

Generating refined segmentations masks via meticulous object segmentation

CN112949614B

( en )

2021-04-29

2021-09-10

æé½å¸å¨èç§ææéå¬å¸

Face detection method and device for automatically allocating candidate areas and electronic equipment

CN113095304B

( en )

2021-06-08

2021-09-03

æé½èææ ç¶ç§ææéå¬å¸

Method for weakening influence of resampling on pedestrian re-identification

US12536733B2

( en )

2021-09-10

2026-01-27

Nvidia Corporation

Single-image inverse rendering

US12020400B2

( en )

2021-10-23

2024-06-25

Adobe Inc.

Upsampling and refining segmentation masks

CN114511846B

( en )

2022-01-24

2025-04-04

æµæ±å¤§å¦

A real-time 3D object detection method based on point cloud cross-view feature conversion

CN115098717A

( en )

2022-05-11

2022-09-23

æ¸åå¤§å¦

Three-dimensional model retrieval method and device, electronic equipment and storage medium

CN114647753B

( en )

2022-05-23

2022-08-12

åä¸å¸èå¤§å¦

A fine-grained sketch retrieval 3D model method for multi-region spatial alignment

US12243095B2

( en )

2022-08-26

2025-03-04

Ebay Inc.

Techniques for generating a three dimensional model for a listing

WO2024112833A1

( en )

2022-11-21

2024-05-30

Georgia Tech Research Corporation

Self-training object perception system

US12086857B2

( en )

2022-12-19

2024-09-10

Google Llc

Search with machine-learned model-generated queries

CN117213469A

( en )

2023-11-07

2023-12-12

ä¸å»ºä¸å±ä¿¡æ¯ç§ææéå¬å¸

Synchronous positioning and mapping methods, systems, equipment and storage media for subway station halls

CN117473105B

( en )

2023-12-28

2024-04-05

æµªæ½®çµåä¿¡æ¯äº§ä¸è¡ä»½æéå¬å¸

Three-dimensional content generation method based on multi-mode pre-training model and related components

US20250336219A1

( en )

2024-04-24

2025-10-30

Varjo Technologies Oy

Three-dimensional object identification and segmentation

Citations (18)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20050276443A1

( en )

2004-05-28

2005-12-15

Slamani Mohamed A

Method and apparatus for recognizing an object within an image

US20050286767A1

( en )

2004-06-23

2005-12-29

Hager Gregory D

System and method for 3D object recognition using range and intensity

US20090034791A1

( en )

2006-12-04

2009-02-05

Lockheed Martin Corporation

Image processing for person and object Re-identification

US20170061252A1

( en )

2015-08-28

2017-03-02

Thomson Licensing

Method and device for classifying an object of an image and corresponding computer program product and computer-readable medium

US20170185823A1

( en )

2014-04-07

2017-06-29

Eyeways Systems Ltd.

Apparatus And Method For Image-Based Positioning, Orientation And Situational Awareness

US20170304732A1

( en )

2014-11-10

2017-10-26

Lego A/S

System and method for toy recognition

US20180182113A1

( en )

2015-08-21

2018-06-28

Panasonic Intellectual Property Management Co., Ltd.

Image processing device and image processing method

US20180204347A1

( en )

2016-12-28

2018-07-19

Volvo Car Corporation

Method and system for vehicle localization from camera image

US20190043172A1

( en )

2017-08-04

2019-02-07

Outward, Inc.

Machine learning based image processing techniques

US20190087976A1

( en )

2017-09-19

2019-03-21

Kabushiki Kaisha Toshiba

Information processing device, image recognition method and non-transitory computer readable medium

US20190147642A1

( en )

2017-11-15

2019-05-16

Google Llc

Learning to reconstruct 3d shapes by rendering many 3d views

US20190156119A1

( en )

2012-10-15

2019-05-23

Tangible Play, Inc.

Virtualization of Tangible Interface Objects

US20190333245A1

( en )

2018-04-27

2019-10-31

Microsoft Technology Licensing, Llc

Location tracking

US20190385352A1

( en )

2016-12-06

2019-12-19

Koninklijke Philips N.V.

Apparatus and method for generating a light intensity image

US10706321B1

( en )

2016-05-20

2020-07-07

Ccc Information Services Inc.

Image processing system to align a target object in a target object image with an object model

US20200250879A1

( en )

2019-02-05

2020-08-06

X Development Llc

Scene recognition using volumetric substitution of real world objects

US20200265238A1

( en )

2019-02-18

2020-08-20

Verizon Patent And Licensing Inc.

Methods and Systems for Identification and Augmentation of Video Content

US20210295594A1

( en )

2016-10-08

2021-09-23

Purdue Research Foundation

Method and Apparatus for Generating Two-Dimensional Image Data Describing a Three-Dimensional Image

Family Cites Families (2)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US10977818B2

( en )

2017-05-19

2021-04-13

Manor Financial, Inc.

Machine learning based model localization system

US10769411B2

( en )

2017-11-15

2020-09-08

Qualcomm Technologies, Inc.

Pose estimation and model retrieval for objects in images

2019

2019-10-16

US16/655,174

patent/US11361505B2/en

active

Active

2020

2020-03-27

PCT/US2020/025415

patent/WO2020247048A1/en

not_active

Ceased

2020-03-27

CN202080040032.8A

patent/CN114041170A/en

active

Pending

Patent Citations (18)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20050276443A1

( en )

2004-05-28

2005-12-15

Slamani Mohamed A

Method and apparatus for recognizing an object within an image

US20050286767A1

( en )

2004-06-23

2005-12-29

Hager Gregory D

System and method for 3D object recognition using range and intensity

US20090034791A1

( en )

2006-12-04

2009-02-05

Lockheed Martin Corporation

Image processing for person and object Re-identification

US20190156119A1

( en )

2012-10-15

2019-05-23

Tangible Play, Inc.

Virtualization of Tangible Interface Objects

US20170185823A1

( en )

2014-04-07

2017-06-29

Eyeways Systems Ltd.

Apparatus And Method For Image-Based Positioning, Orientation And Situational Awareness

US20170304732A1

( en )

2014-11-10

2017-10-26

Lego A/S

System and method for toy recognition

US20180182113A1

( en )

2015-08-21

2018-06-28

Panasonic Intellectual Property Management Co., Ltd.

Image processing device and image processing method

US20170061252A1

( en )

2015-08-28

2017-03-02

Thomson Licensing

Method and device for classifying an object of an image and corresponding computer program product and computer-readable medium

US10706321B1

( en )

2016-05-20

2020-07-07

Ccc Information Services Inc.

Image processing system to align a target object in a target object image with an object model

US20210295594A1

( en )

2016-10-08

2021-09-23

Purdue Research Foundation

Method and Apparatus for Generating Two-Dimensional Image Data Describing a Three-Dimensional Image

US20190385352A1

( en )

2016-12-06

2019-12-19

Koninklijke Philips N.V.

Apparatus and method for generating a light intensity image

US20180204347A1

( en )

2016-12-28

2018-07-19

Volvo Car Corporation

Method and system for vehicle localization from camera image

US20190043172A1

( en )

2017-08-04

2019-02-07

Outward, Inc.

Machine learning based image processing techniques

US20190087976A1

( en )

2017-09-19

2019-03-21

Kabushiki Kaisha Toshiba

Information processing device, image recognition method and non-transitory computer readable medium

US20190147642A1

( en )

2017-11-15

2019-05-16

Google Llc

Learning to reconstruct 3d shapes by rendering many 3d views

US20190333245A1

( en )

2018-04-27

2019-10-31

Microsoft Technology Licensing, Llc

Location tracking

US20200250879A1

( en )

2019-02-05

2020-08-06

X Development Llc

Scene recognition using volumetric substitution of real world objects

US20200265238A1

( en )

2019-02-18

2020-08-20

Verizon Patent And Licensing Inc.

Methods and Systems for Identification and Augmentation of Video Content

Non-Patent Citations (9)

* Cited by examiner, â Cited by third party

Title

GRABNER ALEXANDER; ROTH PETER M.; LEPETIT VINCENT: " Location Field Descriptors: Single Image 3D Model Retrieval in the Wild ", 2019 INTERNATIONAL CONFERENCE ON 3D VISION (3DV), IEEE, 16 September 2019 (2019-09-16), pages 583 - 593, XP033653341, DOI: 10.1109/3DV.2019.00070

Grabnera., et al., " Location Field Descriptors: Single Image 3D Model Retrieval in the Wild ", 2019 International Conference on 3D Vision (3DV), IEEE, Sep. 16, 2019 (Sep. 16, 2019), pp. 583-593, XP033653341, DOI: 10.1109/3DV.2019.00070 [retrieved on Oct. 28, 2019] the whole document.

International Search Report and Written OpinionâPCT/US2020/025415âISA/EPOâdated Jun. 15, 2020.

Lee et al., " Cross-Domain Image-Based 3D Shape Retrieval by View Sequence Learning, " Sep. 2018, IEEE, pp. 258-266 (Year: 2018).

Lu L., et al., " A Three-Tiered Approach to Articulated Object Action Modeling and Recognition ", Neural Information Processing and Systems, Vancouver, B.C. Canada, Dec. 2004, 8 Pages.

Wang Y., et al., " 3D Pose Estimation for Fine-Grained Object Categories ", arxiv.org, Cornell University Library, 201 Olin Library Cornell University Ithaca, NY, 14853, Jun. 12, 2018 (Jun. 12, 2018), 14 Pages, XP081426011, the whole document.

Wang Y., et al., " Improving Annotation for 3D Pose Dataset of Fine-Grained Object Categories ", arxiv.org, Cornell University Library, 201 Olin Library Cornell University Ithaca, NY, 14853, Oct. 19, 2018 (Oct. 19, 2018), 10 Pages, XP081068662, the whole document.

YAMING WANG; XIAO TAN; YI YANG; XIAO LIU; ERRUI DING; FENG ZHOU; LARRY S. DAVIS: " 3D Pose Estimation for Fine-Grained Object Categories ", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, 12 June 2018 (2018-06-12), 201 Olin Library Cornell University Ithaca, NY 14853 , XP081426011

YAMING WANG; XIAO TAN; YI YANG; ZIYU LI; XIAO LIU; FENG ZHOU; LARRY S. DAVIS: " Improving Annotation for 3D Pose Dataset of Fine-Grained Object Categories ", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, 19 October 2018 (2018-10-19), 201 Olin Library Cornell University Ithaca, NY 14853 , XP081068662

Cited By (7)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20220189170A1

( en )

2019-02-22

2022-06-16

Google Llc

Memory-Guided Video Object Detection

US11961298B2

( en )

2019-02-22

2024-04-16

Google Llc

Memory-guided video object detection

US12333806B2

( en )

2019-02-22

2025-06-17

Google Llc

Memory-guided video object detection

US20230100305A1

( en )

2021-09-27

2023-03-30

Baidu Usa Llc

System and process for repainting of planar objects in video

US12131447B2

( en )

2021-09-27

2024-10-29

Baidu Usa Llc

System and process for repainting of planar objects in video

US20230342368A1

( en )

2022-04-24

2023-10-26

Jingdong City (Beijing) Digits Technology Co., Ltd.

Method and apparatus for spatial data processing

US12547635B2

( en )

2022-04-24

2026-02-10

Jingdong City (Beijing) Digits Technology Co., Ltd.

Method and apparatus for spatial data processing

Also Published As

Publication number

Publication date

CN114041170A

( en )

2022-02-11

WO2020247048A1

( en )

2020-12-10

US20200388071A1

( en )

2020-12-10

Model retrieval for objects in images using field descriptors — Qualcomm Technologies, Inc. (US11361505B2)

Related documents