ABSTRACT
Abstract
Provided is a technique for determining a context of an image and an object depicted by the image based on the context. A trained context classification model may determine a context of an image, and a trained object recognition model may determine an object depicted by the image based on the image and the context. Provided is also a technique for determining an object depicted within an image based on an input location of an input detected by a display screen. An object depicted within an image may be detected based on a distance in feature space between an image feature vector of an image and a feature vector of the object, and a distance in pixel-space between an input location of an input and location of the object within the image.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
This patent claims the benefit of U.S. Provisional Patent Application No. 62/781,446, filed Dec. 18, 2018, titled âMULTI-CHANNEL COMPUTER-VISION OBJECT DETECTION,â and also claims the benefit of U.S. Provisional Patent Application No. 62/781,432, filed Dec. 18, 2018, titled âCONTEXT AIDED VISUAL SEARCH.â The entire contents of each afore-listed earlier-filed applications are hereby incorporated by reference for all purposes.
BACKGROUND
1. Field
The present disclosure relates generally to computer vision and, more specifically, to context aided visual search and multi-channel computer-vision object recognition.
2. Description of the Related Art
A surprising discovery by artificial intelligence researchers is that many types of high-level reasoning require relatively few computational resources, while relatively low-level sensorimotor activities require relatively extensive computational resources. In many cases, the skills of a child are exceedingly difficult to implement with a computer, while the added abilities of an adult are relatively straightforward. A canonical example is that of computer vision, where it is relatively simple for a human to parse visual scenes and extract information, while computers struggle with this task.
Notwithstanding these challenges, computer vision algorithms have improved tremendously in recent years, particularly in the realm of object detection and localization within various types of images, such as two-dimensional images, depth images, stereoscopic images, and various forms of video. Variants include unsupervised and supervised computer vision algorithms, with the latter often drawing upon training sets in which objects in images are labeled. In many cases, trained computer-vision models ingest an image, detect an object from among an ontology of objects in the image, and indicate a bounding area in pixel coordinates of the object along with a confidence score.
SUMMARY
The following is a non-exhaustive listing of some aspects of the present techniques. These and other aspects are described in the following disclosure.
Some aspects include a process including: obtaining, with a computer system, an image depicting an object within a context, wherein: the image is captured by a mobile computing device, the object is a member of an ontology of objects including a plurality of objects, and the context is a member of an ontology of contexts including a plurality of contexts; determining, with the computer system, with a trained context classification model, the context depicted by the image; determining, with the computer system, with a trained object recognition model, a first object identifier of the object based on the image and the context; and causing, with the computer system, the first object identifier of the object to be stored in memory.
Some aspects also include a process including: obtaining, with a computer system, (i) an image captured by a mobile computing device and (ii) coordinates indicating an input location of an input detected on a display screen of the mobile computing device, wherein: the input caused the image to be captured, the input location is a location in pixel-space of the image, and the image depicts a first object located at a first location in the image; obtaining, with the computer system, a computer-vision object recognition model trained using a training data set including images depicting objects, wherein: each image of the training data set is labeled with an object identifier, each object identifier indicates an object in an object ontology depicted by a corresponding image, and the object ontology includes the first object; detecting, with the computer system, with the computer-vision object recognition model, the first object based on: a first distance in a feature space of the computer-vision object recognition model between an image feature vector of the image and a first feature vector of the first object in the computer-vision object recognition model; and a first distance in the pixel-space of the image between the input location of the input and the first location of the first object; and causing, with the computer system, a first object identifier of the first object from the object ontology to be stored in memory.
Some aspects include a tangible, non-transitory, machine-readable medium storing instructions that when executed by a data processing apparatus cause the data processing apparatus to perform operations including each of the above-mentioned processes.
Some aspects include a system, including: one or more processors; and memory storing instructions that when executed by the processors cause the processors to effectuate operations of each the above-mentioned processes.
BRIEF DESCRIPTION OF THE DRAWINGS
The above-mentioned aspects and other aspects of the present techniques will be better understood when the present application is read in view of the following figures in which like numbers indicate similar or identical elements:
FIG. 1 illustrates an example system for recognizing an object depicted within an image, in accordance with various embodiments;
FIG. 2 illustrates an example process for recognizing an object depicted within an image based on a context of the image, in accordance with various embodiments;
FIG. 3 illustrates an example data pipeline for determining an object depicted within a context of an image, in accordance with various embodiments;
FIG. 4 illustrates an example of an image depicting a first object and a second object within a context, in accordance with various embodiments;
FIG. 5 illustrates an example of a kiosk device for capturing an image of an object and facilitating a performance of a visual search based on the image, in accordance with various embodiments;
FIG. 6 illustrates an example of an image depicting objects being captured responsive to an input detected by a mobile computing device, in accordance with various embodiments;
FIG. 7 illustrates an example of determining coordinates of an input location of an input detected by a display screen of a mobile computing device, in accordance with various embodiments;
FIG. 8 illustrates an example of a display screen for a mobile computing device segmented into blocks for determining regions of interest and coordinates of an input location of an input detected by the display screen, as well as objects depicted within an image displayed by the display screen, in accordance with various embodiments;
FIG. 9 illustrates an example of determining distances in pixel-space of an image between an input location of an input and objects depicted by the image, in accordance with various embodiments;
FIG. 10 illustrates an example of a process for determining an object identifier of an object to be used as an input for a visual search based on a detected input on a display screen of a mobile computing device, in accordance with various embodiments; and
FIG. 11 illustrates an example of a computing system by which the present techniques may be implemented, in accordance with various embodiments.
While the present techniques are susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. The drawings may not be to scale. It should be understood, however, that the drawings and detailed description thereto are not intended to limit the present techniques to the particular form disclosed, but to the contrary, the intention is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the present techniques as defined by the appended claims.
DETAILED DESCRIPTION OF CERTAIN EMBODIMENTS
To mitigate the problems described herein, the inventors had to both invent solutions and, in some cases just as importantly, recognize problems overlooked (or not yet foreseen) by others in the field of computer vision. Indeed, the inventors wish to emphasize the difficulty of recognizing those problems that are nascent and will become much more apparent in the future should trends in industry continue as the inventors expect. Further, because multiple problems are addressed, it should be understood that some embodiments are problem-specific, and not all embodiments address every problem with traditional systems described herein or provide every benefit described herein. That said, improvements that solve various permutations of these problems are described below.
Context-Aided Visual Search
Many existing computer-vision object detection and localization approaches require larger training sets than are available for some use cases. Many examples are trained with sets of images including tens of thousands of images of each object the model is capable of detecting. Various approaches have been developed for use cases with smaller sets of training images, while candidate objects in an ontology are relatively large. For example, some training data sets may include less than 100 example images of each object, less than 10 example images of each object, or even a single image of each object, while the number of object in the ontology may be more than 1,000, more than 10,000, more than 100,000, or more than 1,000,000. But many of these techniques (e.g., including smaller sets of training images) are less accurate and more computationally expensive than desirable. Or in some cases, ratios of any permutation of these numbers may characterize a relevant scenario. For example, a ratio of example images per object to objects in an ontology of less than 1/100; 1/1,000; 1/10,000; or 1/100,000. Further, many approaches become particularly computationally expensive as more complex convolutional layers are added to models and as the size of an ontology of objects being detected grows.
Some embodiments augment computer-vision object detection by enriching a feature set by which objects are detected with a classification of a context in which the objects appear in an image. Examples include models that upweight kitchen utensils in response to classifying an image as depicting a scene in a kitchen as the image context, or upweight home improvement equipment in response to classifying an image as depicting a scene in a garage as the image context.
In some embodiments, an object recognition model may be positioned downstream from a context classification model in an image processing pipeline. The context classification model may receive an image and output a context classification vector indicative of a confidence that the image depicts a particular context. Some examples of context include scenes, temporal information, geographical information, input information, and the like. In some embodiments, the context classification model may be a scene classification model that receives an image and outputs a scene classification vector indicative of a confidence that the image depicts a particular scene. This scene classification vector may be input to the object recognition model as an enriched feature set along with the corresponding image itself for which objects are to be detected. Some embodiments may classify scenes according to various exclusive or non-exclusive attributes. For example, exclusive attributes may indicate whether an image depicts a scene that is indoors or outdoors, while non-exclusive attributes may classify the scene as a living room decorated for a holiday. The output from the scene classification model may be passed to an object recognition model along with the image for which the scene was classified.
In some embodiments, the object recognition model is trained to recognize (e.g., classify and locate) objects in an ontology of objects, only a small (e.g., less than 0.1%) subset of which may appear in any given image in some cases. Each object in the ontology may have associated therewith a context vector having dimensions indicative of a correlation of that object with a respective context classification (e.g., in the same taxonomy (or other ontology) as is applied by the scene classification model). In some cases, the object recognition model may be trained on a training data set in which both objects depicted are labeled and scenes are labeled according to the context (e.g., scene) ontology or taxonomy, such that the object recognition model is responsive to both pixel values and context classifications when recognizing objects.
In some cases, the context classification model is separately trained from the object recognition model. For example, the context classification model may be trained in a distinct training operation in which a distinct objective function is optimized that differs from the object function to be optimized during training of the object recognition model. As another example, the context classification model may be trained using a different labeled training set of images than that of the object recognition model. The context classification model and the object recognition model may then be combined further down the pipeline of models for classification. Or in some cases, the object recognition model and the context classification model are jointly optimized.
In some embodiments, the context of an image may be identified using metadata obtained with the image. For example, the obtained metadata may include geographic information, ambient lighting information, directionality information, etc. For instance, the metadata may be obtained from geo-positional sensors, motion sensors, ambient light sensors, and other sensors located on a mobile computing device used to capture an image, in communication with the mobile computing device, or a combination thereof. As an example, global positioning data may be used to identify and confirm whether a background of an image includes snow (e.g., by determining a geographical position of the mobile computing device and determining whether that geographical region experienced snowfall recently). If so, this may indicate that it is likely wintertime, and so a vertical of clothing with a focus on winter coats or winter products may be more relevant form a search perspective. Thus, the vertical of clothing with a focus on winter coats or winter products may be upweighted when performing a visual search for an object detected within the image. In some embodiments, metadata may be generated to reflect the context of the image. For example, if an image is taken in a kitchen, then there is a higher likelihood that the item to be searched for is a kitchen-related item. In some embodiments, labels indicative of the context may be obtained and used to either (1) restrict a search to be narrowed to items only related to the identified context's label, or (2) apply a weight to the search to weigh items related to the identified context's label greater than items not related to the identified context's label. The weighting may be accomplished by adding an additional dimension to the feature space or by attaching a multiplier to the distance of certain items.
In some embodiments, upon receiving an image, objects and regions of interests (ROIs) may be detected from the image. The objects and ROIs may be run through a first set of classifiers to identify characteristics such as, but not limited to, pattern, color, material, category, vertical, logo, etc. The information obtained from the first set of classifiers may be provided with the ROIs to a deep learning model to extract a set of features for a given ROI. The information from the first set of classifiers may be used to supplement the features searched against the database of items. The entire image may be used to classify context and then the class
CROSS-REFERENCE TO RELATED APPLICATIONS
This patent claims the benefit of U.S. Provisional Patent Application No. 62/781,446, filed Dec. 18, 2018, titled âMULTI-CHANNEL COMPUTER-VISION OBJECT DETECTION,â and also claims the benefit of U.S. Provisional Patent Application No. 62/781,432, filed Dec. 18, 2018, titled âCONTEXT AIDED VISUAL SEARCH.â The entire contents of each afore-listed earlier-filed applications are hereby incorporated by reference for all purposes.
BACKGROUND
1. Field
The present disclosure relates generally to computer vision and, more specifically, to context aided visual search and multi-channel computer-vision object recognition.
2. Description of the Related Art
A surprising discovery by artificial intelligence researchers is that many types of high-level reasoning require relatively few computational resources, while relatively low-level sensorimotor activities require relatively extensive computational resources. In many cases, the skills of a child are exceedingly difficult to implement with a computer, while the added abilities of an adult are relatively straightforward. A canonical example is that of computer vision, where it is relatively simple for a human to parse visual scenes and extract information, while computers struggle with this task.
Notwithstanding these challenges, computer vision algorithms have improved tremendously in recent years, particularly in the realm of object detection and localization within various types of images, such as two-dimensional images, depth images, stereoscopic images, and various forms of video. Variants include unsupervised and supervised computer vision algorithms, with the latter often drawing upon training sets in which objects in images are labeled. In many cases, trained computer-vision models ingest an image, detect an object from among an ontology of objects in the image, and indicate a bounding area in pixel coordinates of the object along with a confidence score.
SUMMARY
The following is a non-exhaustive listing of some aspects of the present techniques. These and other aspects are described in the following disclosure.
Some aspects include a process including: obtaining, with a computer system, an image depicting an object within a context, wherein: the image is captured by a mobile computing device, the object is a member of an ontology of objects including a plurality of objects, and the context is a member of an ontology of contexts including a plurality of contexts; determining, with the computer system, with a trained context classification model, the context depicted by the image; determining, with the computer system, with a trained object recognition model, a first object identifier of the object based on the image and the context; and causing, with the computer system, the first object identifier of the object to be stored in memory.
Some aspects also include a process including: obtaining, with a computer system, (i) an image captured by a mobile computing device and (ii) coordinates indicating an input location of an input detected on a display screen of the mobile computing device, wherein: the input caused the image to be captured, the input location is a location in pixel-space of the image, and the image depicts a first object located at a first location in the image; obtaining, with the computer system, a computer-vision object recognition model trained using a training data set including images depicting objects, wherein: each image of the training data set is labeled with an object identifier, each object identifier indicates an object in an object ontology depicted by a corresponding image, and the object ontology includes the first object; detecting, with the computer system, with the computer-vision object recognition model, the first object based on: a first distance in a feature space of the computer-vision object recognition model between an image feature vector of the image and a first feature vector of the first object in the computer-vision object recognition model; and a first distance in the pixel-space of the image between the input location of the input and the first location of the first object; and causing, with the computer system, a first object identifier of the first object from the object ontology to be stored in memory.
Some aspects include a tangible, non-transitory, machine-readable medium storing instructions that when executed by a data processing apparatus cause the data processing apparatus to perform operations including each of the above-mentioned processes.
Some aspects include a system, including: one or more processors; and memory storing instructions that when executed by the processors cause the processors to effectuate operations of each the above-mentioned processes.
BRIEF DESCRIPTION OF THE DRAWINGS
The above-mentioned aspects and other aspects of the present techniques will be better understood when the present application is read in view of the following figures in which like numbers indicate similar or identical elements:
FIG. 1 illustrates an example system for recognizing an object depicted within an image, in accordance with various embodiments;
FIG. 2 illustrates an example process for recognizing an object depicted within an image based on a context of the image, in accordance with various embodiments;
FIG. 3 illustrates an example data pipeline for determining an object depicted within a context of an image, in accordance with various embodiments;
FIG. 4 illustrates an example of an image depicting a first object and a second object within a context, in accordance with various embodiments;
FIG. 5 illustrates an example of a kiosk device for capturing an image of an object and facilitating a performance of a visual search based on the image, in accordance with various embodiments;
FIG. 6 illustrates an example of an image depicting objects being captured responsive to an input detected by a mobile computing device, in accordance with various embodiments;
FIG. 7 illustrates an example of determining coordinates of an input location of an input detected by a display screen of a mobile computing device, in accordance with various embodiments;
FIG. 8 illustrates an example of a display screen for a mobile computing device segmented into blocks for determining regions of interest and coordinates of an input location of an input detected by the display screen, as well as objects depicted within an image displayed by the display screen, in accordance with various embodiments;
FIG. 9 illustrates an example of determining distances in pixel-space of an image between an input location of an input and objects depicted by the image, in accordance with various embodiments;
FIG. 10 illustrates an example of a process for determining an object identifier of an object to be used as an input for a visual search based on a detected input on a display screen of a mobile computing device, in accordance with various embodiments; and
FIG. 11 illustrates an example of a computing system by which the present techniques may be implemented, in accordance with various embodiments.
While the present techniques are susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. The drawings may not be to scale. It should be understood, however, that the drawings and detailed description thereto are not intended to limit the present techniques to the particular form disclosed, but to the contrary, the intention is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the present techniques as defined by the appended claims.
DETAILED DESCRIPTION OF CERTAIN EMBODIMENTS
To mitigate the problems described herein, the inventors had to both invent solutions and, in some cases just as importantly, recognize problems overlooked (or not yet foreseen) by others in the field of computer vision. Indeed, the inventors wish to emphasize the difficulty of recognizing those problems that are nascent and will become much more apparent in the future should trends in industry continue as the inventors expect. Further, because multiple problems are addressed, it should be understood that some embodiments are problem-specific, and not all embodiments address every problem with traditional systems described herein or provide every benefit described herein. That said, improvements that solve various permutations of these problems are described below.
Context-Aided Visual Search
Many existing computer-vision object detection and localization approaches require larger training sets than are available for some use cases. Many examples are trained with sets of images including tens of thousands of images of each object the model is capable of detecting. Various approaches have been developed for use cases with smaller sets of training images, while candidate objects in an ontology are relatively large. For example, some training data sets may include less than 100 example images of each object, less than 10 example images of each object, or even a single image of each object, while the number of object in the ontology may be more than 1,000, more than 10,000, more than 100,000, or more than 1,000,000. But many of these techniques (e.g., including smaller sets of training images) are less accurate and more computationally expensive than desirable. Or in some cases, ratios of any permutation of these numbers may characterize a relevant scenario. For example, a ratio of example images per object to objects in an ontology of less than 1/100; 1/1,000; 1/10,000; or 1/100,000. Further, many approaches become particularly computationally expensive as more complex convolutional layers are added to models and as the size of an ontology of objects being detected grows.
Some embodiments augment computer-vision object detection by enriching a feature set by which objects are detected with a classification of a context in which the objects appear in an image. Examples include models that upweight kitchen utensils in response to classifying an image as depicting a scene in a kitchen as the image context, or upweight home improvement equipment in response to classifying an image as depicting a scene in a garage as the image context.
In some embodiments, an object recognition model may be positioned downstream from a context classification model in an image processing pipeline. The context classification model may receive an image and output a context classification vector indicative of a confidence that the image depicts a particular context. Some examples of context include scenes, temporal information, geographical information, input information, and the like. In some embodiments, the context classification model may be a scene classification model that receives an image and outputs a scene classification vector indicative of a confidence that the image depicts a particular scene. This scene classification vector may be input to the object recognition model as an enriched feature set along with the corresponding image itself for which objects are to be detected. Some embodiments may classify scenes according to various exclusive or non-exclusive attributes. For example, exclusive attributes may indicate whether an image depicts a scene that is indoors or outdoors, while non-exclusive attributes may classify the scene as a living room decorated for a holiday. The output from the scene classification model may be passed to an object recognition model along with the image for which the scene was classified.
In some embodiments, the object recognition model is trained to recognize (e.g., classify and locate) objects in an ontology of objects, only a small (e.g., less than 0.1%) subset of which may appear in any given image in some cases. Each object in the ontology may have associated therewith a context vector having dimensions indicative of a correlation of that object with a respective context classification (e.g., in the same taxonomy (or other ontology) as is applied by the scene classification model). In some cases, the object recognition model may be trained on a training data set in which both objects depicted are labeled and scenes are labeled according to the context (e.g., scene) ontology or taxonomy, such that the object recognition model is responsive to both pixel values and context classifications when recognizing objects.
In some cases, the context classification model is separately trained from the object recognition model. For example, the context classification model may be trained in a distinct training operation in which a distinct objective function is optimized that differs from the object function to be optimized during training of the object recognition model. As another example, the context classification model may be trained using a different labeled training set of images than that of the object recognition model. The context classification model and the object recognition model may then be combined further down the pipeline of models for classification. Or in some cases, the object recognition model and the context classification model are jointly optimized.
In some embodiments, the context of an image may be identified using metadata obtained with the image. For example, the obtained metadata may include geographic information, ambient lighting information, directionality information, etc. For instance, the metadata may be obtained from geo-positional sensors, motion sensors, ambient light sensors, and other sensors located on a mobile computing device used to capture an image, in communication with the mobile computing device, or a combination thereof. As an example, global positioning data may be used to identify and confirm whether a background of an image includes snow (e.g., by determining a geographical position of the mobile computing device and determining whether that geographical region experienced snowfall recently). If so, this may indicate that it is likely wintertime, and so a vertical of clothing with a focus on winter coats or winter products may be more relevant form a search perspective. Thus, the vertical of clothing with a focus on winter coats or winter products may be upweighted when performing a visual search for an object detected within the image. In some embodiments, metadata may be generated to reflect the context of the image. For example, if an image is taken in a kitchen, then there is a higher likelihood that the item to be searched for is a kitchen-related item. In some embodiments, labels indicative of the context may be obtained and used to either (1) restrict a search to be narrowed to items only related to the identified context's label, or (2) apply a weight to the search to weigh items related to the identified context's label greater than items not related to the identified context's label. The weighting may be accomplished by adding an additional dimension to the feature space or by attaching a multiplier to the distance of certain items.
In some embodiments, upon receiving an image, objects and regions of interests (ROIs) may be detected from the image. The objects and ROIs may be run through a first set of classifiers to identify characteristics such as, but not limited to, pattern, color, material, category, vertical, logo, etc. The information obtained from the first set of classifiers may be provided with the ROIs to a deep learning model to extract a set of features for a given ROI. The information from the first set of classifiers may be used to supplement the features searched against the database of items. The entire image may be used to classify context and then the classified context of the image may be used for identifying additional labels associated with each ROI detected in the image. In some embodiments, after using the context to identify labels associated with each ROI, a weighted or narrowing search against an object database of items may be performed to identify an object or objects that match the features of the ROI(s) based on the extracted context.
In some embodiments, the system may loop back and use the identified objects from the ROI to further refine the context information. In other words, after identifying the object(s) from the search and the context of the original image, the model can go back and use the images for refining the classifiers for the explicit categories to better understand the aspects that encompass different contexts.
In some embodiments, a context classification model, such as a scene classification model, may be a unified ontology of retail, where retail can encompass various items, products, or services that are searchable and capable of being obtained (e.g., purchased). In some examples, a scene classification model may be configured to identify (e.g., classify or otherwise determine) a vertical to which an image relates. For example, a scene classification model may identify whether an image is related to a fashion vertical, food/beverage vertical, financial vertical, media vertical, technology vertical, etc., based on the scene classified as being depicted by the image. Based on the identified vertical, different attributes (e.g., scores for dimensions) may be added to a feature vector (e.g., increasing its dimensionality) for an object recognition model, or different extant attributes of the feature vector may be weighted based on the vertical (e.g., by scaling the size of various scalars). For example, if (e.g., in response to determining that) an image is identified as being part of the home improvement vertical, then attributes associated with home improvement products, such as saws, drills, hammers, etc., may be added to a feature vector for classifying objects within the image, or the attributes may be upweighted in the object recognition model. In some embodiments, attributes that are not associated with the vertical may be removed from the feature vector or down-weighted. Continuing the previous example, attributes associated with food products may be removed from a feature vector for classifying objects within the image if the image was identified as being part of the home improvement vertical, or attributes associated with food products may be down-weighted in the object recognition model. As another example, in the home improvement vertical, certain features may be irrelevant or have less importance for classifying an object (e.g., pattern), whereas if the scene classification model identified the vertical as being the fashion vertical, then such features may be more relevant for object classification.
In some embodiments, the object recognition model following the context classification model may recognize (e.g., classify, locate, etc.) objects in an ontology of objects that includes cross-reference information connecting objects of one hierarchy to objects in another hierarchy (e.g., in a knowledge graph). For example, if a scene classification model identifies that an image has a garage background and includes a hammer, the scene classification model may access the knowledge graph, select adjacent nodes to hammer having edges indicating complements, and determine that nails are connected to hammers. Some embodiments may increase a weight in the object recognition model for a feature or features related to nails. In some embodiments, a graph may be built from objects detected using a scene classification model, the object recognition model, or both, as a series of single purpose rules. In some embodiments, the graph may be built by extracting information from previously obtained data. For example, a retailer may provide a product catalog that includes product information such as name, description, category, etc., and some embodiments may extract node identifiers and relationship edges for a knowledge graph based on natural language text in the catalog. The graph may even further be enhanced by obtaining information related to a product available through publicly accessible resources, such as the Internet. For instance, some embodiments may crawl Wikipedia or other websites and determine relationships based on term-frequency inverse document frequency (TF-IDF) (e.g., BM25) scores and co-occurrence rates in webpages of terms relating to the nodes, for example by detecting that hammer and nail tend to co-occur, for instance with relationships determined based on (e.g., for those with less than a threshold) distance determined with latent semantic analysis or word2vec (e.g., cosine distance, Minkowski distance, Euclidian distance, or the like). Or these techniques may be applied to co-occurrence of objects in images (e.g., hammers occurring frequently in images with nails and rarely in images without nails may cause embodiments to add an edge to a knowledge graph between a node representing a hammer and a node representing a nail). The graph therefore may be built to include various available information relating to a particular product. When an image is captured and an object or pair of objects is/are identified therein, a likely context for that image may be determined, and the search space for that object may be narrowed or portions of the search space may be more heavily weighted. As an example, if an image of a slice of pizza is captured, the slice of pizza may be identified using the object recognition model. To recommend products to a user, the context may be used to determine that providing products related to purchasing a slice of pizza are less optimal (e.g., because the user already has a slice of pizza), as opposed to information like a food blog, nutritional information, recipes, etc. Using the contextual information surrounding the captured image, the connected features may be used to curate the decisions, recommendations, and actions provided to a user to enhance user experiences.
Tap-to-Search
Existing computer-vision object recognition and localization approaches often fail to align with user-intent when processing an image. In some cases, this is because an image depicts many objects and there is no clear indication of which one prompted the user to select (e.g., capture) the image. And in some cases, such processes suffer from lower accuracy and are more computationally expensive than desirable. Various phenomena contribute to errors, including, but not limited to, blurring, irrelevant objects in a camera's field of view (FOV), novel camera poses relative to a training data set, suboptimal lighting in run-time images, etc. These issues are compounded by many of the aforementioned issues with respect to training data set sizes being limited while object ontology being large.
To mitigate some of the above-described problems, some embodiments may leverage an additional channel of information beyond the image itself to improve object detection, object recognition, object selection, or any combination thereof. Some embodiments may use input information, such as touch location or eye gaze location, to reduce the search space of an image (or modulate the amount of computational effort expended in different areas of the image) when detecting objects therein or inferring user intent from images with multiple objects. Some embodiments may determine whether (and in some cases where) objects appears in an image based on a location of a detected input from a user on a displayed user interface, like a touch screen, an eye gaze sensor, or a pointer in an augmented reality or virtual reality interface. Additionally, or alternatively, some embodiments may select an object among such objects depicted within an image based on a location of the detected input. This is expected to accommodate use of less robust object recognition models based on smaller training sets than versions of such models trained on larger training sets, as the set of inputs may be constrained by the added channel, and in some cases, the search space may be reduced by narrowing which pixels in an image are pertinent to a search. For instance, noise from objects at the periphery of a touch location may be discarded based on the touch location.
In some embodiments, a user may open a search application on their user device to search for information related to an item, like an object that may appear in an image. The application (like a native application on a mobile computing device) may cause a camera on the mobile computing device to sense an image in a field of view (FOV) of the camera (or set of cameras in systems implementing computational photography) and display the image (e.g., a single image or video) sensed by the camera in a user interface (UI) of the application.
In some embodiments, a user may tap (or otherwise select pixel coordinates) on the display screen that displays the UI to cause an image representing the displayed contents to be captured. In some cases, the input may generate a UI event, like an on-touch event, a touch-release event, a draw-circle event, or the like, and the event may include various parameters, like pixel coordinates. In some cases, a coordinate location of where the input was detected on the display screen may also be obtained. Some embodiments may execute an event handler that receives this event and causes responsive actions, like storing the image displayed while the event was received in memory and associating the image with the event in memory.
In some embodiments, the captured image (or set of images, such as those in a video preceding or following (or both) a frame in which the event occurred) and the coordinate location (or other parameters of the UI event) may be provided to a computer-vision object recognition system (which in some cases, may be an object detection and localization system). The object recognition system, which may include or use an object recognition model, may output a score for an object in an ontology of objects indicative of a confidence level that the object was recognized (e.g., the object was detected in the image, the object was selected, or both, where selection indicates that the selected object accords with the user's intent). Some embodiments may output such scores for each of a plurality of objects in an object ontology (e.g., in an object detection vector) and, in some cases, bounding polygons (with vertices expressed in pixel coordinates) of each object. For example, a feature vector may be generated from an input image, where dimensions correspond to features (like edges, blobs, corners, colors, and the like) in the input image. The feature vector may be input into a discriminative computer vision object recognition model, which may match the feature vector to a closest feature vector of an object in a labeled training set of images. Some embodiments may select an object having a highest score based on such a distance (e.g., upon determining the distance is greater than a threshold) as the object in the image. In some embodiments, the score may be used to select an object to be searched for (e.g., against a product catalog or object database). In some embodiments, data associated with the detected event may be used as training data for training an object recognition model to perform object recognition.
In some embodiments, multiple inputs may be detected. For example, two taps on a display screen at two different locations may indicate two different searches to be performed. As another example, a multiple-tap gesture (or pinch gesture or circle event) may indicate a boundary polygon (e.g., box) of an item or items to be searched. As still another example, a front-facing camera may be employed to detect eye gaze, which may also be used to determine an object of interest of the user. In some cases, a user may place an object, like their hand, in a field of view of the camera capturing the image in which objects are to be detected, and the user-placed object may indicate pixel coordinates. For example, by sticking out one's arm and pointing at an object within the field of view of the camera, the hand may appear in the image being processed. In some embodiments, human pose estimation may also be used to determine an object of interest of a user. For example, the OpenPose architecture may be used to identify, from an image, different portions of a human body (e.g., a hand, a face, a leg, etc.), and a determination may be made as to whether the identified portions are directed towards an object depicted within the image.
The image displayed on the UI may include an object or objects with which the user would like to search for and obtain information. In some embodiments, the UI may not include an explicit image capture button. In other words, the entire UI may display the contents viewed by the camera, and no capture image button physically or virtually may be available. In some embodiments, multiple objects may be recognized as being present within the input image, and the coordinate location of the user input (e.g., tap input) may be used to select one of the objects as being reflective of the user's intent based on the location of the user input in pixel coordinates (e.g., selecting the object having a centroid with a closest location to the touch location in pixel coordinates. In some embodiments, a distance from an input's coordinate location with respect to one or more bounding boxes may serve as an additional input for determining an object of interest for the user.
In some embodiments, a visual search system may obtain the image and coordinate location information. The visual search system may extract features from the image and determine, based on locations of objects depicted by the image and the coordinate location information, a likely object (or other type of object) of interest for which the user is searching. Each object detected within the image may be reverse weighted with respect to the coordinate location information, and the object that is most proximate to the coordinate location information may be selected as the likely object of interest. For example, scores of an object recognition model may be adjusted based on the reverse weighting such that a first object for which the model indicates a lower confidence (indicated by a lower) score is selected over a second object with a higher confidence based on the first object being depicted in the image closer to the touch location than the second object (e.g., by multiplying the score by the reverse weighting).
Information about the object may be retrieved by the visual search system and may be provided to the computing device with which the input was detected. For instance, embodiments may access an index keyed to object identifiers (e.g., stock keeping units (SKUs)), and may retrieve and present records related to the object, including a URL of a merchant's website at which the object can be purchased, descriptions of products corresponding to the object, related objects, reviews, and the like. Identifiers of detected objects may be used to a variety of ends, including input to queries in search applications (like product recommendation application), selection of items in augmented reality interfaces, and selections of items for language translation applications that return a name of objects in an image.
In some embodiments, tap point information (or coordinates of other forms of user input) may be used to enhance or selectively process an image prior to being provided to a server. For instance, enhancement may be performed on-device (e.g., on a computing device) to a portion of an image centralized around the tap point. Such enhancements may include light balance enhancement and shadow removal (e.g., embodiments may transform an image in a raw file format (having a relatively wide color gamut) into a file format in a positive file format (having a narrower color gamut), and tradeoffs in white balance, intensity, and other pixel values may be made to favor areas of an image near (e.g., within a threshold distance of, like less than 10%, less than 20%, or less than 50% of an images width in pixels) a touch location. Additionally, patterns and colors may be detected within a region of the image where the tap point is located, which may be used to select an object from an object ontology. In some embodiments, the enhancement or other form of processing may be performed additionally or alternatively by server-side operations of a search system. This may balance the tradeoff between reducing the processing time associated with server side image processing and latency issues associated with transmitting high-quality images to the server.
In some embodiments, the coordinate location of a tap point may be used to compress portions of an image non-proximate to the coordinate location. For example, an image may be segmented into portions (e.g., blocks), and portions further than a threshold distance (like those examples above) from the tap point may be compressed with a first amount of loss, while the portions proximate the tap point may be compressed with a second (e.g., smaller or zero) amount of loss. In some embodiments, the quantization matrices for the compression algorithm may be changed for different portions of the image, which may address file size issues that can increase latency.
For example, in some embodiments, images may be adjusted within the image to reduce bandwidth and memory usage relatively to implementations that naively upload all captured images. Some embodiments may crop images to exclude portions not depicting the object adjacent a tap point or further than a threshold distance from a tap point or other coordinate of user input on a screen (e.g., outside a bounding box of the object in question). Some embodiments may modulate image compression within a frame based on the location of an object. For instance, some embodiments may compress images with a lossy compression algorithm (like JPEG, etc.), and the amount of data loss within different areas of the frame may be modulated such that areas of an image within a threshold distance of a tap point or detected object corresponding thereto (like where the object is detected based on a tap point) have less loss of data (and thus lower compression ratios) while areas not depicting the object or further from the tap point have greater loss of data (and thus higher compression ratios). For example, some embodiments may select different quantization matrices for different areas (e.g., square blocks in a tiling) of an image to modulate data loss/compression (e.g., before dividing a discrete cosine transform coefficient matrix by the quantization matrix in an element-by-element division and rounding to the nearest integer), such that larger values in the quantization matrix tend to result in zero values in the result, and more prevalent use of larger values corresponding to higher frequency components tend to increase compression ratios, thereby increasing efficiency of subsequent run-length encoding of the result. By conserving processing capabilities to only a portion of the image surrounding the tap point and not the entire image, the local processing power may be conserved for only the portion of the image identified as being of interest.
In some embodiments, aspects of a visual search may be executed on client-side (e.g., on a mobile computing device), such as the bounding box proposal aspect of an object detector. For instance, the visual search system may detect the presence of an object within a bounding box, but may not be able to identify the object. In some embodiments, after placing the bounding box around a detected object, computational enhancement techniques may be applied to improve the quality of the portion of the image including the bounding box (e.g., contours, color schemes). Furthermore, the distance from the tap point's coordinate location to a centroid of the bounding box may be computed, and the bounding box may be extended.
As mentioned above, a compressed image may initially be sent to a server for initial processing. In some embodiments, one or more additional images may be obtained in the background, either spatially or temporally, and these images may subsequently be provided to the server as part of the same image processing job as that of the initially provided (compressed) image. By doing so, different objects, backgrounds, contexts, and visualization aspects (e.g., lighting, angle, etc.) may be analyzed in parallel processing with the initially sent image. As an example, an initial image sent to the server may be analyzed to identify a candidate object or objects depicted within a portion of the image surrounding a tap point (a term used generally herein and which should be read as also disclosing coordinates on a display or image corresponding to the other types of user input mentioned). As the initial image is processed, one or more background images may also be captured and provided to the server, and information associated with the background may be used to refine and clarify the candidate item or items that possible were detected in portion of the image surrounding the tap point (e.g., scene classification information indicating a scene within the image).
On the client-side (e.g., on a mobile computing device), object tracking associated with a coordinate location of a tap point may be used to obtain depth information associated with an object. In some embodiments, an object depicted within an image displayed by the UI may be tapped, and in response to the input, a search anchor may be attached to the object in a 3D virtual space. Subsequent movements of the mobile computing device, the object, or both, may be detected and depth information indicated the relative distance from the object to the mobile computing device may be determined on the mobile computing device to provide a relatively low-latency augmented reality display based on the depth information.
Furthermore, in some embodiments a scene graph may be generated to connect contextual information and a detected object to resolve an identity of the object. For example, if an object detected in a background of an image is a stove or microwave, and the object detected proximate to the tap point is identified as possibly being a fork, then this may indicate that the contextual information indicates that a user is likely in a kitchen. Similarly, if a hammer is detected in a same image as other home improvement images (e.g., a drill, saw, etc.) then this may indicate an increased likelihood that the user is currently in a garage. The scene graph may encompass this information to describe the scene with which the image is captured, which may further help in object identification, product recommendation, and contextual analysis.
FIG. 1 illustrates an example system for recognizing an object depicted within an image, in accordance with various embodiments. In some embodiments, system 100 may include a computer system 102 , a mobile computing device 104 , a kiosk device 106 , databases 130 , and other components, each of which may be configured to communicate with one another via one or more networks 150 (e.g., the Internet, an Intranet, etc.). In some embodiments, computer system 102 may be a mobile computing device, which is the same or similar to mobile computing device 104 , however computer system 102 may also refer to a server-side system that receives data from one or more devices (e.g., mobile computing device 104 , kiosk device 106 ), and outputs data to the devices. Although a single instance of kiosk device 106 and mobile computing device 104 is included by system 100 , additional instances of kiosk device 106 and mobile computing device 104 may be present, and are not illustrate so as to simplify the depicted system components. Further still, while only a single computer system 102 is illustrated, the functionality of computer system 102 may be replicated on multiple instances of computer system 102 , shared across such instances of mobile computer system 102 , distributed across multiple instances of computer system 102 , and the like.
In some embodiments, computer system 102 may include a context classification subsystem 112 , an object recognition subsystem 114 , a model subsystem 116 , a visual search subsystem 118 , an input determination subsystem 120 , a distance determination subsystem 122 , and other components. As mentioned above, some or all of the aforementioned subsystems (e.g., subsystems 112 - 122 ) may be offloaded to a mobile computing device (e.g., mobile computing device 104 ), computer system 102 may be a mobile computing device, or both. Therefore, while the functionalities of each subsystem may be described in the context of being performed client-side or server-side, the functionalities of these subsystems are not restricted to be performed only client-side or only server-side.
In some embodiments, context classification subsystem 112 may be configured to classify a context of an image based on a context classification model. As described herein, a context of an image may include, but is not limited to, a scene depicted by an image, geographical information regarding where an image was captured (e.g., from one or more location sensors resident on a device used to capture the image), temporal information indicating a time that an image was captured, input information regarding inputs detected by a device used to capture an image, user information related to a user operating a device used to capture the image, and so on. In some embodiments, scene classification may refer to a process whereby objects depicted by an image, the layout of those objects within the image, and ambient contextual information, are used to determine a scene of an image. A âscene,â as defined herein, may refer to a view of a real-world environment that includes multiple surfaces and objects, which may be organized in a meaningful way. A scene may represent one type of context, and may refer to a physical place (e.g., a geographical location, such as a landmark, address, point of interest, etc.), a type of place (e.g., a home, a school, an office, etc.), a sub-type of place (e.g., a bedroom within a home, a garage of a home, a classroom within a school, etc.), background information (e.g., trees, snow, bodies of water), or any other information, or any combination thereof. For example, context classification subsystem 112 may be configured to determine, based on an input image and a scene classification model, that the image depicts a snow-covered field.
In some embodiments, context classification subsystem 112 may be configured to classify an image as including, describing, being related to, or a combination thereof, a context or contexts of a context ontology including a plurality of contexts. For example, a context ontology may include a scene ontology, a geographical ontology, a season ontology, and so on. In some embodiments, a scene classification model may refer to a genre of context classification models configured to classify an image as depicting a scene from a scene ontology including a plurality of scenes, each scene from the scene ontology being labeled with a scene identifier of the scene. As described below, with respect to model subsystem 116 and training subsystem 120 , a context classification model may be trained to classify an input image as depicting a context from the context ontology using a training data set including a plurality of images including different contexts. In some embodiments, context database 132 may include different contexts with which an image may be classified. For example, context database 132 may include different scenes that an image may be classified as depicting.
In some embodiments, context classification subsystem 112 may obtain a trained context classification model, such as a trained scene classification model, from model database 138 . The scene classification model may be a convolutional neural network (CNN) including a plurality of layers (e.g., 4 or more layers, 5 or more layers, 6 or more layers, 8 or more layers, etc.), which may form a portion of a deep neural network for classifying, or recognizing, a scene. In some embodiments, the CNN may include, in addition to the plurality of convolutional layers, a number of batch normalization layers, a number of ReLU layers, a number of max-pooling layers, one or more fully-connected layers, and one or more Softmax layers. One example of a CNN used to perform automatic scene classification is AlexNet. The AlexNet architecture includes five convolutional layers and three fully connected layers, and a Softmax layer following the last fully connected layer to output a classification distribution, with a ReLU non-linearity applied to the output of every convolutional layer and every fully connected layer. However, the scene classification model used by context classification subsystem 112 may include a same, fewer, or more convolutional layers and fully connected layers. In some embodiments, the kernels may be grouped together as residual blocks, and the kernels may be 1Ã1, 3Ã3, 5Ã5, or other sizes.
In some embodiments, context classification subsystem 112 may output a classification vector including weights representative of the contexts determined for a given input image based on the context classification model. In some embodiments, the classification vector may upweight (e.g., if all weights are initialized to zero) features of the classification vector that were determined to be represented by the image. For example, an image depicting a snow-covered field may have a vector element associated with winter scenes upweighted (e.g., to a non-zero positive value), whereas a vector element associated with a desert or a beach may remain at its initialized value (e.g., zero). In some embodiments, additional features may be added to a feature vector for object recognition based on the contexts identified by context classification subsystem 112 .
In some embodiments, context classification subsystem 112 may receive an image and output a context classification vector indicative of a confidence that the image depicts a particular context. For example, an image captured by an image capture component (e.g., a camera) of computer system 102 , mobile computing device 104 , or kiosk device 106 may be provided to context classification subsystem 112 . Upon receiving the image, context classification subsystem 112 may retrieve a context classification model from model database 138 (e.g., a scene classification model), input the image to the context classification model, and obtain an output from the context classification model of a context classification vector. For instance, context classification subsystem 112 may use a scene classification model to output a context classification vector indicative of a confidence that the image depicts a particular scene. This context classification vector, or a portion of that vector associated with the scene (e.g., a scene classification vector), may be input to an object recognition model as an enriched feature set along with the corresponding image itself for which objects are to be detected.
In some embodiments, context classification subsystem 112 may be configured to receive object identifiers of objects depicted within an image by object recognition subsystem 114 , which may serve, with the classified context vector, as inputs to the context classification model to determine whether the identified context should be updated. For example, a scene classification model may determine a scene of an image, which may be provided to object recognition subsystem 114 . Based on the scene and the image, one or more objects may be recognized within the image. The image, the identified objects, and the initial scene may then be provided back to context classification subsystem</figure-call
CLAIMS
Claims ( 27 )
What is claimed is:
1. A tangible, non-transitory, computer-readable medium storing computer program instructions that when executed by one or more processors effectuate operations comprising:
obtaining, with a computer system, an image depicting an object within a context, wherein:
the context is a scene depicted by the image,
the image is captured by a mobile computing device,
the object is a member of an ontology of objects comprising a plurality of objects, and
the context is a member of an ontology of contexts comprising a plurality of contexts;
determining, with the computer system, with a trained context classification model, the context depicted by the image, wherein:
the trained context classification model comprises a trained scene classification model by which the scene is classified,
the trained scene classification model outputs a scene classification for the image, and
the determined scene comprises background information that is different from the object;
providing, with the computer system, the scene classification and the image to a trained object detection model as inputs to the trained object detection model;
determining, with the computer system, with the trained object detection model, in response to the trained object detection model receiving the scene classification and the image, a first object identifier of the object based on the image and the context scene classification; and
causing, with the computer system, the first object identifier of the object to be stored in memory.
2. The tangible, non-transitory, computer-readable medium of claim 1 , wherein the operations further comprise:
determining, with the computer system, with the trained context classification model, a confidence level associated with the context;
obtaining, with the computer system, the object identifier of the object, the context of image including the scene classification of the image, the confidence level associated with the context, and the image;
providing, with the computer system, the object identifier of the object, the context of the image, the confidence level associated with the context, and the image to the trained context classification model; and
determining, with the computer system, with the trained context classification model, and in response to determining the confidence level is less than a threshold confidence level, an updated context of the image based on the context of the image, the object identifier of the object, and the image, wherein:
the updated context differs from the context, and
the updated context comprises an updated scene classification of the scene depicted by the image.
3. The tangible, non-transitory, computer-readable medium of claim 1 , wherein the operations further comprise:
determining, with the computer system, with the trained object detection model, in response to the trained object detection model receiving the scene classification and the image, a second object identifier of an additional object depicted by the image based on the first object identifier of the object and the scene classification; and
causing, with the computer system, the second object identifier of the additional object to be stored in memory, wherein the first object identifier of the object and the second object identifier of the additional object are stored in the memory in association with the image, a feature vector representing one or more visual features extracted from the image, or the image and the feature vector.
4. The tangible, non-transitory, computer-readable medium of claim 1 , wherein:
the trained object detection model comprises an autoencoder with a continuous vector space representation; and
the trained scene classification model comprises a deep neural network.
5. The tangible, non-transitory, computer-readable medium of claim 4 , wherein the operations further comprise:
causing, with the computer system, an object detection model to be trained to obtain the trained object detection model, wherein:
the objected detection model is trained using a training data set comprising a plurality of images,
each of the plurality of images depicts an object from a subset of objects of the ontology of objects, and
for a given object of the subset of objects, the plurality of images includes three or fewer images depicting the given object.
6. The tangible, non-transitory, computer-readable medium of claim 1 , wherein determining the first object identifier of the object based on the image and the scene classification comprises:
receiving, with the computer system, a feature vector representing features extracted from the image, wherein:
the object identifier of the object is determined, with the trained object detection model, based on the feature vector and the scene classification.
7. The tangible, non-transitory, computer-readable medium of claim 1 , wherein the first object identifier of the object is stored in the memory in association with at least one of: the image, a feature vector representing features extracted from the image, or a scene identifier of the scene classification.
8. The tangible, non-transitory, computer-readable medium of claim 1 , wherein the operations further comprise:
identifying, with the computer system, a region of interest of the object within the image;
cropping, with the computer system, the image to include only the region of interest; and
determining, with the computer system, based on a cropped version of the image comprising the region of interest, a confidence level for the first object identifier of the object, wherein the confidence level indicates how confident the trained object detection model is that the first object identifier represents a label of the object.
9. The tangible, non-transitory, computer-readable medium of claim 8 , wherein the operations further comprise:
determining, with the computer system, that the confidence level is less than a threshold confidence level, indicating that the first object identifier is not able to be the label for the object;
determining, with the computer system, with the trained object detection model, a second object identifier of the object based on the cropped version of the image and the scene classification; and
causing, with the computer system, in response to determining that a confidence level for the second object identifier of the object is greater than or equal to the threshold confidence level, the second object identifier of the object to be stored in the memory in place of the first object identifier.
10. The tangible, non-transitory, computer-readable medium of claim 1 , wherein the operations further comprise:
generating, with the computer system, a training data set for training a computer vision search system to detect the object within a query image, wherein the training data set is generated based on at least one of: the first object identifier of the object, one or more features extracted from the image, or a context label of the context of the image, wherein the context label indicates the scene classification of the scene; and
causing, with the computer system, the computer vision search system to be trained based on the training data to form a trained computer vision search system.
11. The tangible, non-transitory, computer-readable medium of claim 10 , wherein the operations further comprise:
obtaining, with the computer system, a new image; and
determining, with the computer system, with the trained computer vision search system, that the new image depicts the object.
12. The tangible, non-transitory, computer-readable medium of claim 11 , wherein the computer system comprises a kiosk configured to capture the new image and provide information regarding the object for display via the kiosk.
13. A method, comprising:
obtaining, with a computer system, an image depicting an object within a context, wherein:
the context is a scene depicted by the image,
the image is captured by a mobile computing device,
the object is a member of an ontology of objects comprising a plurality of objects, and
the context is a member of an ontology of contexts comprising a plurality of contexts;
determining, with the computer system, with a trained context classification model, the context depicted by the image, wherein:
the trained context classification model comprises a trained scene classification model by which the scene is classified,
the trained scene classification model outputs a scene classification, and
the determined scene comprises background information that is different from the object;
providing, with the computer system, the scene classification and the image to a trained object detection model as input to the trained object detection model;
determining, with the computer system, with the trained object detection model, in response to the trained object detection model receiving the scene classification vector and the image, a first object identifier of the object based on the image and the scene classification; and
causing, with the computer system, the first object identifier of the object to be stored in memory.
14. The tangible, non-transitory, computer-readable medium of claim 1 , wherein providing the image to the trained object detection model is performed by cropping the image and providing a result of the cropping to the trained object detection model.
15. A tangible, non-transitory, computer-readable medium storing computer program instructions that when executed by one or more processors effectuate operations comprising:
obtaining, with a computer system, an image depicting an object within a scene, wherein:
the scene comprises background information that is different from the object,
the image is captured by a mobile computing device,
the object is a member of an ontology of objects comprising a plurality of objects, and
the scene is a member of an ontology of scenes comprising a plurality of scenes;
determining, with the computer system, with a trained scene classification model, a scene classification of the scene depicted by the image, wherein the trained scene classification model outputs the scene classification;
providing, with the computer system, the scene classification and at least one of the image or an adjusted version of the image to a trained object detection model;
determining, with the computer system, with the trained object detection model, in response to the trained object detection model receiving the scene classification and the at least one of the image or the adjusted version of the image, a first object identifier of the object based on the scene classification and the at least one of the image or the adjusted version of the image; and
causing, with the computer system, the first object identifier of the object to be stored in memory.
16. A method, comprising:
obtaining, with a computer system, an image depicting an object within a scene, wherein:
the scene comprises background information that is different from the object,
the image is captured by a mobile computing device,
the object is a member of an ontology of objects comprising a plurality of objects, and
the scene is a member of an ontology of scenes comprising a plurality of scenes;
determining, with the computer system, with a trained scene classification model, a scene classification of the scene depicted by the image, wherein the trained scene classification model outputs the scene classification;
providing, with the computer system, the scene classification and at least one of the image or an adjusted version of the image to a trained object detection model;
determining, with the computer system, with the trained object detection model, in response to the trained object detection model receiving the scene classification and the at least one of the image or the adjusted version of the image, a first object identifier of the object based on the scene classification and the at least one of the image or the adjusted version of the image; and
causing, with the computer system, the first object identifier of the object to be stored in memory.
17. The method of claim 16 , further comprising:
determining, with the computer system, with the trained scene classification model, a confidence level associated with the scene classification;
obtaining, with the computer system, the object identifier of the object, the scene classification, the confidence level associated with the scene classification, and the image;
providing, with the computer system, subsequent to determining the first object identifier of the object, the first object identifier of the object, the scene classification, the confidence level associated with the scene classification, and the image, to the trained scene classification model; and
determining, with the computer system, with the trained scene classification model, and in response to determining the confidence level is less than a threshold confidence level, an updated scene classification based on the first object identifier of the object, the scene classification, and the image, wherein:
the updated scene classification differs from the scene classification, and
the updated scene classification indicates an updated scene with which the image depicts.
18. The method of claim 16 , comprising:
steps for detecting objects based on scene classifications.
19. The method of claim 16 , wherein:
the trained object detection model comprises an autoencoder configured to map images to locations in a continuous vector space representation of the images, the continuous vector space having fewer dimensions than the images; and
the trained scene classification model comprises a deep neural network.
20. The method of claim 19 , further comprising:
causing, with the computer system, an object detection model to be trained to obtain the trained object detection model, wherein:
the objected detection model is trained using a training data set comprising a plurality of images,
each of the plurality of images depicts an object from a subset of objects of the ontology of objects, and
for a given object of the subset of objects, the plurality of images includes three or fewer images depicting the given object.
21. The method of claim 16 , wherein determining the first object identifier of the object based on the scene classification and the at least one of the image or the adjusted version of the image comprises:
receiving, with the computer system, a feature vector representing features extracted from the at least one of the image or the adjusted version of the image, wherein:
the first object identifier of the object is determined, with the trained object detection model, based on the feature vector and the scene classification.
22. The method of claim 16 , comprising:
steps for classifying a scene in an image.
23. The method of claim 16 , further comprising:
identifying, with the computer system, a bounding region of the object within the image;
obtaining, with the computer system, a cropped version of the image by cropping the image to exclude at least some areas outside the bounding region; and
determining, with the computer system, based on the cropped version of the image comprising the region of interest, a confidence level for the first object identifier of the object, wherein the confidence level indicates how confident the trained object detection model is that the first object identifier represents a label of the object.
24. The method of claim 23 , further comprising:
determining, with the computer system, that the confidence level is less than a threshold confidence level, indicating that the first object identifier is not able to be the label for the object;
determining, with the computer system, with the trained object detection model, a second object identifier of the object based on the cropped version of the image and the scene classification; and
causing, with the computer system, in response to determining that a confidence level for the second object identifier of the object is greater than or equal to the threshold confidence level, the second object identifier of the object to be stored in the memory in place of the first object identifier.
25. The method of claim 16 , further comprising:
generating, with the computer system, a training data set for training a computer vision search system to detect the object within a query image, wherein the training data set is generated based on at least one of: the first object identifier of the object, one or more features extracted from the at least one of the image or the adjusted version of the image, or the scene classification of the scene; and
causing, with the computer system, the computer vision search system to be trained based on the training data set to form a trained computer vision search system.
26. The method of claim 25 , further comprising:
obtaining, with the computer system, a new image; and
determining, with the computer system, with the trained computer vision search system, that the new image depicts the object.
27. The method of claim 26 , wherein the computer system comprises a kiosk configured to capture the new image and provide information regarding the object for display via the kiosk.
US16/719,710
2018-12-18
2019-12-18
Scene and user-input context aided visual search
Active
US10755128B2
( en )
Priority Applications (2)
Application Number
Priority Date
Filing Date
Title
US16/719,710
US10755128B2
( en )
2018-12-18
2019-12-18
Scene and user-input context aided visual search
US16/995,531
US11756291B2
( en )
2018-12-18
2020-08-17
Scene and user-input context aided visual search
Applications Claiming Priority (3)
Application Number
Priority Date
Filing Date
Title
US201862781432P
2018-12-18
2018-12-18
US201862781446P
2018-12-18
2018-12-18
US16/719,710
US10755128B2
( en )
2018-12-18
2019-12-18
Scene and user-input context aided visual search
Related Parent Applications (2)
Application Number
Title
Priority Date
Filing Date
US16/719,716
Continuation-In-Part
US10977520B2
( en )
2018-12-18
2019-12-18
Training data collection for computer vision
US16/719,697
Continuation-In-Part
US20200193552A1
( en )
2018-12-18
2019-12-18
Sparse learning for computer vision
Related Child Applications (2)
Application Number
Title
Priority Date
Filing Date
US16/719,697
Continuation-In-Part
US20200193552A1
( en )
2018-12-18
2019-12-18
Sparse learning for computer vision
US16/995,531
Continuation-In-Part
US11756291B2
( en )
2018-12-18
2020-08-17
Scene and user-input context aided visual search
Publications (2)
Publication Number
Publication Date
US20200193206A1
US20200193206A1 ( en )
2020-06-18
US10755128B2
true
US10755128B2 ( en )
2020-08-25
Family
ID=71072700
Family Applications (1)
Application Number
Title
Priority Date
Filing Date
US16/719,710
Active
US10755128B2
( en )
2018-12-18
2019-12-18
Scene and user-input context aided visual search
Country Status (1)
Country
Link
US
( 1 )
US10755128B2
( en )
Cited By (6)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
CN113255628A
( en )
*
2021-07-15
2021-08-13
æé½ç´¢è´æ°ç ç§æè¡ä»½æéå ¬å¸
Scene identification recognition method for news scene
US20220277166A1
( en )
*
2021-02-26
2022-09-01
Changqing ZOU
Methods and systems for rendering virtual objects in user-defined spatial boundary in extended reality environment
US20220292827A1
( en )
*
2021-03-09
2022-09-15
The Research Foundation For The State University Of New York
Interactive video surveillance as an edge service using unsupervised feature queries
US12002290B2
( en )
2022-02-25
2024-06-04
Eyetech Digital Systems, Inc.
Systems and methods for hybrid edge/cloud processing of eye-tracking image data
US20250078463A1
( en )
*
2023-09-05
2025-03-06
The Boeing Company
Region-based object detection with contextual information
US12340580B2
( en )
2021-11-08
2025-06-24
Samsung Electronics Co., Ltd.
Method and electronic device for recognizing image context
Families Citing this family (82)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US11743071B2
( en )
2018-05-02
2023-08-29
View, Inc.
Sensing and communications unit for optically switchable window systems
WO2017089910A1
( en )
*
2015-11-27
2017-06-01
Nz Technologies Inc.
Method and system for interacting with medical information
US10157332B1
( en )
*
2016-06-06
2018-12-18
A9.Com, Inc.
Neural network-based image manipulation
US10353950B2
( en )
2016-06-28
2019-07-16
Google Llc
Visual recognition using user tap locations
US12603091B2
( en )
2017-04-26
2026-04-14
View Operating Corporation
Immersive collaboration of remote participants via media displays
US12034015B2
( en )
2018-05-25
2024-07-09
Meta Platforms Technologies, Llc
Programmable pixel array
US11288733B2
( en )
*
2018-11-14
2022-03-29
Mastercard International Incorporated
Interactive 3D image projection systems and methods
US11888002B2
( en )
2018-12-17
2024-01-30
Meta Platforms Technologies, Llc
Dynamically programmable image sensor
US11962928B2
( en )
2018-12-17
2024-04-16
Meta Platforms Technologies, Llc
Programmable pixel array
JP7164008B2
( en )
*
2019-03-13
2022-11-01
æ¥æ¬é»æ°æ ªå¼ä¼ç¤¾
Data generation method, data generation device and program
JP6651189B1
( en )
2019-03-29
2020-02-19
æ ªå¼ä¼ç¤¾ æ å ±ã·ã¹ãã ã¨ã³ã¸ãã¢ãªã³ã°
Data structure, learning method and information providing system for machine learning
US11206433B2
( en )
*
2019-05-08
2021-12-21
Verizon Media Inc.
Generating augmented videos
US11972466B2
( en )
*
2019-05-20
2024-04-30
Adobe Inc
Computer storage media, method, and system for exploring and recommending matching products across categories
EP3977557A1
( en )
2019-05-31
2022-04-06
View, Inc.
Building antenna
US12108141B2
( en )
2019-08-05
2024-10-01
Meta Platforms Technologies, Llc
Dynamically programmable image sensor
US10885099B1
( en )
*
2019-08-07
2021-01-05
Capital One Services, Llc
Systems and methods for presenting image classification results
US11769180B2
( en )
2019-10-15
2023-09-26
Orchard Technologies, Inc.
Machine learning systems and methods for determining home value
KR102374211B1
( en )
*
2019-10-28
2022-03-15
주ìíì¬ ìì¤ì¤ìì¤ë©
Object recognition method and object recognition device performing the same
US11935291B2
( en )
2019-10-30
2024-03-19
Meta Platforms Technologies, Llc
Distributed sensor system
US11948089B2
( en )
2019-11-07
2024-04-02
Meta Platforms Technologies, Llc
Sparse image sensing and processing
US11386649B2
( en )
2019-11-15
2022-07-12
Maxar Intelligence Inc.
Automated concrete/asphalt detection based on sensor time delay
US11250260B2
( en )
*
2019-11-15
2022-02-15
Maxar Intelligence Inc.
Automated process for dynamic material classification in remotely sensed imagery
CN112887631B
( en )
*
2019-11-29
2022-08-12
å京åèè·³å¨ç½ç»ææ¯æéå ¬å¸
Method and device for displaying object in video, electronic equipment and computer-readable storage medium
US11922252B1
( en )
*
2020-01-14
2024-03-05
Oceanit Laboratories, Inc.
Metamaterials technology for ore tracking
US11645505B2
( en )
*
2020-01-17
2023-05-09
Servicenow Canada Inc.
Method and system for generating a vector representation of an image
JP6846765B1
( en )
*
2020-03-26
2021-03-24
æ ªå¼ä¼ç¤¾ æ å ±ã·ã¹ãã ã¨ã³ã¸ãã¢ãªã³ã°
Information processing program
US11921917B2
( en )
2020-04-07
2024-03-05
Eyetech Digital Systems, Inc.
Compact eye-tracking camera systems and methods
US10996753B1
( en )
2020-04-07
2021-05-04
Eyetech Digital Systems, Inc.
Multi-mode eye-tracking with independently operable illuminators
US11429582B2
( en )
*
2020-04-09
2022-08-30
Capital One Services, Llc
Techniques for creating and utilizing multidimensional embedding spaces
JP7494001B2
( en )
*
2020-05-14
2024-06-03
æ ªå¼ä¼ç¤¾æ¥ç«è£½ä½æ
Training data set creation support device and training data set creation support method
US11825228B2
( en )
2020-05-20
2023-11-21
Meta Platforms Technologies, Llc
Programmable pixel array having multiple power domains
EP4158572A2
( en )
*
2020-06-01
2023-04-05
Trax Technology Solutions Pte Ltd.
Systems and methods for retail environments
CN111950354A
( en )
*
2020-06-30
2020-11-17
æ·±å³å¸éå¸ç§æè¡ä»½æéå ¬å¸
Seal home country identification method and device and electronic equipment
US12075175B1
( en )
2020-09-08
2024-08-27
Meta Platforms Technologies, Llc
Programmable smart sensor with adaptive readout
CN114328501A
( en )
*
2020-09-29
2022-04-12
å为ææ¯æéå ¬å¸
Data processing method, device and equipment
CN112215178B
( en )
*
2020-10-19
2024-05-28
å京大å¦
A chemical experiment recording system based on pen-based interaction
CN112287847B
( en )
*
2020-10-30
2024-12-20
广å·åå¤ç½ç»ç§ææéå ¬å¸
Method, device, terminal device and storage medium for determining image attributes
US11900662B2
( en )
2020-12-16
2024-02-13
Here Global B.V.
Method, apparatus, and computer program product for training a signature encoding module and a query processing module to identify objects of interest within an image utilizing digital signatures
US12073615B2
( en )
*
2020-12-16
2024-08-27
Here Global B.V.
Method, apparatus, and computer program product for identifying objects of interest within an image captured by a relocatable image capture device
EP4017003A1
( en )
*
2020-12-17
2022-06-22
Palantir Technologies Inc.
Approaches for lossy compression using machine learning
US11830103B2
( en )
2020-12-23
2023-11-28
Here Global B.V.
Method, apparatus, and computer program product for training a signature encoding module and a query processing module using augmented data
US11829192B2
( en )
2020-12-23
2023-11-28
Here Global B.V.
Method, apparatus, and computer program product for change detection based on digital signatures
US11587253B2
( en )
2020-12-23
2023-02-21
Here Global B.V.
Method, apparatus, and computer program product for displaying virtual graphical data based on digital signatures
US11861875B2
( en )
*
2021-01-29
2024-01-02
Tata Consultancy Limited Services
System and method for adaptive image transformation
CN112989039A
( en )
*
2021-02-08
2021-06-18
天翼ç©èç§ææéå ¬å¸
Method, system and storage medium for implementing small sample scene artificial intelligence
US20220269947A1
( en )
*
2021-02-25
2022-08-25
Parham Aarabi
System and method for media selection based on class extraction from text
US11164153B1
( en )
*
2021-04-27
2021-11-02
Skyhive Technologies Inc.
Generating skill data through machine learning
CN113269237B
( en )
*
2021-05-10
2022-12-27
éå²ç工大å¦
Assembly change detection method, device and medium based on attention mechanism
JP7004874B1
( en )
*
2021-06-08
2022-02-10
ä¸è±é»æ©æ ªå¼ä¼ç¤¾
Inspection support device, inspection support system, inspection support method and inspection support program
US11373146B1
( en )
2021-06-30
2022-06-28
Skyhive Technologies Inc.
Job description generation based on machine learning
US11961218B2
( en )
2021-07-29
2024-04-16
Zebra Technologies Corporation
Machine vision systems and methods for automatically generating one or more machine vision jobs based on region of interests (ROIs) of digital images
TWI779760B
( en )
*
2021-08-04
2022-10-01
çæ±åå°é«è¡ä»½æéå ¬å¸
Method of data augmentation and non-transitory computer-readable medium
US12282829B2
( en )
2021-08-25
2025-04-22
Capital One Services, Llc
Techniques for data type detection with learned metadata
CN114005020B
( en )
*
2021-11-05
2024-04-26
æ²³åå·¥ä¸å¤§å¦
A designated moving target detection method based on M3-YOLOv5
US11991295B2
( en )
2021-12-07
2024-05-21
Here Global B.V.
Method, apparatus, and computer program product for identifying an object of interest within an image from a digital signature generated by a signature encoding module including a hypernetwork
US12158910B2
( en )
*
2021-12-17
2024-12-03
Microsoft Technology Licensing, Llc
Systems and methods for utilizing object state distributions for image comparison tasks
US12488592B2
( en )
*
2021-12-21
2025-12-02
Waymo Llc
Systems, methods, and apparatus for using remote assistance to classify objects in an environment
WO2023146969A1
( en )
*
2022-01-26
2023-08-03
Meta Platforms Technologies, Llc
On-sensor image processor utilizing contextual data
US12244936B2
( en )
2022-01-26
2025-03-04
Meta Platforms Technologies, Llc
On-sensor image processor utilizing contextual data
US12423988B1
( en )
*
2022-03-16
2025-09-23
Lytx, Inc.
Methodology of building a multi task model into a single model
CN117172297A
( en )
*
2022-05-24
2023-12-05
å为ææ¯æéå ¬å¸
A data processing method and device
US12296846B2
( en )
*
2022-06-01
2025-05-13
Plusai, Inc.
Methods and apparatus for natural language based scenario discovery to train a machine learning model for a driving system
US20240005921A1
( en )
*
2022-06-29
2024-01-04
Apple Inc.
Command Disambiguation based on Environmental Context
WO2024019752A1
( en )
*
2022-07-18
2024-01-25
Google Llc
Systems and methods for place search in augmented reality
US12430788B2
( en )
*
2022-08-29
2025-09-30
Micron Technology, Inc.
Object location determination
US12493705B2
( en )
2022-08-31
2025-12-09
Snap Inc.
Selective collaborative object access based on timestamp
US12148114B2
( en )
2022-08-31
2024-11-19
Snap Inc.
Real-world responsiveness of a collaborative object
US12079395B2
( en )
2022-08-31
2024-09-03
Snap Inc.
Scissor hand gesture for a collaborative object
US12505239B2
( en )
2022-08-31
2025-12-23
Snap Inc.
Collaborative object associated with a geographical location
US12361106B2
( en )
2022-08-31
2025-07-15
Snap Inc.
Authenticating a selective collaborative object
US12299150B2
( en )
2022-08-31
2025-05-13
Snap Inc.
Selective collaborative object access
US12019773B2
( en )
2022-08-31
2024-06-25
Snap Inc.
Timelapse of generating a collaborative object
US12216703B2
( en )
2022-10-18
2025-02-04
Google Llc
Visual search determination for text-to-image replacement
US12254894B2
( en )
*
2022-12-02
2025-03-18
Mirro.ai
Systems and methods for emotion detection, prediction, annotation, and coaching
CN120752905A
( en )
*
2023-01-06
2025-10-03
å¯æ¯æä½å ¬å¸
Occupancy determination technology
US20240259639A1
( en )
*
2023-01-27
2024-08-01
Adeia Guides Inc.
Systems and methods for levaraging machine learning to enable user-specific real-time information services for identifiable objects within a video stream
US12469255B2
( en )
2023-02-13
2025-11-11
Walmart Apollo, Llc
Systems and methods for identifying different product identifiers that correspond to the same product
US12361451B2
( en )
*
2023-02-21
2025-07-15
Adeia Guides Inc.
Visual adwords in augmented reality based on quality and rarity of ambience specification
US11893464B1
( en )
2023-03-16
2024-02-06
edYou
Apparatus and methods for training an educational machine-learning model
KR102742515B1
( en )
*
2023-03-20
2024-12-16
주ìíì¬ ê°ì¹ë©ì¤
Apparatus and method for unstructured data analysis based on artificial intelligence
US12585644B2
( en )
*
2023-09-28
2026-03-24
Capital One Services, Llc
Context-dependent query generation and presentation
WO2025158395A1
( en )
*
2024-01-26
2025-07-31
Shaw Industries Group, Inc.
Method and system for enhanced product analysis through aesthetic, emotional, and material metadata matching
Citations (10)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US20130054603A1
( en )
*
2010-06-25
2013-02-28
U.S. Govt. As Repr. By The Secretary Of The Army
Method and apparatus for classifying known specimens and media using spectral properties and identifying unknown specimens and media
US8644624B2
( en )
2009-07-28
2014-02-04
Samsung Electronics Co., Ltd.
System and method for indoor-outdoor scene classification
US20140172643A1
( en )
2012-12-13
2014-06-19
Ehsan FAZL ERSI
System and method for categorizing an image
US9558268B2
( en )
2014-08-20
2017-01-31
Mitsubishi Electric Research Laboratories, Inc.
Method for semantically labeling an image of a scene using recursive context propagation
US20170083792A1
( en )
*
2015-09-22
2017-03-23
Xerox Corporation
Similarity-based detection of prominent objects using deep cnn pooling layers as features
US20170304732A1
( en )
*
2014-11-10
2017-10-26
Lego A/S
System and method for toy recognition
CN108052966A
( en )
2017-12-08
2018-05-18
éåºé®çµå¤§å¦
Remote sensing images scene based on convolutional neural networks automatically extracts and sorting technique
US20180204111A1
( en )
*
2013-02-28
2018-07-19
Z Advanced Computing, Inc.
System and Method for Extremely Efficient Image and Pattern Recognition and Artificial Intelligence Platform
US20190163982A1
( en )
*
2017-11-28
2019-05-30
Visual Semantics, Inc.
Method and apparatus for integration of detected object identifiers and semantic scene graph networks for captured visual scene be