Machine learning methods and systems for tracking shoppers and interactions … — Gary M. Zalewski (US11620879B2)

ABSTRACT

Abstract

Method and systems are provided for processing actions in a store. One example method includes capturing sensor output from two or more sensors in a shopping environment. One sensor includes a first camera to capture scene data where the scene includes movement of a shopper in the store performing interaction with an item in the store. Another sensor includes a second camera capturing at least part of the scene from a different perspective. The method includes processing, by a processing entity associated with the store, at least one of the camera's output to generate feature data. The feature data is processed by one or more machine learning models to produce engineered feature data. The engineered feature data includes data relating to tracking skeletal movement of shopper. The method includes processing, by a processing entity associated with the store, the feature data including engineered feature data using said one or more machine learning models to produce a prediction that a take of the item has occurred by the shopper. The prediction is based on a characterization that said feature data or engineered feature data is labeled to infer that the movement of the shopper is interaction with the item that is classified as the take as having occurred.

Description

CLAIM OF PRIORITY

This application is a continuation of U.S. patent application Ser. No. 16/137,498, filed on Sep. 20, 2018, entitled “MACHINE LEARNING METHODS AND SYSTEMS FOR MANAGING RETAIL STORE PROCESSES INVOLVING CASHIER-LESS TRANSACTIONS,” which is a continuation of U.S. patent application Ser. No. 15/859,627, filed on Dec. 31, 2017 (now U.S. Pat. No. 10,140,820, issued on Nov. 27, 2018), entitled “DEVICES FOR TRACKING RETAIL INTERACTIONS WITH GOODS AND ASSOCIATION TO USER ACCOUNTS FOR CASHIER-LESS TRANSACTIONS,” which is a continuation of U.S. patent application Ser. No. 15/376,636, filed on Dec. 12, 2016 (now U.S. Pat. No. 9,911,290, issued on Mar. 6, 2018), entitled “WIRELESS CODED COMMUNICATION (WCC) DEVICES FOR TRACKING RETAIL INTERACTIONS WITH GOODS AND ASSOCIATION TO USER ACCOUNTS,” which is a continuation-in-part of U.S. patent application Ser. No. 15/217,972, filed on Jul. 23, 2016 (now U.S. Pat. No. 10,142,822, issued on Nov. 27, 2018) entitled “WIRELESS CODED COMMUNICATION (WCC) DEVICES WITH ENERGY HARVESTING POWER SOURCES TRIGGERED WITH INCIDENTAL MECHANICAL FORCES,” which claims priority from U.S. Provisional Patent Application No. 62/197,003, filed on Jul. 25, 2015, and entitled “WIRELESS CODED COMMUNICATION (WCC) DEVICES AND METHODS, SYSTEMS, DEVICES USING OR IMPLEMENTING WCC DEVICES,” which are incorporated herein by reference.

U.S. patent application Ser. No. 15/217,972, filed on Jul. 23, 2016 claims priority from U.S. Provisional Patent Application No. 62/387,403, filed on Dec. 24, 2015, and entitled “SIGNALING SYSTEMS FOR WIRELESS CODED COMMUNICATION (WCC) DEVICES AND CONNECTED DEVICES,” which are both incorporated by reference for all purposes.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is related to:

(1) U.S. patent application Ser. No. 15/217,973, filed on Jul. 23, 2016, entitled “Wireless Coded Communication (WCC) Devices with Energy Harvesting Power Sources used in Switches,”

(2) U.S. patent application Ser. No. 15/217,974, filed on Jul. 23, 2016, entitled “Wireless Coded Communication (WCC) Devices with Energy Harvesting Power Sources for Processing Biometric Identified Functions,”

(3) U.S. patent application Ser. No. 15/217,975, filed on Jul. 23, 2016, entitled “Wireless Coded Communication (WCC) Devices with Energy Harvesting Power Sources for Wi-Fi Communication,”

(4) U.S. patent application Ser. No. 15/217,976, filed on Jul. 23, 2016, entitled “Wireless Coded Communication (WCC) Devices with Energy Harvesting Power Sources For Processing Internet Purchase Transactions,”

(5) U.S. patent application Ser. No. 15/217,977, filed on Jul. 23, 2016, entitled “Wireless Coded Communication (WCC) Devices with Energy Harvesting Power Sources For Monitoring State Data of Objects,” which are all incorporated herein by reference.

FIELD OF THE EMBODIMENTS

Embodiments are described regarding devices that transmit data, including those that may also sense input, produce output, receive, process and exchange data with end-nodes or networks of end-nodes. Some of the embodiments described herein relate to retail implementations. In such retail implementations, items may be sold to users that simply walk in, select an item or items, and then walk out without stopping at a cash register. Systems associated with a store can include a plurality of sensors of various types, which are used to identify various things, such as user identification, user interaction with items, confirmation that a user actually took an item off a shelf, confirmation that the item is not later dropped off in some other location if the user changes his or her mind, and adding of the item or items to a shopping cart managed by an online application, e.g., cloud, and then transacting payments for items confirmed to have been purchased. In some embodiments, the methods include instructions that are generated to enable pre-gathering of items from a store, for a user.

BACKGROUND

Over the years, there has been much advancement in the area of processing devices and devices that communicate over networks. For example, electronic devices are typically designed for specific applications. Some devices are more versatile, such as the common day smartphone or general purpose computers. These devices, although versatile and sometimes powerful, require network connections to send and receive data. Network connections, for example, are those provided by internet service providers (ISPs). Such connections can be obtained for private use, e.g., homes and businesses, and some can be obtained in public places. In either form, users are required to setup connections or initiate connections, e.g., to the Internet, via their devices and interfaces, before access is enabled.

As a result, devices provided with network access to exchange data must contend with setup procedures for obtaining access, must provide interfaces for managing setups, and must also provide sufficient power to process data and perform the communication. These typical requirements are, unfortunately, impediments to the simplification of devices that could benefit from data exchanges over networks. It is in this context that a need is present for embodiments described in this disclosure.

SUMMARY

The embodiments described herein relate to communications devices, communication methods, and methods for utilizing digital detection, sensing and tracking systems to determine interactions with retail items, in order to enable processing of cashier-less purchase transactions. In one embodiment, methods are provided to enable tracking of items in a store, and detect interactions with said items using one or more sensors. The interactions may be identified and classified, in order to determine if the user is intending to purchase the item and enabling charging of the user's account. In various embodiments described herein, a smart retail outlet is capable of classifying shopping activity for one or more shoppers into one or more categories, where upon a determination that a user fits in any particular category, or changes category, an action can be taken to dynamically respond to the state of the shopper. Described herein, a store tracks the taking of items into the possession of the shopper as well as the detection of returning items to the shelf, and is able to characterize the misplacement of returned items to the wrong shelves. In one embodiment, machine learning algorithms are used to predict a shopping list for a shopper. In one configuration, a retail outlet uses a mix of input data with machine learning model to classify shopping behavior and track events such as takes and returns and enables a family or group of shoppers to participate in the taking and returning of items to a single shopping account. Furthermore, the present invention includes capability to make a determination of the identity of each shopper taking or returning items from the shelf even when multiple shoppers are proximate to the shelf and interacting with the shelf at the same time.

In one embodiment, a method is provided for identifying actions in a retail store. This method includes sampling a shopping environment using one or more sensors. The method includes receiving output of the sampling as feature inputs to a machine learning classifier model to derive one or more labels characterizing a state of an item. At least one processing entity associated with the retail store detects the state of the item to change from one as item taken to one as item returned, and sensor data from said one or more sensors characterizing the item as having been returned to a wrong location in the retail store. In further embodiments, items taken are added to a user's electronic shopping cart to enable processing of a cashier-less transaction.

In some embodiments, a method includes processing of tracking data includes accessing a deep learning model using feature input of at least user location and shopping history, where the feature input includes time-based actions characterizing previous user shopping history and global positioning system data enabled by the user to allow identification of a location of the user based on location data received from the portable device.

In some embodiments, the current route of the portable device is an actual or predicted path in which the user is expected to travel from one location to a location of the store, wherein the predicted path is determined using global positioning system (GPS) dead reckoning or GPS and machine learning to classify that the current route of the portable device of the user remains headed to the store.

In some embodiments, the instructions to create the task for pre-gathering the items includes an identification of items from the shopping list and a time frame in which the user is expected to enter the store or in which the user will receive one or more of said items after entering the store.

In some embodiments, a trigger in sending of instructions is made based on (A) a calculation of expected time required to gather one or more of the items, and (B) a forecasted future time in which the user will either arrive at the store or require the one or more items to be ready for pickup

In some embodiments, the user remains headed to the store if the current route based on received and processed global positioning system (GPS) data of the portable device is on a course headed to the store and at least one processing entity associated with the store has not otherwise received data indicating that the user is not planning to visit or not likely to visit or plans to visit the store at a different time.

In some embodiments, the indicator is a payload, or a signal or a notification on the portable device or a device of the user that is associated with the user account, or is output on an in-store device or output by the store stating that one or more items are ready for pickup.

In some embodiments, the indicator is a state received by at least one of said processing entities in connection with a status of the task and a description of one or more items that have been gathered from the shopping list.

In some embodiments, the indicator is an actual receipt of the pre-gathered items.

In some embodiments, items ready for pickup are identified for collection when the user enters the store, or after having arrived at the store, or prior to exiting the store.

In some embodiments, the items that are ready for pickup are assigned for delivery to the user upon entering the store, or after having arrived at the store, or prior to exiting the store.

In some embodiments, the items that are pre-gathered are ones pre-designated on the user's shopping list, or are a subset of ones on the user's shopping list or are identified as suitable items for the user to purchase in connection with analyzing the user shopping list or user shopping history or data from at least one processing entity capable of keeping track of a consumption or use of items associated with the user.

In some embodiments, an item that is pre-gathered or scheduled to be pre-gathered is removed from the shopping list or identified on the shopping list as being gathered.

In some embodiments, the user remains headed to the store based on receipt of data from the portable device of the user or a processing entity associated with the user indicating confirmation or prediction that the user remains headed to the store or intends to head to the store.

In some embodiments, at least one or more of the said identified item or items on the shopping list are automatically identified by a deep learning model that predicts that one or more items are likely to be purchased by the user.

In some embodiments, the store is further configured with tracking sensors, said tracking sensors being in communication with at least one of said processing entities for monitoring interaction data between the user and one or more items in the store, said one or more processing entities examining the interaction data to determine if the user has taken an item that was already pre-gathered, and, if the processing entity determines the user has taken the item that was already pre-gathered, sending a guidance message to the portable device of the user, or to a store speaker, or to a display, or to a store asset proximate the user.

In some embodiments, the store is further configured with tracking sensors, said tracking sensors being in communication with at least one of said processing entities for monitoring a location of the user in the store, wherein at least one of said processing entities sends a wireless message containing a payload to a device in the store operated by a clerk in the store indicating a location of the user in the store, wherein the package of said one or more items from the store that were ordered and gathered are delivered to the user by the clerk at a current location of the user in the store.

In some embodiments, the store is further configured with tracking sensors, said tracking sensors being in communication with at least one of said processing entities for monitoring a location of the user in the store, wherein at least one of said processing entities provide the location of the user in the store to a robot, said robot having coupled thereto a package of said one or more items from the store that were ordered and gathered, where said robot delivers the items to a current location of the user in the store.

In some embodiments, the store is further configured with tracking sensors, said tracking sensors being in communication with at least one of said processing entities for monitoring a changing location of the user inside the store as well processing interaction data as input features to a machine learning model processed by at least one of said processing entities to classify an action of the user taking a specific item off a shelf, said at least one of said processing entities also access the user account to select additional items that have not been detected to have been taken by the user, said additional items are pre-gathered for the user, selection of items being based in part on a path of the user in the store.

In some embodiments, the selection of items to be pre-gathered is based on said path the user is moving through the store, a time allocation the user has in the store, or a preference of the user, or a promotion of the store.

In some embodiments, the store is further configured with tracking sensors, said tracking sensors being in communication with at least one of said processing entities for monitoring a changing location of the user inside the store and providing navigation guidance to an asset in the store or to the portable device of the user, said guidance aiding the user in traversing through the store along a navigation path through isles that contain one or more items on the shopping list of the user account, said at least one of said processing entities further accessing the user account to select additional items that will be pre-gathered for the user, the selection of items being based in part on a path the user moves within the store, or location of other users in the store, or on an amount of time.

In some embodiments, each of said one or more processing entities is defined by a computer, or a server, or a Lambda function process, or a function, or a process, or a service, or a container, or a container process, or a cloud processing system, or a smartphone, or a tablet, or a laptop, or a smart device, or a smart watch, or an internet of things (IOT) device, or a smart watch, or a router, or an access point, or a repeater, or a vehicle, or a camera, or a weight sensor, or an appliance, or lighting equ

CLAIM OF PRIORITY

This application is a continuation of U.S. patent application Ser. No. 16/137,498, filed on Sep. 20, 2018, entitled “MACHINE LEARNING METHODS AND SYSTEMS FOR MANAGING RETAIL STORE PROCESSES INVOLVING CASHIER-LESS TRANSACTIONS,” which is a continuation of U.S. patent application Ser. No. 15/859,627, filed on Dec. 31, 2017 (now U.S. Pat. No. 10,140,820, issued on Nov. 27, 2018), entitled “DEVICES FOR TRACKING RETAIL INTERACTIONS WITH GOODS AND ASSOCIATION TO USER ACCOUNTS FOR CASHIER-LESS TRANSACTIONS,” which is a continuation of U.S. patent application Ser. No. 15/376,636, filed on Dec. 12, 2016 (now U.S. Pat. No. 9,911,290, issued on Mar. 6, 2018), entitled “WIRELESS CODED COMMUNICATION (WCC) DEVICES FOR TRACKING RETAIL INTERACTIONS WITH GOODS AND ASSOCIATION TO USER ACCOUNTS,” which is a continuation-in-part of U.S. patent application Ser. No. 15/217,972, filed on Jul. 23, 2016 (now U.S. Pat. No. 10,142,822, issued on Nov. 27, 2018) entitled “WIRELESS CODED COMMUNICATION (WCC) DEVICES WITH ENERGY HARVESTING POWER SOURCES TRIGGERED WITH INCIDENTAL MECHANICAL FORCES,” which claims priority from U.S. Provisional Patent Application No. 62/197,003, filed on Jul. 25, 2015, and entitled “WIRELESS CODED COMMUNICATION (WCC) DEVICES AND METHODS, SYSTEMS, DEVICES USING OR IMPLEMENTING WCC DEVICES,” which are incorporated herein by reference.

U.S. patent application Ser. No. 15/217,972, filed on Jul. 23, 2016 claims priority from U.S. Provisional Patent Application No. 62/387,403, filed on Dec. 24, 2015, and entitled “SIGNALING SYSTEMS FOR WIRELESS CODED COMMUNICATION (WCC) DEVICES AND CONNECTED DEVICES,” which are both incorporated by reference for all purposes.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is related to:

(1) U.S. patent application Ser. No. 15/217,973, filed on Jul. 23, 2016, entitled “Wireless Coded Communication (WCC) Devices with Energy Harvesting Power Sources used in Switches,”

(2) U.S. patent application Ser. No. 15/217,974, filed on Jul. 23, 2016, entitled “Wireless Coded Communication (WCC) Devices with Energy Harvesting Power Sources for Processing Biometric Identified Functions,”

(3) U.S. patent application Ser. No. 15/217,975, filed on Jul. 23, 2016, entitled “Wireless Coded Communication (WCC) Devices with Energy Harvesting Power Sources for Wi-Fi Communication,”

(4) U.S. patent application Ser. No. 15/217,976, filed on Jul. 23, 2016, entitled “Wireless Coded Communication (WCC) Devices with Energy Harvesting Power Sources For Processing Internet Purchase Transactions,”

(5) U.S. patent application Ser. No. 15/217,977, filed on Jul. 23, 2016, entitled “Wireless Coded Communication (WCC) Devices with Energy Harvesting Power Sources For Monitoring State Data of Objects,” which are all incorporated herein by reference.

FIELD OF THE EMBODIMENTS

Embodiments are described regarding devices that transmit data, including those that may also sense input, produce output, receive, process and exchange data with end-nodes or networks of end-nodes. Some of the embodiments described herein relate to retail implementations. In such retail implementations, items may be sold to users that simply walk in, select an item or items, and then walk out without stopping at a cash register. Systems associated with a store can include a plurality of sensors of various types, which are used to identify various things, such as user identification, user interaction with items, confirmation that a user actually took an item off a shelf, confirmation that the item is not later dropped off in some other location if the user changes his or her mind, and adding of the item or items to a shopping cart managed by an online application, e.g., cloud, and then transacting payments for items confirmed to have been purchased. In some embodiments, the methods include instructions that are generated to enable pre-gathering of items from a store, for a user.

BACKGROUND

Over the years, there has been much advancement in the area of processing devices and devices that communicate over networks. For example, electronic devices are typically designed for specific applications. Some devices are more versatile, such as the common day smartphone or general purpose computers. These devices, although versatile and sometimes powerful, require network connections to send and receive data. Network connections, for example, are those provided by internet service providers (ISPs). Such connections can be obtained for private use, e.g., homes and businesses, and some can be obtained in public places. In either form, users are required to setup connections or initiate connections, e.g., to the Internet, via their devices and interfaces, before access is enabled.

As a result, devices provided with network access to exchange data must contend with setup procedures for obtaining access, must provide interfaces for managing setups, and must also provide sufficient power to process data and perform the communication. These typical requirements are, unfortunately, impediments to the simplification of devices that could benefit from data exchanges over networks. It is in this context that a need is present for embodiments described in this disclosure.

SUMMARY

The embodiments described herein relate to communications devices, communication methods, and methods for utilizing digital detection, sensing and tracking systems to determine interactions with retail items, in order to enable processing of cashier-less purchase transactions. In one embodiment, methods are provided to enable tracking of items in a store, and detect interactions with said items using one or more sensors. The interactions may be identified and classified, in order to determine if the user is intending to purchase the item and enabling charging of the user's account. In various embodiments described herein, a smart retail outlet is capable of classifying shopping activity for one or more shoppers into one or more categories, where upon a determination that a user fits in any particular category, or changes category, an action can be taken to dynamically respond to the state of the shopper. Described herein, a store tracks the taking of items into the possession of the shopper as well as the detection of returning items to the shelf, and is able to characterize the misplacement of returned items to the wrong shelves. In one embodiment, machine learning algorithms are used to predict a shopping list for a shopper. In one configuration, a retail outlet uses a mix of input data with machine learning model to classify shopping behavior and track events such as takes and returns and enables a family or group of shoppers to participate in the taking and returning of items to a single shopping account. Furthermore, the present invention includes capability to make a determination of the identity of each shopper taking or returning items from the shelf even when multiple shoppers are proximate to the shelf and interacting with the shelf at the same time.

In one embodiment, a method is provided for identifying actions in a retail store. This method includes sampling a shopping environment using one or more sensors. The method includes receiving output of the sampling as feature inputs to a machine learning classifier model to derive one or more labels characterizing a state of an item. At least one processing entity associated with the retail store detects the state of the item to change from one as item taken to one as item returned, and sensor data from said one or more sensors characterizing the item as having been returned to a wrong location in the retail store. In further embodiments, items taken are added to a user's electronic shopping cart to enable processing of a cashier-less transaction.

In some embodiments, a method includes processing of tracking data includes accessing a deep learning model using feature input of at least user location and shopping history, where the feature input includes time-based actions characterizing previous user shopping history and global positioning system data enabled by the user to allow identification of a location of the user based on location data received from the portable device.

In some embodiments, the current route of the portable device is an actual or predicted path in which the user is expected to travel from one location to a location of the store, wherein the predicted path is determined using global positioning system (GPS) dead reckoning or GPS and machine learning to classify that the current route of the portable device of the user remains headed to the store.

In some embodiments, the instructions to create the task for pre-gathering the items includes an identification of items from the shopping list and a time frame in which the user is expected to enter the store or in which the user will receive one or more of said items after entering the store.

In some embodiments, a trigger in sending of instructions is made based on (A) a calculation of expected time required to gather one or more of the items, and (B) a forecasted future time in which the user will either arrive at the store or require the one or more items to be ready for pickup

In some embodiments, the user remains headed to the store if the current route based on received and processed global positioning system (GPS) data of the portable device is on a course headed to the store and at least one processing entity associated with the store has not otherwise received data indicating that the user is not planning to visit or not likely to visit or plans to visit the store at a different time.

In some embodiments, the indicator is a payload, or a signal or a notification on the portable device or a device of the user that is associated with the user account, or is output on an in-store device or output by the store stating that one or more items are ready for pickup.

In some embodiments, the indicator is a state received by at least one of said processing entities in connection with a status of the task and a description of one or more items that have been gathered from the shopping list.

In some embodiments, the indicator is an actual receipt of the pre-gathered items.

In some embodiments, items ready for pickup are identified for collection when the user enters the store, or after having arrived at the store, or prior to exiting the store.

In some embodiments, the items that are ready for pickup are assigned for delivery to the user upon entering the store, or after having arrived at the store, or prior to exiting the store.

In some embodiments, the items that are pre-gathered are ones pre-designated on the user's shopping list, or are a subset of ones on the user's shopping list or are identified as suitable items for the user to purchase in connection with analyzing the user shopping list or user shopping history or data from at least one processing entity capable of keeping track of a consumption or use of items associated with the user.

In some embodiments, an item that is pre-gathered or scheduled to be pre-gathered is removed from the shopping list or identified on the shopping list as being gathered.

In some embodiments, the user remains headed to the store based on receipt of data from the portable device of the user or a processing entity associated with the user indicating confirmation or prediction that the user remains headed to the store or intends to head to the store.

In some embodiments, at least one or more of the said identified item or items on the shopping list are automatically identified by a deep learning model that predicts that one or more items are likely to be purchased by the user.

In some embodiments, the store is further configured with tracking sensors, said tracking sensors being in communication with at least one of said processing entities for monitoring interaction data between the user and one or more items in the store, said one or more processing entities examining the interaction data to determine if the user has taken an item that was already pre-gathered, and, if the processing entity determines the user has taken the item that was already pre-gathered, sending a guidance message to the portable device of the user, or to a store speaker, or to a display, or to a store asset proximate the user.

In some embodiments, the store is further configured with tracking sensors, said tracking sensors being in communication with at least one of said processing entities for monitoring a location of the user in the store, wherein at least one of said processing entities sends a wireless message containing a payload to a device in the store operated by a clerk in the store indicating a location of the user in the store, wherein the package of said one or more items from the store that were ordered and gathered are delivered to the user by the clerk at a current location of the user in the store.

In some embodiments, the store is further configured with tracking sensors, said tracking sensors being in communication with at least one of said processing entities for monitoring a location of the user in the store, wherein at least one of said processing entities provide the location of the user in the store to a robot, said robot having coupled thereto a package of said one or more items from the store that were ordered and gathered, where said robot delivers the items to a current location of the user in the store.

In some embodiments, the store is further configured with tracking sensors, said tracking sensors being in communication with at least one of said processing entities for monitoring a changing location of the user inside the store as well processing interaction data as input features to a machine learning model processed by at least one of said processing entities to classify an action of the user taking a specific item off a shelf, said at least one of said processing entities also access the user account to select additional items that have not been detected to have been taken by the user, said additional items are pre-gathered for the user, selection of items being based in part on a path of the user in the store.

In some embodiments, the selection of items to be pre-gathered is based on said path the user is moving through the store, a time allocation the user has in the store, or a preference of the user, or a promotion of the store.

In some embodiments, the store is further configured with tracking sensors, said tracking sensors being in communication with at least one of said processing entities for monitoring a changing location of the user inside the store and providing navigation guidance to an asset in the store or to the portable device of the user, said guidance aiding the user in traversing through the store along a navigation path through isles that contain one or more items on the shopping list of the user account, said at least one of said processing entities further accessing the user account to select additional items that will be pre-gathered for the user, the selection of items being based in part on a path the user moves within the store, or location of other users in the store, or on an amount of time.

In some embodiments, each of said one or more processing entities is defined by a computer, or a server, or a Lambda function process, or a function, or a process, or a service, or a container, or a container process, or a cloud processing system, or a smartphone, or a tablet, or a laptop, or a smart device, or a smart watch, or an internet of things (IOT) device, or a smart watch, or a router, or an access point, or a repeater, or a vehicle, or a camera, or a weight sensor, or an appliance, or lighting equipment, or a switch, or a container application, or an application, or a wireless coded communication (WCC) device, or a state machine, or by a stateless service, or a combination of two or more thereof,

wherein each one of said one or more processing entities is coupled to or interconnected to a network that provides access to storage, data, or instructions.

In some embodiments, confirming that the current route of the portable device remains headed to the store includes receiving information from the user account, as provided by the user, that the user is headed to the store at a particular time or is currently headed to the store.

In one embodiment, a method is provided for processing requests for items to be pre-gathered from a store. The processing of the requests is executed using one or more processing entities. One method includes receiving tracking data from portable device associated with a user having an online account with the store. One or more items are identified from a shopping list of the user as associated with the online account of the store. Processing the tracking data is made to determine a current route of the portable device to the store. Sending instructions is processed to create a task for pre-gathering one or more items from the store. The sending of instructions is triggered in response to confirming that the current route of the portable device remains headed to the store. The method also includes receiving an indicator that the one or more items from the store have been gathered. The method causes the sending of a notification to the online account of the user that a package of said one or more items from the store that were ordered have been gathered and are ready for pickup at the store.

In one embodiment, a method includes identifying a shopper in a store using sensors and associating the shopper to a shopping account. Monitoring the shopper in the store using one or more sensors, and output of the one or more sensors is an input to one or more deep learning models that generate classification data in connection with a scenario. The method includes receiving natural language voice input containing words spoken from the shopper. And, processing the voice input along with the classification data to generate a response to the words spoken by said shopper. The response is made in a context of the scenario.

In some embodiments, the one or more sensors used for monitoring the shopper are associated with the store and coordinated to identify movement of the shopper while in the store and movement of the shopper relative to one or more items located in the store, and the scenario is associated with actions by the shopper in relation to one or more of said items and said movements of the shopper identified by the one or more sensors, and said response is in context of the scenario.

In some embodiments, the one or more sensors used for monitoring the shopper are associated with the store and coordinated to identify interaction of the shopper with said one or more items located in the store, the scenario is associated with said interactions by the shopper in relation to one or more of said items, and said response is in context of the interactions that relate to the scenario.

In some embodiments, the response is configured to dynamically change based on changes in said interactions by the shopper in relation to said one or more of said items as the shopper is detected to move about the store, wherein movement of the shopper about the store places the user proximate to or further from specific ones of said items.

In some embodiments, said response includes guidance information that communicates directions to the shopper for movement toward a specific item or groups of items and in some cases beam forming is used in a capture of audio.

In some embodiments, the guidance includes one of more of directions to move a view of the shopper toward a direction or directions for the shopper to move to another location within the store in which the specific item or groups of items are located.

In some embodiments, the classification data includes an indication that the shopper has taken an item from a shelf of the store, or has returned an item from the shelf, or is gazing in a particular direction, or is gazing toward a particular product, or has a type of sentiment, or has performed a gesture, or is searching for a product, or is interacting with a mobile or wearable device.

In some embodiments, devices are integrated with a wireless communication chip and integrated power generating or delivering device. In one embodiment, these devices are referred to herein as wireless coded communication (WCC) device. Such devices are configured to harness power to cause or enable activation of a communication device to transmit data. The data can be pre-configured or coded to report occurrence of an event, log an event, log state, cause an action, send a message or request data from one or more end nodes. In some embodiments, the devices enable communication over a wireless network, which enables access to the Internet and further enables cloud processing on data received or processing for data returned or communicated.

Broadly speaking, a WCC device is one that has or is coupled with a wireless transmission capability (e.g., a transmitter, a transceiver, Wi-Fi chip, Bluetooth chip, radio communication chip, etc.), and a power pump or a power supply.

In one embodiment, a method for tracking items in a physical store for processing cashier-less purchase transactions is provided. The method includes detecting a portable wireless coded communication (WCC) device in the physical store. The WCC device is associated with an on-line account of a shopper. Then, receiving, by a server, sensor data regarding the WCC device, its location in the physical store and proximity to items in the physical store. The method includes receiving, by the server, interaction data of an item on a shelf of the physical store by the shopper using one or more sensors of the physical store and the WCC device to determine that the item is one targeted for purchase. The interaction data is configured to identify a type of said item and add it to an electronic shopping cart of the shopper having the on-line account for processing said cashier-less purchase transactions. The method further includes processing, by the server, an electronic charge to a payment service of the shopper for the item. The WCC device may be a portable device of the user, and an application for said cashier-less purchase transactions can be run by the WCC device or portable device of the user.

In one configuration, a power pump device is configured to receive a force or movement input from a user or object. The force can be a direct force that is intentionally input by the user, e.g., by pushing a button, moving a slider, lifting a tab, shifting a lever, etc. Another type of force is an indirect force, which is one that the user is not intending to provide. Instead, force is created when the user moves, opens, lifts, shifts, closes, or somehow changes the position or orientation of a physical object.

For example only, a physical object may be a door, and closing of the door is the intent of the user. In some embodiments, the physical object may be a store shelf, a store item on a shelf, e.g., retail store and retail items. The indirect force of closing the door is then, in one embodiment, transferred as force to a WCC device. As such, even though the user did not have intent to provide input or force to the WCC device, the WCC device received the force. In the case of a battery operated WCC, the WCC device could also receive the input indirectly. That is, when the door is closed, an input trigger or sensor can be set or impacted or triggered, and this input to the WCC device is incidental. The input is then used to activate a process of the WCC device and to automatically transmit data or request data wirelessly over a network.

In one embodiment, a method for tracking items in a store for processing cashier-less purchase transactions is provided. The method includes receiving, by a server, sensor data regarding items associated with shelves of a store, and receiving, by the server, interaction data of an item of a shelf of the store by a user. The interaction data is configured to identify a type of said item and add it to an electronic shopping cart of the user, if a take event is confirmed. The user account is used for processing said cashier-less purchase transactions. The embodiment includes receiving, by the server, data indicative of said user leaving an area of the store that is indicative that the user has purchased the item. The server is configured to process or instruct processing of an electronic charge to a payment service of the user for the item upon confirming that said user left the area of the store.

In some embodiments, a volume or area in which the item is located is tracked to obtain said interaction data. The interaction data is used to determine movement or removal of said item from the shelf and determine if a take event has occurred with respect to the item and the item is added to said electronic shopping cart.

In some embodiments, said movement is identified at least using image data from a camera. The camera is configured to produce image data that is processed as part of said sensor data that is received by said server.

In some embodiments, a confidence level of identifying said movement is increased by using input from one or more other sensors, each sensor is provided a weighting that gives a respective sensor or sensors more or less significance in said identifying of the movement.

In some embodiments, the method further includes receiving, by a server, data indicative of a user device being present in the store. The data from the user device is used to identify the user account of the user for enabling use of the electronic shopping cart for buying one or more items in the store via one of said cashier-less purchase transactions.

In some embodiments, the method further includes receiving, by a server, tracking data regarding movement of the user in the store, the movement of the user is indicative of browsing of said items on one or more of said shelves. The tracking includes tracking a hand of the user, the hand of the user caused by the user to touch, lift, move, and/or remove said item from the shelf, the hand tracking is further usable as said interaction data with the item.

In some embodiments, the method further includes sending, by the server, display information for rendering on a display screen that is proximate to the shelf having the item for which the user is interacting. The display information is customized for the user based on information obtained from a profile of the user associated with the user account.

In some embodiments, the method further includes sending, by the server, audio data for outputting from a region of the store that is proximate to the shelf having the item for which the user is interacting, the audio data being customized for the user based on information obtained from a profile of the user associated with the user account.

In some embodiments, the audio data is configured to be output in a directional tracking format, the directional tracking format implements audio steering to direct said audio data toward a head of the user, wherein said audio steering focuses sound for the user while excluding distribution of sound substantially beyond the head of the user.

In some embodiments, the method further includes receiving, by the server, eye or head gaze information of the user. The eye or head gaze information is indicative of actions taken by the user with the item for which the user is interacting. The eye or head gaze information is collected to determine product information that is of interest to the user. The server is configured to collect the product information that is of interest to the user and using said information to provide recommendations to the user regarding other items that may be of interest and/or to provide discount or promotional information for the item or other items.

In some embodiments, the method further includes receiving, by the server, first data indicative that a take event occurred for the item and second data indicative that the item having a tag communicates with a user device of the user. The user device of the user verifying that the item was subjected to the take even and was not returned prior to the user leaving the area of the store before processing the electronic charge. The user device is configured to communicate with at least one sensor in the store that monitors when the user device is leaving the store.

In some embodiments, the method further includes receiving, by the server, proximity data of the user and a second user, the user and the second user configured to be proximate to each other near the shelf. The proximity data of the user and the second user is usable to distinguish the user and the second user. Cameras are disposed in the store to produce part of the sensor data, and the cameras producing image data that is trackable to identify limbs of each of the user and the second user, and said limbs trackable in motion and used to verify whether the user or the second user is interacting with the item on the self.

In some embodiments, the interaction data includes one or more features that are descriptive of detected interaction. The features are processed as inputs to a model that classifies the interaction to be one of a take event or a returning event of the item from the shelf or back to the shelf.

In some embodiments, classifying is provided, which uses the model to assists in predicting an action to be taken by the user or other users. The model is one that refines its classification over time based on prior interactions and assists in determining whether to add the item or other items to the electronic shopping cart of the user or other users.

In some embodiments, the classifying uses the model to assist in generating recommendations to the user or other users based on learned likely actions to be taken by users when interacting with the item or items, the classifying being a process that continually learns to optimize the model, and said continual learning uses inputs of features from the store and/or other stores that are connected to a shopping service that enables said cashier-less purchase transactions.

In one embodiment, a method for tracking items in a store for processing a cashier-less purchase transaction is provided. The method includes receiving sensor data regarding items associated with shelves of a store. The method also includes identifying a user entering the store. The identifying uses at least a device having wireless communication. The device has an application that is executed in associated with a user account of the user. Also, tracking the user movements while in the store. The tracked movements include detecting proximity of the user to a shelf having an item. And, detecting interaction data for the item on the shelf of the store by the user. The interaction data is configured to identify a type of said item and enable add said item to an electronic shopping cart of the user having said user account for enabling processing of said cashier-less purchase transaction. The method includes receiving data indicative of said user leaving an area of the store while the item is in the electronic shopping cart. Leaving is indicative that the user intends to purchase the item. An exit of the area includes one or more sensors for confirming that the user has left the area. The method includes processing an electronic charge to a payment service associated with the user account of the user for the item based on said confirming.

In one embodiment, if the WCC device uses a power pump, the force or physical pressure or movement received can be imparted or communicated to a mechanically flexible device or element. In one embodiment, the flexible element may be a piezoelectric device. A piezoelectric device is one that can produce a voltage when the force or movement is imparted to it, and the result is that a voltage is produced. That voltage can be harvested and stored to a storage device. In one embodiment, the storage device can be capacitive storage device (e.g., a capacitor). In other embodiments, the storage device can be a battery or a rechargeable battery cell or cells. In one implementation, if the storage device is a capacitive storage device, the device will serve as the power source to an integrated circuit of the WCC device.

In one embodiment, the integrated circuit can be programmed to communicate data or a code or a message to an end node. The end node can store the received data or perform an action, e.g., based on pre-programming.

In one embodiment, the device is a wireless coded communication (WCC) device. The WCC device, in one configuration, is passive, in that the device is not connected to active power (e.g., a battery or power plug). However, once a force, e.g., mechanical force is applied to the power pump of the WCC device, the WCC device is activated for a period of time sufficient to process and transmit coded data to the end point or end-node. In some embodiments, a WCC can be programmed with more than one method or function to perform. For example, the selection of the method or function can be based on a pre-activation setting. In other cases, a method is automatically selected based on the power available. If more power is available, methods that require more power are performed. In certain embodiments, an input feature such as a selection control, switch, slider, touch pad, may be coupled to the WCC, which generate a payload. The payload may be transmitted remotely to an end node or processed by the WCC locally, or both in part or whole.

In some implementations, functions may be selected and triggered according to results of processing of the payload before, during or after transmission of the payload. In certain embodiments, the WCC includes the capability to detect or image the identity or attribute of a user, a biometric signature, a fingerprint, a voice, a sound, a scene, an object position, QR code, RFID code, barcode, status, temperature, pressure, absence or presence of conditions, environmental condition, vibration, and any signal or source coupled to, near or within sensing range of one or more sensors coupled to, or integrated with, a WCC device.

In other embodiments, devices, systems, and method are provided. One such device is a wireless coded communication (WCC) device, which may be configured for wireless communication with other devices, e.g., over a network. A WCC device is a type of internet of things (IOT) device that can sense data, process data, send data, respond to data requests and exchange data with other WCC device, a network device, a user device, and/or systems over the internet. In some configurations, a WCC device may include a power source that enables usage of low power, e.g., to send data that is sensed, request data and/or communicate data wirelessly. WCC devices maybe function as standalone devices or may be integrated into other devices. In some configurations, a WCC device may include power harvesting circuitry. A WCC device may be pre-configured or coded to report occurrence of an event, log an event, log state, cause an action, send a message or request data from one or more end nodes. In some configurations, the devices enable communication over a wireless network, which enables access to the Internet and further enables cloud processing on data received or processing for data returned or communicated.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention may best be understood by reference to the following description taken in conjunction with the accompanying drawings.

FIG. 1 A illustrates an example of a user shopping in a store, in accordance with one embodiment of the present invention.

FIG. 1 B illustrates an example of various types of sensors, and interactions between a network, and compute entities, in accordance with one embodiment.

FIG. 1 C illustrates an example of use of machine learning, in accordance with one embodiment.

FIG. 1 D illustrates one example flowchart, associated with tracking an item that has been selected by a user for purchase, for a cashier-less process.

FIG. 1 E illustrates another example process, which may be processed in order to determine when an item should be charge to the user, in a cashier-less purchase transaction

FIG. 1 F illustrates one example embodiment of a WCC device, in accordance with one embodiment.

FIG. 1 G illustrates an example of a WCC device, which includes a full wave rectifier, in accordance with one embodiment.

FIG. 2 illustrates another example implementation of a WCC device, in accordance with one embodiment.

FIG. 3 illustrates an example of program settings, in accordance with one embodiment.

FIG. 4 A illustrates another embodiment of a WCC device, in accordance with some embodiments.

FIG. 4 B illustrates another example of a WCC, in accordance with some embodiments.

FIG. 4 C illustrates an example of a WCC device that includes a power source and a wireless transceiver, and an input for receiving force input, in accordance with some embodiments.

FIG. 5 A illustrates an example of different types of power sources, which are nonexclusive of other types that are possible, in accordance with some embodiments.

FIG. 5 B illustrates an example of a bridge circuit, in accordance with some embodiments.

FIG. 6 illustrates another example of a WCC device, which can be provided with multiple button presses or different numbers of physical inputs to provide input or charge a power storage device of the WCC, or both, in accordance with some embodiments.

FIG. 7 illustrates another embodiment of a WCC integrated into a device, such as a key fob, in accordance with some embodiments.

FIG. 8 illustrates an example where a WCC device can be programmed, in accordance with some embodiments.

FIG. 9 shows another example of receiving program in the IC in operation, in accordance with some embodiments.

FIG. 10 illustrates another example where program settings can be set at a website or on a mobile application, in accordance with some embodiments.

FIGS. 11 A- 11 B illustrate examples of a WCC device that is integrated with a wall switch or other locations, in accordance with some embodiments.

FIG. 12 A illustrates a general, exaggerated and unscaled, example diagram of one of many potential charging and consumption profiles that may be achieved when charging a power storage cell of a WCC device, when the WCC device includes a power pump, in accordance with some embodiments.

FIG. 12 B illustrates another example similar to FIG. 12 A , where a power pump may be used to charge a storage cell, such as one that can store power in a capacitive form, in accordance with some embodiments.

FIG. 12 C illustrates an example where a WCC device can be programmed or set to operate a mode 3 program, which requires power 3, in accordance with some embodiments.

FIG. 13 A illustrates an example of a WCC device, associated with a local proximity, in accordance with some embodiments.

FIG. 13 B illustrates an example where a WCC device can communicate wirelessly to a connected node, in accordance with some embodiments.

FIG. 13 C illustrates another example where a WCC device can communicate directly to an end node, in accordance with some embodiments.

FIG. 14 illustrates an example of the use of a WCC device, in accordance with one embodiment.

FIG. 15 illustrates an example of a doorway that includes the door and hinges, to illustrate one example of a WCC, in accordance with some embodiments.

FIG. 16 illustrates an example of a hinged box, which may include a hinge, in accordance with some embodiments.

FIG. 17 illustrates an example of the power tool, each may implement a WCC device, in accordance with some embodiments.

FIG. 18 illustrates an example of a WCC device, which can be in the form of a handheld fob, in accordance with some embodiments.

FIG. 19 illustrates another example of a WCC device, which can also include a display, in accordance with some embodiments.

FIG. 20 illustrates an example of program code, which can be dynamically selected by a user, in accordance with some embodiments.

FIG. 21 illustrates an example of the WCC device which may also include a button that includes a biometric sensor, in accordance with some embodiments.

FIGS. 22 A- 22 F illustrates various examples of communications between a WCC device and various nodes over one or more networks or directly between nodes, in accordance with some embodiments.

FIG. 23 illustrates an example of a WCC device utilized by a user to take a picture, and communicate the picture to end node, in accordance with some embodiments.

FIG. 24 illustrates another embodiment of the WCC device, in accordance with one embodiment of the present disclosure, in accordance with some embodiments.

FIGS. 25 A- 25 D illustrate other example uses of WCC devices 100 , in accordance with one embodiment

FIGS. 26 A-E illustrate various examples of WCC devices that can be integrated into various objects or things, in accordance with some embodiments.

FIG. 27 illustrates an example of a wall switch utilize to turn power on and off in a specific room or light fixture, in accordance with some embodiments.

FIG. 28 illustrates an example of a vending machine, which may have a screen, selection input buttons, a slider for dispensing, and multiple sensors, in accordance with one embodiment.

FIGS. 29 A-B illustrate examples of retail objects that may be stored on shelves, such as store shelves, in accordance with some embodiments.

FIG. 30 illustrates an example of a home, which may include a number of WCC devices integrated into various components, objects, things, and wireless communication to a router, in accordance with some embodiments.

FIG. 31 illustrates a bicycle which may include a WCC device, in accordance with some embodiments.

FIG. 32 A illustrates the use of a WCC device for a user, such as integration into an ID badge, in accordance with some embodiments.

FIG. 32 B illustrates another embodiment where a WCC device can be attached to the user in a necklace format, in accordance with some embodiments.

FIGS. 33 a - 33 e illustrates another embodiment for use of WCC device, in accordance with some embodiments.

FIG. 34 A illustrates an example of a user interfacing with the WCC terminal, in accordance with some embodiments.

FIG. 34 B illustrates one example of WCC terminal, in accordance with one embodiment, in accordance with some embodiments.

FIG. 34 C illustrates a different type of display, which can be pressed at different locations to activate the energy harvesting and also activate retrieval of information associated with the option, in accordance with some embodiments.

FIG. 34 D shows another option of a WCC terminal, which includes a rotating knob, in accordance with some embodiments.

FIG. 35 A- 35 C illustrate examples of a WCC terminal, with a display screen and an outer shell that can be rotated, while leaving the screen in its present nonmoving position, in accordance with some embodiments.

FIG. 36 A as an example of the WCC device, which can be used to order goods or services, in accordance with some embodiments.

FIG. 36 B illustrates an example of various types of messages and or responses that can be displayed or presented to the user in the message displayed, in accordance with some embodiments.

FIGS. 37 A- 37 C illustrate examples of a WCC device which can be configured for use by persons with disabilities, in accordance with some embodiments.

FIGS. 38 A- 38 B illustrate examples of a user interfacing with an artificial intelligence (AI) bot, in accordance with some embodiments.

FIG. 39 illustrates an example of a user providing a query to the AI bot, in accordance with some embodiments.

FIG. 40 illustrates another configuration of interface by a user with a voice handler, which can be connected to a network device, in accordance with some embodiments.

FIG. 41 A illustrates an example of a user interfacing with a WCC, which can function as a terminal for communicating with the hub, which in turn communicates with WCC devices over a network, in one embodiment.

FIGS. 41 B and 41 C illustrate examples of queries made to the WCC device at step 1, and the responses received by the user from the WCC device at step 8, in accordance with some embodiments.

FIGS. 42 A- 42 G illustrate examples of WCC devices used in the context of power cords, in accordance with some embodiments.

FIGS. 43 A- 43 C illustrate examples of a WCC device that can be powered by motion and a power pump can be charged or receive power in response to reactive motions of magnets integrated in an object, in accordance with some embodiments.

FIGS. 44 A- 44 E show examples of a control knob, housing, dial, terminal and/or structure that maybe provided for user interfacing, in accordance with some embodiments.

FIGS. 45 A and 45 B will now be described with reference to example methods to manage security associated with data being transferred to and from WCC devices, in accordance with some embodiments.

FIG. 46 shows an example of a system with inductive taps, in

FIG. 47 shows examples of WCC devices that can be disposed in locations to track positions of devices, e.g., to provide a type of internal GPS, in accordance with some embodiments.

FIGS. 48 A- 48 B illustrate examples of WCC devices in electrical boxes and WCC watches, in accordance with some embodiments.

FIGS. 49 A- 49 B illustrate examples of a WCC watch and use of the watch to instruct, command, obtain and interface with data wirelessly, in accordance with some embodiments.

FIG. 50 illustrate examples of obtaining data and use of harvested power, in accordance with some embodiments.

FIG. 51 A illustrates an example store environment, where a user is approaching a shelf in the store</figure-cal

CLAIMS

Claims ( 19 )

What is claimed is:

1. A method for processing actions for a store, comprising:

capturing sensor output from two or more sensors in a shopping environment, one sensor including a first camera to capture scene data where the scene includes movement of a shopper in the store performing interaction with an item in the store, and another sensor including a second camera capturing at least part of the scene from a different perspective;

processing, by a first processing entity associated with the store, at least one of the camera&#39;s output to generate feature data, the feature data processed by one or more machine learning models to track skeletal movement of at least part of a skeleton generated for the shopper;

processing, by a second processing entity associated with the store, the feature data including engineered feature data using said one or more machine learning models to produce a prediction that a take of the item has occurred by the shopper;

wherein the prediction is based on a characterization that said feature data or engineered feature data is labeled to infer that the movement of the shopper is interaction with the item that is classified as the take as having occurred, wherein said inferring uses data related to said tracked skeletal movements of the shopper to process motion of at least part of the skeleton of the shopper along a traversal path in a direction of the item.

2. The method of claim 1 , wherein the said feature data or said engineered feature data is produced from a sensor fusion of two or more outputs from said sensors in the shopping environment.

3. The method of claim 1 , wherein an allocation of weights is used in fusing two or more outputs from said sensors is based, in part, on an output of a machine learning model.

4. The method of claim 1 , further tracking the shopper as they navigate around the store to detect if the shopper returns the item to any location, and if the shopper is not detected to have returned the item to any location in the store, adding the item to a queue for purchase of the item.

5. The method of claim 1 , further comprising,

confirming that the shopper has exited the store and charging the item to an account associated with the shopper, wherein confirming the shopper has exited the store is based on tracking the shopper moving past a region in the store, or is based on a label characterizing a scene in the store associated with a likeliness that the shopper has left the store, or is based on a state of a wireless signal of the shopper, or is based on a message received, or is based on a mobile phone of the shopper, or is based on tracking image data associated with shopper and the image data inferring that the shopper has left the store, or based on one or more sensors that are external to a region associated with the store.

6. The method of claim 1 , wherein each camera producing image data used to track the shopper as the shopper moves from one location to another location in the store, and image data of at least one camera includes depth sensing features.

7. The method of claim 1 , wherein each camera source is coupled to trained machine learning model for inferring that the movement of the shopper has interacted with the item to classify the take as having occurred.

8. The method of claim 1 , wherein each camera source is coupled to trained machine learning model for generating engineered features or labels regarding shopper interaction with the item.

9. The method of claim 8 , wherein a label from one model is used as a feature input to another model and wherein two or more models are nested or cascaded or combined to characterize the interaction of the shopper in connection with the scene.

10. The method of claim 1 , wherein weights are assigned dynamically to give a relative relevance of two or more features and used in a prediction to infer actions associated with a behavior state of the shopper in relation to the item.

11. The method of claim 1 , further comprising,

causing at least one processing entity to send a message identifying the item being chargeable to an electronic shopping cart, the message including descriptive information of the item, the descriptive item used to cause a debit to a credit account that is associated with the electronic shopping cart.

12. The method of claim 1 , wherein feature inputs to the one or more machine learning models used to produce a prediction that the take of the item by the shopper has occurred further includes one or more of audio output features, or microphone array output features, or engineered features, or labeled features, or labels output from another machine learning model, or weight sensor features, or motion sensor features, or biometric sensor features, or infrared sensor features, or proximity detection sensor features, or wireless signal features, or voice features, or recognized gesture features.

13. A method for identifying actions for a store, comprising:

capturing data from one or more sensors in a shopping environment, the one or more sensors include sensors associated to locations related to specific items for sale in the store, and sensors for tracking actions of a shopper in the store;

processing output from the one or more sensors, the output including image data and interaction data in the store, the image data includes shopper image data and item image data, and the interaction data relates to handling of items by the shopper;

labeling features of the output using one or more machine learning models trained to recognize the shopper image data, the item image data and the interaction data, the one or more machine learning models producing labels that characterize the shopper image data and the interaction data, the interaction data includes tracked motion of a limb of a skeleton of the shopper along a traversal path in a direction of an item;

processing the labels to identify the item in the store as being handled by the shopper; and

tracking shopper image data and the item image data while the shopper remains in the store to confirm classification of a take of the item so that the item is made chargeable to an electronic shopping cart of the shopper.

14. The method of claim 13 , wherein the one or more sensors in the shopping environment include one or more cameras, the one or more cameras are assigned to locations in the shopping environment and the tracking of the shopper image data occurs using one or a combination of the one or more cameras that follow movement of the shopper while in the shopping environment.

15. The method of claim 14 , wherein said following movement of the shopper while in the shopping environment is processed to confirm the classification of the take of the item or a put back of the item back from an original location or to a wrong location, and wherein the put back of the item acts to remove the item as chargeable to the electronic shopping cart.

16. The method of claim 13 , wherein the shopper identifying features and shopper actions are tracked temporarily only to facilitate a transaction and non-essential tracking history is not stored by processing systems store processing after shopping activity terminates.

17. The method of claim 13 , wherein the shopper identifying features and shopper actions are tracked over time enabling passive shopper detection upon entry to the store; or wherein the store is configured with at least one shopper aware display configured to present information responsive to interactions of the shopper in a scene.

18. A method for identifying actions for a store, comprising:

capturing data from one or more sensors in a shopping environment, the one or more sensors usable to track items for sale in the store and track actions of a shopper in the store;

processing output from the one or more sensors, the output including image data and interaction data in the store, the image data includes shopper image data and item image data, and the interaction data relates to handling of items by the shopper;

labeling features of the processed output using one or more artificial intelligence models trained to recognize the shopper image data, the item image data and the interaction data, wherein the labels are used to characterize the shopper image data and the interaction data, the interaction data includes tracked motion of a limb of a skeleton of the shopper along a traversal path in a direction of an item;

processing the interaction data to identify a take of the item from the store by the shopper, the take causing the item to be added to a queue of an electronic shopping cart of the shopper, the item is chargeable to the shopping cart when the shopper is determined to have removed the item from store.

19. The method of claim 18 , wherein the one or more sensors include one or more motion sensors, or weight sensors, or audio sensors, or fusions thereof, used to reinforce said interaction data for identifying the take of said item by a limb of the shopper.

US16/714,779

2015-07-25

2019-12-15

Machine learning methods and systems for tracking shoppers and interactions with items in a cashier-less store

Active

2038-03-08

US11620879B2

( en )

Priority Applications (1)

Application Number

Priority Date

Filing Date

Title

US16/714,779

US11620879B2

( en )

2015-07-25

2019-12-15

Machine learning methods and systems for tracking shoppers and interactions with items in a cashier-less store

Applications Claiming Priority (7)

Application Number

Priority Date

Filing Date

Title

US201562197003P

2015-07-25

2015-07-25

US201562387403P

2015-12-24

2015-12-24

US15/217,972

US10142822B1

( en )

2015-07-25

2016-07-23

Wireless coded communication (WCC) devices with power harvesting power sources triggered with incidental mechanical forces

US15/376,636

US9911290B1

( en )

2015-07-25

2016-12-12

Wireless coded communication (WCC) devices for tracking retail interactions with goods and association to user accounts

US15/859,627

US10140820B1

( en )

2015-07-25

2017-12-31

Devices for tracking retail interactions with goods and association to user accounts for cashier-less transactions

US16/137,498

US10510219B1

( en )

2015-07-25

2018-09-20

Machine learning methods and systems for managing retail store processes involving cashier-less transactions

US16/714,779

US11620879B2

( en )

2015-07-25

2019-12-15

Machine learning methods and systems for tracking shoppers and interactions with items in a cashier-less store

Related Parent Applications (1)

Application Number

Title

Priority Date

Filing Date

US16/137,498

Continuation

US10510219B1

( en )

2015-07-25

2018-09-20

Machine learning methods and systems for managing retail store processes involving cashier-less transactions

Publications (2)

Publication Number

Publication Date

US20200118401A1

US20200118401A1 ( en )

2020-04-16

US11620879B2

true

US11620879B2 ( en )

2023-04-04

Family

ID=61257399

Family Applications (17)

Application Number

Title

Priority Date

Filing Date

US15/376,636

Active

US9911290B1

( en )

2015-07-25

2016-12-12

Wireless coded communication (WCC) devices for tracking retail interactions with goods and association to user accounts

US15/859,627

Active

US10140820B1

( en )

2015-07-25

2017-12-31

Devices for tracking retail interactions with goods and association to user accounts for cashier-less transactions

US15/959,274

Active

US10977907B1

( en )

2015-07-25

2018-04-22

Devices for tracking retail interactions with goods including contextual voice input processing and artificial intelligent responses

US15/969,753

Active

2038-11-13

US11195388B1

( en )

2015-07-25

2018-05-02

Machine learning methods and systems for managing retail store processes involving the automatic gathering of items

US16/137,498

Active

US10510219B1

( en )

2015-07-25

2018-09-20

Machine learning methods and systems for managing retail store processes involving cashier-less transactions

US16/143,449

Active

US10573134B1

( en )

2015-07-25

2018-09-26

Machine learning methods and system for tracking label coded items in a retail store for cashier-less transactions

US16/204,057

Active

2038-01-10

US11288933B1

( en )

2015-07-25

2018-11-29

Devices for tracking retail interactions with goods and association to user accounts for cashier-less transactions

US16/221,541

Active

2038-04-16

US11315393B1

( en )

2015-07-25

2018-12-16

Scenario characterization using machine learning user tracking and profiling for a cashier-less retail store

US16/236,344

Active

2038-08-27

US11417179B1

( en )

2015-07-25

2018-12-29

Using image and voice tracking to contextually respond to a user in a shopping environment

US16/714,777

Active

2038-04-23

US11636740B2

( en )

2015-07-25

2019-12-15

Methods and systems for identifying actions of shoppers in stores in relation to items provided for sale in cashier-less transactions

US16/714,779

Active

2038-03-08

US11620879B2

( en )

2015-07-25

2019-12-15

Machine learning methods and systems for tracking shoppers and interactions with items in a cashier-less store

US16/716,466

Active

2038-08-04

US11749073B2

( en )

2015-07-25

2019-12-16

Methods and systems for identifying actions of a shopper to account for taken items in cashier-less transactions

US16/718,177

Active

2038-10-25

US11783682B2

( en )

2015-07-25

2019-12-17

Artificial intelligence systems and methods for cashierless tracking and feedback

US16/732,137

Active

2038-02-16

US11670141B1

( en )

2015-07-25

2019-12-31

Cashierless system using deep learning processing of hand scan biometric feature data for labeling an identity of a tracked shopper

US18/378,102

Pending

US20240038032A1

( en )

2015-07-25

2023-10-09

Artificial intelligence systems and methods for cashierless tracking and feedback

US18/438,497

Pending

US20240265789A1

( en )

2015-07-25

2024-02-11

Electronic shelf labels and retail store integration

US18/747,233

Pending

US20240340034A1

( en )

2015-07-25

2024-06-18

Packet tracking energy harvesting device for location and condition state monitoring

Family Applications Before (10)

Application Number

Title

Priority Date

Filing Date

US15/376,636

Active

US9911290B1

( en )

2015-07-25

2016-12-12

Wireless coded communication (WCC) devices for tracking retail interactions with goods and association to user accounts

US15/859,627

Active

US10140820B1

( en )

2015-07-25

2017-12-31

Devices for tracking retail interactions with goods and association to user accounts for cashier-less transactions

US15/959,274

Active

US10977907B1

( en )

2015-07-25

2018-04-22

Devices for tracking retail interactions with goods including contextual voice input processing and artificial intelligent responses

US15/969,753

Active

2038-11-13

US11195388B1

( en )

2015-07-25

2018-05-02

Machine learning methods and systems for managing retail store processes involving the automatic gathering of items

US16/137,498

Active

US10510219B1

( en )

2015-07-25

2018-09-20

Machine learning methods and systems for managing retail store processes involving cashier-less transactions

US16/143,449

Active

US10573134B1

( en )

2015-07-25

2018-09-26

Machine learning methods and system for tracking label coded items in a retail store for cashier-less transactions

US16/204,057

Active

2038-01-10

US11288933B1

( en )

2015-07-25

2018-11-29

Devices for tracking retail interactions with goods and association to user accounts for cashier-less transactions

US16/221,541

Active

2038-04-16

US11315393B1

( en )

2015-07-25

2018-12-16

Scenario characterization using machine learning user tracking and profiling for a cashier-less retail store

US16/236,344

Active

2038-08-27

US11417179B1

( en )

2015-07-25

2018-12-29

Using image and voice tracking to contextually respond to a user in a shopping environment

US16/714,777

Active

2038-04-23

US11636740B2

( en )

2015-07-25

2019-12-15

Methods and systems for identifying actions of shoppers in stores in relation to items provided for sale in cashier-less transactions

Family Applications After (6)

Application Number

Title

Priority Date

Filing Date

US16/716,466

Active

2038-08-04

US11749073B2

( en )

2015-07-25

2019-12-16

Methods and systems for identifying actions of a shopper to account for taken items in cashier-less transactions

US16/718,177

Active

2038-10-25

US11783682B2

( en )

2015-07-25

2019-12-17

Artificial intelligence systems and methods for cashierless tracking and feedback

US16/732,137

Active

2038-02-16

US11670141B1

( en )

2015-07-25

2019-12-31

Cashierless system using deep learning processing of hand scan biometric feature data for labeling an identity of a tracked shopper

US18/378,102

Pending

US20240038032A1

( en )

2015-07-25

2023-10-09

Artificial intelligence systems and methods for cashierless tracking and feedback

US18/438,497

Pending

US20240265789A1

( en )

2015-07-25

2024-02-11

Electronic shelf labels and retail store integration

US18/747,233

Pending

US20240340034A1

( en )

2015-07-25

2024-06-18

Packet tracking energy harvesting device for location and condition state monitoring

Country Status (1)

Country

Link

US

( 17 )

US9911290B1

( en )

Cited By (7)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20200391501A1

( en )

*

2018-03-06

2020-12-17

Hewlett-Packard Development Company, L.P.

Door locks

US20220309855A1

( en )

*

2021-03-25

2022-09-29

International Business Machines Corporation

Robotic protection barrier

US20220408338A1

( en )

*

2021-06-16

2022-12-22

Medtronic Minimed, Inc.

Medicine administration in dynamic networks

US20230005267A1

( en )

*

2021-06-30

2023-01-05

Fujitsu Limited

Computer-readable recording medium, fraud detection method, and fraud detection apparatus

US20230087730A1

( en )

*

2021-09-22

2023-03-23

Dell Products L.P.

Method for Interference Avoidance

US20230267743A1

( en )

*

2022-02-22

2023-08-24

Fujitsu Limited

Computer-readable recording medium, information processing method, and information processing apparatus

US20240242567A1

( en )

*

2017-07-08

2024-07-18

Gaming Analytics Inc.

Machine-learning platform for marketing promotions decision making

Families Citing this family (532)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US10664904B2

( en )

*

2007-07-20

2020-05-26

Daphne WRIGHT

System, device and method for detecting and monitoring a biological stress response for financial rules behavior

US10535027B2

( en )

2013-11-26

2020-01-14

Paypal, Inc.

Merchant action recommendation system

US12548018B2

( en )

2013-12-18

2026-02-10

Payrange Llc

Device and method for providing external access to multi-drop bus peripheral devices

US11099166B2

( en )

2014-04-04

2021-08-24

Nectar, Inc.

Container content quantity measurement and analysis

US11012764B2

( en )

2014-06-04

2021-05-18

Nectar, Inc.

Interrogation signal parameter configuration

US10324075B2

( en )

2014-04-04

2019-06-18

Nectar, Inc.

Transmitter and receiver configuration for detecting content level

US12248939B2

( en )

2014-06-30

2025-03-11

Payrange Llc

Intermediary communications over non-persistent network connections

US12548032B2

( en )

2014-06-30

2026-02-10

Payrange Llc

Processing interrupted transactions over non-persistent network connections

US9860242B2

( en )

*

2014-08-11

2018-01-02

Vivint, Inc.

One-time access to an automation system

US10027501B1

( en )

*

2014-11-06

2018-07-17

United Services Automobile Association (Usaa)

Systems and methods for pre-configuring a payment vehicle

US10084865B2

( en )

2015-02-26

2018-09-25

Urban Airship, Inc.

Mobile event notifications

US10200486B2

( en )

2015-02-26

2019-02-05

Urban Airship, Inc.

Mobile event notifications for network enabled objects

US10706456B2

( en )

2015-04-22

2020-07-07

Staples, Inc.

Intelligent item tracking and expedited item reordering by stakeholders

US10476972B2

( en )

*

2015-04-23

2019-11-12

Google Llc

User presence determination

SE542202C2

( en )

*

2015-04-27

2020-03-10

Picadeli Ab

A monitoring and controlling system for a product holding arrangement

CN106302330B

( en )

*

2015-05-21

2021-01-05

腾讯科技（深圳）有限公司

Identity verification method, device and system

US11313940B2

( en )

*

2015-06-29

2022-04-26

Here Global B.V.

Advertising ambient pressure information

US10355730B1

( en )

*

2015-07-25

2019-07-16

Gary M. Zalewski

Wireless coded communication (WCC) devices with power harvesting power sources for processing internet purchase transactions

JP2017076338A

( en )

*

2015-10-16

2017-04-20

ソニー株式会社

Information processing device, information processing method, wearable terminal, and program

US10586205B2

( en )

2015-12-30

2020-03-10

Walmart Apollo, Llc

Apparatus and method for monitoring stock information in a shopping space

US20170221296A1

( en )

2016-02-02

2017-08-03

6d bytes inc.

Automated preparation and dispensation of food and beverage products

US20170262933A1

( en )

2016-03-10

2017-09-14

Khai Gan Chuah

Offline to online management system

US10558943B2

( en )

*

2016-04-20

2020-02-11

Wishelf Ltd.

System and method for monitoring stocking shelves

US10497014B2

( en )

*

2016-04-22

2019-12-03

Inreality Limited

Retail store digital shelf for recommending products utilizing facial recognition in a peer to peer network

US10846996B2

( en )

2016-04-25

2020-11-24

Standard Cognition Corp.

Registry verification for a mechanized store using radio frequency tags

US12169818B2

( en )

*

2016-05-04

2024-12-17

Mariella Labels Oy

Simultaneously displaying new prices linked to products via an electronic price label system

JP7009389B2

( en )

2016-05-09

2022-01-25

グラバンゴ コーポレイション

Systems and methods for computer vision driven applications in the environment

WO2018002927A1

( en )

*

2016-06-29

2018-01-04

Yedvab Hagit Hava

Systems and methods for ad-hoc trigger based data sharing group

WO2018013439A1

( en )

2016-07-09

2018-01-18

Grabango Co.

Remote state following devices

US20180047007A1

( en )

*

2016-08-09

2018-02-15

Wal-Mart Stores, Inc.

System and method for paying for goods at a door

CA3034696A1

( en )

*

2016-09-07

2018-03-15

Walmart Apollo, Llc

Apparatus and method for providing item interaction with a virtual store

US10586206B2

( en )

*

2016-09-22

2020-03-10

Walmart Apollo, Llc

Systems and methods for monitoring conditions on shelves

US10128912B2

( en )

*

2016-09-26

2018-11-13

Blackberry Limited

Efficient near field communication (NFC) tag detection and related methods

US10368057B1

( en )

*

2016-09-28

2019-07-30

Amazon Technologies, Inc.

Synchronizing data streams

US11625966B2

( en )

2020-10-14

2023-04-11

1Ahead Technologies

Access management system

US11398120B2

( en )

2020-10-14

2022-07-26

1Ahead Technologies

Security surveillance and entry management system

US12682706B2

( en )

2021-06-30

2026-07-14

1HEAD Technologies

Access management system

US11403901B2

( en )

2020-10-14

2022-08-02

1Ahead Technologies

Entry management system

US11303856B1

( en )

2020-10-14

2022-04-12

1Ahead Technologies

Security entry and delivery system and method of using same

US11756357B2

( en )

2020-10-14

2023-09-12

1Ahead Technologies

Access management system

US11854328B2

( en )

2020-10-14

2023-12-26

1Ahead Technologies

Access management system

US11468723B1

( en )

2020-10-14

2022-10-11

1Ahead Technologies

Access management system

US11151825B1

( en )

2020-10-14

2021-10-19

1Ahead Technologies

Artificial intelligence entry management device, system and method of using same

US11436882B1

( en )

2020-10-14

2022-09-06

1Ahead Technologies

Security surveillance and entry management system

US11393269B2

( en )

2020-10-14

2022-07-19

1Ahead Technologies

Security surveillance and entry management system

US10783682B2

( en )

2016-11-02

2020-09-22

United Parcel Service Of America, Inc.

Displaying items of interest in an augmented reality environment

CN108066078A

( en )

*

2016-11-10

2018-05-25

睿传数据股份有限公司

Intelligent bedside card and control management system thereof

US11120389B2

( en )

2016-11-15

2021-09-14

United Parcel Service Of America, Inc.

Electronically connectable packaging systems configured for shipping items

TWI666598B

( en )

*

2016-12-01

2019-07-21

Institute For Information Industry

Inventory management system and inventory management method

DE102016124964A1

( en )

*

2016-12-20

2018-06-21

Rehau Ag &amp; Co

Device for attachment to a storage device of a goods rack and system with such a device

US10612964B1

( en )

2016-12-21

2020-04-07

Amazon Technologies, Inc.

System to mitigate effects of vibration on load cell

EP3563605B1

( en )

*

2016-12-27

2021-07-21

Denso Corporation

System and method for microlocation sensor communication

US10839203B1

( en )

*

2016-12-27

2020-11-17

Amazon Technologies, Inc.

Recognizing and tracking poses using digital imagery captured from multiple fields of view

US10943285B1

( en )

*

2016-12-28

2021-03-09

Amazon Technologies, Inc.

Updating cart state using auto-generated sensor data

US11068962B1

( en )

*

2016-12-28

2021-07-20

Amazon Technologies, Inc.

Updating cart state using auto-generated sensor data

US11132737B2

( en )

2017-02-10

2021-09-28

Grabango Co.

Dynamic customer checkout experience within an automated shopping environment

US10368306B2

( en )

*

2017-02-17

2019-07-30

Mediatek Inc.

Communicating method and associated electronic device

US20180240180A1

( en )

*

2017-02-20

2018-08-23

Grabango Co.

Contextually aware customer item entry for autonomous shopping applications

US20200012958A1

( en )

*

2017-02-22

2020-01-09

Appnext Ltd.

System and method for real-time generation of predictive models of mobile users&#39; behavior

GB2560177A

( en )

2017-03-01

2018-09-05

Thirdeye Labs Ltd

Training a computational neural network

US10528147B2

( en )

*

2017-03-06

2020-01-07

Microsoft Technology Licensing, Llc

Ultrasonic based gesture recognition

GB2560387B

( en )

2017-03-10

2022-03-09

Standard Cognition Corp

Action identification using neural networks

JP6855862B2

( en )

*

2017-03-22

2021-04-07

日本電気株式会社

Product management device, control method of product management device, and program

JP6870411B2

( en )

2017-03-22

2021-05-12

日本電気株式会社

Product management device, control method of product management device, and program

US11087271B1

( en )

2017-03-27

2021-08-10

Amazon Technologies, Inc.

Identifying user-item interactions in an automated facility

US11494729B1

( en )

*

2017-03-27

2022-11-08

Amazon Technologies, Inc.

Identifying user-item interactions in an automated facility

JP7689357B2

( en )

*

2017-03-28

2025-06-06

オートマトン， インク．

Method and apparatus for locating RFID tags - Patents.com

US10250868B1

( en )

2017-03-29

2019-04-02

Amazon Technologies, Inc.

Synchronizing data streams

US10699421B1

( en )

2017-03-29

2020-06-30

Amazon Technologies, Inc.

Tracking objects in three-dimensional space using calibrated visual cameras and depth cameras

CN107391983B

( en )

2017-03-31

2020-10-16

创新先进技术有限公司

Information processing method and device based on Internet of things

US10896562B2

( en )

*

2017-04-03

2021-01-19

1Ahead Technologies

Secured delivery system and method of using same

US20180286185A1

( en )

*

2017-04-03

2018-10-04

Walmart Apollo, Llc

Automated checkout

US10475259B2

( en )

*

2017-04-03

2019-11-12

Ronald Carter

Security system and method of using same

US10553102B1

( en )

*

2017-04-17

2020-02-04

Ripple Network Technologies, Inc.

Methods and apparatuses for personal safety

US10142165B2

( en )

*

2017-04-24

2018-11-27

Hall Labs Llc

Adaptive communication channel redundancy in a hub-based intermediate-range system

Related documents

• Lie Detector At the Gait: Artificial Intelligence Model#1152348
• Leveraging Machine Learning to Effortlessly Track Patient Movement in the Clinic.#1139741
• AI Eyes on the Road: Cross-Cultural Perspectives on Traffic Surveillance#1062801
• Smart Road Surveillance: Helmet Detection and Triple Ride Detection Using CNN and YOLO#86926
• Behavioural Biometrics for Continuous Authentication in Cybersecurity Systems#100240