Google Patentsopen access

Systems and Methods for Deploying and Updating Neural Networks at the Edge of a … — Ubotica Technologies Limited (US20250363357A1)

ABSTRACT

Abstract

Methods, devices and system for updating a neural network on an edge device that has low-bandwidth uplink capability include a centralized site/device that is configured to train and send the neural network to the edge device. In response, the centralized site/device may receive neural network information from the edge device that includes all or portions of a dataset, output activations, and/or overall inference result that is collected or generated in the edge device. The centralized site/device may use the received neural network information to update all or a part of the trained neural network, generate updated neural network information based on the updated neural network, and send the updated neural network information to the edge device.

Description

RELATED APPLICATIONS

This application claims is a continuation of U.S. patent application Ser. No. 16/797,422 filed on Feb. 21, 2020, which claims the benefit of priority to U.S. Provisional Application 62/809,353 entitled “Systems and Methods for Deploying and Updating Neural Networks at the Edge of a Network” filed Feb. 22, 2019, the entire contents of which are hereby incorporated by reference for all purposes.

BACKGROUND

Artificial intelligence (AI) and related technologies have seen significant advancements in recent years. In particular, neural networks have transitioned from being specialist academic projects to being used in mainstream commercial and consumer facing applications. These applications and technologies have the potential to solve a variety of long-standing technical challenges. However, existing and conventional solutions for deploying and using neural networks still include a number of significant limitations.

SUMMARY

The various aspects include methods of updating a neural network on an edge device that has low-bandwidth uplink capability, which may include a processor in a centralized site/device training the neural network, sending the trained neural network to the edge device, receiving neural network information from the edge device (the received neural network information including at least a portion of at least one or more of a dataset, an activation, or an overall inference result collected or generated in the edge device), using the received neural network information to update all or a part of the trained neural network, generating updated neural network information based on the updated neural network, and sending the updated neural network information to the edge device.

In some aspects, sending the trained neural network to the edge device may include sending the trained neural network to an edge device that has been deployed. In some aspects, using the received neural network information to update all or a part of the trained neural network and generating the updated neural network information based on the updated neural network may include generating a neural network difference model by comparing the updated neural network to the trained neural network.

In some aspects, generating the neural network difference model by comparing the updated neural network to the trained neural network may include generating a patch that identifies the differences between the updated neural network and the trained neural network via one of layer freezing using a minimum size technique, layer freezing using a minimum delta technique, weights freezing using the minimum size technique, or weights freezing using the minimum delta technique. In some aspects, generating the neural network difference model by comparing the updated neural network to the trained neural network may include determining one or more neural network layers or one or more neural network weights of the one or more neural network layers to freeze based on a mean of activations of layers in the neural network.

In some aspects, the methods may include the edge device receiving the trained neural network, collecting the dataset from sensors of the edge device, applying the collected dataset as inputs to the received neural network to generate activations and the overall inference result, storing at least a portion of at least one or more of the collected dataset, the generated activations, or the overall inference result in a memory of the edge device, and sending the neural network information that includes at least a portion of at least one or more of the collected dataset, the generated activations, or the overall inference result to the centralized site/device.

In some aspects, the methods may include the edge device receiving the updated neural network information, generating an updated neural network based on the received trained neural network and the received updated neural network information, and applying a second dataset as input to the updated neural network to generate second inference results. In some aspects, receiving the updated neural network information may include receiving a neural network difference model.

In some aspects, training the neural network may include collecting training data from one or more of a plurality of edge devices, labelling the collected training data, selecting two or more lightweight neural networks, generating an ensemble based on the selected neural networks, and using the labelled training data to train the ensemble, and sending the trained neural network to the edge device may include sending the trained ensemble and an ensemble aggregation function to the edge device.

In some aspects, using the received neural network information to update all or a part of the trained neural network may include adding a neural network to the trained ensemble. In some aspects, using the received neural network information to update all or a part of the trained neural network may include updating the ensemble aggregation function based on a result of analyzing the received neural network information, and updating all or a part of the trained neural network based on the updated ensemble aggregation function.

In some aspects, receiving the trained neural network may include receiving a trained ensemble, and applying the collected dataset as inputs to the received neural network to generate the activations and the overall inference result may include applying the collected dataset as inputs to the received ensemble to generate the activations and the overall inference result. In some aspects, training the neural network may include generating a stratified neural network that includes large data volume parts and small data parts. In some aspects, sending the updated neural network information to the edge device may include sending the small data parts of the stratified neural network to the edge device.

In some aspects, generating the stratified neural network that includes the large data volume parts and the small data parts may include generating the stratified neural network to include a large data volume part that includes a feature identification layer, and a small data part that includes a fully connected layer. In some aspects, generating the stratified neural network that includes the large data volume parts and the small data parts may include generating the stratified neural network to include large data volume parts that include multiple partial layers that are not cross-connected, and small data parts that include cross-connected weights between the multiple partial layers in the large data volume parts. In some aspects, generating the stratified neural network that includes the large data volume parts and the small data parts may include generating the stratified neural network to include large data volume parts that include layers with a higher numerical precision, and small data parts that include layers with a lower numerical precision.

In some aspects, using the received neural network information to update all or a part of the trained neural network and generating the updated neural network information based on the updated neural network may include retraining only the small data parts of the stratified neural network.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated herein and constitute part of this specification, illustrate exemplary aspects of the invention, and together with the general description given above and the detailed description given below, serve to explain the features of the invention.

FIG. 1 A is component block diagram illustrating a system, which includes edge devices in the form of satellites and a centralized site/device connected to a series of transmission sites, that is suitable for implementing various embodiments.

FIG. 1 B is component block diagram illustrating a system that includes different types of edge devices (i.e., a network of heterogeneous edge devices), and is suitable for implementing various embodiments.

FIGS. 2 A and 2 B are block diagrams illustrating edge devices suitable for implementing various embodiments.

FIG. 3 is a process flow diagram illustrating a method of deploying a new edge device in accordance with an embodiment.

FIG. 4 is an activity diagram illustrating a method of performing transfer learning in accordance with an embodiment.

FIG. 5 is a process flow diagram illustrating a method of improving or updating functionality on devices that use narrowband channels in accordance with some embodiments.

FIGS. 6 - 8 are activity diagram illustrating methods of updating neural networks on edge devices that have limited connectivity in accordance with various embodiments.

FIG. 9 is an activity diagram illustrating a method of updating a neural network on one or more edge devices without requiring transmission of neural network data in accordance with an embodiment.

FIG. 10 is an activity diagram illustrating a method of updating an edge device to account for sensor variations and faults in accordance with an embodiment.

FIG. 11 is an activity diagram illustrating a method of using information from proceeding edge devices in a chain of edge devices to update a neural network or otherwise improve or update the functionality of an edge device in accordance with an embodiment.

FIG. 12 is an activity diagram illustrating a method of updating a neural network to extend the mission of a deployed edge device in accordance with an embodiment.

FIGS. 13 A- 13 C are process flow diagrams illustrating methods of updating neural networks on edge devices that have limited connectivity in accordance with some embodiments.

FIG. 14 is a component diagram of server computing device suitable for implementing some embodiments.

DESCRIPTION

The various embodiments will be described in detail with reference to the accompanying drawings. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts. References made to particular examples and implementations are for illustrative purposes, and are not intended to limit the scope of the invention or the claims.

In overview, the various embodiments include a sensor-rich programmable artificial intelligence (AI) inference and compute platform that is suitable for deployment at the extreme network edge, from the oceans of earth to low earth orbit, geosynchronous orbit and deep space. The AI inference and compute platform, alternatively termed the AI inference engine or the AI engine, may use machine learning accelerators, neural network accelerators, convolutional neural network accelerators, neuromorphic accelerators or a combination thereof, or may contain solely general compute.

The word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any implementation described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other implementations.

The term “computing device” may be used herein to refer to any one or all of server computing devices, personal computers, laptop computers, tablet computers, edge devices, user equipment (UE), multimedia Internet enabled cellular telephones, smartphones, smart wearable devices (e.g., smartwatch, smart glasses, fitness tracker, clothes, jewelry, shoes, etc.), Internet-of-Things (IoT) devices (e.g., smart televisions, smart speakers, smart locks, lighting systems, smart switches, smart doorbell cameras or security systems, etc.), connected vehicles, and other similar devices that include a memory and programmable processor for providing the functionality described herein.

The term “edge device” may be used herein to refer to any one or all of computing devices, satellites, connected vehicles (trucks, cars, etc.), electric scooters, trains, trams, metros (which often only have connectivity for brief periods while in stations), aircraft, drones (based on land, in sea, or in the air), high-altitude balloons, smartphones, smart wearable devices, IoT devices, eMobility devices (e.g., electric scooters, electric bikes), robots, nanobots, and other similar computing systems, devices or objects that include a memory, a sensor, a processor, and communications circuity for communicating with computing devices at one or more centralized sites. The processor may be a programmable processor or a fixed programmed processor (e.g., a pre-programmed FPGA or an ASIC) with associated reconfigurable parameters stored in an associated memory. Edge devices are often resource-constrained devices that have limited processing, memory, battery and/or bandwidth resources.

The term “centralized site” may be used herein to refer to a control site that includes one or more computing devices (or “centralized devices”) that are configured to initiate, provision, store data on (e.g., collected data, data obtained from other sources, augmented data, etc.), enable labeling on, train, communicate with and/or control edge devices. For ease of reference and to focus the description on the relevant features or functionalities, some embodiments are described herein with reference to a “centralized site/device” on earth and one or more edge devices deployed in space. However, it should be understood that the described features and functionalities may be applicable to other types of edge devices, systems, configurations or deployments. As such, nothing in this application should be used to limit the claims or disclosures herein to a centralized site/device on earth and edge devices deployed in space unless expressly recited as such within the claims.

The term “AI edge device” may be used herein to refer to an edge device that is configured to perform AI operations locally on the device and/or to work in conjunction with other devices (e.g., another edge device, centralized site/device, etc.) that perform AI operations. For example, an AI edge device may be an edge device that includes an edge AI processor configured to perform “inference” and/or to otherwise deploy or use a neural network that utilizes or accomplishes machine learning locally on the device. As another example, an AI edge device may be configured to collect data (o

RELATED APPLICATIONS

This application claims is a continuation of U.S. patent application Ser. No. 16/797,422 filed on Feb. 21, 2020, which claims the benefit of priority to U.S. Provisional Application 62/809,353 entitled “Systems and Methods for Deploying and Updating Neural Networks at the Edge of a Network” filed Feb. 22, 2019, the entire contents of which are hereby incorporated by reference for all purposes.

BACKGROUND

Artificial intelligence (AI) and related technologies have seen significant advancements in recent years. In particular, neural networks have transitioned from being specialist academic projects to being used in mainstream commercial and consumer facing applications. These applications and technologies have the potential to solve a variety of long-standing technical challenges. However, existing and conventional solutions for deploying and using neural networks still include a number of significant limitations.

SUMMARY

The various aspects include methods of updating a neural network on an edge device that has low-bandwidth uplink capability, which may include a processor in a centralized site/device training the neural network, sending the trained neural network to the edge device, receiving neural network information from the edge device (the received neural network information including at least a portion of at least one or more of a dataset, an activation, or an overall inference result collected or generated in the edge device), using the received neural network information to update all or a part of the trained neural network, generating updated neural network information based on the updated neural network, and sending the updated neural network information to the edge device.

In some aspects, sending the trained neural network to the edge device may include sending the trained neural network to an edge device that has been deployed. In some aspects, using the received neural network information to update all or a part of the trained neural network and generating the updated neural network information based on the updated neural network may include generating a neural network difference model by comparing the updated neural network to the trained neural network.

In some aspects, generating the neural network difference model by comparing the updated neural network to the trained neural network may include generating a patch that identifies the differences between the updated neural network and the trained neural network via one of layer freezing using a minimum size technique, layer freezing using a minimum delta technique, weights freezing using the minimum size technique, or weights freezing using the minimum delta technique. In some aspects, generating the neural network difference model by comparing the updated neural network to the trained neural network may include determining one or more neural network layers or one or more neural network weights of the one or more neural network layers to freeze based on a mean of activations of layers in the neural network.

In some aspects, the methods may include the edge device receiving the trained neural network, collecting the dataset from sensors of the edge device, applying the collected dataset as inputs to the received neural network to generate activations and the overall inference result, storing at least a portion of at least one or more of the collected dataset, the generated activations, or the overall inference result in a memory of the edge device, and sending the neural network information that includes at least a portion of at least one or more of the collected dataset, the generated activations, or the overall inference result to the centralized site/device.

In some aspects, the methods may include the edge device receiving the updated neural network information, generating an updated neural network based on the received trained neural network and the received updated neural network information, and applying a second dataset as input to the updated neural network to generate second inference results. In some aspects, receiving the updated neural network information may include receiving a neural network difference model.

In some aspects, training the neural network may include collecting training data from one or more of a plurality of edge devices, labelling the collected training data, selecting two or more lightweight neural networks, generating an ensemble based on the selected neural networks, and using the labelled training data to train the ensemble, and sending the trained neural network to the edge device may include sending the trained ensemble and an ensemble aggregation function to the edge device.

In some aspects, using the received neural network information to update all or a part of the trained neural network may include adding a neural network to the trained ensemble. In some aspects, using the received neural network information to update all or a part of the trained neural network may include updating the ensemble aggregation function based on a result of analyzing the received neural network information, and updating all or a part of the trained neural network based on the updated ensemble aggregation function.

In some aspects, receiving the trained neural network may include receiving a trained ensemble, and applying the collected dataset as inputs to the received neural network to generate the activations and the overall inference result may include applying the collected dataset as inputs to the received ensemble to generate the activations and the overall inference result. In some aspects, training the neural network may include generating a stratified neural network that includes large data volume parts and small data parts. In some aspects, sending the updated neural network information to the edge device may include sending the small data parts of the stratified neural network to the edge device.

In some aspects, generating the stratified neural network that includes the large data volume parts and the small data parts may include generating the stratified neural network to include a large data volume part that includes a feature identification layer, and a small data part that includes a fully connected layer. In some aspects, generating the stratified neural network that includes the large data volume parts and the small data parts may include generating the stratified neural network to include large data volume parts that include multiple partial layers that are not cross-connected, and small data parts that include cross-connected weights between the multiple partial layers in the large data volume parts. In some aspects, generating the stratified neural network that includes the large data volume parts and the small data parts may include generating the stratified neural network to include large data volume parts that include layers with a higher numerical precision, and small data parts that include layers with a lower numerical precision.

In some aspects, using the received neural network information to update all or a part of the trained neural network and generating the updated neural network information based on the updated neural network may include retraining only the small data parts of the stratified neural network.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated herein and constitute part of this specification, illustrate exemplary aspects of the invention, and together with the general description given above and the detailed description given below, serve to explain the features of the invention.

FIG. 1 A is component block diagram illustrating a system, which includes edge devices in the form of satellites and a centralized site/device connected to a series of transmission sites, that is suitable for implementing various embodiments.

FIG. 1 B is component block diagram illustrating a system that includes different types of edge devices (i.e., a network of heterogeneous edge devices), and is suitable for implementing various embodiments.

FIGS. 2 A and 2 B are block diagrams illustrating edge devices suitable for implementing various embodiments.

FIG. 3 is a process flow diagram illustrating a method of deploying a new edge device in accordance with an embodiment.

FIG. 4 is an activity diagram illustrating a method of performing transfer learning in accordance with an embodiment.

FIG. 5 is a process flow diagram illustrating a method of improving or updating functionality on devices that use narrowband channels in accordance with some embodiments.

FIGS. 6 - 8 are activity diagram illustrating methods of updating neural networks on edge devices that have limited connectivity in accordance with various embodiments.

FIG. 9 is an activity diagram illustrating a method of updating a neural network on one or more edge devices without requiring transmission of neural network data in accordance with an embodiment.

FIG. 10 is an activity diagram illustrating a method of updating an edge device to account for sensor variations and faults in accordance with an embodiment.

FIG. 11 is an activity diagram illustrating a method of using information from proceeding edge devices in a chain of edge devices to update a neural network or otherwise improve or update the functionality of an edge device in accordance with an embodiment.

FIG. 12 is an activity diagram illustrating a method of updating a neural network to extend the mission of a deployed edge device in accordance with an embodiment.

FIGS. 13 A- 13 C are process flow diagrams illustrating methods of updating neural networks on edge devices that have limited connectivity in accordance with some embodiments.

FIG. 14 is a component diagram of server computing device suitable for implementing some embodiments.

DESCRIPTION

The various embodiments will be described in detail with reference to the accompanying drawings. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts. References made to particular examples and implementations are for illustrative purposes, and are not intended to limit the scope of the invention or the claims.

In overview, the various embodiments include a sensor-rich programmable artificial intelligence (AI) inference and compute platform that is suitable for deployment at the extreme network edge, from the oceans of earth to low earth orbit, geosynchronous orbit and deep space. The AI inference and compute platform, alternatively termed the AI inference engine or the AI engine, may use machine learning accelerators, neural network accelerators, convolutional neural network accelerators, neuromorphic accelerators or a combination thereof, or may contain solely general compute.

The word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any implementation described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other implementations.

The term “computing device” may be used herein to refer to any one or all of server computing devices, personal computers, laptop computers, tablet computers, edge devices, user equipment (UE), multimedia Internet enabled cellular telephones, smartphones, smart wearable devices (e.g., smartwatch, smart glasses, fitness tracker, clothes, jewelry, shoes, etc.), Internet-of-Things (IoT) devices (e.g., smart televisions, smart speakers, smart locks, lighting systems, smart switches, smart doorbell cameras or security systems, etc.), connected vehicles, and other similar devices that include a memory and programmable processor for providing the functionality described herein.

The term “edge device” may be used herein to refer to any one or all of computing devices, satellites, connected vehicles (trucks, cars, etc.), electric scooters, trains, trams, metros (which often only have connectivity for brief periods while in stations), aircraft, drones (based on land, in sea, or in the air), high-altitude balloons, smartphones, smart wearable devices, IoT devices, eMobility devices (e.g., electric scooters, electric bikes), robots, nanobots, and other similar computing systems, devices or objects that include a memory, a sensor, a processor, and communications circuity for communicating with computing devices at one or more centralized sites. The processor may be a programmable processor or a fixed programmed processor (e.g., a pre-programmed FPGA or an ASIC) with associated reconfigurable parameters stored in an associated memory. Edge devices are often resource-constrained devices that have limited processing, memory, battery and/or bandwidth resources.

The term “centralized site” may be used herein to refer to a control site that includes one or more computing devices (or “centralized devices”) that are configured to initiate, provision, store data on (e.g., collected data, data obtained from other sources, augmented data, etc.), enable labeling on, train, communicate with and/or control edge devices. For ease of reference and to focus the description on the relevant features or functionalities, some embodiments are described herein with reference to a “centralized site/device” on earth and one or more edge devices deployed in space. However, it should be understood that the described features and functionalities may be applicable to other types of edge devices, systems, configurations or deployments. As such, nothing in this application should be used to limit the claims or disclosures herein to a centralized site/device on earth and edge devices deployed in space unless expressly recited as such within the claims.

The term “AI edge device” may be used herein to refer to an edge device that is configured to perform AI operations locally on the device and/or to work in conjunction with other devices (e.g., another edge device, centralized site/device, etc.) that perform AI operations. For example, an AI edge device may be an edge device that includes an edge AI processor configured to perform “inference” and/or to otherwise deploy or use a neural network that utilizes or accomplishes machine learning locally on the device. As another example, an AI edge device may be configured to collect data (on which to action) on the edge device, send the collected data to a centralized site/device that performs inference to generate an overall inference result, receive the overall inference result from the centralized site/device, and perform an action based on the received overall inference result. An AI edge device may also be part of a group of edge devices (potentially of different types) that work in conjunction with one another to accomplish federated learning.

The term “AI model” may be used herein to refer to wide variety of information structures that may be used by a computing device to perform a computation or evaluate a specific condition, feature, factor, dataset, or behavior on a device. Examples of AI models include network models, neural network models, inference models, neuron models, classifiers, random forest models, spiking neural network (SNN) models, convolutional neural network (CNN) models, recurrent neural network (RNN) models, deep neural network (DNN) models, generative network models, and genetic algorithm models. In some embodiments, an AI model may include an architectural definition (e.g., the neural network architecture, etc.) and one or more weights (e.g., neural network weights, etc.).

The terms “collected data”, “acquired data”, “sensed data”, and “measured data” may all be used herein to refer to data acquired by an edge device (e.g., using its sensors, etc.).

The term “neural network” may be used herein to refer to an interconnected group of processing nodes (or neuron models) that collectively operate as a software application or process that controls a function of a computing device and/or generates an overall inference result as output. Individual nodes in a neural network may attempt to emulate biological neurons by receiving input data, performing simple operations on the input data to generate output data, and passing the output data (also called “activation”) to the next node in the network. Each node may be associated with a weight value that defines or governs the relationship between input data and output data. A neural network may learn to perform new tasks over time by adjusting these weight values. In some cases, the overall structure of the neural network and/or the operations of the processing nodes do not change as the neural network learns a task. Rather, learning is accomplished during a “training” process in which the values of the weights in each layer are determined. As an example, the training process may include causing the neural network to process a task for which an expected/desired output is known, comparing the activations generated by the neural network to the expected/desired output, and determining the values of the weights in each layer based on the comparison results. After the training process is complete, the neural network may begin “inference” to process a new task with the determined weights.

The term “inference” may be used herein to refer to a process that is performed at runtime or during execution of the software application program corresponding to the neural network. Inference may include traversing the processing nodes in the neural network along a forward path to produce one or more values as an overall activation or overall “inference result.”

The term “central inference” may be used herein to refer to inference that is performed at a centralized site/device (or in a server, the cloud, etc.) based on data collected on the edge device or at the edge of the network.

The term “edge-based inference” may be used herein to refer to inference that is performed on the edge device.

The term “deep neural network” may be used herein to refer to a neural network that implements a layered architecture in which the output/activation of a first layer of nodes becomes an input to a second layer of nodes, the output/activation of a second layer of nodes becomes an input to a third layer of nodes, and so on. As such, computations in a deep neural network may be distributed over a population of processing nodes that make up a computational chain. Deep neural networks may also include activation functions and sub-functions between the layers. The first layer of nodes of a multilayered or deep neural network may be referred to as an input layer. The final layer of nodes may be referred to as an output layer. The layers in-between the input and final layer may be referred to as intermediate layers.

The term “convolutional neural network” may be used herein to refer to a deep neural network in which the computation in at least one layer is structured as a convolution. A convolutional neural network may also include multiple convolution-based layers, which allows the neural network to employ a very deep hierarchy of layers. In convolutional neural networks, the weighted sum for each output activation is computed based on a batch of inputs, and the same matrices of weights (called “filters”) are applied to every output. These networks may also implement a fixed feedforward structure in which all the processing nodes that make up a computational chain are used to process every task, regardless of the inputs. In such feed-forward neural networks, all of the computations are performed as a sequence of operations on the outputs of a previous layer. The final set of operations generate the overall inference result of the neural network, such as a probability that an image contains a specific object (e.g., a person, cat, watch, edge, etc.) or information indicating that a proposed action should be taken.

The term “ensemble neural network” may be used herein to refer to a neural network that includes one or more sub-networks. The overall inference result from an ensemble neural network may be a weighted combination of the inference result of the individual neural networks in the ensemble. The processing nodes (e.g., neuron models, etc.) in an ensemble neural network are typically smaller than their corresponding single network equivalents that perform the same or similar prediction functions. As such, ensemble neural networks may be more suitable for receiving updates over a low-bandwidth channel than their non-ensemble counterparts.

The term “feature vector” may be used herein to refer to an information structure that represents or characterizes collected data (e.g., sensor data, etc.) or which represents or characterizes a specific factor, feature, condition, data point, or aspect of collected data. A feature vector may include one or more features and their corresponding feature values. A feature value may be a number or symbol that represents a collected data point. A feature value may be associated with a data type that identifies how a feature (or its feature value) should be measured, analyzed, weighted, or used. For example, a data type may identify a range of possible feature values, meanings of the values, operations that may be performed on those values, etc.

The term “classifier” may be used herein to refer to an AI model and/or information structures that may be used by a device processor to evaluate collected data or a specific feature (or factor, condition, data point, operation, component, etc.). For example, a classifier may include decision nodes (e.g., neural networks, decision stumps, boosted decision trees, etc.) that each include a weight value and a test question/condition suitable for evaluating the collected data. As a simplified example, a classifier may include a decision stump or neural network that evaluates the condition “is road surface roughness greater than 3.5 millimeters per meter (mm/m).” In this example, applying a feature vector that includes a “road surface roughness” feature having a feature value of “3” to the classifier may generate a result that indicates a “no” answer via a number, such as “0.”

A classifier may include multiple decision nodes and a feature vector may include multiple features. As such, applying a feature vector to a classifier may generate a plurality of answers to a plurality of different test conditions. Each of these answers may be represented by a numerical value. The device processor may multiply each of these numerical values with their respective weight value to generate a plurality of weighted answers. The device processor may then compute or determine a weighted average based on the weighted answers, compare the computed weighted average to one or more threshold values, and perform a responsive action (e.g., classify/label the collected data, etc.) based on the results of the comparison. For example, if the computed weighted average is “0.79” and the threshold value for “scooter-specific micromovement (shake)” is “0.75,” the device processor could determine that the collected dataset is suitable (or that it is not suitable) for use in training a neural network for a e-scooter edge device.

The term “ensemble classifier” may be used herein to refer to a group of classifiers that includes an initial classifier and one or more subsequent classifiers. Each classifier in the ensemble classifier may be a different type of classifier, may include different types of decision nodes, may implement different AI models, may focus on evaluating a different feature and/or may focus of evaluating a different aspect of the same feature.

In recent years, the concept of deploying neural networks to edge devices has become a feasible reality. However, as machine learning (ML) and artificial intelligence (AI) move to the edge, it is likely that there will be new challenges that emerge in relation to efficient and effective deployment and operation of neural networks in edge devices. Such challenges are particularly acute in systems or applications where bandwidth between the centralized site/device and the edge device is restricted, limited, intermittent, and/or non-reliable. Moreover, the available compute power at the edge device may be significantly less than that available at centralized site/device, meaning that neural networks suitable for central inference are not always suitable for edge-based inference. Not only is compute power a potential limitation at the edge, but available system electrical power is also typically limited at the edge, meaning that deployed edge devices may be required to use efficient, and potentially customized, neural networks in order to adhere to these limitations.

There are many examples of potential applications for artificial intelligence being implemented at the edge of the network, any or all of which may be implemented, facilitated, supported, enabled, allowed or used by the various embodiments. Some of these examples/applications relate to edge devices that operate at significant distances from the centralized site, such as satellites operating in space (e.g., low earth orbit, geosynchronous orbit, and deep space), submersible drones working on the seabed (e.g., searching for plane wreckage, conducting seismic surveys, oil exploration and extraction), or vehicle mounted IoT devices that continuously transit between communications networks (e.g., vehicle systems monitoring, driver attention and performance monitoring, or cargo monitoring). Consequently, these edge devices may have very limited bandwidth capabilities (e.g., very low throughput, very high round trip time, high latency, etc.).

Further examples relate to edge devices that do not have reliable connections to the centralized site, but yet they need to be able to continue operating autonomously when they experience connectivity failures or transit into/through connectivity blackspots, such as drones performing search and rescue operations for humans in hazardous environments (e.g., in partially collapsed buildings after an earthquake, in deep cave systems, in buildings where there are active fires, explosions, chemical leaks, and radiation leaks).

There are also likely to be commercial applications (e.g., maintenance robots operating in sewers, mining drones operating deep below the surface of the earth, delivery drones delivering online purchases of food and goods), law enforcement applications (e.g., using satellite imaging to detect illegal drug growing and processing facilities, monitoring immigration and smuggling in remote locations) and military applications (e.g., airborne drones must be able to continue operating when they lose their connections to the centralized site due to signaling jamming or due to physical damage, soldiers wearing or using smart equipment must be able to depend on it to function in autonomous situations). In addition, there may be applications relating to assisted and independent living for ill or elderly people (e.g., smart glasses may be able to detect that a cancer patient takes the right medication at the right times of the day, a smart watch may be able to “recall” important notes or reminders throughout the day for a dementia sufferer).

In any or all of the above examples/applications, the edge devices may be extremely small or resource-constrained, and thus not have the power necessary to establish and maintain a connection to a centralized site/device at all times. For example, edge devices in the form of small robots operating within a human's blood stream (called “nanobots”) may only be able to establish a connection to a centralized site/device when they are in a large vein or artery that is close to the surface of the human.

In some applications, the edge device may be connected to a high-performance sensor that generates more data than can be feasibly sent to the centralized site/device (or downlinked, etc.) due to the deployment scenario (such as hyperspectral sensors on satellites). In these cases, a neural network implemented on the edge device may enable digestion of the data at the network edge. The overall inference result of the neural network, along with optionally some of the raw data, may be sent to the centralized site/device, but importantly the volume of data to be sent is much less than for non-AI solutions in which the entire set of hyperspectral data needs to be sent to the centralized site/device. As sensors gain in resolution (e.g., in any one or any combination of spatial, spectral and temporal resolution, etc.) it becomes increasingly important to digest the data on the edge device using AI (e.g., by performing edge-based inference, etc.).

The various embodiments include components (e.g., edge devices, etc.) that are configured to perform edge-based inference so as to overcome the above described challenges and limitations.

The benefits of edge-based inference over central inference include a reduction in latency (to actionable event), a reduction in required transmission bandwidth (compared to centrally based inference), and an increase in data security (e.g., because personal, sensitive, confidential, or secretive data is not required to be transferred off the edge device, etc.).

Edge-based inference may reduce or eliminate many of the data

transmissions associated with central inference, and thus reduce the required transmission bandwidth. The reduction in the required transmission bandwidth may be a direct result of where the inference is performed. For example, central inference may require collecting data (on which to action) from the edge device, sending the collected data to the centralized site/device that performs the inference to generate the overall inference result, and sending the overall inference result from the centralized site/device to the edge device so that it may analyze the overall inference result and/or perform an action based on the overall inference result. In contrast, edge-based inference may include collecting data on the edge device and performing inference locally on the edge device to generate the overall inference result. The edge device may send the overall inference result to the centralized site/device and/or work in conjunction with the centralized site/device to analyze or use the overall inference result. Alternatively, the edge device may perform an action on the inference result directly without transmitting the overall inference result back to the centralized site/device. In all these examples, edge-based inference may reduce the required transmission bandwidth by eliminating or significantly reducing the amount of data that is communicated between the edge device and the centralized site/device.

For example, edge-based inference could be used in an obstacle avoidance application in autonomous vehicle navigation in which the important action is that the vehicle navigates to avoid the obstacle. After this action (navigating to avoid the obstacle), there is no need to send either the collected sensor data or the overall inference results back to the centralized site/device. Since little or no data is transmitted to the centralized site/device, using edge-based inference could significantly reduce the required transmission bandwidth of the obstacle avoidance application, allowing it to be deployed on smaller, more remote, or more resource constrained devices.

Some of the examples and applications above may include the edge device sending the overall inference result to the centralized site/device. Edge-based inference could also reduce the required transmission bandwidth for these examples and applications as well. This is because sending the overall inference result requires significantly less bandwidth and/or power than sending the collected data (as is often required for central inference). For example, image/video data (e.g., a retinal image, a video stream, etc.) collected in an edge device may include megabytes, gigabytes or terabytes of raw data, whereas the overall inference result may be a few bytes representing the probability that person associated with a retinal image (e.g., the collected image/video data) may have early onset of diabetic retinopathy.

Edge-based inference may reduce the required transmission bandwidth by eliminating or significantly reducing the volume of the data that is communicated (e.g., by an order of magnitude or more) between the edge device and the centralized site/device. This reduction in transmission bandwidth may allow for the deployment and use of edge devices that do not have high bandwidth communication resources.

The lack of high bandwidth communication resources could limit an edge device's ability to receive updates (e.g., via over the air updates, etc.). For example, an application update may require updating the entire neural network, but neural networks typically have a large memory footprint (e.g., in the order of megabytes, but possibly up to hundreds of megabytes or more). As a result, the bandwidth and power required to transmit or receive the entire neural network could prevent edge devices that do not have high bandwidth communication resources from receiving updates. Some embodiments may eliminate or reduce the amount of data transmissions required to update an edge device, allowing edge devices that lack high bandwidth communication resources to receive updates after deployment in the field.

As mentioned above, using conventional solutions, the lack of high bandwidth communication resources in an edge device may limit the device's ability to receive updates. A particular case of the above problem occurs for updates that include a new neural network (or new model), such as a neural network produced by retraining a previously deployed neural network with data collected from one or more edge devices. In this case, the network architecture is unchanged by the update. All that is changed is the weights within the network.

An example of this is for an edge device that is deployed to a location from which data has not previously been collected, and in which the initial neural network is trained based on data collected from an alternative location and/or synthetic data and/or augmented data. That is, because data was not available from the deployment location, the training was conducted with data that is only representative of the final deployment location. The edge device is deployed pre-loaded with an initial neural network that is likely to be suboptimal for the final deployment location. Once the edge device is in its final location, the device may collect data from the actual sensors in the deployment location and perform edge-based inference. However, due to the difference between the training data used to train the network and the runtime data acquired at the final edge location with the deployed sensor (and other differences due to environmental effects, etc.), such inference operations may produce inaccurate (or not optimum) results.

In the above example, the accuracy of the inference operations could be improved by sending data captured in the deployment location to the centralized site/device for additional training. The centralized site/device could perform additional training (possibly by transfer techniques) on the neural network to produce a new set of neural network weights that are more accurate or optimal for the data acquired at the edge device location, and send the edge device a new neural network that includes the new network weights. The edge device could then use the new neural network to update the pre-loaded initial neural network.

An example of where the above scenario occurs is for edge devices deployed on satellites, particularly in the field of earth observation. When an earth observation sensor (optical, Synthetic Aperture Radar (SAR) etc.) is integrated into a satellite prior to launch, in-orbit data for this sensor may not already be available. This is particularly the case for new sensors that have not flown in orbit previously. Satellite imagery captured by other ‘similar’ sensors could be used (e.g., after augmentation to map the known characteristics of the sensor to the imagery, etc.) to generate data that mimics the data expected to be captured by the sensor in orbit. The neural network is then trained with this data. This data is typically plentiful. The edge device is then pre-loaded with the resulting neural network prior to launch. Once in orbit, real data captured by the sensor may be sent to the centralized site/device on the ground, where it may be used to update the neural network via additional training cycles. Typically, the amount of such data sent from the satellite to the centralized site/device is much smaller than the original corpus of training data. Once the additional training cycles have been completed, the updated weights of the neural network are sent (e.g., uplinked, uploaded, etc.) to the edge device on the orbiting satellite. Uplink bandwidth (upload bandwidth) is often even more restricted than downlink bandwidth (download bandwidth) for space applications (typically there is an asymmetric data link to the satellite), hence minimizing the size of the update is important.

Another example of a deployment where neural network updates are required, and in which the available bandwidth over which to deploy this neural network is typically limited, is for IoT devices in the field. A further consideration here is that the power required to transmit data can be significant for deployed IoT devices, which are often powered either by battery or solar cell, or both. Reducing the amount of data that is transmitted or received may extend the operational life of the edge device due to power savings.

There are two considerations when examining methods for efficient updating of deployed neural networks. Firstly, the scenario of a neural network that has already been deployed must be considered (termed ‘efficient model update methods’). In this scenario, the task is to develop techniques to reduce the size of the updates or volume of data transmissions. Secondly, the scenario of an application that has not yet been deployed should be considered (termed “model architectures for efficient updates”). In this case the task is to provide network architecture design guidelines to enable the selection of a network architecture that is inherently suited to efficient updates in the future.

Edge-based inference may only require that metadata be transmitted back to the centralized site/device, cloud or users, vastly reducing the time and bandwidth required to generate alerts particularly where video or high resolution image data is being used as input to the inference engine. These alerts may be notifications of forest fires, blight conditions, or destructive insect promulgations. The alerts may be used as inputs for further actionable events that could occur automatically and without any human intervention. These actionable events may be navigation controls for autonomous vehicles or satellites (obstacle avoidance, uncooperative object grasping, collision avoidance), re-orientation or re-configuration of sensors on the edge device (re-calibrating sensors based on environmental changes or wear and tear, re-pointing of satellite earth observation sensors towards ground features of interest or away from cloud-obscured regions), or changing the power state of the device (throttling of device throughput (reducing frame rate or inference rate) to conserve power when certain conditions are detected).

The edge-based AI platform may also run individual networks that have been programmed into the platform before launch time, but more interestingly the platform may be updated in the field with completely new networks, or existing networks may be upgraded, or indeed parallel networks may be combined to form ensemble neural networks that improve accuracy.

As artificial intelligence solutions become enabled for space applications, methods for tuning and improving the neural networks in-orbit may become critical to fully realizing artificial intelligence's potential in space. Continuous (or intermittent) neural network tuning may improve the neural network's effectiveness, may enable the neural network to adapt to the specific sensor(s) of the deployment hardware, and may enable the neural network to adapt to changes in sensor performance over time. A challenge for in-orbit applications of artificial intelligence is to perform these in-flight updates efficiently in the context of the limited bandwidth of the uplink path. A second challenge is to minimize the frequency with which it is required to update neural networks in-flight via the ground uplink channel.

FIG. 1 A illustrates a system 100 that includes edge devices 110 a, 110 b that could be configured in accordance with the embodiments. In the example illustrated in FIG. 1 A , the system 100 includes edge devices 110 a, 110 b that are satellites in space, and a centralized site/device 120 that is connected to a series of transmission sites 130 a, 130 b dispersed around the world to provide suitable coverage.

FIG. 1 B illustrates another system 150 that includes edge devices 110 that could be configured in accordance with the embodiments. In the example illustrated in FIG. 1 B , the system 150 includes various different types of edge devices 110 (i.e., a network of heterogeneous edge devices). These heterogenous devices may be located underground, underwater (submersibles), on land (robots, e-mobility devices, mobile phones, IoT devices, insect traps), on the sea (watercraft, buoys), in the lower atmosphere (drones, planes), in the upper atmosphere (high altitude balloons), in earth orbit (satellites) or in deep space (exploration missions). Data collected from these edge devices 110 may be transmitted to the centralized site/device 120 , from where it can be stored, processed, labelled, delivered, served, queried, analyzed, and used for training. In the AI context, this data may require some level of labelling before training can be initiated. Human-in-the-loop training may be accomplished via a crowd sourced labelling API. Training at the centralized site/device 120 may use general-purpose graphics processing units (GPGPUs) to enhance throughput.

FIGS. 2 A and 2 B illustrate components in an edge device 110 that could be configured in accordance with the various embodiments. In the example illustrated in FIG. 2 A , the edge device 110 includes a printed circuit board (PCB) 202 that includes one or more processors 204 , neural networks (NN) 206 , a controller 208 , storage memory 210 , and sensors 212 . In the example illustrated in FIG. 2 B , the edge device 110 includes sensors 212 , FPGA logic components 250 , 256 , a select multiplexer (MUX) component 252 , a vision processing unit 254 , a spiking neural network component 258 , a global positioning system (GPS) component 270 , additional sensors 272 , a classification component 274 , mass storage 276 , an encryption component 278 , a compression component 280 , a modem 282 , solar panels 284 , batteries 286 , a power management component 288 , and a fault detection component 290 , any or all of which may be implemented in software, hardware, or a combination thereof.

The sensor(s) 212 may capture data about the internal or external environment of the device, or about a remote environment that is targeted by the sensor(s) 212 . In some embodiments, the multiple sensors 212 may capture a variety of data types, or orthogonal data. The combination of the data from various sensor types may be processed by the edge device 110 (or a processor 204 in the edge device 110 , AI processor, etc.) in order to determine a more valuable result (e.g., a heads-up-display used by a fire-fighter may combine multiple types of captured data from multiple sensing paradigms such as visible spectrum, short-wave infrared, thermal infrared, night-vision, ultrasonic sounds to create a more holistic combined result relating to the immediate environment of the firefighter). This is the sensor fusion paradigm. The controller 208 may be configured to manage the system and its interface to the centralized site/device, and to perform an action based on the produced metadata. Storage memory 210 on the edge device 110 may enable autonomous operation, by storing sensed data and/or metadata for later communication to a centralized site/device. The storage memory 210 may also allow the edge device 110 to store the neural network(s) 206 locally on the device.

FIG. 3 illustrates a method 300 for deploying an edge device 110 that is suitable for performing edge-based inference in accordance with some embodiments. Method 300 may be performed by one or more processors in one or more computing devices (e.g., centralized site/device 120 , server computing device, edge device 110 , etc.).

In block 302 , a device processor (e.g., in a centralized site/device 120 , etc.) may collect data that is representative of the data that will be available to the edge device 110 when it is deployed (typically this is sensor data, e.g., images for an image sensor). In block 304 , the processor may label or tag the data, such as by performing or implementing a human-in-the-loop technique. For the example task of object detection in images, labelling in block 304 may take the form of identifying and/or highlighting positive examples of the object to be detected within the images in the dataset.

In block 306 , the processor may (randomly) split the data into training, test, and validation data subsets. In block 308 , the processor may use the training and test datasets to train neural networks (or models). The training dataset may be augmented via mathematical operations, and this augmented data may be added to the training dataset to increase the size of the dataset whilst maintaining its relevance. It may also be increased by using a synthetically generated dataset, e.g., by synthesizing data from a mathematical model. The test dataset may also be used to evaluate the trained neural network (e.g., to trade-off against other neural networks).

Once the neural network has been trained and tested, it may be deployed to the edge device(s) 110 in block 310 . This may involve optimization (e.g., pruning) and conversion of the neural network to target specific hardware of the edge device 110 .

In block 312 , a processor in the edge device 110 may store the neural network in its local memory. In block 314 , the edge device 110 may be deployed to the field, whereupon the neural network may be used in block 316 to perform inference on sensor data on the edge device 110 . It may not be required to modify the neural network once the edge device 110 is deployed. The more similar the sensor data received by the edge device 110 is to the dataset on which the neural network was trained, the more effective the inference (or edge device 110 ) is.

Deploying edge devices 110 in harsh or inaccessible locations, or in deployments that are expensive due to their locations, may result in limited effectiveness of the device (or the inference operations) when it is initially operated and implementing its pretrained neural networks. Due to the relatively few devices deployed prior to any new deployment, there may be a lack of available training data. For example, earth observation sensors on satellites are only infrequently launched into orbit due to the large cost of such missions, and therefore for any particular earth observation sensor there may be very limited data available at the centralized site/device to facilitate training (few devices in orbit therefore limited data available). Further, the data from the limited deployments that is available on the ground (to the centralized site/device) may be sensitive (military), or may require extensive processing (delay in availability), and in general may not be shared between organizations or made available for training.

E-scooters is an example application deployment for which there may be a lack of available training data due to the relatively recent deployment of e-scooters on public roads, and the lack of video data capture capability on these devices. In this case it is not the sensors that are new, nor the difficulty in acquiring the data, but rather that the application itself is new and that it takes time to acquire large and representative datasets for new applications. Additionally, and importantly, due to enhancements in technology, newly deployed edge devices 110 may be more advanced and therefore their neural networks may be most effective if they have training data that is specific to the sensor at the training stage. Examples of such cases are sensors that have higher resolutions, have greater sensitivities, have higher accuracies or that have completely new sensing modalities.

Improved or optimum inference performance may be obtained when the neural network is trained on data from the same sources as those which generate data during inference. For example, only very recently have high spectral resolution hypersp

CLAIMS

Claims ( 15 )

What is claimed is:

1 . A method of updating a neural network on an edge device that has low-bandwidth uplink capability, comprising:

initially training by a processor in a centralized site/device, the neural network to create an initially trained neural network; sending by the processor, the initially trained neural network to the edge device; receiving by the processor, neural network information from the edge device, the received neural network information including at least a portion of at least one or more of a dataset, an activation, or an overall inference result collected or generated in the edge device; using by the processor, the received neural network information to create an updated trained neural network by updating all or a part of the initially trained neural network; generating by the processor, a neural network difference model by comparing the updated trained neural network to the initially trained neural network, wherein the neural network difference model comprises a patch that identifies the differences between the updated trained neural network and the initially trained neural network via one of:

layer freezing using a minimum size technique;

layer freezing using a minimum delta technique;

weights freezing using the minimum size technique; or

weights freezing using the minimum delta technique;

wherein:

the minimum size technique comprises selecting or determining the layers to be frozen based on an aggregate measure of the value of the weights in a layer;

the minimum delta technique comprises selecting or determining the layers to be frozen based on a measure of the aggregate changes in the weight values, as determined by comparing the layer weight values before and after a re-training cycle; and

sending by the processor, the neural network difference model to the edge device.

2 . The method of claim 1 , wherein sending the initially trained neural network to the edge device comprises sending the initially trained neural network to an edge device that has been deployed.

3 . The method of claim 1 , wherein generating the neural network difference model by comparing the updated trained neural network to the initially trained neural network comprises determining one or more neural network layers or one or more neural network weights of the one or more neural network layers to freeze based on a mean of activations of layers in the neural network.

4 . The method of claim 1 , further comprising:

receiving, by the edge device, the initially trained neural network; collecting, by the edge device, the dataset from sensors of the edge device; applying, by the edge device, the collected dataset as inputs to the received initially trained neural network to generate activations and the overall inference result; storing, by the edge device, at least a portion of at least one or more of the collected dataset, the generated activations or the overall inference result in a memory of the edge device; and sending, by the edge device, the neural network information that includes at least a portion of at least one or more of the collected dataset, the generated activations or the overall inference result to the centralized site/device.

5 . The method of claim 4 , further comprising:

receiving, by the edge device, the neural network difference model; generating, by the edge device, an updated neural network based on the received initially trained neural network and the received neural network difference model; and applying a second dataset as input to the updated neural network to generate second inference results.

6 . The method of claim 1 , wherein:

initially training the neural network comprises:

collecting training data from one or more of a plurality of edge devices;

labelling the collected training data;

selecting two or more lightweight neural networks;

generating an ensemble based on the selected neural networks; and

using the labelled training data to train the ensemble; and

sending the trained neural network to the edge device comprises:

sending the trained ensemble and an ensemble aggregation function to the edge device.

7 . The method of claim 6 , wherein using the received neural network information to create an updated trained neural network by updating all or a part of the initially trained neural network comprises adding a neural network to the trained ensemble.

8 . The method of claim 6 , wherein using the received neural network information to create an updated trained neural network by updating all or a part of the initially trained neural network comprises:

updating the ensemble aggregation function based on a result of analyzing the received neural network information; and updating all or a part of the initially trained neural network based on the updated ensemble aggregation function.

9 . The method of claim 4 , wherein:

receiving the initially trained neural network comprises receiving a trained ensemble; and applying the collected dataset as inputs to the received initially trained neural network to generate the activations and the overall inference result comprises applying the collected dataset as inputs to the received trained ensemble to generate the activations and the overall inference result.

10 . The method of claim 1 , wherein initially training the neural network comprises generating a stratified neural network that includes large data volume parts and small data parts.

11 . The method of claim 10 , wherein sending the neural network difference model to the edge device comprises sending the small data parts of the stratified neural network to the edge device.

12 . The method of claim 10 , wherein generating the stratified neural network that includes the large data volume parts and the small data parts comprises:

generating the stratified neural network to include:

a large data volume part that include a feature identification layer; and

a small data part that includes a fully connected layer.

13 . The method of claim 10 , wherein generating the stratified neural network that includes the large data volume parts and the small data parts comprises:

generating the stratified neural network to include:

large data volume parts that include multiple partial layers that are not cross-connected; and

small data parts that include cross-connected weights between the multiple partial layers in the large data volume parts, or generating the stratified neural network to include:

large data volume parts that include layers with a higher numerical precision; and

small data parts that include layers with a lower numerical precision.

14 . A centralized site/device, comprising

a processor is configured with processor-executable instructions to perform operations comprising:

initially training a neural network;

sending the initially trained neural network to an edge device that has low-bandwidth uplink capability;

receiving neural network information from the edge device, the received neural network information including at least a portion of at least one or more of a dataset, an activation, or an overall inference result collected or generated in the edge device;

using the received neural network information to create an updated trained neural network by updating all or a part of the initially trained neural network;

generating a neural network difference model by comparing the updated trained neural network to the initially trained neural network, wherein the neural network difference model comprises a patch that identifies the differences between the updated trained neural network and the initially trained neural network via one of:

layer freezing using a minimum size technique; layer freezing using a minimum delta technique;

weights freezing using the minimum size technique; or weights freezing using the minimum delta technique; wherein:

the minimum size technique comprises selecting or determining the layers to be frozen based on an aggregate measure of the value of the weights in a layer;

the minimum delta technique comprises selecting or determining the layers to be frozen based on a measure of the aggregate changes in the weight values, as determined by comparing the layer weight values before and after a re-training cycle; and

sending the neural network difference model to the edge device.

15 . A non-transitory computer readable storage medium having stored thereon processor-executable software instructions configured to cause a processor in a centralized site/device to perform operations for updating a neural network on an edge device that has low-bandwidth uplink capability, the operations comprising:

initially training a neural network; sending the initially trained neural network to an edge device that has low-bandwidth uplink capability; receiving neural network information from the edge device, the received neural network information including at least a portion of at least one or more of a dataset, an activation, or an overall inference result collected or generated in the edge device; using the received neural network information to create an updated trained neural network by updating all or a part of the initially trained neural network; generating a neural network difference model by comparing the updated trained neural network to the initially trained neural network, wherein the neural network difference model comprises a patch that identifies the differences between the updated trained neural network and the initially trained neural network via one of:

layer freezing using a minimum size technique; layer freezing using a minimum delta technique;

weights freezing using the minimum size technique; or weights freezing using the minimum delta technique;

wherein:

the minimum size technique comprises selecting or determining the layers to be frozen based on an aggregate measure of the value of the weights in a layer;

the minimum delta technique comprises selecting or determining the layers to be frozen based on a measure of the aggregate changes in the weight values, as determined by comparing the layer weight values before and after a re-training cycle; and

sending the neural network difference model to the edge device.

US19/293,758

2019-02-22

2025-08-07

Systems and Methods for Deploying and Updating Neural Networks at the Edge of a Network

Pending

US20250363357A1

( en )

Priority Applications (1)

Application Number

Priority Date

Filing Date

Title

US19/293,758

US20250363357A1

( en )

2019-02-22

2025-08-07

Systems and Methods for Deploying and Updating Neural Networks at the Edge of a Network

Applications Claiming Priority (3)

Application Number

Priority Date

Filing Date

Title

US201962809353P

2019-02-22

2019-02-22

US16/797,422

US20200272899A1

( en )

2019-02-22

2020-02-21

Systems and Methods for Deploying and Updating Neural Networks at the Edge of a Network

US19/293,758

US20250363357A1

( en )

2019-02-22

2025-08-07

Systems and Methods for Deploying and Updating Neural Networks at the Edge of a Network

Related Parent Applications (1)

Application Number

Title

Priority Date

Filing Date

US16/797,422

Continuation

US20200272899A1

( en )

2019-02-22

2020-02-21

Systems and Methods for Deploying and Updating Neural Networks at the Edge of a Network

Publications (1)

Publication Number

Publication Date

US20250363357A1

true

US20250363357A1 ( en )

2025-11-27

Family

ID=69742647

Family Applications (2)

Application Number

Title

Priority Date

Filing Date

US16/797,422

Abandoned

US20200272899A1

( en )

2019-02-22

2020-02-21

Systems and Methods for Deploying and Updating Neural Networks at the Edge of a Network

US19/293,758

Pending

US20250363357A1

( en )

2019-02-22

2025-08-07

Systems and Methods for Deploying and Updating Neural Networks at the Edge of a Network

Family Applications Before (1)

Application Number

Title

Priority Date

Filing Date

US16/797,422

Abandoned

US20200272899A1

( en )

2019-02-22

2020-02-21

Systems and Methods for Deploying and Updating Neural Networks at the Edge of a Network

Country Status (2)

Country

Link

US

( 2 )

US20200272899A1

( en )

EP

( 1 )

EP3699825B1

( en )

Families Citing this family (95)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US12509037B2

( en )

*

2014-11-13

2025-12-30

Faiveley Transport Italia S.P.A.

Vehicle braking assembly

EP3639530B1

( en )

2017-06-15

2022-03-02

Telefonaktiebolaget LM Ericsson (PUBL)

Hardware platform based on fpga partial reconfiguration for wireless communication device

US11386339B2

( en )

*

2019-02-26

2022-07-12

Global Elmeast Inc.

Artificial intelligence delivery edge network

US20200293860A1

( en )

*

2019-03-11

2020-09-17

Infineon Technologies Ag

Classifying information using spiking neural network

US11169532B2

( en )

*

2019-03-26

2021-11-09

Intel Corporation

Computer-assisted (CA)/autonomous driving (AD) vehicle inference model creation

WO2021029891A1

( en )

2019-08-14

2021-02-18

Google Llc

Communicating a neural network formation configuration

EP4014166A1

( en )

2019-08-14

2022-06-22

Google LLC

Base station-user equipment messaging regarding deep neural networks

RU2739483C1

( en )

2019-09-04

2020-12-24

ГУГЛ ЭлЭлСи

Feedback on neural network formation configuration for wireless transmissions

US11880763B2

( en )

*

2019-09-06

2024-01-23

Intel Corporation

Partially-frozen neural networks for efficient computer vision systems

US10957031B1

( en )

*

2019-09-06

2021-03-23

Accenture Global Solutions Limited

Intelligent defect detection from image data

US11711310B2

( en )

2019-09-18

2023-07-25

Tweenznet Ltd.

System and method for determining a network performance property in at least one network

US12075346B2

( en )

2019-10-31

2024-08-27

Google Llc

Determining a machine-learning architecture for network slicing

US11716338B2

( en )

*

2019-11-26

2023-08-01

Tweenznet Ltd.

System and method for determining a file-access pattern and detecting ransomware attacks in at least one computer network

US11886991B2

( en )

2019-11-27

2024-01-30

Google Llc

Machine-learning architectures for broadcast and multicast communications

US11689940B2

( en )

2019-12-13

2023-06-27

Google Llc

Machine-learning architectures for simultaneous connection to multiple carriers

WO2021125063A1

( en )

*

2019-12-18

2021-06-24

ソニーグループ株式会社

Information processing device, information processing method, program, and moving body

IT202000001462A1

( en )

*

2020-01-24

2021-07-24

St Microelectronics Srl

EQUIPMENT TO OPERATE A NEURAL NETWORK, CORRESPONDING PROCEDURE AND IT PRODUCT

US11537885B2

( en )

*

2020-01-27

2022-12-27

GE Precision Healthcare LLC

Freeze-out as a regularizer in training neural networks

US20230093630A1

( en )

*

2020-03-17

2023-03-23

Interdigital Ce Patent Holdings

System and method for adapting to changing constraints

US11586958B2

( en )

*

2020-04-06

2023-02-21

Micron Technology, Inc.

Apparatuses, systems, and methods for machine learning using on-memory pattern matching

US11676052B2

( en )

*

2020-04-15

2023-06-13

Micron Technology, Inc.

Apparatuses and methods for inference processing on edge devices

US12572778B2

( en )

*

2020-06-01

2026-03-10

Nvidia Corporation

Technique to perform neural network architecture search with federated learning

EP4140170B1

( en )

2020-06-05

2025-11-12

Google LLC

End-to-end deep neural network adaptation for edge computing

JP2022007168A

( en )

*

2020-06-25

2022-01-13

富士通株式会社

Learning program, learning method and information processing apparatus

US11551099B1

( en )

*

2020-06-27

2023-01-10

Unicorn Labs Llc

Smart sensor

US11663472B2

( en )

2020-06-29

2023-05-30

Google Llc

Deep neural network processing for a user equipment-coordination set

KR20220010927A

( en )

*

2020-07-20

2022-01-27

삼성전기주식회사

edge artificial intelligence module and method for upgrading weight of edge artificial intelligence module

US20220076133A1

( en )

*

2020-09-04

2022-03-10

Nvidia Corporation

Global federated training for neural networks

US20230244952A1

( en )

*

2020-09-04

2023-08-03

Qualcomm Incorporated

Interface for over the air model aggregation in federated system

CN112101250B

( en )

*

2020-09-18

2022-07-15

电子科技大学

Method for detecting offshore ship target based on context semantic perception

US20220092423A1

( en )

*

2020-09-21

2022-03-24

International Business Machines Corporation

Input ordering neural network decomposition

KR20230079358A

( en )

*

2020-09-29

2023-06-07

소니 세미컨덕터 솔루션즈 가부시키가이샤

Information processing system and information processing method

US11645111B2

( en )

*

2020-10-23

2023-05-09

International Business Machines Corporation

Managing task flow in edge computing environment

WO2022093380A1

( en )

2020-10-28

2022-05-05

Google Llc

Adaptive phase-changing device sharing and handover

US11300652B1

( en )

*

2020-10-30

2022-04-12

Rebellion Defense, Inc.

Systems and methods for generating images from synthetic aperture radar data using neural networks

US11769019B1

( en )

*

2020-11-19

2023-09-26

Amazon Technologies, Inc.

Machine translation with adapted neural networks

US20240095581A1

( en )

*

2020-11-24

2024-03-21

Nippon Telegraph And Telephone Corporation

Processing method, processing system, and processing program

CN112486686B

( en )

*

2020-11-30

2025-02-18

之江实验室

Customized deep neural network model compression method and system based on cloud-edge collaboration

KR102320706B1

( en )

*

2020-11-30

2021-11-02

(주)비스텔리젼스

Method for setting model threshold of facility monitoring system

CN112634927B

( en )

*

2020-12-03

2022-07-29

电子科技大学

Short wave channel voice enhancement method

CN114326781B

( en )

*

2020-12-08

2024-03-15

北京航空航天大学

Fully distributed missile formation collaborative adaptive fault-tolerant compensation controller

CN112532746B

( en )

*

2020-12-21

2021-10-26

北京邮电大学

Cloud edge cooperative sensing method and system

CN112529167B

( en )

*

2020-12-25

2024-05-14

东云睿连(武汉)计算技术有限公司

Neural network interactive automatic training system and method

US20220237415A1

( en )

*

2021-01-28

2022-07-28

International Business Machines Corporation

Priority-based, accuracy-controlled individual fairness of unstructured text

KR20230142469A

( en )

*

2021-02-03

2023-10-11

소니그룹주식회사

Server device, creation method, creation method of electronic device, database creation method, electronic device

CN113033605B

( en )

*

2021-02-08

2024-09-24

广东迅科动力科技有限公司

Motor fault judging method and device, terminal equipment and computer storage medium

DE102021104738A1

( en )

2021-02-26

2022-09-01

Ford Global Technologies, Llc

Method for operating a motor vehicle

CN113792883B

( en )

*

2021-03-03

2024-04-16

京东科技控股股份有限公司

Model training method, device, equipment and medium based on federal learning

EP4307177A4

( en )

*

2021-03-10

2024-08-21

Sony Semiconductor Solutions Corporation

INFORMATION PROCESSING APPARATUS, INFORMATION PROCESSING SYSTEM, INFORMATION PROCESSING METHOD AND RECORDING MEDIUM

CN113099408A

( en )

*

2021-03-15

2021-07-09

西安交通大学

Simulation-based data mechanism dual-drive sensor node deployment method and system

CN113044458B

( en )

*

2021-03-25

2022-05-27

北京物资学院

A method and system for dynamic task assignment of a logistics robot

CN113190345B

( en )

*

2021-03-26

2023-12-05

中国科学院软件研究所

Method and device for deploying neural network model oriented to software defined satellite

US20220335312A1

( en )

*

2021-04-15

2022-10-20

EMC IP Holding Company LLC

System and method for distributed model adaptation

US20220366220A1

( en )

*

2021-04-29

2022-11-17

Nvidia Corporation

Dynamic weight updates for neural networks

CN113139500B

( en )

*

2021-05-10

2023-10-20

重庆中科云从科技有限公司

Smoke detection method, system, medium and equipment

CN115546821A

( en )

*

2021-06-30

2022-12-30

华为技术有限公司

A system and device for configuring a neural network model in an edge server

CN113609912B

( en )

*

2021-07-08

2023-06-20

西华大学

A Fault Diagnosis Method of Transmission Network Based on Multi-source Information Fusion

KR20230013995A

( en )

*

2021-07-20

2023-01-27

삼성전자주식회사

Method and apparatus for generating process simulation model

CN113312183B

( en )

*

2021-07-30

2021-12-21

北京航空航天大学杭州创新研究院

Edge Computing Methods for Deep Neural Networks

CN113993136B

( en )

*

2021-08-04

2023-03-21

东华大学

Redundancy deployment method of application service in edge environment

KR102658912B1

( en )

*

2021-09-24

2024-04-18

한국과학기술원

Energy-efficient Retraining Method of Generative Neural Network for Domain-Specific Optimization

US20240365136A1

( en )

*

2021-10-18

2024-10-31

Qualcomm Incorporated

Reduced capability machine learning with assistance

EP4181081B1

( en )

*

2021-11-16

2024-02-28

Tata Consultancy Services Limited

Energy efficient hierarchical snn architecture for classification and segmentation of high-resolution images

WO2023097022A1

( en )

*

2021-11-23

2023-06-01

Strong Force Ee Portfolio 2022, Llc

Ai-based energy edge platform, systems, and methods

US20230177404A1

( en )

*

2021-12-07

2023-06-08

Ford Global Technologies, Llc

Adaptive model pruning to improve performance of federated learning

US12407808B2

( en )

*

2022-01-20

2025-09-02

Motional Ad Llc

Performance verification of an image sensor mounted to a vehicle

US12107665B2

( en )

*

2022-01-26

2024-10-01

Microsoft Technology Licensing, Llc

Generating model update data at satellite

CN114462623B

( en )

*

2022-02-10

2023-05-26

电子科技大学

Data analysis method, system and platform based on edge calculation

CN116776986A

( en )

*

2022-03-07

2023-09-19

维沃移动通信有限公司

Model accuracy determination method, device and network side equipment

CN114841365A

( en )

*

2022-04-18

2022-08-02

拓元(广州)智慧科技有限公司

Artificial intelligence platform with low training cost and use method thereof

CN114912581A

( en )

*

2022-05-07

2022-08-16

奇安信科技集团股份有限公司

Training method and device for detection model, electronic equipment and storage medium

CN115146775B

( en )

*

2022-07-04

2023-05-23

同方威视技术股份有限公司

Edge device reasoning acceleration method, device and data processing system

US20240028870A1

( en )

*

2022-07-22

2024-01-25

Capital One Services, Llc

Selective reporting of machine learning parameters for federated learning

TWI813419B

( en )

*

2022-08-18

2023-08-21

瑞昱半導體股份有限公司

Neural network system and operation method for neural network system

US12423579B2

( en )

2022-09-26

2025-09-23

International Business Machines Corporation

Reduction of data transmission and data storage using neural network technology

US20240169261A1

( en )

*

2022-11-10

2024-05-23

Lg Electronics Inc.

Method and device for domain generalized incremental learning under covariate shift

US12608928B1

( en )

*

2022-11-14

2026-04-21

Samsara Inc.

On-device machine learning calibration

CN115856873B

( en )

*

2022-11-15

2023-11-07

大连海事大学

Bank-based AIS signal credibility judging model, method and device

US20240169698A1

( en )

*

2022-11-23

2024-05-23

Logitech Europe S.A.

Object detection using artificial intelligence

CN118158737A

( en )

*

2022-12-05

2024-06-07

维沃移动通信有限公司

Information transmission method, information transmission device and communication equipment

EP4387273A1

( en )

*

2022-12-15

2024-06-19

GN Hearing A/S

Fitting system, and method of fitting a hearing device

FI20226119A1

( en )

*

2022-12-19

2024-06-20

Elisa Oyj

Computer-implemented method for performing a computational task using a machine learning model

CN115982163A

( en )

*

2022-12-27

2023-04-18

国网河北省电力有限公司营销服务中心

Power system monitoring data analysis method and device, power system, storage medium

CN116050548B

( en )

*

2023-03-27

2023-07-04

深圳前海环融联易信息科技服务有限公司

Federal learning method and device and electronic equipment

CN116405127B

( en )

*

2023-06-09

2023-09-12

北京理工大学

Compression method and device of underwater acoustic communication preamble signal detection model

CN116468959B

( en )

*

2023-06-15

2023-09-08

清软微视(杭州)科技有限公司

Industrial defect classification method, device, electronic equipment and storage medium

EP4506876A1

( en )

*

2023-08-10

2025-02-12

Hamilton Sundstrand Corporation

Cognitive, learning-based system for monitoring and controlling mission performance

CN116996403B

( en )

*

2023-09-26

2023-12-15

深圳市乙辰科技股份有限公司

Network traffic diagnosis method and system using AI model

US20250118052A1

( en )

*

2023-10-04

2025-04-10

Ubotica Technologies Limited

System and Method for Actively Triaging Data in a Computational Storage System Deployed in an Edge Environment

US20250117646A1

( en )

*

2023-10-04

2025-04-10

Cisco Technology, Inc.

Sustainable training of artificial intelligence and machine learning models

US12566744B2

( en )

2024-01-23

2026-03-03

International Business Machines Corporation

Performing multitask model tuning at edge locations

WO2025158178A1

( en )

*

2024-01-26

2025-07-31

Telefonaktiebolaget Lm Ericsson (Publ)

Feedback-based model compression

WO2025190494A1

( en )

*

2024-03-15

2025-09-18

Huawei Technologies Co., Ltd.

Distributed network and method for collaborative sensing in isac network

GB2640705A

( en )

*

2024-05-02

2025-11-05

Bae Systems Plc

Further improvements in classifying an object

CN120216722B

( en )

*

2025-03-12

2025-09-26

河北政通档案管理有限公司

Intelligent database video retrieval method based on big data technology

Family Cites Families (13)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20160283864A1

( en )

*

2015-03-27

2016-09-29

Qualcomm Incorporated

Sequential image sampling and storage of fine-tuned features

US11144616B2

( en )

*

2017-02-22

2021-10-12

Cisco Technology, Inc.

Training distributed machine learning with selective data transfers

US10409614B2

( en )

*

2017-04-24

2019-09-10

Intel Corporation

Instructions having support for floating point and integer data types in the same register

US11030520B2

( en )

*

2017-06-06

2021-06-08

Ca, Inc.

Distributed secure training of neural network model

US11531932B2

( en )

*

2017-07-06

2022-12-20

Google Llc

Systems and methods for compression and distribution of machine learning models

US10831704B1

( en )

*

2017-10-16

2020-11-10

BlueOwl, LLC

Systems and methods for automatically serializing and deserializing models

EP3740893A1

( en )

*

2018-01-17

2020-11-25

Signify Holding B.V.

System and method for object recognition using neural networks

US11227191B2

( en )

*

2018-02-13

2022-01-18

Slingshot Aerospace, Inc.

Conditional loss function modification in a neural network

US20190286989A1

( en )

*

2018-03-15

2019-09-19

Polarr, Inc.

Distributed neural network model utilization system

US11429862B2

( en )

*

2018-03-20

2022-08-30

Sri International

Dynamic adaptation of deep neural networks

US11373115B2

( en )

*

2018-04-09

2022-06-28

Here Global B.V.

Asynchronous parameter aggregation for machine learning

US20190392300A1

( en )

*

2018-06-20

2019-12-26

NEC Laboratories Europe GmbH

Systems and methods for data compression in neural networks

US20210326399A1

( en )

*

2018-08-29

2021-10-21

Hudson Bay Wireless Llc

System and Method for Search Engine Results Page Ranking with Artificial Neural Networks

2020

2020-02-21

US

US16/797,422

patent/US20200272899A1/en

not_active

Abandoned

2020-02-23

EP

EP20158910.8A

patent/EP3699825B1/en

active

Active

2025

2025-08-07

US

US19/293,758

patent/US20250363357A1/en

active

Pending

Also Published As

Publication number

Publication date

EP3699825A3

( en )

Related documents

• Generating augmented reality experiences utilizing physical objects to … — Meta Platforms Technologies, Llc (US20260037214A1)#653960
• Virtual reality system using multiple force arrays for a solver — DeepMotion, Inc. (US10657696B2)#653961
• Tracking position of device inside-out for virtual reality interactivity — Sony Interactive Entertainment America Llc (US10535153B2)#653962
• Augmented reality system with spatialized audio tied to user manipulated … — Magic Leap, Inc. (US11231770B2)#653963
• Big data-based autonomous flight drone system and autonomous flight method … — Rgblab Co., Ltd. (US20210390867A1)#653964