Google Patentsopen access

Method, apparatus and system to perform action recognition with a spiking neural … — Intel Corporation (US11651199B2)

ABSTRACT

Abstract

Techniques and mechanisms for processing differential video data with a spiking neural network to provide action recognition functionality. In an embodiment, the spiking neural network is coupled to receive and process a first one or more spike trains which represent an encoded version of a sequence comprising frames of differential video data. In turn, the frames of differential video data are each based on a difference between a respective two frames of raw video data. Based on the processing of the first one or more spike trains, the spiking neural network may output a second one or more spike trains. In another embodiment, the second one or more spike trains are provided to train the spiked neural network to recognize an activity type, or to classify a video sequence as including a representation of an instance of the activity type.

Description

CLAIM OF PRIORITY

Claim of Priority

This Application is a National Stage Entry of, and claims priority to, PCT Application No. PCT/US2017/067402, filed on Dec. 19, 2017 and titled “METHOD, APPARATUS AND SYSTEM TO PERFORM ACTION RECOGNITION WITH A SPIKING NEURAL NETWORK”, which claims priority to U.S. Provisional Patent Application No. 62/570,035 filed on Oct. 9, 2017 and titled “METHOD, APPARATUS AND SYSTEM TO PERFORM ACTION RECOGNITION WITH A SPIKING NEURAL NETWORK”, which is incorporated by reference in its entirety for all purposes.

BACKGROUND

Embodiments described herein generally relate to neural networks, and more particularly, but not exclusively, relate to determining a classification of a video sequence with a spiking neural network.

Spiking neural networks (or “SNNs”) are increasingly being adapted to provide next-generation solutions for various applications. SNNs variously rely on signaling techniques wherein information is communicated using a time-based relationship between signal spikes. As compared to typical deep-learning architectures—such as those provided with a convolutional neural network (CNN) or a recurrent neural network (RNN)—a SNN provides an economy of communication which, in turn, allows for orders of magnitude improvement in power efficiency.

Existing neural network technologies—e.g., those which include a recurrent neural network (RNN) such as long short-term memory (LSTM)—are used in some types of object recognition and other image analysis. In a conventional Deep Learning system for classifying a video sequence, features in individual raw video frames are identified (“extracted”), and a deep learning neural network is subsequently used to characterize a temporal relationship of the extracted features.

Configuring a conventional deep learning neural network system to recognize a given object (or activity) has to date required training of the system with a large number of reference images which variously represent objects of the same object/activity type. By contrast, humans are much more adept at learning object/activity types, due in part to our ability to analyze activity dynamically as it takes place over time. As the applications for neural networks continue to grow in number and variety, there is expected to be an increasing premium placed on improvements to efficient training—e.g., for image analysis and other applications.

BRIEF DESCRIPTION OF THE DRAWINGS

The various embodiments of the present invention are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which:

FIG. 1 shows diagrams each illustrating features of a simplified spiking neural network according to an embodiment.

FIG. 2 is a functional block diagram illustrating features of a system to provide action recognition functionality according to an embodiment.

FIG. 3 is a flow diagram illustrating elements of a method to facilitate action recognition with a spiking neural network according to an embodiment.

FIGS. 4 A, 4 B show examples of video data which is prepared for use in action recognition processes with a spiking neural network according to an embodiment.

FIG. 5 shows functional block diagrams illustrating respective stages of processing to configure a spiking network according to an embodiment.

FIG. 6 shows timing diagrams illustrating respective signals communicated to determine a value of a synaptic weight according to an embodiment.

FIG. 7 shows timing diagrams variously showing respective outputs of a spiking neural network each for a corresponding action recognition process according to an embodiment.

FIG. 8 is a functional block diagram illustrating features of a system to provide action recognition functionality according to an embodiment.

FIG. 9 is a functional block diagram illustrating a computing device in accordance with one embodiment.

FIG. 10 is a functional block diagram illustrating an exemplary computer system, in accordance with one embodiment.

DETAILED DESCRIPTION

Embodiments described herein variously provide neural network techniques and mechanisms using an encoding scheme wherein encoded data provides a representation of motion energy, as indicated (for example) by a difference between frames of a video sequence. Training of a spiked neural network using such encoded data may efficiently enable the spiked neural network to classify a real-world video sequence as representing one type of action—e.g., as opposed to sonic other type action which the neural network is also trained to recognize. A spiked neural network can thus be used to recognize an action, with very satisfactory results, based on training with only one reference video sequence that represents the action.

One challenge in machine learning is to build a neural network model that can learn from few examples. Embodiments described herein provide a reservoir based spiking neural model for learning to recognize actions—e.g., based on a limited number of reference videos. The exponential increase in digital data with online media, surveillance cameras among others, creates a growing need to develop intelligent models for complex spatio-temporal processing. State-of-the-art intelligent models require vast quantities of labeled data with extensive, iterative training to learn suitably and yield high performance. While the proliferation of digital media has led to the availability of massive raw and unstructured data, it is often impractical and expensive to gather annotated training datasets for all of them.

In a spiking neural network, nodes (or “neurons”) communicate with one another using a time-based relationship between signal spikes, as opposed to the communication of data values which nodes use in typical deep-learning networks. While the class of spiking neural networks are capable of carrying out the same functions as other neural nets, the communication using spikes provides tremendous economy of information making these networks orders of magnitude more power-efficient. The time-dependent interaction via spikes turn these networks into rich dynamical systems, making them more powerful than the current generation of neural nets such as convolutional neural networks (CNNs) or recurrent neural networks (RNNs).

As used herein, “differential video data” refers to data which specifies, or is otherwise based on, a difference between image sensor outputs that correspond to different respective times. For example, input signals which are provided to a spiking neural network may be based on a sequence of frames of differential video data. One given frame of the sequence may include pixel values which variously correspond each to a different respective pixel of an image sensor field. One such pixel value may specify, or otherwise be based on, a difference between an imaging output which the corresponding pixel generates at one time and another imaging output which the corresponding pixel generates at another time. For example, a pixel of differential video data may indicate whether a difference between such imaging outputs exceeds some minimum threshold value. Accordingly, differential video data may represent motion energy of an action which has been captured by an image sensor. Encoding of raw video data to generate differential video data is referred to herein as differential video encoding or, alternatively, dynamic vision sensor (DVS) encoding.

A spiking neural network may be trained, in various embodiments, to detect any of one or more types of actions (also as referred to herein as an “action type” or “activity types”). Action detection (also referred to herein as “action recognition” or “activity recognition”) may include classifying video information, based on one or more output signals generated with a spiking neural network, as including a representation of an action which is an instance of one such action type.

The technologies described herein may be implemented in one or more electronic devices. Non-limiting examples of electronic devices that may utilize the technologies described herein include any kind of mobile device and/or stationary device, such as cameras, cell phones, computer terminals, desktop computers, electronic readers, facsimile machines, kiosks, laptop computers, netbook computers, notebook computers, internet devices, payment terminals, personal digital assistants, media players and/or recorders, servers (e.g., blade server, rack mount server, combinations thereof, etc.), set-top boxes, smart phones, tablet personal computers, ultra-mobile personal computers, wired telephones, combinations thereof, and the like. More generally, the technologies described herein may be employed in any of a variety of electronic devices including a spiking neural network configured to process differential video data.

FIG. 1 illustrates an example diagram of a simplified neural network 100 , providing an illustration of connections 120 between a first set of nodes 110 (e.g., neurons) and a second set of nodes 130 (e.g., neurons). Some or all of a neural network (such as the simplified neural network 100 ) may be organized into multiple layers—e.g., including input layers and output layers. It will be understood that the simplified neural network 100 only depicts two layers and a small number of nodes, but other forms of neural networks may include a large number of variously configured nodes, layers, connections, and pathways.

Data that is provided into the neutral network 100 may be first processed by synapses of input neurons. Interactions between the inputs, the neuron's synapses and the neuron itself govern whether an output is provided via an axon to another neuron's synapse. Modeling the synapses, neurons, axons, etc., may be accomplished in a variety of ways. In an example, neuromorphic hardware includes individual processing elements in a synthetic neuron (e.g., neurocore) and a messaging fabric to communicate outputs to other neurons. The determination of whether a particular neuron “fires” to provide data to a further connected neuron is dependent on the activation function applied by the neuron and the weight of the synaptic connection (e.g., w ij ) from neuron i (e.g., located in a layer of the first set of nodes 110 ) to neuron j (e.g., located in a layer of the second set of nodes 130 ). The input received by neuron i is depicted as value x i , and the output produced from neuron j is depicted as value Thus, the processing conducted in a neural network is based on weighted connections, thresholds, and evaluations performed among the neurons, synapses, and other elements of the neural network.

In an example, the neural network 100 is established from a network of spiking neural network cores, with the neural network cores communicating via short packetized spike messages sent from core to core. For example, each neural network core may implement some number of primitive nonlinear temporal computing elements as neurons, so that when a neuron's activation exceeds some threshold level, it generates a spike message that is propagated to a fixed set of fanout neurons contained in destination cores. The network may distribute the spike messages to all destination neurons, and in response those neurons update their activations in a transient, time-dependent manner, similar to the operation of real biological neurons.

The neural network 100 further shows the receipt of a spike, represented in the value x i , at neuron i in a first set of neurons (e.g., a neuron of the first set of nodes 110 ). The output of the neural network 100 is also shown as a spike, represented by the value which arrives at neuron j in a second set of neurons (e.g., a neuron of the first set of nodes 110 ) via a path established by the connections 120 . In a spiking neural network all communication occurs over event-driven action potentials, or spikes. In an example, spikes convey no information other than the spike time as well as a source and destination neuron pair. Computations may variously occur in each a respective neuron as a result of the dynamic, nonlinear integration of weighted spike input using real-valued state variables. The temporal sequence of spikes generated by or for a particular neuron may be referred to as its “spike train.”

In an example of a spiking neural network, activation functions occur via spike trains, which means that time is a factor that has to be considered. Further, in a spiking neural network, each neuron may provide functionality similar to that of a biological neuron, as the artificial neuron receives its inputs via synaptic connections to one or more “dendrites” (part of the physical structure of a biological neuron), and the inputs affect an internal membrane potential of the artificial neuron “soma” (cell body). In a spiking neural network, the artificial neuron “fires” (e.g., produces an output spike), when its membrane potential crosses a firing threshold. Thus, the effect of inputs on a spiking neural network neuron operate to increase or decrease its internal membrane potential, making the neuron more or less likely to fire. Further, in a spiking neural network, input connections may be stimulatory or inhibitory. A neuron's membrane potential may also be affected by changes in the neuron's own internal state (“leakage”).

FIG. 1 also illustrates an example inference path NO in a spiking neural network, such as may be implemented by a form of the neural network 110 or other forms of neural networks. The inference path 140 of the neuron includes a pre-synaptic neuron 142 , which is configured to produce a pre-synaptic spike train x i representing a spike input. A spike train is a temporal sequence of discrete spike events, which provides a set of times specifying at which time a neuron fires.

As shown, the spike train x i is produced by the neuron before the synapse (e.g., neuron 142 ), and the spike train x i is evaluated for processing according to the characteristics of a synapse 144 . For example, the synapse may apply one or more weights, e.g., a weight w xi , whi

CLAIM OF PRIORITY

Claim of Priority

This Application is a National Stage Entry of, and claims priority to, PCT Application No. PCT/US2017/067402, filed on Dec. 19, 2017 and titled “METHOD, APPARATUS AND SYSTEM TO PERFORM ACTION RECOGNITION WITH A SPIKING NEURAL NETWORK”, which claims priority to U.S. Provisional Patent Application No. 62/570,035 filed on Oct. 9, 2017 and titled “METHOD, APPARATUS AND SYSTEM TO PERFORM ACTION RECOGNITION WITH A SPIKING NEURAL NETWORK”, which is incorporated by reference in its entirety for all purposes.

BACKGROUND

Embodiments described herein generally relate to neural networks, and more particularly, but not exclusively, relate to determining a classification of a video sequence with a spiking neural network.

Spiking neural networks (or “SNNs”) are increasingly being adapted to provide next-generation solutions for various applications. SNNs variously rely on signaling techniques wherein information is communicated using a time-based relationship between signal spikes. As compared to typical deep-learning architectures—such as those provided with a convolutional neural network (CNN) or a recurrent neural network (RNN)—a SNN provides an economy of communication which, in turn, allows for orders of magnitude improvement in power efficiency.

Existing neural network technologies—e.g., those which include a recurrent neural network (RNN) such as long short-term memory (LSTM)—are used in some types of object recognition and other image analysis. In a conventional Deep Learning system for classifying a video sequence, features in individual raw video frames are identified (“extracted”), and a deep learning neural network is subsequently used to characterize a temporal relationship of the extracted features.

Configuring a conventional deep learning neural network system to recognize a given object (or activity) has to date required training of the system with a large number of reference images which variously represent objects of the same object/activity type. By contrast, humans are much more adept at learning object/activity types, due in part to our ability to analyze activity dynamically as it takes place over time. As the applications for neural networks continue to grow in number and variety, there is expected to be an increasing premium placed on improvements to efficient training—e.g., for image analysis and other applications.

BRIEF DESCRIPTION OF THE DRAWINGS

The various embodiments of the present invention are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which:

FIG. 1 shows diagrams each illustrating features of a simplified spiking neural network according to an embodiment.

FIG. 2 is a functional block diagram illustrating features of a system to provide action recognition functionality according to an embodiment.

FIG. 3 is a flow diagram illustrating elements of a method to facilitate action recognition with a spiking neural network according to an embodiment.

FIGS. 4 A, 4 B show examples of video data which is prepared for use in action recognition processes with a spiking neural network according to an embodiment.

FIG. 5 shows functional block diagrams illustrating respective stages of processing to configure a spiking network according to an embodiment.

FIG. 6 shows timing diagrams illustrating respective signals communicated to determine a value of a synaptic weight according to an embodiment.

FIG. 7 shows timing diagrams variously showing respective outputs of a spiking neural network each for a corresponding action recognition process according to an embodiment.

FIG. 8 is a functional block diagram illustrating features of a system to provide action recognition functionality according to an embodiment.

FIG. 9 is a functional block diagram illustrating a computing device in accordance with one embodiment.

FIG. 10 is a functional block diagram illustrating an exemplary computer system, in accordance with one embodiment.

DETAILED DESCRIPTION

Embodiments described herein variously provide neural network techniques and mechanisms using an encoding scheme wherein encoded data provides a representation of motion energy, as indicated (for example) by a difference between frames of a video sequence. Training of a spiked neural network using such encoded data may efficiently enable the spiked neural network to classify a real-world video sequence as representing one type of action—e.g., as opposed to sonic other type action which the neural network is also trained to recognize. A spiked neural network can thus be used to recognize an action, with very satisfactory results, based on training with only one reference video sequence that represents the action.

One challenge in machine learning is to build a neural network model that can learn from few examples. Embodiments described herein provide a reservoir based spiking neural model for learning to recognize actions—e.g., based on a limited number of reference videos. The exponential increase in digital data with online media, surveillance cameras among others, creates a growing need to develop intelligent models for complex spatio-temporal processing. State-of-the-art intelligent models require vast quantities of labeled data with extensive, iterative training to learn suitably and yield high performance. While the proliferation of digital media has led to the availability of massive raw and unstructured data, it is often impractical and expensive to gather annotated training datasets for all of them.

In a spiking neural network, nodes (or “neurons”) communicate with one another using a time-based relationship between signal spikes, as opposed to the communication of data values which nodes use in typical deep-learning networks. While the class of spiking neural networks are capable of carrying out the same functions as other neural nets, the communication using spikes provides tremendous economy of information making these networks orders of magnitude more power-efficient. The time-dependent interaction via spikes turn these networks into rich dynamical systems, making them more powerful than the current generation of neural nets such as convolutional neural networks (CNNs) or recurrent neural networks (RNNs).

As used herein, “differential video data” refers to data which specifies, or is otherwise based on, a difference between image sensor outputs that correspond to different respective times. For example, input signals which are provided to a spiking neural network may be based on a sequence of frames of differential video data. One given frame of the sequence may include pixel values which variously correspond each to a different respective pixel of an image sensor field. One such pixel value may specify, or otherwise be based on, a difference between an imaging output which the corresponding pixel generates at one time and another imaging output which the corresponding pixel generates at another time. For example, a pixel of differential video data may indicate whether a difference between such imaging outputs exceeds some minimum threshold value. Accordingly, differential video data may represent motion energy of an action which has been captured by an image sensor. Encoding of raw video data to generate differential video data is referred to herein as differential video encoding or, alternatively, dynamic vision sensor (DVS) encoding.

A spiking neural network may be trained, in various embodiments, to detect any of one or more types of actions (also as referred to herein as an “action type” or “activity types”). Action detection (also referred to herein as “action recognition” or “activity recognition”) may include classifying video information, based on one or more output signals generated with a spiking neural network, as including a representation of an action which is an instance of one such action type.

The technologies described herein may be implemented in one or more electronic devices. Non-limiting examples of electronic devices that may utilize the technologies described herein include any kind of mobile device and/or stationary device, such as cameras, cell phones, computer terminals, desktop computers, electronic readers, facsimile machines, kiosks, laptop computers, netbook computers, notebook computers, internet devices, payment terminals, personal digital assistants, media players and/or recorders, servers (e.g., blade server, rack mount server, combinations thereof, etc.), set-top boxes, smart phones, tablet personal computers, ultra-mobile personal computers, wired telephones, combinations thereof, and the like. More generally, the technologies described herein may be employed in any of a variety of electronic devices including a spiking neural network configured to process differential video data.

FIG. 1 illustrates an example diagram of a simplified neural network 100 , providing an illustration of connections 120 between a first set of nodes 110 (e.g., neurons) and a second set of nodes 130 (e.g., neurons). Some or all of a neural network (such as the simplified neural network 100 ) may be organized into multiple layers—e.g., including input layers and output layers. It will be understood that the simplified neural network 100 only depicts two layers and a small number of nodes, but other forms of neural networks may include a large number of variously configured nodes, layers, connections, and pathways.

Data that is provided into the neutral network 100 may be first processed by synapses of input neurons. Interactions between the inputs, the neuron's synapses and the neuron itself govern whether an output is provided via an axon to another neuron's synapse. Modeling the synapses, neurons, axons, etc., may be accomplished in a variety of ways. In an example, neuromorphic hardware includes individual processing elements in a synthetic neuron (e.g., neurocore) and a messaging fabric to communicate outputs to other neurons. The determination of whether a particular neuron “fires” to provide data to a further connected neuron is dependent on the activation function applied by the neuron and the weight of the synaptic connection (e.g., w ij ) from neuron i (e.g., located in a layer of the first set of nodes 110 ) to neuron j (e.g., located in a layer of the second set of nodes 130 ). The input received by neuron i is depicted as value x i , and the output produced from neuron j is depicted as value Thus, the processing conducted in a neural network is based on weighted connections, thresholds, and evaluations performed among the neurons, synapses, and other elements of the neural network.

In an example, the neural network 100 is established from a network of spiking neural network cores, with the neural network cores communicating via short packetized spike messages sent from core to core. For example, each neural network core may implement some number of primitive nonlinear temporal computing elements as neurons, so that when a neuron's activation exceeds some threshold level, it generates a spike message that is propagated to a fixed set of fanout neurons contained in destination cores. The network may distribute the spike messages to all destination neurons, and in response those neurons update their activations in a transient, time-dependent manner, similar to the operation of real biological neurons.

The neural network 100 further shows the receipt of a spike, represented in the value x i , at neuron i in a first set of neurons (e.g., a neuron of the first set of nodes 110 ). The output of the neural network 100 is also shown as a spike, represented by the value which arrives at neuron j in a second set of neurons (e.g., a neuron of the first set of nodes 110 ) via a path established by the connections 120 . In a spiking neural network all communication occurs over event-driven action potentials, or spikes. In an example, spikes convey no information other than the spike time as well as a source and destination neuron pair. Computations may variously occur in each a respective neuron as a result of the dynamic, nonlinear integration of weighted spike input using real-valued state variables. The temporal sequence of spikes generated by or for a particular neuron may be referred to as its “spike train.”

In an example of a spiking neural network, activation functions occur via spike trains, which means that time is a factor that has to be considered. Further, in a spiking neural network, each neuron may provide functionality similar to that of a biological neuron, as the artificial neuron receives its inputs via synaptic connections to one or more “dendrites” (part of the physical structure of a biological neuron), and the inputs affect an internal membrane potential of the artificial neuron “soma” (cell body). In a spiking neural network, the artificial neuron “fires” (e.g., produces an output spike), when its membrane potential crosses a firing threshold. Thus, the effect of inputs on a spiking neural network neuron operate to increase or decrease its internal membrane potential, making the neuron more or less likely to fire. Further, in a spiking neural network, input connections may be stimulatory or inhibitory. A neuron's membrane potential may also be affected by changes in the neuron's own internal state (“leakage”).

FIG. 1 also illustrates an example inference path NO in a spiking neural network, such as may be implemented by a form of the neural network 110 or other forms of neural networks. The inference path 140 of the neuron includes a pre-synaptic neuron 142 , which is configured to produce a pre-synaptic spike train x i representing a spike input. A spike train is a temporal sequence of discrete spike events, which provides a set of times specifying at which time a neuron fires.

As shown, the spike train x i is produced by the neuron before the synapse (e.g., neuron 142 ), and the spike train x i is evaluated for processing according to the characteristics of a synapse 144 . For example, the synapse may apply one or more weights, e.g., a weight w xi , which are used in evaluating the data from the spike train x i . Input spikes from the spike train x i enter a synapse such as synapse 144 which has a weight w jj . This weight scales what the impact of the presynaptic spike has on the post-synaptic neuron (e.g., neuron 146 ). If the integral contribution of all input connections to a post-synaptic neuron exceeds a threshold, then the post-synaptic neuron 146 will fire and produce a spike. As shown, y j is the post-synaptic spike train produced by the neuron following the synapse (e.g., neuron 146 ) in response to some number of input connections. As shown, the post-synaptic spike train y j is distributed from the neuron 146 to other post-synaptic neurons.

Nodes of such a spiking neural network 100 may be of a Leaky Integrate-and-Fire (LIF) type—e.g., wherein, based on one or more spiking signals received at a given node j, the value of a membrane potential v m of that node j may spike and then decay over time. The spike and decay behavior of such a membrane potential v m may, for example, be according to the following:

τ m

(

dv m

dt

)

∝

-

(

v m

-

v rest

)

+

w ij

·

I ij

+

J b

,

( 1 )

where v rest is a resting potential toward which membrane potential v m is to settle, τ m is a time constant for decay of v m w ij is a weight assigned to a synapse coupled between another node i and node j, I ij is a spiking signal (or “spike train”) communicated to node j via said synapse, and J b is a value that, for example, is based on another signal provided to node j from some other external node/source. In some embodiments, J b may be a bias current provided to node j. The spiking neural network may operate based on a pre-defined threshold voltage V threshold , wherein the node j is configured to output a signal spike in response to its membrane potential v m being greater than (or equal to, in some embodiments) V threshold .

Embodiments variously provide a reservoir-based spiking neural model that reliably learns to recognize actions in video data—e.g., by detecting time-based signatures of an action based on a small number of training examples. Some embodiments provide a novel encoding to generate spike signal information based on differential video data, while preserving a temporal correlation across different frames of raw video data. The reservoir may be configured to generalize a representation of a dynamic action toward signature action/movements, enabling it to learn from few training examples. Some embodiments provide an adaptation of a direct/autonomous model (D/A) based reservoir construction approach which introduces an approximate delay apportioning to configure a recurrent spiking model for practical action recognition based on limited video data. In this particular context, the terms “Auto” and “Autonomous” are used interchangeably herein.

In order to facilitate limited example training of a reservoir spiking framework, some embodiments variously provide a spike-based encoding/pre-processing method to convert the raw pixel valued video information in a dataset into spiking information that preserves the temporal statistics and correlation across different raw video frames. An emitted synchronous spike volley may thus rapidly input, as neural network stimulus, relatively more salient edges of image information, which often constitute more crucial information for action detection/recognition. In some embodiments, encoding may eliminate or otherwise mitigate less relevant spiking information due to ambiguity such as, noisy background activity or jitter (due to unsteady camera movement), further making the technique more robust and reliable. Hence, the encoding, in general, may capture more signature action or movement of a subject across different videos as spiking information. This, in turn, may enable the reservoir to recognize/generalize over motion cues from spiking data, to enable learning various types of actions from few video samples per action type/class. As compared to conventional solutions, the spiking model may be much more adept at learning object/activity types, due in part to an ability to analyze activity dynamically as it takes place over time.

FIG. 2 shows a functional block diagram illustrating processing by a system 200 which, according to an embodiment, is to evaluate video information to provide action recognition functionality. System 200 includes a spiking neural network 230 coupled via input synapses 222 to receive input signaling such as the illustrative one or more input spike trains 220 shown. Spiking neural network 230 may include features of spiking neural network 100 , for example.

In an embodiment, spiking neural network 230 is configured to process video information for at least one type of action detection. Nodes (or “neurons”) of spiking neural network 230 may be variously coupled, each via a respective one of synapses 222 , to receive a respective one of one or more input spike trains 220 —some or all of which may represent, or otherwise be based on, differential video data. Based on the one or more input spike trains 220 , one or more output signals may be generated with the spiking neural network, where the one or more output signals are subsequently used to train the network 230 to perform an action detection or, alternatively, are subsequently evaluated as part of such an action detection.

For example, as described herein, spiking neural network 230 may be trained, based on one or more input spike trains 220 , to generate a particular type of output signaling which has been preassigned to correspond to (e.g., to be indicative of) a particular action type. Alternatively, an already-trained spiking neural network 230 may communicate (for example, via the illustrative synapses 232 shown) one or more output spike trains which are based on one or more input spike trains 220 . Such one or more output spike trains may include an instance of the type of output signaling which corresponds to the action type.

By way of illustration and not limitation, system 200 may further comprise or couple to hardware and/or executing software (such as the illustrative selector logic 240 shown) which is to receive the one or more output spike trains via synapses 232 . Selector logic 240 may include any of various processors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) and/or other circuitry configured to identify an action type as corresponding to the one or more output spike trains—e.g., wherein such identifying includes selecting one action type from a plurality of actions types which system 200 is trained to recognize. In the example embodiment shown, selector logic 240 may include or couple to reference data 242 which identifies one or more reference signal characteristics as being indicative of a given action type. Selector logic 240 may evaluate the one or more output spike trains based on such one or more reference characteristics to determine whether (or not) a particular action type is indicated by the one or more output spike trains. Selector logic 240 may output a signal 244 representing a result of such a determination.

In the embodiment shown, frames 212 of differential video data may be generated or otherwise determined—e.g., each based on frames 210 of raw video data variously representing images in a monochrome, RGB or other color space. Each of frames 212 may correspond to a respective two of frames 210 , wherein any one of frames 212 represents a difference (weighted or otherwise) between the corresponding two of frames 210 . In a simple example, a differential video frame DF x may be based on a difference (RF b −RF a ) between raw video frames RF a , RF b —e.g., wherein a value of a pixel of DE (the subscript “ij” indicating an ith pixel row and a ith pixel column) is based on a difference between respective values of a corresponding pixel p ij of RF b and a corresponding pixel p ij of RF a . Such two raw video frames RF a , RF b may be successive frames in a video sequence, or may otherwise be offset from one another by a time difference.

The one or more input spike trains 220 may be generated based on frames 212 —e.g., wherein pixels of frames 212 are variously encoded as respective signal spikes of one or more input spike trains 220 . Although some embodiments are not limited in this regard, system 200 may include (or alternatively, be coupled to) an encoder 204 to generate the one or more input spike trains 220 based on frames 212 . Alternatively or in addition, system 200 may include or couple to an image sensor, integrated circuitry and/or other hardware such as the illustrative video data processor logic 202 shown—to generate frames 212 (and in some embodiments, frames 210 ).

FIG. 3 shows features of a method 300 to provide video classification (e.g., including action detection/recognition) based on spiked signaling which represents differential video data. Method 300 may be performed with circuit logic and/or other such hardware such as that of system 200 or neural network 100 , for example. In an embodiment, method 300 is performed at least in part by an execution of software with a processor and a memory device coupled thereto.

Method 300 may include operations 305 to process signals with a spiking neural network. Although some embodiments are not limited in this regard, method 300 may additionally or alternatively include other processes to configuring the signals and/or the spiking neural network for operations 305 . For example, such other processes may include (at 310 ) preparing a spiking neural network to receive training to detect an instance of a particular action type. The preparing at 310 may include building the spiking neural network using one or more neural network models. Building the spiking neural network may include changing synaptic weights to facilitate the generation of a particular neural network output e.g., where such output is predetermined a priori as the one to be associated with a particular action type. As discussed herein, some embodiments successively utilize two types of models (referred to as a “Direct” model type and an “Autonomous” model type) to build a spiking neural network.

Additionally or alternatively, such other processes may include (at 320 ) generating frames of differential video data based on raw video data. The generating may include calculating differences each between a respective two frames of video data. In some embodiments, pixel values of the differential video data may be variously weighted and/or selectively included in (or excluded from) an encoding process to generate input signaling for the spiking neural network.

In one embodiment, operations 305 comprises (at 330 ) receiving, at a spiking neural network, one or more input signals including an encoded version of frames of differential video data which, for example, are generated at 320 . The frames of differential video data may be each include a respective pixel value representing a difference between a respective two frames of raw video data of a video sequence. Method 300 may further comprise (at 340 ) applying a spiking neural network to the one or more input signals, wherein the applying includes communicating one or more spike trains each between respective nodes of the spiking neural network. The one or more spike trains may each be based on the frames of differential video data—e.g., wherein a sequence of signal spiking by a given spike train is based on a sequence of the frames of differential video data.

The method may further comprise (at 350 ) detecting an output signal of the spiking neural network, wherein the output is based on the one or more spike trains. For example, the one or more output signals generated by the spiking neural network may be selected in preference to one or more other output signals of the neural network (e.g., wherein such output signals are all variously based on the one or more spike trains).

Based on the output signal, method 300 may (at 360 ) perform one of training the spiked neural network to recognize an action type, or classifying a video sequence as including a representation of an instance of the action type. A training of the spiked neural network to recognize a particular action type may be based, for example, on only a single reference video sequence which represents an instance of the action type. Alternatively, such training may be based on multiple reference video sequences which each include a respective representation of the same instance or different instances of the action type. In one embodiment, of all reference video sequences used to train the network to recognize a given action type, only one such reference video sequence may represent an instance of the action type as imaged from a first point-of-view. For example, each reference video sequence used to train the network to recognize the given action type may represent an instance of the action type as imaged from a different respective point-of-view.

Operations 305 may further comprise processing (not shown) to generate the one or more input signals based on frames of differential video data which, for example, are generated at 320 . In such an embodiment, this additional processing may comprise, for each of one or more of the frames of differential video data, selectively excluding at least a portion the frame from a conversion to generate the one or more input signals. For example, a frame of differential video data may be selectively excluded, on a per-frame basis, from encoding to generate the one or more input signals. Additionally or alternatively, a sub-frame portion of a given differential video frame may be selectively excluded from the conversion on a per-sub-frame basis (e.g., where another sub-frame portion of that same frame is included in the encoding/conversion).

Selectively filtering frames on a per-frame basis may include foregoing an encoding of an entire frame where a total amount of action indicated by the frame (e.g., by a sum of pixel values thereof) is above some predetermined maximum threshold level. Alternatively or in addition, an entire frame may be filtered on a per-frame basis where a total amount of action indicated by the frame is below some predetermined minimum threshold level. Similarly, selectively filtering a sub-frame portion of a given frame on a per-sub-frame basis may include foregoing an encoding of the sub-frame—but not necessarily another sub-frame of that same frame—where a total amount of activity indicated by the sub-frame (e.g., by a sum of pixel values thereof) is above some predetermined maximum threshold level. Alternatively or in addition, a sub-frame of a given frame may be filtered on a per-sub-frame basis where a total amount of activity indicated by the sub-frame is below some predetermined minimum threshold level.

Embodiments described herein variously implement a processing of video information (e.g., including video differential data and, in some embodiments, raw video data on which such video differential data may be based) to detect for time-based indicia of a given action type. FIGS. 4 A- 4 B show example frames 400 - 405 of image information which illustrate features of processing to generate differential video data. In various embodiments, an encoded version of such differential video data may be communicated, as one or more spike trains, to facilitate action recognition with a spiking neural network. Video information such as that variously represented by one or more of frames 400 - 405 may be communicated in system 200 , for example, and/or may be processed with a spiking neural network according to method 300 .

Frames 400 - 405 variously illustrate a video sequence—such as that of prerecorded and/or streaming video—which includes a representation of a given action (such as the golf swing activity shown). Frames

400 , 401 , which are different respective raw video frames of the video sequence, include representations of respective positions

410 , 411 by a golfer who is performing an instance of a golf swing action type/class. The difference between positions

410 , 411 may be somewhat small—e.g., wherein frame 400 precedes frame 401 in the video frame sequence immediately, or by only a few frames. FIGS. 4 A, 4 B also include frames

402 , 403 to more clearly show this difference between positions

410 , 411 . For example, frame 402 shows an outline 410 a of the representation of position 410 , where the outline 410 a is superimposed on the representation of position 411 . Frame 403 shows outline 410 a superimposed on an outline 411 a of the representation of position 411 . Frames

402 , 403 are shown merely for illustrative purposes, and one or both of frames

402 , 403 may not actually be generated during processing to determine differential video data such as that of frames

404 , 405 .

Some embodiments variously use differential video data whereby moving edge artefacts in a video sequence are given more relevance than relatively slow or even static portions of an imaged environment. For example, frame 404 is a frame of differential video data which is calculated based on a difference between frames

401 , 402 . For a given pixel of frame 404 , a value of that pixel may be based on a difference between two corresponding pixels of frames

401 , 402 . In one example embodiment, a pixel of frame 404 may be white if such a difference is above some minimum threshold amount—e.g., wherein the pixel is otherwise to be black. Frame 405 shows another version of such differential video data according to some embodiments—e.g., wherein frame 405 is a photonegative of frame 404 .

As variously shown in frames

404 , 405 , differential video data provides a filtering effect which effectively removes or otherwise mitigates the representation of static or relatively slow-moving portions of imaged objects. In some embodiments, respective values for pixels of frame 404 (or of frame 405 ) may be variously weighted—e.g., based on a magnitude of such a difference, based on the value of adjoining pixels and/or the like.

In some embodiments, sub-frame portions of a given differential video frame may be selectively given more or less weight, or may be selectively prevented from representation by spike train signals which are to be communicated to a spiking neural network. By way of illustration and not limitation, video data processor logic 202 (or other such hardware and/or executing software) may define

sub-frame portions

420 , 422 , 424 , 426 , 428 of frame 405 . The particular number, sizes, shapes, and relative configuration of

sub-frame portions

420 , 422 , 424 , 426 , 428 is merely illustrative, and may vary in other embodiments. In such an embodiment, whether and/or how a given one of

sub-frame portions

420 , 422 , 424 , 426 , 428 is to be subsequently encoded and/or otherwise processed may be determined based on the pixel information of that sub-frame portion. For example, as described herein, the inclusion of a given sub-frame portion may depend on whether the pixels of that sub-frame portion represent some minimum threshold level of motion energy.

Pixels, representing differential motion information across different times, may be tracked so that a basic action signature is captured. For example, this may be done by monitoring a difference of pixel intensity values (P diff ) between consecutive frames of raw video data, and comparing the difference against some threshold (ε, that is user-defined) to detect the moving edges (for each location (x, y) within a frame)—e.g., according to the following:

P

diff

(

x

,

y

)

=

Frame

j

-

Frame

j

-

1

;

(

2

)

(

where

⁢

j

=

2

⁢

to

⁢

NumberofFrames

)

,

If

⁢

P

diff

(

x

,

y

)

≥

ε

,

Spike

diff

(

x

,

y

)

=

1

,

and

CLAIMS

Claims ( 25 )

What is claimed is:

1. A computer device for video classification using a spiking neural network, the computer device comprising circuitry to:

receive at a spiking neural network one or more input signals comprising an encoded version of a first plurality of frames of differential video data, wherein the first plurality of frames each correspond to both:

a respective first frame of a second plurality of frames of a sequence of raw video data; and

a respective second frame of the second plurality of frames; and

wherein, for each frame of the first plurality of frames:

multiple pixels of the frame each correspond to both:

a respective first pixel of the respective first frame; and

a respective second pixel of the respective second frame; and

wherein, for each of the multiple pixels of the frame, a value of the pixel is to represent a difference between a value of the respective first pixel, and a value of the respective second pixel;

apply a spiking neural network to the one or more input signals, comprising circuitry to communicate one or more spike trains each between respective nodes of the spiking neural network, the one or more spike trains each based on the first plurality of frames; and

detect an output signal of the spiking neural network, the output signal based on the one or more spike trains; and

based on the output signal:

train the spiking neural network to recognize an activity type; or

communicate a signal which classifies the sequence of the raw video data as comprising a representation of an instance of the activity type.

2. The computer device of claim 1 , wherein the circuitry is to train the spiking neural network to recognize the activity type based on multiple reference video sequences which each represent a respective instance of the activity type, wherein, of all reference video sequences used to train the spiking neural network to recognize the activity type, only one reference video sequence represents the activity type from a first point-of-view.

3. The computer device of claim 2 , wherein, of all reference video sequences used to train the spiking neural network to recognize the activity type, each reference video sequence represents the activity type from a different respective point-of-view.

4. The computer device of claim 1 , further comprising circuitry to prepare the spiking neural network to be trained, comprising circuitry to configure the spiking neural network based on a driven network model and an autonomous network model.

5. The computer device of claim 1 , further comprising circuitry to encode the raw video data of the video sequence into the first plurality of frames.

6. The computer device of claim 5 , further comprising circuitry to convert the raw video data from a polychromatic color space format to a monochromatic color space format.

7. The computer device of claim 1 , further comprising circuitry to generate the one or more input signals based on the first plurality of frames, comprising circuitry to selectively exclude at least a portion of a frame from a conversion to generate the one or more input signals.

8. The computer device of claim 7 , wherein the circuitry to selectively exclude at least the portion of the frame includes circuitry to selectively exclude the frame on a per-frame basis.

9. The computer device of claim 7 , wherein the circuitry to selectively exclude at least the portion of the frame includes circuitry to selectively exclude only a sub-frame portion of the frame on a per-sub-frame basis.

10. The computer device of claim 7 , wherein the circuitry to selectively exclude at least the portion of the frame based on a predetermined maximum threshold level amount of activity to be indicated by the frame or by a sub-frame portion of the frame.

11. The computer device of claim 7 , wherein the circuitry to selectively exclude at least the portion of the frame based on a predetermined minimum threshold level amount of activity to be indicated by the frame or by a sub-frame portion of the frame.

12. The computer device of claim 1 , further comprising circuitry to:

train a plurality of spiking neural networks each to recognize a different respective activity type, comprising circuitry to train a first spiking neural network to recognize a first activity type;

compare, after the plurality of spiking neural networks are trained, multiple output signals each from a different respective one of the plurality of spiking neural networks, the output signals each based on a test video sequence;

select among from the multiple output signals a first output signal of the first spiking neural network; and

signal, based on a selection of the first output signal, that the test video sequence includes a representation of an instance of the first activity type.

13. The computer device of claim 12 , wherein the circuitry to compare the multiple output signals comprises circuitry to compare a measure of output signal spiking from each of the plurality of trained spiking neural networks.

14. The computer device of claim 1 , wherein the circuitry is to train the spiking neural network to recognize the activity type based on one and only one reference video sequence.

15. At least one non-transitory_machine readable medium including instructions that, when executed by a machine, cause the machine to perform operations for video classification using a spiking neural network, the operations comprising:

receiving at a spiking neural network one or more input signals comprising an encoded version of a first plurality of frames of differential video data, wherein the first plurality of frames each correspond to both:

a respective first frame of a second plurality of frames of a sequence of raw video data; and

a respective second frame of the second plurality of frames; and

wherein, for each frame of the first plurality of frames:

multiple pixels of the frame each correspond to both:

a respective first pixel of the respective first frame; and

a respective second pixel of the respective second frame; and

wherein, for each of the multiple pixels of the frame, a value of the pixel represents a difference between a value of the respective first pixel, and a value of the respective second pixel;

applying a spiking neural network to the one or more input signals, comprising communicating one or more spike trains each between respective nodes of the spiking neural network, the one or more spike trains each based on the first plurality of frames; and

detecting an output signal of the spiking neural network, the output signal based on the one or more spike trains; and

based on the output signal:

training the spiking neural network to recognize an activity type; or

communicating a signal which classifies the sequence of the raw video data as comprising a representation of an instance of the activity type.

16. The at least one non-transitory machine readable medium of claim 15 , wherein training the spiking neural network to recognize the activity type is based on multiple reference video sequences which each represent a respective instance of the activity type, wherein, of all reference video sequences used to train the spiking neural network to recognize the activity type, only one reference video sequence represents the activity type from a first point-of-view.

17. The at least one non-transitory machine readable medium of claim 16 , wherein, of all reference video sequences used to train the spiking neural network to recognize the activity type, each reference video sequence represents the activity type from a different respective point-of-view.

18. The at least one non-transitory machine readable medium of claim 15 , wherein training the spiking neural network to recognize the activity type is based on one and only one reference video sequence.

19. The at least one non-transitory machine readable medium of claim 15 , the operations further comprising preparing the spiking neural network to receive the training, comprising configuring the spiking neural network based on a driven network model and an autonomous network model.

20. The at least one non-transitory machine readable medium of claim 15 , the operations further comprising encoding the raw video data of the video sequence into the first plurality of frames of differential video data.

21. A method for video classification using a spiking neural network, the method comprising:

receiving at a spiking neural network one or more input signals comprising an encoded version of a first plurality of frames of differential video data respective pixel value representing, wherein the first plurality of frames each correspond to both:

a respective first frame of a second plurality of frames of a sequence of raw video data; and

a respective second frame of the second plurality of frames; and

wherein, for each frame of the first plurality of frames:

multiple pixels of the frame each correspond to both:

a respective first pixel of the respective first frame; and

a respective second pixel of the respective second frame; and

wherein, for each of the multiple pixels of the frame, a value of the pixel represents a difference between a value of the respective first pixel, and a value of the respective second pixel;

applying a spiking neural network to the one or more input signals, comprising communicating one or more spike trains each between respective nodes of the spiking neural network, the one or more spike trains each based on the first plurality of frames; and

detecting an output signal of the spiking neural network, the output signal based on the one or more spike trains; and

based on the output signal:

training the spiking neural network to recognize an activity type; or

communicating a signal which classifies the sequence of the raw video data as comprising a representation of an instance of the activity type.

22. The method of claim 21 , wherein training the spiking neural network to recognize the activity type is based on multiple reference video sequences which each represent a respective instance of the activity type, wherein, of all reference video sequences used to train the spiking neural network to recognize the activity type, only one reference video sequence represents the activity type from a first point-of-view.

23. The method of claim 22 , wherein, of all reference video sequences used to train the spiking neural network to recognize the activity type, each reference video sequence represents the activity type from a different respective point-of-view.

24. The method of claim 21 , wherein training the spiking neural network to recognize the activity type is based on one and only one reference video sequence.

25. The method of claim 21 , further comprising preparing the spiking neural network to receive the training, including comprising configuring the spiking neural network based on a driven network model and an autonomous network model.

US16/644,446

2017-10-09

2017-12-19

Method, apparatus and system to perform action recognition with a spiking neural network

Active

2038-12-03

US11651199B2

( en )

Priority Applications (1)

Application Number

Priority Date

Filing Date

Title

US16/644,446

US11651199B2

( en )

2017-10-09

2017-12-19

Method, apparatus and system to perform action recognition with a spiking neural network

Applications Claiming Priority (3)

Application Number

Priority Date

Filing Date

Title

US201762570035P

2017-10-09

2017-10-09

PCT/US2017/067402

WO2019074532A1

( en )

2017-10-09

2017-12-19

Method, apparatus and system to perform action recognition with a spiking neural network

US16/644,446

US11651199B2

( en )

2017-10-09

2017-12-19

Method, apparatus and system to perform action recognition with a spiking neural network

Publications (2)

Publication Number

Publication Date

US20200218959A1

US20200218959A1 ( en )

2020-07-09

US11651199B2

true

US11651199B2 ( en )

2023-05-16

Family

ID=66101626

Family Applications (1)

Application Number

Title

Priority Date

Filing Date

US16/644,446

Active

2038-12-03

US11651199B2

( en )

2017-10-09

2017-12-19

Method, apparatus and system to perform action recognition with a spiking neural network

Country Status (2)

Country

Link

US

( 1 )

US11651199B2

( en )

WO

( 1 )

WO2019074532A1

( en )

Families Citing this family (40)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US10051344B2

( en )

*

2016-09-27

2018-08-14

Clarifai, Inc.

Prediction model training via live stream concept association

US11150327B1

( en )

*

2018-01-12

2021-10-19

Hrl Laboratories, Llc

System and method for synthetic aperture radar target recognition using multi-layer, recurrent spiking neuromorphic networks

US11544577B1

( en )

*

2018-01-26

2023-01-03

Amazon Technologies, Inc.

Adaptable filtering for edge-based deep learning models

US11074453B2

( en )

*

2018-01-31

2021-07-27

Hewlett Packard Enterprise Development Lp

Video active region batching

US11019355B2

( en )

*

2018-04-03

2021-05-25

Electronics And Telecommunications Research Institute

Inter-prediction method and apparatus using reference frame generated based on deep learning

US10970882B2

( en )

2019-07-24

2021-04-06

At&T Intellectual Property I, L.P.

Method for scalable volumetric video coding

US10979692B2

( en )

*

2019-08-14

2021-04-13

At&T Intellectual Property I, L.P.

System and method for streaming visible portions of volumetric video

US11449578B2

( en )

*

2019-09-27

2022-09-20

Botty Todorov DIMANOV

Method for inspecting a neural network

US20230019839A1

( en )

*

2019-12-11

2023-01-19

Inait Sa

Constructing and operating an artificial recurrent neural network

CN115552307B

( en )

2020-05-14

2026-03-27

索尼集团公司

Autofocus imaging circuit, autofocus imaging device and method for event cameras

EP3910558A3

( en )

*

2020-05-14

2022-03-02

Xephor Solutions GmbH

Computer system and method

EP3913534B1

( en )

2020-05-22

2023-11-22

Tata Consultancy Services Limited

System and method for real-time radar-based action recognition using spiking neural network(snn)

SE544261C2

( en )

2020-06-16

2022-03-15

IntuiCell AB

A computer-implemented or hardware-implemented method of entity identification, a computer program product and an apparatus for entity identification

EP3926544B1

( en )

2020-06-18

2024-03-13

Tata Consultancy Services Limited

System and method of gesture recognition using a reservoir based convolutional spiking neural network

US11863221B1

( en )

*

2020-07-14

2024-01-02

Hrl Laboratories, Llc

Low size, weight and power (swap) efficient hardware implementation of a wide instantaneous bandwidth neuromorphic adaptive core (NeurACore)

US12566244B1

( en )

*

2020-07-13

2026-03-03

Hrl Laboratories, Llc

System and method for real-time radar range-doppler map learning

US11776319B2

( en )

2020-07-14

2023-10-03

Fotonation Limited

Methods and systems to predict activity in a sequence of images

US12057989B1

( en )

*

2020-07-14

2024-08-06

Hrl Laboratories, Llc

Ultra-wide instantaneous bandwidth complex neuromorphic adaptive core processor

CN111967577B

( en )

*

2020-07-29

2024-04-05

华北电力大学

Energy Internet scene generation method based on variation self-encoder

US11282221B1

( en )

*

2020-09-22

2022-03-22

Varian Medical Systems, Inc.

Image contouring using spiking neural networks

US11972699B1

( en )

*

2020-09-25

2024-04-30

Nathaniel McLaughlin

Virtualized education system that tracks student attendance and provides a remote learning platform

CN112464807A

( en )

*

2020-11-26

2021-03-09

北京灵汐科技有限公司

Video motion recognition method and device, electronic equipment and storage medium

KR102635968B1

( en )

*

2020-11-27

2024-02-13

충북대학교 산학협력단

Apparatus and method for generating spike tarin signal

CN112953972A

( en )

*

2021-04-08

2021-06-11

周士博

Network intrusion detection method of single-pulse neural network time domain coding neurons

CN115545063A

( en )

*

2021-06-29

2022-12-30

北京灵汐科技有限公司

Method and device for generating DVS data of dynamic visual sensor

SE2151099A1

( en )

*

2021-09-03

2023-03-04

IntuiCell AB

A computer-implemented or hardware-implemented method, a computer program product, an apparatus, a transfer function unit and a system for identification or separation of entities

KR102907798B1

( en )

*

2021-10-25

2026-01-07

한국전자통신연구원

Encoder and operation method thereof

CN114155478B

( en )

*

2022-02-09

2022-05-10

苏州浪潮智能科技有限公司

Emotion recognition method, device, system and computer-readable storage medium

US12579809B2

( en )

*

2022-02-15

2026-03-17

Sony Group Corporation

Information processing devices, information processing methods and video streaming system

US12526538B2

( en )

*

2022-04-13

2026-01-13

Shenzhen SynSense Technology Co., Ltd.

Frame image conversion spike system

CN114466153B

( en )

*

2022-04-13

2022-09-09

深圳时识科技有限公司

Self-adaptive pulse generation method and device, brain-like chip and electronic equipment

CN114495178B

( en )

*

2022-04-14

2022-06-21

深圳时识科技有限公司

Pulse sequence randomization method and device, brain-like chip and electronic equipment

US20250317661A1

( en )

*

2022-05-06

2025-10-09

IntuiCell AB

A data processing system for processing pixel data to be indicative of contrast

CN114638360A

( en )

*

2022-05-19

2022-06-17

之江实验室

Computing platform and method for impulse neural network learning and simulation

CN115146765A

( en )

*

2022-07-07

2022-10-04

成都信息工程大学

Spike recognition method based on deep learning and stride feature extraction

CN115171221B

( en )

*

2022-09-06

2022-12-06

上海齐感电子信息科技有限公司

Action recognition method and action recognition system

DE102023102528A1

( en )

2023-02-02

2024-08-08

Valeo Schalter Und Sensoren Gmbh

METHOD FOR RECONSTRUCTING A PROPERTY OF AN IMAGE AND COMPUTER DEVICE

CN116080688B

( en )

*

2023-03-03

2023-07-11

北京航空航天大学

Brain-inspiring-like intelligent driving vision assisting method, device and storage medium

CN116681725B

( en )

*

2023-05-30

2024-11-22

电子科技大学

A human body movement detection method and system based on pulse neural network

CN118587561B

( en )

*

2024-08-06

2024-10-22

杭州电子科技大学

Action recognition migration attack method based on self-adaptive gradient time sequence characteristic pruning

Citations (6)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

WO2012167158A1

( en )

2011-06-02

2012-12-06

Brain Corporation

Apparatus and methods for pulse-code invariant object recognition

US20140129496A1

( en )

2012-11-06

2014-05-08

Qualcomm Incorporated

Methods and apparatus for identifying spectral peaks in a neuronal spiking representation of a signal

US20140229411A1

( en )

*

2013-02-08

2014-08-14

Brain Corporation

Spiking network apparatus and method with bimodal spike-timing dependent plasticity

US20150269481A1

( en )

*

2014-03-24

2015-09-24

Qualcomm Incorporated

Differential encoding in neural networks

US20150341633A1

( en )

*

2014-05-22

2015-11-26

Brain Corporation

Apparatus and methods for real time estimation of differential motion in live video

US20160232430A1

( en )

*

2014-05-29

2016-08-11

International Business Machines Corporation

Scene understanding using a neurosynaptic system

2017

2017-12-19

US

US16/644,446

patent/US11651199B2/en

active

Active

2017-12-19

WO

PCT/US2017/067402

patent/WO2019074532A1/en

not_active

Ceased

Patent Citations (6)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

WO2012167158A1

( en )

2011-06-02

2012-12-06

Brain Corporation

Apparatus and methods for pulse-code invariant object recognition

US20140129496A1

( en )

2012-11-06

2014-05-08

Qualcomm Incorporated

Methods and apparatus for identifying spectral peaks in a neuronal spiking representation of a signal

US20140229411A1

( en )

*

2013-02-08

2014-08-14

Brain Corporation

Spiking network apparatus and method with bimodal spike-timing dependent plasticity

US20150269481A1

( en )

*

2014-03-24

2015-09-24

Qualcomm Incorporated

Differential encoding in neural networks

US20150341633A1

( en )

*

2014-05-22

2015-11-26

Brain Corporation

Apparatus and methods for real time estimation of differential motion in live video

US20160232430A1

( en )

*

2014-05-29

2016-08-11

International Business Machines Corporation

Scene understanding using a neurosynaptic system

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party

Title

Abbott, LF et al., " Building functional networks of spiking model neurons ", Published in Nature Neuroscience 19(3): 350-355, Mar. 2016, 16 pgs.

International Preliminary Report on Patentability from PCT/US2017/067402 dated Apr. 23, 2020, 9 pgs.

International Search Report and Written Opinion from PCT/US2017/067402 dated Jul. 9, 2018, 14 pgs.

Also Published As

Publication number

Publication date

WO2019074532A1

( en )

2019-04-18

US20200218959A1

( en )

2020-07-09

Similar Documents

Publication

Publication Date

Title

US20200218959A1

( en )

2020-07-09

Method, apparatus and system to perform action recognition with a spiking neural network

Cordone et al.

2022

Object detection with spiking neural networks on automotive event data

US11403479B2

( en )

2022-08-02

Feedback signaling to facilitate data classification functionality of a spiking neural network

US10902615B2

( en )

2021-01-26

Hybrid and self-aware long-term object tracking

US11568241B2

( en )

2023-01-31

Device, system and method for varying a synaptic weight with a phase differential of a spiking neural network

Yang et al.

2019

Unsupervised moving object detection via contextual information separation

Schliebs et al.

2013

Evolving spiking neural network—a survey

US20170236027A1

( en )

2017-08-17

Intelligent biomorphic system for pattern recognition with autonomous visual feature extraction

US20190251360A1

( en )

2019-08-15

Method for analysing media content

US20170337469A1

( en )

2017-11-23

Anomaly detection using spiking neural networks

US20190042942A1

( en )

2019-02-07

Hybrid spiking neural network and support vector machine classifier

US11256954B2

( en )

2022-02-22

System and method of gesture recognition using a reservoir based convolutional spiking neural network

US20250013877A1

( en )

2025-01-09

Data Processing Method and Apparatus

JP2017516192A

( en )

2017-06-15

Differential coding in neural networks.

Leroux et al.

2015

Resource-constrained classification using a cascade of neural network layers

Jain et al.

2021

Human activity recognition with videos using deep learning

CN119888431B

( en )

2025-07-18

Improved flying object target detection method and device based on spiking neural network

Fallas-Moya et al.

2018

Object recognition using hierarchical temporal memory

Kumar et al.

2025

Maize Leaf Disease Detection and Classification Using CNN

Sravanthi et al.

2020

An efficient classifier using machine learning technique for individual action identification

Jain et al.

2021

Face emotion detection using deep learning

Desai et al.

2023

Enhancing plant leaf disease classification performance through efficientNetB3 and hyperparameter optimization

Rasamuel et al.

2019

Specialized visual sensor coupled to a dynamic neural field for embedded attentional process

Limberg et al.

2018

Efficient accuracy estimation for instance-based incremental active learning.

Haindavi et al.

2023

Human action recognition by learning spatio-temporal features with deep neural networks

Legal Events

Date

Code

Title

Description

2020-03-04

FEPP

Fee payment procedure

Free format text : ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

2020-04-06

STPP

Information on status: patent application and granting procedure in general

Free format text : APPLICATION DISPATCHED FROM PREEXAM, NOT YET DOCKETED

2021-08-20

STPP

Information on status: patent application and granting procedure in general

Free format text : DOCKETED NEW CASE - READY FOR EXAMINATION

2022-05-11

STPP

Information on status: patent application and granting procedure in general

Free format text : NON FINAL ACTION MAILED

2022-08-11

STPP

Information on status: patent application and granting procedure in general

Free format text : RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER

2022-10-21

STPP

Information on status: patent application and granting procedure in general

Free format text : FINAL REJECTION MAILED

2022-12-30

STPP

Information on status: patent application and granting procedure in general

Free format text : DOCKETED NEW CASE - READY FOR EXAMINATION

2023-04-26

STCF

Information on status: patent grant

Free format text : PATENTED CASE

Related documents

• Generating augmented reality experiences utilizing physical objects to … — Meta Platforms Technologies, Llc (US20260037214A1)#653960
• Virtual reality system using multiple force arrays for a solver — DeepMotion, Inc. (US10657696B2)#653961
• Tracking position of device inside-out for virtual reality interactivity — Sony Interactive Entertainment America Llc (US10535153B2)#653962
• Augmented reality system with spatialized audio tied to user manipulated … — Magic Leap, Inc. (US11231770B2)#653963
• Big data-based autonomous flight drone system and autonomous flight method … — Rgblab Co., Ltd. (US20210390867A1)#653964