Conceptio › Archive › Google Patents

Google Patentsopen access

Neural network and method of neural network training — Progress, Inc. (US9619749B2)

Progress, Inc. · Google Patents

Google Patents · Patents · License: Open Access

patent, google patents, intellectual property, US9619749B2, Progress, Inc., Dmitri Pescianschi, en, 2017

ABSTRACT

Abstract

A neural network includes a plurality of inputs for receiving input signals, and synapses connected to the inputs and having corrective weights established by a memory element that retains a respective weight value. The network additionally includes distributors. Each distributor is connected to one of the inputs for receiving the respective input signal and selects one or more corrective weights in correlation with the input value. The network also includes neurons. Each neuron has an output connected with at least one of the inputs via one synapse and generates a neuron sum by summing corrective weights selected from each synapse connected to the respective neuron. The output of each neuron provides the respective neuron sum to establish operational output signal of the network. A method of operating a neural network includes processing data thereby and using modified corrective weight values established by a separate analogous neural network during training thereof.

Description

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation in part of U.S. Utility application Ser. No. 14/862,337 which is a continuation of International Application Serial No. PCT/US 2015/19236, which claims the benefit of U.S. Provisional Application Ser. No. 61/949,210 filed Mar. 6, 2014, and U.S. Provisional Application Ser. No. 62/106,389 filed Jan. 22, 2015, the entire content of which is hereby incorporated by reference. This application also claims the benefit of U.S. Provisional Application Ser. No. 62/173,163 filed Jun. 9, 2015, the entire content of which is similarly incorporated by reference.

TECHNICAL FIELD

The disclosure relates to an artificial neural network and a method of training the same.

BACKGROUND

In machine learning, the term âneural networkâ generally refers to software and/or computer architecture, i.e., the overall design or structure of a computer system or a microprocessor, including the hardware and software required to run it. Artificial neural networks can be a family of statistical learning algorithms inspired by biological neural networks, a.k.a., the central nervous systems of animals, in particular the brain. Artificial neural networks are primarily used to estimate or approximate generally unknown functions that can depend on a large number of inputs. Such neural networks have been used for a wide variety of tasks that are difficult to resolve using ordinary rule-based programming, including computer vision and speech recognition.

Artificial neural networks are generally presented as systems of âneuronsâ which can compute values from inputs, and, as a result of their adaptive nature, are capable of machine learning, as well as pattern recognition. Each neuron frequently connects with several inputs through synapses having synaptic weights.

Neural networks are not programmed as typical software and hardware, but are trained. Such training is typically accomplished via analysis of a sufficient number of representative examples and by statistical or algorithmic selection of synaptic weights, so that a given set of input images corresponds to a given set of output images. A common criticism of classical neural networks is that significant time and other resources are frequently required for their training.

Various artificial neural networks are described in the following U.S. Pat. Nos. 4,979,124; 5,479,575; 5,493,688; 5,566,273; 5,682,503; 5,870,729; 7,577,631; and 7,814,038.

SUMMARY

A neural network includes a plurality of network inputs, such that each input is configured to receive an input signal having an input value. The neural network also includes a plurality of synapses, wherein each synapse is connected to one of the plurality of inputs and includes a plurality of corrective weights, wherein each corrective weight is established by a memory element that retains a respective weight value. The neural network additionally includes a set of distributors. Each distributor is operatively connected to one of the plurality of inputs for receiving the respective input signal and is configured to select one or more corrective weights from the plurality of corrective weights in correlation with the input value. The neural network also includes a set of neurons. Each neuron has at least one output and is connected with at least one of the plurality of inputs via one of the plurality of synapses synapse and is configured to add up the weight values of the corrective weights selected from each synapse connected to the respective neuron and thereby generate a neuron sum. The output of each neuron provides the respective neuron sum to establish an operational output signal of the neural network.

The neural network may also include a weight correction calculator configured to receive a desired output signal having a value, determine a deviation of the neuron sum from the desired output signal value, and modify respective corrective weight values established by the corresponding memory elements using the determined deviation. In such a case, adding up the modified corrective weight values to determine the neuron sum is intended to minimize the deviation of the neuron sum from the desired output signal value to thereby generate a trained neural network.

The trained neural network can be configured to receive supplementary training using solely a supplementary input signal having a value and a corresponding supplementary desired output signal.

Either during training or before the supplementary training of the neural network, each of the plurality of synapses can be configured to accept one or more additional corrective weights established by the respective memory elements.

The neural network can be configured to remove from the respective synapses, during or after training of the neural network, one or more corrective weights established by the respective memory elements. Such removal of some corrective weights can permit the neural network to retain only a number of memory elements required to operate the neural network.

The neural network can be configured to accept at least one of an additional input, an additional synapse, and an additional neuron before or during training of the neural network to thereby expand operational parameters of the neural network.

The neural network can be configured to remove at least one of an input, a synapse, and a neuron before, during, or after training of the neural network. Such ability to remove neural network elements that are not being used by the network is intended to simplify structure and modify operational parameters of the neural network without loss of the network's output quality.

Each memory element can be established by an electrical device characterized by an electrical and/or a magnetic characteristic configured to define a respective weight value. Such a characteristic can be resistance, impedance, capacity, magnetic field, induction, electric field intensity, etc. The respective electrical and/or magnetic characteristic of each device can be configured to be varied during training of the neural network. Additionally, the weight correction calculator can modify the respective corrective weight values by varying the respective at least one of the electrical and the magnetic characteristic of the corresponding electrical devices.

The electrical device can be configured as one of a resistor, a memistor, a memristor, a transistor, a capacitor, a field-effect transistor, a photoresistor, such as a light-dependent resistor (LDR), or a magnetic dependent resistor (MDR).

Each memory element can be established by a block of electrical resistors and include a selector device configured to select one or more electrical resistors from the block using the determined deviation to establish each corrective weight.

The block of electrical resistors can additionally include electrical capacitors. In other words, each memory element can be established by a block having both, electrical resistors and electrical capacitors. The selector device can then be additionally configured to select capacitors using the determined deviation to establish each corrective weight.

The neural network can be configured as one of an analog, digital, and digital-analog network. In such a network, at least one of the plurality of inputs, the plurality of synapses, the memory elements, the set of distributors, the set of neurons, the weight correction calculator, and the desired output signal can be configured to operate in an analog, digital, and digital-analog format.

In the case where the neural network is configured as the analog network, each neuron can be established by one of a series and a parallel communication channel, such as an electrical wire, or a series or parallel bus.

The weight correction calculator can be established as a set of differential amplifiers. Furthermore, each differential amplifier can be configured to generate a respective correction signal.

Each of the distributors can be a demultiplexer configured to select one or more corrective weights from the plurality of corrective weights in response to the received input signal.

Each distributor can be configured to convert the received input signal into a binary code and select one or more corrective weights from the plurality of corrective weights in correlation with the binary code.

The neural network can be programmed into an electronic device having a memory, and wherein each memory element is stored in the memory of the electronic device.

A method of operating a utility neural network is also disclosed. The method includes processing data via the utility neural network using modified corrective weight values established by a separate analogous neural network during training thereof. The method also includes establishing an operational output signal of the utility neural network using the modified corrective weight values established by the separate analogous neural network.

For use of modified corrective weight values by the utility neural network, the separate analogous neural network was trained via receiving, via an input to the neural network, a training input signal having a training input value; communicating the training input signal to a distributor operatively connected to the input; selecting, via the distributor, in correlation with the training input value, one or more corrective weights from a plurality of corrective weights, wherein each corrective weight is defined by a weight value and is positioned on a synapse connected to the input; adding up the weight values of the selected corrective weights, via a neuron connected with the input via the synapse and having at least one output, to generate a neuron sum; receiving, via a weight correction calculator, a desired output signal having a value; determining, via the weight correction calculator, a deviation of the neuron sum from the desired output signal value; and modifying, via the weight correction calculator, respective corrective weight values using the determined deviation to establish the modified corrective weight values, such that adding up the modified corrective weight values to determine the neuron sum minimizes the deviation of the neuron sum from the desired output signal value to thereby train the neural network.

The utility neural network and the trained separate analogous neural network can include a matching neural network structure including a number of inputs, corrective weights, distributors, neurons, and synapses.

In each of the utility neural network and the trained separate analogous neural network, each corrective weight can be established by a memory element that retains a respective weight value.

The above features and advantages, and other features and advantages of the present disclosure, will be readily apparent from the following detailed description of the embodiment(s) and best mode(s) for carrying out the described disclosure when taken in connection with the accompanying drawings and appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic illustration of a prior art, classical artificial neural network.

FIG. 2 is a schematic illustration of a âprogressive neural networkâ (p-net) having a plurality of synapses, a set of distributors, and a plurality of corrective weights associated with each synapse.

FIG. 3A is a schematic illustration of a portion of the p-net shown in FIG. 2 , having a plurality of synapses and one synaptic weight positioned upstream of each distributor.

FIG. 3B is a schematic illustration of a portion of the p-net shown in FIG. 2 , having a plurality of synapses and a set of synaptic weights positioned downstream of the respective plurality of corrective weights.

FIG. 3C is a schematic illustration of a portion of the p-net shown in FIG. 2 , having a plurality of synapses and one synaptic weight positioned upstream of each distributor and a set of synaptic weights positioned downstream of the respective plurality of corrective weights.

FIG. 4A is a schematic illustration of a portion of the p-net shown in FIG. 2 , having a single distributor for all synapses of a given input and one synaptic weight positioned upstream of each distributor.

FIG. 4B is a schematic illustration of a portion of the p-net shown in FIG. 2 , having a single distributor for all synapses of a given input and a set of synaptic weights positioned downstream of the respective plurality of corrective weights.

FIG. 4C is a schematic illustration of a portion of the p-net shown in FIG. 2 , having a single distributor for all synapses of a given input, and having one synaptic weight positioned upstream of each distributor and a set of synaptic weights positioned downstream of the respective plurality of corrective weights.

FIG. 5 is a schematic illustration of division of input signal value range into individual intervals in the p-net shown in FIG. 2 .

FIG. 6A is a schematic illustration of one embodiment of a distribution for values of coefficient of impact of corrective weights in the p-net shown in FIG. 2 .

FIG. 6B is a schematic illustration of another embodiment of the distribution for values of coefficient of impact of corrective weights in the p-net shown in FIG. 2 .

FIG. 6C is a schematic illustration of yet another embodiment of the distribution for values of coefficient of impact of corrective weights in the p-net shown in FIG. 2 .

FIG. 7 is a schematic illustration of an input image for the p-net shown in FIG. 2 , as well as one corresponding table representing the image in the form of digital codes and another corresponding table representing the same image as a set of respective intervals.

FIG. 8 is a schematic illustration of an embodiment of the p-net shown in FIG. 2 trained for recognition of two distinct images, wherein the p-net is configured to recognize a picture that includes some features of each image;

FIG. 9 is a schematic illustration of an embodiment of the p-net shown in FIG. 2 with an example of distribution of synaptic weights around a âcentralâ neuron.

FIG. 10 is a schematic illustration of an embodiment of the p-net shown in FIG. 2 , depicting a uniform distribution of training deviation between correct

CROSS-REFERENCE TO RELATED APPLICATION(S)

TECHNICAL FIELD

The disclosure relates to an artificial neural network and a method of training the same.

BACKGROUND

Various artificial neural networks are described in the following U.S. Pat. Nos. 4,979,124; 5,479,575; 5,493,688; 5,566,273; 5,682,503; 5,870,729; 7,577,631; and 7,814,038.

SUMMARY

The trained neural network can be configured to receive supplementary training using solely a supplementary input signal having a value and a corresponding supplementary desired output signal.

The weight correction calculator can be established as a set of differential amplifiers. Furthermore, each differential amplifier can be configured to generate a respective correction signal.

Each of the distributors can be a demultiplexer configured to select one or more corrective weights from the plurality of corrective weights in response to the received input signal.

The neural network can be programmed into an electronic device having a memory, and wherein each memory element is stored in the memory of the electronic device.

In each of the utility neural network and the trained separate analogous neural network, each corrective weight can be established by a memory element that retains a respective weight value.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic illustration of a prior art, classical artificial neural network.

FIG. 3A is a schematic illustration of a portion of the p-net shown in FIG. 2 , having a plurality of synapses and one synaptic weight positioned upstream of each distributor.

FIG. 5 is a schematic illustration of division of input signal value range into individual intervals in the p-net shown in FIG. 2 .

FIG. 6A is a schematic illustration of one embodiment of a distribution for values of coefficient of impact of corrective weights in the p-net shown in FIG. 2 .

FIG. 6B is a schematic illustration of another embodiment of the distribution for values of coefficient of impact of corrective weights in the p-net shown in FIG. 2 .

FIG. 6C is a schematic illustration of yet another embodiment of the distribution for values of coefficient of impact of corrective weights in the p-net shown in FIG. 2 .

FIG. 9 is a schematic illustration of an embodiment of the p-net shown in FIG. 2 with an example of distribution of synaptic weights around a âcentralâ neuron.

FIG. 10 is a schematic illustration of an embodiment of the p-net shown in FIG. 2 , depicting a uniform distribution of training deviation between corrective weights.

FIG. 11 is a schematic illustration of an embodiment of the p-net shown in FIG. 2 , employing modification of the corrective weights during p-net training.

FIG. 12 is a schematic illustration of an embodiment of the p-net shown in FIG. 2 , wherein the basic algorithm generates a primary set of output neuron sums, and wherein the generated set is used to generate several âwinnerâ sums with either retained or increased values and the contribution of remaining sums is negated.

FIG. 13 is a schematic illustration of an embodiment of the p-net shown in FIG. 2 recognizing a complex image with elements of multiple images.

FIG. 14 is a schematic illustration of a model for object oriented programming for the p-net shown in FIG. 2 using Unified Modeling Language (UML).

FIG. 15 is a schematic illustration of a general formation sequence of the p-net shown in FIG. 2 .

FIG. 16 is a schematic illustration of representative analysis and preparation of data for formation of the p-net shown in FIG. 2 .

FIG. 17 is a schematic illustration of representative input creation permitting interaction of the p-net shown in FIG. 2 with input data during training and p-net application.

FIG. 18 is a schematic illustration of representative creation of neuron units for the p-net shown in FIG. 2 .

FIG. 19 is a schematic illustration of representative creation of each synapse connected with the neuron units.

FIG. 20 is a schematic illustration of training the p-net shown in FIG. 2 .

FIG. 21 is a schematic illustration of neuron unit training in the p-net shown in FIG. 2 .

FIG. 22 is a schematic illustration of extending of neuron sums during training of the p-net shown in FIG. 2 .

FIG. 23 is a flow diagram of a method used to train the p-net shown in FIGS. 2-22 .

FIG. 24 is a schematic illustration of a specific embodiment of the p-net having each of the plurality of corrective weights established by a memory element; the p-net being depicted in the process of network training.

FIG. 25 is a schematic illustration of a specific embodiment of the p-net having each of the plurality of corrective weights established by the memory element; the p-net being depicted in the process of image recognition.

FIG. 26 is a schematic illustration of a representative p-net using memristors during a first stage of training.

FIG. 27 is a schematic illustration of the representative p-net using memristors during a second stage of training.

FIG. 28 is a schematic illustration of twin parallel branches of memristors in the representative p-net.

FIG. 29 is a schematic illustration of the representative p-net using resistors.

FIG. 30 is a schematic illustration of one embodiment of the memory element configured as a resistor in the p-net.

FIG. 31 is a schematic illustration of another embodiment of the memory element configured as a resistor in the p-net.

FIG. 32 is a schematic illustration of another embodiment of the memory element configured as variable impedance in the p-net.

FIG. 33 is a flow diagram of a method used to operate the neural network shown in FIGS. 2-22 and 24-32 .

DETAILED DESCRIPTION

A classical artificial neural network 10 , as shown in FIG. 1 , typically includes input devices 12 , synapses 14 with synaptic weights 16 , neurons 18 , including an adder 20 and activation function device 22 , neuron outputs 24 and weight correction calculator 26 . Each neuron 18 is connected through synapses 14 to two or more input devices 12 . The values of synaptic weights 16 are commonly represented using electrical resistance, conductivity, voltage, electric charge, magnetic property, or other parameters.

Supervised training of the classical neural network 10 is generally based on an application of a set of training pairs 28 . Each training pair 28 commonly consists of an input image 28 - 1 and a desired output image 28 - 2 , a.k.a., a supervisory signal. Training of the classical neural network 10 is typically provided as follows. An input image in the form of a set of input signals (I 1 -I m ) enters the input devices 12 and is transferred to the synaptic weights 16 with initial weights (W 1 ). The value of the input signal is modified by the weights, typically by multiplying or dividing each signal (I 1 -I m ) value by the respective weight. From the synaptic weights 16 , modified input signals are transferred either to the respective neurons 18 . Each neuron 18 receives a set of signals from a group of synapses 14 related to the subject neuron 18 . The adder 20 included in the neuron 18 sums up all the input signals modified by the weights and received by the subject neuron. Activation function devices 22 receive the respective resultant neuron sums and modify the sums according to mathematical function(s), thus forming respective output images as sets of neuron output signals (Î£F 1 . . . F n ).

The obtained neuron output image defined by the neuron output signals (Î£F 1 . . . Î£F n ) is compared by a weight correction calculator 26 with pre-determined desired output images (O 1 -O n ). Based on the determined difference between the obtained neuron output image Î£F n and the desired output image O n , correction signals for changing the synaptic weights 16 are formed using a pre-programmed algorithm. After corrections are made to all the synaptic weights 16 , the set of input signals is reintroduced to the neural network 10 and new corrections are made. The above cycle is repeated until the difference between the obtained neuron output image (I 1 -I m ) and the desired output image O n is determined to be less than some predetermined error. One cycle of network training with all the individual images is typically identified as a âtraining epochâ. Generally, with each training epoch, the magnitude of error is reduced. However, depending on the number of individual inputs (I 1 -I m ), as well as the number of inputs and outputs, training of the classical neural network 10 may require a significant number of training epochs, which, in some cases, may be as great as hundreds of thousands.

A variety of classical neural networks exist, including Hopfield network, Restricted Boltzmann Machine, Radial basis function network, and recurrent neural network. Specific tasks of classification and clustering require a specific type of neural network, the Self-Organizing Maps that use only input images as network input training information, whereas the desired output image, corresponding to a certain input image is formed directly during the training process based on a single winning neuron having an output signal with the maximum value.

As noted above, one of the main concerns with existing, classical neural networks, such as the neural network 10 , is that successful training thereof may require a significant duration of time. Some additional concerns with classical networks may be a large consumption of computing resources, which would in turn drive the need for powerful computers. Additional concerns are an inability to increase the size of the network without full retraining of the network, a predisposition to such phenomena as ânetwork paralysisâ and âfreezing at a local minimumâ, which make it impossible to predict if a specific neural network would be capable of being trained with a given set of images in a given sequence. Also there may be limitations related to specific sequencing of images being introduced during training, where changing the order of introduction of training images may lead to network freezes, as well as an inability to perform additional training of an already trained network.

Referring to the remaining drawings, wherein like reference numbers refer to like components, FIG. 2 shows a schematic view of a progressive neural network, thereafter âprogressive networkâ, or âp-netâ 100 . The p- net 100 includes a plurality or a set of inputs 102 of the p-net. Each input 102 is configured to receive an input signal 104 , wherein the input signals are represented as I 1 , I 2 . . . I m in FIG. 2 . Each input signal I 1 , I 2 . . . I m represents a value of some characteristic(s) of an input image 106 , for example, a magnitude, frequency, phase, signal polarization angle, or association with different parts of the input image 106 . Each input signal 104 has an input value, wherein together the plurality of input signals 104 generally describes the input image 106 .

Each input value may be within a value range that lies between ââ and +â and can be set in digital and/or analog forms. The range of the input values may depend on a set of training images. In the simplest case, the range input values could be the difference between the smallest and largest values of input signals for all training images. For practical reasons, the range of the input values may be limited by eliminating input values that are deemed too high. For example, such limiting of the range of the input values may be accomplished via known statistical methods for variance reduction, such as importance sampling. Another example of limiting the range of the input values may be designation of all signals that are lower than a predetermined minimum level to a specific minimum value and designation of all signals exceeding a predetermined maximum level to a specific maximum value.

The p- net 100 also includes a plurality or a set of synapses 118 . Each synapse 118 is connected to one of the plurality of inputs 102 , includes a plurality of corrective weights 112 , and may also include a synaptic weight 108 , as shown in FIG. 2 . Each corrective weight 112 is defined by a respective weight value. The p- net 100 also includes a set of distributors 114 . Each distributor 114 is operatively connected to one of the plurality of inputs 102 for receiving the respective input signal 104 . Additionally, each distributor 114 is configured to select one or more corrective weights from the plurality of corrective weights 112 in correlation with the input value.

The p- net 100 additionally includes a set of neurons 116 . Each neuron 116 has at least one output 117 and is connected with at least one of the plurality of inputs 102 via one synapse 118 . Each neuron 116 is configured to add up or sum the corrective weight values of the corrective weights 112 selected from each synapse 118 connected to the respective neuron 116 and thereby generate and output a neuron sum 120 , otherwise designated as Î£n. A separate distributor 114 can be used for each synapse 118 of a given input 102 , as shown in FIGS. 3A, 3B, and 3C , or a single distributor can be used for all such synapses, as shown in FIGS. 4A, 4B, and 4C . During formation or setup of the p- net 100 , all corrective weights 112 are assigned initial values, which can change during the process of p-net training. The initial value of the corrective weight 112 may be assigned as in the classical neural network 10 , for example, the weights may be selected randomly, calculated with the help of a pre-determined mathematical function, selected from a predetermined template, etc.

The p- net 100 also includes a weight correction calculator 122 . The weight correction calculator 122 is configured to receive a desired, i.e., predetermined, output signal 124 having a signal value and representing a portion of an output image 126 . The weight correction calculator 122 is also configured to determine a deviation 128 of the neuron sum 120 from the value of the desired output signal 124 , a.k.a., training error, and modify respective corrective weight values using the determined deviation 128 . Thereafter, summing the modified corrective weight values to determine the neuron sum 120 minimizes the deviation of the subject neuron sum from the value of the desired output signal 124 and, as a result, is effective for training the p- net 100 .

For analogy with the classical network 10 discussed with respect to FIG. 1 , the deviation 128 may also be described as the training error between the determined neuron sum 120 and the value of the desired output signal 124 . In comparison with the classical neural network 10 discussed with respect to FIG. 1 , in the p-net 100 the input values of the input signal 104 only change in the process of general network setup, and are not changed during training of the p-net. Instead of changing the input value, training of the p- net 100 is provided by changing the values 112 of the corrective weights 112 . Additionally, although each neuron 116 includes a summing function, where the neuron adds up the corrective weight values, the neuron 116 does not require, and, in fact, is characterized by absence of an activation function, such as provided by the activation function device 22 in the classical neural network 10 .

In the classical neural network 10 , weight correction during training is accomplished by changing synaptic weights 16 , while in the p-net 100 corresponding weight correction is provided by changing corrective weights values 112 , as shown in FIG. 2 . The respective corrective weights 112 may be included in weight correction blocks 110 positioned on all or some of the synapses 118 . In neural network computer emulations, each synaptic and corrective weight may be represented either by a digital device, such as a memory cell, and/or by an analog device. In neural network software emulations, the values of the corrective weights 112 may be provided via an appropriate programmed algorithm, while in hardware emulations, known methods for memory control could be used.

In the p- net 100 , the deviation 128 of the neuron sum 120 from the desired output signal 124 may be represented as a mathematically computed difference therebetween. Additionally, the generation of the respective modified corrective weights 112 may include apportionment of the computed difference to each corrective weight used to generate the neuron sum 120 . In such an embodiment, the generation of the respective modified corrective weights 112 will permit the neuron sum 120 to be converged on the desired output signal value within a small number of epochs, in some cases needing only a single epoch, to rapidly train the p- net 100 . In a specific case, the apportionment of the mathematical difference among the corrective weights 112 used to generate the neuron sum 120 may include dividing the determined difference equally between each corrective weight used to generate the respective neuron sum 120 .

In a separate embodiment, the determination of the deviation 128 of the neuron sum 120 from the desired output signal value may include division of the desired output signal value by the neuron sum to thereby generate a deviation coefficient. In such a specific case, the modification of the respective modified corrective weights 112 includes multiplication of each corrective weight used to generate the neuron sum 120 by the deviation coefficient. Each distributor 114 may additionally be configured to assign a plurality of coefficients of impact 134 to the plurality of corrective weights 112 . In the present embodiment, each coefficient of impact 134 may be assigned to one of the plurality of corrective weights 112 in some predetermined proportion to generate the respective neuron sum 120 . For correspondence with each respective corrective weight 112 , each coefficient of impact 134 may be assigned a âC i,d,n â nomenclature, as shown in the Figures.

Each of the plurality of coefficients of impact 134 corresponding to the specific synapse 118 is defined by a respective impact distribution function 136 . The impact distribution function 136 may be same either for all coefficients of impact 134 or only for the plurality of coefficients of impact 134 corresponding a specific synapse 118 . Each of the plurality of input values may be received into a value range 138 divided into intervals or sub-divisions âdâ according to an interval distribution function 140 , such that each input value is received within a respective interval âdâ and each corrective weight corresponds to one of such intervals. Each distributor 114 may use the respective received input value to select the respective interval âdâ, and to assign the respect

CLAIMS

Claims ( 20 )

The invention claimed is:

1. A neural network comprising:

a plurality of inputs of the neural network, each input configured to receive an input signal having an input value;

a plurality of synapses, wherein each synapse is connected to one of the plurality of inputs and includes a plurality of corrective weights, wherein each corrective weight is established by a memory element that retains a respective weight value;

a set of distributors, wherein each distributor is operatively connected to one of the plurality of inputs for receiving the respective input signal and is configured to select one or more corrective weights from the plurality of corrective weights in correlation with the input value; and

a set of neurons, wherein:

each neuron has at least one output and is connected with at least one of the plurality of inputs via one of the plurality of synapses;

each neuron is configured to add up the weight values of the corrective weights selected from each synapse connected to the respective neuron and thereby generate a neuron sum; and

the output of each neuron provides the respective neuron sum to establish an operational output signal of the neural network.

2. The neural network of claim 1 , further comprising a weight correction calculator configured to receive a desired output signal having a value, determine a deviation of the neuron sum from the desired output signal value, and modify respective corrective weight values established by the corresponding memory elements using the determined deviation, such that adding up the modified corrective weight values to determine the neuron sum minimizes the deviation of the neuron sum from the desired output signal value to thereby generate a trained neural network.

3. The neural network of claim 2 , wherein the trained neural network is configured to receive supplementary training using solely a supplementary input signal having a value and a corresponding supplementary desired output signal.

4. The neural network of claim 3 , wherein, during training or before the supplementary training of the neural network, each of the plurality of synapses is configured to accept one or more additional corrective weights established by the respective memory elements.

5. The neural network of claim 2 , wherein the neural network is configured to remove from the respective synapses, during or after training of the neural network, one or more corrective weights established by the respective memory elements to retain only a number of memory elements required to operate the neural network.

6. The neural network of claim 2 , wherein the neural network is configured to accept at least one of an additional input, an additional synapse, and an additional neuron before or during training of the neural network to thereby expand operational parameters of the neural network.

7. The neural network of claim 2 , wherein the neural network is configured to remove at least one of an input, a synapse, and a neuron that is not being used by the neural network before, during, or after training of the neural network to thereby simplify a structure of the neural network and modify operational parameters of the neural network.

8. The neural network of claim 2 , wherein each memory element is established by an electrical device characterized by at least one of an electrical and a magnetic characteristic configured to define a respective weight value wherein the respective at least one of the electrical and the magnetic characteristic of each electrical device is configured to be varied during training of the neural network, and wherein the weight correction calculator modifies the respective corrective weight values by varying the respective at least one of the electrical and the magnetic characteristic of the corresponding electrical devices.

9. The neural network of claim 8 , wherein the electrical device is configured as one of a resistor, a memistor, a memristor, a transistor, a capacitor, a field-effect transistor, a photoresistor, or a magnetic dependent resistor.

10. The neural network of claim 2 , wherein each memory element is established by a block of electrical resistors and includes a selector device configured to select one or more electrical resistors from the block using the determined deviation to establish each corrective weight.

11. The neural network of claim 10 , wherein each memory element is additionally established by a block of electrical capacitors, and wherein the selector device is additionally configured to select capacitors using the determined deviation to establish each corrective weight.

12. The neural network of claim 2 , wherein the neural network is configured as one of an analog, digital, and digital-analog network, such that at least one of the plurality of inputs, the plurality of synapses, the memory elements, the set of distributors, the set of neurons, the weight correction calculator, and the desired output signal is configured to operate in an analog, digital, and digital-analog format.

13. The neural network of claim 12 , wherein the neural network is configured as the analog network, and wherein each neuron is established by one of a series and a parallel communication channel.

14. The neural network of claim 2 , wherein the weight correction calculator is established as a set of differential amplifiers, and wherein each differential amplifier is configured to generate a respective correction signal.

15. The neural network of claim 1 , wherein each of the distributors is a demultiplexer configured to select one or more corrective weights from the plurality of corrective weights in response to the received input signal.

16. The neural network of claim 1 , wherein each distributor is configured to convert the received input signal into a binary code and select one or more corrective weights from the plurality of corrective weights in correlation with the binary code.

17. The neural network of claim 1 , wherein the neural network is programmed into an electronic device having a memory, and wherein each memory element is stored in the memory of the electronic device.

18. A method of operating a utility neural network, the method comprising:

processing data via the utility neural network using modified corrective weight values established by a separate analogous neural network during training thereof; and

establishing an operational output signal of the utility neural network using the modified corrective weight values established by the separate analogous neural network;

wherein the separate analogous neural network was trained via:

receiving, via an input to the neural network, a training input signal having a training input value;

communicating the training input signal to a distributor operatively connected to the input;

selecting, via the distributor, in correlation with the training input value, one or more corrective weights from a plurality of corrective weights, wherein each corrective weight is defined by a weight value and is positioned on a synapse connected to the input;

adding up the weight values of the selected corrective weights, via a neuron connected with the input via the synapse and having at least one output, to generate a neuron sum;

receiving, via a weight correction calculator, a desired output signal having a value;

determining, via the weight correction calculator, a deviation of the neuron sum from the desired output signal value; and

modifying, via the weight correction calculator, respective corrective weight values using the determined deviation to establish the modified corrective weight values, such that adding up the modified corrective weight values to determine the neuron sum minimizes the deviation of the neuron sum from the desired output signal value to thereby train the neural network.

19. The method according to claim 18 , wherein the utility neural network and the trained separate neural network include a matching neural network structure including a number of inputs, corrective weights, distributors, neurons, and synapses.

20. The method according to claim 18 , wherein, in each of the utility neural network and the trained separate neural network, each corrective weight is established by a memory element that retains a respective weight value, and wherein in the separate neural network the memory element retains its respective weight value following training.

US15/178,137

2014-03-06

2016-06-09

Neural network and method of neural network training

Active

US9619749B2

( en )

Priority Applications (8)

Application Number

Priority Date

Filing Date

Title

US15/178,137

US9619749B2

( en )

2014-03-06

2016-06-09

Neural network and method of neural network training

US15/449,614

US10423694B2

( en )

2014-03-06

2017-03-03

Neural network and method of neural network training

PCT/US2017/036758

WO2017214507A1

( en )

2016-06-09

2017-06-09

Neural network and method of neural network training

CN201780035716.7A

CN109416758A

( en )

2016-06-09

2017-06-09

The method of neural network and neural metwork training

EP17811082.1A

EP3469521A4

( en )

2016-06-09

2017-06-09

NEURONAL NETWORK AND METHOD FOR TRAINING A NEURONAL NETWORK

JP2018564317A

JP7041078B2

( en )

2016-06-09

2017-06-09

Neural networks, and neural network training methods

KR1020197000226A

KR102558300B1

( en )

2016-06-09

2017-06-09

Neural Networks and How to Train Neural Networks

US16/523,584

US11494653B2

( en )

2014-03-06

2019-07-26

Neural network and method of neural network training

Applications Claiming Priority (6)

Application Number

Priority Date

Filing Date

Title

US201461949210P

2014-03-06

US201562106389P

2015-01-22

PCT/US2015/019236

WO2015134900A1

( en )

2014-03-06

2015-03-06

Neural network and method of neural network training

US201562173163P

2015-06-09

US14/862,337

US9390373B2

( en )

2014-03-06

2015-09-23

Neural network and method of neural network training

US15/178,137

US9619749B2

( en )

2014-03-06

2016-06-09

Neural network and method of neural network training

Related Parent Applications (1)

Application Number

Title

Priority Date

Filing Date

US14/862,337

Continuation-In-Part

US9390373B2

( en )

2014-03-06

2015-09-23

Neural network and method of neural network training

Related Child Applications (1)

Application Number

Title

Priority Date

Filing Date

US15/449,614

Continuation-In-Part

US10423694B2

( en )

2014-03-06

2017-03-03

Neural network and method of neural network training

Publications (2)

Publication Number

Publication Date

US20160283842A1

US20160283842A1 ( en )

2016-09-29

US9619749B2

true

US9619749B2 ( en )

2017-04-11

Family

ID=56976540

Family Applications (1)

Application Number

Title

Priority Date

Filing Date

US15/178,137

Active

US9619749B2

( en )

2014-03-06

2016-06-09

Neural network and method of neural network training

Country Status (1)

Country

Link

( 1 )

US9619749B2

( en )

Cited By (3)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US11074499B2

( en )

2017-11-20

2021-07-27

International Business Machines Corporation

Synaptic weight transfer between conductance pairs with polarity inversion for reducing fixed device asymmetries

US11334671B2

( en )

2019-10-14

2022-05-17

International Business Machines Corporation

Adding adversarial robustness to trained machine learning models

US11694072B2

( en )

2017-05-19

2023-07-04

Nvidia Corporation

Machine learning technique for automatic modeling of multiple-valued outputs

Families Citing this family (42)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

CN108073978A

( en )

2016-11-14

2018-05-25

é¡¾æ³½è

A kind of constructive method of the ultra-deep learning model of artificial intelligence

US11275996B2

( en )

2017-06-21

2022-03-15

Arm Ltd.

Systems and devices for formatting neural network parameters

CN109472344A

( en )

2017-09-08

2019-03-15

åå®ç§æè¡ä»½æéå¬å¸

Design method of neural network system

WO2019055380A1

( en )

2017-09-15

2019-03-21

Mythic, Inc.

System and methods for mixed-signal computing

CN111837145B

( en )

2017-12-18

2024-02-02

ç±³è¥¿åæéå¬å¸

System and method for mapping matrix calculations to matrix multiplication accelerators

CA3020884A1

( en )

2018-01-24

2019-07-24

Wai Kit Ricky Fok

Parallel residual neural network architechture and system and method for trainning a residual neural network

US11308397B2

( en )

2018-02-16

2022-04-19

Ilya Sorokin

System and method of training a neural network

CN108962230B

( en )

2018-07-27

2019-04-23

éåºå æ®ä¹ç§ææéå¬å¸

Memristor-based speech recognition method

US11562231B2

( en )

2018-09-03

2023-01-24

Tesla, Inc.

Neural networks for embedded devices

JP7259253B2

( en )

2018-10-03

2023-04-18

æ ªå¼ä¼ç¤¾ãã³ã½ã¼

artificial neural network circuit

DE102018221703A1

( en )

2018-12-13

2020-06-18

HELLA GmbH & Co. KGaA

Verification and identification of a neural network

RU2729878C2

( en )

2019-01-21

2020-08-13

ÐÐ¸ÑÐ°Ð¸Ð» ÐÑÐ¸Ð¼Ð¾Ð²Ð¸Ñ ÐÐ°Ð·ÑÑÐ¾Ð²

Instrumental method of training selective neural network without mathematics and without a teacher with use of self-organization

DE102019101617A1

( en )

2019-01-23

2020-07-23

Rockwell Collins Deutschland Gmbh

Device and method for training a neural network

US10896726B2

( en )

2019-04-02

2021-01-19

Junsung KIM

Method for reading a cross-point type memory array comprising a two-terminal switching material

CN110610708B

( en )

2019-08-31

2021-10-15

æµæ±å·¥ä¸å¤§å¦

A voiceprint recognition attack defense method based on cuckoo search algorithm

CN111008699B

( en )

2019-12-05

2022-06-07

é¦é½å¸èå¤§å¦

Neural network data storage method and system based on automatic driving

US11514326B2

( en )

2020-06-18

2022-11-29

International Business Machines Corporation

Drift regularization to counteract variation in drift coefficients for analog accelerators

US20210406661A1

( en )

2020-06-25

2021-12-30

PolyN Technology Limited

Analog Hardware Realization of Neural Networks

KR20220088845A

( en )

2020-06-25

2022-06-28

í´ë¦¬ì¤ íí¬ëë¡ì§ ë¦¬ë¯¸í°ë

Analog hardware implementation of neural networks

US12347421B2

( en )

2020-06-25

2025-07-01

PolyN Technology Limited

Sound signal processing using a neuromorphic analog signal processor

US11885271B2

( en )

2020-06-25

2024-01-30

PolyN Technology Limited

Systems and methods for detonation control in spark ignition engines using analog neuromorphic computing hardware

US20240039766A1

( en )

2021-04-16

2024-02-01

Nokia Solutions And Networks Oy

An apparatus, a method, and a non-transitory computer readable medium for determining a random access preamble identifier

TWI767757B

( en )

2021-06-18

2022-06-11

ä¸è¯é»ä¿¡è¡ä»½æéå¬å¸

Method for determining weight initial value of binary classification problem in neural network and computer readable medium thereof

CN113553293B

( en )

2021-07-21

2024-09-03

æ¸åå¤§å¦

Storage and calculation integrated device and calibration method thereof

CN113887135B

( en )

2021-09-30

2025-01-10

æ±èå¦æ¸æºé ç§ææéå¬å¸

Conductivity and flow pattern background prediction method, image reconstruction method, equipment and medium

CN116472575B

( en )

2021-11-18

2025-08-05

çä»ªåçµ(èå·)æéå¬å¸

Display device, current correction value establishment method and current correction system

US12179608B2

( en )

2021-12-03

2024-12-31

Sustainable Energy Technologies, Inc.

Monitoring and managing temperature of power packs

KR102579571B1

( en )

2021-12-03

2023-09-15

ìì£¼ëíêµì°ííë ¥ë¨

Pixelated monolithic photoplethysmography sensor with spiking neural network structure and method for operating the same

WO2023102265A1

( en )

2021-12-03

2023-06-08

Sustainable Energy Technologies, Inc.

System and method for energy management of a power pack for an electric vehicle including photovoltaic charging

US12194886B2

( en )

2021-12-06

2025-01-14

Sustainable Energy Technologies, Inc.

System and method for analyzing temperature changes in supercapacitor battery storage for electric vehicle

US12296721B2

( en )

2021-12-10

2025-05-13

Sustainable Energy Technologies, Inc.

Method and systems to trickle charge electric vehicle's supercapacitors using solar energy

US12224611B2

( en )

2021-12-14

2025-02-11

Sustainable Energy Technologies, Inc.

Supercapacitor to electrochemical hybrid system

WO2023114429A1

( en )

2021-12-17

2023-06-22

Sustainable Energy Technologies, Inc.

Intelligent hybrid power system for electrical vehicles

WO2023128792A1

( en )

2021-12-30

2023-07-06

PolyN Technology Limited

Transformations, optimizations, and interfaces for analog hardware realization of neural networks

US12533968B2

( en )

2021-12-30

2026-01-27

Sustainable Energy Technologies, Inc.

Supercapacitor to electrochemical hybrid system with failsafe safety capability

WO2023129616A1

( en )

2021-12-30

2023-07-06

Sustainable Energy Technologies, Inc.

Supercapacitor to electrochemical hybrid system with a supercapacitor battery management capability

WO2023129614A1

( en )

2021-12-30

2023-07-06

Sustainable Energy Technologies, Inc.

Supercapacitor to electrochemical hybrid system with smart self-discharge capability

CN115358389B

( en )

2022-09-01

2024-08-20

æ¸åå¤§å¦

Method, device, electronic device and medium for reducing training error of neural network

CN116009248B

( en )

2023-02-07

2025-05-30

ä¸å½ç§å¦é¢åå¯¼ä½ç ç©¶æ

Power divider design method, power divider and device

US11823037B1

( en )

2023-02-28

2023-11-21

PolyN Technology Limited

Optocoupler-based flexible weights in neuromorphic analog signal processors

CN116992938A

( en )

2023-08-08

2023-11-03

è¯åå¾®çµå(ä¸æµ·)è¡ä»½æéå¬å¸

A neural network pruning method, device, electronic equipment and storage medium

CN119904344B

( en )

2024-12-30

2025-10-10

å®å¾½å¤§å¦

Medical image encryption and hiding method and system based on fractional-order memristor Hopfield neural network

Citations (13)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US4918618A

( en )

1988-04-11

1990-04-17

Analog Intelligence Corporation

Discrete weight neural network

US4979124A

( en )

1988-10-05

1990-12-18

Cornell Research Foundation

Adaptive, neural-based signal processor

US5253329A

( en )

1991-12-26

1993-10-12

The United States Of America As Represented By The Administrator Of The National Aeronautics And Space Administration

Neural network for processing both spatial and temporal data with time based back-propagation

US5299285A

( en )

1992-01-31

1994-03-29

The United States Of America As Represented By The Administrator, National Aeronautics And Space Administration

Neural network with dynamically adaptable neurons

US5479575A

( en )

1991-02-12

1995-12-26

Mitsubishi Denki Kabushiki Kaisha

Self-organizing neural network for pattern classification

US5493688A

( en )

1991-07-05

1996-02-20

Booz, Allen & Hamilton, Inc.

Pattern categoritzation system having self-organizing analog fields

US5566273A

( en )

1993-12-30

1996-10-15

Caterpillar Inc.

Supervised training of a neural network

US5671337A

( en )

1992-12-24

1997-09-23

Olympus Optical Co., Ltd.

Neuron unit

US7577631B2

( en )

2001-09-10

2009-08-18

Feldhake Michael J

Cognitive image filtering

US20100088263A1

( en )

2005-09-29

2010-04-08

Gustavo Deco

Method for Computer-Aided Learning of a Neural Network and Neural Network

US7814038B1

( en )

2007-12-06

2010-10-12

Dominic John Repici

Feedback-tolerant method and device producing weight-adjustment factors for pre-synaptic neurons in artificial neural networks

US7904398B1

( en )

2005-10-26

2011-03-08

Dominic John Repici

Artificial synapse component using multiple distinct learning means with distinct predetermined learning acquisition times

US20120166374A1

( en )

2006-12-08

2012-06-28

Medhat Moussa

Architecture, system and method for artificial neural network implementation

2016

2016-06-09

US15/178,137

patent/US9619749B2/en

active

Active

Patent Citations (15)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US4918618A

( en )

1988-04-11

1990-04-17

Analog Intelligence Corporation

Discrete weight neural network

US4979124A

( en )

1988-10-05

1990-12-18

Cornell Research Foundation

Adaptive, neural-based signal processor

US5682503A

( en )

1991-02-12

1997-10-28

Mitsubishi Denki Kabushiki Kaisha

Self-organizing neural network for pattern classification

US5479575A

( en )

1991-02-12

1995-12-26

Mitsubishi Denki Kabushiki Kaisha

Self-organizing neural network for pattern classification

US5870729A

( en )

1991-02-12

1999-02-09

Mitsubishi Denki Kabushiki Kaisha

Self-organizing neural network for pattern classification

US5493688A

( en )

1991-07-05

1996-02-20

Booz, Allen & Hamilton, Inc.

Pattern categoritzation system having self-organizing analog fields

US5253329A

( en )

1991-12-26

1993-10-12

The United States Of America As Represented By The Administrator Of The National Aeronautics And Space Administration

Neural network for processing both spatial and temporal data with time based back-propagation

US5299285A

( en )

1992-01-31

1994-03-29

The United States Of America As Represented By The Administrator, National Aeronautics And Space Administration

Neural network with dynamically adaptable neurons

US5671337A

( en )

1992-12-24

1997-09-23

Olympus Optical Co., Ltd.

Neuron unit

US5566273A

( en )

1993-12-30

1996-10-15

Caterpillar Inc.

Supervised training of a neural network

US7577631B2

( en )

2001-09-10

2009-08-18

Feldhake Michael J

Cognitive image filtering

US20100088263A1

( en )

2005-09-29

2010-04-08

Gustavo Deco

Method for Computer-Aided Learning of a Neural Network and Neural Network

US7904398B1

( en )

2005-10-26

2011-03-08

Dominic John Repici

Artificial synapse component using multiple distinct learning means with distinct predetermined learning acquisition times

US20120166374A1

( en )

2006-12-08

2012-06-28

Medhat Moussa

Architecture, system and method for artificial neural network implementation

US7814038B1

( en )

2007-12-06

2010-10-12

Dominic John Repici

Feedback-tolerant method and device producing weight-adjustment factors for pre-synaptic neurons in artificial neural networks

Cited By (5)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US11694072B2

( en )

2017-05-19

2023-07-04

Nvidia Corporation

Machine learning technique for automatic modeling of multiple-valued outputs

US12254410B2

( en )

2017-05-19

2025-03-18

Nvidia Corporation

Machine learning technique for automatic modeling of multiple-valued outputs

US11074499B2

( en )

2017-11-20

2021-07-27

International Business Machines Corporation

Synaptic weight transfer between conductance pairs with polarity inversion for reducing fixed device asymmetries

US11915132B2

( en )

2017-11-20

2024-02-27

International Business Machines Corporation

Synaptic weight transfer between conductance pairs with polarity inversion for reducing fixed device asymmetries

US11334671B2

( en )

2019-10-14

2022-05-17

International Business Machines Corporation

Adding adversarial robustness to trained machine learning models

Also Published As

Publication number

Publication date

US20160283842A1

( en )

2016-09-29

Neural network and method of neural network training — Progress, Inc. (US9619749B2)

Related documents