Conceptio › Archive › Google Patents

Google Patentsopen access

Hardware neural network conversion method, computing device, compiling method … — Tsinghua University (US11544539B2)

Tsinghua University · Google Patents

Google Patents · Patents · License: Open Access

patent, google patents, intellectual property, US11544539B2, Tsinghua University, Youhui Zhang, en, 2023

ABSTRACT

Abstract

A hardware neural network conversion method, a computing device, a compiling method and a neural network software and hardware collaboration system for converting a neural network application into a hardware neural network fulfilling a hardware constraint condition are disclosed. The method comprises: obtaining a neural network connection diagram corresponding to the neural network application; splitting the neural network connection diagram into neural network basic units; converting each of the neural network basic units so as to form a network having equivalent functions thereto and formed by connecting basic module virtual entities of neural network hardware; and connecting the obtained basic unit hardware network according to the sequence of splitting so as to create a parameter file for the hardware neural network. The present disclosure provides a novel neural network and a brain-like computing software and hardware system.

Description

CROSS REFERENCE TO RELATED APPLICATION

This application is a 35 USC 371 application of International PCT Patent Application No. PCT/CN2016/100783, filed on Sep. 29, 2016; the content of which is hereby incorporated by reference herein in its entirety.

TECHNICAL FIELD

The present disclosure generally relates to a neural network technical field, and more particularly, to a technology of implementing a software neural network with a neural network chip.

BACKGROUND

In recent years, a deep learning technology has made a breakthrough progress, and has achieved a very high precision rate in image recognition, speech recognition, natural language processing and many other fields; however, deep learning requires massive computing resources, and it is very difficult for a traditional general-purpose processor to fulfill computing requirements of deep learning, so hardware conversion of deep learning and its application specific integrated circuit (ASIC) design have become an important direction of development. Meanwhile, with development of brain science, because a brain has characteristics such as ultra-low power consumption, and high error tolerance as compared with a traditional von Neumann computer, and has significant advantages in processing unstructured information and intelligent tasks, drawing on a computing mode of the brain to construct a novel brain-like computing system and a brain-like computing chip has become an emerging development direction.

Whether it is deep learning or brain-like computing, the underlying computing model is a Neural Network (NN); the two differ from each other mainly in that: the deep learning mainly uses an Artificial Neural Network (ANN), while the brain-like computing mainly uses a Spiking Neural Network (SNN); and both take a neuron as a basic component unit, with a large number of neurons interconnected to form a network. A connection between neurons may be considered as a weighted directed edge, an output of a neuron is weighted by a connection between neurons, and then transmitted to a neuron connected therewith, and all inputs received by each neuron will be accumulated for further processing, to generate an output of the neuron. The ANN and the SNN differ from each other mainly that: a neuron of the ANN outputs a numerical value, which is multiplied by an edge weight for weighting, while a neuron of the SNN outputs electric pulse signals one by one, which are weighted to form current signals of different intensities; with respect to an input of any other neuron, the neuron of the ANN will directly calculate an output value of the neuron through an activation function, while the neuron of the SNN, when receiving a current signal input by any other neuron, will update its state according to its neuron model, discharge an electrical pulse when a certain state is reached, and reset the state.

The neural network is usually modeled with a plurality of neurons as a layer, with layers interconnected; FIG. 10 shows a chain-like neural network, in which each circle represents one neuron, each arrow indicates a connection between neurons, and each connection has a weight; however, a structure of an actual neural network is not limited to the chain-like network structure.

A core computation of the neural network is a matrix vector multiplication operation, wherein, an output generated by layer L n including n neurons may be represented by vector V n with a length of n, the layer is in full connection with layer L m including m neurons, a connection weight may be expressed as a matrix M nÃm , the matrix has a size of n rows and m columns, and each matrix element represents a weight of one connection. Then, a vector input into L m after weighting is M nÃm V n , and such a matrix vector multiplication is the most core computation of the neural network.

Because the matrix vector multiplication is extraordinarily complex, and plenty of time has to be spent doing a large amount of matrix multiplications on a traditional general-purpose processor, a neural network acceleration chip and a brain-like chip both take an accelerated matrix multiplication as a main design objective; in specific implementation, a matrix vector multiplication module of a certain size (for example, a basic module for multiplying a matrix with a size of 256Ã256 by a vector with a length of 256) is usually implemented by hardware, and then basic modules are connected by using network on chip (NoC) and other technologies. Through hardware conversion of the matrix vector multiplication, an operational speed may be greatly improved.

However, hardware conversion also constrains freedom of a neural network application it may support, which also poses an important issue that: it is difficult to run an actual neural network application with such a chip. Although the neural network chip may efficiently perform matrix vector multiplications, there are still significant differences between the neural network application and the underlying chip, for example:

(1) A neural network hardware basic module is usually a fixed-size matrix vector multiplication, while a size of a matrix operation in the actual neural network application is arbitrary.

(2) The neural network application usually computes with a 32-bit floating point number, while hardware is sometimes designed to have a lower precision, and even to compute with an integer in order to increase efficiency.

(3) An activation function (with respect to the ANN) or a neuron model (with respect to the SNN) of the neural network hardware is usually fixed, while an activation function or a neuron model of the neural network application is usually very flexible, and new activation functions and neuron models will be constantly brought into the neural network application.

Hereinafter, an overview of hardware chip series in the prior art is provided.

1. Technical Solution I in the Prior Art: Cambrian Chip Series

1 (1) Technical Solution I in the Prior Art

A computing core of the Cambrian chip implements a matrix vector multiplication with a size of 16Ã16 and a nonlinear activation function through a high-speed three-stage pipeline, the chip is further configured with three dedicated storage modules, which are respectively used for storing input data, output data and weight data, and the data is retrieved from an on-chip storage module by a controller and sent to the computing core for computation. With respect to an operation of a matrix with larger size of, for example, 32, it is split into 4 matrices, each with a size of 16Ã16, according to the technical solution, which are sequentially loaded into the computing core by the controller to complete computation, and finally computation results are accumulated and combined. A neural network of an arbitrary size is supported through multiplexing the computing core in a time division manner. On the other hand, in a third-stage pipeline step of the Cambrian chip computing core, various common activation functions are provided to support the vast majority of neural network applications.

1 (2). Disadvantages of the Technical Solution I in the Prior Art

In the Cambrian chip approach, the weight of the neural network is separated from the computing core, and time division multiplexing of computing resources and access to a memory is controlled through software. Since the method still separates computation from storage, it is essentially a customized solution under the von Neumann architecture, such that weight data still has to be transmitted back and forth between a computing unit and a storage unit, and it is still subjected to a Von Neumann bottleneck. Although the Cambrian chip has made great effort in improving a bandwidth between the computing core and the storage unit, yet as the size of the neural network application increases, access to the weight data will eventually become a system bottleneck.

In addition, due to relatively large overhead of computational logic and on-chip storage, chip integration cannot reach a very high level, and the number of computing cores integrated on each chip is very limited.

2. Technical Solution II in the Prior Art Related to the Present Disclosure: TrueNorth Chip

2 (1). The Technical Solution II in Prior Art

TrueNorth is IBM's neuromorphic chip, each chip has 4096 synaptic nuclei integrated thereon, and each synaptic nucleus may process 256 synaptic computations (i.e., matrix vector multiplications). In order to improve an integration level, TrueNorth's synaptic nucleus has been greatly streamlined, wherein, a very simple leaky integrate-and-fire (LIF) neuron model (a commonly used SNN neuron model) is applied, a weight is also greatly compressed, each neuron may have up to 256 input synapses, and there are only 3 optional values for weights of the 256 input synapses.

In order to apply TrueNorth to run the actual neural network, IBM has designed a set of Corelet language to program TrueNorth, which decomposes a large task into connections between small tasks in a stepwise manner, so that a smallest task is just on a synaptic nucleus. Corelet has various hardware constraints exposed to an application layer, so that constraints of TrueNorth hardware itself have to be considered during neural network design.

2 (2). Disadvantages of the Technical Solution II in the Prior Art

In the TrueNorth chip design, in order to improve the integration level of the chip, more synaptic nuclei are placed within a limited area, and the synaptic nuclei of the TrueNorth chip has a strong constraint on the neural network. Therefore, it is very difficult to run the existing neural network application on the TrueNorth chip; with respect to various intelligent tasks, it is necessary to redesign and train a neural network dedicated to the TrueNorth chip, and due to the hardware constraints on the application layer, currently it is very difficult for the redesigned and trained neural network dedicated to TrueNorth to achieve a precision rate comparable to the most advanced neural network at present in image recognition and other fields.

3. Technical Solution III in the Prior Art Related to the Present Disclosure: A Novel DeviceâMemristor

3 (1). Technical Solution III in the Prior Art

The memristor is a novel semiconductor device, whose resistance value may be changed at a specific input current. The resistance value of the memristor may be used for storing data; as compared with traditional Dynamic Random Access Memory (DRAM) and Static Random Access Memory (SRAM), it has a characteristic of high storage density; and since its data is stored through the resistance value, data will not be lost even when power supply fails. In addition, the memristor may also perform computation, which is an ideal device for integrating computation and storage.

FIG. 11 shows a structural schematic diagram of a crossbar based on the memristor.

As shown in FIG. 11 , by arranging lines into the crossbar and connecting them at intersection points with memristors, setting a conductance value (a reciprocal of a resistance) of the memristor to a matrix element value of a weight matrix, and inputting a voltage value at an input end, the matrix vector multiplication may be completed at an output end. Based on such a basic unit, a neuromorphic chip based on a novel device may be constructed. Due to its characteristics of high integration level and integration of computation and storage, it does not have to transmit weight data back and forth, and has great potential in constructing a large-size neuromorphic chip.

3 (2). Disadvantages of the Technical Solution III

Since computation of the memristor is based on an analog circuit, precision its analog signal may achieve is limited, and a value range of the weight also depends on a resistance change range of the memristor. In addition, it is also constrained by connectivity just as TrueNorth, and it is still difficult to directly run the existing neural network applied thereto.

To sum up, the technical solution I in the prior art, the Cambrian chip, is dedicated to adapting the chip to the demand of the neural network application, so that the chip can support a neural network of an arbitrary size in a time division multiplexing manner, and support the existing neural network through a built-in common activation function. On the one hand, due to its characteristic of separation of storage from computation, it is always subjected to the Von Neumann bottleneck, and as the application size increases, its efficiency will be limited by the transmission bandwidth between storage and computation; on the other hand, because the commonly used activation function is fixed thereby, with development of the neural network application technology, new activation functions and neuron models require the chip to constantly adapt to the development of the application and make corresponding modifications; moreover, due to its high degree of freedom, the logic is relatively complex logic, and cannot achieve a very high integration level. The technical solution II in prior art, TrueNorth, is dedicated to adapting the application to the neural network chip, while the underlying chip is dedicated to improving integration and efficiency, to reduce power consumption. The neuron model it supports is streamlined, so that millions of neurons are inherited within a very small chip area with very low power consumption. In addition, it may be combined with the technical solution III, to further improve the integration level with novel devices and processes. However, the category of solutions poses too many constraints on the application, so they cannot be well combined with the existing application, and may hardly achieve an effect equivalent to that of the most advanced neural network at present on complex tasks.

It may be seen that, the existing neural network hardware is usually directly connected with the neural network application, resulting in either an issue that the hardware is too simple, which restrains the freedom degree of the application, or an issue that the hardware has a high freedom degree, which is relatively complicated, so that it is difficult to improve integration and efficiency.

A more universal all-purpose technology which applies an arbitrary neural network application to an arbitrary neural network chip is required.

SUMMARY

The present disclosure has been made in view of the above-described circumstances.

CROSS REFERENCE TO RELATED APPLICATION

TECHNICAL FIELD

The present disclosure generally relates to a neural network technical field, and more particularly, to a technology of implementing a software neural network with a neural network chip.

BACKGROUND

(1) A neural network hardware basic module is usually a fixed-size matrix vector multiplication, while a size of a matrix operation in the actual neural network application is arbitrary.

Hereinafter, an overview of hardware chip series in the prior art is provided.

1. Technical Solution I in the Prior Art: Cambrian Chip Series

1 (1) Technical Solution I in the Prior Art

1 (2). Disadvantages of the Technical Solution I in the Prior Art

2. Technical Solution II in the Prior Art Related to the Present Disclosure: TrueNorth Chip

2 (1). The Technical Solution II in Prior Art

2 (2). Disadvantages of the Technical Solution II in the Prior Art

3. Technical Solution III in the Prior Art Related to the Present Disclosure: A Novel DeviceâMemristor

3 (1). Technical Solution III in the Prior Art

FIG. 11 shows a structural schematic diagram of a crossbar based on the memristor.

3 (2). Disadvantages of the Technical Solution III

A more universal all-purpose technology which applies an arbitrary neural network application to an arbitrary neural network chip is required.

SUMMARY

The present disclosure has been made in view of the above-described circumstances.

According to one aspect of the present disclosure, there is provided a hardware neural network conversion method for converting a neural network application into a hardware neural network fulfilling a hardware constraint condition, which may comprise: a neural network connection diagram obtaining step of obtaining a neural network connection diagram corresponding to the neural network application, wherein, the neural network connection diagram is a directed diagram, each node in the diagram represents a layer of neurons, and each edge represents an inter-layer connection relationship; a neural network connection diagram splitting step of splitting the neural network connection diagram into neural network basic units, wherein, each neural network basic unit has only ingress nodes and egress nodes without any intermediate layer node, the ingress nodes are in full connection with the egress nodes, all fan-outs of each neuron in the ingress node are within the basic unit, and all fan-ins of each neuron in the egress node are within the basic unit; a neural network basic unit conversion step of converting each of the neural network basic units so as to form a network having equivalent functions thereto and formed by connecting basic module virtual entities of neural network hardware, which is referred to as a basic unit hardware network, wherein, one neural network basic unit corresponds to one or more basic module virtual entities of the neural network hardware, and each basic module virtual entity of the neural network hardware fulfills a connectivity constraint condition of a neural network hardware basic module, and is capable of being directly mapped to the neural network hardware basic module; and a basic unit hardware network connection step of connecting the obtained basic unit hardware networks according to the sequence of splitting so as to create a parameter file for the hardware neural network.

The above-described hardware neural network conversion method, may further comprise: in a case where the neural network application has a convolution layer, before the neural network connection diagram splitting step, performing network compression on the convolution layer of the neural network application; and the network compression operation may include: obtaining a plurality of feature maps of each convolution layer; taking similarity between outputs generated by the feature maps on all samples as a DPP algorithm-associated matrix element by using a DPP method for extracting a diversity subset, obtaining a highest diversity subset by using the DPP, reserving the subset, discarding other feature map nodes, projecting a vector corresponding to the discarded feature map into a linear space spanned by the reserved feature map, taking a ratio of a projection length of the discarded feature map to its original vector length as a weighting coefficient, and accumulating a connection weight between the discarded feature map and a next layer of neurons in a weighted manner onto a connection weight between the reserved feature map and the next layer of neurons.

According to the above-described hardware neural network conversion method, the neural network basic unit conversion step includes: reconstructing a network topology for each neural network basic unit; and determining a weight parameter for the reconstructed network topology.

According to the above-described hardware neural network conversion method, the reconstructing a network topology includes a fully expanded operation; after fully expansion, the neural network basic unit is decomposed into interconnections between basic module virtual entities; and the fully expanded operation includes: in a case where a first-size matrix multiplication and/or a large matrix convolution operation associated with the neural network basic unit exceeds a second-size small matrix operation supported by the neural network hardware basic module, executing operations below: splitting the first-size large matrix operation into the third number of second-size small matrix operations, each small matrix operation being completed by one basic module virtual entity; decomposing input data for the first-size large matrix operation into the third number of copies, and transmitting the same to the third number of second-size small matrix operations, which is a multicast operation; aggregating operation results from the third number of second-size small matrix operations to be equivalent to an operation result of the first-size large matrix operation, which is a reduction operation; in a case that the neural network hardware chip has a first additional module supporting the multicast operation, assigning the multicast operation to be executed by the first additional module virtual entity, otherwise, completing the multicast operation by a first set of basic module virtual entities; in a case that the neural network hardware chip has a second additional module supporting the reduction operation, assigning the reduction operation to be executed by the second additional module virtual entity, otherwise, completing the reduction operation by a second set of basic module virtual entities.

According to the above-described hardware neural network conversion method, in a case where the number of basic modules on the neural network hardware chip is insufficient, the basic module is multiplexed in a time division manner.

According to the above-described hardware neural network conversion method, the reconstructing a network topology, before the fully expanded operation, further includes a recoding operation, which may include: recoding inter-layer data with an autoencoder, wherein, the autoencoder is a neural network, consisting of three layers of neurons, including an input layer, a hidden layer and an output layer, the number of nodes of the output layer is equal to the number of nodes of the input layer, and the number of nodes of the hidden layer is greater than dimensionality of inter-layer vector data; training the network, so that a value of the output layer is as approximate to a value of the input layer as possible, wherein, precision of the input layer and the output layer is precision of the neural network application, and precision of the hidden layer is precision of data transmitted between the neural network hardware basic modules; converting the autoencoder into a combination of an encoder and a decoder; and forming a connection matrix by combining a decoder of an input node, a weight matrix of an original connection, and an encoder of an output node, with respect to an expression that an inter-layer vector transmitted from a Kth layer to a (K+1)th layer is a hidden layer of the autoencoder used in the kth layer.

The above-described hardware neural network conversion method, in a case where there is a special function in the neural network application and the neural network hardware chip does not support the special function, before the fully expanded operation, further comprises: constructing a specialized neural network for the special function.

According to the above-described hardware neural network conversion method, the determining a weight parameter for the reconstructed network topology includes: initializing a weight of the network obtained by reconstructing the network topology, according to the weight of the original neural network; and performing fine adjustment on the weight parameter such that the weight fulfills a hardware weight constraint. According to the above-described hardware neural network conversion method, the performing fine adjustment on a weight parameter such that the weight fulfills a hardware weight constraint includes: (1) firstly, expressing the weight with floating point precision, and retraining the constructed network, so that an error from the original network is as small as possible; (2) in a case where the neural network hardware chip has a configurable parameter P, determining a best P and by using an EM algorithm, according to the parameter obtained in the training of step (1), expressing all weight parameters as a function of P, and retraining to adjust P, where P is a configurable parameter of hardware abstraction, and k ij is an index of values of respective matrix elements in a set S P ; (3) in a case where weight precision of the neural network hardware chip is lower than a predetermined threshold, fixing P obtained in the training of step (2), initializing all weights to corresponding S k

P , retraining to adjusts k ij , storing all the weights with floating point precision, but rounding all the weight parameters to a most approximate value in S P in a feedforward process of the training, then bringing into feedforward computation, while feeding back and updating the weights still with floating point precision, and updating the weight values of floating point precision, where, a value range of a weight matrix W of the neural network hardware basic module is considered as a set S P , each element in the set is a function of parameter P, where P is a hardware configurable parameter, each element W ij in the weight matrix may be independently selected from S P , that is, an index k ij is capable of being independently configured such that W ij =S k

P , so what the weight matrix W is capable of configuring is a set parameter P and index k ij of values of respective weights in the set.

According to the above-described hardware neural network conversion method, the converting each of the neural network basic units so as to form a network having equivalent functions thereto and formed by connecting basic module virtual entities of neural network hardware may include: in a case where the neural network connection diagram is a directed acyclic diagram, converting the respective neural network basic units one by one, in a topological order of the neural network connection diagram; in a case where the neural network connection diagram is a cyclic directed diagram, firstly breaking a cycle of the cyclic directed diagram, so that the neural network connection diagram is changed into a directed acyclic diagram, then converting the respective neural network basic units one by one, in a topological order of the directed acyclic diagram; and training the respective converted neural network basic units in the topological order, sources of training data required for retraining being that: training input data is an output generated by a training sample having undergone a topological order-preceded basic unit hardware network, and training output label is an output generated by the training sample in a corresponding layer of the original neural network application.

According to the above-described hardware neural network conversion method, when the neural network application is an SNN, the training data used in the neural network basic unit conversion step is obtained as follows: taking an electrical pulse of a stable frequency as an input with respect to the original network, recording electrical pulse discharge frequencies of respective neurons, and taking the same as the training data used in the neural network basic unit conversion step.

According to the above-described hardware neural network conversion method, when a neural network involved in the neural network hardware chip is of an SNN type, a functional relation of the SNN on the pulse discharge frequency is derived according to a neuron model of the SNN, and based on that the functional relation is continuous and derivable, training is performed by using a back-propagation algorithm.

According to another aspect of the present disclosure, there is provided a computing device, for converting a neural network application into a hardware neural network fulfilling a hardware constraint condition, comprising a memory and a processor, the memory having computer-executable instructions stored therein, and when executing the computer-executable instructions, the processor executing a method below: a neural network connection diagram obtaining step of obtaining a neural network connection diagram corresponding to the neural network application, wherein, the neural network connection diagram is a directed diagram, each node in the diagram represents a layer of neurons, and each edge represents an inter-layer connection relationship; a neural network connection diagram splitting step of splitting the neural network connection diagram into neural network basic units, wherein, each neural network basic unit has only ingress nodes and egress nodes without any intermediate layer node, the ingress nodes are in full connection with the egress nodes, all fan-outs of each neuron in the ingress node are within the basic unit, and all fan-ins of each neuron in the egress node are within the basic unit; a neural network basic unit conversion step of converting each of the neural network basic units so as to form a network having equivalent functions thereto and formed by connecting basic module virtual entities of neural network hardware, which is referred to as a basic unit hardware network, wherein, one neural network basic unit corresponds to one or more basic module virtual entities of the neural network hardware, and each basic module virtual entity of the neural network hardware fulfills a connectivity constraint condition of a neural network hardware basic module, and is capable of being directly mapped to the neural network hardware basic module; and a basic unit hardware network connection step of connecting the obtained basic unit hardware networks according to the sequence of splitting so as to create a parameter file for the hardware neural network.

According to the above-described computing device, the method executed further comprises: in a case where the neural network application has a convolution layer, before the neural network connection diagram splitting step, performing network compression on the convolution layer of the neural network application, including: obtaining a plurality of feature maps of each convolution layer; taking similarity between outputs generated by the feature maps on all samples as a DPP algorithm-associated matrix element by using a DPP method for extracting a diversity subset, obtaining a highest diversity subset by using the DPP, reserving the subset, discarding other feature map nodes, projecting a vector corresponding to the discarded feature map into a linear space spanned by the reserved feature map, taking a ratio of a projection length of the discarded feature map to its original vector length as a weighting coefficient, and accumulating a connection weight between the discarded feature map and a next layer of neurons in a weighted manner onto a connection weight between the reserved feature map and the next layer of neurons.

According to the above-described computing device, the neural network basic unit conversion step may include: reconstructing a network topology for each neural network basic unit; and determining a weight parameter for the reconstructed network topology.

According to the above-described computing device, the reconstructing a network topology includes a fully expanded operation; after fully expansion, the neural network basic unit is decomposed into interconnections between basic module virtual entities; and the fully expanded operation includes:

In a case where a first-size matrix multiplication and/or a large matrix convolution operation associated with the neural network basic unit exceeds a second-size small matrix operation supported by the neural network hardware basic module, executing operations below: splitting the first-size large matrix operation into the third number of second-size small matrix operations, each small matrix operation being completed by one basic module virtual entity; decomposing input data for the first-size large matrix operation into the third number of copies, and transmitting the same to the third number of second-size small matrix operations, which is a multicast operation; aggregating operation results from the third number of second-size small matrix operations to be equivalent to an operation result of the first-size large matrix operation, which is a reduction operation; in a case that the neural network hardware chip has a first additional module supporting the multicast operation, assigning the multicast operation to be executed by the first additional module virtual entity, otherwise, completing the multicast operation by a first set of basic module virtual entities; in a case that the neural network hardware chip has a second additional module supporting the reduction operation, assigning the reduction operation to be executed by the second additional module virtual entity, otherwise, completing the multicast operation by a second set of basic module virtual entities.

According to the above-described computing device, in a case where the number of basic modules on the neural network hardware chip is insufficient, the basic module is multiplexed in a time division manner.

According to the above-described computing device, the reconstructing a network topology, before the fully expanded operation, further includes a recoding operation, including: recoding inter-layer data with an autoencoder, wherein, the autoencoder is a neural network, consisting of three layers of neurons, including an input layer, a hidden layer and an output layer, the number of nodes of the output layer is equal to the number of nodes of the input layer, and the number of nodes of the hidden layer is greater than dimensionality of inter-layer vector data; training the network, so that a value of the output layer is as approximate to a value of the input layer as possible, wherein, precision of the input layer and the output layer is precision of the neural network application, and precision of the hidden layer is precision of data transmitted between the neural network hardware basic modules; converting the autoencoder into a combination of an encoder and a decoder; and forming a connection matrix by combining a decoder of an input node, a weight matrix of an original connection, and an encoder of an output node, with respect to an expression that an inter-layer vector transmitted from a Kth layer to a (K+1)th layer is a hidden layer of the autoencoder used in the kth layer.

According to the above-described computing device, in a case where there is a special function in the neural network application and the neural network hardware chip does not support the special function, before the fully expanded operation, the method further comprises: constructing a specialized neural network for the special function.

According to the above-described computing device, the determining a weight parameter for the reconstructed network topology includes: initializing a weight of the network obtained by reconstructing the network topology, according to the weight of the original neural network; and performing fine adjustment on the weight parameter such that the weight fulfills a hardware weight constraint.

According to the above-described computing device, the performing fine adjustment on a weight parameter such that the weight fulfills a hardware weight constraint includes: (1) firstly, expressing the weight with floating point precision, and retraining the constructed network, so that an error from the original network is as small as possible; (2) in a case where the neural network hardware chip has a configurable parameter P, determining a best P and k ij by using an EM algorithm, according to the parameter obtained in the training of step (1), expressing all weight parameters as a function of P, and retraining to adjust P, where P is a configurable parameter of hardware abstraction, and k ij is an index of values of respective matrix elements in a set S P ; (3) in a case where weight precision of the neural network hardware chip is lower than a predetermined threshold, fixing P obtained in the training of step (2), initializing all weights to corresponding S k

P , retraining to adjust k ij , storing all the weights with floating point precision, but rounding all the weight parameters to a most approximate value in S P in a feedforward process of the training, then bringing into feedforward computation, while feeding back and updating the weights still with floating point precision, and updating the weight values of floating point precision, where, a value range of a weight matrix W ij of the neural network hardware basic module is considered as a set S P each element in the set is a function of parameter P, where P is a hardware configurable parameter, each element W ij in the weight matrix is capable of being independently selected from that is S P , an index k ij is capable of being independently configured such that W ij =S k

P , so what the weight matrix W is capable of configuring is a set parameter P and an index k ij of values of respective weights in the set.

According to the above-described computing device, the converting each of the neural network basic units so as to form a network having equivalent functions thereto and formed by connecting basic module virtual entities of neural network hardware includes: in a case where the neural network connection diagram is a directed acyclic diagram, converting the respective neural network basic units one by one, in a topological order of the neural network connection diagram; in a case where the neural network connection diagram is a cyclic directed diagram, firstly breaking a cycle of the cyclic directed diagram, so that the neural network connection diagram is changed into a directed acyclic diagram, then converting the respective neural network basic units one by one, in a topological order of the directed acyclic diagram; and training the respective converted neural network basic units in the topological order, sources of training data required for retraining being that: training input data is an output generated by a training sample having undergone a topological order-preceded basic unit hardware network, and training output label is an output generated by the training sample in a corresponding layer of the original neural network application.

According to the above-described computing device, when the neural network application is an SNN, the training data used in the neural network basic unit conversion step is obtained as follows: taking an electrical pulse of a stable frequency as an input with respect to the original network, recording electrical pulse discharge frequencies of respective neurons, and taking the same as the training data used in the neural network basic unit conversion step.

According to the above-described computing device, when a neural network involved in the neural network hardware chip is of an SNN type, a functional relation of the SNN on the pulse discharge frequency is derived according to a neuron model of the SNN, and based on that the functional relation is continuous and derivable, training is performed by using a back-propagation algorithm.

According to still another aspect of the present disclosure, there is provided a compiling method for compiling a neural network software application into a hardware neural network, which may comprise: obtaining a neural network software application and a neural network hardware chip configuration situation; converting the neural network software application into the hardware neural network, based on the neural network hardware configuration situation, the hardware neural network being formed by connecting basic modules of the neural network hardware chip; and outputting a parameter file for the hardware neural network, the parameter file describing a connection relationship between the basic modules and a parameter configuration situation of respective basic modules.

According to a further aspect of the present disclosure, there is provided a neural network software and hardware collaboration system, which may comprise: a neural network hardware chip, wherein, the neural network hardware chip has a basic module thereon, the basic module executes a matrix vector multiplication and an activation function operation in a form of hardware, and a parameter of the basic module on the neural network hardware chip and a connection between basic modules may be configured by a configuration file of a determined format; a compiling layer unit, configured to compile a neural network application into a parameter file for a hardware neural network, wherein, the hardware neural network may be mapped to one or more neural network hardware chips based on the parameter file, and the one or more mapped neural network hardware chips are capable of running a function of the neural network application.

According to the above-described neural network software and hardware collaboration system, the compiling layer unit is configured to execute a method below: a hardware configuration data obtaining step of obtaining configuration situation data of the neural network hardware chip; a neural network connection diagram obtaining step of obtaining a neural network connection diagram corresponding to the neural network application, wherein, the neural network connection diagram is a directed diagram, each node in the diagram represents a layer of neurons, and each edge represents an inter-layer connection relationship; a neural network connection diagram splitting step of splitting the neural network connection diagram into neural network basic units, wherein, each neural network basic unit has only ingress nodes and egress nodes without any intermediate layer node, the ingress nodes are in full connection with the egress nodes, all fan-outs of each neuron in the ingress node are within the basic unit, and all fan-ins of each neuron in the egress node are within the basic unit; a neural network basic unit conversion step of converting each of the neural network basic units so as to form a network having equivalent functions thereto and formed by connecting basic module virtual entities of neural network hardware, which is referred to as a basic unit hardware network, wherein, one neural network basic unit corresponds to one or more basic module virtual entities of the neural network hardware, and each basic module virtual entity of the neural network hardware fulfills a connectivity constraint condition of a neural network hardware basic module, and is capable of being directly mapped to the neural network hardware basic module; and a basic unit hardware network connection step of connecting the obtained basic unit hardware networks according to the sequence of splitting so as to create a parameter file for the hardware neural network.

The present disclosure provides a novel neural network and a brain-like computing software and hardware system.

As described above, the existing technical route is, without exception, to directly adapt the neural network application to the chip: either directly adapt the chip to the freedom degree of the application, which brings a performance bottleneck; or expose the chip constraint to the application, which constrains capabilities of the application. In contrast, in the hardware neural network conversion method according to the embodiment of the present disclosure, the addition of an intermediate layer between the neural network application and the neural network chip resolves the issue of adaptation between the neural network application and the neural network application chip by means equivalent to a compiling technology in the traditional computer system, and at a same time, decouples applications from chip development.

In addition, in the hardware neural network conversion method according to the embodiment of the present disclosure, with respect to an arbitrary complex neural network and arbitrary hardware fulfilling hardware abstraction, there is provided a universal process, which may convert the complex neural network into a specific network fulfilling the hardware constraint condition, and having substantially equivalent functions to the original network. The core of the process is to decompose the complex network; since an operation executed by each basic unit is relatively simple, the conversion process may be more securely converged than direct conversion of the entire network, with a faster convergence speed.

Moreover, in the hardware neural network conversion method according to the embodiment of the present disclosure, the nodes in the neural network connection diagram are grouped, and the neural network is split into a plurality of basic units, so that ingress edges or egress edges of an arbitrary node within the basic unit are all within the basic unit, and thus, after the issue of connectivity is resolved within the basic unit, the converted basic units are reconnected, and the obtained network still can fulfill the connectivity requirement.

In addition, in one example as described above, modules are converted one by one in the topological order, and the previously generated error is introduced into subsequent fine adjustment, so that the error introduced by conversion of the respective basic modules will not be accumulated layer by layer.

In addition, in one example, in a case where the neural network application has a convolution layer, before the neural network connection diagram splitting step, network compression may be performed on the convolution layer of the neural network application, so as to reduce a network size, and save hardware resources.

BRIEF DESCRIPTION OF THE DRAWINGS

These and/or other aspects and advantages of the present disclosure will become more apparent and more comprehensible, from the following detailed description of embodiments of the present disclosure in conjunction with the accompanying drawings, in which:

FIG. 1 shows a schematic diagram of an application context 1000 of a hardware neural network conversion technology according to an embodiment of the present disclosure.

FIG. 2 shows a general flow chart of a hardware neural network conversion method 200 executed by a compiling layer 1200 according to an embodiment of the present disclosure.

FIG. 3 shows an example of a neural network connection diagram, in which each of

nodes

1, 2, 3, 4 and 5 represents a layer of neurons.

FIG. 4 shows an exemplary schematic diagram of a neural network basic unit 400 .

FIG. 5 ( a ) to FIG. 5 ( c ) show process schematic diagrams of splitting the neural network connection diagram into a plurality of neural network basic units.

FIG. 6 shows a schematic diagram of a network topology reconstruction operation and a weight parameter fine adjustment operation in neural network basic unit conversion.

FIG. 7 shows a process schematic diagram of recoding a three-layer neural network with an autoencoder to obtain an expanded three-layer neural network.

FIG. 8 shows neural network replacement of a max operation.

FIG. 9 shows an exemplary schematic diagram of fully expansion 2313 for a large-size matrix multiplication operation, according to the embodiment of the present disclosure.

FIG. 10 shows a schematic diagram of a chain-like neural network.

FIG. 11 shows a structural schematic diagram of a crossbar based on a memristor.

DETAILED DESCRIPTION

In order that those skilled in the art better understand the present disclosure, it will be further described in detail below in conjunction with accompanying drawings and specific implementation modes.

Before the respective embodiments are described in detail, terms used herein are explained.

Hardware neural network: it refers to a neural network fulfilling a hardware constraint condition.

Neural network hardware chip: it refers to a chip that takes a neural network as a target application.

Neural network connection diagram: a neural network connection diagram is a directed diagram, each node in the diagram represents a layer of neurons, and each edge represents an inter-layer connection relationship; in a case of an ANN neural network application, a corresponding neural network connection diagram is an acyclic directed diagram, and in a case of an SNN neural network application, a corresponding neural network connection diagram is a directed cyclic diagram.

Neural network basic unit: each neural network basic unit has only ingress nodes and egress nodes without any intermediate layer node, the ingress nodes are in full connection with the egress nodes, all fan-outs of each neuron in the ingress node are within the basic unit, and all fan-ins of each neuron in the egress node are within the basic unit.

Neural network hardware chip: it is connected by a large number of physical cores through an interconnection system, may have various topologies, and may accept certain configuration.

Physical core: it is a neural network hardware basic module consisting of a matrix vector multiplication+an activation function, and plays a role in receiving an input, performing matrix weighting firstly, and then generating an output through the activation function.

Parameter file for hardware neural network: it includes information describing a parameter of a virtual core and a connection relationship between virtual cores, wherein, the parameter of the virtual core includes, for example, a connection matrix, and the like.

Virtual core: a virtual core corresponds to the physical core, and is an abstraction of the physical core; herein, it refers to each and every one hardware basic module virtual entity in a connection diagram finally obtained by using an algorithm. After the conversion algorithm ends, a bunch of virtual cores and a connection relationship therebetween are obtained, and then the virtual cores are distributed onto the physical cores of the neural network hardware chip through a mapping algorithm.

Mapping: a process of distributing the virtual cores onto the physical cores.

Connectivity constraint: each neural network hardware basic module may only support a fixed-size matrix operation, so an fan-in of a neuron should not exceed the number of inputs of the hardware basic module, and an fan-out of a neuron should not exceed an fan-out of the hardware basic module. Another point is that, a connection between hardware basic modules only supports one-to-one connection, that is, one output of the hardware basic module may only be sent to an input of one other hardware basic module, which is also a connectivity constraint; however, not all neural network hardware is subjected to the constraint.

The present disclosure provides an idea of introducing an intermediate layer between the hardware and the application, and provides a universal method and process for transparently converting and adapting an arbitrary neural network (regardless whether an ANN or an SNN) to an arbitrary neural network chip, which is similar to a role of a compiler in a traditional computer system. In the present disclosure, development of the neural network application may be decoupled from research and development of the neural network chip, so that the hardware may be made simple enough, dedicated to improving efficiency and integration, while capable of supporting any neural network application.

Target hardware herein refers to a variety of neural network accelerators and brain-like computing chips. These chips usually consist of a plurality of processing cores, each processing core may accept M inputs, which are subjected to a matrix vector multiplication with an MÃN matrix, resulting in results, which undergo a hardware built-in activation function or a built-in hardware neuron model, resulting in final N outputs. The target hardware consists of a large number of such processing cores, and the processing cores may communicate with one another. The hardware neural network conversion technology (the compiling layer 1200 in FIG. 1 ) according to the present disclosure only requires that each ou

CLAIMS

Claims ( 13 )

The invention claimed is:

1. A hardware neural network conversion method for converting a neural network application into a hardware neural network fulfilling a hardware constraint condition, comprising:

a neural network connection diagram obtaining step of obtaining a neural network connection diagram corresponding to the neural network application, wherein, the neural network connection diagram is a directed diagram, each node in the diagram represents a layer of neurons from the neural network application and each edge represents an inter-layer connection relationship;

a neural network connection diagram splitting step of splitting the neural network connection diagram into neural network basic units, wherein each neural network basic unit has only ingress nodes and egress nodes without any intermediate layer node, the ingress nodes are in full connection with the egress nodes, all fan-outs of each neuron in the ingress node are within the basic unit, and all fan-ins of each neuron in the egress node are within the basic unit;

a neural network basic unit conversion step of:

converting each of the neural network basic units into a basic unit hardware network having equivalent functions thereto and formed by connecting basic module virtual entities of neural network hardware, wherein one neural network basic unit corresponds to one or more basic module virtual entities of the neural network hardware, and each basic module virtual entity of the neural network hardware (1) fulfills a connectivity constraint condition of a neural network hardware basic module and (2) is capable of being directly mapped to a neural network hardware basic module; and

reconstructing a network topology for each neural network basic unit, including:

recoding inter-layer data with an autoencoder, the autoencoder being a neural network consisting of three layers of neurons including an input layer, a hidden layer and an output layer, a number of nodes of the output layer being equal to a number of nodes of the input layer, and a number of nodes of the hidden layer being greater than a dimensionality of inter-layer vector data;

training the network by reducing an error between a value of the output layer and a value of the input layer below a predetermined threshold, wherein precision of the input layer and the output layer is precision of the neural network application, and precision of the hidden layer is precision of data transmitted between the neural network hardware basic modules;

converting the autoencoder into a combination of an encoder and a decoder; and

forming a connection matrix by combining a decoder of an input node, a weight matrix of a connection of the neural network application, and an encoder of an output node, with respect to an expression that an inter-layer vector transmitted from a Kth layer to a (K+1)th layer is a hidden layer of the autoencoder used in the Kth layer, and

performing a fully expanded operation which decomposes the neural network basic unit into interconnections between basic module virtual entities;

determining a weight parameter for the reconstructed network topology, including:

initializing a weight of the network obtained by reconstructing the network topology, according to a weight of the neural network application; and

performing adjustment on the weight parameter such that the weight fulfills a hardware weight constraint;

wherein when there is a special function in the neural network application and a neural network hardware chip does not support the special function, before the fully expanded operation, constructing a specialized neural network for the special function; and

a basic unit hardware network connection step of connecting the obtained basic unit hardware networks according to a sequence of splitting in order to create a parameter file for the hardware neural network.

2. The hardware neural network conversion method according to claim 1 , further comprising: when the neural network application has a convolution layer, before the neural network connection diagram splitting step, performing network compression on the convolution layer of the neural network application, including:

obtaining a plurality of feature maps of each convolution layer; and

taking similarity between outputs generated by the feature maps on all samples as a determinantal point process (DPP) algorithm-associated matrix element by using a DPP method of: extracting a diversity subset, obtaining a highest diversity subset by using the DPP, reserving the subset, discarding other feature map nodes, projecting a vector corresponding to the discarded feature map into a linear space spanned by the reserved feature map, taking a ratio of a projection length of the discarded feature map to its original vector length as a weighting coefficient, and accumulating a connection weight between the discarded feature map and a next layer of neurons in a weighted manner onto a connection weight between the reserved feature map and the next layer of neurons.

3. The hardware neural network conversion method according to claim 1 , wherein, the fully expanded operation includes:

when a first-size matrix operation associated with the neural network basic unit exceeds a second-size matrix operation supported by the neural network hardware basic module, executing operations below, wherein a size of the first-size matrix operation exceeds a second predefined threshold, and a size of the second-size matrix operation is less than the second predefined threshold:

splitting the first-size matrix operation into a predetermined number of second-size small matrix operations, each second-size matrix operation being completed by one basic module virtual entity;

decomposing input data for the first-size matrix operation into the predetermined number of copies, and transmitting the predetermined number of copies to the predetermined number of second-size matrix operations, which is a multicast operation;

aggregating operation results from the predetermined number of second-size matrix operations to be equivalent to an operation result of the first-size matrix operation, which is a reduction operation;

when the neural network hardware chip has a first additional module supporting the multicast operation, assigning the multicast operation to be executed by a virtual entity of the first additional module; otherwise, completing the multicast operation by a first set of virtual entities of basic modules; and

when the neural network hardware chip has a second additional module supporting the reduction operation, assigning the reduction operation to be executed by a virtual entity of the second additional module virtual entity; otherwise, completing the reduction operation by a second set of virtual entities of basic modules.

4. The hardware neural network conversion method according to claim 3 , wherein, when a number of basic modules on a neural network hardware chip is less than a third predetermined threshold, a basic module is multiplexed in a time division manner.

5. The hardware neural network conversion method according to claim 1 , wherein, the performing adjustment on the weight parameter such that the weight fulfills a hardware weight constraint, includes:

(1) firstly, expressing the weight with floating point precision, and retraining the constructed network, so that an error of the constructed network from the original network is below a second predetermined threshold;

(2) when a neural network hardware chip has a configurable parameter P, determining P and k ij satisfying a condition in an Expectation Maximization (EM) algorithm, according to the parameter obtained in the retraining of step (1), wherein all weight parameters are expressed as a function of P, and are retrained to adjust P, where P is a configurable parameter of hardware abstraction, and k ij is an index of values of respective matrix elements in a set S P ;

(3) when weight precision of the neural network hardware chip is lower than the second predetermined threshold, fixing P obtained in step (2), initializing all weights to corresponding S k

P , retraining to adjust k ij , storing all the weights with floating point precision, but rounding all the weight parameters to a most approximate value in S P in a feedforward process of the training, then bringing all the weight parameters into feedforward computation, and still using floating point precision while feeding back and updating the weights, and updating weight values of floating point precision,

where, a value range of a weight matrix W of the neural network hardware basic module is considered as a set S P , each element in the set is a function of parameter P, where P is a hardware configurable parameter, each element W ij in the weight matrix is capable of being independently selected from S P , that is, index k ij is capable of being independently configured such that W ij =S k

P , so what the weight matrix W is capable of configuring is a set parameter P and an index k ij of values of respective weights in the set.

6. The hardware neural network conversion method according to claim 1 , wherein, the converting each of the neural network basic units into a network having equivalent functions thereto and formed by connecting basic module virtual entities of neural network hardware includes:

in a case where the neural network connection diagram is a directed acyclic diagram, converting the respective neural network basic units one by one, in a topological order of the neural network connection diagram;

in a case where the neural network connection diagram is a cyclic directed diagram, firstly breaking a cycle of the cyclic directed diagram, so that the neural network connection diagram is changed into a directed acyclic diagram, then converting the respective neural network basic units one by one, in a topological order of the directed acyclic diagram; and

training the respective converted neural network basic units in the topological order, wherein training input data is an output generated by a training sample having undergone a topological order-preceded basic unit hardware network, and training output label is an output generated by the training sample in a corresponding layer of the original neural network application.

7. The hardware neural network conversion method according to claim 1 , wherein,

when the neural network application is a Spiking Neural Network (SNN), the training data used in the neural network basic unit conversion step is obtained as follows: taking an electrical pulse of a stable frequency as an input with respect to the original network, recording electrical pulse discharge frequencies of respective neurons, and taking the recorded electrical pulse discharge frequencies as the training data used in the neural network basic unit conversion step.

8. The hardware neural network conversion method according to claim 1 , wherein, when a neural network involved in the neural network hardware chip is of a Spiking Neural Network (SNN) type, a functional relation of the SNN on the pulse discharge frequency is derived according to a neuron model of the SNN, and based on that the functional relation is continuous and derivable, training is performed by using a back-propagation algorithm.

9. A computing device, for converting a neural network application into a hardware neural network fulfilling a hardware constraint condition, comprising a memory and a processor, the memory having computer-executable instructions stored therein, and when executing the computer-executable instructions, the processor executing a method below:

a neural network basic unit conversion step of converting each of the neural network basic units so as to form a basic unit hardware network having equivalent functions thereto and formed by connecting basic module virtual entities of neural network hardware, wherein one neural network basic unit corresponds to one or more basic module virtual entities of the neural network hardware, and each basic module virtual entity of the neural network hardware (1) fulfills a connectivity constraint condition of a neural network hardware basic module and (2) is capable of being directly mapped to the neural network hardware basic module; and

wherein, the neural network basic unit conversion step includes: reconstructing a network topology for each neural network basic unit;

wherein, the reconstructing a network topology includes:

recoding inter-layer data with an autoencoder, the autoencoder being a neural network consisting of three layers of neurons including an input layer, a hidden layer and an output layer, the number of nodes of the output layer being equal to a number of nodes of the input layer, and a number of nodes of the hidden layer being greater than dimensionality of inter-layer vector data;

training the network by reducing an error between a value of the output layer and a value of the input layer below a predetermined threshold, precision of the input layer and the output layer is precision of the neural network application, and precision of the hidden layer is precision of data transmitted between the neural network hardware basic modules;

converting the autoencoder into a combination of an encoder and a decoder; and forming a connection matrix by combining a decoder of an input node, a weight matrix of a connection of the neural network application, and an encoder of an output node, with respect to an expression that an inter-layer vector transmitted from a Kth layer to a (K+1)th layer is a hidden layer of the autoencoder used in the Kth layer,

performing a fully expanded operation which decomposes the neural network basic unit into interconnections between basic module virtual entities;

determining a weight parameter for the reconstructed network topology, including:

initializing a weight of the network obtained by reconstructing the network topology, according to a weight of the neural network application; and

performing adjustment on the weight parameter such that the weight fulfills a hardware weight constraint;

further comprising connecting the obtained basic unit hardware networks according to a sequence of splitting in order to create a parameter file for the hardware neural network.

10. The computing device according to claim 9 , the method executed further comprising: when the neural network application has a convolution layer, before the neural network connection diagram splitting step, performing network compression on the convolution layer of the neural network application, including:

obtaining a plurality of feature maps of each convolution layer; and

11. The computing device according to claim 9 , wherein, the fully expanded operation includes:

when a first-size matrix multiplication and/or a matrix convolution operation associated with the neural network basic unit exceeds a second-size matrix operation supported by the neural network hardware basic module, executing operations below, wherein a size of the first-size matrix operation exceeds a second predefined threshold, and a size of the second-size matrix operation is less than the second predefined threshold:

splitting the first-size matrix operation into the predetermined number of second-size matrix operations, each second-size matrix operation being completed by one basic module virtual entity;

when the neural network hardware chip has a first additional module supporting the multicast operation, assigning the multicast operation to be executed by a virtual entity of the first additional module virtual entity; otherwise, completing the multicast operation by a first set of virtual entities of basic modules; and

12. A compiling method for compiling a neural network software application into a hardware neural network, comprising:

obtaining a neural network software application and a neural network hardware chip configuration situation;

converting the neural network software application into the hardware neural network, based on the neural network hardware chip configuration situation, the hardware neural network being formed by connecting basic modules of the neural network hardware chip; and

outputting a parameter file for the hardware neural network, the parameter file describing a connection relationship between the basic modules and a parameter configuration situation of respective basic modules,

wherein converting the neural network software application into the hardware neural network comprises:

a neural network basic unit conversion step of converting each of the neural network basic units into a basic unit hardware network having equivalent functions thereto and formed by connecting basic module virtual entities of neural network hardware, wherein one neural network basic unit corresponds to one or more basic module virtual entities of the neural network hardware, and each basic module virtual entity of the neural network hardware (1) fulfills a connectivity constraint condition of a neural network hardware basic module and (2) is capable of being directly mapped to a neural network hardware basic module; and

wherein, the neural network basic unit conversion step includes: reconstructing a network topology for each neural network basic unit;

wherein, the reconstructing a network topology includes:

training the network by reducing an error between a value of the output layer and a value of the input layer below a predetermined threshold, wherein precision of the input layer and the output layer being precision of the neural network application, and precision of the hidden layer being precision of data transmitted between the neural network hardware basic modules;

converting the autoencoder into a combination of an encoder and a decoder; and

performing a fully expanded operation which decomposes the neural network basic unit into interconnections between basic module virtual entities;

determining a weight parameter for the reconstructed network topology, including:

initializing a weight of the network obtained by reconstructing the network topology, according to a weight of the neural network application; and

performing adjustment on the weight parameter such that the weight fulfills a hardware weight constraint;

connecting the obtained basic unit hardware networks according to a sequence of splitting in order to create a parameter file for the hardware neural network.

13. A neural network software and hardware collaboration system, comprising:

a neural network hardware chip, wherein, the neural network hardware chip has a basic module thereon, the basic module executes a matrix vector multiplication and an activation function operation in a form of hardware, and a parameter of the basic module on the neural network hardware chip and a connection between basic modules are capable of being configured by a configuration file of a determined format;

a compiling layer unit, configured to compile a neural network application into a parameter file for a hardware neural network, wherein, the hardware neural network is capable of being mapped to one or more neural network hardware chips based on the parameter file, and the one or more mapped neural network hardware chips are capable of running a function of the neural network application,

the compiling layer unit being configured to execute a method below:

a hardware configuration data obtaining step of obtaining configuration situation data of the neural network hardware chip;

a neural network basic unit conversion step of converting each of the neural network basic units so as to form a basic unit hardware network having equivalent functions thereto and formed by connecting basic module virtual entities of neural network hardware, wherein one neural network basic unit corresponds to one or more basic module virtual entities of the neural network hardware, and each basic module virtual entity of the neural network hardware (1) fulfills a connectivity constraint condition of a neural network hardware basic module and (2) is capable of be directly mapped to the neural network hardware basic module; and

wherein the neural network basic unit conversion step includes: reconstructing a network topology for each neural network basic unit;

wherein the reconstructing a network topology includes:

training the network by reducing an error between a value of the output layer and a value of the input layer below a predetermined threshold, precision of the input layer and the output layer being precision of the neural network application, and precision of the hidden layer being precision of data transmitted between the neural network hardware basic modules;

converting the autoencoder into a combination of an encoder and a decoder; and

performing a fully expanded operation which decomposes the neural network basic unit into interconnections between basic module virtual entities;

determining a weight parameter for the reconstructed network topology, including:

initializing a weight of the network obtained by reconstructing the network topology, according to a weight of the neural network application; and

performing adjustment on the weight parameter such that the weight fulfills a hardware weight constraint;

US16/337,981

2016-09-29

Hardware neural network conversion method, computing device, compiling method and neural network software and hardware collaboration system

Active

2039-05-07

US11544539B2

( en )

Applications Claiming Priority (1)

Application Number

Priority Date

Filing Date

Title

PCT/CN2016/100783

WO2018058426A1

( en )

2016-09-29

Hardware neural network conversion method, computing device, compiling method and neural network software and hardware collaboration system

Publications (2)

Publication Number

Publication Date

US20200026992A1

US20200026992A1 ( en )

2020-01-23

US11544539B2

true

US11544539B2 ( en )

2023-01-03

Family

ID=61762937

Family Applications (1)

Application Number

Title

Priority Date

Filing Date

US16/337,981

Active

2039-05-07

US11544539B2

( en )

2016-09-29

Hardware neural network conversion method, computing device, compiling method and neural network software and hardware collaboration system

Country Status (2)

Country

Link

( 1 )

US11544539B2

( en )

( 1 )

WO2018058426A1

( en )

Cited By (1)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20210397770A1

( en )

2018-11-09

2021-12-23

Adagos

Method of neural network construction for the simulation of physical systems

Families Citing this family (114)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

WO2018125928A1

( en )

2016-12-29

2018-07-05

DeepScale, Inc.

Multi-channel sensor simulation for autonomous control systems

JP6720402B2

( en )

2017-03-21

2020-07-08

æ ªå¼ä¼ç¤¾ï¼°ï½ï½ï½ï½ï½ï½ï½ï½ ï¼®ï½ï½ï½ï½ï½ï½ï½

Server device, learned model providing program, learned model providing method, and learned model providing system

WO2018176000A1

( en )

2017-03-23

2018-09-27

DeepScale, Inc.

Data synthesis for autonomous control systems

US10387298B2

( en )

2017-04-04

2019-08-20

Hailo Technologies Ltd

Artificial neural network incorporating emphasis and focus techniques

US11615297B2

( en )

2017-04-04

2023-03-28

Hailo Technologies Ltd.

Structured weight based sparsity in an artificial neural network compiler

US12430543B2

( en )

2017-04-04

2025-09-30

Hailo Technologies Ltd.

Structured sparsity guided training in an artificial neural network

CN107239315B

( en )

2017-04-11

2019-11-15

èµçµæå¬å¸

Programming Model for Neural Network Heterogeneous Computing Platform

US10929749B2

( en )

2017-04-24

2021-02-23

Intel Corporation

Neural network optimization mechanism

US11017291B2

( en )

2017-04-28

2021-05-25

Intel Corporation

Training with adaptive runtime and precision profiling

WO2018222900A1

( en )

2017-05-31

2018-12-06

Intel Corporation

Computationally-efficient quaternion-based machine-learning system

US11580376B2

( en )

2017-06-09

2023-02-14

Korea Advanced Institute Of Science And Technology

Electronic apparatus and method for optimizing trained model

US11893393B2

( en )

2017-07-24

2024-02-06

Tesla, Inc.

Computational array microprocessor system with hardware arbiter managing memory requests

US11157441B2

( en )

2017-07-24

2021-10-26

Tesla, Inc.

Computational array microprocessor system using non-consecutive data formatting

US11409692B2

( en )

2017-07-24

2022-08-09

Tesla, Inc.

Vector computational unit

US10671349B2

( en )

2017-07-24

2020-06-02

Tesla, Inc.

Accelerated mathematical engine

US11587099B2

( en )

2017-07-27

2023-02-21

Ripple Luxembourg S.A.

Electronic payment network security

KR102931058B1

( en )

2017-12-05

2026-02-24

ì¼ì±ì ìì£¼ìíì¬

Method and apparatus for processing convolution operation in neural network

US12307350B2

( en )

2018-01-04

2025-05-20

Tesla, Inc.

Systems and methods for hardware-based pooling

US11561791B2

( en )

2018-02-01

2023-01-24

Tesla, Inc.

Vector computational unit receiving data elements in parallel from a last row of a computational array

US11533272B1

( en )

2018-02-06

2022-12-20

Amesite Inc.

Computer based education methods and apparatus

US11263517B1

( en )

2018-02-28

2022-03-01

Amazon Technologies, Inc.

Flexible weight expansion

KR102497238B1

( en )

2018-06-11

2023-02-07

ì´ëì´í¸ ìì¤ì

Characterization of activity in recurrent artificial neural networks and encoding and decoding of information

US12412072B2

( en )

2018-06-11

2025-09-09

Inait Sa

Characterizing activity in a recurrent artificial neural network

US11215999B2

( en )

2018-06-20

2022-01-04

Tesla, Inc.

Data pipeline and deep learning system for autonomous driving

US11361457B2

( en )

2018-07-20

2022-06-14

Tesla, Inc.

Annotation cross-labeling for autonomous control systems

US11636333B2

( en )

2018-07-26

2023-04-25

Tesla, Inc.

Optimizing neural network structures for embedded systems

US12387082B2

( en )

2018-07-31

2025-08-12

International Business Machines Corporation

Scheduler for mapping neural networks onto an array of neural cores in an inference processing unit

CN109344840B

( en )

2018-08-07

2022-04-01

æ·±å³å¸åæ±¤ç§ææéå¬å¸

Image processing method and apparatus, electronic device, storage medium, and program product

US11562231B2

( en )

2018-09-03

2023-01-24

Tesla, Inc.

Neural networks for embedded devices

CN109543140B

( en )

2018-09-20

2020-07-10

ä¸å½ç§å¦é¢è®¡ç®ææ¯ç ç©¶æ

Convolutional neural network accelerator

ES3063986T3

( en )

2018-10-11

2026-04-21

Tesla Inc

Systems and methods for training machine models with augmented data

US11196678B2

( en )

2018-10-25

2021-12-07

Tesla, Inc.

QOS manager for system on a chip communications

CN111104120B

( en )

2018-10-29

2023-12-22

èµçµæå¬å¸

Neural network compiling method and system and corresponding heterogeneous computing platform

CN109558564B

( en )

2018-11-30

2022-03-11

ä¸æµ·å¯æ¦çºªä¿¡æ¯ç§ææéå¬å¸

Operation method, device and related product

US11816585B2

( en )

2018-12-03

2023-11-14

Tesla, Inc.

Machine learning models operating at different frequencies for autonomous vehicles

US11537811B2

( en )

2018-12-04

2022-12-27

Tesla, Inc.

Enhanced object detection for autonomous vehicles based on field view

US11610117B2

( en )

2018-12-27

2023-03-21

Tesla, Inc.

System and method for adapting a neural network model on a hardware platform

US11150664B2

( en )

2019-02-01

2021-10-19

Tesla, Inc.

Predicting three-dimensional features for autonomous driving

US10997461B2

( en )

2019-02-01

2021-05-04

Tesla, Inc.

Generating ground truth for machine learning from time series elements

US11567514B2

( en )

2019-02-11

2023-01-31

Tesla, Inc.

Autonomous and user controlled vehicle summon to a target

US10956755B2

( en )

2019-02-19

2021-03-23

Tesla, Inc.

Estimating object properties using visual image data

US11652603B2

( en )

2019-03-18

2023-05-16

Inait Sa

Homomorphic encryption

US11569978B2

( en )

2019-03-18

2023-01-31

Inait Sa

Encrypting and decrypting information

KR102782971B1

( en )

2019-05-08

2025-03-18

ì¼ì±ì ìì£¼ìíì¬

A computing device for training an artificial neural network model, a method for training an artificial neural network model, and a memory system for storing the same

US11687789B2

( en )

2019-05-31

2023-06-27

Apple Inc.

Decomposition of machine learning operations

US11836635B2

( en )

2019-05-31

2023-12-05

Apple Inc.

Mutable parameters for machine learning models during runtime

US11080200B2

( en )

2019-05-31

2021-08-03

Apple Inc.

Allocation of machine learning tasks into a shared cache

CN112116083B

( en )

2019-06-20

2024-03-08

å°å¹³çº¿(ä¸æµ·)äººå·¥æºè½ææ¯æéå¬å¸

Neural network accelerator and detection method and device thereof

CN110377472B

( en )

2019-07-25

2021-05-18

éåºä¸æå¾®äººå·¥æºè½è¯çææ¯æéå¬å¸

Method and device for positioning operation error of chip

EP3798919A1

( en )

2019-09-26

2021-03-31

Hardware architecture for spiking neural networks and method of operating

WO2021107360A2

( en )

2019-11-29

2021-06-03

ìëªì¬ìëíêµì°ííë ¥ë¨

Electronic device for determining similarity degree and control method thereof

US11797827B2

( en )

2019-12-11

2023-10-24

Inait Sa

Input into a neural network

US11651210B2

( en )

2019-12-11

2023-05-16

Inait Sa

Interpreting and improving the processing results of recurrent neural networks

US11580401B2

( en )

2019-12-11

2023-02-14

Inait Sa

Distance metrics and clustering in recurrent neural networks

CN111126582B

( en )

2019-12-20

2024-04-05

ä¸æµ·å¯æ¦çºªä¿¡æ¯ç§ææéå¬å¸

Data processing method and related product

US12072806B2

( en )

2020-01-22

2024-08-27

Alibaba Group Holding Limited

Compression and decompression module in a cache controller for reducing off-chip data traffic

TWI727641B

( en )

2020-02-03

2021-05-11

è¯é¦é»åè¡ä»½æéå¬å¸

Memory apparatus and operation method thereof

CN111368994B

( en )

2020-02-12

2024-05-28

å¹³å®ç§æï¼æ·±å³ï¼æéå¬å¸

Node parameter updating method and related equipment based on neural network model

CN111414993B

( en )

2020-03-03

2024-03-01

ä¸æï¼ä¸å½ï¼åå¯¼ä½æéå¬å¸

Clipping and convolution calculation methods and devices for convolutional neural networks

CN111460906B

( en )

2020-03-05

2023-05-26

éåºå¤§å¦

Pulse neural network mode identification method and system based on ensemble learning

CN111461316A

( en )

2020-03-31

2020-07-28

ä¸ç§å¯æ¦çºªç§æè¡ä»½æéå¬å¸

Method and device for calculating neural network, board card and computer readable storage medium

CN111563583B

( en )

2020-04-03

2022-08-09

æ¸åå¤§å¦

Optical computing chip design method based on neural ordinary differential equation and optical computing chip

FR3109651B1

( en )

2020-04-23

2022-12-16

St Microelectronics Rousset

METHOD FOR IMPLEMENTING AN ARTIFICIAL NEURON NETWORK IN AN INTEGRATED CIRCUIT

US12347421B2

( en )

2020-06-25

2025-07-01

PolyN Technology Limited

Sound signal processing using a neuromorphic analog signal processor

US20210406661A1

( en )

2020-06-25

2021-12-30

PolyN Technology Limited

Analog Hardware Realization of Neural Networks

CN111831354B

( en )

2020-07-09

2023-05-16

åäº¬çµæ±ç§ææéå¬å¸

Data precision configuration method, device, chip array, equipment and medium

CN111831359B

( en )

2020-07-10

2023-06-23

åäº¬çµæ±ç§ææéå¬å¸

Weight precision configuration method, device, equipment and storage medium

CN111950710B

( en )

2020-08-12

2024-12-13

æ·±å³å¸åæ±¤ç§ææéå¬å¸

Neural network optimization method, device, electronic device and storage medium

CN112036564B

( en )

2020-08-28

2024-01-09

è¾è®¯ç§æï¼æ·±å³ï¼æéå¬å¸

Image recognition method, device, equipment and storage medium

CN112580774B

( en )

2020-09-01

2022-10-21

æµæ±å¤§å¦

Neural network layout method for reconfigurable neural network processor

CN112130057B

( en )

2020-09-14

2024-04-16

åå°æ»¨å·¥ä¸å¤§å¦ï¼å¨æµ·ï¼

Radiation effect diagnosis system based on memristor neural network

WO2022060908A1

( en )

2020-09-15

2022-03-24

Arizona Board Of Regents On Behalf Of The University Of Arizona

Frequency multiplexed photonic neural networks

US11811421B2

( en )

2020-09-29

2023-11-07

Hailo Technologies Ltd.

Weights safety mechanism in an artificial neural network processor

US11874900B2

( en )

2020-09-29

2024-01-16

Hailo Technologies Ltd.

Cluster interlayer safety mechanism in an artificial neural network processor

US12248367B2

( en )

2020-09-29

2025-03-11

Hailo Technologies Ltd.

Software defined redundant allocation safety mechanism in an artificial neural network processor

CN112181867B

( en )

2020-09-29

2022-07-26

è¥¿å®çµåç§æå¤§å¦

On-chip network memory controller layout method based on multi-target genetic algorithm

CN112232484B

( en )

2020-10-15

2022-08-05

A method and system for identifying and capturing space debris based on brain-like neural network

CN112232486B

( en )

2020-10-19

2025-03-18

An optimization method for YOLO spiking neural network

US20220147811A1

( en )

2020-11-06

2022-05-12

Micron Technology, Inc.

Implement the computation of an artificial neural network using multiple deep learning accelerators

US12456049B2

( en )

2020-11-06

2025-10-28

Micron Technology, Inc.

Compiler with an artificial neural network to optimize instructions generated for execution on a deep learning accelerator of artificial neural networks

CN112270406B

( en )

2020-11-11

2023-05-23

æµæ±å¤§å¦

Nerve information visualization method of brain-like computer operating system

CN112269751B

( en )

2020-11-12

2022-08-23

æµæ±å¤§å¦

Chip expansion method for hundred million-level neuron brain computer

CN114491913B

( en )

2020-11-13

2025-10-21

åä¸ºææ¯æéå¬å¸

A model processing method and related equipment

US11972003B2

( en )

2020-12-03

2024-04-30

Capital One Services, Llc

Systems and methods for processing requests for access

US12190235B2

( en )

2021-01-29

2025-01-07

Microsoft Technology Licensing, Llc

System for training an artificial neural network

KR102956564B1

( en )

2021-02-25

2026-04-24

ì¼ì±ì ìì£¼ìíì¬

Neural network operation method and apparatus

CN113128682B

( en )

2021-04-14

2022-10-21

Automatic neural network model adaptation method and device

CN113077829B

( en )

2021-04-20

2023-04-28

æ¸åå¤§å¦

Data processing method based on memristor array and electronic device

CN113222121B

( en )

2021-05-31

2023-08-29

æå·æµ·åº·å¨è§æ°åææ¯è¡ä»½æéå¬å¸

A data processing method, device and equipment

CN113378009B

( en )

2021-06-03

2023-12-01

ä¸æµ·ç§æå¤§å¦

Quantitative analysis method of binary neural network based on binary decision diagram

CN113469336A

( en )

2021-06-29

2021-10-01

ä¸æµ·å¯æ¦çºªä¿¡æ¯ç§ææéå¬å¸

Compiling method and execution method for optimizing neural network model and related products

US12462575B2

( en )

2021-08-19

2025-11-04

<span itemprop="assigneeOrigin

Hardware neural network conversion method, computing device, compiling method … — Tsinghua University (US11544539B2)

Related documents