Conceptio › Archive › Google Patents

Google Patents open access

Neural network search method and related device — Huawei Technologies Co., Ltd. (US20240152770A1)

Huawei Technologies Co., Ltd. · Google Patents

Google Patents · Patents

en google-patents hang-xu huawei-technologies-co intellectual-property patent

. The method according to claim 1 , wherein the plurality of candidate neural networks comprise a target candidate neural network; the obtaining theft plurality of candidate neural networks comprises: constructing the target attention head in the target candidate neural network, the constructing the … Neural network search method and related device — Huawei Technologies Co., Ltd. (US20240152770A1)

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

ABSTRACT

Abstract

This application relates to the artificial intelligence field, and discloses a neural network search method and a related apparatus. The neural network search method includes: constructing attention heads in transformer layers by sampling a plurality of candidate operators during model search, to construct a plurality of candidate neural networks, and comparing performance of the plurality of candidate neural networks to select a target neural network with higher performance. In this application, a transformer model is constructed with reference to model search, so that a new attention structure with better performance than an original self-attention mechanism can be generated, and effect in a wide range of downstream tasks is significantly improved.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of International Application No. PCT/CN2022/105115, filed on Jul. 12, 2022, which claims priority to Chinese Patent Application No. 202110803202.X, filed on Jul. 15, 2021. The disclosures of the aforementioned applications are hereby incorporated by reference in their entireties.

TECHNICAL FIELD

This application relates to the artificial intelligence field, and in particular, to a neural network search method and a related device.

BACKGROUND

Artificial intelligence (AI) is a theory, a method, a technology, and an application system that simulates, extends, and expands human intelligence by using a digital computer or a machine controlled by a digital computer, to perceive an environment, obtain knowledge, and obtain an optimal result by using the knowledge. In other words, the artificial intelligence is a branch of computer science, and is intended to understand essence of intelligence and produce a new intelligent machine that can react in a manner similar to the human intelligence. The artificial intelligence is intended to study design principles and implementation methods of various intelligent machines, so that the machines have perception, inference, and decision-making functions.

With continuous development of artificial intelligence technologies, a natural language human-machine interaction system that enables human-machine interaction to be performed by using a natural language becomes increasingly important. The human-machine interaction to be performed by using the natural language requires the system to recognize a specific meaning of a human natural language. Generally, the system extracts key information from a natural language sentence to recognize a specific meaning of the sentence.

A transformer structure has a powerful semantic representation capability and can capture a dependency between pieces of long text. Since the transformer structure was proposed, the transformer structure has significantly surpassed previous models in terms of a series of natural language processing tasks represented by translation. A pre-trained language model based on the transformer structure has also achieved very good effect in fields such as a question-answering system and a voice assistant.

With rapid development of artificial intelligence technologies, a neural network with excellent performance usually has a delicate network architecture, which requires that experts with high skills and extensive experience need to make great efforts to construct the neural network. A neural architecture search (NAS) method is proposed to better construct a neural network, and automatic search is performed on neural network architectures, to obtain a neural network architecture with excellent performance.

An existing neural network search method for a transformer model has limited performance improvement for the transformer model.

SUMMARY

According to a first aspect, this application provides a neural network search method, and the method includes:

obtaining a plurality of candidate neural networks, where at least one candidate neural network in the plurality of candidate neural networks includes a target transformer layer, the target transformer layer includes a target attention head, the target attention head includes a plurality of operators, and the plurality of operators are obtained by sampling a plurality of candidate operators included in a first search space.

It may be determined, in a sampling or fixed setting manner, that a type of a network layer in the candidate neural network is a transformer layer (which may be referred to as the target transformer layer in this embodiment of this application), and a structure of the target attention head in the target transformer layer is determined by sampling the operators in the first search space.

In a possible implementation, a source of operator sampling may be the first search space, and the first search space may include the plurality of candidate operators. When an attention head is constructed, the plurality of candidate operators in the first search space may be sampled, and candidate operators obtained through sampling are combined to obtain the attention head in one transformer layer. After a plurality of times of sampling, attention heads in a plurality of transformer layers may be obtained. Specifically, in a possible implementation, the target attention head may be constructed by sampling the candidate operators in the first search space. Specifically, the plurality of operators and a connection relationship between the plurality of operators may be sampled from the first search space. In other words, when the target attention head is constructed, a type of each operator, a quantity of operators, and the connection relationship between the operators that are included in the target attention head may be determined in a sampling manner. Further, the target attention head (head) may be constructed based on the plurality of operators obtained through sampling and the sampled connection relationship between the plurality of operators.

It should be understood that sampling in this embodiment of this application may be random sampling or non-random sampling. Random sampling may be a sampling method for selecting a sample from a population according to a randomization principle. Random sampling includes, for example, but is not limited to, simple random sampling, systematic sampling, cluster sampling, and stratified sampling. Non-random sampling may be a sampling method for making probabilities of different pieces of sampled information in each sampling not exactly the same based on specific probability distribution or in another manner of guiding a sampling process.

Sampling in this embodiment of this application may include sampling of a network layer type (the network layer type specifically includes the transformer layer and a target network layer), sampling of a plurality of operators in an attention head (specifically including sampling of a type of the operator, sampling of a quantity of operators, and sampling of a connection relationship between the operators), sampling of a quantity of transformation matrices in the attention head, and sampling of a size of a convolution kernel of a convolutional layer in the target network layer. The foregoing sampling process may be partially implemented through random sampling, and partially implemented through non-random sampling, or may be completely implemented through random sampling, or may be completely implemented through non-random sampling.

Sampling of the network layer type is used to determine network types of network layers connected in series in the candidate neural network (the network type may be the transformer layer, the target network layer, or another network type).

Sampling of the plurality of operators in the attention head is used to determine the type of the operator, the quantity of operators, and the connection relationship between the operators that are included in the attention head. The type of the operator is obtained by sampling the plurality of candidate operators in the first search space. The quantity of operators is obtained through sampling within a preset range. For example, a quantity of operators ranging from 5 to 20 is determined for target head sampling. The connection relationship between the operators is obtained by performing sampling based on a quantity flow direction between the plurality of operators obtained through sampling. For example, the plurality of operators obtained through sampling include an operator A and an operator B. In this case, whether the operator A is connected to the operator B may be determined in a sampling manner. When a connection relationship between the operator A and the operator B exists, whether an output of the operator A is used as an input of the operator B or an output of the operator B is used as an input of the operator A may also be determined in the sampling manner.

Sampling of the quantity of transformation matrices in the attention head (head) is used to determine the quantity of transformation matrices included in the attention head (head). The sampled quantity may have an upper limit and/or a lower limit, for example, is determined through sampling in a quantity interval of 1 to 4, or is determined through sampling in a quantity interval of 2 to 4, or is determined through sampling in a quantity interval of 2 and 3. This is not limited herein. The method further includes: selecting a target neural network from the plurality of candidate neural networks based on performance of the plurality of candidate neural networks.

After the plurality of candidate neural networks are obtained, the plurality of neural networks may be trained to obtain performance of each candidate neural network, and then the target neural network may be selected from the plurality of candidate neural networks based on the performance of each candidate neural network, where there is at least one target neural network. When there is one target neural network, the target neural network may be a model with best performance in the plurality of candidate neural networks. When there are a plurality of target neural networks, the target neural networks may be a plurality of models with best performance in the plurality of candidate neural networks.

In the foregoing manner, combined with model search, a new attention structure that is stronger than an original self-attention mechanism can be generated, and effect in a wide range of downstream tasks is significantly improved.

In one embodiment, the first search space includes the plurality of candidate operators, and the candidate operators are unary operators or binary operators. The unary operator refers to performing an operation on only one piece of data, for example, a negative (neg) number operation, a square root (sqrt) operation, a transpose operation, a softmax operation, a logsigmoid operation, and a softsign operation. The binary operator (binary operation) refers to a rule for performing an operation on two pieces of data to obtain a third piece of data, for example, an add operation, a dot multiplication (matmul) operation, a cosine similarity operation, and a euclidean distance operation.

Compared with operator types of an existing attention head, types of the foregoing candidate operators are more abundant, which greatly increases a structure type of a candidate transformer layer, and further increases a possibility of finding a transformer model with better performance.

In one embodiment, the plurality of candidate operators include a softmax operator and a dot multiplication operator.

On a premise that operator types are enriched, a softmax operator and a dot multiplication operator in the existing attention head (head) are retained. Because the softmax operator and the dot multiplication operator are important operator types in an attention mechanism, when operator sampling is performed for the attention head (head), the absence of the two operator types may cause an excessively large difference between a structure of a found attention head and the existing attention head. On one hand, it is difficult to sample an attention head structure with excellent performance. On the other hand, it also increases time and computing overheads of a search process.

In one embodiment, the target attention head may be constructed by sampling the candidate operators in the first search space. Specifically, the plurality of operators and the connection relationship between the plurality of operators may be sampled from the first search space. In other words, when the target attention head is constructed, the type of each operator, the quantity of operators, and the connection relationship between the operators that are included in the target attention head may be determined in the sampling manner. Further, the target attention head may be constructed based on the plurality of operators obtained through sampling and the sampled connection relationship between the plurality of operators.

Compared with the operator types of the existing attention head, the types of the foregoing candidate operators are more abundant, which greatly increases the structure type of the candidate transformer layer, and further increases the possibility of finding the transformer model with better performance.

In one embodiment, the target transformation matrix includes only X transformation matrices, X is a positive integer less than or equal to 4, and a quantity of X is determined in a sampling manner.

For example, the target transformation matrix may include only one of a Q transformation matrix, a V transformation matrix, and a K transformation matrix. Alternatively, the target transformation matrix includes only two of a Q transformation matrix, a V transformation matrix, and a K transformation matrix. Alternatively, the target transformation matrix includes a Q transformation matrix, a V transformation matrix, and a K transformation matrix.

For example, another transformation matrix (for example, referred to as a P transformation matrix) may be constructed. A structure of the P transformation matrix is similar to or completely consistent with those of the other transformation matrices. Further, the target transformation matrix may include at least one of the Q transformation matrix, the V transformation matrix, the K transformation matrix, and the P transformation matrix.

In one embodiment, a matrix type of a transformation matrix included in the target transformation matrix may be determined in a sampling manner, where the matrix type is the Q transformation matrix, the K transformation matrix, or the V transformation matrix, or the matrix type of the transformation matrix included in the target transformation matrix is preset. This is not limited herein. When the matrix type of the transformation matrix included in the target transformation matrix is determined in the sampling manner, a possibility of the structure of the target attention head can be increased, and a model with better performance can be found.

In one embodiment, the target attention head further includes a second linear transformation layer, and the second linear transformation layer is used to perform linear transformation on the data processing result of the plurality of operators, to obtain an output vector of the target attention head.

In one embodiment, sizes of the input vector of the target attention head and the output vector of the target attention head are the same.

When search is performed only for an attention head in a transformer layer to re-determine a structur

CROSS-REFERENCE TO RELATED APPLICATIONS

TECHNICAL FIELD

This application relates to the artificial intelligence field, and in particular, to a neural network search method and a related device.

BACKGROUND

An existing neural network search method for a transformer model has limited performance improvement for the transformer model.

SUMMARY

According to a first aspect, this application provides a neural network search method, and the method includes:

In one embodiment, the plurality of candidate operators include a softmax operator and a dot multiplication operator.

In one embodiment, the target transformation matrix includes only X transformation matrices, X is a positive integer less than or equal to 4, and a quantity of X is determined in a sampling manner.

In one embodiment, sizes of the input vector of the target attention head and the output vector of the target attention head are the same.

When search is performed only for an attention head in a transformer layer to re-determine a structure, to ensure that work of another network layer of the transformer layer is not affected, a relationship characteristic between an input and an output of the attention head in the conventional technology may be retained. In other words, it is ensured that the input vector of the target attention head and the output vector of the target attention head have the same size. In the foregoing manner, costs of adaptive modification at the another network layer of the transformer layer are reduced, thereby improving network search efficiency.

In one embodiment, the quantity of operators included in the target attention head is less than a preset value. For example, the preset value may be 10, 11, 12, 14, 15, 20, or 21.

In a process of searching for an attention head, an upper limit of a sampled quantity of operators is set, to ensure that a size of a found network is not excessively large, and further, a model with good performance on a premise that a specific model size constraint is met can be found.

In one embodiment, the target transformer layer in the transformer model may be constructed in a manner of operator sampling. Specifically, the target attention head in the target transformer layer in the transformer model may be constructed in the manner of operator sampling. The target transformer layer may include a plurality of attention heads (heads), and the target attention head may be any one of the plurality of attention heads (heads). In one embodiment, structures of all attention heads (heads) in the plurality of attention heads (heads) are the same. In one embodiment, the at least one candidate neural network includes a plurality of network layers connected in series, the plurality of network layers include the target transformer layer, and a location of the target transformer layer in the plurality of network layers is determined in a sampling manner.

Compared with network layer types of network layers connected in series in an existing transformer model, the network type of the network layer determined in the sampling manner (for example, the target transformer layer or the subsequent target network layer) can greatly increase the structure type of the candidate transformer layer, and the possibility of finding the transformer model with better performance is further increased.

In one embodiment, the at least one candidate neural network includes the plurality of network layers connected in series, the plurality of network layers include the target transformer layer and the target network layer, and the target network layer includes a convolutional layer. A convolution kernel in the convolutional layer may be obtained by sampling convolution kernels of a plurality of sizes included in a second search space.

In one embodiment, it may be determined, in the sampling or fixed setting manner, that the type of the network layer in the candidate neural network is the target network layer including the convolutional layer, and a size of the convolution kernel in the convolution layer in the target network layer is determined through sampling from the second search space.

In this embodiment of this application, diversified search spaces are designed, and include both a local operator (the convolution kernel in the convolutional layer) operator and a global operator (an operator in the transformer layer). The global operator can construct a new attention mechanism in combination with a mathematical basic operator, and the local operator includes a plurality of convolution kernels of different sizes. The global operator and the local operator are combined, so that an association relationship between words or sentences can be captured more effectively, and performance of a found model can be improved. In addition, the neural network model in this embodiment of this application may be used as a pre-training model, and is applicable to a plurality of downstream tasks.

In one embodiment, because lightweight convolution achieves good performance on a series of natural language understanding tasks (such as machine translation), the convolution kernel may use a lightweight convolution architecture to improve model performance.

In one embodiment, the target network layer further includes a first addition and normalization layer, a feed-forward layer (FFN), and a second addition and normalization layer. The first addition and normalization layer is used to process an input vector of the target network layer and an output vector of the convolutional layer, and the feed-forward layer FFN is used to process the output vector of the first addition and normalization layer. The second addition and normalization layer is used to process the output vector of the first addition and normalization layer and an output vector of the feed-forward layer FFN. In other words, an addition and normalization layer and an FFN in an existing transformer layer, and a residual connection architecture may be retained, and an attention head is replaced with a convolutional layer, so that the target network layer in this embodiment of this application can be obtained. A type of the replaced convolution layer may be obtained by performing convolution kernel sampling in the second search space.

In one embodiment, the plurality of candidate neural networks include a target candidate neural network; and the obtaining a plurality of candidate neural networks specifically includes: constructing a target attention head in the target candidate neural network; and

the constructing a target attention head in the target candidate neural network includes: obtaining a first neural network, where the first neural network includes a first transformer layer, the first transformer layer includes a first attention head, and a plurality of operators included in the first attention head are obtained by sampling a plurality of candidate operators included in the first search space; and determining replacement operators from M candidate operators based on positive impact on performance of the first neural network when target operators in the first attention head are replaced with the M candidate operators in the first search space; and replacing the target operators in the first attention head with the replacement operators, to obtain the target attention head.

In one embodiment, the plurality of candidate neural networks may be constructed through sampling. To select a model with good performance, a quantity of sampled candidate neural networks is large. The performance of the plurality of candidate neural networks may be determined through training, and a specific quantity of networks are preliminarily selected from the plurality of candidate neural networks as parent networks based on the performance of the plurality of candidate neural networks. Then, operators in the parent networks can be replaced (if transformer layers are used, operators in attention heads are replaced. If target network layers are used, convolution kernels may be replaced), to obtain a plurality of sub-networks, the plurality of sub-networks are trained to determine performance of the plurality of sub-networks, the target neural network is determined from the plurality of sub-networks based on the performance of the plurality of sub-networks, and the target neural network is used as a search result of the neural network.

The foregoing initially constructed candidate neural network may be referred to as a second neural network, the parent network may be referred to as the first neural network, and the sub-network may be referred to as the candidate neural network.

In this embodiment of this application, a plurality of second neural networks may be obtained in a sampling manner (for details, refer to the descriptions of obtaining the candidate neural networks through sampling in the foregoing embodiment, and details are not described herein again), and the plurality of second neural networks are trained to obtain a plurality of trained second neural networks and performance of the plurality of trained second neural networks. Specifically, random parameter initialization may be performed on the plurality of second neural networks, and fast search training (for example, 4w operations of training) is performed on the plurality of second neural networks, to obtain the plurality of trained second neural networks. In addition, evaluation is performed on the plurality of trained second neural networks by using a GLUE task, to obtain the performance of the plurality of second neural networks, N optimal networks are selected as parent networks, and training parameters of the parent networks are stored. The N parent networks may include the first neural network. The first neural network may include the first transformer layer, the first transformer layer includes the first attention head, and the first attention head includes the target operators. Then, the replacement operators may be determined from the M candidate operators based on the positive impact on the performance of the first neural network when the target operators in the first attention head are replaced with the M candidate operators in the first search space, and the target operators in the first attention head are replaced with the replacement operators, to obtain the target attention head.

In one embodiment, the obtaining a first neural network includes:

obtaining the plurality of second neural networks, and training the plurality of second neural networks to obtain the plurality of trained second neural networks and the performance of the plurality of trained second neural networks; and selecting, from the plurality of trained second neural networks based on the performance of the plurality of trained second neural networks, N trained second neural networks whose performance meets a preset requirement, where the N trained second neural networks include the first neural network.

The target operator is used as an example. The target operator may be located at a target operator location of the second neural network. The target operator location may represent, to some extent, a location of a distance from an input of the head, and the target operator location may be related to a manner of representing a location between network operators in code. When positive impact is calculated, a manner of calculating a location of each operator in the second neural network is consistent with a manner of calculating the target operator location of the target operator in the second neural network, and each of the manners can express a degree of positive impact of a different location of the operator in the attention head on model performance. When the positive impact is calculated, the positive impact on the performance of the first neural network may be determined based on an operator that is in each of the plurality of trained second neural networks and that is located at the target operator location and the performance of the plurality of trained second neural networks, and/or an occurrence frequency of the operator that is in each trained second neural network and that is located at the target operator location when the target operators in the first attention head are replaced with the M candidate operators in the first search space.

For example, the positive impact may be represented by using an upper confidence bound UCB (upper confidence bound), and a specific UCB score calculation manner may be as follows:

Î¼

â¢

log

â¢

where

Î¼ i represents a score obtained by an operator i in a current location of a network structure, N i represents a quantity of times that the operator i is sampled in history (when the second neural network is sampled), and N represents a quantity of times that all operators are sampled. When an operator is rarely sampled, a larger value is obtained in the right half of the formula, and the current operator is selected with a higher probability. It should be understood that after UCB scores of all operators at all locations are calculated, softmax calculation may be performed on these scores to obtain probability distribution. A probability is set to a probability that the operator i is activated at the current location.

In this embodiment of this application, operator replacement is performed by using the positive impact, so that search precision and search breadth of an algorithm can be balanced, and a local optimal network architecture can be avoided, and a better network architecture can be continuously found.

In one embodiment, the method further includes:

performing parameter initialization on the target candidate neural network based on the first neural network, to obtain an initialized target candidate neural network, where an updatable parameter in the initialized target candidate neural network is obtained by performing parameter sharing on an updatable parameter at a same location in the first neural network; and training the target candidate neural network on which parameter initialization is performed, to obtain performance of the target candidate neural network.

When a parameter of the attention head is shared, an updatable parameter is a parameter in a transformation matrix in the attention head. When a parameter of the convolutional layer is shared, an updatable parameter is a convolution kernel. It should be understood that a corresponding parameter at a centralmost location of the convolution kernel may be selected for parameter sharing.

In this embodiment of this application, parameter initialization is performed in a parameter sharing manner, so that a search speed can be accelerated, repeated training can be avoided, and search efficiency can be greatly improved.

In one embodiment, the target neural network is used to implement at least one of the following task types:

reading comprehension, text translation, retelling recognition, named entity recognition, text sentiment analysis, natural language inference, automatic text question answering, text intent recognition, text classification, text simplification, or text story generation.

According to a second aspect, this application provides a model providing method. The method includes:

receiving a performance requirement sent by a device side, where the performance requirement indicates a performance requirement of a neural network, and the performance requirement may include at least one of the following: data processing precision, a model size, and an implemented task type; obtaining, from a plurality of candidate neural networks based on the performance requirement, a target neural network that meets the performance requirement, where at least one candidate neural network in the plurality of candidate neural networks includes a target transformer layer, the target transformer layer includes a target attention head, the target attention head includes a plurality of operators, and the plurality of operators are obtained by sampling a plurality of candidate operators included in a first search space; and sending the target neural network to the device side.

In one embodiment, the first search space includes the plurality of candidate operators, and the candidate operators are unary operators or binary operators. The target attention head is constructed based on the plurality of operators and an arrangement relationship between the plurality of operators, and the arrangement relationship between the plurality of operators is determined in a sampling manner.

In one embodiment, the target attention head further includes a first linear transformation layer, the first linear transformation layer is used to process an input vector of the target attention head by using a target transformation matrix, and the plurality of operators are used to perform an operation on a data processing result of the first linear transformation layer. The target transformation matrix includes only X transformation matrices, X is a positive integer less than or equal to 4, and a quantity of X is determined in a sampling manner.

In one embodiment, the at least one candidate neural network includes a plurality of network layers connected in series, the plurality of network layers include the target transformer layer, and a location of the target transformer layer in the plurality of network layers is determined in a sampling manner.

In one embodiment, a location of the target network layer in the plurality of network layers is determined in a sampling manner.

In one embodiment, a convolution kernel in the convolutional layer is obtained by sampling convolution kernels of a plurality of sizes included in a second search space.

In one embodiment, a type of the convolution kernel in the convolutional layer is lightweight convolution (lightweight convolution).

According to a third aspect, this application provides a neural network search method. The method includes:

obtaining a plurality of candidate neural networks, where at least one candidate neural network includes a plurality of network layers connected in series, the plurality of network layers include a target transformer layer and a target network layer, the target network layer includes a convolutional layer, and a convolution kernel in the convolution layer is obtained by sampling convolution kernels of a plurality of sizes included in a second search space; and selecting a target neural network from the plurality of candidate neural networks based on performance of the plurality of candidate neural networks.

In one embodiment, a type of the convolution kernel in the convolutional layer is lightweight convolution.

In one embodiment, the target network layer further includes a first addition and normalization layer, a feed-forward layer FFN, and a second addition and normalization layer. The first addition and normalization layer is used to process an input vector of the target network layer and an output vector of the convolutional layer, and the feed-forward layer FFN is used to process the output vector of the first addition and normalization layer. The second addition and normalization layer is used to process the output vector of the first addition and normalization layer and an output vector of the feed-forward layer FFN.

In one embodiment, the target neural network is used to implement at least one of the following task types:

According to a fourth aspect, this application provides a data processing method. The method includes:

obtaining a target neural network, where the target neural network includes a plurality of serial network layers, the plurality of network layers include a target transformer layer and a target network layer, and the target network layer includes a convolutional layer; and obtaining to-be-processed data, and processing the to-be-processed data by using the target neural network, to obtain a data processing result.

In one embodiment, the target transformer layer includes a target attention head, the target attention head includes a plurality of operators, and the plurality of operators are unary operators or binary operators.

In one embodiment, the target attention head includes the plurality of operators, and the plurality of operators are obtained by sampling a plurality of candidate operators included in a first search space.

In one embodiment, the target transformation matrix includes only X transformation matrices, and X is a positive integer less than or equal to 4.

In one embodiment, a quantity of X is determined in a sampling manner. Specifically, a matrix type of a transformation matrix included in the target transformation matrix may be determined in a sampling manner, where the matrix type is the Q transformation matrix, the K transformation matrix, or the V transformation matrix, or the matrix type of the transformation matrix included in the target transformation matrix is preset. This is not limited herein. When the matrix type of the transformation matrix included in the target transformation matrix is determined in the sampling manner, a possibility of a structure of the target attention head can be increased, and a model with better performance can be found.

In one embodiment, sizes of the input vector of the target attention head and the output vector of the target attention head are the same.

In one embodiment, a quantity of operators included in the target attention head is less than a preset value. For example, the preset value may be 10, 11, 12, 14, 15, 20, or 21.

In one embodiment, the target transformer layer may include a plurality of attention heads (heads), and the target attention head may be any one of the plurality of attention heads (heads). In one embodiment, structures of all attention heads (heads) in the plurality of attention heads (heads) are the same.

In one embodiment, a location of the target transformer layer in the plurality of network layers is determined in a sampling manner.

In one embodiment, a convolution kernel in the convolutional layer may be obtained by sampling convolution kernels of a plurality of sizes included in a second search space.

In one embodiment, the convolutional layer is included in the target network layer in the plurality of network layers, and the target network layer further includes a first addition and normalization layer, a feed-forward layer FFN, and a second addition and normalization layer. The first addition and normalization layer is used to process an input vector of the target network layer and an output vector of the convolutional layer, and the feed-forward layer FFN is used to process the output vector of the first addition and normalization layer. The second addition and normalization layer is used to process the output vector of the first addition and normalization layer and an output vector of the feed-forward layer FFN.

In one embodiment, a location of the target network layer in the plurality of network layers is determined in a sampling manner.

According to a fifth aspect, this application provides a model providing method. The method includes:

receiving a performance requirement sent by a device side, where the performance requirement indicates a performance requirement of a neural network, and the performance requirement may include at least one of the following: data processing precision, a model size, and an implemented task type; obtaining, from a plurality of candidate neural networks based on the performance requirement, a target neural network that meets the performance requirement, where the target neural network includes a target transformer layer and a target network layer, the target network layer includes a convolutional layer, and a convolution kernel in the convolution layer is obtained by sampling convolution kernels of a plurality of sizes included in a second search space; and sending the target neural network to the device side.

In one embodiment, a type of the convolution kernel in the convolutional layer is lightweight convolution.

In one embodiment, the target neural network is used to implement at least one of the following task types:

According to a sixth aspect, a neural network search apparatus is provided. The apparatus includes:

an obtaining module, configured to obtain a plurality of candidate neural networks, where at least one candidate neural network in the plurality of candidate neural networks includes a target transformer layer, the target transformer layer includes a target attention head, the target attention head includes a plurality of operators, and the plurality of operators are obtained by sampling a plurality of candidate operators included in a first search space; and a model selection module, configured to select a target neural network from the plurality of candidate neural networks based on performance of the plurality of candidate neural networks.

In one embodiment, the first search space includes the plurality of candidate operators, and the candidate operators are unary operators or binary operators. The unary operator refers to performing an operation on only one piece of data, for example, a negative (neg) number operation, a square root (sqrt) operation, a transpose operation, a softmax operation, a logsigmoid operation, and a softsign operation. The binary operator refers to a rule for performing an operation on two pieces of data to obtain a third piece of data, for example, an add operation, a dot multiplication (matmul) operation, a cosine similarity operation, and a euclidean distance operation.

In one embodiment, the plurality of candidate operators include a softmax operator and a dot multiplication operator.

In one embodiment, the target transformation matrix includes only X transformation matrices, X is a positive integer less than or equal to 4, and a quantity of X is determined in a sampling manner.

In one embodiment, sizes of the input vector of the target attention head and the output vector of the target attention head are the same.

<div i

CLAIMS

Claims ( 20 )

1 . A method of neural network search, comprising:

obtaining a plurality of candidate neural networks, wherein at least one candidate neural network in the plurality of candidate neural networks comprises a target transformer layer, the target transformer layer comprises a target attention head comprising a plurality of operators, and the plurality of operators are obtained by sampling a plurality of candidate operators comprised in a first search space; and selecting a target neural network from the plurality of candidate neural networks based on performance of the plurality of candidate neural networks.

2 . The method according to claim 1 , wherein the target attention head is constructed based on the plurality of operators and an arrangement relationship between the plurality of operators determined in a sampling manner.

3 . The method according to claim 1 , wherein the target attention head further comprises a first linear transformation layer to process an input vector of the target attention head by using a target transformation matrix, and the plurality of operators are used to perform an operation on a data processing result of the first linear transformation layer.

4 . The method according to claim 3 , wherein the target transformation matrix comprises only X transformation matrices, X is a positive integer less than or equal to 4, and a quantity of X is determined in a sampling manner.

5 . The method according to claim 3 , wherein a sizes of the input vector of the target attention head and a size of an output vector of the target attention head are the same.

6 . The method according to claim 1 , wherein a quantity of operators comprised in the target attention head is less than a preset value.

7 . The method according to claim 1 , wherein the at least one candidate neural network comprises a plurality of network layers connected in series, the plurality of network layers comprise the target transformer layer, and a location of the target transformer layer in the plurality of network layers is determined in a sampling manner.

8 . The method according to claim 1 , wherein the at least one candidate neural network comprises the plurality of network layers connected in series, the plurality of network layers comprise the target transformer layer and a target network layer, and the target network layer comprises a convolutional layer.

9 . The method according to claim 8 , wherein a location of the target network layer in the plurality of network layers is determined in a sampling manner.

10 . The method according to claim 8 , wherein a convolution kernel in the convolutional layer is obtained by sampling convolution kernels of a plurality of sizes comprised in a second search space.

11 . The method according to claim 1 , wherein the plurality of candidate neural networks comprise a target candidate neural network; the obtaining theft plurality of candidate neural networks comprises:

constructing the target attention head in the target candidate neural network, the constructing the target attention head in the target candidate neural network comprising:

obtaining a first neural network, wherein the first neural network comprises a first transformer layer comprising a first attention head, and a plurality of operators comprised in the first attention head are obtained by sampling the plurality of candidate operators comprised in the first search space; and

determining replacement operators from M candidate operators of the plurality of candidate operators based on positive impact on performance of the first neural network when target operators in the first attention head are replaced with the M candidate operators in the first search space; and replacing the target operators in the first attention head with the replacement operators, to obtain the target attention head, wherein M is a positive integer.

12 . The method according to claim 11 , further comprising:

in response to a target operator of the target operators is located at a target operator location of a second neural network, determining, based on an operator that is in each of a plurality of trained second neural networks and that is located at the target operator location and performance of the plurality of trained second neural networks, and/or an occurrence frequency of the operator that is in each trained second neural network and that is located at the target operator location, the positive impact on the performance of the first neural network when the target operators in the first attention head are replaced with the M candidate operators in the first search space.

13 . The method according to claim 11 , further comprising:

performing parameter initialization on the target candidate neural network based on the first neural network, to obtain an initialized target candidate neural network, wherein an updatable parameter in the initialized target candidate neural network is obtained by performing parameter sharing on an updatable parameter at a same location in the first neural network; and training the target candidate neural network on which parameter initialization is performed, to obtain performance of the target candidate neural network.

14 . A method of model providing, wherein the method comprising:

receiving, from a device side, a performance requirement indicating a performance requirement of a neural network; obtaining, from a plurality of candidate neural networks based on the performance requirement, a target neural network that meets the performance requirement, wherein at least one candidate neural network in the plurality of candidate neural networks comprises a target transformer layer, the target transformer layer comprises a target attention head comprising a plurality of operators, and the plurality of operators are obtained by sampling a plurality of candidate operators comprised in a first search space; and sending, to the device side, the target neural network.

15 . The method according to claim 14 , wherein the performance requirement comprises at least one of data processing precision, a model size, or an implemented task type.

16 . The method according to claim 14 , wherein the target attention head is constructed based on the plurality of operators and an arrangement relationship between the plurality of operators determined in a sampling manner.

17 . An apparatus for neural network search, comprising:

at least one processor; and one or more memories coupled to the at least one processor and storing programming instructions for execution by the at least one processor to cause the apparatus to: obtain a plurality of candidate neural networks, wherein at least one candidate neural network in the plurality of candidate neural networks comprises a target transformer layer, the target transformer layer comprises a target attention head comprising a plurality of operators, and the plurality of operators are obtained by sampling a plurality of candidate operators comprised in a first search space; and select a target neural network from the plurality of candidate neural networks based on performance of the plurality of candidate neural networks.

18 . The apparatus according to claim 17 , wherein the target attention head is constructed based on the plurality of operators and an arrangement relationship between the plurality of operators determined in a sampling manner.

19 . The apparatus according to claim 17 , wherein the target attention head further comprises a first linear transformation layer to process an input vector of the target attention head by using a target transformation matrix; and the plurality of operators are used to perform an operation on a data processing result of the first linear transformation layer.

20 . The apparatus according to claim 19 , wherein the target transformation matrix comprises only X transformation matrices, X is a positive integer less than or equal to 4, and a quantity of X is determined in a sampling manner.

US18/411,616

2021-07-15

2024-01-12

Neural network search method and related device

Pending

US20240152770A1

( en )

Applications Claiming Priority (3)

Application Number

Priority Date

Filing Date

Title

CN202110803202.X

2021-07-15

CN202110803202.XA

CN113656563B

( en )

2021-07-15

A neural network search method and related equipment

PCT/CN2022/105115

WO2023284716A1

( en )

2021-07-15

2022-07-12

Neural network searching method and related device

Related Parent Applications (1)

Application Number

Title

Priority Date

Filing Date

PCT/CN2022/105115

Continuation

WO2023284716A1

( en )

2021-07-15

2022-07-12

Neural network searching method and related device

Publications (1)

Publication Number

Publication Date

US20240152770A1

true

US20240152770A1 ( en )

2024-05-09

Family

ID=78489439

Family Applications (1)

Application Number

Title

Priority Date

Filing Date

US18/411,616

Pending

US20240152770A1

( en )

2021-07-15

2024-01-12

Neural network search method and related device

Country Status (4)

Country

Link

( 1 )

US20240152770A1

( en )

( 1 )

EP4361843A4

( en )

( 1 )

CN113656563B

( en )

( 1 )

WO2023284716A1

( en )

Cited By (1)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

CN121119548A

( en )

2025-08-29

2025-12-12

ä¸æ³°æºäº(åäº¬)ç§ææéå¬å¸

Artificial Intelligence-Based Intelligent Dispatch Method and System for Transportation Vehicles

Families Citing this family (14)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

CN113656563B

( en )

2021-07-15

2024-06-28

åä¸ºææ¯æéå¬å¸

A neural network search method and related equipment

CN114781621A

( en )

2022-05-06

2022-07-22

åäº¬ç¾åº¦ç½è®¯ç§ææéå¬å¸

Neural network determining method and device, electronic equipment and storage medium

CN117592509A

( en )

2022-08-08

2024-02-23

ç¬¬åèå¼ï¼åäº¬ï¼ææ¯æéå¬å¸

Text recognition network generation method and device

CN116168249A

( en )

2023-02-28

2023-05-26

å²åºç§æ(ä¸æµ·)æéå¬å¸

Search method and device, chip and computer readable storage medium

CN117273074A

( en )

2023-04-14

2023-12-22

åä¸ºææ¯æéå¬å¸

A data processing method and its device

US12688245B2

( en )

2023-05-18

2026-07-21

Cisco Technology, Inc.

Generative knowledge search engine for multi-query enabled network knowledge completion

CN116959575A

( en )

2023-05-18

2023-10-27

è¾è®¯ç§æï¼æ·±å³ï¼æéå¬å¸

Affinity prediction method, device, computer equipment and storage medium

CN116945191B

( en )

2023-09-11

2024-08-02

åäº¬é¾ç¿¼é£ç§ææéå¬å¸

Robot control method based on artificial intelligence

CN117239743B

( en )

2023-11-15

2024-02-27

éå²é¼ä¿¡éè®¯è¡ä»½æéå¬å¸

A method, device, equipment and medium for obtaining electrical load of an electric energy meter

CN120012906A

( en )

2023-11-15

2025-05-16

åä¸ºææ¯æéå¬å¸

A data processing method and device thereof

CN117708568B

( en )

2024-02-02

2024-07-12

æºæ§ç¼ç§æè¡ä»½æéå¬å¸

Feature extraction method and device for large language model, computer equipment and medium

CN118628499B

( en )

2024-08-14

2024-10-29

æ¹åå¤§å¦

Aeroengine blade defect detection method based on network architecture search

CN119272022B

( en )

2024-12-10

2025-03-25

åå·çç§å¦åä¹ä¿¡ç§ææéå¬å¸

Flexible DC support capacitor quality analysis method and system based on artificial intelligence

CN122347210A

( en )

2025-01-06

2026-07-07

åä¸ºææ¯æéå¬å¸

Model reasoning method and device and electronic equipment

Family Cites Families (9)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20220121906A1

( en )

2019-01-30

2022-04-21

Google Llc

Task-aware neural network architecture search

CN112445823B

( en )

2019-09-04

2024-11-26

åä¸ºææ¯æéå¬å¸

Neural network structure search method, image processing method and device

CN112561027B

( en )

2019-09-25

2025-02-07

åä¸ºææ¯æéå¬å¸

Neural network architecture search method, image processing method, device and storage medium

CN111242176B

( en )

2019-12-31

2023-10-13

åäº¬è¿æ ¼å¨ç§ææéå¬å¸

Method and device for processing computer vision task and electronic system

CN112541159A

( en )

2020-09-30

2021-03-23

åä¸ºææ¯æéå¬å¸

Model training method and related equipment

CN112579063B

( en )

2021-03-01

2021-06-08

ä¹æ±å®éªå®¤

Acceleration method for exploring optimization space in deep learning compiler

CN112949832B

( en )

2021-03-25

2024-04-16

é¼å¯æºè½ç§ææéå¬å¸

Network structure searching method and device, electronic equipment and storage medium

CN113065645B

( en )

2021-04-30

2024-04-09

åä¸ºææ¯æéå¬å¸

Twin attention network, image processing method and device

CN113656563B

( en )

2021-07-15

2024-06-28

åä¸ºææ¯æéå¬å¸

A neural network search method and related equipment

2021

2021-07-15

CN202110803202.XA

patent/CN113656563B/en

active

Active

2022

2022-07-12

PCT/CN2022/105115

patent/WO2023284716A1/en

not_active

Ceased

2022-07-12

EP22841352.2A

patent/EP4361843A4/en

active

Pending

2024

2024-01-12

US18/411,616

patent/US20240152770A1/en

active

Pending

Cited By (1)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

CN121119548A

( en )

2025-08-29

2025-12-12

ä¸æ³°æºäº(åäº¬)ç§ææéå¬å¸

Artificial Intelligence-Based Intelligent Dispatch Method and System for Transportation Vehicles

Also Published As

Publication number

Publication date

EP4361843A4

( en )

2024-11-06

CN113656563A

( en )

2021-11-16

CN113656563B

( en )

2024-06-28

WO2023284716A1

( en )

2023-01-19

EP4361843A1

( en )

2024-05-01

Neural network search method and related device — Huawei Technologies Co., Ltd. (US20240152770A1)

Analyze & explore related research in Conceptio

Related documents