Conceptio › Archive › Google Patents

Google Patentsopen access

Method and apparatus of training natural language processing model, and method … — Beijing Baidu Netcom Science Technology Co., Ltd. (US12131728B2)

Beijing Baidu Netcom Science Technology Co., Ltd. · Google Patents

Google Patents · Patents · License: Open Access

beijingbaidunetcomsciencetechnologyco.ltd.

patent, google patents, intellectual property, US12131728B2, Beijing Baidu Netcom Science Technology Co., Ltd., Siyu DING, en, 2024

ABSTRACT

Abstract

The present application provides a method of training a natural language processing model, which relates to a field of artificial intelligence, and in particular to a field of natural language processing. A specific implementation scheme includes: performing a semantic learning for multi-tasks on an input text, so as to obtain a semantic feature for the multi-tasks, wherein the multi-tasks include a plurality of branch tasks; performing a feature learning for each branch task based on the semantic feature, so as to obtain a first output result for each branch task; calculating a loss for each branch task according to the first output result for the branch task; and adjusting a parameter of the natural language processing model according to the loss for each branch task. The present application further provides a method of processing a natural language, an electronic device, and a storage medium.

Description

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application claims the priority of Chinese Patent Application No. 202110747046.X, filed on Jun. 30, 2021, the entire contents of which are hereby incorporated by reference.

TECHNICAL FIELD

The present disclosure relates to a field of an artificial intelligence technology, in particular to a natural language processing technology. More specifically, the present disclosure provides a method and an apparatus of training a natural language processing model, a method and an apparatus of processing a natural language, an electronic device, and a storage medium.

BACKGROUND

Recently, with a continuous development of a natural language processing technology, a pre-trained language model based on large-scale corpus has gradually become a classic framework. However, a current research on language models for different tasks such as a semantic understanding and a language generation is generally carried out independently in the industry. A semantic understanding model does not have an ability of language generation, and vice versa.

Therefore, how to make a model have both the ability of semantic understanding and the ability of language generation has become a problem to be solved.

SUMMARY

The present disclosure provides a method and an apparatus of training a natural language processing model, a method and an apparatus of processing a natural language, an electronic device, and a storage medium.

According to a first aspect, there is provided a method of training a natural language processing model, including: performing a semantic learning for multi-tasks on an input text, so as to obtain a semantic feature for the multi-tasks, wherein the multi-tasks include a plurality of branch tasks; performing a feature learning for each branch task based on the semantic feature, so as to obtain a first output result for each branch task; calculating a loss for each branch task according to the first output result for the branch task; and adjusting a parameter of the natural language processing model according to the loss for each branch task.

According to a second aspect, there is provided a method of processing a natural language, including: acquiring a to-be-processed text for a preset branch task; and performing a semantic learning for multi-tasks on the to-be-processed text by using a natural language processing model, so as to obtain a semantic feature for the multi-tasks, and performing a feature learning for the preset branch task according to the semantic feature, so as to obtain a processing result for the preset branch task, wherein the multi-tasks include the preset branch task, and the natural language processing model is trained using the method of training the natural language processing model described above.

According to a third aspect, there is provided an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions, when executed by the at least one processor, cause the at least one processor to implement the method provided by the present disclosure.

According to a fourth aspect, there is provided a non-transitory computer-readable storage medium having computer instructions stored thereon, wherein the computer instructions allow a computer to implement the method provided by the present disclosure.

According to a fifth aspect of the present disclosure, there is provided a computer program product containing a computer program that, when executed by a processor, causes the processor to implement the method provided by the present disclosure.

It should be understood that content described in this section is not intended to identify key or important features in the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will be easily understood through the following description.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings are used to understand the solution better and do not constitute a limitation to the present disclosure. wherein:

FIG. 1 A shows a schematic structural diagram of a unified pre-trained language model according to an embodiment of the present disclosure.

FIG. 1 B shows a schematic system diagram of a method of training a natural language processing model and/or a method of processing a natural language according to an embodiment of the present disclosure.

FIG. 2 shows a flowchart of a method of training a natural language processing model according to an embodiment of the present disclosure.

FIG. 3 A , FIG. 3 B and FIG. 3 C show schematic diagrams of performing different branch tasks based on the above-mentioned natural language processing model according to an embodiment of the present disclosure.

FIG. 4 A and FIG. 4 B show schematic diagrams of performing a semantic generation task based on the above-mentioned natural language processing model according to an embodiment of the present disclosure.

FIG. 5 shows a schematic system diagram of a method of training a natural language processing model and/or a method of processing a natural language according to another embodiment of the present disclosure.

FIG. 6 shows a flowchart of a method of processing a natural language according to an embodiment of the present disclosure.

FIG. 7 shows a block diagram of an apparatus of training a natural language processing model according to an embodiment of the present disclosure.

FIG. 8 shows a block diagram of an apparatus of processing a natural language according to an embodiment of the present disclosure.

FIG. 9 shows a block diagram of a method of training a natural language processing model and/or a method of processing a natural language according to an embodiment of the present disclosure.

DETAILED DESCRIPTION OF EMBODIMENTS

Exemplary embodiments of the present disclosure will be described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding, and should be considered as merely exemplary. Therefore, those of ordinary skilled in the art should realize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Likewise, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

In the technical solution of the present disclosure, an acquisition, a storage and an application of user personal information involved comply with provisions of relevant laws and regulations, and do not violate public order and good custom.

Natural language processing models are used more and more widely. As branches of natural language processing, a semantic understanding and a language generation have been significantly improved in their respective research fields. For example, in a field of the semantic understanding, classic pre-trained language models such as BERT, ERNIE and XLNet have been produced, which greatly improves effects of various natural semantic understanding tasks (such as text matching, sentiment classification, retrieval, etc.). In a field of the language generation, classic pre-trained language models such as GPT, BART and T5 have been produced, which greatly improves effects of various natural language generation tasks (such as text generation, text summary generation, generative question and answer, etc.).

However, a current research on language models for different tasks such as the semantic understanding and the language generation is generally carried out independently in the industry. A semantic understanding model does not have an ability of language generation, and vice versa. Therefore, when faced with a scenario in which both the ability of semantic understanding and the ability of language generation are required, two models need to be trained, which may greatly increase a consumption of resources and time.

At present, a scheme of using a unified pre-trained language model to perform a joint training of a semantic understanding task and a language generation task has been proposed in the industry. For example, UNILM (UNIfied pre-trained Language Model) may integrate a Seq2Seq (Sequence to Sequence) task on the basis of BERT, which may initially achieve a joint training of the semantic understanding and the language generation. Using the BERT as a model structure, the UNILM may unify a unidirectional mask language model, a bidirectional mask language model and a Seq2Seq model by using a mask mechanism.

FIG. 1 A shows a schematic structural diagram of a unified pre-trained language model according to an embodiment of the present disclosure.

As shown in FIG. 1 A , the unified pre-trained language model (e.g., the UNILM) may include an embedding layer 111 and a transformer layer 112 . For example, using X 1 , X 2 . . . X 5 as an input text, the embedding layer 111 is used to perform a feature extraction and a feature vectorization representation of the input text, and output a feature vector of the input text. The transformer layer 112 is used to perform a semantic learning based on the feature vector of the input text, and output semantic features h 1 , h 2 . . . h 5 .

For example, the embedding layer 111 may include a statement embedding layer (e.g., Segment Embedding), a position information embedding layer (e.g., Position Embedding) and a word embedding layer (e.g., Token Embedding), which are respectively used to perform a statement vectorization representation of the input text, a position information vectorization representation of each word in the input text and a vectorization representation of each word, so as to output a statement feature, a position feature and a word feature.

For example, the transformer layer 112 may include a plurality of layers of transformer modules. The transformer module is a classic model architecture in the field of natural language processing and may learn a correlation between words (or phrases) in a sentence. After the statement feature, the position feature and the word feature of the input text pass through the plurality of layers of transformer modules, a semantic feature for representing a meaning of the input text may be output.

It should be understood that the unified pre-trained language model may be used for the joint training of the semantic understanding task and the language generation task. For different language processing tasks, a calculation process for the input text X 1 , X 2 . . . X 5 is different, and the meaning of the output semantic features h 1 , h 2 . . . h 5 is also different. However, different language processing tasks may share a parameter of the unified pre-trained language model. That is, the training of different language processing tasks jointly affects the parameter of the unified pre-trained language model, and the trained unified pre-trained language model may be used for both the semantic understanding task and the language generation task.

By completely sharing a model parameter, the unified pre-trained language model may achieve the unified training of the semantic understanding task and the language generation task. However, as two different research modes, the semantic understanding and the language generation have a similarity and a difference. For the similarity, sharing the model parameter may complement each other, but for the difference, sharing the model parameter may have a negative effect.

Specifically, the unified pre-trained language model learns a plurality of types of tasks under a set of parameters. For the plurality of types of tasks, sharing the parameter at a model bottom for extracting a basic feature may not affect each other, but sharing the parameter when extracting a task-specific feature may have a negative effect.

In view of this, the embodiments of the present disclosure propose a method of training a natural language processing model and a method of processing a natural language.

As shown in FIG. 1 B , a system for the method of training the natural language processing model and/or the method of processing the natural language may include a unified training layer 110 , a connection layer 120 and an independent training layer 130 . The independent training layer 130 may include a plurality of task sub-layers, and each task sub-layer is used for a language processing branch task. For example, the independent training layer 130 may include a semantic understanding task sub-layer 131 and a language <figure-callout i

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application claims the priority of Chinese Patent Application No. 202110747046.X, filed on Jun. 30, 2021, the entire contents of which are hereby incorporated by reference.

TECHNICAL FIELD

BACKGROUND

Therefore, how to make a model have both the ability of semantic understanding and the ability of language generation has become a problem to be solved.

SUMMARY

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings are used to understand the solution better and do not constitute a limitation to the present disclosure. wherein:

FIG. 1 A shows a schematic structural diagram of a unified pre-trained language model according to an embodiment of the present disclosure.

FIG. 2 shows a flowchart of a method of training a natural language processing model according to an embodiment of the present disclosure.

FIG. 6 shows a flowchart of a method of processing a natural language according to an embodiment of the present disclosure.

FIG. 7 shows a block diagram of an apparatus of training a natural language processing model according to an embodiment of the present disclosure.

FIG. 8 shows a block diagram of an apparatus of processing a natural language according to an embodiment of the present disclosure.

FIG. 9 shows a block diagram of a method of training a natural language processing model and/or a method of processing a natural language according to an embodiment of the present disclosure.

DETAILED DESCRIPTION OF EMBODIMENTS

FIG. 1 A shows a schematic structural diagram of a unified pre-trained language model according to an embodiment of the present disclosure.

In view of this, the embodiments of the present disclosure propose a method of training a natural language processing model and a method of processing a natural language.

The unified training layer 110 may include a unified pre-trained language model for a unified training of a plurality of language processing branch tasks. The connection layer 120 is used to connect the unified training layer 110 and the independent training layer 130 , and may transmit, for different tasks, an output result from the unified training layer 110 to a corresponding task sub-layer in the independent training layer 130 .

Each task sub-layer in the independent training layer 130 may include a model for an independent training for different branch tasks (referred to as a branch task model). Each branch task model may include a plurality of transformer modules for analyzing and calculating the semantic feature so as to perform the language processing branch task.

According to the embodiments of the present disclosure, the independent training of each branch task is performed on the basis of the unified pre-trained language model, so as to achieve a purpose of performing a unified training for the similarity of multiple-tasks and performing an independent training for the difference of the multi-tasks, which may improve a processing effect of the natural language processing model on each language processing task.

It should be understood that the embodiments of the present disclosure propose a collaborative unified pre-trained framework including the unified training layer and the independent training layer, so that a plurality of types of tasks may jointly train the unified training layer to extract a basic feature information, and each type of task may train a specific task branch to extract a high-level semantic information required by the task. In this way, a model may have a good effect on the plurality of types of tasks.

FIG. 2 shows a flowchart of a method of training a natural language processing model according to an embodiment of the present disclosure.

As shown in FIG. 2 , a method 200 of training a natural language processing model may include operation S 210 to operation S 240 .

In operation S 210 , a semantic learning for multi-tasks is performed on an input text, so as to obtain a semantic feature for the multi-tasks.

For example, the multi-tasks may include a plurality of types of tasks, and each type of task may correspond to a branch. For example, a semantic understanding task, a language generation task, a common sense reasoning task and a knowledge graph task are all branch tasks of natural language processing. The natural language processing model may include a unified pre-trained language model (e.g., UNILM), and the plurality of branch tasks (i.e., a plurality of types of tasks) may be jointly trained through the unified pre-trained language model.

For example, the input text may contain at least one statement, and each statement may contain at least one word. Since different branch tasks have different task forms, a calculation process for the input text may be different, and then the semantic feature output by the unified pre-trained language model may have different meaning. The unified pre-trained language model may perform a statement vectorization representation of the input text, a position information vectorization representation of each word in the input text, and a vectorization representation of each word so as to obtain a statement feature, a position feature and a word feature, then learn a correlation between words (or phrases) according to the statement feature, the position feature and the word feature of the input text, and then output the semantic feature.

For example, for the semantic understanding task, the output semantic feature may represent a semantic understanding information of the input text. For another example, for the language generation task, the output semantic feature may represent a following-text information of the input text.

In operation S 220 , a feature learning for each branch task is performed based on the semantic feature, so as to obtain a first output result for each branch task.

For example, the natural language processing model may further include a plurality of branch task models, and each branch task model is used to perform an independent training for the corresponding branch task. For different branch tasks, the semantic feature output based on the unified pre-trained language model may be input into the corresponding branch task model, and the branch task model may be used for the independent training for the branch task, so as to achieve the purpose of performing a joint training for the similarity of the plurality of branch tasks and performing an independent training for the difference of the plurality of branch tasks.

It should be understood that since the unified pre-trained language model jointly trains the plurality of branch tasks, the plurality of branch tasks share the parameter of the unified pre-trained language model. That is, the training of the plurality of branch tasks jointly affect the parameter of the unified pre-trained language model. Therefore, although the semantic feature output by the unified pre-trained language model may be used to represent the meaning for different branch tasks, the semantic feature is also affected by different branch tasks.

In the embodiment of the present disclosure, the independent training for each branch task is performed separately on the basis of the semantic feature output by the unified pre-trained language model, so as to achieve the purpose of performing the joint training for the similarity of the plurality of branch tasks and performing the independent training for the difference of the plurality of branch tasks. In this way, a negative impact of different branch tasks on the parameter of the natural language processing model may be avoided, and a processing effect of the natural language processing model on each branch task may be improved.

In operation S 230 , a loss for each branch task is calculated according to the first output result for the branch task.

For example, each branch task model may perform the independent training for each branch task and output a first output result for the branch task. A loss for each branch task may be calculated according to a difference between a label of the input text for the branch task and the first output result.

For example, for the semantic understanding task, the input text may have a label indicating a semantic understanding information, and the first output result may be output after the unified training of the unified pre-trained language model and the independent training for the semantic understanding branch task. The first output result may indicate the semantic understanding information of the input text predicted by the natural language processing model. According to a difference between the semantic understanding information indicated by the label and the predicted semantic understanding information, a loss of the natural language processing model processing the semantic understanding task, that is, the loss for the semantic understanding task, may be calculated.

Similarly, a loss for the language generation task, a loss for the common sense reasoning task and a loss for the knowledge graph task may be obtained.

In operation S 240 , a parameter of the natural language processing model is adjusted according to the loss for each branch task.

For example, the parameter of the natural language processing model may be adjusted based on a sum of losses generated by the independent training of various branch tasks, such as the loss for the semantic understanding task, the loss for the language generation task, the loss for the common sense reasoning task and the loss for the knowledge graph task, so as to update the natural language processing model. Then, for a next input text, the process may return to the step of performing the semantic learning for multi-tasks on the input text by using the updated natural language processing model, until a preset condition is met. The preset condition may be a convergence of the sum of losses for the various branch tasks.

It should be understood that an adjustment of the parameter of the natural language processing model may include at least one of an adjustment of the parameter of the unified pre-trained language model and an adjustment of the parameter of the branch task model for the independent training of each branch task.

According to the embodiments of the present disclosure, the independent training of each branch task is performed on the basis of the unified pre-trained language model, so as to achieve the purpose of performing the joint training for the similarity of the plurality of branch tasks and performing the independent training for the difference of the plurality of branch tasks, which may improve a processing effect of the natural language processing model on each language processing task.

Different branch tasks performed based on the above-described natural language processing model will be described below with reference to FIG. 3 A to FIG. 3 C .

FIG. 3 A shows a schematic diagram of performing a semantic understanding task based on the above-described natural language processing model according to an embodiment of the present disclosure.

FIG. 3 B shows a schematic diagram of performing a word-level semantic generation task based on the above-described natural language processing model according to an embodiment of the present disclosure.

FIG. 3 C shows a schematic diagram of performing a sentence-level semantic generation task based on the above-described natural language processing model according to an embodiment of the present disclosure.

The above-described natural language processing model includes a unified pre-trained language model and a plurality of branch task models for performing various branch tasks. Each branch task model may include a plurality of transformer modules for analyzing and calculating based on the semantic feature so as to perform the language processing branch task.

As shown in FIG. 3 A to FIG. 3 C , S 1 and S 2 are two sentences (or statements) in the input text. S 1 contains words (or phrases) X 1 , X 2 and X 3 , and S 2 contains words (or phrases) X 4 and X 5 . Each box may represent a meaning of a word (or phrase). A white box indicates that the meaning of the word represented by the box may be known through the calculation of the transformer module, and a gray box indicates that the meaning of the word represented by the box may not be known through the calculation of the transformer module.

For FIG. 3 A , a branch task model for performing the semantic understanding may be a bidirectional MLM (Mask Language Model), a principle of which is to randomly mask a preset percentage (e.g., 15%) of words (or phrases) in the input text, encode and input the input text into the transformer module, and then obtain the information of all words (or phrases) in the input text through the calculation of the transformer module.

As shown in FIG. 3 A , for X 1 , the meaning of X 1 . . . X 5 may be obtained through the calculation of the transformer module. Similarly, for X 2 , the meaning of X 1 . . . X 5 may also be obtained through the calculation of the transformer module. In other words, for each word in the input text, the transformer module may output a preceding-text information and a following-text information for the word. That is, an output of the transformer module may contain the meaning of each word in the input text, which may be understood as the semantics of the entire input text, so as to achieve the semantic understanding of the input text.

For example, the branch task model for performing the semantic understanding may further perform a sentence-level semantic understanding task based on the preceding-text information and the following-text information for each word in the input text. For example, a logical distance between the statement S 1 and the statement S 2 may be calculated, and the logical distance indicates whether the sentences have a logical order or not. For another example, for the input text containing a plurality of statements, the logical order of the statements may be determined, so as to reorder the plurality of statements, and so on.

For FIG. 3 B , for example, a branch task model for performing a word-level semantic generation may be a unidirectional MLM (Unidirectional Mask Language Model). The unidirectional MLM may include a left-to-right language model (L2R LM) and a right-to-left language model (R2L LM).

FIG. 3 B may be a schematic diagram of the left-to-right language model, a principle of which is to randomly mask a preset percentage (e.g., 0Ë100%) of words (or phrases) in the input text, encode and input the input text into the transformer module, and then obtain an information of the word (or phrase) and an information of all words (or phrases) prior to the word (that is, the preceding-text information (or historical information)) through the calculation of the transformer module.

As shown in FIG. 3 B , for X 5 , the meaning of X 1 . . . X 5 may be obtained through the calculation of the transformer module. Similarly, for X 4 , the meaning of X 1 . . . X 4 may be obtained through the calculation of the transformer module. In other words, for each word in the input text, the transformer module may output the word and the preceding-text information for the word. That is, the output of the transformer module contains all the preceding-text information. The following-text information may be predicted based on the preceding-text information, so as to perform the unidirectional-MLM-based word-level language generation task.

Similarly, a principle of the right-to-left language model is to randomly mask a preset percentage (e.g., 0Ë100%) of words (or phrases) in the input text, encode and input the input text into the transformer module, and then obtain the information of the word (or phrase) and the information of all words (or phrases) subsequent to the word (that is, the following-text information) through the calculation of the transformer module. The preceding-text information may be predicted based on the following-text information, so as to perform the unidirectional-MLM-based word-level language generation task.

For FIG. 3 C , for example, a branch task model for performing a sentence-level semantic generation may be a Seq2Seq language model (Sequence to Sequence Language Model, S2S LM), a principle of which is to randomly mask a preset percentage (e.g., 0Ë100%) of words (or phrases) in a second sentence of the input text, encode and input the input text into the transformer module, so that through the calculation of the transformer module, each word (or phrase) in a first sentence may obtain the meaning of all other words (or phrases) in the first sentence but may not obtain the meaning of words (or phrases) in the second sentence, while the masked word in the second sentence may obtain the meaning of all words (or phrases) in the first sentence and the preceding-text information prior to the word in the second sentence. That is, the output of the transformer module contains the meaning of all words (or phrases) in the first sentence and contains the preceding-text information for the masked word (or phrase) in the second sentence. Therefore, the Seq2Seq language model may be used to predict a sentence-level following-text information and perform a sentence-level language generation task.

As shown in FIG. 3 C , for each word in the first sentence S 1 , the meaning of all words X 1 , X 2 and X 3 may be obtained through the calculation of the transformer module. For X 4 in the second sentence S 2 , the meaning of X 1 , X 2 , X 3 and X 4 may be obtained through the calculation of the transformer module. For X 5 in the second sentence S 2 , the meaning of X 1 , X 2 , X 3 , X 4 and X 5 may be obtained through the calculation of the transformer module. That is, the output of the transformer module contains the meaning of the first sentence and the preceding-text information for all masked words in the second sentence, so that the next sentence may be predicted and the sentence-level language generation task may be performed.

It should be noted that the bidirectional MLM task is characterized in that 15% of words (tokens) is involved in each training, while the unidirectional MLM task and the S2S LM task are characterized in that 100% of tokens may be predicted. The unified pre-trained language model may unify three language processing models including the bidirectional MLM, the unidirectional MLM and the Seq2Seq LM. However, in order to adapt to different tasks, a training form of the task needs to be adjusted. For example, only 15% of tokens may be involved in each training for the unidirectional MLM task and the S2S LM task, so that 85% loss may be generated compared with the training for an original MLM task and S2S LM task. Therefore, a model convergence may be slow, which may result in a waste of resources and time.

The embodiments of the present disclosure may be implemented to perform independent training for each branch task on the basis of the unified pre-trained language model, that is, each branch task may retain a task characteristic without making an adaptation adjustment during the training. For example, the unidirectional MLM task and the S2S LM task are characterized in that 100% of tokens may be predicted, so that an error may be avoided and a model training efficiency may be improved.

It should be noted that the semantic generation task performed based on the above-described natural language processing model may be used to achieve a prediction of one or more words in the preceding-text information or the following-text information.

FIG. 4 A shows a schematic diagram of performing a semantic generation task based on the above-described natural language processing model according to an embodiment of the present disclosure.

FIG. 4 B shows a schematic diagram of performing a semantic generation task based on the above-described natural language processing model according to another embodiment of the present disclosure.

The natural language processing model includes a unified pre-trained language model and a plurality of branch task models for performing various branch tasks. The branch task model for performing the semantic generation may be a left-to-right language model, which may learns each word (or phrase) of the input text, so that each word (or phrase) may only obtain the word (or phrase) and the preceding-text information for the word (or phrase), and the following-text information may be predicted based on the preceding-text information, so as to perform the language generation task.

The left-to-right language model may further include a left-to-right N-Gram language model (N-GLM). A traditional left-to-right language model may predict a word (or phrase) in the following-text information based on the preceding-text information, and the left-to-right N-Gram language model may simultaneously predict a plurality of words (or phrases) in the following-text information based on the preceding-text information.

FIG. 4 A shows a schematic diagram of predicting a word (or phrase) in the following-text information. For example, the input text may be â

(Chinese characters indicating Harbin is the capital of Heilongjiang province)â. For each word in the input text, a word subsequent to the word may be predicted through a semantic learning of a transformer module 410 of the left-to-right language model (such as GPT-3). For example, â

â may be predicted for â

â, â

â may be predicted for â

â, and so on.

FIG. 4 B shows a schematic diagram of simultaneously predicting a plurality of words (or phrases) in the following-text information. The input text may be â

â. For each word in the input text, a plurality of words subsequent to the word may be predicted through a semantic learning of a transformer module 420 of the left-to-right N-Gram language model (such as ERNIE 3.0). For example, â

â may be predicted for â

â, â

â may be predicted for â

â, and so on.

According to the embodiments of the present disclosure, the training for the language generation task may be performed based on the unified pre-trained language model, and one or more words in the following-text information may be output for each word in the input text, so as to perform the language generation task.

As shown in FIG. 5 , a system for the method of training the natural language processing model and/or the method of processing the natural language may include a unified training layer 510 , a connection layer 520 , an independent training layer 530 and a joint loss calculation layer 540 .

The independent training layer 530 may include a plurality of task sub-layers, and each task sub-layer is used for a language processing branch task. For example, the independent training layer 530 may include a semantic understanding task sub-layer 531 , a language generation task sub-layer 532 and a common sense reasoning task sub-layer 533 . It should be noted that the independent training layer 530 may further expand more branch tasks, such as a knowledge graph task sub-layer, according to actual needs.

On the one hand, for the input text, after the unified training of the unified training layer 510 , the output semantic feature may be transmitted to a corresponding task sub-layer in the independent training layer 530 through the connection layer 520 . For example, for the input text for the semantic understanding task, a semantic feature for the semantic understanding task may be output by the unified training layer 510 , and the connection layer 520 may input the semantic feature into the semantic understanding task sub-layer 531 to perform an independent training for the semantic understanding task. An output result for the independent training of each task sub-layer is a first output result 501 .

On the other hand, for the input text, after the unified training of the unified training layer 510 , the obtained semantic feature may be used as a second output result 502 . For different branch tasks, the second output result 502 may represent different meanings.

The joint loss calculation layer 540 may calculate a joint loss based on the first output result 501 and the second output result 502 . Then, the loss for the independent training and the loss for the unified training may be integrated as an overall loss of the natural language processing model, which may be used for the adjustment of the parameter of the natural language processing model. In this way, the natural language processing model may converge faster, and the training efficiency of the natural language processing model may be improved.

FIG. 6 shows a flowchart of a method of processing a natural language according to an embodiment of the present disclosure.

As shown in FIG. 6 , a method 600 of processing a natural language includes operation S 610 to operation S 620 .

In operation S 610 , a to-be-processed text for a preset branch task is acquired.

In operation S 620 , a semantic learning for multi-tasks is performed on the to-be-processed text by using a natural language processing model, so as to obtain a semantic feature for the multi-tasks, and a feature learning for the preset branch task is performed according to the semantic feature, so as to obtain a proces

CLAIMS

Claims ( 16 )

What is claimed is:

1. A method of training a natural language processing model, comprising:

performing a semantic learning for multi-tasks on an input text, so as to obtain a semantic feature for the multi-tasks, wherein the multi-tasks comprise a plurality of branch tasks;

performing a feature learning for each branch task based on the semantic feature, so as to obtain a first output result for each branch task;

calculating a loss for each branch task according to the first output result for the branch task;

adjusting a parameter of the natural language processing model according to the loss for each branch task; and

determining a second output result for each branch task based on the semantic feature, wherein

the multi-tasks comprise a first branch task for a semantic understanding; and

the determining a second output result for each branch task based on the semantic feature comprises one of:

determining a semantic understanding information for the input text as the second output result for the first branch task based on the semantic feature;

calculating a logical distance between a plurality of statements in the input text as the second output result for the first branch task based on the semantic feature; and

determining a logical order of a plurality of statements in the input text as the second output result for the first branch task based on the semantic feature.

2. The method of claim 1 , wherein the performing a feature learning for each branch task based on the semantic feature, so as to obtain a first output result for each branch task comprises:

determining, for each word in the input text, a preceding-text information prior to the word and a following-text information subsequent to the word based on the semantic feature; and

determining a semantic understanding information for the input text as a first output result for the first branch task according to the preceding-text information and the following-text information.

3. The method of claim 2 , further comprising:

calculating a logical distance between a plurality of statements in the input text as the first output result for the first branch task according to the preceding-text information and the following-text information.

4. The method of claim 2 , further comprising:

determining a logical order of a plurality of statements in the input text as the first output result for the first branch task according to the preceding-text information and the following-text information.

5. The method of claim 1 , wherein the multi-tasks further comprise a second branch task for a language generation; and the performing a feature learning for each branch task based on the semantic feature, so as to obtain a first output result for each branch task comprises: for each word in the input text,

determining a preceding-text information prior to the word based on the semantic feature; and

predicting a following-text information subsequent to the word as a first output result for the second branch task based on the preceding-text information.

6. The method of claim 1 , wherein the calculating a loss for each branch task according to the first output result for the branch task comprises:

calculating the loss for the branch task according to the first output result for the branch task and the second output result for the branch task.

7. The method of claim 1 , wherein the multi-tasks further comprise a second branch task for a language generation; and the determining a second output result for each branch task based on the semantic feature further comprises:

predicting, for each word in the input text, a following-text information subsequent to the word as the second output result for the second branch task based on the semantic feature.

8. A method of processing a natural language, comprising:

acquiring a to-be-processed text for a preset branch task; and

performing a semantic learning for multi-tasks on the to-be-processed text by using a natural language processing model, so as to obtain a semantic feature for the multi-tasks, and performing a feature learning for the preset branch task according to the semantic feature, so as to obtain a processing result for the preset branch task,

wherein the multi-tasks comprise the preset branch task, and the natural language processing model is trained using operations of training a natural language processing model, comprising: