Google Patentsopen access

Method and system for training document-level natural language processing models — Naver Corporation (US11625544B2)

ABSTRACT

Abstract

In methods for training a natural language generation (NLG) model using a processor a document-level machine translation (MT) model is provided by training an MT model to receive as input, token sequences in a first language, and to generate as output, token sequences in a second language. An augmented document-level MT model is provided by training the document-level MT model to receive as input, paired language-independent structured data and token sequences in the first language, and to generate as output, token sequences in the second language. The augmented document-level MT model is trained to receive as input, language-independent structured data, and to generate as output, token sequences in the second language.

Description

PRIORITY CLAIM AND REFERENCE TO RELATED APPLICATION

This application claims priority to U.S. Provisional Patent Application Ser. No. 63/065,138, filed Aug. 13, 2020, which application is incorporated herein by reference in its entirety.

FIELD

The present disclosure relates to processor-based natural language methods and systems for machine-based natural language generation using neural models, and more particularly methods and systems for training natural language generation (NLG) models using machine translation (MT) models.

BACKGROUND

Recent advances in processor-based Natural Language Processing (NLP) include neural models for machine translation (MT) and Natural Language Generation (NLG). Such neural models have led to significant improvements in both MT and NLG tasks. However, existing neural NLG and MT models are believed to be insufficient or sub-optimal for performing certain NLP tasks.

For NLG models, for instance, the generation of text such as for descriptive summaries conditioned on structured data (or data-to-text) remains an open challenge. Existing NLG models lack accuracy, coherence, and/or adequacy to the source material.

Two significant aspects to address in data-to-text generation techniques are 1) identifying the most important information from input data, and 2) verbalizing data as a coherent document. These two challenges have been addressed separately as different modules in pipeline systems, or in an end-to-end manner with neural generation models.

However, although known end-to-end NLG generation models can generate fluent text, such models perform weakly in terms of best content selection. Recently, Puduppully et al. (2019) disclosed a method for training an end-to-end data-to-document generation model on the Rotowire dataset (https://github.com/harvardnlp/boxscore-data; English summaries of basketball games with structured information). This disclosure aimed to overcome the shortcomings of end-to-end neural NLG models by explicitly modeling content selection and planning in its architecture.

Additionally, most existing MT models translate sentences independently (that is, at sentence-level), without access to larger context that may be provided, for instance, from other sentences from the same document or from structured information. Although existing MT systems have reported impressive performance on several languages, there are still many challenges in the art especially when factors beyond sentence-level text are considered.

SUMMARY

Example systems and methods provided in present embodiments can leverage data from both MT and NLG tasks. Particular example systems and methods can exploit transfer learning between MT and NLG.

According to one aspect of the disclosed embodiments, methods are provided for training a natural language generation (NLG) model using a processor. A document-level machine translation (MT) model is provided by training an MT model to receive as input, token sequences in a first language, and to generate as output, token sequences in a second language. An augmented document-level MT model is provided by training the document-level MT model to receive as input, paired language-independent structured data and token sequences in the first language, and to generate as output, token sequences in the second language. The language-independent structured data is understood in the first language and the second language. The NLG model is provided by training the augmented document-level MT model to receive as input, language-independent structured data (e.g., without the paired token sequences in the first language), and to generate as output, token sequences in the second language. Methods are also provided for generating an output text from new structured data using trained NLG models.

According to a complementary aspect, the present disclosure provides a computer program product, comprising code instructions to execute a method according to the previously described aspects; and a computer-readable medium, on which is stored a computer program product comprising code instructions to execute a method according to the previously described aspects.

Other features and advantages of the invention will be apparent from the following specification taken in conjunction with the following drawings.

DESCRIPTION OF THE DRAWINGS

The present disclosure will become more fully understood from the detailed description and the accompanying drawings, wherein:

FIG. 1 illustrates an example of a system architecture in which example methods according to the present disclosure may be performed;

FIG. 2 illustrates an example architecture of the processor in FIG. 1 ;

FIG. 3 sets forth an example method for training a natural language generation (NLG) model according to the present disclosure;

FIG. 4 sets forth an example method for providing a trained document-level machine translation model;

FIG. 5 sets forth an example method for augmenting a document-level machine translation model;

FIG. 6 sets forth an example method for pre-processing structured data;

FIG. 7 sets forth an example method for training an augmented document-level machine translation model to provide a natural language generation model (NLG model);

FIG. 8 sets forth an example runtime (inference) method for an NLG model trained according to the method set forth in FIG. 3 ;

FIGS. 9 A- 9 C show data from an experimental NLG training method, including: example metadata encoding used in an NLG training method (Metadata) that is shown in FIG. 9 A ; a story from a training set (Reference Story) that is shown in FIG. 9 B ; and an output of the trained English NLG model (Generated Story) that is shown in FIG. 9 C ,

FIG. 10 shows DGT-valid BLEU (by the best checkpoint) depending on the maximum number of selected players for an experimental English NLG track; and

FIG. 11 shows correctly predicted information from an experimental NLG model that was not explicitly in provided metadata (+) or hallucinations (−), where “REF” signifies “reference” text and “NLG” signifies “natural language generated” text.

In the drawings, reference numbers may be reused to identify similar and/or identical elements.

DETAILED DESCRIPTION

Introduction

Embodiments herein provide, among other things, methods, systems, and program products for training document-level natural language generation (NLG) models by training a different type of neural NLP model, namely a machine-translation (MT) model. A MT model that is document-level can share an output (document-level text in a selected language) with the NLG model. This allows the use of transfer learning between the MT models and the NLG models to train the NLG, and thus allows the use of training data that otherwise may not be available or suitable for training NLGs. This is particularly useful in the typical case where available NLG model training data is limited relative to MT model training data.

Example methods herein exploit various features of MT and NLG models to use datasets from both MT and NLG models. Methods for training MT models, and methods for using such trained models (e.g., for inference) are also provided.

A “model” as used herein can be defined by an algorithm executable by a processor and memory based on one or more neural network layers and a corresponding set of model parameters (e.g., weights). Example natural language processing (NLP) models including machine translation (MT) models and natural language generation (NLG) models, may be configured for processing one or more inputs, such as input sequences, and for generating one or more outputs, such as output sequences. Models can be trained by determining the parameters using training datasets having source data (e.g., source sequences in a first language) and target data (e.g., target sequences in a second language). Available datasets are known for training certain types of models. However, in example methods disclosed herein, datasets typically used to train certain types of NLP models can be adapted for training other types of NLP models.

As used herein, a “machine translation model” or “MT model” is a neural network-based natural language processing (NLP) model executable by a processor and memory that is configured to receive input tokens in a first language and after processing by the model generate (either directly or after further processing) output tokens in a second language. Tokens can include, for instance, words, characters (e.g., letters, numbers, punctuation, special characters (such as an end-of-sentence character), or others, as will be appreciated by an artisan. Both the input tokens and the output tokens can be provided by sequences of tokens (i.e., token sequences). “Language” with reference to the first language and the second language is intended to be generally construed as any suitable natural language. The first and second languages can be any two selected natural languages (including two variants of the same, more general language) so long as they are different from one another in some respect.

Some MT models can generally be considered “sentence-level,” in which the sequences of input tokens are processed by the MT model, and the output tokens generated, on a per-sentence basis (the end of sentences can be indicated, for instance, by specialized end-of-sentence tokens) to provide one or more input sentences, and the sequences of output tokens are accordingly provided as one or more corresponding output sentences. Other MT models can generally be considered “document-level,” in which the sequences of input tokens are processed, and the output tokens generated, on a larger basis than that of individual sentences, up to and including a complete document (e.g., a complete set of tokens, or set of tokens representing a sequence greater than individual sentences).

A “natural language generation model” or “NLG model” is a neural network-based NLP model executed by a processor and memory that is configured to receive as input, structured data, and to generate as output, a sequence of output tokens in a selected language. Structured data, which is language-independent, can include, for instance, metadata, data in table form, data in tree form, data in graph form, formatted data, etc. The output sequence of tokens (i.e., token sequences) can be, for instance, one or more sentences of generated text, and can be document-level; for instance, the output sequence of tokens can be a document-level text sequence based on the input structured data.

In some example embodiments herein, a document-level machine translation (MT) model is provided, e.g., generated, that is trained for receiving input token sequences in a first language and generating output token sequences in a second language. Such document-level MT models may be initially acquired, or they may be provided by acquiring and then training sentence-level MT models, e.g., using parallel document-level corpus data in the first language and the second language. Example methods for providing document-level machine translation models are explained in more detail herein.

Additional example methods train a document-level MT model for receiving input tokens in a first language and generating output tokens in a second language. An MT model may be trained using a dataset that includes token sequences in the first language and the second language that are concatenated groups of sentences randomly selected from a sentence-level parallel corpus. The document-level MT model may be alternatively or additionally trained using one or more datasets that include document-level source data in the first language and document-level token data in the second language. Example methods for providing the dataset(s) are provided herein. Training the document-level MT model using concatenated groups of sentences increases the available amount of training data and can exploit information beyond sentence-level information when there exists little or no parallel document-level corpus data.

After generating a document-level MT model, an augmented document-level MT model can be provided by training (e.g., MT training) a document-level MT model to receive as input, language-independent structured data paired with token sequences in a first language, and to generate as output, target token sequences in a second language. Such data can be provided at least in part, for instance, by one or more training datasets conventionally used for NLG models. The language-independent structured data may be pre-processed, e.g., using example methods as provided herein, to generate one or more text sequences to be consumed by the augmented document-level MT model during training (and, for new structured data, during runtime). In some example methods, pre-processing may further include generating language-agnostic structured data from language-dependent structured data (as a nonlimiting example, written dates).

An NLG model can be provided by training the augmented document-level MT model to receive as input, language-independent structured data, and to generate as output, token sequences in the second language. This can be performed, for instance, by NLG training of the augmented document-level MT model using text sequences generated from language-independent structured data (e.g., without further including paired token sequences in the first language as input).

In an example method of generating output text from structured data (e.g., during runtime or inference), the structured data is input into the trained NLG model. The structured data may be pre-processed to generate one or more text sequences. In response to this input, the trained NLG model generates (e.g., document-level) output text in the second language.

While this invention is susceptible of embodiments in many different forms, there is shown in the drawings and will herein be described in detail preferred embodiments of the invention with the understanding that the present disclosure is to be considered as an exemplification of the principles of the invention and is not intended to limit the broad aspects of the invention to the embodiments illustrated.

References

The following documents are incorporated by reference herein in their entirety, without an admission that any of these documents constitute prior art:

Alexandre Berard, Calapodescu Iona, and Claude Roux, 2019. NAVER LABS Europe Systems for the WMT19 Machine Translation Robustness Task. In WMT-Shared Task Paper. Sergey Edunov, Myle Ott, Michael Auli, and David Grangier, 2018. Understanding Back-Translation at Scale. In EMNLP. Marcin Junczys-Dowmunt. 2019. Microsoft Translator at WMT 2019: Towards Large-Scale Document-Level Neural Machine Translation. In WMT-Shared Task Paper. Taku Kudo and John Richardson, 2018. Sentence Piece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. In EMNLP. <li id="ul00

PRIORITY CLAIM AND REFERENCE TO RELATED APPLICATION

This application claims priority to U.S. Provisional Patent Application Ser. No. 63/065,138, filed Aug. 13, 2020, which application is incorporated herein by reference in its entirety.

FIELD

The present disclosure relates to processor-based natural language methods and systems for machine-based natural language generation using neural models, and more particularly methods and systems for training natural language generation (NLG) models using machine translation (MT) models.

BACKGROUND

Recent advances in processor-based Natural Language Processing (NLP) include neural models for machine translation (MT) and Natural Language Generation (NLG). Such neural models have led to significant improvements in both MT and NLG tasks. However, existing neural NLG and MT models are believed to be insufficient or sub-optimal for performing certain NLP tasks.

For NLG models, for instance, the generation of text such as for descriptive summaries conditioned on structured data (or data-to-text) remains an open challenge. Existing NLG models lack accuracy, coherence, and/or adequacy to the source material.

Two significant aspects to address in data-to-text generation techniques are 1) identifying the most important information from input data, and 2) verbalizing data as a coherent document. These two challenges have been addressed separately as different modules in pipeline systems, or in an end-to-end manner with neural generation models.

However, although known end-to-end NLG generation models can generate fluent text, such models perform weakly in terms of best content selection. Recently, Puduppully et al. (2019) disclosed a method for training an end-to-end data-to-document generation model on the Rotowire dataset (https://github.com/harvardnlp/boxscore-data; English summaries of basketball games with structured information). This disclosure aimed to overcome the shortcomings of end-to-end neural NLG models by explicitly modeling content selection and planning in its architecture.

Additionally, most existing MT models translate sentences independently (that is, at sentence-level), without access to larger context that may be provided, for instance, from other sentences from the same document or from structured information. Although existing MT systems have reported impressive performance on several languages, there are still many challenges in the art especially when factors beyond sentence-level text are considered.

SUMMARY

Example systems and methods provided in present embodiments can leverage data from both MT and NLG tasks. Particular example systems and methods can exploit transfer learning between MT and NLG.

According to one aspect of the disclosed embodiments, methods are provided for training a natural language generation (NLG) model using a processor. A document-level machine translation (MT) model is provided by training an MT model to receive as input, token sequences in a first language, and to generate as output, token sequences in a second language. An augmented document-level MT model is provided by training the document-level MT model to receive as input, paired language-independent structured data and token sequences in the first language, and to generate as output, token sequences in the second language. The language-independent structured data is understood in the first language and the second language. The NLG model is provided by training the augmented document-level MT model to receive as input, language-independent structured data (e.g., without the paired token sequences in the first language), and to generate as output, token sequences in the second language. Methods are also provided for generating an output text from new structured data using trained NLG models.

According to a complementary aspect, the present disclosure provides a computer program product, comprising code instructions to execute a method according to the previously described aspects; and a computer-readable medium, on which is stored a computer program product comprising code instructions to execute a method according to the previously described aspects.

Other features and advantages of the invention will be apparent from the following specification taken in conjunction with the following drawings.

DESCRIPTION OF THE DRAWINGS

The present disclosure will become more fully understood from the detailed description and the accompanying drawings, wherein:

FIG. 1 illustrates an example of a system architecture in which example methods according to the present disclosure may be performed;

FIG. 2 illustrates an example architecture of the processor in FIG. 1 ;

FIG. 3 sets forth an example method for training a natural language generation (NLG) model according to the present disclosure;

FIG. 4 sets forth an example method for providing a trained document-level machine translation model;

FIG. 5 sets forth an example method for augmenting a document-level machine translation model;

FIG. 6 sets forth an example method for pre-processing structured data;

FIG. 7 sets forth an example method for training an augmented document-level machine translation model to provide a natural language generation model (NLG model);

FIG. 8 sets forth an example runtime (inference) method for an NLG model trained according to the method set forth in FIG. 3 ;

FIGS. 9 A- 9 C show data from an experimental NLG training method, including: example metadata encoding used in an NLG training method (Metadata) that is shown in FIG. 9 A ; a story from a training set (Reference Story) that is shown in FIG. 9 B ; and an output of the trained English NLG model (Generated Story) that is shown in FIG. 9 C ,

FIG. 10 shows DGT-valid BLEU (by the best checkpoint) depending on the maximum number of selected players for an experimental English NLG track; and

FIG. 11 shows correctly predicted information from an experimental NLG model that was not explicitly in provided metadata (+) or hallucinations (−), where “REF” signifies “reference” text and “NLG” signifies “natural language generated” text.

In the drawings, reference numbers may be reused to identify similar and/or identical elements.

DETAILED DESCRIPTION

Introduction

Embodiments herein provide, among other things, methods, systems, and program products for training document-level natural language generation (NLG) models by training a different type of neural NLP model, namely a machine-translation (MT) model. A MT model that is document-level can share an output (document-level text in a selected language) with the NLG model. This allows the use of transfer learning between the MT models and the NLG models to train the NLG, and thus allows the use of training data that otherwise may not be available or suitable for training NLGs. This is particularly useful in the typical case where available NLG model training data is limited relative to MT model training data.

Example methods herein exploit various features of MT and NLG models to use datasets from both MT and NLG models. Methods for training MT models, and methods for using such trained models (e.g., for inference) are also provided.

A “model” as used herein can be defined by an algorithm executable by a processor and memory based on one or more neural network layers and a corresponding set of model parameters (e.g., weights). Example natural language processing (NLP) models including machine translation (MT) models and natural language generation (NLG) models, may be configured for processing one or more inputs, such as input sequences, and for generating one or more outputs, such as output sequences. Models can be trained by determining the parameters using training datasets having source data (e.g., source sequences in a first language) and target data (e.g., target sequences in a second language). Available datasets are known for training certain types of models. However, in example methods disclosed herein, datasets typically used to train certain types of NLP models can be adapted for training other types of NLP models.

As used herein, a “machine translation model” or “MT model” is a neural network-based natural language processing (NLP) model executable by a processor and memory that is configured to receive input tokens in a first language and after processing by the model generate (either directly or after further processing) output tokens in a second language. Tokens can include, for instance, words, characters (e.g., letters, numbers, punctuation, special characters (such as an end-of-sentence character), or others, as will be appreciated by an artisan. Both the input tokens and the output tokens can be provided by sequences of tokens (i.e., token sequences). “Language” with reference to the first language and the second language is intended to be generally construed as any suitable natural language. The first and second languages can be any two selected natural languages (including two variants of the same, more general language) so long as they are different from one another in some respect.

Some MT models can generally be considered “sentence-level,” in which the sequences of input tokens are processed by the MT model, and the output tokens generated, on a per-sentence basis (the end of sentences can be indicated, for instance, by specialized end-of-sentence tokens) to provide one or more input sentences, and the sequences of output tokens are accordingly provided as one or more corresponding output sentences. Other MT models can generally be considered “document-level,” in which the sequences of input tokens are processed, and the output tokens generated, on a larger basis than that of individual sentences, up to and including a complete document (e.g., a complete set of tokens, or set of tokens representing a sequence greater than individual sentences).

A “natural language generation model” or “NLG model” is a neural network-based NLP model executed by a processor and memory that is configured to receive as input, structured data, and to generate as output, a sequence of output tokens in a selected language. Structured data, which is language-independent, can include, for instance, metadata, data in table form, data in tree form, data in graph form, formatted data, etc. The output sequence of tokens (i.e., token sequences) can be, for instance, one or more sentences of generated text, and can be document-level; for instance, the output sequence of tokens can be a document-level text sequence based on the input structured data.

In some example embodiments herein, a document-level machine translation (MT) model is provided, e.g., generated, that is trained for receiving input token sequences in a first language and generating output token sequences in a second language. Such document-level MT models may be initially acquired, or they may be provided by acquiring and then training sentence-level MT models, e.g., using parallel document-level corpus data in the first language and the second language. Example methods for providing document-level machine translation models are explained in more detail herein.

Additional example methods train a document-level MT model for receiving input tokens in a first language and generating output tokens in a second language. An MT model may be trained using a dataset that includes token sequences in the first language and the second language that are concatenated groups of sentences randomly selected from a sentence-level parallel corpus. The document-level MT model may be alternatively or additionally trained using one or more datasets that include document-level source data in the first language and document-level token data in the second language. Example methods for providing the dataset(s) are provided herein. Training the document-level MT model using concatenated groups of sentences increases the available amount of training data and can exploit information beyond sentence-level information when there exists little or no parallel document-level corpus data.

After generating a document-level MT model, an augmented document-level MT model can be provided by training (e.g., MT training) a document-level MT model to receive as input, language-independent structured data paired with token sequences in a first language, and to generate as output, target token sequences in a second language. Such data can be provided at least in part, for instance, by one or more training datasets conventionally used for NLG models. The language-independent structured data may be pre-processed, e.g., using example methods as provided herein, to generate one or more text sequences to be consumed by the augmented document-level MT model during training (and, for new structured data, during runtime). In some example methods, pre-processing may further include generating language-agnostic structured data from language-dependent structured data (as a nonlimiting example, written dates).

An NLG model can be provided by training the augmented document-level MT model to receive as input, language-independent structured data, and to generate as output, token sequences in the second language. This can be performed, for instance, by NLG training of the augmented document-level MT model using text sequences generated from language-independent structured data (e.g., without further including paired token sequences in the first language as input).

In an example method of generating output text from structured data (e.g., during runtime or inference), the structured data is input into the trained NLG model. The structured data may be pre-processed to generate one or more text sequences. In response to this input, the trained NLG model generates (e.g., document-level) output text in the second language.

While this invention is susceptible of embodiments in many different forms, there is shown in the drawings and will herein be described in detail preferred embodiments of the invention with the understanding that the present disclosure is to be considered as an exemplification of the principles of the invention and is not intended to limit the broad aspects of the invention to the embodiments illustrated.

References

The following documents are incorporated by reference herein in their entirety, without an admission that any of these documents constitute prior art:

Alexandre Berard, Calapodescu Iona, and Claude Roux, 2019. NAVER LABS Europe Systems for the WMT19 Machine Translation Robustness Task. In WMT-Shared Task Paper. Sergey Edunov, Myle Ott, Michael Auli, and David Grangier, 2018. Understanding Back-Translation at Scale. In EMNLP. Marcin Junczys-Dowmunt. 2019. Microsoft Translator at WMT 2019: Towards Large-Scale Document-Level Neural Machine Translation. In WMT-Shared Task Paper. Taku Kudo and John Richardson, 2018. Sentence Piece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. In EMNLP. Rémi Lebret, David Grangier, and Michael Auli, 2016. Neural Text Generation from Structured Data with Application to the Biography Domain. In EMNLP. Marco Lui and Timothy Baldwin, 2012. Langid.py: An off-the-shelf language identification tool. In proceedings of the ACL 2012 System Demonstrations, ACL. Kathleen R. McKeown, 1985. Text Generation: Using Discourse Strategies and Focus Constraints to Generate Natural Language Text. Cambridge University Press, New York, N.Y., USA. Hongyuan Mei, Mohit Bansal, and Matthew R Walter, 2016. What to talk about and how? Selective generation using LSTMs with Coarse-to-Fine Alignment. In NAACL-HLT. Myle Ott, Sergey Edunov, David Grangier, and Michael Auli, 2018. Scaling Neural Machine Translation. In WMT. Ratish Puduppully, Li Dong, and Mirella Lapata, 2019. Data-to-Text Generation with Content Selection and Planning. In Proceedings of the AAAI Conference on Artificial Intelligence. Ehud Reiter and Robert Dale, 2000. Building Natural Language Generation Systems. Cambridge University Press.

Rico Sennrich, Barry Haddow, and Alexandra Birch, 2016. Neural Machine Translation of Rare Words with Subword Units. In ACL.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, ukasz Kaiser, and Illia Polosukhin, 2017. Attention is All You Need. In NIPS. Sam Wiseman, Stuart Shieber, and Alexander Rush, 2017. Challenges in Data-to-Document Generation. In EMNLP.

System Architecture

Referring now to the figures, example methods disclosed hereunder may be implemented within a system 100 architected as illustrated in FIG. 1 . The system 100 includes a processor 102 , e.g., a computing device, that is configured to perform example natural language processing (NLP) and NLP model training tasks using one or more neural networks. The processor 102 can communicate with one or more databases 104 that store datasets used for training example models. It will be appreciated that the processor 102 can include either a single processor or multiple processors operating in series or in parallel, and that the database 104 can include one or more databases.

During an operation of NLP tasks, such as training, validation, testing, and/or inference of the machine translation (MT) or natural language generation (NLG) models, the processor 102 can receive input data from another, connected processor (not shown), from the databases 104 , and/or from one or more user terminals 106 connected via a network 108 , or any combination. The processor 102 can process the input data using the model, and then output results of such processing to the additional processor(s), the databases 104 , and/or the one or more user terminals

106 a , 106 b . In some example methods, the processor 102 can be configured as a server (or cloud computing device) and one or more of the additional processors or the one or more user terminals 106 can be configured as clients. The databases 104 may be local to the processor, or connected remotely, e.g., via the network 108 .

User terminals

106 a , 106 b include, but are not limited to, personal computers 106 a , client computers, client terminals, mobile communication devices 106 b , etc., or any other computing device that can be configured for sending and receiving data to the processor 102 according to methods herein. The user terminals 106 may include a display for displaying results of processing by the processor 102 according to example methods.

FIG. 2 shows components of an example processor 200 , such as may be embodied in the processor 102 . The processor 200 includes a processing unit 202 and a memory 204 , which can include any combination of random-access memory, non-volatile memory, and storage media. A database 206 , such as the database 104 , may be provided in communication with the processing unit 202 . NLP model configuration data (e.g., models, parameters), datasets (e.g., for training, testing, and/or validation), generated input data, generated output data, or other data can be stored in and retrieved from any combination of the memory 204 and the database 206 as needed.

The processing unit 202 , executing code stored in the memory 204 , provides modules for performing steps of example methods herein. Example operations of such modules are explained in further detail below with reference to example methods.

A natural language processing (NLP) module 208 executes a neural network based NLP model, such as a sentence-level machine translation (MT) model, a document-level MT model, an augmented document-level MT model, or a natural language generation (NLG) model, according to the particular method used. The NLP module 208 receives input data generated from one or more datasets 210 (e.g., stored in the memory 204 ), and processes the input data using the NLP model to generate output data.

An input data processing module 214 receives, processes, and/or generates input data from the datasets 210 to provide to the NLP module 208 for use in training, testing, validation, and/or inference. The input data processing module 214 can include one or more data processing modules such as a token sequence generation module 216 , a structured data pre-processing module 218 , and a concatenation/ aggregation module 220 .

The token sequence generation module 216 provides one or more source and/or target token sequences, e.g., text sequences, from the datasets 210 or other sources. For a dataset provided from a parallel corpus, for instance, the token sequence generation module 216 can provide sentence-level text sequences or document-level text sequences as needed for training, including any pre-processing desired for such text sequences as will be appreciated by an artisan. The token sequence generation module 216 may further provide, e.g., by selection at random or other selection methods, a set of text sequences, e.g., sentence-level text sequences, that may then be concatenated by the concatenation/ aggregation module 220 to provide document-level text During runtime, the token sequence generation module may perform any desired pre-processing of newly received token (e.g., text) sequence for input to the NLP model, such as for MT tasks.

The structured data pre-processing module 218 performs one or more pre-processing steps for received structured data from the datasets 210 or other sources, to provide a sequence of language-agnostic data, e.g., a text or token sequence. Pre-processing may include, as nonlimiting examples, generating language-agnostic data from language-dependent structured data, filtering structured data, supplementing structured data with additional structured data that may be inferred from input structured data, and/or placing the structured data into a sequential form.

The concatenation/ aggregation module 220 concatenates, aggregates, or otherwise combines or assembles (pairs) document-level, sentence-level, or other text sequences from the token sequence generation module 216 and/or the structured data pre-processing module 218 to provide input and/or output text sequences for the NLP model executed by the NLP module 208 . Example concatenation or aggregation methods are provided herein.

An NLP training module 222 trains the NLP model executed by the NLP module 208 using the received input data (e.g., as processed by the input data processing module 214 ) in accordance with one or more training methods as provided herein. The NLP training module 222 may perform multiple training methods, including sentence-level MT training, document-level MT training, augmented document-level MT training, or NLG training, examples of which are described herein. The NLP training module 222 can also be configured for testing and/or validation of the NLP model using additional input data.

An NLP inference module 224 provides new input data from the received datasets 210 or from any other data source (which input data optionally may be processed by the input data processing module 214 ) to the NLP module 208 for performing inference during runtime. The processor 200 may further process the output data generated by the NLP model during inference, e.g., for storage, for providing for display on a display, etc.

Training an NLG Model

FIG. 3 shows an example method 300 for training a document-level NLG model (an example NLP model) using the processor

102 , 200 .

At 310 , a document-level MT model is provided by training an MT model, e.g., using NLP training module 222 , for receiving input token sequences in a first language (source) and generating output token sequences in a second language (target). Training the MT model can use parallel source and target token sequences, e.g., from a parallel corpus.

The document-level MT model can be initially provided in any suitable manner (e.g., previously generated and stored within the processor 200 (e.g., in NLP module 208 ), previously generated and stored outside of the processor and then received or acquired by the processor 200 , etc.). In some example methods, the document-level MT model can be provided by training a sentence-level MT model, e.g., using NLP training module 222 . Alternatively, an existing document-level MT model can be trained to provide the document-level MT model.

As a nonlimiting example, the document-level MT model can be configured, e.g., generated or received, and trained, for translating input tokens such as a sequence of text (e.g., words) in a first language (L1) (e.g., English) and generating output tokens such as a sequence of text (e.g., words) in a second language (L2) (e.g., French). The input and/or output tokens can further include specialized tokens, such as those indicating an end of sentence, as will be appreciated by an artisan. As the MT model provided in this step 310 is document-level, the MT model is trained by the NLP training module 222 to receive input tokens that collectively represent sequences on a larger level than sentence-level, and similarly generate output tokens that represent sequences on a larger level than sentence-level.

At 312 , the processor 102 , e.g., using the NLP training module 208 in combination with the input data processing module 214 , provides an augmented document-level MT model by training the document-level MT model (e.g., as trained at step 310 ) using structured data paired with source token data, e.g., source token sequences. The training at 312 involves training the augmented document-level MT model to receive as input, the paired structured data and source token sequences, and to generate as output, target token sequences. The training at step 312 to provide the augmented document-level MT model can be performed using MT training methods.

For example, the processor 200 , e.g., the input data processing module 214 , can provide one or more datasets having structured data and source token data that are paired with one another, such that one or more pairs are provided, each pair having structured data in a language-independent form and source token data in the first language (L1). A language-independent form is a form where the structured data can be understood in both the source language (L1) and the target language (L2). The pairs provide a source for training in this step 312 , and text in the target language L2 provides a target.

In some example methods, the structured data is or includes data that is associated with the document. For instance, the structured data can be data that represents conceptual or semantic information regarding the document. Structured data can be, as nonlimiting examples, table record data, tree data, graph data, metadata, formatted data, or any combination.

An example source of the structured data is a training set that is typically used for training natural language generation (NLG) models, examples of which are provided herein. However, other sources of data may be used. Multiple datasets, from the same source or multiple sources, can be combined, e.g., concatenated, to provide the paired structured data and/or source token data.

The structured data may be pre-processed before it is paired with the source token sequences. Example pre-processing methods are provided herein.

The source token data can be or include, for instance, text summarizing the document associated with the structured data, in the first language (L1). For instance, the structured data can represent facts for an event, which event is summarized by the text. Token target data can be received as token data by the processor 200 , or can be provided, for instance, by the processor receiving non-tokenized target text, e.g., from a training dataset, and tokenizing the non-tokenized data. Example methods for tokenizing data are provided herein, and others will be appreciated by those of ordinary skill in the art.

Target token data can be or include, for instance, text summarizing the document associated with the structured data, in the second language (L2); that is, the language of the output sequence of tokens generated by the trained document-level MT model. In other words, the source token data and the target token data can be text summarizing a document in languages L1 and L2 respectively, while the structured data paired with the source token data can be data associated with that document.

The target token data can be considered document-level data. As recognized by the present inventors, this helps align the output of the augmented document-level MT model and that of an NLG model to provide text summaries in the second language L2, and allows transfer learning between a first type of NLP model that processes input text (such as the document-level MT model) for translation between text of languages L1 and L2, and a second type of NLP model that processes structured data (such as an NLG model) and generates text in language L2.

An NLG model is provided at 314 by training the augmented document-level MT model trained at 312 to produce target token sequences as output while receiving structured data as input without paired source token sequences. Training the augmented document-level MT model, and accordingly the NLG model, in this training step 314 can be conducted by the NLP training module 222 using known methods for training neural NLG models. Example training methods (including example hyperparameters) are provided herein.

FIG. 4 shows an example method 400 for training a document-level MT model according to step 310 . At 402 an MT model, which may be implemented using the NLP module 208 , is provided using any suitable method. The provided MT model provided at step 402 may be a sentence-level MT model (processing input text on a sentence-by-sentence basis) or a document-level MT model (processing input text on a document basis).

This provided MT model may be initially trained by the NLP training module 222 using MT training methods that will be apparent to those of ordinary skill in the art. Example methods are provided herein. For instance, if the MT model is a sentence-level MT model, it may be initially trained using sentence-level data, while a document-level MT model may be initially trained using document-level data. If initial training is sentence based, text sequences can be provided by sentence-level text and/or by segmenting document-level text as desired. Training data can be provided by receiving datasets for training MT models as will be appreciated by those of ordinary skill in the art. Document-level token data for extracting (e.g., segmenting) into sentence-level token data can be provided by document-level MT model training sets. Sentence-level or document-level source token data for initial training may be in the first language (L1), while sentence-level target token data for initial training may be in the second language (L2).

At 404 it is determined whether document-level training data (e.g., existing document-level training data) is to be used to train the document-level MT model. If document-level training data is to be used (typically a higher level of training), at 406 document-level token sentences (e.g., text sequences, with any pre-processing desired as will be appreciated by an artisan) in the first and second languages L1, L2, are respectively selected, e.g., from a parallel corpus, to provide source and target data (that is, token sentences in L1 as source; token sentences in L2 as target). Example sources for such datasets include existing document-level MT training datasets or aggregated sentence data from sentence-level MT training datasets. Additional sources for document-level token data include upsampled and/or back-translated target data from NLG training sets. Selection and/or pre-processing of training data can be provided by the input data processing module 214 .

If it is determined at step 404 that existing document-level training data is not to be used, then document-level data can be simulated. At 408 random sentence-level token sequences (e.g., text sequences, with any pre-processing desired as will be appreciated by an artisan) in the first and second languages L1, L2 are selected, e.g., from a parallel corpus. For instance, a set of randomly selected sentence-level token sequences in first language L1 and their associated sentence-level token sequences in second language L2 may be selected. Alternatively or additionally, randomly selected sentence-level token sequences in second language L2 and their associated sentence-level token sequences in first language L1 may be selected.

Continuing from 408 , at 410 , the selected sentence-level token sequences in L1 and L2 are respectively concatenated, e.g., by the input data processing module 214 , to provide simulated document-level source and target data. For example, the source sequence can include concatenated randomly-selected sentences in L1, while the target sequence can include concatenated sentences in L2 respectively associated with (i.e., a translation of) the concatenated sentences in L1.

At 412 , the MT model provided at step 402 is trained on the selected data from steps

406 , or 408 / 410 , using MT training methods. It is possible that a combination of document-level and simulated-document-level data can be used to train the MT model at training step 412 . The result of training step 412 is a document-level MT model that receives an input sequence of text in first language L1 and outputs a sequence of text in second language L2.

FIG. 5 sets forth an example method 500 for augmenting a document-level machine translation model in step 312 . At 502 , structured data can be pre-processed by the processor 200 , e.g., the structured data pre-processing module 218 , to provide text sequences (or other token sequences more generally). The text sequences resulting from the pre-processing 502 can be language-agnostic, as explained above.

At 504 , the processor 200 , e.g., the concatenation/ aggregation module 220 , pairs the text sequences from the pre-processing 502 with source token data in the first language L1 (e.g., document-level) to provide source data for a training dataset. The pairing 504 can include, for instance, concatenating the text sequences with the source token data using concatenation methods that will be appreciated by those of ordinary skill in the art. Example concatenation methods are provided herein.

At 506 , the document-level MT model, for instance, the document-level MT model trained at step 310 , is trained by the NLP training module 222 using the paired source data from step 504 as source data, and target token data in second language L2 as target data as processed by the input data processing module 214 . The target token data can be token sequences (e.g., text sequences) in the second language L2 that correspond to (e.g., a translation of) the source token data in language L1 that is paired with the (language-agnostic) structured data-based text sequence in step 504 . For instance, source token data can be provided by back-translating target text that is in the second language L2 from an NLG training set. As another example, some NLG training sets include target text in more than one language, and this target text typically is document-level data. As another example, the source token data can be provided by receiving or generating text in the first language L1 from an MT training set having text in languages L1 and L2. Source token data and target token data can also be provided by both MT training data and NLG training data, concatenated with one another.

Providing the source and/or target token sequences can include pre-processing of token data, e.g., by the token sequence generation module 214 and/or concatenation/ aggregation module 220 , as will be appreciated by an artisan.

During the training 506 , to enhance consideration of the structured data (as opposed to the source token sequences), the pairing at 504 can optionally further include randomly masking a plurality of tokens in the source token sequences, and concatenating this source token sequences (having the randomly masked tokens) with the pre-processed text sequences. Masking may be performed using known methods.

In some example training methods, the training at 506 can take place over a plurality of epochs. For each epoch, a different (e.g., randomly selected) plurality of tokens may optionally be masked when source data is provided at step 504 . This ensures that the augmented document-level MT model is trained in consideration of the structured data.

The augmented MT model when trained is configured to receive an input sequence that includes both input tokens (e.g., text, special characters, etc.) to be translated in a first language and language-independent structured data, and to generate an output sequence that includes document-level output tokens (e.g., text, special characters, etc.) in a second language.

FIG. 6 shows an example method 600 for pre-processing structured data to provide text sequences using the structured data pre-processing module 218 . Typically, structured data is provided in table form, graph form, tree form, or other formats that are not easily processed by conventional machine learning models. Thus, the example pre-processing method 600 transforms structural data to a text sequence form (as a nonlimiting example, using tuples to transform graph

CLAIMS

Claims ( 26 )

What is claimed is:

1. A method of training a natural language generation (NLG) model using a processor, the method comprising:

providing a document-level machine translation (MT) model by training an MT model to receive as input, token sequences in a first language, and to generate as output, token sequences in a second language;

providing an augmented document-level MT model by training the document-level MT model to receive as input, paired language-independent structured data and token sequences in the first language, and to generate as output, token sequences in the second language; and

providing the NLG model by training the augmented document-level MT model to receive as input, language-independent structured data, and to generate as output, token sequences in the second language, wherein said training the augmented document-level MT model uses language-independent structured data as a source, without pairing the paired token sequences in the first language, and uses the token sequences in the second language as a target; and

wherein the language-independent structured data is understood in the first language and the second language.

2. The method of claim 1 , wherein the document-level MT model is trained using token sequences in the first language and the second language that are concatenated groups of sentences randomly selected from a sentence-level parallel corpus in the first language and the second language.

3. The method of claim 2 , wherein the document-level MT model is trained using token sequences in the first language and the second language that are document-level token sequences selected from a document-level parallel corpus in the first language and the second language.

4. The method of claim 1 , wherein the NLG model receives as input, language-independent structured data concerning facts surrounding an event, and generates as output, a multi-sentence summary of the event with token sequences in the second language that reference the facts surrounding the event.

5. The method of claim 4 , wherein the event comprises a sporting event.

6. The method of claim 1 , wherein the language-independent structured data is in a token sequence form.

7. The method of claim 1 , wherein the language-independent structured data is in a text sequence form.

8. The method of claim 1 , further comprising:

pre-processing structured data to provide the language-independent structured data as one or more text sequences.

9. The method of claim 8 , wherein the structured data before said pre-processing comprises one or more of table record data, graph data, metadata, or formatted data.

10. The method of claim 8 , wherein said pre-processing structured data comprises:

receiving the structured data from one or more NLG training sets; and

generating the one or more text sequences from the structured data.

11. The method of claim 10 , wherein said pre-processing further comprises filtering the received structured data.

12. The method of claim 10 , wherein said pre-processing further comprises supplementing the received structured data with additional structured data inferred from the received structured data.

13. The method of claim 10 , wherein said pre-processing further comprises converting or normalizing the received structured data into text.

14. The method of claim 10 , wherein said pre-processing further comprises converting language-dependent structured data among the received structured data into language-independent structured data that is understood in the first language and the second language.

15. The method of claim 1 , further comprising:

segmenting one or more of the language-independent structured data or the token sequences in the first language.

16. The method of claim 15 , wherein said segmenting uses byte-pair encoding (BPE).

17. The method of claim 1 , wherein the document-level MT model is initially trained as a sentence-level MT model.

18. The method of claim 1 , wherein said the document-level machine translation model is trained using token sequences in the first language that are provided by back-translating token sequences in the second language.

19. The method of claim 1 , wherein said training the document-level MT model uses an MT training method.

20. A method of generating an output text, the method comprising:

inputting new language-independent structured data into the NLG model trained according to the method of claim 1 ; and

in response to said inputting, the trained NLG model generating output text in the second language.

21. The method of claim 20 , further comprising:

receiving new structured data; and

pre-processing the received new structured data to provide the new language-independent structured data.

22. The method of claim 20 , further comprising one or more of storing or displaying the generated output text.

23. An apparatus for training a natural language generation (NLG) model to receive as input, token sequences in a first language and to generate as output, token sequences in a second language, comprising:

one or more processors; and

memory including code that, when executed by the one or more processors, performs functions including:

providing a document-level machine translation (MT) model by training an MT model to receive as input, token sequences in a first language, and to generate as output, token sequences in a second language;

providing an augmented document-level MT model by training the document-level MT model to receive as input, paired language-independent structured data and token sequences in the first language, and to generate as output, token sequences in the second language; and

providing the NLG model by training the augmented document-level MT model to receive as input, language-independent structured data, and to generate as output, token sequences in the second language, wherein said training the augmented document-level MT model uses language-independent structured data as a source, without pairing the paired token sequences in the first language, and uses the token sequences in the second language as a target; and

wherein the language-independent structured data is understood in the first language and the second language.

24. The apparatus of claim 23 , wherein the function of training the document-level MT model uses an MT training method.

25. A method of generating an output text, the method comprising:

inputting language-independent structured data into a natural language generation (NLG) model; and

in response to said inputting, the trained NLG model generating output text in a second language;

wherein the NLG model is trained with an augmented document-level machine translation (MT) model to receive as input, language-independent structured data, and to generate as output, token sequences in the second language;

wherein the augmented document-level MT model is trained with a document-level MT model to receive as input, paired language-independent structured data and token sequences in a first language, and to generate as output, token sequences in the second language; and

wherein the document-level MT model is trained with an MT model to receive as input, token sequences in the first language, and to generate as output, token sequences in the second language;

wherein said training the NLG model with the augmented document-level MT model uses language-independent structured data as a source, without pairing the paired token sequences in the first language, and uses the token sequences in the second language as a target.

26. The method of claim 25 , wherein the augmented document-level MT model is trained with a document-level MT model using an MT training method.

US17/023,660

2020-08-13

2020-09-17

Method and system for training document-level natural language processing models

Active

2040-11-28

US11625544B2

( en )

Priority Applications (3)

Application Number

Priority Date

Filing Date

Title

US17/023,660

US11625544B2

( en )

2020-08-13

2020-09-17

Method and system for training document-level natural language processing models

KR1020200139569A

KR102957039B1

( en )

2020-08-13

2020-10-26

Method and system for training document-level natural language processing models

JP2020183754A

JP7337770B2

( en )

2020-08-13

2020-11-02

Method and system for training a document-level natural language processing model

Applications Claiming Priority (2)

Application Number

Priority Date

Filing Date

Title

US202063065138P

2020-08-13

2020-08-13

US17/023,660

US11625544B2

( en )

2020-08-13

2020-09-17

Method and system for training document-level natural language processing models

Publications (2)

Publication Number

Publication Date

US20220050973A1

US20220050973A1 ( en )

2022-02-17

US11625544B2

true

US11625544B2 ( en )

2023-04-11

Family

ID=80224338

Family Applications (1)

Application Number

Title

Priority Date

Filing Date

US17/023,660

Active

2040-11-28

US11625544B2

( en )

2020-08-13

2020-09-17

Method and system for training document-level natural language processing models

Country Status (3)

Country

Link

US

( 1 )

US11625544B2

( en )

JP

( 1 )

JP7337770B2

( en )

KR

( 1 )

KR102957039B1

( en )

Cited By (1)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US12517960B1

( en )

2024-11-22

2026-01-06

Bank Of America Corporation

Integrated conditioning and machine-learning model for natural language processing

Families Citing this family (11)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US12153618B2

( en )

2015-11-02

2024-11-26

Salesforce, Inc.

Applied artificial intelligence technology for automatically generating narratives from visualization data

US11003866B1

( en )

2018-01-17

2021-05-11

Narrative Science Inc.

Applied artificial intelligence technology for narrative generation using an invocable analysis service and data re-organization

US11734517B1

( en )

*

2021-03-09

2023-08-22

Yseop Sa

Systems and methods for measuring automatability of report generation using a natural language generation system

US12001786B2

( en )

2021-05-24

2024-06-04

Salesforce, Inc.

Applied artificial intelligence technology for natural language generation using a graph data structure with follow-up capabilities

JP2026509732A

( en )

*

2023-02-15

2026-03-25

株式会社ティビング

Method and apparatus for providing similar content in a content streaming system

KR102704034B1

( en )

*

2023-09-06

2024-09-05

주식회사 티빙

Method and apparatus for providing similar contents in contents streaming system

KR102615165B1

( en )

*

2023-02-24

2023-12-15

주식회사 티빙

Method and apparatus for providing similar contents in contents streaming system

KR102615164B1

( en )

*

2023-02-15

2023-12-15

주식회사 티빙

Method and apparatus for providing similar contents in contents streaming system

WO2025017467A1

( en )

*

2023-07-14

2025-01-23

De Silva Sisilal

A method of connecting multilingual translation module/s with large language model/s for optimized output generation and output translation of user inputs in user preferred language

KR102640449B1

( en )

*

2023-08-02

2024-02-27

(주)유알피

Apparatus for hallucination removal of deep learning based generative document summary

KR102640452B1

( en )

*

2023-08-02

2024-02-27

(주)유알피

Method for generating ai based generative document summary with hallucination removed

Citations (3)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

JP2018513481A

( en )

*

2015-04-08

2018-05-24

Ｌｉｓｕｔｏ株式会社

Data transformation system and method

US20200302023A1

( en )

*

2019-03-22

2020-09-24

Fortia Financial Solutions

Generation of natural language text from structured data using a fusion model

US11222184B1

( en )

*

2015-11-02

2022-01-11

Narrative Science Inc.

Applied artificial intelligence technology for using narrative analytics to automatically generate narratives from bar charts

Family Cites Families (2)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US10789546B2

( en )

*

2016-06-23

2020-09-29

International Business Machines Corporation

Cognitive machine learning classifier generation

US10817650B2

( en )

*

2017-05-19

2020-10-27

Salesforce.Com, Inc.

Natural language processing using context specific word vectors

2020

2020-09-17

US

US17/023,660

patent/US11625544B2/en

active

Active

2020-10-26

KR

KR1020200139569A

patent/KR102957039B1/en

active

Active

2020-11-02

JP

JP2020183754A

patent/JP7337770B2/en

active

Active

Patent Citations (3)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

JP2018513481A

( en )

*

2015-04-08

2018-05-24

Ｌｉｓｕｔｏ株式会社

Data transformation system and method

US11222184B1

( en )

*

2015-11-02

2022-01-11

Narrative Science Inc.

Applied artificial intelligence technology for using narrative analytics to automatically generate narratives from bar charts

US20200302023A1

( en )

*

2019-03-22

2020-09-24

Fortia Financial Solutions

Generation of natural language text from structured data using a fusion model

Non-Patent Citations (20)

* Cited by examiner, † Cited by third party

Title

Anastasia Shimorina, Claire Gardent " Handling Rare Items in Data-to-Text Generation " Proceedings of The 11th International Natural Language Generation Conference, pp. 360-370 (Year: 2018).

*

Berard, A., et al., " Naver Labs Europe's Systems for the WMT19 Machine Translation Robustness Task, " Proceedings of the Fourth Conference on Machine Translation, 2019, (vol. 2: Shared Task Papers, Day 1), pp. 526-532.

Dowmunt, M., " Microsoft Translator at WMT 2019: Towards Large-Scale Document-Level Neural Machine Translation, " In Proceedings of the Fourth Conference on Machine Translation, 2019, (vol. 2: Shared ask Papers, Day 1), pp. 225-233. (Year: 2019).

*

Edunov, S., et al., " Understanding Back-Translation at Scale, " Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, pp. 489-500.

Jorg Tiedemann, Yves Scherrer " Neural Machine Translation with Extended Context " Proceedings of the Third Workshop on Discourse in Machine Translation, pp. 82-92 (Year: 2017).

*

Junczys-Dowmunt, M., " Microsoft Translator at WMT 2019: Towards Large-Scale Document-Level Neural Machine Translation, " In Proceedings of the Fourth Conference on Machine Translation, 2019, (vol. 2: Shared Task Papers, Day 1), pp. 225-233.

Kudo, T., et al., " SentencePiece: A Simple and Language Independent Subword Tokenizer and Detokenizer for Neural Text Processing, " In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2018, pp. 66-71.

Lebret, R., et al., " Neural Text Generation from Structured Data with Application to the Biography Domain, " In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 2016, pp. 1203-1213.

Lui, M., et al., " Langid.Py: An Off-the-Shelf Language Identification Tool, " Proceeding of the 50th Annual Meeting of the Association for Computational Linguistics, Jul. 2012, pp. 25-30.

McKeown, K., " Discourse Strategies for Generating Natural-Language Text, " Artificial Intelligence 27, No. 1, Sep. 1985, pp. 1-41.

Mei, H., et al., " What to Talk about and How? Selective Generation Using LSTMs with Coarse-to-Fine Alignment, " Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics Human Language Technologies, 2016, pp. 720-730.

Mihir Kale, Scott Roy " Machine Translation Pre-training for Data-to-Text Generation—A Case Study in Czech " arXiv:2004.02077 (Year: 2020).

*

Ott, M., et al., " Scaling Neural Machine Translation, " Proceedings of the Third Conference on Machine Translation: Research Papers, 2018, pp. 1-9.

Puduppully, R., et al., " Data-to-Text Generation with Content Selection and Planning, " Proceedings of the AAAI Conference on Artificial Intelligence 33, Jul. 17, 2019, pp. 6908-6915.

Reiter, E., et al., " Building Natural Language Generation Systems. Studies in Natural Language Processing, ", Chapter 3: The Architecture of a Natural Language Generation System Cambridge, Cambridge Univ. Press, 2000, pp. 41-70.

Sebastian Gehrmann, Falcon Z. Dai, Henry Elder, Alexander M. Rush " End-to-End Content and Plan Selection for Data-to-Text Generation " arXiv:1810.04700 (Year: 2018).

*

Sennrich, R., et al., " Neural Machine Translation of Rare Words with Subword Units, " Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, 2016, vol. 1: Long Papers, pp. 1715-1725.

Vaswani, A., et al., " Attention is All You Need, " 31st Conference on Neural Information Processing Systems, 2017, 11 pages.

Wiseman, S., et al., " Challenges in Data-to-Document Generation, " In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2017, pp. 2253-2263.

Zhiyu Chen, Harini Eavani, Wenhu Chen, Yinyin Liu, William Yang Wang " Few-Shot NLG with Pre-Trained Language Model " arXiv:1904.09521 (Year: 2020).

*

Cited By (1)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US12517960B1

( en )

2024-11-22

2026-01-06

Bank Of America Corporation

Integrated conditioning and machine-learning model for natural language processing

Also Published As

Publication number

Publication date

KR102957039B1

( en )

2026-04-24

US20220050973A1

( en )

2022-02-17

KR20220021360A

( en )

2022-02-22

JP7337770B2

( en )

2023-09-04

JP2022032910A

( en )

2022-02-25

Similar Documents

Publication

Publication Date

Title

US20220050973A1

( en )

2022-02-17

Method and system for training document-level natural language processing models

Lin et al.

2021

Common sense beyond English: Evaluating and improving multilingual language models for commonsense reasoning

Zhang et al.

2021

AMBERT: A pre-trained language model with multi-grained tokenization

Mave et al.

2018

Language identification and analysis of code-switched social media text

US20220383159A1

( en )

2022-12-01

Systems and methods for open domain multi-hop question answering

Inoue et al.

2022

Morphosyntactic tagging with pre-trained language models for Arabic and its dialects

US20140163951A1

( en )

2014-06-12

Hybrid adaptation of named entity recognition

Masmoudi et al.

2019

Transliteration of Arabizi into Arabic script for Tunisian dialect

US11669691B2

( en )

2023-06-06

Information processing apparatus, information processing method, and computer readable recording medium

CN106407211A

( en )

2017-02-15

Method and device for classifying semantic relationships among entity words

US20230367977A1

( en )

2023-11-16

Word alignment apparatus, learning apparatus, word alignment method, learning method and program

Liu et al.

2022

Singlish message paraphrasing: A joint task of creole translation and text normalization

Tafreshi et al.

2024

Emotion classification in low and moderate resource languages

Ezeani et al.

2019

Leveraging pre-trained embeddings for Welsh taggers

Novák et al.

2025

Findings of the Fourth Shared Task on Multilingual Coreference Resolution: Can LLMs Dethrone Traditional Approaches?

Tran et al.

2020

Webnlg 2020 challenge: semantic template mining for generating references from rdf

Simonsen

2024

Improving Machine Translation for Faroese using ChatGPT-Generated Parallel Data

KR102743434B1

( en )

2024-12-17

Apparatus and method for learning language model based on tokenization and embedding optimized for korean, and method for generaging vocabulary

Nguyen et al.

2023

Natural language generation from Universal Dependencies using data augmentation and pre-trained language models

Stankevičius et al.

2022

Towards Lithuanian grammatical error correction

CN111090720A

( en )

2020-05-01

Hot word adding method and device

Kwok et al.

2023

Cantonese to written chinese translation via huggingface translation pipeline

Shingi et al.

2021

AtteSTNet--An attention and subword tokenization based approach for code-switched text hate speech detection

Jha et al.

2023

Deep Learning Based Text Translation and Summarization Tool for Hearing Impaired Using Indian Sign Language.

Degu et al.

2024

Fine‐Tuned Pretrained Transformer for Amharic News Headline Generation

Legal Events

Date

Code

Title

Description

2020-09-17

FEPP

Fee payment procedure

Free format text : ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

2020-10-01

AS

Assignment

Owner name : NAVER CORPORATION, KOREA, REPUBLIC OF

Free format text : ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:CALAPODESCU, IOAN;BERARD, ALEXANDRE;SALEH, FAHIMEH;AND OTHERS;SIGNING DATES FROM 20200925 TO 20200929;REEL/FRAME:053949/0576

2021-08-20

STPP

Information on status: patent application and granting procedure in general

Free format text : DOCKETED NEW CASE - READY FOR EXAMINATION

2022-03-28

STPP

Information on status: patent application and granting procedure in general

Free format text : NON FINAL ACTION MAILED

2022-09-01

STPP

Information on status: patent application and granting procedure in general

Free format text : RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER

2022-12-09

STPP

Information on status: patent application and granting procedure in general

Free format text : NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS

2023-03-22

STCF

Information on status: patent grant

Free format text : PATENTED CASE

Related documents

• Generating augmented reality experiences utilizing physical objects to … — Meta Platforms Technologies, Llc (US20260037214A1)#653960
• Virtual reality system using multiple force arrays for a solver — DeepMotion, Inc. (US10657696B2)#653961
• Tracking position of device inside-out for virtual reality interactivity — Sony Interactive Entertainment America Llc (US10535153B2)#653962
• Augmented reality system with spatialized audio tied to user manipulated … — Magic Leap, Inc. (US11231770B2)#653963
• Big data-based autonomous flight drone system and autonomous flight method … — Rgblab Co., Ltd. (US20210390867A1)#653964