Google Patentsopen access

Natural language text generation from a set of keywords using machine learning … — International Business Machines Corporation (US11586829B2)

ABSTRACT

Abstract

An embodiment of the present invention generates natural language content from a set of keywords in accordance with a template. Keyword vectors representing a context for the keywords are generated. The keywords are associated with language tags, while the template includes a series of language tags indicating an arrangement for the generated natural language content. Template vectors are generated from the series of language tags of the template and represent a context for the template. Contributions from the contexts for the keywords and the template are determined based on a comparison of the series of language tags of the template with the associated language tags of the keywords. One or more words for each language tag of the template are generated to produce the natural language content based on combined contributions from the contexts for the keywords and the template.

Description

BACKGROUND

1. Technical Field

Present invention embodiments relate to natural language generation, and more specifically, to generating natural language content (e.g., sentences, clauses, phrases, etc.) from a set of keywords using machine learning and templates.

2. Discussion of the Related Art

Generating realistic sentences is a challenging task since this requires forming syntactically, well-structured sentences, while capturing complex and plausible semantic structures underlying those sentences. Some previous approaches regarding language generation focus on task-specific applications in supervised settings (e.g., machine translation, image captioning, etc.). Further approaches address generic text generation (e.g., political speech generation, poem generation, etc.), where training samples are obtained from a corresponding task specific corpus. Generation of semantically equivalent literal or sarcastic paraphrased sentences, or even simpler sentences, has also been explored.

However, generating sentences from a few given keywords is a difficult task. An example input for this task is a set of keywords (e.g., “victim”, “Vanessa”, “demons”), and an acceptable output for the example input is the sentence “Vanessa can also become a victim of demons”. This generation is complex since natural plausible sentences cannot be generated by merely reordering a few keywords. Unlike sentences typically provided as input to traditional language generation systems, keywords do not inform the system how to learn semantic, lexical and syntactic aspects to produce a valid text containing those keywords. Also, keywords can come from a structured input (such as tables, data-structures, knowledgebases, etc.) in any order creating additional challenges for the learning system. This is not the case for input sentences in traditional systems, since words in a sentence are grammatically ordered.

Controllable text generation has been explored mainly for the text-to-text domain. Existing approaches for text-to-text controllable text generation only receive input parameters, and generate text by sampling text randomly from language models. However, these approaches are not applicable when input text is given, or the input is in a non-sentential form (e.g., not in the form of a sentence).

Another form of controllable text generation includes rule-based controllable styling. This approach receives input text and style parameters, and requires a domain expert to design rules that transform the input text into user-specified stylized text. Although the rule-based control styling approach is intuitive, this approach is not feasible to scale to various domains, data formats, and template patterns.

Accordingly, existing controllable text generation approaches are special purpose (e.g., style specification specific, etc.), where the design strongly depends on the input parameters. The extension of these approaches to adapt to even a slightest variation (e.g., to add a new tunable parameter) is extremely difficult.

SUMMARY

According to one embodiment of the present invention, a system comprises a processor to generate natural language content from a set of keywords in accordance with a template. Keyword vectors representing a context for the keywords are generated. The keywords are associated with language tags, while the template includes a series of language tags indicating an arrangement for the generated natural language content. Templates can be created from sentences of a large unlabeled text corpus. In other words, any sentence can be used to create a template. Template vectors are generated from the series of language tags of the template and represent a context for the template. Contributions from the contexts for the keywords and the template are determined based on a comparison of the series of language tags of the template with the associated language tags of the keywords. One or more words for each language tag of the template are generated to produce the natural language content based on combined contributions from the contexts for the keywords and the template. Embodiments of the present invention further include a method and computer program product for generating natural language content from a set of keywords and a template in substantially the same manner described above.

The embodiment of the present invention lessens the burden of learning deep syntactic structures in a language due to the template, thereby reducing training time and improving computer performance.

An embodiment of the present invention may further employ language tags of the keywords and templates that include part-of-speech (POS) tags. The part-of-speech (POS) tag sequences of the template provide additional hints about the morphological, lexical and syntactic changes to be made to the keywords and the function words to be additionally used. From a few possible part-of-speech (POS) tags (e.g., NOUNS, VERBS, ADJECTIVES, etc.), numerous POS categories may map to a few candidate words in a vocabulary. This significantly limits the number of choices for a large number of decoding steps, thereby reducing processing for the decoding and enabling a language model of a decoder to have a strong influence on the generation process.

An embodiment of the present invention may also determine a probability for each language tag of the template indicating a likelihood of that language tag of the template matching one of the associated language tags of the keywords. The probability for a corresponding language tag of the template indicates the contribution for the context of the keywords for generating a word for the corresponding language tag of the template, and a complement of the probability indicates the contribution for the context of the template for generating the word for the corresponding language tag of the template. This enables a decoder to shift focus between keyword representations and generic language (e.g., POS, etc.) representations based on the language (e.g., POS, etc.) representations in the template. This helps the decoder decide whether to produce a word related to the input keywords and the morphological form, or to simply introduce a function word (e.g., preposition, etc.) based on the language (e.g., POS, etc.) tags in the template and previously generated words.

An embodiment of the present invention may further determine the associated language tags for the keywords via a machine learning model. The machine learning model is trained with a data set including complete sentences and the complete sentences without function words. This provides a context oblivious approach that improves the accuracy of the tagging, thereby enabling natural language content to be generated that more closely aligns with the keywords and template.

An embodiment of the present invention may also provide language translation, where the keywords are in a first natural language and the generated natural language content is in a second different natural language. The present invention embodiment is indifferent to syntactic order of the keywords or the underlying natural language, thereby enabling flexibility in terms of natural languages processed.

An embodiment of the present invention may further generate the keyword vectors by generating word embeddings for each of the keywords, encoding the word embeddings using a machine learning model to produce encoded vector representations of the keywords, and generating the keyword vectors based on the encoded vector representations. The machine learning model is trained to produce the same encoded vector representations for a set of keywords regardless of an order of the keywords. This enables the natural language content generation to be agnostic of the order of the keywords, thereby generating consistent content from the same keywords (regardless of order) that closely aligns with the template.

An embodiment of the present invention may generate the keyword vectors based on the encoded vector representations by applying attention weights to the encoded vector representations of the keywords to produce a keyword vector for a corresponding language tag of the template as a weighted combination of the encoded vector representations. The attention weights indicate importance of individual keywords and are based on the corresponding language tag of the template. This ensures that the template tags influence the attention mechanism and selection of natural language content, and enables the decoder to provide greater flexibility in extracting contextual information from either the keywords or the template, while ensuring that unnecessary attention is not given to the keywords (unless a current step of generation requires additional attention).

An embodiment of the present invention may still further generate the template vectors by generating word embeddings for each of the language tags of the template, encoding the word embeddings for the language tags of the template using a bidirectional recurrent machine learning model, and producing the template vectors based on the encoded word embeddings for the language tags of the template. Each template vector is produced based on adjacent language tags within the template. This enables the template vectors to more accurately capture the template context since adjacent tags are considered for generating the template vectors.

An embodiment of the present invention may also generate one or more words for each language tag of the template by determining for each language tag of the template a probability distribution over a word vocabulary using a recurrent machine learning model, and selecting one or more words from the word vocabulary for a corresponding language tag of the template based on the probability distribution. The word vocabulary is learned from training data during training of the recurrent machine learning model. This enables function words to be introduced in the generated natural language content from the learned word vocabulary, and produces content that more accurately captures the template since previous words are considered for selecting subsequent words.

BRIEF DESCRIPTION OF THE DRAWINGS

Generally, like reference numerals in the various figures are utilized to designate like components.

FIG. 1 is a diagrammatic illustration of an example computing environment according to an embodiment of the present invention.

FIG. 2 is a block diagram of an example computing device of the computing environment of FIG. 1 according to an embodiment of the present invention.

FIG. 3 is a block diagram illustrating a flow of the transform module of FIG. 1 for generating natural language content according to an embodiment of the present invention.

FIG. 4 is a procedural flowchart illustrating a manner of generating natural language content according to an embodiment of the present invention.

FIG. 5 is a block diagram illustrating an alternative flow of the transform module of FIG. 1 for generating natural language content using context oblivious part-of speech (POS) tagging of keywords according to an embodiment of the present invention.

FIG. 6 is a procedural flowchart illustrating a manner of tagging keywords with part-of-speech (POS) tags using a context oblivious approach according to an embodiment of the present invention.

FIG. 7 illustrates example training data for tagging keywords with part-of-speech (POS) tags according to an embodiment of the present invention.

FIG. 8 illustrates example metrics for natural language content generation produced from present invention embodiments in comparison with other systems.

FIG. 9 illustrates example natural language content produced from a present invention embodiment in comparison with other systems.

DETAILED DESCRIPTION

Most natural languages provide numerous manners for constructing sentences that convey the same meaning. However, constructs within a natural language are often repetitive and follow a finite set of syntactic and stylistic variations (e.g., a manner that humans greet each other may be captured through a finite set of predominant greeting styles). Modern language generators, especially neural network models, are over-burdened due to a requirement of meticulously memorizing linguistic syntax and styles apart from performing content selection from input and surface realization (e.g., all in a single episode of learning).

Present invention embodiments guide language generators by using generic structures within a language that are easily available, shallow in nature, and interpretable by humans. This avoids a difficult learning curve of modeling syntax, and enables language generation to be more tractable and accurate.

A data-to-text generator according to an embodiment of the present invention is not concerned about syntax. Rather, the data-to-text generator according to the present invention embodiment focuses on a manner to extract vital content from input keywords and determine an order in which a selected context should appear in the output. The data-to-text generator according to the present invention embodiment logically derives appropriate morphological forms of content words, and intermittently introduces function words as necessary. These processes are made smoother by guiding the data-to-text generator according to the present invention embodiment with a template (e.g., part-of-speech (POS) tag sequence of an example sentence).

Present invention embodiments employ a control specification, in the form of templates, that uses an input-style agnostic approach for which a domain expert is not required. This control specification, in terms of templates, is dynamic since templates in a natural language (even though finite) may be considerably large in number (e.g., millions, etc.).

A present invention embodiment uses natural language generation (NLG) to summarize data in natural language text, where input may be values in tables, lists of keywords, key-value pairs, knowledge graph entries, etc. The present invention embodiment transforms input text according to an input template. The present invention embodiment does not require explicit supervision (e.g., training data can be automatically generated, etc.), and may utilize easily implementable or available natural language processing (NLP) systems as a source for generating training data. The present invention embodiment may detect part-of-speech (POS) tags of keywords accurately using a context oblivious approach, and may be used to generate a sentence in one language from keywords in another language by using universal POS tags.

An embodiment of the present invention provides for unsupervised template controllable data. Input data are received including a set of keywords and a human interpretable template (e.g., represented by a sequence of part-of-speech (POS) tags). POS tags for the input keywords may be detected using a deep learning-based context oblivious POS tagging approach. Using the input data (e.g., the set of keywords, the detected POS tags, and the template), output summarization text is generated using a deep learning-based natural language generation (NLG) approach. This approach enables output text to be semantically related to the input data, thereby conveying the desired meaning. Further, the output text is transformed according to the template, and knowledge of adhering to the template style may be obtained from easily implement

BACKGROUND

1. Technical Field

Present invention embodiments relate to natural language generation, and more specifically, to generating natural language content (e.g., sentences, clauses, phrases, etc.) from a set of keywords using machine learning and templates.

2. Discussion of the Related Art

Generating realistic sentences is a challenging task since this requires forming syntactically, well-structured sentences, while capturing complex and plausible semantic structures underlying those sentences. Some previous approaches regarding language generation focus on task-specific applications in supervised settings (e.g., machine translation, image captioning, etc.). Further approaches address generic text generation (e.g., political speech generation, poem generation, etc.), where training samples are obtained from a corresponding task specific corpus. Generation of semantically equivalent literal or sarcastic paraphrased sentences, or even simpler sentences, has also been explored.

However, generating sentences from a few given keywords is a difficult task. An example input for this task is a set of keywords (e.g., “victim”, “Vanessa”, “demons”), and an acceptable output for the example input is the sentence “Vanessa can also become a victim of demons”. This generation is complex since natural plausible sentences cannot be generated by merely reordering a few keywords. Unlike sentences typically provided as input to traditional language generation systems, keywords do not inform the system how to learn semantic, lexical and syntactic aspects to produce a valid text containing those keywords. Also, keywords can come from a structured input (such as tables, data-structures, knowledgebases, etc.) in any order creating additional challenges for the learning system. This is not the case for input sentences in traditional systems, since words in a sentence are grammatically ordered.

Controllable text generation has been explored mainly for the text-to-text domain. Existing approaches for text-to-text controllable text generation only receive input parameters, and generate text by sampling text randomly from language models. However, these approaches are not applicable when input text is given, or the input is in a non-sentential form (e.g., not in the form of a sentence).

Another form of controllable text generation includes rule-based controllable styling. This approach receives input text and style parameters, and requires a domain expert to design rules that transform the input text into user-specified stylized text. Although the rule-based control styling approach is intuitive, this approach is not feasible to scale to various domains, data formats, and template patterns.

Accordingly, existing controllable text generation approaches are special purpose (e.g., style specification specific, etc.), where the design strongly depends on the input parameters. The extension of these approaches to adapt to even a slightest variation (e.g., to add a new tunable parameter) is extremely difficult.

SUMMARY

According to one embodiment of the present invention, a system comprises a processor to generate natural language content from a set of keywords in accordance with a template. Keyword vectors representing a context for the keywords are generated. The keywords are associated with language tags, while the template includes a series of language tags indicating an arrangement for the generated natural language content. Templates can be created from sentences of a large unlabeled text corpus. In other words, any sentence can be used to create a template. Template vectors are generated from the series of language tags of the template and represent a context for the template. Contributions from the contexts for the keywords and the template are determined based on a comparison of the series of language tags of the template with the associated language tags of the keywords. One or more words for each language tag of the template are generated to produce the natural language content based on combined contributions from the contexts for the keywords and the template. Embodiments of the present invention further include a method and computer program product for generating natural language content from a set of keywords and a template in substantially the same manner described above.

The embodiment of the present invention lessens the burden of learning deep syntactic structures in a language due to the template, thereby reducing training time and improving computer performance.

An embodiment of the present invention may further employ language tags of the keywords and templates that include part-of-speech (POS) tags. The part-of-speech (POS) tag sequences of the template provide additional hints about the morphological, lexical and syntactic changes to be made to the keywords and the function words to be additionally used. From a few possible part-of-speech (POS) tags (e.g., NOUNS, VERBS, ADJECTIVES, etc.), numerous POS categories may map to a few candidate words in a vocabulary. This significantly limits the number of choices for a large number of decoding steps, thereby reducing processing for the decoding and enabling a language model of a decoder to have a strong influence on the generation process.

An embodiment of the present invention may also determine a probability for each language tag of the template indicating a likelihood of that language tag of the template matching one of the associated language tags of the keywords. The probability for a corresponding language tag of the template indicates the contribution for the context of the keywords for generating a word for the corresponding language tag of the template, and a complement of the probability indicates the contribution for the context of the template for generating the word for the corresponding language tag of the template. This enables a decoder to shift focus between keyword representations and generic language (e.g., POS, etc.) representations based on the language (e.g., POS, etc.) representations in the template. This helps the decoder decide whether to produce a word related to the input keywords and the morphological form, or to simply introduce a function word (e.g., preposition, etc.) based on the language (e.g., POS, etc.) tags in the template and previously generated words.

An embodiment of the present invention may further determine the associated language tags for the keywords via a machine learning model. The machine learning model is trained with a data set including complete sentences and the complete sentences without function words. This provides a context oblivious approach that improves the accuracy of the tagging, thereby enabling natural language content to be generated that more closely aligns with the keywords and template.

An embodiment of the present invention may also provide language translation, where the keywords are in a first natural language and the generated natural language content is in a second different natural language. The present invention embodiment is indifferent to syntactic order of the keywords or the underlying natural language, thereby enabling flexibility in terms of natural languages processed.

An embodiment of the present invention may further generate the keyword vectors by generating word embeddings for each of the keywords, encoding the word embeddings using a machine learning model to produce encoded vector representations of the keywords, and generating the keyword vectors based on the encoded vector representations. The machine learning model is trained to produce the same encoded vector representations for a set of keywords regardless of an order of the keywords. This enables the natural language content generation to be agnostic of the order of the keywords, thereby generating consistent content from the same keywords (regardless of order) that closely aligns with the template.

An embodiment of the present invention may generate the keyword vectors based on the encoded vector representations by applying attention weights to the encoded vector representations of the keywords to produce a keyword vector for a corresponding language tag of the template as a weighted combination of the encoded vector representations. The attention weights indicate importance of individual keywords and are based on the corresponding language tag of the template. This ensures that the template tags influence the attention mechanism and selection of natural language content, and enables the decoder to provide greater flexibility in extracting contextual information from either the keywords or the template, while ensuring that unnecessary attention is not given to the keywords (unless a current step of generation requires additional attention).

An embodiment of the present invention may still further generate the template vectors by generating word embeddings for each of the language tags of the template, encoding the word embeddings for the language tags of the template using a bidirectional recurrent machine learning model, and producing the template vectors based on the encoded word embeddings for the language tags of the template. Each template vector is produced based on adjacent language tags within the template. This enables the template vectors to more accurately capture the template context since adjacent tags are considered for generating the template vectors.

An embodiment of the present invention may also generate one or more words for each language tag of the template by determining for each language tag of the template a probability distribution over a word vocabulary using a recurrent machine learning model, and selecting one or more words from the word vocabulary for a corresponding language tag of the template based on the probability distribution. The word vocabulary is learned from training data during training of the recurrent machine learning model. This enables function words to be introduced in the generated natural language content from the learned word vocabulary, and produces content that more accurately captures the template since previous words are considered for selecting subsequent words.

BRIEF DESCRIPTION OF THE DRAWINGS

Generally, like reference numerals in the various figures are utilized to designate like components.

FIG. 1 is a diagrammatic illustration of an example computing environment according to an embodiment of the present invention.

FIG. 2 is a block diagram of an example computing device of the computing environment of FIG. 1 according to an embodiment of the present invention.

FIG. 3 is a block diagram illustrating a flow of the transform module of FIG. 1 for generating natural language content according to an embodiment of the present invention.

FIG. 4 is a procedural flowchart illustrating a manner of generating natural language content according to an embodiment of the present invention.

FIG. 5 is a block diagram illustrating an alternative flow of the transform module of FIG. 1 for generating natural language content using context oblivious part-of speech (POS) tagging of keywords according to an embodiment of the present invention.

FIG. 6 is a procedural flowchart illustrating a manner of tagging keywords with part-of-speech (POS) tags using a context oblivious approach according to an embodiment of the present invention.

FIG. 7 illustrates example training data for tagging keywords with part-of-speech (POS) tags according to an embodiment of the present invention.

FIG. 8 illustrates example metrics for natural language content generation produced from present invention embodiments in comparison with other systems.

FIG. 9 illustrates example natural language content produced from a present invention embodiment in comparison with other systems.

DETAILED DESCRIPTION

Most natural languages provide numerous manners for constructing sentences that convey the same meaning. However, constructs within a natural language are often repetitive and follow a finite set of syntactic and stylistic variations (e.g., a manner that humans greet each other may be captured through a finite set of predominant greeting styles). Modern language generators, especially neural network models, are over-burdened due to a requirement of meticulously memorizing linguistic syntax and styles apart from performing content selection from input and surface realization (e.g., all in a single episode of learning).

Present invention embodiments guide language generators by using generic structures within a language that are easily available, shallow in nature, and interpretable by humans. This avoids a difficult learning curve of modeling syntax, and enables language generation to be more tractable and accurate.

A data-to-text generator according to an embodiment of the present invention is not concerned about syntax. Rather, the data-to-text generator according to the present invention embodiment focuses on a manner to extract vital content from input keywords and determine an order in which a selected context should appear in the output. The data-to-text generator according to the present invention embodiment logically derives appropriate morphological forms of content words, and intermittently introduces function words as necessary. These processes are made smoother by guiding the data-to-text generator according to the present invention embodiment with a template (e.g., part-of-speech (POS) tag sequence of an example sentence).

Present invention embodiments employ a control specification, in the form of templates, that uses an input-style agnostic approach for which a domain expert is not required. This control specification, in terms of templates, is dynamic since templates in a natural language (even though finite) may be considerably large in number (e.g., millions, etc.).

A present invention embodiment uses natural language generation (NLG) to summarize data in natural language text, where input may be values in tables, lists of keywords, key-value pairs, knowledge graph entries, etc. The present invention embodiment transforms input text according to an input template. The present invention embodiment does not require explicit supervision (e.g., training data can be automatically generated, etc.), and may utilize easily implementable or available natural language processing (NLP) systems as a source for generating training data. The present invention embodiment may detect part-of-speech (POS) tags of keywords accurately using a context oblivious approach, and may be used to generate a sentence in one language from keywords in another language by using universal POS tags.

An embodiment of the present invention provides for unsupervised template controllable data. Input data are received including a set of keywords and a human interpretable template (e.g., represented by a sequence of part-of-speech (POS) tags). POS tags for the input keywords may be detected using a deep learning-based context oblivious POS tagging approach. Using the input data (e.g., the set of keywords, the detected POS tags, and the template), output summarization text is generated using a deep learning-based natural language generation (NLG) approach. This approach enables output text to be semantically related to the input data, thereby conveying the desired meaning. Further, the output text is transformed according to the template, and knowledge of adhering to the template style may be obtained from easily implementable custom or conventional natural language processing (NLP) systems (e.g., deterministic or machine learning based classifiers), regressors, and/or metrics.

A present invention embodiment considers a set of keywords, a part-of-speech (POS) tag sequence in the form of a template, and may further consider universal POS forms of the keywords. The keywords and template are encoded into vector forms using neural network encoding techniques. A decoder generates a finite number of words according to the POS tags in the template. During generation, the decoder shifts focus between keyword representations and generic POS representations based on the POS representations in the template. This helps the decoder decide whether to produce a word related to the input keywords and the morphological form, or to simply introduce a function word (e.g., preposition, etc.) based on the POS tags in the template and previously generated words.

A present invention embodiment provides a machine learning framework for controllable natural language generation (NLG) from a set of keywords. Realization of keywords in natural language form may be performed in various (but potentially finite) manners according to diverse, lexical, and syntactic structures in a language. The framework accepts an additional input in the form of human-interpretable templates, and generates output that conforms to the template syntax. The template not only lessens the burden of the framework towards learning language structure, but also enables sentence generation to be controllable and tractable. The framework is based on an encode-attend-decode paradigm, where keywords and templates are encoded using linear and recurrent units. A decoder attends over contexts derived from the encoded keywords and templates. Words are produced by either generating morpho-syntactic variations of the keywords (or words related to the keywords), or selecting suitable function words from a vocabulary. The templates include part-of-speech (POS) tag sequences of valid English or other natural language sentences. The framework relies on keywords and templates for training that are easily extracted from a large volume of unlabeled data.

Present invention embodiments provide several advantages. For example, the decoder expects to generate a finite number of words, which should ideally distribute the probability masses over the whole sentence. The burden of learning deep syntactic structures in a language is lessened due to the additional template input. The part-of-speech (POS) tag sequences of templates provide additional hints about the morphological, lexical and syntactic changes to be made to the keywords and the function words to be additionally used. From a few possible part-of-speech POS tags (e.g., NOUNS, VERBS, ADJECTIVES, etc.), numerous POS categories may map to a few candidate words in the vocabulary. This significantly limits the number of choices for a large number of decoding steps, and enables a language model of the decoder to have a strong influence on the generation process.

An example computing environment for use with present invention embodiments is illustrated in FIG. 1 . Specifically, computing environment 100 includes one or more server systems 110 , and one or more client or end- user systems 114 . Server systems 110 and client systems 114 may be remote from each other and communicate over a network 112 . The network may be implemented by any number of any suitable communications media (e.g., wide area network (WAN), local area network (LAN), Internet, Intranet, etc.). Alternatively, server systems 110 and client systems 114 may be local to each other, and communicate via any appropriate local communication medium (e.g., local area network (LAN), hardwire, wireless link, Intranet, etc.).

Client systems 114 enable users to submit sets of keywords and templates (and optionally part-of-speech (POS) tags for the keywords) to server systems 110 for generation of natural language content (e.g., sentences, clauses, phrases, etc.). The client systems include an interface module or browser 120 to interact with server systems 110 . The server systems include a transform module 116 to generate natural language content (e.g., sentence, clause, phrase, etc.) from a set of keywords and a template provided to the transform module. Transform module 116 may also be provided with part-of-speech (POS) tags for the keywords, or generate the part-of-speech (POS) tags for the keywords based on the provided set of keywords. A database system 118 , memory of the client and/or server systems, and/or other storage units may store various information for the analysis (e.g., training data, vocabulary, machine learning or other parameters, dictionaries, etc.). The database system may be implemented by any conventional or other database or storage unit, may be local to or remote from server systems 110 and client systems 114 , and may communicate via any appropriate communication medium (e.g., local area network (LAN), wide area network (WAN), Internet, hardwire, wireless link, Intranet, etc.). The client systems may present a graphical user (e.g., GUI, etc.) or other interface (e.g., command line prompts, menu screens, etc.) to solicit information from users pertaining to the sets of keywords, corresponding POS tags, and templates, and may provide reports including analysis results (e.g., generated natural language content, metrics pertaining to the analysis, etc.).

Server systems 110 and client systems 114 may be implemented by any conventional or other computer systems (e.g., FIG. 2 as described below) preferably equipped with a display or monitor, a base (e.g., including at least one hardware processor 115 (e.g., microprocessor, controller, central processing unit (CPU), etc.), one or more memories 135 and/or internal or external network interfaces or communications devices 125 (e.g., modem, network cards, etc.)), optional input devices (e.g., a keyboard, mouse or other input device), and any commercially available and custom software (e.g., server/communications software, transform module 116 , interface module 120 , etc.).

Alternatively, one or more client systems 114 may generate natural language content (e.g., sentence, clause, phrase, etc.) from a set of keywords and a template when operating as a stand-alone unit. In a stand-alone mode of operation, the client system stores or has access to the data (e.g., training data, vocabulary, machine learning or other parameters, dictionaries, etc.), and includes transform module 116 to generate natural language content (e.g., sentence, clause, phrase, etc.) from a set of keywords and a template provided to the transform module. Transform module 116 may also be provided with part-of-speech (POS) tags for the keywords, or generate the part-of-speech (POS) tags for the keywords based on the provided set of keywords. The graphical user (e.g., GUI, etc.) or other interface (e.g., command line prompts, menu screens, etc.) solicits information from a corresponding user pertaining to the desired set of keywords (and optionally the corresponding part-of-speech (POS) tags for the keywords) and template, and may provide reports including the generated natural language content.

Transform and interface modules

116 , 120 may include one or more modules or units to perform the various functions of present invention embodiments described below. The various modules (e.g., transform module 116 , interface module 120 , etc.) may be implemented by any combination of any quantity of software and/or hardware modules or units, and may reside within memory 135 of the server and/or client systems for execution by processor 115 .

Referring now to FIG. 2 , a schematic of an example of a computing device 210 of computing environment 100 (e.g., implementing server system 110 and/or client system 114 ) is shown. The computing device is only one example of a suitable computing device for computing environment 100 and is not intended to suggest any limitation as to the scope of use or functionality of embodiments of the invention described herein. Regardless, computing device 210 is capable of being implemented and/or performing any of the functionality set forth herein.

In computing device 210 , there is a computer system 212 which is operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of computing systems, environments, and/or configurations that may be suitable for use with computer system 212 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, hand-held or laptop devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or devices, and the like.

Computer system 212 may be described in the general context of computer system executable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, and so on that perform particular tasks or implement particular abstract data types.

As shown in FIG. 2 , computer system 212 is shown in the form of a general-purpose computing device. The components of computer system 212 may include, but are not limited to, one or more processors or processing units 115 , a system memory 135 , and a bus 218 that couples various system components including system memory 135 to processor 115 .

Bus 218 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnects (PCI) bus.

Computer system 212 typically includes a variety of computer system readable media. Such media may be any available media that is accessible by computer system 212 , and includes both volatile and non-volatile media, removable and non-removable media.

System memory 135 can include computer system readable media in the form of volatile memory, such as random access memory (RAM) 230 and/or cache memory 232 . Computer system 212 may further include other removable/non-removable, volatile/non-volatile computer system storage media. By way of example only, storage system 234 can be provided for reading from and writing to a nonremovable, non-volatile magnetic media (not shown and typically called a “hard drive”). Although not shown, a magnetic disk drive for reading from and writing to a removable, non-volatile magnetic disk (e.g., a “floppy disk”), and an optical disk drive for reading from or writing to a removable, non-volatile optical disk such as a CD-ROM, DVD-ROM or other optical media can be provided. In such instances, each can be connected to bus 218 by one or more data media interfaces. As will be further depicted and described below, memory 135 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the invention.

Program/ utility 240 , having a set (at least one) of program modules 242 (e.g., transform module 116 , interface module 120 , etc.) may be stored in memory 135 by way of example, and not limitation, as well as an operating system, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data or some combination thereof, may include an implementation of a networking environment. Program modules 242 generally carry out the functions and/or methodologies of embodiments of the invention as described herein.

Computer system 212 may also communicate with one or more external devices 214 such as a keyboard, a pointing device, a display 224 , etc.; one or more devices that enable a user to interact with computer system 212 ; and/or any devices (e.g., network card, modem, etc.) that enable computer system 212 to communicate with one or more other computing devices. Such communication can occur via Input/Output (I/O) interfaces 222 . Still yet, computer system 212 can communicate with one or more networks such as a local area network (LAN), a general wide area network (WAN), and/or a public network (e.g., the Internet) via network adapter 125 . As depicted, network adapter 125 communicates with the other components of computer system 212 via bus 218 . It should be understood that although not shown, other hardware and/or software components could be used in conjunction with computer system 212 . Examples, include, but are not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

According to one embodiment of the present invention, a system comprises a processor to generate natural language content from a set of keywords in accordance with a template. Keyword vectors representing a context for the keywords are generated. The keywords are associated with language tags (e.g., part-of-speech (POS) or other tags, etc.), while the template includes a series of language tags (e.g., part-of-speech (POS) or other tags, etc.) indicating an arrangement for the generated natural language content. Template vectors are generated from the series of language tags of the template and represent a context for the template. Contributions from the contexts for the keywords and the template are determined based on a comparison of the series of language tags of the template with the associated language tags of the keywords. One or more words for each language tag of the template are generated to produce the natural language content based on combined contributions from the contexts for the keywords and the template. Embodiments of the present invention further include a method and computer program product for generating natural language content from a set of keywords and a template in substantially the same manner described above.

The embodiment of the present invention lessens the burden of learning deep syntactic structures in a language due to the template, thereby reducing training time and improving computer performance.

An embodiment of the present invention may further employ language tags of the keywords and templates that include part-of-speech (POS) tags. The part-of-speech (POS) tag sequences of the template provide additional hints about the morphological, lexical and syntactic changes to be made to the keywords and the function words to be additionally used. From a few possible part-of-speech POS tags (e.g., NOUNS, VERBS, ADJECTIVES, etc.), numerous POS categories may map to a few candidate words in a vocabulary. This significantly limits the number of choices for a large number of decoding steps, thereby reducing processing for the decoding and enabling a language model of a decoder to have a strong influence on the generation process.

An embodiment of the present invention may also determine a probability for each language tag of the template indicating a likelihood of that language tag of the template matching one of the associated language tags of the keywords. The probability for a corresponding language tag of the template indicates the contribution for the context of the keywords for generating a word for the corresponding language tag of the template, and a complement of the probability indicates the contribution for the context of the template for generating the word for the corresponding language tag of the template. This enables a decoder to shift focus between keyword representations and generic language (e.g., POS, etc.) representations based on the language (e.g., POS, etc.) representations in the template. This helps the decoder decide whether to produce a word related to the input keywords and the morphological form, or to simply introduce a function word (e.g., preposition, etc.) based on the language (e.g., POS, etc.) tags in the template and previously generated words.

An embodiment of the present invention may further determine the associated language tags for the keywords via a machine learning model. The machine learning model is trained with a data set including complete sentences and the complete sentences without function words. This provides a context oblivious approach that improves the accuracy of the tagging, thereby enabling natural language content to be generated that more closely aligns with the keywords and template.

An embodiment of the present invention may also provide language translation, where the keywords are in a first natural language and the generated natural language content is in a second different natural language. The present invention embodiment is indifferent to syntactic order of the keywords or the underlying natural language, thereby enabling flexibility in terms of natural languages processed.

An embodiment of the present invention may further generate the keyword vectors by generating word embeddings for each of the keywords, encoding the word embeddings using a machine learning model to produce encoded vector representations of the keywords, and generating the keyword vectors based on the encoded vector representations. The machine learning model is trained to produce the same encoded vector representations for a set of keywords regardless of an order of the keywords. This enables the natural language content generation to be agnostic of the order of the keywords, thereby generating consistent content from the same keywords (regardless of order) that closely aligns with the template.

An embodiment of the present invention may generate the keyword vectors based on the encoded vector representations by applying attention weights to the encoded vector representations of the keywords to produce a keyword vector for a corresponding language tag of the template as a weighted combination of the encoded vector representations. The attention weights indicate importance of individual keywords and are based on the corresponding language tag of the template. This ensures that the template tags influence the attention mechanism and selection of natural language content, and enables the decoder to provide greater flexibility in extracting contextual information from either the keywords or the template, while ensuring that unnecessary attention is not given to the keywords (unless a current step of generation requires additional attention).

An embodiment of the present invention may still further generate the template vectors by generating word embeddings for each of the language tags of the template, encoding the word embeddings for the language tags of the template using a bidirectional recurrent machine learning model, and producing the template vectors based on the encoded word embeddings for the language tags of the template. Each template vector is produced based on adjacent language tags within the template. This enables the template vectors to more accurately capture the template context since adjacent tags are considered for generating the template vectors.

An embodiment of the present invention may also generate one or more words for each language tag of the template by determining for each language tag of the template a probability distribution over a word vocabulary using a recurrent machine learning model, and selecting one or more words from the word vocabulary for a corresponding language tag of the template based on the probability distribution. The word vocabulary is learned from training data during training of the recurrent machine learning model. This enables function words to be introduced in the generated natural language content from the learned word vocabulary, and produces content that more accurately captures the template since previous words are considered for selecting subsequent words.

Transform module 116 ( FIG. 1 ) according to an embodiment of the present invention receives a set of keywords and a template (and optionally part-of-speech (POS) tags for the keywords), and generates natural language content (e.g., a sentence, clause, phrase, etc.) in accordance with the template. The template includes a sequence of part-of-speech (POS) tags for the resulting generated natural language content. For example, a set of keywords may include (PLAYER1, goals, 15), while a template may include (subject, verb, qualifier, object, complement). Resulting natural language content based on the set of keywords and template includes words corresponding to the part-of-speech (POS) tags in the template. With respect to the example set of keywords and template, resulting natural language content may include “PLAYER1 scored total 15 goals in the tournament.” The additional words are selected and inserted based on learning a word vocabulary from training data including example sentences as described below.

Referring to FIG. 3 , transform module 116 according to an embodiment of the present invention processes various data to generate natural language content (e.g., sentences, clauses, phrases, etc.). The data include a set of one or more keywords 305 (e.g., a set of N keywords represented by K=[k 1 , k 2 , k 3 , . . . k N ] as viewed in FIG. 3 ), a set of corresponding unique part-of-speech (POS) tags for the keywords 310 (e.g., a set of U POS tags for the keywords represented by KT=[kt 1 , kt 2 , kt 3 , . . . kt U ] as viewed in FIG. 3 ), and a sequence of part-of-speech (POS) tags forming a template 315 indicating an arrangement for the generated natural language content (e.g., a sequence of M POS tags for the template represented by TT=[tt 1 , tt 2 , tt 3 , . . . tt M ] as viewed in FIG. 3 ). The set of POS tags for the keywords 310 may be provided to transform module 116 , or generated by the transform module from the set of keywords 305 using any conventional natural language processing (NLP) techniques and/or tools.

The generated natural language content (e.g., sentence, clause, phrase, etc.) may be represented as a sequence of words 360 (e.g., a sequence of M words represented by Y=[y 1 , y 2 , . . . yM] as viewed in FIG. 3 ) corresponding to the arrangement indicated by template 315 . The variables N, U, and M indicate the length of (or quantity of elements within) set of keywords 305 , corresponding part-of-speech (POS) tags 310 , template 315 , and sequence of words 360 , and may be any desired integers of the same or different values. By way of example, transform module 116 of FIG. 3 may be configured to generate a corresponding word for each POS tag in template 315 (e.g., the length of the sequence of words 360 is the same as the length of template 315 ). However, the length of (or quantity of words in) the generated sequence of words 360 may vary from the length of (or quantity of POS tags in) template 315 . In other words, the generated sequence of words 360 may include a fewer or greater quantity of words than the POS tags in the template and/or the keywords in set 305 .

Transform module 116 includes a keyword encoder 320 , a template encoder 345 , and a decoder 350 . Keyword encoder 320 generates a transformed or encoded vector for each keyword in the set of keywords 305 (e.g., K=[k 1 , k 2 , k 3 , . . . k N ] as viewed in FIG. 3 ) that captures contextual representations of that keyword. The encoding is accomplished in a manner that is agnostic to an order of the keywords in the set. In other words, the resulting transformed vector for a keyword is basically the same regardless of the order of keywords in the set. This may be accomplished by initially passing the keywords through an embedding layer, and non-linearly transforming the embedding of each keyword independently to produce the transformed vectors. The keywords may be provided to the embedding layer as a series of hot-form vectors, where each hot-form vector includes one keyword in a vector position corresponding to the order of the keyword in the set with remaining vector elements set to a zero or null value. Decoder 350 extracts appropriate context from these transformed vectors through an attention mechanism 354 described below.

Keyword encoder 320 generates a transformed or encoded vector for each keyword in the set of keywords 305<

CLAIMS

Claims ( 20 )

What is claimed is:

1. A method of generating natural language content from a set of keywords in accordance with a template comprising:

generating, via a processor, word embeddings for the keywords;

generating, via the processor, keyword vectors representing a context for the keywords based on the word embeddings for the keywords, wherein the keywords are syntactically unordered and associated with language tags, and wherein the template includes a series of language tags indicating an arrangement for words of the generated natural language content;

generating, via the processor, word embeddings for the series of language tags of the template;

generating, via the processor, template vectors based on the word embeddings for the series of language tags of the template, wherein the template vectors represent a context for the template;

generating, via the processor, word embeddings for the associated language tags of the keywords;

determining, via the processor, contributions from the context for the keywords represented by the keyword vectors and the context for the template represented by the template vectors based on a comparison of the word embeddings of the series of language tags of the template with the word embeddings of the associated language tags of the keywords; and

generating, via a machine learning model of the processor, one or more words for each language tag of the template from a word vocabulary to produce the natural language content based on combined contributions from the context for the keywords represented by the keyword vectors and the context for the template represented by the template vectors, wherein the machine learning model includes a recurrent neural network and the word vocabulary is learned from training data during training of the machine learning model.

2. The method of claim 1 , wherein the language tags of the template and the associated language tags of the keywords include part-of-speech tags.

3. The method of claim 1 , wherein determining contributions comprises:

determining a probability for each language tag of the template indicating a likelihood of that language tag of the template matching one of the associated language tags of the keywords, wherein the probability for a corresponding language tag of the template indicates the contribution for the context of the keywords for generating a word for the corresponding language tag of the template, and wherein a complement of the probability indicates the contribution for the context of the template for generating the word for the corresponding language tag of the template.

4. The method of claim 3 , further comprising:

applying the probability for the corresponding language tag of the template to a keyword vector associated with the corresponding language tag of the template to produce the contribution of the context for the keywords;

applying the complement of the probability for the corresponding language tag of the template to a template vector associated with the corresponding language tag of the template to produce the contribution of the context for the template; and

combining the contributions of the contexts for the keywords and the template to produce the combined contributions.

5. The method of claim 1 , further comprising:

determining the associated language tags for the keywords via a second machine learning model, wherein the second machine learning model is trained with a data set including complete sentences and the complete sentences without function words.

6. The method of claim 1 , wherein the keywords are in a first natural language, and the generated natural language content is in a second different natural language.

7. The method of claim 1 , wherein generating the keyword vectors comprises:

encoding the word embeddings for the keywords using a second machine learning model to produce encoded vector representations of the keywords, wherein the second machine learning model is trained to produce the same encoded vector representations for a corresponding set of keywords regardless of an order of keywords in the corresponding set; and

generating the keyword vectors based on the encoded vector representations.

8. The method of claim 7 , wherein generating the keyword vectors based on the encoded vector representations further comprises:

applying attention weights to the encoded vector representations of the keywords to produce a keyword vector for a corresponding language tag of the template as a weighted combination of the encoded vector representations, wherein the attention weights indicate importance of individual keywords and are based on the corresponding language tag of the template.

9. The method of claim 1 , wherein generating the template vectors comprises:

encoding the word embeddings for the series of language tags of the template using a bidirectional recurrent machine learning model; and

producing the template vectors based on the encoded word embeddings for the series of language tags of the template, wherein each template vector is produced based on adjacent language tags within the template.

10. The method of claim 1 , wherein generating one or more words for each language tag of the template comprises:

determining for each language tag of the template a probability distribution over the word vocabulary using the machine learning model; and

selecting one or more words from the word vocabulary for a corresponding language tag of the template based on the probability distribution.

11. A system for generating natural language content from a set of keywords in accordance with a template comprising:

a processor configured to:

generate word embeddings for the keywords;

generate keyword vectors representing a context for the keywords based on the word embeddings for the keywords, wherein the keywords are syntactically unordered and associated with language tags, and wherein the template includes a series of language tags indicating an arrangement for words of the generated natural language content;

generate word embeddings for the series of language tags of the template;

generate template vectors based on the word embeddings for the series of language tags of the template, wherein the template vectors represent a context for the template;

generate word embeddings for the associated language tags of the keywords;

determine contributions from the context for the keywords represented by the keyword vectors and the context for the template represented by the template vectors based on a comparison of the word embeddings of the series of language tags of the template with the word embeddings of the associated language tags of the keywords; and

generate, via a machine learning model, one or more words for each language tag of the template from a word vocabulary to produce the natural language content based on combined contributions from the context for the keywords represented by the keyword vectors and the context for the template represented by the template vectors, wherein the machine learning model includes a recurrent neural network and the word vocabulary is learned from training data during training of the machine learning model.

12. The system of claim 11 , wherein determining contributions comprises:

determining a probability for each language tag of the template indicating a likelihood of that language tag of the template matching one of the associated language tags of the keywords, wherein the probability for a corresponding language tag of the template indicates the contribution for the context of the keywords for generating a word for the corresponding language tag of the template, and wherein a complement of the probability indicates the contribution for the context of the template for generating the word for the corresponding language tag of the template.

13. The system of claim 11 , wherein the processor is further configured to:

determine the associated language tags for the keywords via a second machine learning model, wherein the second machine learning model is trained with a data set including complete sentences and the complete sentences without function words.

14. The system of claim 11 , wherein generating the keyword vectors comprises:

encoding the word embeddings for the keywords using a second machine learning model to produce encoded vector representations of the keywords, wherein the second machine learning model is trained to produce the same encoded vector representations for a corresponding set of keywords regardless of an order of keywords in the corresponding set; and

generating the keyword vectors based on the encoded vector representations, wherein generating the keyword vectors based on the encoded vector representations further comprises:

applying attention weights to the encoded vector representations of the keywords to produce a keyword vector for a corresponding language tag of the template as a weighted combination of the encoded vector representations, wherein the attention weights indicate importance of individual keywords and are based on the corresponding language tag of the template; and

wherein generating the template vectors comprises:

encoding the word embeddings for the series of language tags of the template using a bidirectional recurrent machine learning model; and

producing the template vectors based on the encoded word embeddings for the series of language tags of the template, wherein each template vector is produced based on adjacent language tags within the template.

15. The system of claim 11 , wherein generating one or more words for each language tag of the template comprises:

determining for each language tag of the template a probability distribution over the word vocabulary using the machine learning model; and

selecting one or more words from the word vocabulary for a corresponding language tag of the template based on the probability distribution.

16. A computer program product for generating natural language content from a set of keywords in accordance with a template, the computer program product comprising one or more computer readable storage media having program instructions collectively stored on the one or more computer readable storage media, the program instructions executable by a processor to cause the processor to:

generate word embeddings for the keywords;

generate keyword vectors representing a context for the keywords based on the word embeddings for the keywords, wherein the keywords are syntactically unordered and associated with language tags, and wherein the template includes a series of language tags indicating an arrangement for words of the generated natural language content;

generate word embeddings for the series of language tags of the template;

generate template vectors based on the word embeddings for the series of language tags of the template, wherein the template vectors represent a context for the template;

generate word embeddings for the associated language tags of the keywords;

determine contributions from the context for the keywords represented by the keyword vectors and the context for the template represented by the template vectors based on a comparison of the word embeddings of the series of language tags of the template with the word embeddings of the associated language tags of the keywords; and

generate, via a machine learning model, one or more words for each language tag of the template from a word vocabulary to produce the natural language content based on combined contributions from the context for the keywords represented by the keyword vectors and the context for the template represented by the template vectors, wherein the machine learning model includes a recurrent neural network and the word vocabulary is learned from training data during training of the machine learning model.

17. The computer program product of claim 16 , wherein determining contributions comprises:

determining a probability for each language tag of the template indicating a likelihood of that language tag of the template matching one of the associated language tags of the keywords, wherein the probability for a corresponding language tag of the template indicates the contribution for the context of the keywords for generating a word for the corresponding language tag of the template, and wherein a complement of the probability indicates the contribution for the context of the template for generating the word for the corresponding language tag of the template.

18. The computer program product of claim 16 , wherein the program instructions further cause the processor to:

determine the associated language tags for the keywords via a second machine learning model, wherein the second machine learning model is trained with a data set including complete sentences and the complete sentences without function words.

19. The computer program product of claim 16 , wherein generating the keyword vectors comprises:

encoding the word embeddings for the keywords using a second machine learning model to produce encoded vector representations of the keywords, wherein the second machine learning model is trained to produce the same encoded vector representations for a corresponding set of keywords regardless of an order of keywords in the corresponding set; and

generating the keyword vectors based on the encoded vector representations, wherein generating the keyword vectors based on the encoded vector representations further comprises:

applying attention weights to the encoded vector representations of the keywords to produce a keyword vector for a corresponding language tag of the template as a weighted combination of the encoded vector representations, wherein the attention weights indicate importance of individual keywords and are based on the corresponding language tag of the template; and

wherein generating the template vectors comprises:

encoding the word embeddings for the series of language tags of the template using a bidirectional recurrent machine learning model; and

producing the template vectors based on the encoded word embeddings for the series of language tags of the template, wherein each template vector is produced based on adjacent language tags within the template.

20. The computer program product of claim 16 , wherein generating one or more words for each language tag of the template comprises:

determining for each language tag of the template a probability distribution over the word vocabulary using the machine learning model; and

selecting one or more words from the word vocabulary for a corresponding language tag of the template based on the probability distribution.

US16/864,253

2020-05-01

2020-05-01

Natural language text generation from a set of keywords using machine learning and templates

Active

2041-03-13

US11586829B2

( en )

Priority Applications (1)

Application Number

Priority Date

Filing Date

Title

US16/864,253

US11586829B2

( en )

2020-05-01

2020-05-01

Natural language text generation from a set of keywords using machine learning and templates

Applications Claiming Priority (1)

Application Number

Priority Date

Filing Date

Title

US16/864,253

US11586829B2

( en )

2020-05-01

2020-05-01

Natural language text generation from a set of keywords using machine learning and templates

Publications (2)

Publication Number

Publication Date

US20210342552A1

US20210342552A1 ( en )

2021-11-04

US11586829B2

true

US11586829B2 ( en )

2023-02-21

Family

ID=78293007

Family Applications (1)

Application Number

Title

Priority Date

Filing Date

US16/864,253

Active

2041-03-13

US11586829B2

( en )

2020-05-01

2020-05-01

Natural language text generation from a set of keywords using machine learning and templates

Country Status (1)

Country

Link

US

( 1 )

US11586829B2

( en )

Cited By (3)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US11803710B1

( en )

*

2023-03-28

2023-10-31

SurgeTech, LLC

Multi-modal machine learning architectures integrating language models and computer vision systems

US20230367976A1

( en )

*

2022-05-12

2023-11-16

Alipay (Hangzhou) Information Technology Co., Ltd.

Model training methods and apparatuses, text processing methods and apparatuses, and computer devices

US12437505B2

( en )

*

2022-10-13

2025-10-07

Adobe Inc.

Generating templates using structure-based matching

Families Citing this family (30)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US11599721B2

( en )

*

2020-08-25

2023-03-07

Salesforce, Inc.

Intelligent training set augmentation for natural language processing tasks

US11934795B2

( en )

2021-01-29

2024-03-19

Oracle International Corporation

Augmented training set or test set for improved classification model robustness

US20230032208A1

( en )

*

2021-07-30

2023-02-02

Oracle International Corporation

Augmenting data sets for machine learning models

US20230129467A1

( en )

*

2021-10-22

2023-04-27

Citrix Systems, Inc.

Systems and methods to analyze audio data to identify different speakers

CN114239568B

( en )

*

2021-11-10

2026-04-14

国网福建省电力有限公司信息通信分公司

Automatic generation method and device for power grid document based on self-adaptive attention mechanism

US11726750B1

( en )

*

2021-11-17

2023-08-15

Outsystems—Software Em Rede, S.A.

Constrained decoding and ranking of language models for code generation

CN114065722A

( en )

*

2021-11-26

2022-02-18

支付宝(杭州)信息技术有限公司

System, method and device for generating transaction report and electronic equipment

CN113918708B

( en )

*

2021-12-15

2022-03-22

深圳市迪博企业风险管理技术有限公司

A method for extracting abstracts

CN114428851A

( en )

*

2021-12-29

2022-05-03

北京百度网讯科技有限公司

Abstract generation method and device, electronic equipment and storage medium

CN114282515B

( en )

*

2022-02-18

2022-07-08

北京语言大学

End-to-end vocabulary limited text generation method based on incremental segment prediction

CN114566156B

( en )

*

2022-02-28

2025-08-01

恒玄科技(上海)股份有限公司

Voice recognition method and device for keywords

CN114611505B

( en )

*

2022-03-07

2025-06-03

云知声智能科技股份有限公司

Non-autoregressive neural machine translation decoding method, device, equipment and storage medium

CN114691866B

( en )

*

2022-03-09

2025-05-30

航空工业信息中心

Text classification method, device, equipment and storage medium for multi-level labels

CN114972848A

( en )

*

2022-05-10

2022-08-30

中国石油大学(华东)

Image semantic understanding and text generation based on fine-grained visual information control network

CN115130461B

( en )

*

2022-05-16

2025-07-18

腾讯科技（深圳）有限公司

Text matching method and device, electronic equipment and storage medium

CN114818641B

( en )

*

2022-05-17

2025-11-25

广州欢聚时代信息科技有限公司

Copywriting template generation methods, devices, equipment, media, and products

US12242568B2

( en )

2022-06-14

2025-03-04

Oracle International Corporation

Guided augmentation of data sets for machine learning models

WO2023244514A1

( en )

*

2022-06-14

2023-12-21

Oracle International Corporation

Guided augmention of data sets for machine learning models

CN115358224B

( en )

*

2022-07-19

2026-04-03

南京航空航天大学

A gradient search attack method based on prompt template

US20240070402A1

( en )

*

2022-08-31

2024-02-29

Viettel Group

Method for factual event detection from online news based on deep learning

CN115687609B

( en )

*

2022-09-06

2025-08-19

北京交通大学

Zero sample relation extraction method based on Prompt multi-template fusion

US12131367B2

( en )

*

2022-10-18

2024-10-29

Wesco Distribution, Inc.

Intelligent product matching based on a natural language query

CN115905510B

( en )

*

2022-11-08

2025-03-21

中图科信数智技术(北京)有限公司

A text summary generation method and system

US20240311424A1

( en )

*

2023-03-17

2024-09-19

Adobe Inc.

Generative artificial intelligence assistant

CN116432663B

( en )

*

2023-06-12

2023-10-13

山东山大鸥玛软件股份有限公司

Controllable diversity professional text generation method and system based on element diagram

US20250232496A1

( en )

*

2024-01-12

2025-07-17

The Boeing Company

Machine learning to generate multimedia visualizations

CN117594038B

( en )

*

2024-01-19

2024-04-02

壹药网科技(上海)股份有限公司

Voice service improvement method and system

CN118427309B

( en )

*

2024-07-03

2024-08-27

云储新能源科技有限公司

Natural language interaction-based energy storage management system parameter extraction method

WO2026059432A1

( en )

*

2024-09-10

2026-03-19

Республиканское Государственное Предприятие На Праве Хозяйственного Ведения &#34;Центр Поддержки Цифрового Правительства&#34; Министерства Цифрового Развития, Инноваций И Аэрокосмической Промышленности Республики Казахстан

Method and system for processing and generating machine-readable text in kazakh using articial intelligence

CN119578411B

( en )

*

2024-11-21

2025-09-30

河海大学

A Chinese text grammatical error correction method combining syntactic information and pre-trained language model

Citations (8)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US6289304B1

( en )

1998-03-23

2001-09-11

Xerox Corporation

Text summarization using part-of-speech

US6658377B1

( en )

2000-06-13

2003-12-02

Perspectus, Inc.

Method and system for text analysis based on the tagging, processing, and/or reformatting of the input text

US20110184727A1

( en )

2010-01-25

2011-07-28

Connor Robert A

Prose style morphing

CN108073576A

( en )

*

2016-11-09

2018-05-25

上海诺悦智能科技有限公司

Intelligent search method, searcher and search engine system

US20180329883A1

( en )

*

2017-05-15

2018-11-15

Thomson Reuters Global Resources Unlimited Company

Neural paraphrase generator

US20190027133A1

( en )

*

2017-11-07

2019-01-24

Intel Corporation

Spoken language understanding using dynamic vocabulary

US10467274B1

( en )

*

2016-11-10

2019-11-05

Snap Inc.

Deep reinforcement learning-based captioning with embedding reward

US20190370396A1

( en )

*

2018-05-31

2019-12-05

Wipro Limited

Method and Device for Identifying Relevant Keywords from Documents

2020

2020-05-01

US

US16/864,253

patent/US11586829B2/en

active

Active

Patent Citations (8)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US6289304B1

( en )

1998-03-23

2001-09-11

Xerox Corporation

Text summarization using part-of-speech

US6658377B1

( en )

2000-06-13

2003-12-02

Perspectus, Inc.

Method and system for text analysis based on the tagging, processing, and/or reformatting of the input text

US20110184727A1

( en )

2010-01-25

2011-07-28

Connor Robert A

Prose style morphing

CN108073576A

( en )

*

2016-11-09

2018-05-25

上海诺悦智能科技有限公司

Intelligent search method, searcher and search engine system

US10467274B1

( en )

*

2016-11-10

2019-11-05

Snap Inc.

Deep reinforcement learning-based captioning with embedding reward

US20180329883A1

( en )

*

2017-05-15

2018-11-15

Thomson Reuters Global Resources Unlimited Company

Neural paraphrase generator

US20190027133A1

( en )

*

2017-11-07

2019-01-24

Intel Corporation

Spoken language understanding using dynamic vocabulary

US20190370396A1

( en )

*

2018-05-31

2019-12-05

Wipro Limited

Method and Device for Identifying Relevant Keywords from Documents

Non-Patent Citations (38)

* Cited by examiner, † Cited by third party

Title

(Diab, M. (Apr. 2009). Second generation AMIRA tools for Arabic processing: Fast and robust tokenization, POS tagging, and base phrase chunking. In 2nd International Conference on Arabic Language Resources and Tools (vol. 110, p. 198).) (Year: 2009).

*

A. Laha, et al., " Scalable Micro-planned Generation of Discourse from Structured Data ", Computational Linguistics, vol. 45, No. 4, 2019, 28 pages.

A. Mishra, et al., " A Modular Architecture for Unsupervised Sarcasm Generation ", Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pp. 6144-6154, Hong Kong, China, Nov. 3-7, 2019, 11 pages.

A. Vaswani, et al., " Attention Is All You Need ", 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA, Dec. 4-9, 2017, 11 pages.

Agarwal, V. Aggarwal, A. R. Akula, G. B. Dasgupta and G. Sridhara, " Automatic problem extraction and analysis from unstructured text in IT tickets, " in IBM Journal of Research and Development, vol. 61, No. 1, pp. 4:41-4:52, Jan. 1-Feb. 2017, doi: 10.1147/JRD.2016.2629318. (Year: 2017).

*

Bird et al., " Categorizing and Tagging Words ", Sep. 4, 2019, from Natural Language Processing with Python, https://www.nltk.org/book/ch05.html, 29 pages.

Brownlee. (Oct. 21, 2019). A Gentle Introduction to Cross-Entropy for Machine Learning. Machinelearningmastery.com. Retrieved Apr. 25, 2022, from https://machinelearningmastery.com/cross-entropy-for-machine-learning/ (Year: 2019).

*

Chin-Yew Lin, " ROUGE: A Package for Automatic Evaluation of Summaries ", https://www.aclweb.org/anthology/W04-1013/, 2004, 8 pages.

D. Bahdanau, et al., " Neural Machine Translation by Jointly Learning to Align and Translate ", Published as a conference paper at ICLR 2015, arXiv:1409.0473v7 [cs.CL], May 19, 2016, 15 pages.

E. Reiter, et al., " Building Natural Language Generation Systems ", Computational Linguistics, vol. 27, No. 2, Cambridge University Press (Studies in natural language processing), 2000, xxi+248 pp; hardbound, ISBN 0-521-62036-8, 3 pages.

G. Klein, et al., " OpenNMT: Open-Source Toolkit for Neural Machine Translation ", arXiv:1701.02810v2 [cs.CL], Mar. 6, 2017, 6 pages.

H. Jhamtani, et al., " Shakespearizing Modern Language Using Copy-Enriched Sequence-to-Sequence Models ", arXiv.1707.01161v2 [cs.CL], Jul. 20, 2017, 10 pages.

J. Chung, et al., " Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling ", arXiv:1412.3555v1 [cs.NE], Dec. 11, 2014, 9 pages.

J. Ficler, et al., " Controlling Linguistic Style Aspects in Neural Language Generation ", arXiv.1707.02633v1 [cs.CL], Jul. 9, 2017, 14 pages.

J. Kabbara, et al., " Stylistic Transfer in Natural Language Generation Systems Using Recurrent Neural Networks ", Proceedings of EMNLP 2016 Workshop on Uphill Battles in Language Processing: Scaling Early Achievements to Robust Methods, pp. 43-47, Austin, TX, Nov. 5, 2016, 5 p.

J. Wieting, et al., " PARANMT-50M: Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations ", arXiv:1711.05732v2 [cs.CL], Apr. 20, 2018, 15 pages.

K. Papineni, et al., " BLEU: a Method for Automatic Evaluation of Machine Translation ", Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), Philadelphia, Jul. 2002, 8 pages.

K. Uchimoto, et al., " Text Generation from Keywords ", COLING '02: Proceedings of the 19th international conference on Computational linguistics—vol. 1, Aug. 2002, https://doi.org/10.3115/1072228.1072292, 7 pages.

Kolokas, Nikolaos, et al., " Keywords-To-Text Synthesis Using Recurrent Neural Network, " IFIP International Conference on Artificial Intelligence Applications and Innovations, pp. 85-96, Springer, Cham, 2018, 12 pages.

L. Yu, et al., " SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient ", Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (AAAI-17), Feb. 4-9, 2017, San Francisco, CA, 7 pages.

M. Chen, et al., " Controllable Paraphrase Generation with a Syntactic Exemplar ", arXiv:1906.00565v1 [cs.CL], Jun. 3, 2019, 13 pages.

M. Gupta, " NLP | WordNet for tagging ", GeeksforGeeks, https://www.geeksforgeeks.org/nlp-wordnet-for-tagging/, downloaded from the internet on Apr. 20, 2020, 5 pages.

M. Iyyer, et al., " Adversarial Example Generation with Syntactically Controlled Paraphrase Networks ", arXiv:1804.06059v1 [cs.CL], Apr. 17, 2018, 11 pages.

O. Vinyals, et al., " Show and Tell: A Neural Image Caption Generator ", https://www.cv-foundation.org/openaccess/content_cvpr_2015/html/Vinyals_Show_and_Tell_2015_CVPR_paper.html, The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, 9 pages.

S. Banerjee, et al., " METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments ", Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translatior and/or Summarization, pp. 65-72, Ann Arbor, Jun. 2005, 8 pages.

S. Ghosh, et al., " Affect-LM: A Neural Language Model for Customizable Affective Text Generation ", arXiv:1704.06851v1 [cs.CL], Apr. 22, 2017, 9 pages.

S. Surya, et al., " Unsupervised Neural Text Simplification ", arXiv:1810.07931v6 [cs.CL], Aug. 21, 2019, 11 pages.

S. Wiseman, et al., " Challenges in Data-to-Document Generation ", arXiv:1707.8052v1 [cs.CL], Jul. 25, 2017, 13 pages.

W. Wang, et al., " Toward unsupervised text content manipulation ", arXiv:1901.09501v2 [cs.CL], Feb. 8, 2019, 12 pages.

Wikipedia, " F1 score ", https://en.wikipedia.org/wiki/F1_score, downloaded from the internet on Apr. 20, 2020, 3 pages.

Wikipedia, " Function word ", https://en.wikipedia.org/wiki/Function_word, downloaded from the internet on Apr. 20, 2020, 3 pages.

Wikipedia, " Part-of-speech tagging ", https://en.wikipedia.org/wiki/Part-of-speech_tagging, downloaded from the internet on Apr. 20, 2020, 7 pages.

X. Schmaltz, et al., " Word OrderingWithout Syntax ", Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2319-2324, Austin, Texas, Nov. 1-5, 2016, https://www.aclweb.org/anthology/D16-1255.pdf, 6 pages.

X. Zhang, et al., " Chinese Poetry Generation with Recurrent Neural Networks ", Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 670-680, Oct. 25-29, 2014, Doha, Qatar, 11 pages.

Z. Hu, et al., " Controllable Text Generation ", arXiv:1703.00955v1 [cs.LG], Mar. 2, 2017, 10 pages.

Z. Hu, et al., " Toward Controlled Generation of Text ", Proceedings of the 34 th International Conference on Machine Learning, Sydney, Australia, PMLR 70, 2017, 10 pages.

Z. Li, et al., " Paraphrase Generation with Deep Reinforcement Learning ", arXiv:1711.00279v3 [cs.CL], Aug. 23, 2018, 14 pages.

Z. Song, et al., " A Neural Network Model for Chinese Sentence Generation with Key Word ", 2019 IEEE 9th International Conference on Electronics Information and Emergency Communication (ICEIEC), Jul. 12-14, 2019, Beijing, China, 4 pages.

Cited By (5)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20230367976A1

( en )

*

2022-05-12

2023-11-16

Alipay (Hangzhou) Information Technology Co., Ltd.

Model training methods and apparatuses, text processing methods and apparatuses, and computer devices

US12561534B2

( en )

*

2022-05-12

2026-02-24

Alipay (Hangzhou) Information Technology Co., Ltd.

Model training methods and apparatuses, text processing methods and apparatuses, and computer devices

US12437505B2

( en )

*

2022-10-13

2025-10-07

Adobe Inc.

Generating templates using structure-based matching

US11803710B1

( en )

*

2023-03-28

2023-10-31

SurgeTech, LLC

Multi-modal machine learning architectures integrating language models and computer vision systems

US11900067B1

( en )

*

2023-03-28

2024-02-13

SurgeTech, LLC

Multi-modal machine learning architectures integrating language models and computer vision systems

Also Published As

Publication number

Publication date

US20210342552A1

( en )

2021-11-04

Similar Documents

Publication

Publication Date

Title

US20210342552A1

( en )

2021-11-04

Natural language text generation from a set of keywords using machine learning and templates

US12136037B2

( en )

2024-11-05

Non-transitory computer-readable storage medium and system for generating an abstractive text summary of a document

Pakray et al.

2025

Natural language processing applications for low-resource languages

Adelia et al.

2019

Indonesian abstractive text summarization using bidirectional gated recurrent unit

US12299389B2

( en )

2025-05-13

Proficiency and native language-adapted grammatical error correction

Aliero et al.

2023

Systematic review on text normalization techniques and its approach to non-standard words

CN101002198A

( en )

2007-07-18

Systems and methods for spell correction of non-roman characters and words

Rijhwani et al.

2021

Lexically aware semi-supervised learning for OCR post-correction

ElSabagh et al.

2025

A comprehensive survey on Arabic text augmentation: approaches, challenges, and applications

K. Nambiar et al.

2023

Abstractive summarization of text document in Malayalam language: enhancing attention model using POS tagging feature

Inbasekaran et al.

2021

Using transfer learning to contextually optimize optical character recognition (OCR) output and perform new feature extraction on a digitized cultural and historical dataset

Beshirov et al.

2025

Post-OCR text correction for Bulgarian historical documents

García-Martínez et al.

2017

Neural machine translation by generating multiple linguistic factors

Li et al.

2021

HG-news: News headline generation based on a generative pre-training model

Yadav et al.

2023

Different models of transliteration-a comprehensive review

Ahmed et al.

2025

Hybrid text summarization: Integrating extractive and abstractive models for enhanced cross-domain summarization

Lauc et al.

2024

AyutthayaAlpha: A Thai-Latin script transliteration transformer

Philip et al.

2022

A Brief Survey on Natural Language Processing Based Text Generation and Evaluation Techniques

Sharma et al.

2018

Language identification for hindi language transliterated text in roman script using generative adversarial networks

Babhulgaonkar et al.

2022

Experimenting with factored language model and generalized back-off for Hindi

Anbukkarasi et al.

2023

Phonetic-based forward online transliteration tool from English to Tamil language

Aydinov et al.

2022

Investigation of automatic part-of-speech tagging using CRF, HMM and LSTM on misspelled and edited texts

Faheem et al.

2025

Part of speech (POS) tagging in Roman Urdu: Datasets and models

Sardarov

2022

Development and design of deep learning-based parts-of-speech tagging system for Azerbaijani language

You et al.

2020

Syllable-based Korean named entity recognition using convolutional neural network

Legal Events

Date

Code

Title

Description

2020-05-01

Related documents

• Generating augmented reality experiences utilizing physical objects to … — Meta Platforms Technologies, Llc (US20260037214A1)#653960
• Virtual reality system using multiple force arrays for a solver — DeepMotion, Inc. (US10657696B2)#653961
• Tracking position of device inside-out for virtual reality interactivity — Sony Interactive Entertainment America Llc (US10535153B2)#653962
• Augmented reality system with spatialized audio tied to user manipulated … — Magic Leap, Inc. (US11231770B2)#653963
• Big data-based autonomous flight drone system and autonomous flight method … — Rgblab Co., Ltd. (US20210390867A1)#653964