ABSTRACT
Abstract
Disclosed herein is computer technology that applies natural language processing (NLP) techniques to training data to generate information used to train a natural language generation (NLG) system to produce output that stylistically resembles the training data. In this fashion, the NLG system can be readily trained with training data supplied by a user so that the NLG system is adapted to produce output that stylistically resembles such training data. In an example, an NLP system detects a plurality of linguistic features in the training data. These detected linguistic features are then aggregated into a specification data structure that is arranged for training the NLG system to produce natural language output that stylistically resembles the training data. Parameters in the specification data structure can be linked to objects in an ontology used by the NLG system to facilitate the training of the NLG system based on the detected linguistic features.
Description
CROSS-REFERENCE AND PRIORITY CLAIM TO RELATED PATENT APPLICATIONS
This patent application is a divisional of U.S. patent application Ser. No. 16/444,649, filed Jun. 18, 2019, and entitled âApplied Artificial Intelligence Technology for Using Natural Language Processing and Concept Expression Templates to Train a Natural Language Generation Systemâ, now U.S. Pat. No. 10,706,236, which claims priority to U.S. provisional patent application Ser. No. 62/691,197, filed Jun. 28, 2018, and entitled âApplied Artificial Intelligence Technology for Using Natural Language Processing to Train a Natural Language Generation Systemâ, the entire disclosures of each of which are incorporated herein by reference.
This patent application is related to (1) U.S. patent application Ser. No. 16/444,689, filed Jun. 18, 2019, and entitled âApplied Artificial Intelligence Technology for Using Natural Language Processing to Train a Natural Language Generation System With Respect to Numeric Style Featuresâ, (2) U.S. patent application Ser. No. 16/444,718, filed Jun. 18, 2019, and entitled âApplied Artificial Intelligence Technology for Using Natural Language Processing to Train a Natural Language Generation System With Respect to Date and Number Textual Features, and (3) U.S. patent application Ser. No. 16/444,748, filed Jun. 18, 2019, and entitled âApplied Artificial Intelligence Technology for Using Natural Language Processing to Train a Natural Language Generation Systemâ, the entire disclosures of each of which are incorporated herein by reference.
INTRODUCTION
There is an ever-growing need in the art for improved natural language generation (NLG) technology. However, one of the challenges for developing a robust NLG system as a platform that is to be used by many different users is that each user may have different stylistic preferences regarding how content should be presented in NLG output. For example, Company A and Company B may both use the same underlying NLG technology to produce performance reports about its salespeople, but each may have different stylistic preferences for such reports. However, configuring the NLG system to differentiate its stylistic output for different users is a challenging task technologically.
As a technical advance in the art, the inventors disclose the use of natural language processing (NLP) techniques that are applied to training data to generate information used to train an NLG system to produce output that stylistically resembles the training data. In other words, the NLP techniques discussed herein permit an NLG system to be trained via automated learning techniques in a manner that will satisfy a user who wants the NLG system to âwrite like meâ.
NLG is a subfield of artificial intelligence (AI) concerned with technology that produces language as output on the basis of some input information or structure (e.g., where the input constitutes data about a situation to be analyzed and expressed in natural language).
NLP is a subfield of AI concerned with technology that interprets natural language inputs, and natural language understanding (NLU) is a subfield of NLP concerned with technology that draws conclusions on the basis of some input information or structure.
A computer system that trains an NLG system to flexibly produce style-specific natural language outputs needs to combine these difficult areas of NLG and NLP/NLU so that the system not only understands the deeper meanings and styles that underlie the training data but also is able to translate these stylistic understandings and meanings into a configuration that is usable by the NLG system. The inventors disclose herein a number of technical advances with respect to the use of NLP technology to train an NLG system.
For example, the inventors disclose an NLP system that is able to detect a plurality of linguistic features in the training data, wherein the training data comprises a plurality of words arranged in a natural language. These detected linguistic features are then aggregated into a specification data structure that is arranged for training an NLG system to produce natural language output that stylistically resembles the training data. This specification data structure can comprise a machine-readable representation of the detected linguistic features. Parameters in the specification data structure can be linked to objects in an ontology used by the NLG system to facilitate the training of the NLG system based on the detected linguistic features.
The detected linguistic features can include numeric styles in the training data as well as date and number textual expressions in the training data. Examples of such linguistic features include decimal precision features, decimal separator features, digit grouping delimiter features, currency symbol features, day expressions features, month expression features, currency expressions features, and numeric expressions features.
The detected linguistic features can also include ontological vocabulary derived from the training data. Such ontological vocabulary can be used to train the NLG system to use expressions for ontological objects known by the NLG system that match up with how those ontological objects are expressed in the training data.
In a particularly powerful example embodiment discussed herein, the detected linguistic features can include concept expression templates that model how a concept is expressed in the training data. Examples of concepts that can be modeled in this fashion from the training data include change concepts, compare concepts, driver concepts, and rank concepts. In an example embodiment, to detect and extract such concept expression templates from the training data, the training data can be scanned for the presence of one or more anchor words, where each anchor word is associated with a concept understood by the system. If an anchor word is present in the training data, the system can then process the training data to extract an expression template that models how the concept associated with the present anchor word is discussed in the training data. NLP parsing can be applied to the training data and linkages to NLG ontologies can be employed to facilitate this concept expression template extraction.
Further still, the inventors disclose how user interfaces can be employed that permit a user to selectively control which of the detected linguistic features will be used to train the NLG system. Such user interfaces can also permit users to create concept expression templates âon the flyâ in response to text inputs from the user (e.g., where a user types in a sentence from which a concept expression template is to be extracted).
Through these and other features, example embodiments of the invention provide significant technical advances in the NLP and NLG arts by harnessing computer technology to improve how natural language training data is processed to train an NLG system for producing natural language outputs in a manner that stylistically resembles the training data.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 discloses an example AI computer system in accordance with an example embodiment.
FIG. 2 discloses an example process flow for NLP-based training of an NLG system.
FIG. 3 shows an example process flow for extracting linguistic features from training data and aggregating the extracted linguistic features into a specification data structure.
FIG. 4 A discloses an example process flow for entity identification in support of NLP in accordance with an example embodiment.
FIG. 4 B discloses an example prefix tree that can be used for identifying entities in training data.
FIG. 5 shows an example process flow for detecting and extracting concept expression templates from training data.
FIGS. 6 A- 6 D show examples of parse tree structures at various stages of the FIG. 5 process flow.
FIG. 6 E shows an example process flow for transforming and/or tagging tokens in a parse tree with NLG-compatible labels.
FIG. 7 shows another example schematic for end-to-end detection and extraction of concept expression templates from training data.
FIGS. 8 A- 8 I show examples of parse tree structures and other text examples at various stages of the FIG. 7 process.
FIGS. 9 A- 9 J show examples of different portions of a specification data structure that can be produced by the NLP training system.
FIGS. 10 A- 10 G show various example of user interfaces for controlling and operating the training system.
FIG. 10 H shows an example narrative produced by the trained NLG system.
FIGS. 11 A and 11 B show example specification data structures that illustrate how user inputs via user interfaces can modify a base specification data structure.
FIGS. 12 A and 12 B show example commands for an HTTP API with respect to an example embodiment.
DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS
FIG. 1 shows an example computer system 100 in accordance with an example embodiment. The computer system 100 may comprise a training data gateway 102 that links an artificial intelligence (AI) platform 104 with one or more sources of input training data such as document(s) 120 , text input 122 , and/or speech input 124 . The training data gateway 102 then provides such information to an NLP-based training system 106 as training data 126 . As an example, the training data gateway 102 can receive an upload of one or more documents 120 , where the document(s) 120 serve as training data 126 . The training data gateway 102 can also receive user input in the form of text 122 (such as text input through a graphical user interface) wherein the text 122 serves as training data 126 . The training data gateway 102 can also receive user input in the form of speech 124 , where speech recognition is performed by the gateway 102 to convert the speech into training data 126 . For example, software such as the Transcribe application available from Amazon could be employed to transcribe speech data into text data for processing. The document(s) 120 , text input 122 , and/or speech input 124 can take the form of unstructured data arranged as a plurality of words in a natural language format. The NLP-based training system 106 applies NLP to the training data to determine linguistic styles that are present in the training data and uses the determined linguistic styles to generate configuration data 128 that is used to train the NLG system 108 to produce natural language output that stylistically resembles the training data 126 .
To aid the NLP-based training system 106 and the NLG system 108 in their operations, the NLP-based training system 106 and the NLG system 108 can access supporting data 110 . This supporting data 110 can include the ontological and project data that serves as a knowledge base for the AI platform 104 .
The computer system 100 comprises one or more processors and associated memories that cooperate together to implement the operations discussed herein. The computer system 100 may also include a data source that serves as a repository of data for analysis by the AI platform 104 when processing inputs a
CROSS-REFERENCE AND PRIORITY CLAIM TO RELATED PATENT APPLICATIONS
This patent application is a divisional of U.S. patent application Ser. No. 16/444,649, filed Jun. 18, 2019, and entitled âApplied Artificial Intelligence Technology for Using Natural Language Processing and Concept Expression Templates to Train a Natural Language Generation Systemâ, now U.S. Pat. No. 10,706,236, which claims priority to U.S. provisional patent application Ser. No. 62/691,197, filed Jun. 28, 2018, and entitled âApplied Artificial Intelligence Technology for Using Natural Language Processing to Train a Natural Language Generation Systemâ, the entire disclosures of each of which are incorporated herein by reference.
This patent application is related to (1) U.S. patent application Ser. No. 16/444,689, filed Jun. 18, 2019, and entitled âApplied Artificial Intelligence Technology for Using Natural Language Processing to Train a Natural Language Generation System With Respect to Numeric Style Featuresâ, (2) U.S. patent application Ser. No. 16/444,718, filed Jun. 18, 2019, and entitled âApplied Artificial Intelligence Technology for Using Natural Language Processing to Train a Natural Language Generation System With Respect to Date and Number Textual Features, and (3) U.S. patent application Ser. No. 16/444,748, filed Jun. 18, 2019, and entitled âApplied Artificial Intelligence Technology for Using Natural Language Processing to Train a Natural Language Generation Systemâ, the entire disclosures of each of which are incorporated herein by reference.
INTRODUCTION
There is an ever-growing need in the art for improved natural language generation (NLG) technology. However, one of the challenges for developing a robust NLG system as a platform that is to be used by many different users is that each user may have different stylistic preferences regarding how content should be presented in NLG output. For example, Company A and Company B may both use the same underlying NLG technology to produce performance reports about its salespeople, but each may have different stylistic preferences for such reports. However, configuring the NLG system to differentiate its stylistic output for different users is a challenging task technologically.
As a technical advance in the art, the inventors disclose the use of natural language processing (NLP) techniques that are applied to training data to generate information used to train an NLG system to produce output that stylistically resembles the training data. In other words, the NLP techniques discussed herein permit an NLG system to be trained via automated learning techniques in a manner that will satisfy a user who wants the NLG system to âwrite like meâ.
NLG is a subfield of artificial intelligence (AI) concerned with technology that produces language as output on the basis of some input information or structure (e.g., where the input constitutes data about a situation to be analyzed and expressed in natural language).
NLP is a subfield of AI concerned with technology that interprets natural language inputs, and natural language understanding (NLU) is a subfield of NLP concerned with technology that draws conclusions on the basis of some input information or structure.
A computer system that trains an NLG system to flexibly produce style-specific natural language outputs needs to combine these difficult areas of NLG and NLP/NLU so that the system not only understands the deeper meanings and styles that underlie the training data but also is able to translate these stylistic understandings and meanings into a configuration that is usable by the NLG system. The inventors disclose herein a number of technical advances with respect to the use of NLP technology to train an NLG system.
For example, the inventors disclose an NLP system that is able to detect a plurality of linguistic features in the training data, wherein the training data comprises a plurality of words arranged in a natural language. These detected linguistic features are then aggregated into a specification data structure that is arranged for training an NLG system to produce natural language output that stylistically resembles the training data. This specification data structure can comprise a machine-readable representation of the detected linguistic features. Parameters in the specification data structure can be linked to objects in an ontology used by the NLG system to facilitate the training of the NLG system based on the detected linguistic features.
The detected linguistic features can include numeric styles in the training data as well as date and number textual expressions in the training data. Examples of such linguistic features include decimal precision features, decimal separator features, digit grouping delimiter features, currency symbol features, day expressions features, month expression features, currency expressions features, and numeric expressions features.
The detected linguistic features can also include ontological vocabulary derived from the training data. Such ontological vocabulary can be used to train the NLG system to use expressions for ontological objects known by the NLG system that match up with how those ontological objects are expressed in the training data.
In a particularly powerful example embodiment discussed herein, the detected linguistic features can include concept expression templates that model how a concept is expressed in the training data. Examples of concepts that can be modeled in this fashion from the training data include change concepts, compare concepts, driver concepts, and rank concepts. In an example embodiment, to detect and extract such concept expression templates from the training data, the training data can be scanned for the presence of one or more anchor words, where each anchor word is associated with a concept understood by the system. If an anchor word is present in the training data, the system can then process the training data to extract an expression template that models how the concept associated with the present anchor word is discussed in the training data. NLP parsing can be applied to the training data and linkages to NLG ontologies can be employed to facilitate this concept expression template extraction.
Further still, the inventors disclose how user interfaces can be employed that permit a user to selectively control which of the detected linguistic features will be used to train the NLG system. Such user interfaces can also permit users to create concept expression templates âon the flyâ in response to text inputs from the user (e.g., where a user types in a sentence from which a concept expression template is to be extracted).
Through these and other features, example embodiments of the invention provide significant technical advances in the NLP and NLG arts by harnessing computer technology to improve how natural language training data is processed to train an NLG system for producing natural language outputs in a manner that stylistically resembles the training data.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 discloses an example AI computer system in accordance with an example embodiment.
FIG. 2 discloses an example process flow for NLP-based training of an NLG system.
FIG. 3 shows an example process flow for extracting linguistic features from training data and aggregating the extracted linguistic features into a specification data structure.
FIG. 4 A discloses an example process flow for entity identification in support of NLP in accordance with an example embodiment.
FIG. 4 B discloses an example prefix tree that can be used for identifying entities in training data.
FIG. 5 shows an example process flow for detecting and extracting concept expression templates from training data.
FIGS. 6 A- 6 D show examples of parse tree structures at various stages of the FIG. 5 process flow.
FIG. 6 E shows an example process flow for transforming and/or tagging tokens in a parse tree with NLG-compatible labels.
FIG. 7 shows another example schematic for end-to-end detection and extraction of concept expression templates from training data.
FIGS. 8 A- 8 I show examples of parse tree structures and other text examples at various stages of the FIG. 7 process.
FIGS. 9 A- 9 J show examples of different portions of a specification data structure that can be produced by the NLP training system.
FIGS. 10 A- 10 G show various example of user interfaces for controlling and operating the training system.
FIG. 10 H shows an example narrative produced by the trained NLG system.
FIGS. 11 A and 11 B show example specification data structures that illustrate how user inputs via user interfaces can modify a base specification data structure.
FIGS. 12 A and 12 B show example commands for an HTTP API with respect to an example embodiment.
DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS
FIG. 1 shows an example computer system 100 in accordance with an example embodiment. The computer system 100 may comprise a training data gateway 102 that links an artificial intelligence (AI) platform 104 with one or more sources of input training data such as document(s) 120 , text input 122 , and/or speech input 124 . The training data gateway 102 then provides such information to an NLP-based training system 106 as training data 126 . As an example, the training data gateway 102 can receive an upload of one or more documents 120 , where the document(s) 120 serve as training data 126 . The training data gateway 102 can also receive user input in the form of text 122 (such as text input through a graphical user interface) wherein the text 122 serves as training data 126 . The training data gateway 102 can also receive user input in the form of speech 124 , where speech recognition is performed by the gateway 102 to convert the speech into training data 126 . For example, software such as the Transcribe application available from Amazon could be employed to transcribe speech data into text data for processing. The document(s) 120 , text input 122 , and/or speech input 124 can take the form of unstructured data arranged as a plurality of words in a natural language format. The NLP-based training system 106 applies NLP to the training data to determine linguistic styles that are present in the training data and uses the determined linguistic styles to generate configuration data 128 that is used to train the NLG system 108 to produce natural language output that stylistically resembles the training data 126 .
To aid the NLP-based training system 106 and the NLG system 108 in their operations, the NLP-based training system 106 and the NLG system 108 can access supporting data 110 . This supporting data 110 can include the ontological and project data that serves as a knowledge base for the AI platform 104 .
The computer system 100 comprises one or more processors and associated memories that cooperate together to implement the operations discussed herein. The computer system 100 may also include a data source that serves as a repository of data for analysis by the AI platform 104 when processing inputs and generating outputs. These components can interconnect with each other in any of a variety of manners (e.g., via a bus, via a network, etc.). For example, the computer system 100 can take the form of a distributed computing architecture where one or more processors implement the NLP tasks described herein (see NLP-based training system 106 ), one or more processors implement the NLG tasks described herein (see NLG system 108 ). Furthermore, different processors can be used for NLP and NLG tasks, or alternatively some or all of these processors may implement both NLP and NLG tasks. It should also be understood that the computer system 100 may include additional or different components if desired by a practitioner. The one or more processors may comprise general-purpose processors (e.g., a single-core or multi-core microprocessor), special-purpose processors (e.g., an application-specific integrated circuit or digital-signal processor), programmable-logic devices (e.g., a field programmable gate array), etc. or any combination thereof that are suitable for carrying out the operations described herein. The associated memories may comprise one or more non-transitory computer-readable storage mediums, such as volatile storage mediums (e.g., random access memory, registers, and/or cache) and/or non-volatile storage mediums (e.g., read-only memory, a hard-disk drive, a solid-state drive, flash memory, and/or an optical-storage device). The memory may also be integrated in whole or in part with other components of the system 100 . Further, the memory may be local to the processor(s), although it should be understood that the memory (or portions of the memory) could be remote from the processor(s), in which case the processor(s) may access such remote memory through a network interface. The memory may store software programs or instructions that are executed by the processor(s) during operation of the system 100 . Such software programs can take the form of a plurality of instructions configured for execution by processor(s). The memory may also store project or session data generated and used by the system 100 . The data source can be any source of data, such as one or more databases, file systems, computer networks, etc. which may be part of the memory accessed by the processor(s).
The NLP-based training system 106 can be designed to work end-to-end without any human supervision, although it should be understood that a practitioner may choose to provide a user interface that allows users to review and update the determined linguistic features before they are applied to the NLG system 108 .
FIG. 2 depicts an example process flow for execution by one or more processors that implement the NLP-based training system 106 . At step 200 , a processor ingests the training data 126 . For example, as noted, the training data 126 can take the form of a corpus of documents that are represented by files. The documents can be ingested, converted into raw text strings, and saved for use by the training system 106 (for example, in a relational database as one document per row). The same process can be followed for text inputs and speech inputs, albeit the volume of data will likely be lower in such instances. Also, if desired, multiple files can be ingested at step 200 using techniques such as multi-part, form-encoded HTTP POST.
At step 202 , a processor extracts linguistic features from the ingested training data using a variety of pattern matchers and rule-based NLP heuristics, examples of which are discussed below. Using these techniques, specific linguistic features can be detected in and extracted from each document, and each document can be converted into a data structure (e.g., a JSON data structure) that contains linguistic feature metadata.
At step 204 , a processor aggregates the extracted linguistic features produced from the documents at step 202 by iterating over the document-specific data structures. This can include deriving totals, percentages, grouping, and sorting, which operates to produce a specification data structure (e.g., a JSON specification data structure, which is a machine-readable description of the linguistic features extracted from the ingested training data 126 .
At step 206 , a user interface (e.g., a browser-based graphical user interface (GUI)) can process the specification data structure and present a user with the linguistic features discovered by steps
202 and 204 . Through the user interface, the user can elect to discard any of the discovered linguistic features. In example embodiments, the user can also enter custom sentences into the user interface to add additional ontological vocabulary to the system and/or add concept expressions to the specification. However, as noted above, such user interaction can be omitted if desired by a practitioner.
At step 208 , a processor configures the NLG system 108 based on the specification data structure to thereby train the NLG system 108 to produce language that stylistically resembles the training data 126 . In an example embodiment, a platform-specific applicator can take the JSON specification data structure (and any user preferences) as inputs and update the appropriate configuration within the NLG system 108 .
The NLG system 108 can then use the specification data structure to update its configuration information to control how it produces natural language output. In an example embodiment, the NLG system 108 can produce NLG output about a data set based on defined configurations such as parameterized communication goal statements. An example of NLG technology that can be used as the NLG system 108 is the QUILL⢠narrative generation platform from Narrative Science Inc. of Chicago, IL. Aspects of this technology are described in the following patents and patent applications: U.S. Pat. Nos. 8,374,848, 8,355,903, 8,630,844, 8,688,434, 8,775,161, 8,843,363, 8,886,520, 8,892,417, 9,208,147, 9,251,134, 9,396,168, 9,576,009, 9,697,178, 9,697,197, 9,697,492, 9,720,884, 9,720,899, and 9,977,773, 9,990,337, and 10,185,477; and U.S. patent application Ser. No. 15/253,385 (entitled âApplied Artificial Intelligence Technology for Using Narrative Analytics to Automatically Generate Narratives from Visualization Data, filed Aug. 31, 2016), 62/382,063 (entitled âApplied Artificial Intelligence Technology for Interactively Using Narrative Analytics to Focus and Control Visualizations of Dataâ, filed Aug. 31, 2016), Ser. No. 15/666,151 (entitled âApplied Artificial Intelligence Technology for Interactively Using Narrative Analytics to Focus and Control Visualizations of Dataâ, filed Aug. 1, 2017), Ser. No. 15/666,168 (entitled âApplied Artificial Intelligence Technology for Evaluating Drivers of Data Presented in Visualizationsâ, filed Aug. 1, 2017), Ser. No. 15/666,192 (entitled âApplied Artificial Intelligence Technology for Selective Control over Narrative Generation from Visualizations of Dataâ, filed Aug. 1, 2017), 62/458,460 (entitled âInteractive and Conversational Data Explorationâ, filed Feb. 13, 2017), Ser. No. 15/895,800 (entitled âInteractive and Conversational Data Explorationâ, filed Feb. 13, 2018), 62/460,349 (entitled âApplied Artificial Intelligence Technology for Performing Natural Language Generation (NLG) Using Composable Communication Goals and Ontologies to Generate Narrative Storiesâ, filed Feb. 17, 2017), Ser. No. 15/897,331 (entitled âApplied Artificial Intelligence Technology for Performing Natural Language Generation (NLG) Using Composable Communication Goals and Ontologies to Generate Narrative Storiesâ, filed Feb. 15, 2018), Ser. No. 15/897,350 (entitled âApplied Artificial Intelligence Technology for Determining and Mapping Data Requirements for Narrative Stories to Support Natural Language Generation (NLG) Using Composable Communication Goalsâ, filed Feb. 15, 2018), Ser. No. 15/897,359 (entitled âApplied Artificial Intelligence Technology for Story Outline Formation Using Composable Communication Goals to Support Natural Language Generation (NLG)â, filed Feb. 15, 2018), Ser. No. 15/897,364 (entitled âApplied Artificial Intelligence Technology for Runtime Computation of Story Outlines to Support Natural Language Generation (NLG)â, filed Feb. 15, 2018), Ser. No. 15/897,373 (entitled âApplied Artificial Intelligence Technology for Ontology Building to Support Natural Language Generation (NLG) Using Composable Communication Goalsâ, filed Feb. 15, 2018), Ser. No. 15/897,381 (entitled âApplied Artificial Intelligence Technology for Interactive Story Editing to Support Natural Language Generation (NLG)â, filed Feb. 15, 2018), 62/539,832 (entitled âApplied Artificial Intelligence Technology for Narrative Generation Based on Analysis Communication Goalsâ, filed Aug. 1, 2017), Ser. No. 16/047,800 (entitled âApplied Artificial Intelligence Technology for Narrative Generation Based on Analysis Communication Goalsâ, filed Jul. 27, 2018), Ser. No. 16/047,837 (entitled âApplied Artificial Intelligence Technology for Narrative Generation Based on a Conditional Outcome Frameworkâ, filed Jul. 27, 2018), 62/585,809 (entitled âApplied Artificial Intelligence Technology for Narrative Generation Based on Smart Attributes and Explanation Communication Goalsâ, filed Nov. 14, 2017), Ser. No. 16/183,230 (entitled âApplied Artificial Intelligence Technology for Narrative Generation Based on Smart Attributesâ, filed Nov. 7, 2018), Ser. No. 16/183,270 (entitled âApplied Artificial Intelligence Technology for Narrative Generation Based on Explanation Communication Goalsâ, filed Nov. 7, 2018), 62/632,017 (entitled âApplied Artificial Intelligence Technology for Conversational Inferencing and Interactive Natural Language Generationâ, filed Feb. 19, 2018), Ser. No. 16/277,000 (entitled âApplied Artificial Intelligence Technology for Conversational Inferencingâ, filed Feb. 15, 2019), Ser. No. 16/277,003 (entitled âApplied Artificial Intelligence Technology for Conversational Inferencing and Interactive Natural Language Generationâ, filed Feb. 15, 2019), Ser. No. 16/277,004 (entitled âApplied Artificial Intelligence Technology for Contextualizing Words to a Knowledge Base Using Natural Language Processingâ, filed Feb. 15, 2019), Ser. No. 16/277,006 (entitled âApplied Artificial Intelligence Technology for Conversational Inferencing Using Named Entity Reductionâ, filed Feb. 15, 2019), and Ser. No. 16/277,008 (entitled âApplied Artificial Intelligence Technology for Building a Knowledge Base Using Natural Language Processingâ, filed Feb. 15, 2019); the entire disclosures of each of which are incorporated herein by reference. As explained in the above-referenced and incorporated Ser. No. 16/183,230 patent application, the NLG system 108 can employ a conditional outcome framework to determine the ideas that should be expressed in the narrative that is produced in response to the parameterized communication goal statement. Once the ideas have been generated by the conditional outcome framework of the NLG system 108 , the NLG system can then form these ideas into a narrative using the techniques described in the above-referenced and incorporated Ser. No. 16/183,230 patent application to generate the natural language output. Through the training techniques discussed herein, this natural language output will stylistically resemble the training data by including one or more expressions that are derived from the linguistic features detected in and extracted from the training data.
I. Linguistic Features
FIG. 3 depicts an example architecture for implementing systems
202 and 204 within the training system 106 . A variety of different pattern matchers can be employed to detect and extract linguistic features from the training data 126 . These pattern matchers can be implemented in software code within the training system 106 . In an example embodiment, the pattern matchers can employ regular expression (RegEx) pattern matching where regular expressions are used to define the patterns sought via the matching process. In example embodiments, the training system 106 can include numeric style pattern matchers 300 , date and number pattern matchers 310 , ontological vocabulary pattern matchers 320 , and concept expressions pattern matchers 330 . Examples of each of these will be discussed below. The linguistic features detected and extracted via the pattern matchers can then be aggregated into a specification data structure 370 .
I(A). Numeric Styles
The numeric styles class of linguistic features is concerned with how numeric values are rendered in text. Numeric style pattern matchers 300 can detect and extract different aspects of numeric style expressed by numbers within the training data. The pattern matchers within 300 (examples of which are discussed below) can use regular expressions to define the generalized patterns sought within the training data 126 so that specific instances of the patterns can be recognized. Each pattern matcher can be run against the full text of each document within the training data 126 , and the constituents of each match can be captured for aggregation into the specification data structure 370 .
One or more decision precision pattern matchers 302 can be configured to determine the number of digits contained in the fractional part of a number written in decimal form. For example, the number â5.5â exhibits a single digit of decimal precision, while the number â5.539â exhibits 3 digits of decimal precision. Regular expressions can be employed to detect numbers written in decimal form, and then associated logic can be used to count how many digits are to the right of the decimal.
One or more decimal separator pattern matchers 304 can be configured to determine the character that is used by a number string to separate the integer part of the number from the fractional part of the number. For example, often times a period â.â is used to denote the decimal in a number, but sometimes other characters are used, such as a comma â,â. Regular expressions can be employed to detect numbers written in decimal form, and then associated logic can be used to determine the character being used to separate the integer and fractional portions. For example, the decimal separator pattern matcher 304 can return a period as the decimal separator if the input number is â305.59â, and it can return a comma as the decimal separator if the input number is â305,59â.
One or more digit grouping delimiter pattern matchers 306 can be configured to determine the character that is used by a number string to divide groups of integers in large integers that represent values over 1000. For example, often times a comma â,â is used to separate rightmost groupings of 3 digits in an integer, but sometimes other characters are used, such as a period â.â or white space. Regular expressions can be employed to detect the presence of large integers that represent values over 1000, and then associated logic can be used to determine the character being used to separate the integer portions in groups of 3 digits starting from the rightmost integer digit. For example, the digit grouping delimiter pattern matcher 306 can return a comma as the digit grouping delimiter if the input number is â30,000â; it can return a period as the digit grouping delimiter if the input number is â30.000â; and it can return white space as the digit grouping delimiter if the input number is â30 000â. Disambiguation techniques can be applied to distinguish between numbers that may be ambiguous as to whether they are large integers or small integers with a fractional component following a decimal. As an example, if the decimal separator character is unknown, then the number â5,536â could be interpreted as five thousand five hundred thirty six (if the decimal separator is a period) or it could be interpreted as five point five three six (if the decimal separator is a comma). Possible disambiguation options can include resolving decimal separation and digit grouping hierarchically (e.g., excluding a character found to be a decimal separator from consideration as a digit grouping delimiter), or flagging ambiguous cases for resolution via user input, etc.
One or more currency symbol pattern matchers 308 can be configured to determine the character that is used as a currency symbol within a string that expresses a currency value. Regular expressions can be employed to detect the currency values, and then associated logic can return the character used as the currency symbol (e.g., $, Â¥, â¬, etc.).
I(B). Date and Number Expressions
The date and numbers class of linguistic features is concerned with the form of how numbers are dates are expressed in text. Date and number pattern matchers 310 can detect and extract different aspects of the formats for dates and numbers within the training data. The pattern matchers within 310 (examples of which are discussed below) can use regular expressions to define the generalized patterns sought within the training data 126 so that specific instances of the patterns can be recognized. Each pattern matcher can be run against the full text of each document within the training data 126 , and the constituents of each match can be captured for aggregation into the specification data structure 370 .
One or more day expressions pattern matchers 312 can be configured to determine the textual form in which days of the year are expressed (e.g., âMonday, January 13 th 2018â, â01/13/2018â, â13/01/2018â, â Jan 13, 2018â, etc.). Regular expressions can be employed to detect which of a set of possible day expression patterns are present within the training data.
One or more month expressions pattern matchers 314 can be configured to determine the textual form in which months of the year are expressed (e.g., âJanuary 2018â, âJan. 2018â, â01/2018â, etc.). Regular expressions can be employed to detect which of a set of possible month expression patterns are present within the training data.
One or more currency expressions pattern matchers 316 can be configured to determine the textual form in which currency values are expressed (e.g., â$20â, â20 USDâ, â20 US Dollarsâ, etc.). Regular expressions can be employed to detect which of a set of possible currency expression patterns are present within the training data.
One or more numeric expressions pattern matchers 318 can be configured to determine the textual form in which integer and decimal values are expressed (e.g., âThree Thousand Eighteenâ, â3018â, etc.). Regular expressions can be employed to detect which of a set of possible numeric expression patterns are present within the training data.
I(C). Ontological Vocabulary
The ontological vocabulary class of linguistic features is concerned with the words used to represent ontological entities and relationships within the training data. Different information domains might refer to the same notional entity using different lexicons (e.g., Company A might refer to sales personnel as âsalespeopleâ while Company B might refer to sales personnel as âsales associatesâ). The ontological vocabulary pattern matchers 320 can use data accessible to the underlying NLG system (e.g., supporting data 110 ) to automatically detect ontologically-significant words, particularly nouns and verbs. For example, the ontological vocabulary pattern matchers 320 can leverage an ontology used by the NLG system 108 , which can contain a rich ontology that may include human-readable labels and linguistic expression forms that span one or more domains. Other data sources that can be tapped can include data sources that contain named instances of ontological entities, as well as name attribute values related to known entities. Although specific named instances may not have any relevance to vocabulary features and NLG expressions, they can help disambiguate relationship and/or attribute words. Such data sources can be used to build a text search index that maps specific words back to their corresponding ontological entities, where the text search index is for use by the ontological vocabulary pattern matchers 320 . The system can build the index by traversing all nodes in the ontology as well as all fields in the underlying data sources via a data access layer for the training system 106 .
As an example, consider the following ontology:
Entity: salesperson
Expressions: salesperson, account executive
Entity: sale
Expressions: sale, transaction, auction
Relationship: sells
Participating Entities: salesperson, sale
Expressions: sells, achieves, earns
As well as the following dataset, in tabular form:
salesperson sales region year
Aaron Young 50000
East
2018
Daisy Bailey 51000
West
2018
Once the data above is loaded into the system, the ontological vocabulary pattern matchers 320 can extract vocabulary features and infer preferences from any of the following examples of unstructured text:
âIn 2018, the top account executive was Tom Reynolds, with a total of 56,000â.
Identified: âaccount executiveâ
Result: Express salesperson entities as âaccount executiveâ
âIn 2018, Aaron Young achieved 50,000 transactionsâ
Identified: âAaron Youngâ, âachievedâ, âtransactionsâ
Result: Express relationship of sales+salespeople as âachieveâ
FIG. 4 A discloses an example process flow for performing ontological vocabulary pattern matching. As used herein, the term ânamed entityâ refers to any ontological or data atom that the NLP system 106 recognizes in training data. As such, it should be understood that the term named entity refers to more than just the entities that are described as part of an ontology 410 within the supporting data 110 . Examples of different types of named entities can include entity types (e.g., salesperson), entity instances (e.g., John, who is an instance of a salesperson), attributes (e.g., sales, which are an attribute of a salesperson), attribute values, timeframes, relationship types, relationships, qualifiers, outcomes, entity bindings, and predicate bindings.
At step 400 of FIG. 4 A , the system builds a tree structure that can be used for recognizing named entities in the training data (e.g., the sentences of a training document or other training input), for example a prefix tree. This tree can pull information from the knowledge base such as the sources shown in FIG. 4 A , which may include an ontology 410 , project data 412 , linguistic/ deictic context 414 , and general knowledge 416 .
The ontology 410 can be the ontology for a data set addressed by the message, an example of such an ontology is described in the above-referenced and incorporated Ser. No. 16/183,230 patent application.
The project data 412 represents the data set that serves as a project-specific knowledge base. For example, the project data 412 can be the sales data for the salespeople of a company. Thus, the project data 412 may include a number of entity instances and attribute values for the entity types and attributes of the ontology 410 .
The deictic context 414 can be a data structure that maps referring terms such as pronouns and demonstratives in the training data to specific named entities in the supporting data 110 . This linguistic/deictic context can help the system know how to map referring terms such as pronouns that are mentioned in the training data to specific entities that are mentioned in the training data. An example of technology that can be used to build such a linguistic/deictic context is described in (1) U.S. patent application 62/612,820, filed Jan. 2, 2018, and entitled âContext Saliency-Based Deictic Parser for Natural Language Generation and Natural Language Processingâ, (2) U.S. patent application Ser. No. 16/233,746, filed Dec. 27, 2018, and entitled âContext Saliency-Based Deictic Parser for Natural Language Generationâ, and (3) U.S. patent application Ser. No. 16/233,776, filed Dec. 27, 2018, and entitled âContext Saliency-Based Deictic Parser for Natural Language Processingâ, the entire disclosures of each of which are incorporated herein by reference.
The general knowledge 416 can be a data structure that identifies the words that people commonly use to describe data and timeframes (e.g., âhighestâ, etc.).
Step 400 can operate to read through these data sources and extract each unique instance of a named entity that is found to be present in the data sources, and build the prefix tree that allows the system to later recognize these named entities in the words of the training data and then map those named entities to elements in the ontology 410 , project data 412 , deictic context 414 , and/or general knowledge that are understood by the system. Also, if desired by a practitioner, it should be understood that step 400 can be performed as a pre-processing step that happens before any training data is received by the NLP training system 106 .
FIG. 4 B shows a simple example of a prefix tree that can be built as a result of step 400 . It should be understood that for many projects, the prefix tree would be much larger. In this example, it can be seen that the name âAaron Youngâ was found in the knowledge base of data sources as an entity instance, the word âgenerateâ was found in the knowledge base of data sources as an attribute of sales value, the pronoun âheâ was found to be contextually relevant to the entity instance of Aaron Young, and so on for other named entities as shown by FIG. 4 B . Given that the ontology 410 may include a variety of different expressions for ontological elements (as described in the above-referenced and incorporated Ser. No. 16/183,230 patent application), it should be understood that the prefix tree can be highly robust at recognizing the meaning of a large number of words within the context of a training data set. For example, expressions such as âsalesâ, âsellsâ, âdealsâ, âmovesâ, âtransactionsâ, etc. can be linked to an attribute such as the sales of a salesperson to allow the system to recognize a wide variety of words in training data that relates to sales data. In general, it can be expected that (1) nouns will often map to entity types, entity instances, characterizations, attributes, and qualifiers, (2) verbs will often map to attributes and relationships, (3) adjectives will often map to qualifiers and characterizations, and (4) prepositions will often map to relationships; however this need not always be the case and will depend on the nature of the data sources accessed by step 400 .
Then, step 402 maps words in the training data to named entities in the prefix tree. Thus, if the word âAaronâ appears in the training data, this can be recognized and mapped via the prefix tree to the entity instance of Aaron Young, and if the word âgenerateâ appears in the training data, this can be recognized and mapped via the prefix tree to the attribute of sales value.
I(D). Concept Expressions
The concept expressions class of linguistic features is
CLAIMS
Claims ( 33 )
What is claimed is:
1. A method of training a natural language generation system, the method comprising:
determining a prefix tree via a processor based on a training data source that includes a plurality of natural language sentences, the prefix tree linking one or more named entities included in the training data source to one or more attributes in the training data source;
determining a plurality of concepts via the processor based on the training data source;
parsing the plurality of natural language sentences via the processor based on an ontology to determine a plurality of parse tree structures representing the plurality of natural language sentences, the ontology including an ontological vocabulary class of linguistic features identifying words used to represent ontological entities and relationships within the training data source;
determining a plurality of concept expression templates via the processor by collapsing and parameterizing the plurality of parse tree structures based on the prefix tree, the ontology, and the plurality of concepts, the plurality of concept expression templates modeling how the plurality of natural language sentences express the plurality of concepts as applied to the named entities and the corresponding attributes; and
training the natural language generation system based on the plurality of concept expression templates to generate natural language output expressing a concept of the plurality of concepts about an input data set.
2. The method recited in claim 1 , the method further comprising:
receiving one or more of the plurality of natural language sentences as user input through a user interface.
3. The method recited in claim 1 , the method further comprising:
linking a plurality of parameters included in the concept expression templates to a plurality of objects in the ontology used by the natural language generation system to produce the natural language output.
4. The method recited in claim 1 , the method further comprising:
receiving the plurality of natural language sentences as part of a plurality of documents in a document corpus.
5. The method recited in claim 1 , wherein each of the plurality of concept expression templates correspond with a document including the plurality of natural language sentences.
6. The method recited in claim 1 , wherein the plurality of concepts are determined by analyzing the prefix tree.
7. The method recited in claim 1 , wherein determining the plurality of concepts comprises processing the plurality of natural language sentences against a plurality of pattern matchers to identify a plurality of matches, each of the plurality of matches matching a respective sentence of the plurality of natural language sentences to a pattern matcher of the plurality of pattern matchers.
8. The method recited in claim 7 , wherein the plurality of pattern matchers has associated concepts and wherein the plurality of concepts correspond to the associated concepts.
9. The method recited in claim 8 , wherein the plurality of pattern matchers includes a pattern matcher associated with a change concept, a compare concept, a driver concept, or a rank concept.
10. The method recited in claim 7 , wherein the plurality of pattern matchers is associated with a plurality of anchor words such that that different pattern matchers are associated with different anchor words.
11. The method recited in claim 10 , wherein determining the plurality of concepts comprises matching words in the plurality of natural language sentences against the anchor words.
12. One or more non-transitory computer readable media having instructions stored thereon for performing a method of training a natural language generation system, the method comprising:
determining a prefix tree via a processor based on a training data source that includes a plurality of natural language sentences, the prefix tree linking one or more named entities included in the training data source to one or more attributes in the training data source;
determining a plurality of concepts via the processor based on the training data source;
parsing the plurality of natural language sentences via the processor based on an ontology to determine a plurality of parse tree structures representing the plurality of natural language sentences, the ontology including an ontological vocabulary class of linguistic features identifying words used to represent ontological entities and relationships within the training data source;
determining a plurality of concept expression templates via the processor by collapsing and parameterizing the plurality of parse tree structures based on the prefix tree, the ontology, and the plurality of concepts, the plurality of concept expression templates modeling how the plurality of natural language sentences express the plurality of concepts as applied to the named entities and the corresponding attributes; and
training the natural language generation system based on the plurality of concept expression templates to generate natural language output expressing a concept of the plurality of concepts about an input data set.
13. The one or more non-transitory computer readable media recited in claim 12 , the method further comprising:
receiving one or more of the plurality of natural language sentences as user input through a user interface.
14. The one or more non-transitory computer readable media recited in claim 12 , the method further comprising:
linking a plurality of parameters included in the concept expression templates to a plurality of objects in the ontology used by the natural language generation system to produce the natural language output.
15. The one or more non-transitory computer readable media recited in claim 12 , the method further comprising:
receiving the plurality of natural language sentences as part of a plurality of documents in a document corpus.
16. The one or more non-transitory computer readable media recited in claim 12 , wherein each of the plurality of concept expression templates correspond with a document including the plurality of natural language sentences.
17. The one or more non-transitory computer readable media recited in claim 12 , wherein the plurality of concepts are determined by analyzing the prefix tree.
18. The one or more non-transitory computer readable media recited in claim 12 , wherein determining the plurality of concepts comprises processing the plurality of natural language sentences against a plurality of pattern matchers to identify a plurality of matches, each of the plurality of matches matching a respective sentence of the plurality of natural language sentences to a pattern matcher of the plurality of pattern matchers.
19. The one or more non-transitory computer readable media recited in claim 18 , wherein the plurality of pattern matchers has associated concepts and wherein the plurality of concepts correspond to the associated concepts.
20. The one or more non-transitory computer readable media recited in claim 19 , wherein the plurality of pattern matchers includes a pattern matcher associated with a change concept, a compare concept, a driver concept, or a rank concept.
21. The one or more non-transitory computer readable media recited in claim 18 , wherein the plurality of pattern matchers is associated with a plurality of anchor words such that that different pattern matchers are associated with different anchor words.
22. The one or more non-transitory computer readable media recited in claim 21 , wherein determining the plurality of concepts comprises matching words in the plurality of natural language sentences against the anchor words.
23. A system including a processor and memory, the system configured to train a natural language generation system by performing a method comprising:
determining a prefix tree via the processor based on a training data source that includes a plurality of natural language sentences, the prefix tree linking one or more named entities included in the training data source to one or more attributes in the training data source;
determining a plurality of concepts via the processor based on the training data source;
parsing the plurality of natural language sentences via the processor based on an ontology to determine a plurality of parse tree structures representing the plurality of natural language sentences, the ontology including an ontological vocabulary class of linguistic features identifying words used to represent ontological entities and relationships within the training data source;
determining a plurality of concept expression templates via the processor by collapsing and parameterizing the plurality of parse tree structures based on the prefix tree, the ontology, and the plurality of concepts, the plurality of concept expression templates modeling how the plurality of natural language sentences express the plurality of concepts as applied to the named entities and the corresponding attributes; and
training the natural language generation system based on the plurality of concept expression templates to generate natural language output expressing a concept of the plurality of concepts about an input data set.
24. The system recited in claim 23 , wherein the system is further configured to receiveone or more of the plurality of natural language sentences as user input through a user interface.
25. The system recited in claim 23 , wherein the system is further configured to link a plurality of parameters included in the concept expression templates to a plurality of objects in the ontology used by the natural language generation system to produce the natural language output.
26. The system recited in claim 23 , wherein the system is further configured to receive the plurality of natural language sentences as part of a plurality of documents in a document corpus.
27. The system recited in claim 23 , wherein each of the plurality of concept expression templates correspond with a document including the plurality of natural language sentences.
28. The system recited in claim 23 , wherein the plurality of concepts are determined by analyzing the prefix tree.
29. The system recited in claim 23 , wherein determining the plurality of concepts comprises processing the plurality of natural language sentences against a plurality of pattern matchers to identify a plurality of matches, each of the plurality of matches matching a respective sentence of the plurality of natural language sentences to a pattern matcher of the plurality of pattern matchers.
30. The system recited in claim 29 , wherein the plurality of pattern matchers has associated concepts and wherein the plurality of concepts correspond to the associated concepts.
31. The system recited in claim 30 , wherein the plurality of pattern matchers includes a pattern matcher associated with a change concept, a compare concept, a driver concept, or a rank concept.
32. The system recited in claim 29 , wherein the plurality of pattern matchers is associated with a plurality of anchor words such that that different pattern matchers are associated with different anchor words.
33. The system recited in claim 32 , wherein determining the plurality of concepts comprises matching words in the plurality of natural language sentences against the anchor words.
US16/916,248
2018-06-28
2020-06-30
Applied artificial intelligence technology for using natural language processing and concept expression templates to train a natural language generation system
Active
US11989519B2
( en )
Priority Applications (1)
Application Number
Priority Date
Filing Date
Title
US16/916,248
US11989519B2
( en )
2018-06-28
2020-06-30
Applied artificial intelligence technology for using natural language processing and concept expression templates to train a natural language generation system
Applications Claiming Priority (3)
Application Number
Priority Date
Filing Date
Title
US201862691197P
2018-06-28
2018-06-28
US16/444,649
US10706236B1
( en )
2018-06-28
2019-06-18
Applied artificial intelligence technology for using natural language processing and concept expression templates to train a natural language generation system
US16/916,248
US11989519B2
( en )
2018-06-28
2020-06-30
Applied artificial intelligence technology for using natural language processing and concept expression templates to train a natural language generation system
Related Parent Applications (1)
Application Number
Title
Priority Date
Filing Date
US16/444,649
Division
US10706236B1
( en )
2018-06-28
2019-06-18
Applied artificial intelligence technology for using natural language processing and concept expression templates to train a natural language generation system
Publications (2)
Publication Number
Publication Date
US20200334418A1
US20200334418A1 ( en )
2020-10-22
US11989519B2
true
US11989519B2 ( en )
2024-05-21
Family
ID=71408486
Family Applications (5)
Application Number
Title
Priority Date
Filing Date
US16/444,689
Active
2039-09-06
US11232270B1
( en )
2018-06-28
2019-06-18
Applied artificial intelligence technology for using natural language processing to train a natural language generation system with respect to numeric style features
US16/444,748
Active
US11042713B1
( en )
2018-06-28
2019-06-18
Applied artificial intelligence technology for using natural language processing to train a natural language generation system
US16/444,718
Active
2039-12-21
US11334726B1
( en )
2018-06-28
2019-06-18
Applied artificial intelligence technology for using natural language processing to train a natural language generation system with respect to date and number textual features
US16/444,649
Active
US10706236B1
( en )
2018-06-28
2019-06-18
Applied artificial intelligence technology for using natural language processing and concept expression templates to train a natural language generation system
US16/916,248
Active
US11989519B2
( en )
2018-06-28
2020-06-30
Applied artificial intelligence technology for using natural language processing and concept expression templates to train a natural language generation system
Family Applications Before (4)
Application Number
Title
Priority Date
Filing Date
US16/444,689
Active
2039-09-06
US11232270B1
( en )
2018-06-28
2019-06-18
Applied artificial intelligence technology for using natural language processing to train a natural language generation system with respect to numeric style features
US16/444,748
Active
US11042713B1
( en )
2018-06-28
2019-06-18
Applied artificial intelligence technology for using natural language processing to train a natural language generation system
US16/444,718
Active
2039-12-21
US11334726B1
( en )
2018-06-28
2019-06-18
Applied artificial intelligence technology for using natural language processing to train a natural language generation system with respect to date and number textual features
US16/444,649
Active
US10706236B1
( en )
2018-06-28
2019-06-18
Applied artificial intelligence technology for using natural language processing and concept expression templates to train a natural language generation system
Country Status (1)
Country
Link
US
( 5 )
US11232270B1
( en )
Cited By (2)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US20250225328A1
( en )
*
2024-01-10
2025-07-10
International Business Machines Corporation
Artificial Intelligence Based Metadata Semantic Enrichment
US12518095B1
( en )
*
2022-06-30
2026-01-06
Amazon Technologies, Inc.
Dynamic entity catalog update for natural language processing
Families Citing this family (48)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US8355903B1
( en )
2010-05-13
2013-01-15
Northwestern University
System and method for using data and angles to automatically generate a narrative story
US9208147B1
( en )
2011-01-07
2015-12-08
Narrative Science Inc.
Method and apparatus for triggering the automatic generation of narratives
US11989659B2
( en )
2010-05-13
2024-05-21
Salesforce, Inc.
Method and apparatus for triggering the automatic generation of narratives
US10657201B1
( en )
2011-01-07
2020-05-19
Narrative Science Inc.
Configurable and portable system for generating narratives
US9720899B1
( en )
2011-01-07
2017-08-01
Narrative Science, Inc.
Automatic generation of narratives from data using communication goals and narrative analytics
US10185477B1
( en )
2013-03-15
2019-01-22
Narrative Science Inc.
Method and system for configuring automatic generation of narratives from data
US11475076B2
( en )
2014-10-22
2022-10-18
Narrative Science Inc.
Interactive and conversational data exploration
US11922344B2
( en )
2014-10-22
2024-03-05
Narrative Science Llc
Automatic generation of narratives from data using communication goals and narrative analytics
US11238090B1
( en )
2015-11-02
2022-02-01
Narrative Science Inc.
Applied artificial intelligence technology for using narrative analytics to automatically generate narratives from visualization data
US12153618B2
( en )
2015-11-02
2024-11-26
Salesforce, Inc.
Applied artificial intelligence technology for automatically generating narratives from visualization data
US11222184B1
( en )
2015-11-02
2022-01-11
Narrative Science Inc.
Applied artificial intelligence technology for using narrative analytics to automatically generate narratives from bar charts
US11170038B1
( en )
2015-11-02
2021-11-09
Narrative Science Inc.
Applied artificial intelligence technology for using narrative analytics to automatically generate narratives from multiple visualizations
US11232268B1
( en )
2015-11-02
2022-01-25
Narrative Science Inc.
Applied artificial intelligence technology for using narrative analytics to automatically generate narratives from line charts
US10853583B1
( en )
2016-08-31
2020-12-01
Narrative Science Inc.
Applied artificial intelligence technology for selective control over narrative generation from visualizations of data
US11568148B1
( en )
*
2017-02-17
2023-01-31
Narrative Science Inc.
Applied artificial intelligence technology for narrative generation based on explanation communication goals
US10572606B1
( en )
2017-02-17
2020-02-25
Narrative Science Inc.
Applied artificial intelligence technology for runtime computation of story outlines to support natural language generation (NLG)
US11954445B2
( en )
2017-02-17
2024-04-09
Narrative Science Llc
Applied artificial intelligence technology for narrative generation based on explanation communication goals
US10943069B1
( en )
2017-02-17
2021-03-09
Narrative Science Inc.
Applied artificial intelligence technology for narrative generation based on a conditional outcome framework
US11068661B1
( en )
2017-02-17
2021-07-20
Narrative Science Inc.
Applied artificial intelligence technology for narrative generation based on smart attributes
US11042709B1
( en )
2018-01-02
2021-06-22
Narrative Science Inc.
Context saliency-based deictic parser for natural language processing
US11003866B1
( en )
2018-01-17
2021-05-11
Narrative Science Inc.
Applied artificial intelligence technology for narrative generation using an invocable analysis service and data re-organization
US10755046B1
( en )
2018-02-19
2020-08-25
Narrative Science Inc.
Applied artificial intelligence technology for conversational inferencing
US11232270B1
( en )
*
2018-06-28
2022-01-25
Narrative Science Inc.
Applied artificial intelligence technology for using natural language processing to train a natural language generation system with respect to numeric style features
US10839164B1
( en )
*
2018-10-01
2020-11-17
Iqvia Inc.
Automated translation of clinical trial documents
CN110162604B
( en )
*
2019-01-24
2023-09-12
è ¾è®¯ç§æï¼æ·±å³ï¼æéå ¬å¸
Statement generation method, device, equipment and storage medium
US11341330B1
( en )
2019-01-28
2022-05-24
Narrative Science Inc.
Applied artificial intelligence technology for adaptive natural language understanding with term discovery
US10796104B1
( en )
*
2019-07-03
2020-10-06
Clinc, Inc.
Systems and methods for constructing an artificially diverse corpus of training data samples for training a contextually-biased model for a machine learning-based dialogue system
CN110309289B
( en )
*
2019-08-23
2019-12-06
æ·±å³å¸ä¼å¿ éç§æè¡ä»½æéå ¬å¸
Sentence generation method, sentence generation device and intelligent equipment
US11394799B2
( en )
2020-05-07
2022-07-19
Freeman Augustus Jackson
Methods, systems, apparatuses, and devices for facilitating for generation of an interactive story based on non-interactive data
US11314503B2
( en )
*
2020-06-08
2022-04-26
Bank Of America Corporation
Software development documentation using machine learning
US11556705B2
( en )
*
2020-10-29
2023-01-17
International Business Machines Corporation
Natural language processing payload generation
US12511491B2
( en )
*
2021-01-21
2025-12-30
Servicenow, Inc.
System and method for managing and optimizing lookup source templates in a natural language understanding (NLU) framework
US11734517B1
( en )
*
2021-03-09
2023-08-22
Yseop Sa
Systems and methods for measuring automatability of report generation using a natural language generation system
CH718451A2
( en )
*
2021-03-16
2022-09-30
Ella Media Ag
Procedure for generating files.
US12001786B2
( en )
2021-05-24
2024-06-04
Salesforce, Inc.
Applied artificial intelligence technology for natural language generation using a graph data structure with follow-up capabilities
US11769007B2
( en )
2021-05-27
2023-09-26
International Business Machines Corporation
Treebank synthesis for training production parsers
CN113361644B
( en )
*
2021-07-03
2024-05-14
ä¸æµ·çæ³ä¿¡æ¯äº§ä¸ï¼éå¢ï¼æéå ¬å¸
Model training method, telecommunication service characteristic information extraction method, device and equipment
CN113658585B
( en )
*
2021-08-13
2024-04-09
å京ç¾åº¦ç½è®¯ç§ææéå ¬å¸
Training method of voice interaction model, voice interaction method and device
US12204545B1
( en )
*
2021-08-13
2025-01-21
Xsqrd, Inc.
System and method for electronic document parsing
US12609102B2
( en )
*
2021-09-30
2026-04-21
Sap Se
Training dataset generation for speech-to-text service
US12511477B2
( en )
*
2021-12-30
2025-12-30
Huawei Technologies Co., Ltd.
Methods and devices for generating sensitive text detectors
CN114510578A
( en )
*
2021-12-31
2022-05-17
å¯èæºè½å·¥å(éå·)æéå ¬å¸
Triple prediction model training method, medium, generating device and server
CN114428823B
( en )
*
2022-01-28
2022-11-01
å京å½çµéç½ç»ææ¯æéå ¬å¸
Data linkage method, device, equipment and medium based on multidimensional variable expression
US12462114B2
( en )
2022-01-31
2025-11-04
Salesforce, Inc.
Applied artificial intelligence technology for integrating natural language narrative generation with newsfeeds
CN114896983B
( en )
*
2022-05-12
2025-10-28
æ¯ä»å®(æå·)ä¿¡æ¯ææ¯æéå ¬å¸
Model training method, text processing method, device and computer equipment
US20260072931A1
( en )
*
2022-09-26
2026-03-12
FORMIC Al LTD.
Application of an ai-based model to a preprocessed data set
US12496524B2
( en )
2023-05-03
2025-12-16
Gamechanger Media, Inc.
Systems and methods for automatic media generation for game sessions
US12405983B1
( en )
*
2024-02-29
2025-09-02
Palantir Technologies Inc.
Interacting with ontology-based databases using machine learning
Citations (396)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US4992939A
( en )
1988-08-05
1991-02-12
Tyler Brian G
Method of producing narrative analytical report
WO1996030844A1
( en )
1995-03-28
1996-10-03
Takashi Ogata
Support system for automation of story structure preparation
US5619631A
( en )
1995-06-07
1997-04-08
Binaryblitz
Method and apparatus for data alteration by manipulation of representational graphs
US5734916A
( en )
1994-06-01
1998-03-31
Screenplay Systems, Inc.
Method and apparatus for identifying, predicting, and reporting object relationships
US5794050A
( en )
1995-01-04
1998-08-11
Intelligent Text Processing, Inc.
Natural language understanding system
US5802495A
( en )
1996-03-01
1998-09-01
Goltra; Peter
Phrasing structure for the narrative display of findings
US5999664A
( en )
1997-11-14
1999-12-07
Xerox Corporation
System for searching a corpus of document images by user specified document layout components
US6006175A
( en )
1996-02-06
1999-12-21
The Regents Of The University Of California
Methods and apparatus for non-acoustic speech characterization and recognition
US6144938A
( en )
1998-05-01
2000-11-07
Sun Microsystems, Inc.
Voice user interface with personality
US6278967B1
( en )
1992-08-31
2001-08-21
Logovista Corporation
Automated system for generating natural language translations that are domain-specific, grammar rule-based, and/or based on part-of-speech analysis
US6289363B1
( en )
1996-08-23
2001-09-11
International Business Machines Corporation
Navigation editor framework for building mulitmedia titles
US20020046018A1
( en )
2000-05-11
2002-04-18
Daniel Marcu
Discourse parsing and summarization
US20020083025A1
( en )
1998-12-18
2002-06-27
Robarts James O.
Contextual responses based on automated learning techniques
US20020099730A1
( en )
2000-05-12
2002-07-25
Applied Psychology Research Limited
Automatic text classification system
US20020107721A1
( en )
2000-10-24
2002-08-08
International Business Machines Corporation
Story-based organizational assessment and effect system
US20030004706A1
( en )
2001-06-27
2003-01-02
Yale Thomas W.
Natural language processing system and method for knowledge management
US20030061029A1
( en )
2001-08-29
2003-03-27
Efraim Shaket
Device for conducting expectation based mixed initiative natural language dialogs
US20030110186A1
( en )
2001-04-26
2003-06-12
Michael Markowski
Dynamic generation of personalized presentation of domain-specific information content
US20030182102A1
( en )
2002-03-20
2003-09-25
Simon Corston-Oliver
Sentence realization model for a natural language generation system
US20030212543A1
( en )
2002-05-07
2003-11-13
International Business Machines Corporation
Integrated development tool for building a natural language understanding application
US6651218B1
( en )
1998-12-22
2003-11-18
Xerox Corporation
Dynamic content database for multiple document genres
US20030217335A1
( en )
2002-05-17
2003-11-20
Verity, Inc.
System and method for automatically discovering a hierarchy of concepts from a corpus of documents
US20030216905A1
( en )
2002-05-20
2003-11-20
Ciprian Chelba
Applying a structured language model to information extraction
US6665666B1
( en )
1999-10-26
2003-12-16
International Business Machines Corporation
System, method and program product for answering questions using a search engine
US20040015342A1
( en )
2002-02-15
2004-01-22
Garst Peter F.
Linguistic support for a recognizer of mathematical expressions
US20040029977A1
( en )
2000-11-30
2004-02-12
Rolf Kawa
Fine-grained emulsions
US20040034520A1
( en )
2002-03-04
2004-02-19
Irene Langkilde-Geary
Sentence generator
US6697998B1
( en )
2000-06-12
2004-02-24
International Business Machines Corporation
Automatic labeling of unlabeled text data
US20040068691A1
( en )
2002-04-19
2004-04-08
Mark Asbury
System and method for client-side locale specific numeric format handling in a web environment
US20040083092A1
( en )
2002-09-12
2004-04-29
Valles Luis Calixto
Apparatus and methods for developing conversational applications
US20040103116A1
( en )
2002-11-26
2004-05-27
Lingathurai Palanisamy
Intelligent retrieval and classification of information from a product manual
US6757362B1
( en )
2000-03-06
2004-06-29
Avaya Technology Corp.
Personal virtual assistant
US20040138899A1
( en )
2003-01-13
2004-07-15
Lawrence Birnbaum
Interactive task-sensitive assistant
US6771290B1
( en )
1998-07-17
2004-08-03
B.E. Technology, Llc
Computer interface method and apparatus with portable network organization system and targeted advertising
US20040174397A1
( en )
2003-03-05
2004-09-09
Paul Cereghini
Integration of visualizations, reports, and data
US20040225651A1
( en )
2003-05-07
2004-11-11
Musgrove Timothy A.
System and method for automatically generating a narrative product summary
US20040230989A1
( en )
2003-05-16
2004-11-18
Macey William H.
Method and apparatus for survey processing
US20040255232A1
( en )
2003-06-11
2004-12-16
Northwestern University
Networked presentation system
US20050028156A1
( en )
2003-07-30
2005-02-03
Northwestern University
Automatic method and system for formulating and transforming representations of context used by information services
US20050027704A1
( en )
2003-07-30
2005-02-03
Northwestern University
Method and system for assessing relevant properties of work contexts for use by information services
US20050033582A1
( en )
2001-02-28
2005-02-10
Michael Gadd
Spoken language interface
US20050049852A1
( en )
*
2003-09-03
2005-03-03
Chao Gerald Cheshun
Adaptive and scalable method for resolving natural language ambiguities
US20050125213A1
( en )
2003-12-04
2005-06-09
Yin Chen
Apparatus, system, and method for modeling and analyzing a plurality of computing workloads
US20050137854A1
( en )
2003-12-18
2005-06-23
Xerox Corporation
Method and apparatus for evaluating machine translation quality
US6917936B2
( en )
2002-12-18
2005-07-12
Xerox Corporation
Method and apparatus for measuring similarity between documents
US6968316B1
( en )
1999-11-03
2005-11-22
Sageworks, Inc.
Systems, methods and computer program products for producing narrative financial analysis reports
US20050273362A1
( en )
2004-06-02
2005-12-08
Catalis, Inc.
Method and system for generating medical narrative
US6976207B1
( en )
1999-04-28
2005-12-13
Ser Solutions, Inc.
Classification method and apparatus
US6976031B1
( en )
1999-12-06
2005-12-13
Sportspilot, Inc.
System and method for automatically generating a narrative report of an event, such as a sporting event
US20060031182A1
( en )
2004-08-05
2006-02-09
First Look Networks Llc
Method and apparatus for automatically providing expert analysis-based advice
<span itemp