ABSTRACT
Abstract
A system performs named entity recognition for performing natural language processing, for example, for conversation engines. The system uses context information in named entity recognition. The system includes the context of a sentence during model training and execution. The system generates high quality contextual data for training NER models. The system utilizes labeled and unlabeled contextual data for training NER models. The system provides NER models for execution in production environments. The system uses heuristics to determine whether to use a context-based NER model or a simple NER model that does not use context information. This allows the system to use simple NER models when the likelihood of improving the accuracy of prediction based on context is low.
Description
CROSS-REFERENCES TO RELATED APPLICATIONS
This application claims the benefits of U.S. Provisional Application No. 63/136,831, filed on Jan. 13, 2021, which is incorporated by reference in its entirety.
BACKGROUND
Field of Art
The disclosure relates in general to machine learning based natural language processing, and more specifically to generating training data set for machine learning based named entity recognition for natural language processing.
Description of the Related Art
Named entity recognition (NER) is a commonly used operation performed by many natural language processing (NLP) tasks. Named entity recognition is used in NLP services such as conversational BOTs that allow users to interact with online systems using unstructured text, for example, natural language sentences.
Named entity recognition is performed using linguistic grammar-based techniques as well as machine learning based models such as neural networks. Machine learning based techniques for NER typically require a large amount of manually annotated training data. This requires significant resources in obtaining the training data. Furthermore, trained machine learning based models for performing named entity recognition have lower accuracy in production since they are trained using training data that is different from real conversations. For example, real conversations may include short utterances such as single word responses. Conventional techniques typically fail to perform accurate named entity recognition in such situations.
BRIEF DESCRIPTION OF DRAWINGS
FIG. 1 is a block diagram of a system environment illustrating use of an NER model in a conversation engine, according to one embodiment.
FIG. 2 is a block diagram of the system architecture of an online system, according to an embodiment.
FIG. 3 shows a flowchart illustrating the overall process of generating training data for training the NER model according to an embodiment.
FIG. 4 shows a flowchart illustrating the process of generating context expressions according to an embodiment.
FIG. 5 shows a flowchart illustrating the process of generating answer expressions according to an embodiment.
FIG. 6 shows a flowchart illustrating the process of concatenating context and answer expressions to generate training data according to an embodiment.
FIG. 7 shows a flowchart illustrating the process of training the NER model according to an embodiment.
FIG. 8 shows a flowchart illustrating the overall process of using the NER model for processing sentences according to an embodiment.
FIG. 9 is a block diagram illustrating a functional view of a typical computer system according to one embodiment.
The figures depict various embodiments for purposes of illustration only. One skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles of the embodiments described herein.
The figures use like reference numerals to identify like elements. A letter after a reference numeral, such as â 104 A,â indicates that the text refers specifically to the element having that particular reference numeral. A reference numeral in the text without a following letter, such as â 104 ,â refers to any or all of the elements in the figures bearing that reference numeral.
DETAILED DESCRIPTION
Named entity recognition (NER) models are used for various natural language processing tasks, for example, in chatbots or conversation engines. Conversational engines or allow users to interact with web services through text or speech. During a conversation between the chatbot and a user, an NER model is invoked to help extract entity information and formalize free text to structured text. The structured text is further processed by the system, for example, to determine the user intent and to automatically generate a response for providing to the user.
Named entity recognition is also referred to as named entity identification or entity extraction. Named entity recognition includes locating and classifying named entities mentioned in unstructured text into pre-defined categories such as person names, organizations, locations, time expressions, quantities, monetary values, percentages, and so on. An example of unstructured text is an utterance referring to a sentence, a phrase, or a group thereof.
Existing techniques for NER perform fail to perform accurate named entity recognition in various situations. For example, when the chatbot asks the user: âwhat's your name?â, and the user replies: âAprilâ. In this scenario, conventional NER models may fail to determine that âAprilâ is a name of a person for various reasons. This is so, because most NER models are trained using public open datasets. These training datasets typically have full sentences. Conventional NER models typically learn from other words within the sentence to help identify the entity. However, in typical interactions with chatbots, user reply may have very short sentences, for example single word utterances. During the training phase, the NER model may never see such short answer data. A typical NER model may mis-classify the word âAprilâ as a time expression or date since most training datasets typically include a Date entity âApril.â However, in the above example, âAprilâ is the name of a person.
Embodiments use context information in named entity recognition. Embodiments include the context of a sentence during NER model training and execution. The system according to various embodiments generates high quality contextual data for training NER models. The system utilizes labeled and unlabeled contextual data for training NER models. The NER models are provided for execution in production environments. The system uses heuristics to determine whether to use a context-based NER model or a simple NER model that does not use context information.
Context information helps the system determine the intent of an utterance and boosts the performance for named entity recognition. Accordingly, the accuracy of the context-based NER model is higher than conventional NER models, thereby allowing the online system 120 to determine the named entities in a sentence of phrase more accurately.
Although applications of NER are described in connection with conversational engines, the techniques disclosed are not limited to conversational engines but can be applied to other domains. For example, analysis of documents that processes an unstructured document and annotates it with additional information describing various keywords that occur within the document.
System Environment
FIG. 1 is a block diagram of a system environment illustrating use of an NER model in a conversation engine, according to one embodiment. The system environment 100 includes an online system 120 that communicates with users via client devices 115 . The online system 120 includes a conversation engine 130 , a named entity recognition model 140 , and a data store 160 . The online system 120 may include other components not shown in FIG. 1 , for example, various applications, other types of data stores, and so on. The system environment 100 may include other elements not shown in FIG. 1 , for example, a network.
The conversation engine 130 stores the instructions that allow the online system 120 to perform natural language conversations with users 110 . A conversation engine may also be referred to as a chatbot, a dialog system, virtual assistant, or artificial intelligence (AI) assistant. The conversation engine 130 receives a natural language request from a user and determines the action that the user requested via the natural language request. The conversation engine 130 uses the named entity recognition model 140 for performing NER tasks for use in analyzing user utterances to determine how to respond to user utterances. During a conversation between the conversation engine 130 and user, the conversation engine 130 invokes NER model 140 to perform NER for extracting key information from user utterances and to formalize free text in user utterances to structured text.
The conversation engine 130 determines a context based on the current conversation with a user. The context of a conversation is represented by a sequence of recent utterances that may have been provided by the user or automatically generated by the online system 120 . The conversation engine 130 analyzes a natural language text by combining the text with the context to obtain a combined string 135 . The combined string 135 is provided as input to the named entity recognition model 140 . The named entity recognition model 140 provides an output text 145 annotated with entity information. The named entity recognition model 140 may remove the context from the combined string 135 . The output text 145 annotated with entity information is used by the conversation engine 130 for further processing. For example, if the input text was an utterance from a user, the conversation engine 130 may use the entity information to accurately determine an intent of the user. Accordingly, the conversation engine 130 may analyze the utterance based on the entity information to determine a type of information that the user may be requesting. The conversation engine 130 may perform processing to determine the information and provide to the user as a generated response.
In some embodiments, the online system 120 is a multi-tenant system that stores data of multiple tenants. Chatbots are often used by multi-tenant systems to allow tenants of the multi-tenant systems to configure chatbots for specific purposes. Each tenant may be an enterprise. For example, a tenant may be a company that employs people that use the multi-tenant system to manage their processes. A multi-tenant system may store data for multiple tenants in the same physical database but enforce a logical separation of data across tenants. As a result, one tenant does not have access to another tenant's data, unless such data is expressly shared.
The online system 120 stores data in data store 160 that may be processed in order to respond to requests from users. For example, a permitted action may update data stored in data store 160 , search for data stored in data store 160 , and so on. The data store 160 stores data for users of t
CROSS-REFERENCES TO RELATED APPLICATIONS
This application claims the benefits of U.S. Provisional Application No. 63/136,831, filed on Jan. 13, 2021, which is incorporated by reference in its entirety.
BACKGROUND
Field of Art
The disclosure relates in general to machine learning based natural language processing, and more specifically to generating training data set for machine learning based named entity recognition for natural language processing.
Description of the Related Art
Named entity recognition (NER) is a commonly used operation performed by many natural language processing (NLP) tasks. Named entity recognition is used in NLP services such as conversational BOTs that allow users to interact with online systems using unstructured text, for example, natural language sentences.
Named entity recognition is performed using linguistic grammar-based techniques as well as machine learning based models such as neural networks. Machine learning based techniques for NER typically require a large amount of manually annotated training data. This requires significant resources in obtaining the training data. Furthermore, trained machine learning based models for performing named entity recognition have lower accuracy in production since they are trained using training data that is different from real conversations. For example, real conversations may include short utterances such as single word responses. Conventional techniques typically fail to perform accurate named entity recognition in such situations.
BRIEF DESCRIPTION OF DRAWINGS
FIG. 1 is a block diagram of a system environment illustrating use of an NER model in a conversation engine, according to one embodiment.
FIG. 2 is a block diagram of the system architecture of an online system, according to an embodiment.
FIG. 3 shows a flowchart illustrating the overall process of generating training data for training the NER model according to an embodiment.
FIG. 4 shows a flowchart illustrating the process of generating context expressions according to an embodiment.
FIG. 5 shows a flowchart illustrating the process of generating answer expressions according to an embodiment.
FIG. 6 shows a flowchart illustrating the process of concatenating context and answer expressions to generate training data according to an embodiment.
FIG. 7 shows a flowchart illustrating the process of training the NER model according to an embodiment.
FIG. 8 shows a flowchart illustrating the overall process of using the NER model for processing sentences according to an embodiment.
FIG. 9 is a block diagram illustrating a functional view of a typical computer system according to one embodiment.
The figures depict various embodiments for purposes of illustration only. One skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles of the embodiments described herein.
The figures use like reference numerals to identify like elements. A letter after a reference numeral, such as â 104 A,â indicates that the text refers specifically to the element having that particular reference numeral. A reference numeral in the text without a following letter, such as â 104 ,â refers to any or all of the elements in the figures bearing that reference numeral.
DETAILED DESCRIPTION
Named entity recognition (NER) models are used for various natural language processing tasks, for example, in chatbots or conversation engines. Conversational engines or allow users to interact with web services through text or speech. During a conversation between the chatbot and a user, an NER model is invoked to help extract entity information and formalize free text to structured text. The structured text is further processed by the system, for example, to determine the user intent and to automatically generate a response for providing to the user.
Named entity recognition is also referred to as named entity identification or entity extraction. Named entity recognition includes locating and classifying named entities mentioned in unstructured text into pre-defined categories such as person names, organizations, locations, time expressions, quantities, monetary values, percentages, and so on. An example of unstructured text is an utterance referring to a sentence, a phrase, or a group thereof.
Existing techniques for NER perform fail to perform accurate named entity recognition in various situations. For example, when the chatbot asks the user: âwhat's your name?â, and the user replies: âAprilâ. In this scenario, conventional NER models may fail to determine that âAprilâ is a name of a person for various reasons. This is so, because most NER models are trained using public open datasets. These training datasets typically have full sentences. Conventional NER models typically learn from other words within the sentence to help identify the entity. However, in typical interactions with chatbots, user reply may have very short sentences, for example single word utterances. During the training phase, the NER model may never see such short answer data. A typical NER model may mis-classify the word âAprilâ as a time expression or date since most training datasets typically include a Date entity âApril.â However, in the above example, âAprilâ is the name of a person.
Embodiments use context information in named entity recognition. Embodiments include the context of a sentence during NER model training and execution. The system according to various embodiments generates high quality contextual data for training NER models. The system utilizes labeled and unlabeled contextual data for training NER models. The NER models are provided for execution in production environments. The system uses heuristics to determine whether to use a context-based NER model or a simple NER model that does not use context information.
Context information helps the system determine the intent of an utterance and boosts the performance for named entity recognition. Accordingly, the accuracy of the context-based NER model is higher than conventional NER models, thereby allowing the online system 120 to determine the named entities in a sentence of phrase more accurately.
Although applications of NER are described in connection with conversational engines, the techniques disclosed are not limited to conversational engines but can be applied to other domains. For example, analysis of documents that processes an unstructured document and annotates it with additional information describing various keywords that occur within the document.
System Environment
FIG. 1 is a block diagram of a system environment illustrating use of an NER model in a conversation engine, according to one embodiment. The system environment 100 includes an online system 120 that communicates with users via client devices 115 . The online system 120 includes a conversation engine 130 , a named entity recognition model 140 , and a data store 160 . The online system 120 may include other components not shown in FIG. 1 , for example, various applications, other types of data stores, and so on. The system environment 100 may include other elements not shown in FIG. 1 , for example, a network.
The conversation engine 130 stores the instructions that allow the online system 120 to perform natural language conversations with users 110 . A conversation engine may also be referred to as a chatbot, a dialog system, virtual assistant, or artificial intelligence (AI) assistant. The conversation engine 130 receives a natural language request from a user and determines the action that the user requested via the natural language request. The conversation engine 130 uses the named entity recognition model 140 for performing NER tasks for use in analyzing user utterances to determine how to respond to user utterances. During a conversation between the conversation engine 130 and user, the conversation engine 130 invokes NER model 140 to perform NER for extracting key information from user utterances and to formalize free text in user utterances to structured text.
The conversation engine 130 determines a context based on the current conversation with a user. The context of a conversation is represented by a sequence of recent utterances that may have been provided by the user or automatically generated by the online system 120 . The conversation engine 130 analyzes a natural language text by combining the text with the context to obtain a combined string 135 . The combined string 135 is provided as input to the named entity recognition model 140 . The named entity recognition model 140 provides an output text 145 annotated with entity information. The named entity recognition model 140 may remove the context from the combined string 135 . The output text 145 annotated with entity information is used by the conversation engine 130 for further processing. For example, if the input text was an utterance from a user, the conversation engine 130 may use the entity information to accurately determine an intent of the user. Accordingly, the conversation engine 130 may analyze the utterance based on the entity information to determine a type of information that the user may be requesting. The conversation engine 130 may perform processing to determine the information and provide to the user as a generated response.
In some embodiments, the online system 120 is a multi-tenant system that stores data of multiple tenants. Chatbots are often used by multi-tenant systems to allow tenants of the multi-tenant systems to configure chatbots for specific purposes. Each tenant may be an enterprise. For example, a tenant may be a company that employs people that use the multi-tenant system to manage their processes. A multi-tenant system may store data for multiple tenants in the same physical database but enforce a logical separation of data across tenants. As a result, one tenant does not have access to another tenant's data, unless such data is expressly shared.
The online system 120 stores data in data store 160 that may be processed in order to respond to requests from users. For example, a permitted action may update data stored in data store 160 , search for data stored in data store 160 , and so on. The data store 160 stores data for users of the multi-tenant system. In an embodiment, the data store 160 stores information describing entities represented in the multi-tenant system. Each entity has an entity type. Each entity stores values in fields or data fields. Examples of entity types include account, case, opportunity, and so on. An entity representing an account stores information describing a user or an enterprise; an entity representing an opportunity represents a potential customer and stores fields representing information identifying the potential customer, a status of the opportunity, the last interaction with the potential customer; an entity representing a case stores information describing an interaction between an enterprise and a customer and stores fields comprising the date of the interaction, the type of the interaction, and other details of the interaction.
The data store 160 may be implemented as a relational database storing one or more tables. Each table contains one or more data categories logically arranged as columns or fields. Each row or record of a table contains an instance of data for each category defined by the fields. For example, a data store 160 may include a table that describes a customer with fields for basic contact information such as name, address, phone number, fax number, etc. Another table might describe a purchase order, including fields for information such as customer, product, sale price, date, etc.
In an embodiment, multi-tenant system implements a web-based customer relationship management (CRM) system. For example, the multi-tenant system may include application servers configured to implement and execute CRM software applications as well as provide related data, code, forms, webpages and other information to and from client devices 115 and to store to, and retrieve from, a database system related data, objects, and webpage content.
The online system 120 and client devices 115 shown in FIG. 1 represent computing devices. A computing device can be a conventional computer system executing, for example, a Microsoft⢠Windowsâ¢-compatible operating system (OS), Apple⢠OS X, and/or a Linux OS. A computing device can also be a device such as a personal digital assistant (PDA), mobile telephone, video game system, etc.
The client devices 115 may interact with the online system 120 via a network (not shown in FIG. 1 ). The network uses a networking protocol such as the transmission control protocol/Internet protocol (TCP/IP), the user datagram protocol (UDP), internet control message protocol (ICMP), etc. The data exchanged over the network can be represented using technologies and/or formats including the hypertext markup language (HTML), the extensible markup language (XML), etc.
System Architecture
FIG. 2 is a block diagram of the system architecture of an online system, according to an embodiment. The online system 120 includes the conversation engine 150 , the named entity recognition model 140 , a training data generation module 210 , a training module 220 , and a training data store 230 . Other embodiments can have different and/or other components than the ones described here, and that the functionalities can be distributed among the components in a different manner.
The training data generation module 210 generates training data for training the named entity recognition model 140 . For example, a tenant of a multi-tenant system may not have sufficient data to train the named entity recognition model 140 . The training data generation module 210 can take a small amount of training data and generates additional training data for training the training data generation module 210 . The training data is stored in the training data store 230 . The training module 220 trains the named entity recognition model 140 using the training data. The various modules of the online system 120 execute the various processes further described herein.
Generation of Training Dataset Including Contextual Data
FIG. 3 shows a flowchart illustrating the overall process of generating training data for training the NER model according to an embodiment. Other embodiments can perform the operations of FIG. 3 in different orders. Other embodiments can include different and/or additional steps than the ones described herein.
The training data generation module 210 generates data by performing the
steps
310 , 315 , and 320 . The training data generation module 210 generates 310 context expressions. The training data generation module 210 generates 315 answer expressions. The training data generation module 210 concatenates 320 the generated context expressions and the generated answer expressions to generate training data that is stored in the training data store 230 . The training module 220 uses the generated training data to train 325 the named entity recognition model 140 . The details of these steps are further described herein.
Context Generation
FIG. 4 shows a flowchart illustrating the process of generating 310 context expressions according to an embodiment. Other embodiments can perform the operations of FIG. 4 in different orders. Other embodiments can include different and/or additional steps than the ones described herein.
The training data generation module 210 uses context information for contextual data generation. Accordingly, the training data generation module 210 uses high quality context data that: (1) conveys the information relevant for understanding the user's answer, and (2) covers as many expressions as possible for comprehensive training and evaluation. The use of high quality data improves NER performance.
The process illustrated in FIG. 4 is a two phase context generation method. In the first stage, the training data generation module 210 generates a high quality context seed list. For this stage, the training data generation module 210 receives manually generated commonly used context expressions based on domain knowledge. For example, to improve for PERSON entity detection performance, the training data generation module 210 starts with common expressions that ask for a person's name, such as, âwhat's your name?â, âMay I know your nameâ, and so on. For most domains, in this stage, the training data generation module 210 may use 10 to 30 common expressions.
In the second stage, the training data generation module 210 diversifies context expressions. For this stage, the training data generation module 210 diversifies the context expressions with data augmentation techniques. After this stage with more context expressions, the training data generation module 210 trains a better generalized NER model and evaluates the model more fairly.
According to various embodiments, the data augmentation methods used by the system include: (1) Casing Optimization: the training data generation module 210 converts the first letter of multiple words to uppercase, not just the first word of the sentence, e.g. transforming âWhat's your nameâ=>[âwhat's your nameâ, âWhat's Your Nameâ]. This helps NER model to focus on the context semantics rather than casing. (2) Question Mark: the training data generation module 210 randomly removes question marks from context questions, e.g. transforming âwhat's your name?â=>[âwhat's your nameâ]. This helps the NER model avoid overfitting on question mark. (3) Word switching: The training data generation module 210 switches certain words with other words that could occur in that context, whether or not they are equivalent words âMay I know your first name?â=>[âMay I know your last name?â, âCan I know your first name?â]. (4) Phrase re-generation: the training data generation module 210 generates an alternative statement that carries the same meaning but with different words. The system may use a store of equivalent words and phrases to convert a sentence to alternative sentences.
After the first stage, the training data generation module 210 gets high quality context data as it is based on manual generation. However, the number of the expressions may be limited. During the second stage, the training data generation module 210 extends the number of expressions while not losing its information. After the second stage, the training data generation module 210 may get a few hundred meaningful context expressions.
Answer Generation
FIG. 5 shows a flowchart illustrating the process of generating answer expressions according to an embodiment. In addition to generating context, the training data generation module 210 builds a pipeline to generate answers. The training data generation module 210 receives 510 a training dataset comprising utterances. The training data generation module 210 extracts 515 entities from the training dataset. The training data generation module 210 generates 520 answers based on the extracted entities.
For user answers, the training data generation module 210 uses (1) positive answers, which contains a potential entity, and (2) negative answers, which are common user responses but do not contain any entities. Negative data is used during data generation, to prevent the model from overfitting and avoiding learning that any user answer after context contains an entity.
For positive data generation, the training data generation module 210 may start from open public NER datasets. From the open dataset, the training data generation module 210 extracts entities based on the label for specific domains. The extracted entities (e.g., names like âJohn Doeâ) could either be used directly as single/short word utterance or combined with simple utterance prefixes, for example, âmy name isâ, âI amâ, to build a full sentence containing entities.
For negative data generation, the training data generation module 210 executes a process similar to that discussed above for context generation. For first stage, the training data generation module 210 manually generates common negative expressions not containing any entities, e.g. âOk, sureâ, âWhy?â. For second stage, the training data generation module 210 extends the number of negative expressions via data augmentation.
The context may include text that occurs before a term as well as text that occurs after the term. For example, for processing a document the system may look at context after answers.
Context Answer Concatenation
After the system generates context and user answers, the system concatenates context and answer. FIG. 6 shows a flowchart illustrating the process of concatenating context and answer expressions to generate training data according to an embodiment. The training data generation module 210 receives 610 a list of contexts and answers. The training data generation module 210 assigns each context in the list to different answers and concatenates 630 the respective contexts and answer pairs.
In some embodiments, the NER model is transformer based and the training data generation module 210 puts a separator ([SEP]) token between context and answer. For example, a NER model input is â[CLS] context [SEP] answer [SEP]â. Besides [SEP] token, the training data generation module 210 may add additional token or tokens to distinguish between context and answer, for example, â[CLS] [CTX] context [SEP] [ASW] answer [SEP]â. The additional tokens help the model to focus on the answer for named entity recognition and simplify post-processing. In some embodiments, the training data generation module 210 directly concatenates the context and the answer via <space> or another white space character, for example, âcontext <space> answerâ, and lets the NER model learn to detect the entity.
The training data generation module 210 may use various methods for assigning context to user answer. In an embodiment, the training data generation module 210 performs random assignment. If a system has enough context and answer data, the training data generation module 210 randomly chooses a context to assign to an answer. In an embodiment, the training data generation module 210 performs cross product assignment. If the system has less than a threshold amount of context and answer data, the system iteratively assigns each context to each answer. The generated context answer pair is a cross product of the set of contexts and the set of answers.
As an example, if the training data generation module 210 has two contexts: [âwhat's your name?â, âmay I know your name?â] and two answers: [âJohnâ, âJimâ], the above methods will generate corresponding results. Random Assignment may generate two inputs â[CLS] what's your name [SEP] Jimâ and â[CLS] may I know your name? [SEP] Johnâ. Cross Product Assignment generates four inputs: â[CLS] what's your name [SEP] Johnâ, â[CLS] may I know your name? [SEP] Jimâ, â[CLS] what's your name [SEP] Johnâ, and â[CLS] may I know your name? [SEP] Jimâ.
Contextual Model Training
The training module 220 implements an end-to-end model training pipeline based on deep learning NLP (natural language processing) techniques. FIG. 7 shows a flowchart illustrating the process of training the NER model according to an embodiment. The training module 220 performs an upstream pre-training 710 of the NER model using unlabeled data. The training module 220 further performs downstream training 720 that results in fine-tuning of NER model for specific tasks. The training module 220 provides 730 the trained model for execution for various NLP tasks.
Upstream Pre-Training with Unlabeled Data
Upstream pre-training 710 is used to train a task agnostic base NER model which can be used for multiple downstream tasks. Since the pre-training is task agnostic, the model training uses (1) language model type of loss to train the model to understand the text, and (2) next sentence prediction and contrastive loss type of loss to train the model to understand the relationship between two sentences. An upstream base model pre-trained with such losses generalizes well on multiple different downstream tasks.
Besides improved downstream performances, another advantage of upstream pre-training is that it does not require task specific labels. As the pre-training step 710 uses language model and next sentence prediction or contrastive loss, the training module 220 may use unlabeled contextual data directly. For example, the training module 220 starts with open public conversational datasets.
Downstream Fine-Tuning with Labeled Data
With the pre-trained upstream NER model and labeled data for specific downstream task, the system fine-tunes the NER model for specific downstream task. The labeled data may be generated via the above described methods performed for contextual data generation by the training data generation module 210 . For model downstream training for fine-tuning, labeled data is used as an additional dataset.
The system may use an NLP model, for example, a transformer based deep learning model. For the described contextual NER task, the model encodes the input in the format of â[CLS] context [SEP] answer [SEP]â. The encoded input is processed by the model and for each token of the encoded input, the model generates an output embedding. The output embedding is used as input for another softmax layer to help decide the entity type tag for each token. The number of the softmax layer outputs corresponds to the label space, the number of the entity types. In an embodiment, the NER model architecture is a stacked model of multiple transformer encoder layers.
The system uses unlabeled data for pre-training of a raw model to obtain a base model that is trained to process conversation and context. The system fine-tunes the base NER model to obtain the final context based NER model. During training the model uses both unlabeled data as well as labeled data.
Contextual Model Serving
To use context as part of the service for performing entity recognition, the system performs following steps (described in further details below): (1) Passing context as part of the request to the NER service; (2) Making decision whether to use context; (3) Passing context along with input text to the model; (4) Preparing the final response.
Passing Context as Part of the Request to the Service
An API (application programming interface) supports an optional parameter that has the context and is passed to the service from the client along with the actual text/utterance being processed. An example request body is as follows:
{
ââmodelIdâ: âENTITIESâ,
â// additional optional parameter - context list
ââcontextâ: [âMay I have the name of your company to locate your
âaccount?â],
ââutteranceâ: âMyCompanyâ
}
Decision to Use Context
The system may use the different heuristics to determine whether or not to pass the context to the model. The system may use one or more of (1) utterance length or (2) context intent to determine whether to use the context or not.
According to one embodiment, the system uses the length of the utterance to determine if the system should pass context to the model. Short utterances may not contain enough information to enable the model to predict entities. In some cases, for example, the conversational use case, the short utterance may include only one keyword corresponding to the entity. There is no and no other text surrounding it. To overcome this, the system adds context to utterances that are short in length (typically less than 3 words but the threshold is configurable). The advantage to this approach is that it is simple and easy to implement.
According to another embodiment, the system examines the context and determines whether the information present in the context is required by the model to make a correct prediction for the actual utterance. For each prediction request, the system first uses an ML (machine learning) model to predict whether the context is important for that particular prediction request. The system trains a relatively lightweight binary intent model such as a neural network based model on a curated dataset of contexts for this purpose. For example, the system may use a simple convolutional neural network based model, or even logistic regression. One of the use cases for this is conversational texts. For example, âWhat is your name?â is a context that has relevant information for the model to make a correct prediction on an utterance, âJohn Doeâ. The advantage of this approach is that it is a more sophisticated heuristic, and it helps ensure that the system only adds context when necessary. This helps reduce overall NER model prediction times by supplying shorter utterances.
These two techniques may be used together or separately, depending on an empirical evaluation.
Passing Context Along with Input Text to the Model
The conversation engine 150 or any NLP task performed by a system that executes the NER model executes pre-processing phase to prepare the input to the NER model. If the system decides to use context, the system combines the context and the input text together and passes the combined text to the NER model for prediction. The process used for combining the context and input text depends on the process used for training the NER model. For example, if the NER model was trained using training data that was combined as â[CLS] context [SEP] text [SEP]â the system combines the context and answer using the same approach during execution of the NER model.
Since the NER model is trained to receive context, the system uses the context as supplemental information to predict the entities present in the combined text.
Apart from preparing the input, the system may store the length of the context that is being prepended to the utterance. The system uses the length information as explained in the section below to extract the processed input text from the output.
Preparing the Final Response
The system executes a post-processing step that involves removing the entities present in the context from the output response generated using the NER model.
After the NER model makes the prediction on the combined text that also includes context, the generated response includes the tags/entities present in context as well. Since the requester, for example, the user that invokes the service is interested in the entities present in the input text, the system removes the tags predicted by the model for the context (and any special tokens that were also passed).
To achieve this, the system maintains the length of the context as described in connection with the pre-processing step. The system uses the length to strip off the predictions comprising the tags included in the context portion of the response generated using the NER. Remaining tags/entities represent the tags/entities in the input text/utterance and are provided to the requestor as the final response.
Following is an example illustrating output scores for various terms that occur in a sentence. The B-name score for the term John is high indicating that this term is a B-name named entity.
Score indicating Score indicating Score indicating
keyword is keyword is keyword is
Keywords Not entity B-Name I-name.
what. 0.99 0.05 0.05
is 0.99 0.05 0.05
your 0.99 0.05 0.05
name 0.99 0.05 0.05
John 0.05 0.99 0.05
Technical Improvements
The system according to various embodiments improves upon the accuracy of conventional systems. A conventional NER model that does not use the context information to determine the entity information failed on utterances that were short, for example, for a conversation in which the agent asked a question âwhat is you
CLAIMS
Claims ( 20 )
We claim:
1. A computer implemented method for generating a training dataset for machine learning based models for natural language processing, the method comprising:
receiving a set of natural language phrases;
generating a plurality of user responses from the set of natural language phrases, the generating comprising:
selecting a natural language phrase from the set of natural language phrases;
processing the selected natural language phrase to extract an entity of a particular named entity type;
generating a user response associated with the particular named entity type based on the extracted entity; and
determining whether to collect a plurality of context strings, each context string associated with a named entity type, wherein the determining is based on whether a length of a natural language phrase associated with the named entity type is shorter than a threshold length;
receiving, based on determining whether to collect the plurality of context strings, the plurality of context strings;
generating the training dataset by repeating for a named entity type:
selecting a context string associated with the named entity type from the plurality of context strings;
selecting a user response associated with the named entity type from the plurality of user responses, wherein the selecting is random in response to an amount of context strings in the plurality of context strings and an amount of user responses in the plurality of user responses being above a threshold amount, and wherein the selecting is according to a pattern such that each context string in the plurality of context strings is selected with each user response in the plurality of user responses in response to the amount of context strings in the plurality of context strings and the amount of user responses in the plurality of user responses being below a threshold amount; and
adding a pair of the selected context string and the selected user response to the training dataset; and
using the training dataset for training a machine learning based model for use in named entity recognition.
2. The computer implemented method of claim 1 , wherein the machine learning based model is configured to receive as input, a combined string obtained by combining a context string and a user response, wherein different portions of the combined string are identified using separator tokens.
3. The computer implemented method of claim 1 , further comprising, generating context sentences by performing:
receiving a set of seed context sentences, each seed context associated with a named entity type; and
for each of one or more seed context sentences, generating a plurality of context sentences by modifying the seed context sentence.
4. The computer implemented method of claim 3 , wherein modifying the seed context sentence comprises converting a first letter of multiple keywords of the seed context sentence to uppercase.
5. The computer implemented method of claim 3 , wherein modifying the seed context sentence comprises adding one or more punctuation marks to the seed context sentence.
6. The computer implemented method of claim 3 , wherein modifying the seed context sentence comprises switches one or more words of the seed context with other words that could occur in that context.
7. The computer implemented method of claim 3 , wherein modifying the seed context sentence comprises generating an alternative statement that has equivalent meaning using different words.
8. The computer implemented method of claim 1 , wherein the machine learning based model is used for generating responses in a conversation engine in response to a prediction of a pretrained binary intent model that a context of a conversation of the conversation engine is relevant.
9. A non-transitory computer readable storage medium storing instructions that when executed by a computer processor, cause the computer processor to perform steps comprising:
receiving a set of natural language phrases;
generating a plurality of user responses from the set of natural language phrases, the generating comprising:
selecting a natural language phrase from the set of natural language phrases;
processing the selected natural language phrase to extract an entity of a particular named entity type;
generating a user response associated with the particular named entity type based on the extracted entity; and
determining whether to collect a plurality of context strings, each context string associated with a named entity type, wherein the determining is based whether on a length of a natural language phrase associated with the named entity type is shorter than a threshold length;
receiving, based on determining whether to collect the plurality of context strings, the plurality of context strings;
generating a training dataset by repeating for a named entity type:
selecting a context string associated with the named entity type from the plurality of context strings;
selecting a user response associated with the named entity type from the plurality of user responses, wherein the selecting is random in response to an amount of context strings in the plurality of context strings and an amount of user responses in the plurality of user responses being above a threshold amount, and wherein the selecting is according to a pattern such that each context string in the plurality of context strings is selected with each user response in the plurality of user responses in response to the amount of context strings in the plurality of context strings and the amount of user responses in the plurality of user responses being below a threshold amount; and
adding a pair of the selected context string and the selected user response to the training dataset; and
using the training dataset for training a machine learning based model for use in named entity recognition.
10. The non-transitory computer readable storage medium of claim 9 , wherein the machine learning based model is configured to receive as input, a combined string obtained by combining a context string and a user response, wherein different portions of the combined string are identified using separator tokens.
11. The non-transitory computer readable storage medium of claim 9 , further comprising, generating context sentences by performing:
receiving a set of seed context sentences, each seed context associated with a named entity type; and
for each of one or more seed context sentences, generating a plurality of context sentences by modifying the seed context sentence.
12. The non-transitory computer readable storage medium of claim 11 , wherein modifying the seed context sentence comprises converting a first letter of multiple keywords of the seed context sentence to uppercase.
13. The non-transitory computer readable storage medium of claim 11 , wherein modifying the seed context sentence comprises adding one or more punctuation marks to the seed context sentence.
14. The non-transitory computer readable storage medium of claim 11 , wherein modifying the seed context sentence comprises switches one or more words of the seed context with other words that could occur in that context.
15. The non-transitory computer readable storage medium of claim 11 , wherein modifying the seed context sentence comprises generating an alternative statement that has equivalent meaning using different words.
16. The non-transitory computer readable storage medium of claim 11 , wherein the machine learning based model is used for generating responses in a conversation engine.
17. A computer system comprising:
a computer processor; and
a non-transitory computer readable storage medium storing instructions that when executed by a computer processor, cause the computer processor to perform steps comprising:
receiving a set of natural language phrases;
generating a plurality of user responses from the set of natural language phrases, the generating comprising:
selecting a natural language phrase from the set of natural language phrases;
processing the selected natural language phrase to extract an entity of a particular named entity type;
generating a user response associated with the particular named entity type based on the extracted entity; and
determining whether to collect a plurality of context strings, each context string associated with a named entity type, wherein the determining is based on whether a length of a natural language phrase associated with the named entity type;
receiving, based on determining whether to collect the plurality of context strings, the plurality of context strings;
generating a training dataset by repeating for a named entity type:
selecting a context string associated with the named entity type from the plurality of context strings;
selecting a user response associated with the named entity type from the plurality of user responses, wherein the selecting is random in response to an amount of context strings in the plurality of context strings and an amount of user responses in the plurality of user responses being above a threshold amount, and wherein the selecting is according to a pattern such that each context string in the plurality of context strings is selected with each user response in the plurality of user responses in response to the amount of context strings in the plurality of context strings and the amount of user responses in the plurality of user responses being below a threshold amount; and
adding a pair of the selected context string and the selected user response to the training dataset; and
using the training dataset for training a machine learn-ing based model for use in named entity recognition.
18. The computer system of claim 17 , wherein the machine learning based model is configured to receive as input, a combined string obtained by combining a context string and a user response, wherein different portions of the combined string are identified using separator tokens.
19. The computer system of claim 17 , further comprising, generating context sentences by performing:
receiving a set of seed context sentences, each seed context associated with a named entity type; and
for each of one or more seed context sentences, generating a plurality of context sentences by modifying the seed context sentence.
20. The computer system of claim 17 , wherein the machine learning based model is used for generating responses in a conversation engine.
US17/202,188
2021-01-13
2021-03-15
Generation of training data for machine learning based models for named entity recognition for natural language processing
Active
2041-12-28
US12001798B2
( en )
Priority Applications (1)
Application Number
Priority Date
Filing Date
Title
US17/202,188
US12001798B2
( en )
2021-01-13
2021-03-15
Generation of training data for machine learning based models for named entity recognition for natural language processing
Applications Claiming Priority (2)
Application Number
Priority Date
Filing Date
Title
US202163136831P
2021-01-13
2021-01-13
US17/202,188
US12001798B2
( en )
2021-01-13
2021-03-15
Generation of training data for machine learning based models for named entity recognition for natural language processing
Publications (2)
Publication Number
Publication Date
US20220222489A1
US20220222489A1 ( en )
2022-07-14
US12001798B2
true
US12001798B2 ( en )
2024-06-04
Family
ID=82321935
Family Applications (2)
Application Number
Title
Priority Date
Filing Date
US17/202,188
Active
2041-12-28
US12001798B2
( en )
2021-01-13
2021-03-15
Generation of training data for machine learning based models for named entity recognition for natural language processing
US17/202,183
Abandoned
US20220222441A1
( en )
2021-01-13
2021-03-15
Machine learning based named entity recognition for natural language processing
Family Applications After (1)
Application Number
Title
Priority Date
Filing Date
US17/202,183
Abandoned
US20220222441A1
( en )
2021-01-13
2021-03-15
Machine learning based named entity recognition for natural language processing
Country Status (1)
Country
Link
US
( 2 )
US12001798B2
( en )
Families Citing this family (28)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
CN112270167B
( en )
*
2020-10-14
2022-02-08
å京ç¾åº¦ç½è®¯ç§ææéå ¬å¸
Role labeling method and device, electronic equipment and storage medium
US11880659B2
( en )
2021-01-29
2024-01-23
Salesforce, Inc.
Hierarchical natural language understanding systems
KR20220161821A
( en )
*
2021-05-31
2022-12-07
ì¼ì±ìì¤ëìì¤ ì£¼ìíì¬
Apparatus and method for training model for document summarization
US12229510B2
( en )
*
2021-08-31
2025-02-18
Paypal, Inc.
Named entity recognition in chat dialogues for customer relationship management systems
US12518112B2
( en )
2021-12-20
2026-01-06
Adeia Guides Inc.
Methods and systems for responding to a natural language query
US12223951B2
( en )
*
2021-12-20
2025-02-11
Adeia Guides Inc.
Methods and systems for responding to a natural language query
US12417348B2
( en )
*
2022-03-31
2025-09-16
Oracle International Corporation
Training data augmentation using gazetteers and perturbations to facilitate training named entity recognition models
US12136413B1
( en )
*
2022-03-31
2024-11-05
Amazon Technologies, Inc.
Domain-specific parameter pre-fixes for tuning automatic speech recognition
US12307188B2
( en )
2022-04-13
2025-05-20
Servicenow, Inc.
Labeled clustering preprocessing for natural language processing
US12271699B2
( en )
*
2022-04-13
2025-04-08
Servicenow, Inc.
Multi-dimensional N-gram preprocessing for natural language processing
US20230419127A1
( en )
*
2022-06-22
2023-12-28
Oracle International Corporation
Techniques for negative entity aware augmentation
US12511498B2
( en )
*
2022-08-16
2025-12-30
Microsoft Technology Licensing, Llc
Pre-training a unified natural language model with corrupted span and replaced token detection
JP2025528391A
( en )
*
2022-08-22
2025-08-28
ãªã©ã¯ã«ã»ã¤ã³ã¿ã¼ãã·ã§ãã«ã»ã³ã¼ãã¬ã¤ã·ã§ã³
Adaptive training data augmentation to facilitate training of named entity recognition models
US12499385B2
( en )
2022-08-22
2025-12-16
Oracle International Corporation
Adaptive training data augmentation to facilitate training named entity recognition models
US20240078431A1
( en )
*
2022-08-25
2024-03-07
Nec Laboratories America, Inc.
Prompt-based sequential learning
CN115422939B
( en )
*
2022-10-14
2024-04-02
è½ç±³ç§æ(广å·)æéå ¬å¸
A fine-grained commodity named entity recognition method based on big data
US20240143941A1
( en )
*
2022-10-27
2024-05-02
Adobe Inc.
Generating subject lines from keywords utilizing a machine-learning model
US20240143925A1
( en )
*
2022-10-31
2024-05-02
Uniphore Technologies, Inc.
Method and apparatus for automatic entity recognition in customer service environments
KR20240080879A
( en )
*
2022-11-30
2024-06-07
ì¼ì±ì ì주ìíì¬
Method and apparatus for linking entities
US11822894B1
( en )
*
2022-12-30
2023-11-21
Fmr Llc
Integrating common and context-specific natural language understanding processing in a virtual assistant application
CN115952274B
( en )
*
2023-03-10
2023-06-27
å京ç¾åº¦ç½è®¯ç§ææéå ¬å¸
Data generation method, training method and device based on deep learning model
US12608652B2
( en )
*
2023-03-10
2026-04-21
International Business Machines Corporation
Tuning a trained data record matching model using customer data and representation learning
CN116029299B
( en )
*
2023-03-30
2023-06-30
å京èä¿¡æ°èç§ææéå ¬å¸
Named entity recognition method, system and storage medium based on polysemous words
US20240419903A1
( en )
*
2023-06-16
2024-12-19
Nvidia Corporation
Processing sensor data using language models in map generation systems and applications
CN116646041B
( en )
*
2023-07-21
2023-11-21
å京æ æ¯äºç§ææéå ¬å¸
Method and system for improving matching precision of clinical test subjects based on large model
US12380736B2
( en )
*
2023-08-29
2025-08-05
Ben Avi Ingel
Generating and operating personalized artificial entities
US12561314B2
( en )
*
2023-11-17
2026-02-24
Goldman Sachs & Co. LLC
Retrieval-augmented generation (RAG) system optimization
US12517941B1
( en )
*
2025-04-04
2026-01-06
Poma Ai Gmbh
Retrieval-augmented generation for large language models
Citations (45)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US20100010960A1
( en )
*
2008-07-09
2010-01-14
Yahoo! Inc.
Operations of Multi-Level Nested Data Structure
US20110131205A1
( en )
*
2009-11-28
2011-06-02
Yahoo! Inc.
System and method to identify context-dependent term importance of queries for predicting relevant search advertisements
US20130055372A1
( en )
*
2011-08-31
2013-02-28
International Business Machines Corporation
Dynamically providing algorithm-based password/challenge authentication
US20140278362A1
( en )
*
2013-03-15
2014-09-18
International Business Machines Corporation
Entity Recognition in Natural Language Processing Systems
US9009134B2
( en )
*
2010-03-16
2015-04-14
Microsoft Technology Licensing, Llc
Named entity recognition in query
US9116940B1
( en )
*
2013-04-15
2015-08-25
Google Inc.
Searching for join candidates
US20170039176A1
( en )
*
2015-08-03
2017-02-09
BlackBoiler, LLC
Method and System for Suggesting Revisions to an Electronic Document
US9613004B2
( en )
*
2007-10-17
2017-04-04
Vcvc Iii Llc
NLP-based entity recognition and disambiguation
US20180004843A1
( en )
*
2010-03-01
2018-01-04
Vcvc Iii Llc
Content recommendation based on collections of entities
US9946789B1
( en )
*
2017-04-28
2018-04-17
Shenzhen Cestbon Technology Co. Limited
Classifying electronic messages using individualized artificial intelligence techniques
US20180225281A1
( en )
*
2017-02-06
2018-08-09
Thomson Reuters Global Resources Unlimited Company
Systems and Methods for Automatic Semantic Token Tagging
US20190012302A1
( en )
2014-10-20
2019-01-10
Google Inc.
Annotations of textual segments based on user feedback
US20190065576A1
( en )
*
2017-08-23
2019-02-28
Rsvp Technologies Inc.
Single-entity-single-relation question answering systems, and methods
US20190180195A1
( en )
*
2015-01-23
2019-06-13
Conversica, Inc.
Systems and methods for training machine learning models using active learning
US20190180196A1
( en )
*
2015-01-23
2019-06-13
Conversica, Inc.
Systems and methods for generating and updating machine hybrid deep learning models
US20190251169A1
( en )
2017-02-12
2019-08-15
Seyed Ali Loghmani
Convolutional state modeling for planning natural language conversations
US20200067861A1
( en )
*
2014-12-09
2020-02-27
ZapFraud, Inc.
Scam evaluation system
US10607042B1
( en )
*
2019-02-12
2020-03-31
Live Objects, Inc.
Dynamically trained models of named entity recognition over unstructured data
US20200111021A1
( en )
*
2018-10-03
2020-04-09
Camelot Uk Bidco Limited
Systems and Methods for Training and Employing Machine Learning Models for Unique String Generation and Prediction
US20200279159A1
( en )
*
2019-03-01
2020-09-03
Fujitsu Limited
Learning method, extraction method, and information processing apparatus
US10769374B1
( en )
2019-04-24
2020-09-08
Honghui CHEN
Answer selection method for question answering system and the system
US20200285683A1
( en )
*
2019-03-05
2020-09-10
Corinne Chantal David
Method and system to filter out harassment from incoming social media data
US20200349919A1
( en )
2019-04-30
2020-11-05
Microsoft Technology Licensing, Llc
Using dialog context to improve language understanding
US20200364412A1
( en )
*
2018-05-10
2020-11-19
Tencent Technology (Shenzhen) Company Limited
Translation model training method, sentence translation method, device, and storage medium
US20200401794A1
( en )
*
2018-02-16
2020-12-24
Nippon Telegraph And Telephone Corporation
Nonverbal information generation apparatus, nonverbal information generation model learning apparatus, methods, and programs
CN112199511A
( en )
*
2020-09-28
2021-01-08
西åçµåææ¯ç 究æï¼ä¸å½çµåç§æéå¢å ¬å¸ç¬¬åç 究æï¼
Cross-language multi-source vertical domain knowledge graph construction method
US20210026924A1
( en )
*
2019-07-23
2021-01-28
International Business Machines Corporation
Natural language response improvement in machine assisted agents
US10929392B1
( en )
*
2018-11-16
2021-02-23
Amazon Technologies, Inc.
Artificial intelligence system for automated generation of realistic question and answer pairs
US20210064821A1
( en )
*
2019-08-27
2021-03-04
Ushur, Inc.
System and method to extract customized information in natural language text
US20210157872A1
( en )
*
2019-03-05
2021-05-27
Corinne David
Method and system to filter out unwanted content from incoming social media data
US20210216577A1
( en )
*
2020-01-13
2021-07-15
Adobe Inc.
Reader-retriever approach for question answering
US20210224486A1
( en )
*
2020-01-17
2021-07-22
Samsung Electronics Co., Ltd.
Structured adversarial training for natural language machine learning tasks
US20210248321A1
( en )
*
2020-02-12
2021-08-12
Home Depot Product Authority, Llc
Named entity recognition in search queries
US20210274255A1
( en )
*
2020-02-27
2021-09-02
Rovi Guides, Inc.
Systems and methods for generating dynamic annotations
US20210334459A1
( en )
*
2020-04-24
2021-10-28
Deepmind Technologies Limited
Robustness to adversarial behavior for text classification models
US20210365640A1
( en )
*
2020-05-19
2021-11-25
Samsung Sds Co., Ltd.
Method and apparatus for customizing natural language processing model
US20210365837A1
( en )
*
2020-05-19
2021-11-25
Arizona Board Of Regents On Behalf Of Arizona State University
Systems and methods for social structure construction of forums using interaction coherence
US20220121710A1
( en )
*
2020-10-21
2022-04-21
International Business Machines Corporation
Training a question-answer dialog sytem to avoid adversarial attacks
US20220121633A1
( en )
*
2020-10-15
2022-04-21
International Business Machines Corporation
Learning-based workload resource optimization for database management systems
US11321538B1
( en )
*
2021-10-15
2022-05-03
Dovel Technologies, Llc
Ensemble natural language processing model with compliance verification
US11423897B1
( en )
*
2020-01-30
2022-08-23
Amazon Technologies, Inc.
Techniques for providing adaptive responses
US20220269939A1
( en )
*
2021-02-25
2022-08-25
Robert Bosch Gmbh
Graph-based labeling rule augmentation for weakly supervised training of machine-learning-based named entity recognition
US20220277290A1
( en )
*
2021-03-01
2022-09-01
Bank Of America Corporation
Electronic system for automatically generating resource distributions based on sms-based instructions using machine learning
US20220277143A1
( en )
*
2021-02-27
2022-09-01
Walmart Apollo, Llc
Methods and apparatus for natural language understanding in conversational systems using machine learning processes
US11580379B1
( en )
*
2019-01-29
2023-02-14
Amazon Technologies, Inc.
Phased deployment of deep-learning models to customer facing APIs
Family Cites Families (1)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
CN110032634A
( en )
*
2019-04-17
2019-07-19
å京ç工大å¦
A kind of Chinese name entity recognition method based on polymorphic type depth characteristic
2021
2021-03-15
US
US17/202,188
patent/US12001798B2/en
active
Active
2021-03-15
US
US17/202,183
patent/US20220222441A1/en
not_active
Abandoned
Patent Citations (45)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US9613004B2
( en )
*
2007-10-17
2017-04-04
Vcvc Iii Llc
NLP-based entity recognition and disambiguation
US20100010960A1
( en )
*
2008-07-09
2010-01-14
Yahoo! Inc.
Operations of Multi-Level Nested Data Structure
US20110131205A1
( en )
*
2009-11-28
2011-06-02
Yahoo! Inc.
System and method to identify context-dependent term importance of queries for predicting relevant search advertisements
US20180004843A1
( en )
*
2010-03-01
2018-01-04
Vcvc Iii Llc
Content recommendation based on collections of entities
US9009134B2
( en )
*
2010-03-16
2015-04-14
Microsoft Technology Licensing, Llc
Named entity recognition in query
US20130055372A1
( en )
*
2011-08-31
2013-02-28
International Business Machines Corporation
Dynamically providing algorithm-based password/challenge authentication
US20140278362A1
( en )
*
2013-03-15
2014-09-18
International Business Machines Corporation
Entity Recognition in Natural Language Processing Systems
US9116940B1
( en )
*
2013-04-15
2015-08-25
Google Inc.
Searching for join candidates
US20190012302A1
( en )
2014-10-20
2019-01-10
Google Inc.
Annotations of textual segments based on user feedback
US20200067861A1
( en )
*
2014-12-09
2020-02-27
ZapFraud, Inc.
Scam evaluation system
US20190180195A1
( en )
*
2015-01-23
2019-06-13
Conversica, Inc.
Systems and methods for training machine learning models using active learning
US20190180196A1
( en )
*
2015-01-23
2019-06-13
Conversica, Inc.
Systems and methods for generating and updating machine hybrid deep learning models
US20170039176A1
( en )
*
2015-08-03
2017-02-09
BlackBoiler, LLC
Method and System for Suggesting Revisions to an Electronic Document
US20180225281A1
( en )
*
2017-02-06
2018-08-09
Thomson Reuters Global Resources Unlimited Company
Systems and Methods for Automatic Semantic Token Tagging
US20190251169A1
( en )
2017-02-12
2019-08-15
Seyed Ali Loghmani
Convolutional state modeling for planning natural language conversations
US9946789B1
( en )
*
2017-04-28
2018-04-17
Shenzhen Cestbon Technology Co. Limited
Classifying electronic messages using individualized artificial intelligence techniques
US20190065576A1
( en )
*
2017-08-23
2019-02-28
Rsvp Technologies Inc.
Single-entity-single-relation question answering systems, and methods
US20200401794A1
( en )
*
2018-02-16
2020-12-24
Nippon Telegraph And Telephone Corporation
Nonverbal information generation apparatus, nonverbal information generation model learning apparatus, methods, and programs
US20200364412A1
( en )
*
2018-05-10
2020-11-19
Tencent Technology (Shenzhen) Company Limited
Translation model training method, sentence translation method, device, and storage medium
US20200111021A1
( en )
*
2018-10-03
2020-04-09
Camelot Uk Bidco Limited
Systems and Methods for Training and Employing Machine Learning Models for Unique String Generation and Prediction
US10929392B1
( en )
*
2018-11-16
2021-02-23
Amazon Technologies, Inc.
Artificial intelligence system for automated generation of realistic question and answer pairs
US11580379B1
( en )
*
2019-01-29
2023-02-14
Amazon Technologies, Inc.
Phased deployment of deep-learning models to customer facing APIs
US10607042B1
( en )
*
2019-02-12
2020-03-31
Live Objects, Inc.
Dynamically trained models of named entity recognition over unstructured data
US20200279159A1
( en )
*
2019-03-01
2020-09-03
Fujitsu Limited
Learning method, extraction method, and information processing apparatus
US20200285683A1
( en )
*
2019-03-05
2020-09-10
Corinne Chantal David
Method and system to filter out harassment from incoming social media data
US20210157872A1
( en )
*
2019-03-05
2021-05-27
Corinne David
Method and system to filter out unwanted content from incoming social media data
US10769374B1
( en )
2019-04-24
2020-09-08
Honghui CHEN
Answer selection method for question answering system and the system
US20200349919A1
( en )
2019-04-30
2020-11-05
Microsoft Technology Licensing, Llc
Using dialog context to improve language understanding
US20210026924A1
( en )
*
2019-07-23
2021-01-28
International Business Machines Corporation
Natural language response improvement in machine assisted agents
US20210064821A1
( en )
*
2019-08-27
2021-03-04
Ushur, Inc.
System and method to extract customized information in natural language text
US20210216577A1
( en )
*
2020-01-13
2021-07-15
Adobe Inc.
Reader-retriever approach for question answering
US20210224486A1
( en )
*
2020-01-17
2021-07-22
Samsung Electronics Co., Ltd.
Structured adversarial training for natural language machine learning tasks
US11423897B1
( en )
*
2020-01-30
2022-08-23
Amazon Technologies, Inc.
Techniques for providing adaptive responses
US20210248321A1
( en )
*
2020-02-12
2021-08-12
Home Depot Product Authority, Llc
Named entity recognition in search queries
<a href="