Conceptio › Archive › Google Patents

Google Patentsopen access

System and method for performing a meaning search using a natural language … — Servicenow, Inc. (US11556713B2)

Servicenow, Inc. · Google Patents

Google Patents · Patents · License: Open Access

Open Source ↗

edwinsapugay servicenow

patent, google patents, intellectual property, US11556713B2, Servicenow, Inc., Edwin Sapugay, en, 2023

ABSTRACT

Abstract

The present disclosure is directed to an agent automation framework that is capable of extracting meaning from user utterances and suitably responding using a search-based natural language understanding (NLU) framework. The NLU framework includes a meaning extraction subsystem capable of detecting multiple alternative meaning representations for a given natural language utterance. Furthermore, the NLU framework includes a meaning search subsystem that enables elastic confidence thresholds (e.g., elastic beam-width meaning searches), forced diversity, and cognitive construction grammar (CCG)-based predictive scoring functions to provide an efficient and effective meaning search. As such, the disclosed meaning extraction subsystem and meaning search subsystem improve the performance, the domain specificity, the inference quality, and/or the efficiency of the NLU framework.

Description

CROSS-REFERENCES

This application claims priority from and the benefit of U.S. Provisional Application No. 62/869,864, entitled âSYSTEM AND METHOD FOR PERFORMING A MEANING SEARCH USING A NATURAL LANGUAGE UNDERSTANDING (NLU) FRAMEWORK,â filed Jul. 2, 2019, which is incorporated by reference herein in its entirety for all purposes. This application is also related to U.S. Provisional Application No. 62/869,817, entitled âPREDICTIVE SIMILARITY SCORING SUBSYSTEM IN A NATURAL LANGUAGE UNDERSTANDING (NLU) FRAMEWORKâ; U.S. Provisional Application No. 62/869,826, entitled âDERIVING MULTIPLE MEANING REPRESENTATIONS FOR AN UTTERANCE IN A NATURAL LANGUAGE UNDERSTANDING (NLU) FRAMEWORKâ; and U.S. Provisional Application No. 62/869,811, entitled âPINNING ARTIFACTS FOR EXPANSION OF SEARCH KEYS AND SEARCH SPACES IN A NATURAL LANGUAGE UNDERSTANDING (NLU) FRAMEWORK,â which were each filed Jul. 2, 2019 and are incorporated by reference herein in their entirety for all purposes.

BACKGROUND

The present disclosure relates generally to the fields of natural language understanding (NLU) and artificial intelligence (AI), and more specifically, to a hybrid learning system for NLU.

This section is intended to introduce the reader to various aspects of art that may be related to various aspects of the present disclosure, which are described and/or claimed below. This discussion is believed to be helpful in providing the reader with background information to facilitate a better understanding of the various aspects of the present disclosure. Accordingly, it should be understood that these statements are to be read in this light, and not as admissions of prior art.

Cloud computing relates to the sharing of computing resources that are generally accessed via the Internet. In particular, a cloud computing infrastructure allows users, such as individuals and/or enterprises, to access a shared pool of computing resources, such as servers, storage devices, networks, applications, and/or other computing based services. By doing so, users are able to access computing resources on demand that are located at remote locations and these resources may be used to perform a variety computing functions (e.g., storing and/or processing large quantities of computing data). For enterprise and other organization users, cloud computing provides flexibility in accessing cloud computing resources without accruing large up-front costs, such as purchasing expensive network equipment or investing large amounts of time in establishing a private network infrastructure. Instead, by utilizing cloud computing resources, users are able redirect their resources to focus on their enterprise's core functions.

Such a cloud computing service may host a virtual agent, such as a chat agent, that is designed to automatically respond to issues with the client instance based on natural language requests from a user of the client instance. For example, a user may provide a request to a virtual agent for assistance with a password issue, wherein the virtual agent is part of a Natural Language Processing (NLP) or Natural Language Understanding (NLU) system. NLP is a general area of computer science and AI that involves some form of processing of natural language input. Examples of areas addressed by NLP include language translation, speech generation, parse tree extraction, part-of-speech identification, and others. NLU is a sub-area of NLP that specifically focuses on understanding user utterances. Examples of areas addressed by NLU include question-answering (e.g., reading comprehension questions), article summarization, and others. For example, a NLU may use algorithms to reduce human language (e.g., spoken or written) into a set of known symbols for consumption by a downstream virtual agent. NLP is generally used to interpret free text for further analysis. Current approaches to NLP are typically based on deep learning, which is a type of AI that examines and uses patterns in data to improve the understanding of a program.

Certain NLU systems are based on linguistic principles of cognitive constructive grammar. One of these principles is that the shape or form of an utterance is closely related to the meaning of the utterance. As such, it is presently recognized that it is important for NLU systems to be capable of extracting or determining multiple probable ways of understanding utterances to improve the operation of the NLU system. Additionally, it is recognized that, intent inference presents a hard problem that can be solved by being transformed into a more manageable search problem.

One main issue with certain NLU systems is static search scope when attempting to locate a meaning match in a search space. For example, when a NLU system is performing a meaning search of a search space in the form of a tree-like data structure, the system may seek the terminal points or leaves of the tree, each representing specific search solutions or meaning matches. The NLU system may start from the root of the data structure, and may consider at each branch to decide which branches to follow based on defined search criteria. The selected branches may then be expanded, and the search criteria may be applied to corresponding sub-branches of the data structure to select the next branches for exploration. The meaning search progresses in this manner until the NLU system reaches the leaves of the data structure, and the selected leaves or meaning matches form a result set.

However, in such a meaning search operation, the data structure representing the search space may be quite large (e.g., gigabytes or terabytes in size). Additionally, in certain cases, certain search criteria may be expensive to apply in terms of resource usage (e.g., processing time, memory space). For such situations, it is impractical or unfeasible to explore all branches of the data structure using all defined search criteria. As such, the NLU system may apply a fixed or static search width that limits a number of branches that can be considered at any one time, resulting one or more branches of the data structure being pruned or removed from consideration. However, it is possible that a branch may demonstrate a low score when the search criteria is applied even though the branch actually includes high-scoring, and potentially the best matching, leaves or meaning matches. As such, aggressive or ineffective pruning can result in such a branch being removed from consideration, such that the high-scoring solution fails to be located by the meaning search. Additionally, searching down branches tends to be self-fulfilling, meaning that searches tend to be rewarded by digging deep along a few narrow pathways. As a result, a meaning search can move toward becoming a depth-first tree traversal that undesirably biases the meaning search. Therefore, it is recognized that an improved meaning search technique is desired to avoid premature pruning potential meaning matches from consideration, and to address the deep-tree-search problem and associated bias, during a meaning search operation.

SUMMARY

A summary of certain embodiments disclosed herein is set forth below. It should be understood that these aspects are presented merely to provide the reader with a brief summary of these certain embodiments and that these aspects are not intended to limit the scope of this disclosure. Indeed, this disclosure may encompass a variety of aspects that may not be set forth below.

Present embodiments are directed to an agent automation framework that is capable of extracting meaning from user utterances, such as requests received by a virtual agent (e.g., a chat agent), and suitably responding to these user utterances. To do this, the agent automation framework includes a NLU framework and an intent-entity model having defined intents and entities that are associated with sample utterances. The NLU framework includes a meaning extraction subsystem that is designed to generate meaning representations for the sample utterances of the intent-entity model to construct an understanding model, as well as generate meaning representations for a received user utterance to construct an utterance meaning model. Additionally, the disclosed NLU framework includes a meaning search subsystem that is designed to search the meaning representations of the understanding model (also referred to as the search space) to locate matches for meaning representations of the utterance meaning model (also referred to as the search key). As such, present embodiments generally address the hard problem posed by NLU by transforming it into a manageable search problem.

To address the shortcomings of other NLU systems set forth above, it is presently recognized that a focus threshold (e.g., beam width) can be used to establish limits or bounds on resource consumption of a meaning search. It is also presently recognized that, since different search criteria may be associated with different resource consumption or cost, the search criteria should be organized into progressive search-criteria prediction functions, wherein the initially-applied search criteria is relatively cheap and imprecise, while the later-applied search criteria is relatively more expensive and more precise. Additionally, it is presently recognized that it is desirable for the focus threshold (e.g., beam width) to be dynamically adjusted based on the level of precision of the search-criteria predictive function being applied. For example, the focus threshold may be lower (e.g., a dynamically-relaxed focus threshold, a wider beam width) when the precision of the applied search-criteria predictive function is relatively low, and may be higher (e.g., a dynamically-restricted focus threshold, a more narrow beam width) when the precision of the applied search-criteria predictive function is relatively higher. This enables a branch with a low initial search-criteria prediction function score to remain under consideration, preventing the early loss of potentially good meaning matches, while still reducing resource consumption of the search operation. Furthermore, it is presently recognized that the deep-tree-search problem and associated bias can be mitigated by using forcible diversity, which ensures that considered solutions are sufficiently different from one another. Additionally, it may be appreciated that the disclosed dynamic focus thresholds, progressive search-criteria prediction functions, and forcible diversity can be applied in two stages, such as in a first search stage across intent-entity artifacts in an understanding model, and also in a subsequent second search stage within each intent-entity artifact from the first stage that merited further exploration.

Furthermore, it is presently recognized that the performance of the meaning search is enhanced by having multiple meaning representations of a user utterance in the utterance meaning model, as well as multiple meaning representations of the sample utterances in the understanding model, for comparison. Furthermore, while certain alternative meaning representations improve the likelihood of identifying matches during a meaning search, it is also presently recognized that the meaning search can also consume additional computing resources searching clearly erroneous or substantially redundant meaning representations. Therefore, it is recognized that certain, high-value alternative meaning representations of an utterance should be pursued to enhance the meaning search, while other alternative meaning representations should be discarded to limit computing resource usage and improve the efficiency of the NLU framework.

With the foregoing in mind, present embodiments are directed to a NLU framework that includes a meaning extraction subsystem capable of generating multiple meaning representations for utterances, including sample utterances in the intent-entity model and utterances received from a user. The disclosed meaning extraction subsystem includes a number of different components, which may be implemented as plug-ins for enhanced flexibility. In certain embodiments, these include: a part of speech (POS) plug-in, a correction plug-in, a variation filter (VF) plug-in, a parser plug-in, and a final scoring and filtering (FSF) plug-in. For example, the POS plug-in may include a machine-learning (ML)-based component that receives an utterance and a POS threshold value, and generates a set of potential POS taggings for the utterance, along with corresponding confidence scores for these POS taggings, based on the POS threshold value. The correction plug-in may receive the set of potential POS taggings and apply rules-based or ML-based techniques to modify or remove entries in the set of potential POS taggings. The VF plug-in may receive the set of potential POS taggings and a variation threshold value, and may remove certain entries from the set of potential POS taggings that are not sufficiently different from other entries based on the variation threshold value. The parser plug-in may include a ML-based or rules-based component that receives the set of potential POS taggings (e.g., after correction and/or variability filtering) and generates a respective meaning representation (e.g., dependency parse tree) and corresponding confidence score for each entry in the set of potential POS taggings. The FSF plug-in may receive the set of meaning representations, the confidence scores from POS tagging and parsing, and a FSF threshold, and determine a final set of meaning representations for the utterance based on the FSF threshold.

Furthermore, it is presently recognized that the performance of the meaning search can be enhanced by using elastic (e.g., non-static, variable) thresholds when performing various stages of the meaning search. For example, once the meaning representations have been generated for a user utterance and a sample space has been populated with meaning representations of sample utterances, a first search operation may identify a set of contender matching meaning representations using an elastic beam-width CCG form-based search. This search has an elastic confidence threshold that is based on an aggregate minimum confidence threshold (AMCT) value, which is the minimum confidence value of a collection of potential confidence values (e.g., inference-level confidence values, model-level confidence values, artifact-level confidence values). The AMCT value is further expanded by a relaxation function to yield a relaxed AMCT value that defines a dynamic threshold for the CCG form-based search. It is presently recognized that, by using the elastic confidence threshold, the CCG form-based search gives a potential matching meaning representation an opportunity to survive receiving a low confidence score from an initial comparison of the search operation, such that the meaning representation can increase its confidence score in a subsequent comparison of the search operation, and be eligible to reach a final set of matching meaning representations for eventual artifact (e.g., intent and/or entity) extraction.

Similarly, other aspects of the meaning search can be enhanced by using elastic (e.g., non-static, variable) thresholds. For example, a second search operation may compare each potential matching meaning representation identified by the elastic beam-width CCG form-based search to portions of the search space that share a common intent with the potential matching meaning representation using an elastic beam-width intra-artifact search. Like the CCG form-based search, the elastic beam-width intra-artifact search has an elastic confidence threshold that is based on a model artifact confidence threshold (MACT) value, which is the artifact-level confidence value in the understanding model for the particular artifact (e.g., the intent or entity) being compared. The MACT value is further expanded by a second relaxation function to yield a relaxed MACT value that defines a dynamic threshold for the intra-artifact search. It is presently recognized that, by using the elastic confidence threshold, the intra-artifact search gives a potential matching meaning representation an opportunity to survive receiving a low confidence score from an initial comparison of the search operation, such that the meaning representation can increase its confidence score in a subsequent comparison of the search operation, and be eligible to reach a final set of matching meaning representations for eventual artifact (e.g., intent and/or entity) extraction.

BRIEF DESCRIPTION OF THE DRAWINGS

Various aspects of this disclosure may be better understood upon reading the following detailed description and upon reference to the drawings in which:

FIG. 1 is a block diagram of an embodiment of a cloud computing system in which embodiments of the present technique may operate;

FIG. 2 is a block diagram of an embodiment of a multi-instance cloud architecture in which embodiments of the present technique may operate;

<figref idref

CROSS-REFERENCES

BACKGROUND

The present disclosure relates generally to the fields of natural language understanding (NLU) and artificial intelligence (AI), and more specifically, to a hybrid learning system for NLU.

SUMMARY

BRIEF DESCRIPTION OF THE DRAWINGS

Various aspects of this disclosure may be better understood upon reading the following detailed description and upon reference to the drawings in which:

FIG. 1 is a block diagram of an embodiment of a cloud computing system in which embodiments of the present technique may operate;

FIG. 2 is a block diagram of an embodiment of a multi-instance cloud architecture in which embodiments of the present technique may operate;

FIG. 3 is a block diagram of a computing device utilized in a computing system that may be present in FIG. 1 or 2 , in accordance with aspects of the present technique;

FIG. 4 A is a schematic diagram illustrating an embodiment of an agent automation framework including an NLU framework that is part of a client instance hosted by the cloud computing system, in accordance with aspects of the present technique;

FIG. 4 B is a schematic diagram illustrating an alternative embodiment of the agent automation framework in which portions of the NLU framework are part of an enterprise instance hosted by the cloud computing system, in accordance with aspects of the present technique;

FIG. 5 is a flow diagram illustrating an embodiment of a process by which an agent automation framework, including an NLU framework and a Behavior Engine framework, extracts intents and/or entities from and responds to a user utterance, in accordance with aspects of the present technique;

FIG. 6 is a block diagram illustrating an embodiment of the NLU framework, including a meaning extraction subsystem and a meaning search subsystem, wherein the meaning extraction subsystem generates meaning representations from a received user utterance to yield an utterance meaning model and generates meaning representations from sample utterances of an understanding model to yield understanding model, and wherein the meaning search subsystem compares meaning representations of the utterance meaning model to meaning representations of the understanding model to extract artifacts (e.g., intents and entities) from the received user utterance, in accordance with aspects of the present technique;

FIG. 7 is a block diagram illustrating an embodiment of the meaning extraction subsystem using a combination of rules-based methods and machine-learning (ML)-based methods within a vocabulary subsystem, a structure subsystem, and a prosody subsystem of the NLU framework, to generate a meaning representation for an utterance, in accordance with aspects of the present technique;

FIG. 8 is a diagram illustrating an example of an utterance tree generated for an utterance, in accordance with aspects of the present technique;

FIG. 9 is a flow diagram illustrating an embodiment of a process whereby the meaning extraction subsystem generates an utterance meaning model (search key) based on a received user utterance and generates a search space based on one or more understanding models, and whereby the meaning search subsystem searches the search space, based on the search key, to extract artifacts of the user utterance, in accordance with aspects of the present technique;

FIG. 10 is an information flow diagram illustrating an embodiment of the meaning extraction subsystem generating the search space from multiple understanding models and generating multiple meaning representations of a user utterance as a search key of a meaning search to extract the artifacts of the user utterance, in accordance with aspects of the present technique;

FIG. 11 is a flow diagram illustrating an embodiment of a process by which the meaning extraction subsystem of the NLU framework, including the vocabulary subsystem and the structure subsystem, generates re-expressions and meaning representations of an utterance, in accordance with aspects of the present technique;

FIG. 12 is a flow diagram illustrating an embodiment of a process by which the structure subsystem of the NLU framework generates different meaning representations for an utterance, in accordance with aspects of the present technique;

FIG. 13 is an flow diagram illustrating an embodiment of a similarity scoring subsystem that may be implemented within the meaning search subsystem of the NLU framework to retrieve mathematical comparison function lists that enable efficient comparisons between any suitable number of meaning representations, in accordance with aspects of the present technique;

FIG. 14 is a flow diagram of an embodiment of a process by which the meaning search subsystem performs an elastic beam-width CCG form-based search, in accordance with aspects of the present technique;

FIG. 15 is a schematic diagram illustrating an embodiment of the meaning search subsystem applies the comparison function list to selectively narrow a search subspace of interest for identifying meaning representations that match an utterance-based meaning representation, in accordance with aspects of the present technique; and

FIG. 16 is a schematic diagram illustrating an effective beam-width of an embodiment provided by the relaxed aggregate minimum confidence threshold (AMCT) used during the elastic beam-width CCG form-based search, in accordance with aspects of the present technique;

FIG. 17 is a flow diagram of an embodiment of a process by which the meaning search subsystem performs an elastic beam-width intra-artifact search, in accordance with aspects of the present technique; and

FIG. 18 is a schematic diagram illustrating an effective beam-width of an embodiment provided by the relaxed model artifact confidence threshold (MACT) used during the elastic beam-width intra-artifact search, in accordance with aspects of the present technique.

DETAILED DESCRIPTION OF SPECIFIC EMBODIMENTS

One or more specific embodiments will be described below. In an effort to provide a concise description of these embodiments, not all features of an actual implementation are described in the specification. It should be appreciated that in the development of any such actual implementation, as in any engineering or design project, numerous implementation-specific decisions must be made to achieve the developers' specific goals, such as compliance with system-related and business-related constraints, which may vary from one implementation to another. Moreover, it should be appreciated that such a development effort might be complex and time consuming, but would nevertheless be a routine undertaking of design, fabrication, and manufacture for those of ordinary skill having the benefit of this disclosure.

As used herein, the term âcomputing systemâ or âcomputing deviceâ refers to an electronic computing device such as, but not limited to, a single computer, virtual machine, virtual container, host, server, laptop, and/or mobile device, or to a plurality of electronic computing devices working together to perform the function described as being performed on or by the computing system. As used herein, the term âmachine-readable mediumâ may include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store one or more instructions or data structures. The term ânon-transitory machine-readable mediumâ shall also be taken to include any tangible medium that is capable of storing, encoding, or carrying instructions for execution by the computing system and that cause the computing system to perform any one or more of the methodologies of the present subject matter, or that is capable of storing, encoding, or carrying data structures utilized by or associated with such instructions. The term ânon-transitory machine-readable mediumâ shall accordingly be taken to include, but not be limited to, solid-state memories, and optical and magnetic media. Specific examples of non-transitory machine-readable media include, but are not limited to, non-volatile memory, including by way of example, semiconductor memory devices (e.g., Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), and flash memory devices), magnetic disks such as internal hard disks and removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

As used herein, the terms âapplication,â âengine,â and âplug-inâ refer to one or more sets of computer software instructions (e.g., computer programs and/or scripts) executable by one or more processors of a computing system to provide particular functionality. Computer software instructions can be written in any suitable programming languages, such as C, C++, C#, Pascal, Fortran, Perl, MATLAB, SAS, SPSS, JavaScript, AJAX, and JAVA. Such computer software instructions can comprise an independent application with data input and data display modules. Alternatively, the disclosed computer software instructions can be classes that are instantiated as distributed objects. The disclosed computer software instructions can also be component software, for example JAVABEANS or ENTERPRISE JAVABEANS. Additionally, the disclosed applications or engines can be implemented in computer software, computer hardware, or a combination thereof.

As used herein, the term âframeworkâ refers to a system of applications and/or engines, as well as any other supporting data structures, libraries, modules, and any other supporting functionality, that cooperate to perform one or more overall functions. In particular, a ânatural language understanding frameworkâ or âNLU frameworkâ comprises a collection of computer programs designed to process and derive meaning (e.g., intents, entities, artifacts) from natural language utterances based on an understanding model. As used herein, a âbehavior engineâ or âBE,â also known as a reasoning agent or RA/BE, refers to a rule-based agent, such as a virtual agent, designed to interact with users based on a conversation model. For example, a âvirtual agentâ may refer to a particular example of a BE that is designed to interact with users via natural language requests in a particular conversational or communication channel. With this in mind, the terms âvirtual agentâ and âBEâ are used interchangeably herein. By way of specific example, a virtual agent may be or include a chat agent that interacts with users via natural language requests and responses in a chat room environment. Other examples of virtual agents may include an email agent, a forum agent, a ticketing agent, a telephone call agent, and so forth, which interact with users in the context of email, forum posts, and autoreplies to service tickets, phone calls, and so forth.

As used herein, an âintentâ refers to a desire or goal of a user which may relate to an underlying purpose of a communication, such as an utterance. As used herein, an âentityâ refers to an object, subject, or some other parameterization of an intent. It is noted that, for present embodiments, certain entities are treated as parameters of a corresponding intent. More specifically, certain entities (e.g., time and location) may be globally recognized and extracted for all intents, while other entities are intent-specific (e.g., merchandise entities associated with purchase intents) and are generally extracted only when found within the intents that define them. As used herein, âartifactâ collectively refers to both intents and entities of an utterance. As used herein, an âunderstanding modelâ is a collection of models used by the NLU framework to infer meaning of natural language utterances. An understanding model may include a vocabulary model that associates certain tokens (e.g., words or phrases) with particular word vectors, an intent-entity model, an entity model, or a combination thereof. As used herein an âintent-entity modelâ refers to a model that associates particular intents with particular sample utterances, wherein entities associated with the intent may be encoded as a parameter of the intent within the sample utterances of the model. As used herein, the term âagentsâ may refer to computer-generated personas (e.g. chat agents or other virtual agents) that interact with one another within a conversational channel. As used herein, a âcorpusâ refers to a captured body of source data that includes interactions between various users and virtual agents, wherein the interactions include communications or conversations within one or more suitable types of media (e.g., a help line, a chat room or message string, an email string). As used herein, an âutterance treeâ refers to a data structure that stores a meaning representation of an utterance. As discussed, an utterance tree has a tree structure (e.g., a dependency parse tree structure) that represents the syntactic structure of the utterance, wherein nodes of the tree structure store vectors (e.g., word vectors, subtree vectors) that encode the semantic meaning of the utterance.

As used herein, âsource dataâ or âconversation logsâ may include any suitable captured interactions between various agents, including but not limited to, chat logs, email strings, documents, help documentation, frequently asked questions (FAQs), forum entries, items in support ticketing, recordings of help line calls, and so forth. As used herein, an âutteranceâ refers to a single natural language statement made by a user or agent that may include one or more intents. As such, an utterance may be part of a previously captured corpus of source data, and an utterance may also be a new statement received from a user as part of an interaction with a virtual agent. As used herein, âmachine learningâ or âMLâ may be used to refer to any suitable statistical form of artificial intelligence capable of being trained using machine learning techniques, including supervised, unsupervised, and semi-supervised learning techniques. For example, in certain embodiments, ML-based techniques may be implemented using an artificial neural network (ANN) (e.g., a deep neural network (DNN), a recurrent neural network (RNN), a recursive neural network, a feedforward neural network). In contrast, ârules-basedâ methods and techniques refer to the use of precision rule-sets and manually-crafted ontologies that enable precise adjudication of linguistic structure and semantic understanding to derive meaning representations from utterances. As used herein, a âvectorâ (e.g., a word vector, an intent vector, a subject vector, a subtree vector) refers to a linear algebra vector that is an ordered n-dimensional list (e.g., a 300 dimensional list) of floating point values (e.g., a 1ÃN or an NÃ1 matrix) that provides a mathematical representation of the semantic meaning of a portion (e.g., a word or phrase, an intent, an entity, a token) of an utterance. As used herein, âdomain specificityâ refers to how attuned a system is to correctly extracting intents and entities expressed in actual conversations in a given domain and/or conversational channel.

As mentioned, a computing platform may include a chat agent, or another similar virtual agent, that is designed to automatically respond to user requests to perform functions or address issues on the platform. When attempting to derive user intent from a natural language utterance, it is presently recognized that a single utterance can have different potential part-of-speech (POS) taggings for the tokens of the utterance, and that these different POS taggings represent different form-derivations of the utterance. As a consequence, a number of different meaning representations (e.g., utterance trees) can be generated for a single utterance, each representing a distinct form-derivation of the utterance. It is presently recognized that giving all possible form-derivations of the utterance due consideration improves intent inference abilities of a NLU framework. However, it is also presently recognized that certain form-derivations of the utterance may be erroneous or substantially redundant to other forms already being considered, which can substantially increase processing and memory resource consumption without substantially improving the intent inference abilities of the NLU framework.

Accordingly, present embodiments are directed to a NLU framework that includes a meaning extraction subsystem that applies ML-based and rules-based techniques to generate multiple meaning representations for a natural language utterance. The meaning extraction subsystem includes a structure subsystem capable of generating multiple meaning representations of sample utterances of an understanding model to expand the search space and/or capable of generating multiple meaning representations of a received user utterance to expand the search key. The disclosed structure subsystem uses ML-based techniques to generate multiple potential part-of-speech (POS) taggings for the utterance, wherein only potential POS taggings having corresponding confidence scores greater than a predefined threshold value are advanced. The disclosed structure subsystem may apply rule-based and/or ML-based correction techniques to modify or eliminate erroneous potential POS taggings. The disclosed structure subsystem may also apply a variability filter to eliminate potential POS taggings that are not sufficiently different from one another based on a predefined variation threshold value. After correction and/or variability filtering, the disclosed structure subsystem uses ML-based or rule-based techniques to generate a respective meaning representation (e.g., an utterance tree) for each remaining potential POS tagging, wherein only meaning representations having corresponding confidence scores greater than a predefined threshold are advanced. Finally, the disclosed structure subsystem applies a final scoring and filtering step that considers the confidence scores of the advanced meaning representations, as well as the underlying confidence scores of the corresponding POS taggings, to generate a final score, and removes meaning representations having final scores below a predefined threshold. As such, the disclosed structure subsystem effectively expands the number of form-derivations that are generated for the search key and/or search space of the meaning search, improving the intent inference capabilities of the NLU framework, while eliminating erroneous or substantially redundant form-derivations to reduce resource consumption and improve efficiency of the NLU framework.

Furthermore, it is presently recognized that the quality of the meaning search can be enhanced by using elastic (e.g., non-static, variable) thresholds when performing various stages of the meaning search. As such, present embodiments are directed to a NLU framework that includes a meaning search subsystem capable of performing meaning search operations using elastic thresholds. For example, once the meaning representations have been generated for a user utterance and a sample space has been populated with meaning representations of sample utterances, a first search operation may identify a set of contender matching meaning representations using an elastic beam-width CCG form-based search. Each CCG comparison of the search has an elastic confidence threshold that is based on an aggregate minimum confidence threshold (AMCT) value, which is the minimum confidence value of a collection of potential confidence values (e.g., inference-level confidence values, model-level confidence values, artifact-level confidence values). The AMCT value is further expanded by a relaxation function to yield a relaxed AMCT value that defines a dynamic threshold for the CCG form-based search. It is presently recognized that, by using the elastic confidence threshold, the CCG form-based search gives a potential matching meaning representation an opportunity to survive receiving a low confidence score from an initial comparison of the search operation, such that the meaning representation can increase its confidence score in a subsequent comparison of the search operation, and be eligible to reach a final set of matching meaning representations for eventual artifact (e.g., intent and/or entity) extraction.

Similarly, other aspects of the meaning search can be enhanced by using elastic (e.g., non-static, variable) thresholds. For example, a second search operation performed by the meaning search subsystem may compare each potential matching meaning representation identified by the elastic beam-width CCG form-based search to portions of the search space that share a common intent with the potential matching meaning representation using an elastic beam-width intra-artifact search. Like the CCG form-based search, each comparison operation in the elastic beam-width intra-artifact search has an elastic confidence threshold that is based on a model artifact confidence threshold (MACT) value, which is the artifact-level confidence value for the artifact (e.g., the intent or entity) in the understanding model. The MACT value is further expanded by a second relaxation function to yield a relaxed MACT value that defines a dynamic threshold for the intra-artifact search. It is presently recognized that, by using the elastic confidence threshold, the intra-artifact search gives a potential matching meaning representation an opportunity to survive receiving a low confidence score from an initial comparison of the search operation, such that the meaning representation can increase its confidence score in a subsequent comparison of the search operation, and be eligible to reach a final set of matching meaning representations for eventual artifact (e.g., intent and/or entity) extraction. As such, the disclosed meaning search subsystem more effectively and comprehensively identifies matching meaning representations that might otherwise be prematurely culled during a meaning search, improving the operation and performance of the NLU framework.

With the preceding in mind, the following figures relate to various types of generalized system architectures or configurations that may be employed to provide services to an organization in a multi-instance framework and on which the present approaches may be employed. Correspondingly, these system and platform examples may also relate to systems and platforms on which the techniques discussed herein may be implemented or otherwise utilized. Turning now to FIG. 1 , a schematic diagram of an embodiment of a cloud computing system 10 where embodiments of the present disclosure may operate, is illustrated. The cloud computing system 10 may include a client network 12 , a network 18 (e.g., the Internet), and a cloud-based platform 20 . In some implementations, the cloud-based platform 20 may be a configuration management database (CMDB) platform. In one embodiment, the client network 12 may be a local private network, such as local area network (LAN) having a variety of network devices that include, but are not limited to, switches, servers, and routers. In another embodiment, the client network 12 represents an enterprise network that could include one or more LANs, virtual networks, data centers 22 , and/or other remote networks. As shown in FIG. 1 , the client network 12 is able to connect to one or

more client devices

14 A, 14 B, and 14 C so that the client devices are able to communicate with each other and/or with the network hosting the platform 20 . The client devices 14 may be computing systems and/or other types of computing devices generally referred to as Internet of Things (IoT) devices that access cloud computing services, for example, via a web browser application or via an edge device 16 that may act as a gateway between the client devices 14 and the platform 20 . FIG. 1 also illustrates that the client network 12 includes an administration or managerial device, agent, or server, such as a management, instrumentation, and discovery (MID) server 17 that facilitates communication of data between the network hosting the platform 20 , other external applications, data sources, and services, and the client network 12 . Although not specifically illustrated in FIG. 1 , the client network 12 may also include a connecting network device (e.g., a gateway or router) or a combination of devices that implement a customer firewall or intrusion protection system.

For the illustrated embodiment, FIG. 1 illustrates that client network 12 is coupled to a network 18 . The network 18 may include one or more computing networks, such as other LANs, wide area networks (WAN), the Internet, and/or other remote networks, to transfer data between the client devices 14 A-C and the network hosting the platform 20 . Each of the computing networks within network 18 may contain wired and/or wireless programmable devices that operate in the electrical and/or optical domain. For example, network 18 may include wireless networks, such as cellular networks (e.g., Global System for Mobile Communications (GSM) based cellular network), IEEE 802.11 networks, and/or other suitable radio-based networks. The network 18 may also employ any number of network communication protocols, such as Transmission Control Protocol (TCP) and Internet Protocol (IP). Although not explicitly shown in FIG. 1 , network 18 may include a variety of network devices, such as servers, routers, network switches, and/or other network hardware devices configured to transport data over the network 18 .

In FIG. 1 , the network hosting the platform 20 may be a remote network (e.g., a cloud network) that is able to communicate with the client devices 14 via the client network 12 and network 18 . The network hosting the platform 20 provides additional computing resources to the client devices 14 and/or the client network 12 . For example, by utilizing the network hosting the platform 20 , users of the client devices 14 are able to build and execute applications for various enterprise, IT, and/or other organization-related functions. In one embodiment, the network hosting the platform 20 is implemented on the one or more data centers 22 , where each data center could correspond to a different geographic location. Each of the data centers 22 includes a plurality of virtual servers 24 (also referred to herein as application nodes, application servers, virtual server instances, application instances, or application server instances), where each virtual server 24 can be implemented on a physical computing system, such as a single electronic computing device (e.g., a single physical hardware server) or across multiple-computing devices (e.g., multiple physical hardware servers). Examples of virtual servers 24 include, but are not limited to a web server (e.g., a unitary Apache installation), an application server (e.g., unitary JAVA Virtual Machine), and/or a database server (e.g., a unitary relational database management system (RDBMS) catalog).

To utilize computing resources within the platform 20 , network operators may choose to configure the data centers 22 using a variety of computing infrastructures. In one embodiment, one or more of the data centers 22 are configured using a multi-tenant cloud architecture, such that one of the server instances 24 handles requests from and serves multiple customers. Data centers 22 with multi-tenant cloud architecture commingle and store data from multiple customers, where multiple customer instances are assigned to one of the virtual servers 24 . In a multi-tenant cloud architecture, the particular virtual server 24 distinguishes between and segregates data and other information of the various customers. For example, a multi-tenant cloud architecture could assign a particular identifier for each customer in order to identify and segregate the data from each customer. Generally, implementing a multi-tenant cloud architecture may suffer from various drawbacks, such as a failure of a particular one of the server instances 24 causing outages for all customers allocated to the particular server instance.

In another embodiment, one or more of the data centers 22 are configured using a multi-instance cloud architecture to provide every customer its own unique customer instance or instances. For example, a multi-instance cloud architecture could provide each customer instance with its own dedicated application server and dedicated database server. In other examples, the multi-instance cloud architecture could deploy a single physical or virtual server 24 and/or other combinations of physical and/or virtual servers 24 , such as one or more dedicated web servers, one or more dedicated application servers, and one or more database servers, for each customer instance. In a multi-instance cloud architecture, multiple customer instances could be installed on one or more respective hardware servers, where each customer instance is allocated certain portions of the physical server resources, such as computing memory, storage, and processing power. By doing so, each customer instance has its own unique software stack that provides the benefit of data isolation, relatively less downtime for customers to access the platform 20 , and customer-driven upgrade schedules. An example of implementing a customer instance within a multi-instance cloud architecture will be discussed in more detail below with reference to FIG. 2 .

FIG. 2 is a schematic diagram of an embodiment of a multi-instance cloud architecture 40 where embodiments of the present disclosure may operate. FIG. 2 illustrates that the multi-instance cloud architecture 40 includes the client network 12 and the network 18 that connect to two (e.g., paired) <figure-cal

CLAIMS

Claims ( 20 )

What is claimed is:

1. An agent automation system, comprising:

a memory configured to store a natural language understanding (NLU) framework and a search space, wherein the NLU framework includes a meaning extraction subsystem and a meaning search subsystem, and wherein the search space includes sample meaning representations; and

a processor configured to execute instructions of the NLU framework to cause the agent automation system to perform actions comprising:

generating, via the meaning extraction subsystem, an utterance meaning model from a user utterance, wherein the utterance meaning model includes a set of search key meaning representations;

performing, via the meaning search subsystem, an elastic beam-width cognitive construction grammar (CCG) form-based search to compare the set of search key meaning representations of the utterance meaning model to the sample meaning representations of the search space and select a set of contender matching meaning representations;

performing, via the meaning search subsystem, an elastic beam-width intra-artifact search to compare the set of contender matching meaning representations to sample meaning representations of the search space and select a final set of matching meaning representations; and

extracting, via the meaning search subsystem, intents, entities, or a combination thereof, of the user utterance from the final set of matching meaning representations.

2. The system of claim 1 , wherein the meaning extraction subsystem of the NLU framework includes a vocabulary subsystem and a structure subsystem, wherein the structure subsystem includes a part-of-speech (POS) component, a correction component, a variability filter component, a parser component, and a final scoring and filtering component.

3. The system of claim 2 , wherein, to generate the utterance meaning model, the processor is configured to execute the instructions of the NLU framework to cause the agent automation system to perform actions comprising:

using the vocabulary subsystem to perform vocabulary injection to generate a set of utterances based on the user utterance, wherein the set of utterances includes the user utterance and one or more re-expressions of the user utterance having different phraseology, different terminology, or a combination thereof;

using the POS component to perform part-of-speech (POS) tagging of the set of utterances to generate a set of potential POS taggings from the set of utterances;

using the variability filter component to remove one or more POS taggings from the set of potential POS taggings that are substantially similar to other POS taggings in the set of potential POS taggings;

using the parser component to generate a set of potential meaning representations from the set of potential POS taggings; and

using the final scoring and filtering component to calculate a respective final score for each potential meaning representation in the set of potential meaning representations and selecting potential meaning representations from the set of potential meaning representations as the set of search key meaning representations of the utterance meaning model based on their respective final score.

4. The system of claim 2 , wherein the processor is configured to execute the instructions of the NLU framework to cause the agent automation system to perform actions comprising:

generating the search space from at least one understanding model stored in the memory, by:

using the vocabulary subsystem to perform vocabulary injection to generate a set of utterances based on sample utterances of an intent-entity model of the at least one understanding model, wherein the set of utterances includes the sample utterances and one or more re-expressions of the user utterance having different phraseology, different terminology, or a combination thereof;

using the POS component to perform part-of-speech (POS) tagging of the set of utterances to generate a set of potential POS taggings from the set of utterances;

using the parser component to generate a set of potential meaning representations from the set of potential POS taggings; and

using the final scoring and filtering component to calculate a respective final score for each potential meaning representation in the set of potential meaning representations and selecting potential meaning representations from the set of potential meaning representations as the sample meaning representations of the sample space based on their respective final score.

5. The system of claim 1 , wherein, to perform the elastic beam-width CCG form-based search, the processor is configured to execute the instructions of the NLU framework to cause the agent automation system to perform actions comprising:

for each search key meaning representation of the set of search key meaning representations of the utterance meaning model:

(A) determining a CCG form and a comparison function list associated with the search key meaning representation;

(B) selecting a first comparison function from the comparison function list and define a search subspace as an entirety of the search space;

(C) using the selected comparison function to compare the search key meaning representation to comparable sample meaning representations of the search space to calculate or refine a set of corresponding similarity scores;

(D) pruning meaning representations from the search subspace having corresponding similarity scores in the set of corresponding similarity scores that are below a relaxed aggregate minimum confidence threshold (AMCT);

(E) in response to determining that additional comparison functions from the comparison function list should be applied, selecting a next comparison function from the comparison function list and returning to step (C); and

(F) adding the sample meaning representations remaining in the search subspace to an initial set of contender matching meaning representations; and

performing diversity scoring of the initial sets of contender matching meaning representations to select meaning representations from the initial sets of contender matching meaning representations as the set of contender matching meaning representations.

6. The system of claim 1 , wherein, to perform the elastic beam-width intra-artifact search, the processor is configured to execute the instructions of the NLU framework to cause the agent automation system to perform actions comprising:

for each contender matching meaning representation of the set of contender matching meaning representations:

(A) determining an intent, a CCG form, and a comparison function list associated with the contender matching meaning representation;

(B) selecting a first comparison function from the comparison function list and define a search subspace as all sample meaning representations of the search space having the intent;

(C) using the selected comparison function to compare the contender matching meaning representation to comparable sample meaning representations of the search space to calculate or refine a set of corresponding similarity scores;

(D) pruning meaning representations from the search subspace having corresponding similarity scores in the set of corresponding similarity scores that are below a relaxed model artifact confidence threshold (MACT);

(F) adding the sample meaning representations remaining in the search subspace to the final set of matching meaning representations.

7. The system of claim 1 , wherein the sample meaning representations and the search key meaning representations comprise utterance trees.

8. A method of performing a meaning search of a user utterance, comprising:

generating an utterance meaning model from the user utterance, wherein the utterance meaning model includes a set of search key meaning representations;

performing an elastic beam-width cognitive construction grammar (CCG) form-based search to compare the set of search key meaning representations of the utterance meaning model to sample meaning representations of a search space and select a set of contender matching meaning representations;

performing an elastic beam-width intra-artifact search to compare the set of contender matching meaning representations to sample meaning representations of the search space and select a final set of matching meaning representations; and

extracting intents, entities, or a combination thereof, of the user utterance from the final set of matching meaning representations.

9. The method of claim 8 , wherein generating the utterance meaning model comprises:

performing vocabulary injection to generate a set of utterances based on the user utterance, wherein the set of utterances includes the user utterance and one or more re-expressions of the user utterance;

performing part-of-speech (POS) tagging of the set of utterances to generate a set of potential POS taggings for the set of utterances;

performing variability filtering to remove POS taggings from the set of potential POS taggings that are substantially similar to other POS taggings in the set of potential POS taggings;

parsing the set of potential POS taggings to generate a set of potential meaning representations; and

calculating a respective final score for each potential meaning representation in the set of potential meaning representations and selecting the set of search key meaning representations for the utterance meaning model from the set of potential meaning representations based on the respective final score of each potential meaning representation in the set of potential meaning representations.

10. The method of claim 8 , wherein performing the elastic beam-width CCG form-based search comprises:

for each search key meaning representation of the set of search key meaning representations:

(A) determining a CCG form and a comparison function list associated with the search key meaning representation;

(B) selecting a first comparison function from the comparison function list and define a search subspace as an entirety of the search space;

(F) adding the sample meaning representations remaining in the search subspace to an initial set of contender matching meaning representations; and

11. The method of claim 10 , comprising:

determining the relaxed AMCT by:

selecting an AMCT value to be a minimum confidence value from a set of confidence values, wherein the set of confidence values comprise at least one of a confidence value associated with the user utterance, a confidence value defined in an understanding model associated with the search space, or a default confidence value; and

applying a relaxation function to expand the AMCT value and generate the relaxed AMCT.

12. The method of claim 10 , wherein performing diversity scoring comprises:

determining an understanding model and an intent associated with each contender matching meaning representation in the initial sets of contender matching meaning representations; and

selecting meaning representations from the initial sets of contender matching meaning representations having a unique understanding model, a unique intent, or a combination thereof.

13. The method of claim 10 , wherein the comparison function list is ordered in terms of increasing computational cost from a least computationally expensive comparison function to a most computationally expensive comparison function.

14. The method of claim 13 , wherein at least a portion of the increasing computation cost is the result of comparing additional nodes of the search key meaning representation, or additional nodes of the comparable sample meaning representations of the search space, relative to a previous comparison function.

15. The method of claim 8 , wherein performing the elastic beam-width intra-artifact search comprises:

for each contender matching meaning representation of the set of contender matching meaning representations:

(A) determining an intent, a CCG form, and a comparison function list associated with the contender matching meaning representation;

(B) selecting a first comparison function from the comparison function list and define a search subspace as all sample meaning representations of the search space having the intent;

(F) adding the sample meaning representations remaining in the search subspace to the final set of matching meaning representations.

16. The method of claim 15 , comprising:

determining the relaxed MACT by:

selecting a MACT value to be a minimum artifact-level confidence value defined in an understanding model for artifacts of the comparable sample meaning representations; and

applying a relaxation function to expand the MACT value and generate the relaxed MACT.

17. The method of claim 8 , comprising:

providing the extracted intents, entities, or the combination thereof, to a behavior engine (BE), wherein the BE performs one or more actions in response to the extracted intents, entities, or the combination thereof, of the user utterance.

18. A non-transitory, computer-readable medium storing instructions of a natural language understanding (NLU) framework executable by one or more processors of a computing system, the instructions comprising instructions to:

generate a search space from at least one understanding model, wherein the search space comprises sample meaning representations;

generate an utterance meaning model from a user utterance, wherein the utterance meaning model includes a set of search key meaning representations;

perform an elastic beam-width cognitive construction grammar (CCG) form-based search to compare the set of search key meaning representations of the utterance meaning model to the sample meaning representations of the search space and select a set of contender matching meaning representations;

perform an elastic beam-width intra-artifact search to compare the set of contender matching meaning representations to sample meaning representations of the search space and select a final set of matching meaning representations; and

extract intents, entities, or a combination thereof, of the user utterance from the final set of matching meaning representations.

19. The medium of claim 18 , wherein the instructions to perform the elastic beam-width CCG form-based search comprise instructions to:

for each search key meaning representation of the set of search key meaning representations:

(A) determine a CCG form and a comparison function list associated with the search key meaning representation;

(B) select a first comparison function from the comparison function list and define a search subspace as an entirety of the search space;

(C) use the selected comparison function to compare the search key meaning representation to comparable sample meaning representations of the search space to calculate or refine a set of corresponding similarity scores;

(D) prune meaning representations from the search subspace having corresponding similarity scores in the set of corresponding similarity scores that are below a relaxed aggregate minimum confidence threshold (AMCT);

(E) in response to determining that additional comparison functions from the comparison function list should be applied, select a next comparison function from the comparison function list and returning to step (C); and

(F) adding the sample meaning representations remaining in the search subspace to an initial set of contender matching meaning representations; and

20. The medium of claim 18 , wherein the instructions to perform the elastic beam-width intra-artifact search comprise instructions to:

for each contender matching meaning representation of the set of contender matching meaning representations:

(A) determine an intent, a CCG form, and a comparison function list associated with the contender matching meaning representation;

(B) selecting a first comparison function from the comparison function list and define a search subspace as all sample meaning representations of the search space having the intent;

(F) adding the sample meaning representations remaining in the search subspace to the final set of matching meaning representations.

US16/749,828

2019-07-02

2020-01-22

System and method for performing a meaning search using a natural language understanding (NLU) framework

Active

2041-07-17

US11556713B2

( en )

Priority Applications (5)

Application Number

Priority Date

Filing Date

Title

US16/749,828

US11556713B2

( en )

2019-07-02

2020-01-22

System and method for performing a meaning search using a natural language understanding (NLU) framework

KR1020227002950A

KR102844617B1

( en )

2019-07-02

2020-07-02

System and method for performing semantic search using a natural language understanding (NLU) framework

PCT/US2020/040728

WO2021003408A1

( en )

2019-07-02

2020-07-02

System and method for performing a meaning search using a natural language understanding (nlu) framework

JP2021577214A

JP7346609B2

( en )

2019-07-02

2020-07-02

Systems and methods for performing semantic exploration using natural language understanding (NLU) frameworks

AU2020299634A

AU2020299634B2

( en )

2019-07-02

2020-07-02

System and method for performing a meaning search using a natural language understanding (NLU) framework

Applications Claiming Priority (5)

Application Number

Priority Date

Filing Date

Title

US201962869826P

2019-07-02

US201962869864P

2019-07-02

US201962869817P

2019-07-02

US201962869811P

2019-07-02

US16/749,828

US11556713B2

( en )

2019-07-02

2020-01-22

System and method for performing a meaning search using a natural language understanding (NLU) framework

Publications (2)

Publication Number

Publication Date

US20210004537A1

US20210004537A1 ( en )

2021-01-07

US11556713B2

true

US11556713B2 ( en )

2023-01-17

Family

ID=74065244

Family Applications (3)

Application Number

Title

Priority Date

Filing Date

US16/570,506

Active

2040-08-26

US11487945B2

( en )

2019-07-02

2019-09-13

Predictive similarity scoring subsystem in a natural language understanding (NLU) framework

US16/584,499

Abandoned

US20210004443A1

( en )

2019-07-02

2019-09-26

Pinning artifacts for expansion of search keys and search spaces in a natural language understanding (nlu) framework

US16/749,828

Active

2041-07-17

US11556713B2

( en )

2019-07-02

2020-01-22

System and method for performing a meaning search using a natural language understanding (NLU) framework

Family Applications Before (2)

Application Number

Title

Priority Date

Filing Date

US16/570,506

Active

2040-08-26

US11487945B2

( en )

2019-07-02

2019-09-13

Predictive similarity scoring subsystem in a natural language understanding (NLU) framework

US16/584,499

Abandoned

US20210004443A1

( en )

2019-07-02

2019-09-26

Pinning artifacts for expansion of search keys and search spaces in a natural language understanding (nlu) framework

Country Status (5)

Country

Link

( 3 )

US11487945B2

( en )

( 3 )

JP7420842B2

( en )

( 3 )

KR102842345B1

( en )

( 3 )

AU2020299384B2

( en )

( 3 )

WO2021003311A1

( en )

Cited By (1)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US12462798B1

( en )

2021-09-27

2025-11-04

Amazon Technologies, Inc.

Evaluation of speech processing components

Families Citing this family (46)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

WO2021057690A1

( en )

2019-09-24

2021-04-01

åä¸ºææ¯æéå¬å¸

Neural network building method and device, and image processing method and device

US11593608B2

( en )

2019-10-28

2023-02-28

Paypal, Inc.

Systems and methods for predicting and providing automated online chat assistance

US11615645B2

( en )

2019-11-19

2023-03-28

International Business Machines Corporation

Automated presentation contributions

US11651250B2

( en )

2019-11-20

2023-05-16

International Business Machines Corporation

Automatically generated conversation output

US11487948B2

( en )

2020-02-19

2022-11-01

Conduent Business Services, Llc

Method and system for automated autonomous intent mining

US11301273B2

( en )

2020-03-11

2022-04-12

Atlassian Pty Ltd.

Computer user interface for a virtual workspace having multiple application portals displaying context-related content

US11507862B2

( en )

2020-04-12

2022-11-22

International Business Machines Corporation

Determining skill adjacencies using a machine learning model

US11037549B1

( en )

2020-06-05

2021-06-15

Inference Communications Pty Ltd

System and method for automating the training of enterprise customer response systems using a range of dynamic or generic data sets

US12230406B2

( en )

2020-07-13

2025-02-18

Vignet Incorporated

Increasing diversity and engagement in clinical trails through digital tools for health data collection

US11263407B1

( en )

2020-09-01

2022-03-01

Rammer Technologies, Inc.

Determining topics and action items from conversations

US11568862B2

( en )

2020-09-29

2023-01-31

Cisco Technology, Inc.

Natural language understanding model with context resolver

US11093718B1

( en )

2020-12-01

2021-08-17

Rammer Technologies, Inc.

Determining conversational structure from speech

US11636134B2

( en )

2021-01-18

2023-04-25

Fidelity Information Services, Llc

Systems and methods for automatic change request management using similarity metric computations

US20220229986A1

( en )

2021-01-21

2022-07-21

Servicenow, Inc.

System and method for compiling and using taxonomy lookup sources in a natural language understanding (nlu) framework

US12265796B2

( en )

2021-01-21

2025-04-01

Servicenow, Inc.

Lookup source framework for a natural language understanding (NLU) framework

US12499313B2

( en )

2021-01-21

2025-12-16

Servicenow, Inc.

Ensemble scoring system for a natural language understanding (NLU) framework

US12175196B2

( en )

2021-01-21

2024-12-24

Servicenow, Inc.

Operational modeling and optimization system for a natural language understanding (NLU) framework

US12175193B2

( en )

2021-01-21

2024-12-24

Servicenow, Inc.

System and method for lookup source segmentation scoring in a natural language understanding (NLU) framework

US12299391B2

( en )

2021-01-21

2025-05-13

Servicenow, Inc.

System and method for repository-aware natural language understanding (NLU) using a lookup source framework

US12511491B2

( en )

2021-01-21

2025-12-30

Servicenow, Inc.

System and method for managing and optimizing lookup source templates in a natural language understanding (NLU) framework

US12197869B2

( en )

2021-01-21

2025-01-14

Servicenow, Inc.

Concept system for a natural language understanding (NLU) framework

US12374325B2

( en )

2021-01-21

2025-07-29

Servicenow, Inc.

Domain-aware vector encoding (DAVE) system for a natural language understanding (NLU) framework

US11296971B1

( en )

2021-02-03

2022-04-05

Vignet Incorporated

Managing and adapting monitoring programs

US11316941B1

( en )

2021-02-03

2022-04-26

Vignet Incorporated

Remotely managing and adapting monitoring programs using machine learning predictions

US11789837B1

( en )

2021-02-03

2023-10-17

Vignet Incorporated

Adaptive data collection in clinical trials to increase the likelihood of on-time completion of a trial

US11196656B1

( en )

2021-02-03

2021-12-07

Vignet Incorporated

Improving diversity in cohorts for health research

US11521714B1

( en )

2021-02-03

2022-12-06

Vignet Incorporated

Increasing diversity of participants in health research using adaptive methods

US11361846B1

( en )

2021-02-03

2022-06-14

Vignet Incorporated

Systems and methods for customizing monitoring programs involving remote devices

CN113468330B

( en )

2021-07-06

2023-04-28

åäº¬æç«¹å±ç½ç»ææ¯æéå¬å¸

Information acquisition method, device, equipment and medium

US11823666B2

( en )

2021-10-04

2023-11-21

International Business Machines Corporation

Automatic measurement of semantic similarity of conversations

US20230127907A1

( en )

2021-10-22

2023-04-27

International Business Machines Corporation

Intention identification in dialogue system

US11302314B1

( en )

2021-11-10

2022-04-12

Rammer Technologies, Inc.

Tracking specialized concepts, topics, and activities in conversations

US12135945B2

( en )

2021-11-30

2024-11-05

Kore.Ai, Inc.

Systems and methods for natural language processing using a plurality of natural language models

US12582366B2

( en )

2022-02-06

2026-03-24

Pritam Kumar Sarkar

Robotic transfer device and related method

JP7761124B2

( en )

2022-02-15

2025-10-28

ï¼®ï½ï½æ ªå¼ä¼ç¤¾

Validity determination device, validity determination method, and validity determination program

EP4383252A4

( en )

2022-02-18

2025-01-29

Samsung Electronics Co., Ltd.

ELECTRONIC DEVICE AND EXPRESSION PROCESSING METHOD THEREFOR

US12250622B2

( en )

2022-05-23

2025-03-11

T-Mobile Innovations Llc

Automated discovery schedules for telecommunication service networks

WO2023235565A1

( en )

2022-06-02

2023-12-07

University Of Central Florida Research Foundation, Inc.

Medical language model

US11599713B1

( en )

2022-07-26

2023-03-07

Rammer Technologies, Inc.

Summarizing conversational speech

WO2024049336A1

( en )

2022-08-31

2024-03-07

Oxide Ab

Software for searching digital information, a system comprising a plurality of software, and a user device connected to the system

KR102590099B1

( en )

2023-05-26

2023-10-17

ë¦¬ë²¨ë¦¬ì¨ ì£¼ìíì¬

Method and system for controlling operation of machine learning model

US12541650B2

( en )

2023-06-28

2026-02-03

Quantiphi, Inc.

Method and system for training a virtual agent using optimal utterances

US12430374B1

( en )

2024-04-01

2025-09-30

ServiceNow Inc.

Platform agnostic scalable and high-performance semantic search framework

WO2025216340A1

( en )

2024-04-11

2025-10-16

ì£¼ìíì¬ ìì¨ìíí¸

Electronic device and method for providing utterance content determined using search of persona data store, and non-transitory computer-readable storage medium

US20260052119A1

( en )

2024-08-14

2026-02-19

Servicenow, Inc.

Systems and methods for generating spokes using large language models

US12249322B1

( en )

2024-10-25

2025-03-11

Beijing Waterdrop Technology Group Co., Ltd.

Intent recognition method, apparatus and storage medium

Citations (120)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US5978594A

( en )

1994-09-30

1999-11-02

Bmc Software, Inc.

System for managing computer resources across a distributed computing environment by first reading discovery information about how to determine system resources presence

US20030009331A1

( en )

2001-07-05

2003-01-09

Johan Schalkwyk

Grammars for speech recognition

US6609122B1

( en )

2000-08-01

2003-08-19

Bmc Software, Inc.

Navigation of view relationships in database system

US6816898B1

( en )

2000-08-16

2004-11-09

Proactivenet, Inc.

Interfacing external metrics into a performance management system

US20050033574A1

( en )

2003-08-06

2005-02-10

Samsung Electronics Co., Ltd.

Method and apparatus handling speech recognition errors in spoken dialogue systems

US7020706B2

( en )

2002-06-17

2006-03-28

Bmc Software, Inc.

Method and system for automatically updating multiple servers

US7027411B1

( en )

2000-10-31

2006-04-11

Hewlett-Packard Development Company, L.P.

Method and system for identifying and processing changes to a network topology

US7028301B2

( en )

2000-12-08

2006-04-11

Bmc Software, Inc.

System and method for automatic workload characterization

US7062683B2

( en )

2003-04-22

2006-06-13

Bmc Software, Inc.

Two-phase root cause analysis

US20060130038A1

( en )

2004-12-15

2006-06-15

Claussen Christopher S

Apparatus, system, and method for facilitating dynamic modification of existing software objects defined in a strongly-typed programming language

US7131037B1

( en )

2002-06-05

2006-10-31

Proactivenet, Inc.

Method and system to correlate a specific alarm to one or more events to identify a possible cause of the alarm

US7170864B2

( en )

2001-03-08

2007-01-30

Bmc Software, Inc.

System and method for WAP server management using a single console

US20070261065A1

( en )

2006-04-20

2007-11-08

Astl Kenneth L

Framework for generating pre-packaged business integration component group pattern-based applications

US20080075444A1

( en )

2006-09-25

2008-03-27

Murali Subbarao

Blur equalization for auto-focusing

US7392300B2

( en )

2004-01-08

2008-06-24

Hewlett-Packard Development Company, L.P.

Method and system for modelling a communications network

US7509653B2

( en )

2004-01-22

2009-03-24

International Business Machines Corporation

Efficient and scalable event partitioning in business integration applications using multiple delivery queues

US20090119102A1

( en )

2007-11-01

2009-05-07

At&T Labs

System and method of exploiting prosodic features for dialog act tagging in a discriminative modeling framework

US20090258333A1

( en )

2008-03-17

2009-10-15

Kai Yu

Spoken language learning systems

US7610512B2

( en )

2006-01-06

2009-10-27

Hewlett-Packard Development Company, L.P.

System and method for automated and assisted resolution of it incidents

US7617073B2

( en )

2002-03-01

2009-11-10

Bmc Software, Inc.

System and method for assessing and indicating the health of components

US7617500B2

( en )

2005-05-20

2009-11-10

International Business Machines Corporation

Generic framework for integrating components with different interfaces in an enterprise application integration environment

US7630784B2

( en )

2005-07-22

2009-12-08

International Business Machines Corporation

Method and apparatus for independent deployment of roles

US7653650B2

( en )

2005-12-13

2010-01-26

International Business Machines Corporation

Apparatus, system, and method for synchronizing change histories in enterprise applications

US7689628B2

( en )

2005-05-19

2010-03-30

Atul Garg

Monitoring several distributed resource elements as a resource pool

US7716353B2

( en )

2005-12-21

2010-05-11

Bmc Software, Inc.

Web services availability cache

US7769718B2

( en )

2003-11-26

2010-08-03

Bmc Software, Inc.

Unobtrusive point-in-time consistent copies

US7783744B2

( en )

2003-06-03

2010-08-24

Bmc Software, Inc.

Facilitating root cause analysis for abnormal behavior of systems in a networked environment

US7849201B1

( en )

2005-06-14

2010-12-07

Billeo, Inc

Method and system for capturing, organizing, searching and sharing web pages

US7877783B1

( en )

2001-11-15

2011-01-25

Bmc Software, Inc.

System and method for secure communications with a remote software program

US7899670B1

( en )

2006-12-21

2011-03-01

Escription Inc.

Server-based speech recognition

US7925981B2

( en )

2003-05-14

2011-04-12

Hewlett-Packard Development Company, L.P.

Systems and methods for managing web services via a framework of interfaces

US7930396B2

( en )

2000-07-10

2011-04-19

Bmc Software, Inc.

System and method of enterprise systems and business impact management

US7941506B2

( en )

2004-12-06

2011-05-10

Bmc Software, Inc.

User interface for network discovery operations

US7945860B2

( en )

2003-05-14

2011-05-17

Hewlett-Packard Development Company, L.P.

Systems and methods for managing conversations between information technology resources

US7958031B2

( en )

2005-12-13

2011-06-07

International Business Machines Corporation

Apparatus, system, and method for automated identity relationship maintenance

US7966398B2

( en )

2001-03-30

2011-06-21

Bmc Software, Inc.

Synthetic transaction monitor with replay capability

US7971147B2

( en )

2005-07-25

2011-06-28

Billeo, Inc.

Methods and systems for automatically creating a site menu

US8006240B2

( en )

2005-12-01

2011-08-23

International Business Machines Corporation

Support continuous availability by allowing the use of multiple concurrent versions of shared artifact libraries, with proper bind-drain semantics, for long-lived process application consumers

US20110238410A1

( en )

2010-03-26

2011-09-29

Jean-Marie Henri Daniel Larcheveque

Semantic Clustering and User Interfaces

US20110238408A1

( en )

2010-03-26

2011-09-29

Jean-Marie Henri Daniel Larcheveque

Semantic Clustering

US8051164B2

( en )

2007-12-14

2011-11-01

Bmc Software, Inc.

Impact propagation in a directed acyclic graph having restricted views

US8112354B2

( en )

2005-04-08

2012-02-07

Billeo, Inc

Method and system for virtual consolidation of biller direct web sites

System and method for performing a meaning search using a natural language … — Servicenow, Inc. (US11556713B2)

Related documents