ABSTRACT
Abstract
A natural language understanding (NLU) framework includes a domain-aware vector encoding (DAVE) framework. The DAVE framework enables a designer to create a DAVE system having a domain-agnostic semantic (DAS) model and a corresponding trained vector translator (VT) model. The DAVE system uses the DAS model to generate domain-agnostic semantic vectors for portions of a user utterance, and then uses the VT model to translate the domain-agnostic semantic vectors into a domain-aware semantic vectors to be used by a NLU system of the NLU framework during a meaning search operation. The VT model is also designed to provide predicted intent classifications for the portions the user utterance. Both the NLU system and the DAVE system of the NLU framework are highly configurable and refer to various NLU constraints during operation, including performance constraints and resource constraints provided by a designer or user of the NLU framework.
Description
CROSS-REFERENCE
This application claims priority from and the benefit of U.S. Provisional Patent Application No. 63/140,098, entitled âDOMAIN-AWARE VECTOR ENCODING (DAVE) SYSTEM FOR A NATURAL LANGUAGE UNDERSTANDING (NLU) FRAMEWORK,â filed Jan. 21, 2021, which is herein incorporated by reference in its entirety for all purposes.
BACKGROUND
The present disclosure relates generally to the fields of natural language understanding (NLU) and artificial intelligence (AI), and more specifically, to a hybrid learning system for NLU.
This section is intended to introduce the reader to various aspects of art that may be related to various aspects of the present disclosure, which are described and/or claimed below. This discussion is believed to be helpful in providing the reader with background information to facilitate a better understanding of the various aspects of the present disclosure. Accordingly, it should be understood that these statements are to be read in this light, and not as admissions of prior art.
Cloud computing relates to the sharing of computing resources that are generally accessed via the Internet. In particular, a cloud computing infrastructure allows users, such as individuals and/or enterprises, to access a shared pool of computing resources, such as servers, storage devices, networks, applications, and/or other computing based services. By doing so, users are able to access computing resources on demand that are located at remote locations and these resources may be used to perform a variety computing functions (e.g., storing and/or processing large quantities of computing data). For enterprise and other organization users, cloud computing provides flexibility in accessing cloud computing resources without accruing large up-front costs, such as purchasing expensive network equipment or investing large amounts of time in establishing a private network infrastructure. Instead, by utilizing cloud computing resources, users are able redirect their resources to focus on their enterprise's core functions.
In modern communication networks, examples of cloud computing services a user may utilize include so-called infrastructure as a service (IaaS), software as a service (SaaS), and platform as a service (PaaS) technologies. IaaS is a model in which providers abstract away the complexity of hardware infrastructure and provide rapid, simplified provisioning of virtual servers and storage, giving enterprises access to computing capacity on demand. In such an approach, however, a user may be left to install and maintain platform components and applications. SaaS is a delivery model that provides software as a service rather than an end product. Instead of utilizing a local network or individual software installations, software is typically licensed on a subscription basis, hosted on a remote machine, and accessed by client customers as needed. For example, users are generally able to access a variety of enterprise and/or information technology (IT)-related software via a web browser. PaaS acts an extension of SaaS that goes beyond providing software services by offering customizability and expandability features to meet a user's needs. For example, PaaS can provide a cloud-based developmental platform for users to develop, modify, and/or customize applications and/or automating enterprise operations without maintaining network infrastructure and/or allocating computing resources normally associated with these functions.
Such a cloud computing service may host a virtual agent, such as a chat agent, that is designed to automatically respond to issues with the client instance based on natural language requests from a user of the client instance. For example, a user may provide a request to a virtual agent for assistance with a password issue, wherein the virtual agent is part of a Natural Language Processing (NLP) or Natural Language Understanding (NLU) system. NLP is a general area of computer science and AI that involves some form of processing of natural language input. Examples of areas addressed by NLP include language translation, speech generation, parse tree extraction, part-of-speech identification, and others. NLU is a sub-area of NLP that specifically focuses on understanding user utterances. Examples of areas addressed by NLU include question-answering (e.g., reading comprehension questions), article summarization, and others. For example, a NLU may use algorithms to reduce human language (e.g., spoken or written) into a set of known symbols for consumption by a downstream virtual agent. NLP is generally used to interpret free text for further analysis. Current approaches to NLP are typically based on deep learning, which is a type of AI that examines and uses patterns in data to improve the understanding of a program.
As such, it is presently recognized that there is a need to improve the ability of virtual agents to apply NLU techniques to properly derive meaning from complex natural language utterances. For example, it may be advantageous to create a virtual agent capable of comprehending complex language and executing contextually relevant requests, which could afford substantial advantages in terms of reduced operational cost and increased responsiveness to client issues. Additionally, it is recognized that it is advantageous for virtual agents to be customizable and adaptable to various communication channels and styles.
SUMMARY
A summary of certain embodiments disclosed herein is set forth below. It should be understood that these aspects are presented merely to provide the reader with a brief summary of these certain embodiments and that these aspects are not intended to limit the scope of this disclosure. Indeed, this disclosure may encompass a variety of aspects that may not be set forth below.
NLU systems are used by a wide variety of clients for various domains, such as Information Technology Management (ITSM), Customer Service Management (CSM), Human Resource Management (HRM), Finance, and so forth. However, in certain embodiments, a NLU system may utilize one or more machine learning (ML)-based word vector distribution models (also referred to herein as semantic models or neural language models) that are trained based on a domain-agnostic corpus, such as an encyclopedia, a dictionary, a newspaper, to generate semantic vectors (also referred to as encodings or embeddings) for portions of utterances, including tokens of utterances, phrases of utterances, and/or entire utterances. It is presently recognized that, while this enables large, existing vector spaces to be leveraged that capture important relationships between many words and phrases (e.g., frequently used terms) within a given language, these domain-agnostic semantic models can fail to provide suitable semantic vectors for domain-specific terminology. For example, a term that is rarely or never used outside of a particular domain (e.g., a domain-specific term) may not be sufficiently represented within the domain-agnostic corpus to enable the domain-agnostic semantic model to learn a high-quality semantic vector that suitably represents the meaning of the term relative to other terms of the generic corpus represented within the vector space. Additionally, it may be desirable to leverage an existing semantic model that generates semantic vectors in a vector space having a different number of dimensions than the vector space(s) utilized by the NLU system.
With this in mind, the disclosed NLU framework includes a domain-aware vector encoding (DAVE) framework. The DAVE framework enables a designer to create a DAVE system having a domain-agnostic semantic (DAS) model and a corresponding trained vector translator (VT) model. The DAVE system uses the DAS model to generate a domain-agnostic semantic vector for a user utterance or a portion of a NLU-processed user utterance, and then uses the VT model to translate the domain-agnostic semantic vector into a domain-aware semantic vector to be used by a NLU system of the NLU framework during a meaning search operation. The VT model is also designed to provide one or more predicted intent classifications for the user utterance or the portion of a NLU-processed user utterance. Both the NLU system and the DAVE system of the NLU framework are highly configurable and refer to various NLU constraints during operation, including performance constraints and resource constraints provided by a designer or user of the NLU framework. As such, the disclosed designs ensure the NLU framework provides the desired level of performance (e.g., desired prediction latency, desired precision, desired recall, desired operational explainability) without exceeding a desired level of computational resource usage (e.g., processing time, memory usage, storage usage). The disclosed DAVE system enhances the performance (e.g., precision and/or recall) of the NLU system within the specific domain of the client, improves the quality of predictions of the NLU system for various tasks, such as intent recognition, entity recognition, and so forth. Additionally, since the DAVE system enables the use of existing DAS models in the NLU framework regardless of dimensionality, the disclosed DAVE system gives the designer freedom in selecting and immediately leveraging best-of-breed DAS models as they become available.
BRIEF DESCRIPTION OF THE DRAWINGS
Various aspects of this disclosure may be better understood upon reading the following detailed description and upon reference to the drawings in which:
FIG. 1 is a block diagram of an embodiment of a cloud computing system in which embodiments of the present technique may operate;
FIG. 2 is a block diagram of an embodiment of a multi-instance cloud architecture in which embodiments of the present technique may operate;
FIG. 3 is a block diagram of a computing device utilized in a computing system that may be present in FIG. 1 or 2 , in accordance with aspects of the present technique;
FIG. 4 A is a schematic diagram illustrating an embodiment of an agent automation framework including a NLU framework that is part of a client instance hosted by the cloud computing system, in accordance with aspects of the present technique;
FIG. 4 B is a schematic diagram illustrating an alternative embodiment of the agent automation framework in which portions of the NLU framework are part of an enterprise instance hosted by the cloud computing system, in accordance with aspects of the present technique;
FIG. 5 is a flow diagram illustrating an embodiment of a process by which an agent automation framework, including an NLU framework and a Reasoning Agent/Behavior Engine (RA/BE) framework, extracts intent/entities from and responds to a user utterance, in accordance with aspects of the present technique;
FIG. 6 is a block diagram illustrating an embodiment of the NLU framework including a meaning extraction subsystem and a meaning search subsystem, wherein the meaning extraction subsystem generates meaning representations from a received user utterance to yield an utterance meaning model and generates meaning representations from sample utterances of an intent/entity model to yield understanding model, and wherein the meaning search subsystem compares meaning representations of the utterance meaning model to meaning representations of the understanding model to extract intents and entities from the received user utterance, in accordance with aspects of the present technique;
FIG. 7 is a block diagram illustrating an embodiment of the meaning extraction subsystem using a combination of rules-based methods and machine-learning (ML)-based methods within a vocabulary subsystem, a structure subsystem, and a prosody subsystem, to generate an annotated utterance tree for an utterance, in accordance with aspects of the present technique;
FIG. 8 is a flow diagram illustrating an example process by which the meaning extraction subsystem performs error correction of an annotated utterance tree of an utterance before generating the corresponding meaning representation of the utterance, in accordance with aspects of the present technique;
FIG. 9 is a flow diagram illustrating an example process by which the meaning extraction subsystem generates a meaning representations of the understanding model or the utterance meaning model based on the annotated utterance trees and a compilation model template, in accordance with aspects of the present technique;
FIG. 10 is a block diagram illustrating an embodiment of the compilation model template, in accordance with aspects of the present technique;
FIG. 11 is a block diagram illustrating example operation of an embodiment of a tree substructure vectorization algorithm to generate a combined subtree vector for a subtree of an annotated utterance tree, in accordance with aspects of the present technique;
FIG. 12 is a flow diagram illustrating example process by which the meaning search subsystem searches the meaning representations of the understanding model for matches to the meaning representation of the user utterance, in accordance with aspects of the present technique;
FIG. 13 is a flow diagram illustrating an embodiment of a process by which a tree-model comparison algorithm compares an intent subtree of a first meaning representation to an intent subtree of a second meaning representation, based on the compilation model template, to generate an intent subtree similarity score, in accordance with aspects of the present technique;
FIG. 14 is a block diagram illustrating an embodiment of a process by which the agent automation system continuously improves a structure learning model, such as a recurrent neural network associated with a ML-based parser of the NLU framework, for improved domain specificity, based on a collection of utterances, in accordance with aspects of the present technique;
FIG. 15 is a block diagram illustrating an embodiment of a process by which the agent automation system continuously learns new words and/or refines word understanding for improved domain specificity based on a collection of utterances, in accordance with aspects of the present technique;
FIG. 16 is a diagram illustrating an embodiment of an annotated utterance tree, in accordance with aspects of the present technique; and
FIG. 17 is a diagram illustrating an embodiment of a meaning representation, in accordance with aspects of the present technique.
FIG. 18 is a block diagram illustrating how the vocabulary subsystem of FIG. 7 operates within the NLU framework, in accordance with aspects of the present techniques;
FIG. 19 is a diagram illustrating what may be included in a vocabulary model template, in accordance with aspects of the present techniques;
FIG. 20 is a block diagram illustrating example operation of an embodiment of a multi-vector aggregation algorithm to generate a combined sub-phrase vector for a subtree of an annotated utterance tree, in accordance with aspects of the present techniques;
FIG. 21 is a flow diagram illu
CROSS-REFERENCE
This application claims priority from and the benefit of U.S. Provisional Patent Application No. 63/140,098, entitled âDOMAIN-AWARE VECTOR ENCODING (DAVE) SYSTEM FOR A NATURAL LANGUAGE UNDERSTANDING (NLU) FRAMEWORK,â filed Jan. 21, 2021, which is herein incorporated by reference in its entirety for all purposes.
BACKGROUND
The present disclosure relates generally to the fields of natural language understanding (NLU) and artificial intelligence (AI), and more specifically, to a hybrid learning system for NLU.
This section is intended to introduce the reader to various aspects of art that may be related to various aspects of the present disclosure, which are described and/or claimed below. This discussion is believed to be helpful in providing the reader with background information to facilitate a better understanding of the various aspects of the present disclosure. Accordingly, it should be understood that these statements are to be read in this light, and not as admissions of prior art.
Cloud computing relates to the sharing of computing resources that are generally accessed via the Internet. In particular, a cloud computing infrastructure allows users, such as individuals and/or enterprises, to access a shared pool of computing resources, such as servers, storage devices, networks, applications, and/or other computing based services. By doing so, users are able to access computing resources on demand that are located at remote locations and these resources may be used to perform a variety computing functions (e.g., storing and/or processing large quantities of computing data). For enterprise and other organization users, cloud computing provides flexibility in accessing cloud computing resources without accruing large up-front costs, such as purchasing expensive network equipment or investing large amounts of time in establishing a private network infrastructure. Instead, by utilizing cloud computing resources, users are able redirect their resources to focus on their enterprise's core functions.
In modern communication networks, examples of cloud computing services a user may utilize include so-called infrastructure as a service (IaaS), software as a service (SaaS), and platform as a service (PaaS) technologies. IaaS is a model in which providers abstract away the complexity of hardware infrastructure and provide rapid, simplified provisioning of virtual servers and storage, giving enterprises access to computing capacity on demand. In such an approach, however, a user may be left to install and maintain platform components and applications. SaaS is a delivery model that provides software as a service rather than an end product. Instead of utilizing a local network or individual software installations, software is typically licensed on a subscription basis, hosted on a remote machine, and accessed by client customers as needed. For example, users are generally able to access a variety of enterprise and/or information technology (IT)-related software via a web browser. PaaS acts an extension of SaaS that goes beyond providing software services by offering customizability and expandability features to meet a user's needs. For example, PaaS can provide a cloud-based developmental platform for users to develop, modify, and/or customize applications and/or automating enterprise operations without maintaining network infrastructure and/or allocating computing resources normally associated with these functions.
Such a cloud computing service may host a virtual agent, such as a chat agent, that is designed to automatically respond to issues with the client instance based on natural language requests from a user of the client instance. For example, a user may provide a request to a virtual agent for assistance with a password issue, wherein the virtual agent is part of a Natural Language Processing (NLP) or Natural Language Understanding (NLU) system. NLP is a general area of computer science and AI that involves some form of processing of natural language input. Examples of areas addressed by NLP include language translation, speech generation, parse tree extraction, part-of-speech identification, and others. NLU is a sub-area of NLP that specifically focuses on understanding user utterances. Examples of areas addressed by NLU include question-answering (e.g., reading comprehension questions), article summarization, and others. For example, a NLU may use algorithms to reduce human language (e.g., spoken or written) into a set of known symbols for consumption by a downstream virtual agent. NLP is generally used to interpret free text for further analysis. Current approaches to NLP are typically based on deep learning, which is a type of AI that examines and uses patterns in data to improve the understanding of a program.
As such, it is presently recognized that there is a need to improve the ability of virtual agents to apply NLU techniques to properly derive meaning from complex natural language utterances. For example, it may be advantageous to create a virtual agent capable of comprehending complex language and executing contextually relevant requests, which could afford substantial advantages in terms of reduced operational cost and increased responsiveness to client issues. Additionally, it is recognized that it is advantageous for virtual agents to be customizable and adaptable to various communication channels and styles.
SUMMARY
A summary of certain embodiments disclosed herein is set forth below. It should be understood that these aspects are presented merely to provide the reader with a brief summary of these certain embodiments and that these aspects are not intended to limit the scope of this disclosure. Indeed, this disclosure may encompass a variety of aspects that may not be set forth below.
NLU systems are used by a wide variety of clients for various domains, such as Information Technology Management (ITSM), Customer Service Management (CSM), Human Resource Management (HRM), Finance, and so forth. However, in certain embodiments, a NLU system may utilize one or more machine learning (ML)-based word vector distribution models (also referred to herein as semantic models or neural language models) that are trained based on a domain-agnostic corpus, such as an encyclopedia, a dictionary, a newspaper, to generate semantic vectors (also referred to as encodings or embeddings) for portions of utterances, including tokens of utterances, phrases of utterances, and/or entire utterances. It is presently recognized that, while this enables large, existing vector spaces to be leveraged that capture important relationships between many words and phrases (e.g., frequently used terms) within a given language, these domain-agnostic semantic models can fail to provide suitable semantic vectors for domain-specific terminology. For example, a term that is rarely or never used outside of a particular domain (e.g., a domain-specific term) may not be sufficiently represented within the domain-agnostic corpus to enable the domain-agnostic semantic model to learn a high-quality semantic vector that suitably represents the meaning of the term relative to other terms of the generic corpus represented within the vector space. Additionally, it may be desirable to leverage an existing semantic model that generates semantic vectors in a vector space having a different number of dimensions than the vector space(s) utilized by the NLU system.
With this in mind, the disclosed NLU framework includes a domain-aware vector encoding (DAVE) framework. The DAVE framework enables a designer to create a DAVE system having a domain-agnostic semantic (DAS) model and a corresponding trained vector translator (VT) model. The DAVE system uses the DAS model to generate a domain-agnostic semantic vector for a user utterance or a portion of a NLU-processed user utterance, and then uses the VT model to translate the domain-agnostic semantic vector into a domain-aware semantic vector to be used by a NLU system of the NLU framework during a meaning search operation. The VT model is also designed to provide one or more predicted intent classifications for the user utterance or the portion of a NLU-processed user utterance. Both the NLU system and the DAVE system of the NLU framework are highly configurable and refer to various NLU constraints during operation, including performance constraints and resource constraints provided by a designer or user of the NLU framework. As such, the disclosed designs ensure the NLU framework provides the desired level of performance (e.g., desired prediction latency, desired precision, desired recall, desired operational explainability) without exceeding a desired level of computational resource usage (e.g., processing time, memory usage, storage usage). The disclosed DAVE system enhances the performance (e.g., precision and/or recall) of the NLU system within the specific domain of the client, improves the quality of predictions of the NLU system for various tasks, such as intent recognition, entity recognition, and so forth. Additionally, since the DAVE system enables the use of existing DAS models in the NLU framework regardless of dimensionality, the disclosed DAVE system gives the designer freedom in selecting and immediately leveraging best-of-breed DAS models as they become available.
BRIEF DESCRIPTION OF THE DRAWINGS
Various aspects of this disclosure may be better understood upon reading the following detailed description and upon reference to the drawings in which:
FIG. 1 is a block diagram of an embodiment of a cloud computing system in which embodiments of the present technique may operate;
FIG. 2 is a block diagram of an embodiment of a multi-instance cloud architecture in which embodiments of the present technique may operate;
FIG. 3 is a block diagram of a computing device utilized in a computing system that may be present in FIG. 1 or 2 , in accordance with aspects of the present technique;
FIG. 4 A is a schematic diagram illustrating an embodiment of an agent automation framework including a NLU framework that is part of a client instance hosted by the cloud computing system, in accordance with aspects of the present technique;
FIG. 4 B is a schematic diagram illustrating an alternative embodiment of the agent automation framework in which portions of the NLU framework are part of an enterprise instance hosted by the cloud computing system, in accordance with aspects of the present technique;
FIG. 5 is a flow diagram illustrating an embodiment of a process by which an agent automation framework, including an NLU framework and a Reasoning Agent/Behavior Engine (RA/BE) framework, extracts intent/entities from and responds to a user utterance, in accordance with aspects of the present technique;
FIG. 6 is a block diagram illustrating an embodiment of the NLU framework including a meaning extraction subsystem and a meaning search subsystem, wherein the meaning extraction subsystem generates meaning representations from a received user utterance to yield an utterance meaning model and generates meaning representations from sample utterances of an intent/entity model to yield understanding model, and wherein the meaning search subsystem compares meaning representations of the utterance meaning model to meaning representations of the understanding model to extract intents and entities from the received user utterance, in accordance with aspects of the present technique;
FIG. 7 is a block diagram illustrating an embodiment of the meaning extraction subsystem using a combination of rules-based methods and machine-learning (ML)-based methods within a vocabulary subsystem, a structure subsystem, and a prosody subsystem, to generate an annotated utterance tree for an utterance, in accordance with aspects of the present technique;
FIG. 8 is a flow diagram illustrating an example process by which the meaning extraction subsystem performs error correction of an annotated utterance tree of an utterance before generating the corresponding meaning representation of the utterance, in accordance with aspects of the present technique;
FIG. 9 is a flow diagram illustrating an example process by which the meaning extraction subsystem generates a meaning representations of the understanding model or the utterance meaning model based on the annotated utterance trees and a compilation model template, in accordance with aspects of the present technique;
FIG. 10 is a block diagram illustrating an embodiment of the compilation model template, in accordance with aspects of the present technique;
FIG. 11 is a block diagram illustrating example operation of an embodiment of a tree substructure vectorization algorithm to generate a combined subtree vector for a subtree of an annotated utterance tree, in accordance with aspects of the present technique;
FIG. 12 is a flow diagram illustrating example process by which the meaning search subsystem searches the meaning representations of the understanding model for matches to the meaning representation of the user utterance, in accordance with aspects of the present technique;
FIG. 13 is a flow diagram illustrating an embodiment of a process by which a tree-model comparison algorithm compares an intent subtree of a first meaning representation to an intent subtree of a second meaning representation, based on the compilation model template, to generate an intent subtree similarity score, in accordance with aspects of the present technique;
FIG. 14 is a block diagram illustrating an embodiment of a process by which the agent automation system continuously improves a structure learning model, such as a recurrent neural network associated with a ML-based parser of the NLU framework, for improved domain specificity, based on a collection of utterances, in accordance with aspects of the present technique;
FIG. 15 is a block diagram illustrating an embodiment of a process by which the agent automation system continuously learns new words and/or refines word understanding for improved domain specificity based on a collection of utterances, in accordance with aspects of the present technique;
FIG. 16 is a diagram illustrating an embodiment of an annotated utterance tree, in accordance with aspects of the present technique; and
FIG. 17 is a diagram illustrating an embodiment of a meaning representation, in accordance with aspects of the present technique.
FIG. 18 is a block diagram illustrating how the vocabulary subsystem of FIG. 7 operates within the NLU framework, in accordance with aspects of the present techniques;
FIG. 19 is a diagram illustrating what may be included in a vocabulary model template, in accordance with aspects of the present techniques;
FIG. 20 is a block diagram illustrating example operation of an embodiment of a multi-vector aggregation algorithm to generate a combined sub-phrase vector for a subtree of an annotated utterance tree, in accordance with aspects of the present techniques;
FIG. 21 is a flow diagram illustrating how the agent automation framework continuously improves a word vector distribution model, which may be plugged into the vocabulary subsystem of the meaning extraction subsystem shown in FIG. 7 , in accordance with aspects of the present techniques; and
FIG. 22 is a flow diagram illustrating a process for receiving the user utterance, determining which meanings of one or more words or phrases appearing in the utterance were intended, and outputting one or more associated semantic word vectors, in accordance with aspects of the present technique.
FIG. 23 is a flow diagram illustrating an example process by which a model augmentation subsystem of the NLU framework augments one or more models (e.g., the utterance meaning model, the understanding model, or a combination thereof) before performing a meaning search operation, in accordance with aspects of the present technique;
FIG. 24 is a diagram of an embodiment of a model augmentation template storing generalizing rule-sets, refining rule-sets, and model applicability criteria used by the model augmentation subsystem to augment the one or more models, in accordance with aspects of the present technique;
FIG. 25 is a diagram illustrating an embodiment of model augmentation in which meaning representations of the one or more models are generalized and/or refined to yield an augmented model, in accordance with aspects of the present technique;
FIG. 26 is a flow diagram illustrating an embodiment of a process whereby the model augmentation subsystem performs rule-based generalization of the meaning representations of the one or more models, in accordance with aspects of the present technique; and
FIG. 27 is a flow diagram illustrating an embodiment of a process whereby the model augmentation subsystem performs rule-based refinement of the meaning representations of the one or more models, in accordance with aspects of the present technique.
FIG. 28 is a flow diagram illustrating an embodiment of the prosody subsystem digesting conversation logs into a number of different outputs for consumption by various components of the NLU framework, in accordance with aspects of the present technique;
FIG. 29 is a block diagram illustrating an embodiment of a process by which the agent automation system continuously improves a structure learning model, such as a recurrent neural network associated with a ML-based parser of the NLU framework, for improved domain specificity, based on intent segments identified by the prosody subsystem, in accordance with aspects of the present technique;
FIG. 30 is a block diagram illustrating an embodiment of a process by which the agent automation system continuously learns new words and/or refines word understanding for improved domain specificity, based on in-context utterances identified by the prosody subsystem, in accordance with aspects of the present technique;
FIG. 31 is a block diagram depicting a high-level view of certain components of the agent automation framework, including a semantic mining framework, in accordance with aspects of the present technique;
FIG. 32 is a block diagram of a semantic mining pipeline of the semantic mining framework illustrating a number of processing steps of a semantic mining process, in accordance with aspects of the present technique;
FIG. 33 is a diagram illustrating the prosody subsystem supporting the RA/BE in segmenting episodic context information from conversation logs, in accordance with aspects of the present technique;
FIG. 34 is a flow diagram illustrating how a persona of the RA/BE uses the prosody subsystem to manage episodic context within the agent automation framework, in accordance with aspects of the present technique;
FIG. 35 is a flow diagram illustrating an embodiment of a NLU framework that includes a NLU system and a domain-aware vector encoding (DAVE) system processing a user utterance, in accordance with aspects of the present technique;
FIG. 36 is a flow diagram illustrating an embodiment of a process whereby a DAVE framework of the NLU framework generates a trained vector translator (VT) model for a domain-agnostic semantic (DAS) model of the DAVE system, in accordance with aspects of the present technique; and
FIG. 37 is a flow diagram illustrating an embodiment of a process whereby the DAVE system uses the DAS model and the corresponding trained VT model to generate a suitable domain-aware semantic vector and a set of predicted intents from a received user utterance or a NLU-processed portion thereof, in accordance with aspects of the present technique.
DETAILED DESCRIPTION OF SPECIFIC EMBODIMENTS
One or more specific embodiments will be described below. In an effort to provide a concise description of these embodiments, not all features of an actual implementation are described in the specification. It should be appreciated that in the development of any such actual implementation, as in any engineering or design project, numerous implementation-specific decisions must be made to achieve the developers' specific goals, such as compliance with system-related and business-related constraints, which may vary from one implementation to another. Moreover, it should be appreciated that such a development effort might be complex and time consuming, but would nevertheless be a routine undertaking of design, fabrication, and manufacture for those of ordinary skill having the benefit of this disclosure.
As used herein, the terms âapplicationâ, âengineâ, âprogramâ, or âpluginâ refers to one or more sets of computer software instructions (e.g., computer programs and/or scripts) executable by one or more processors of a computing system to provide particular functionality. Computer software instructions can be written in any suitable programming languages, such as C, C++, C #, Pascal, Fortran, Perl, MATLAB, SAS, SPSS, JavaScript, AJAX, and JAVA. Such computer software instructions can comprise an independent application with data input and data display modules. Alternatively, the disclosed computer software instructions can be classes that are instantiated as distributed objects. The disclosed computer software instructions can also be component software, for example JAVABEANS or ENTERPRISE JAVABEANS. Additionally, the disclosed applications or engines can be implemented in computer software, computer hardware, or a combination thereof.
As used herein, the term âframeworkâ refers to a system of applications and/or engines, as well as any other supporting data structures, libraries, modules, and any other supporting functionality, that cooperate to perform one or more overall functions. In particular, a ânatural language understanding frameworkâ or âNLU frameworkâ comprises a collection of computer programs designed to process and derive meaning (e.g., intents, entities, artifacts) from natural language utterances using one or more machine-learning (ML) components and one or more rule-based components. As used herein, a âbehavior engineâ or âBE,â also known as a reasoning agent or RA/BE, refers to a rule-based agent, such as a virtual agent, designed to interact with users based on a conversation model. For example, a âvirtual agentâ may refer to a particular example of a BE that is designed to interact with users via natural language requests in a particular conversational or communication channel. With this in mind, the terms âvirtual agentâ and âBEâ are used interchangeably herein. By way of specific examples, a virtual agent may be or include a chat agent that interacts with users via natural language requests and responses in a chat room environment, or that provides recommended answers to requests or queries made in a search text box. Other examples of virtual agents may include an email agent, a forum agent, a ticketing agent, a telephone call agent, a search agent, a genius search result agent, and so forth, which interact with users in the context of email, forum posts, search queries, autoreplies to service tickets, phone calls, and so forth.
As used herein, an âintentâ refers to a desire or goal of a user which may relate to an underlying purpose of a communication, such as an utterance. As used herein, an âentityâ refers to an object, subject, or some other parameterization of an intent. It is noted that, for present embodiments, certain entities are treated as parameters of a corresponding intent within an intent-entity model. More specifically, certain entities (e.g., time and location) may be globally recognized and extracted for all intents, while other entities are intent-specific (e.g., merchandise entities associated with purchase intents) and are generally extracted only when found within the intents that define them. As used herein, âartifactâ collectively refers to both intents and entities of an utterance. As used herein, an âunderstanding modelâ is a collection of models used by the NLU framework to infer meaning of natural language utterances. An understanding model may include a vocabulary model that associates certain tokens (e.g., words or phrases) with particular word vectors, an intent-entity model, an intent model, an entity model, a taxonomy model, other models, or a combination thereof. As used herein an âintent-entity modelâ refers to a model that associates particular intents with particular entities and particular sample utterances, wherein entities associated with the intent may be encoded as a parameter of the intent within the sample utterances of the model. As used herein, the term âagentsâ may refer to computer-generated personas (e.g. chat agents or other virtual agents) that interact with human users within a conversational channel. As used herein, a âcorpusâ may refer to a captured body of source data that can include interactions between various users and virtual agents, wherein the interactions include communications or conversations within one or more suitable types of media (e.g., a help line, a chat room or message string, an email string). As used herein, an âutterance treeâ refers to a data structure that stores a representation of the meaning of an utterance. As discussed, an utterance tree has a tree structure (e.g., a dependency parse tree structure) that represents the syntactic structure of the utterance, wherein nodes of the tree structure store vectors (e.g., word vectors, subtree vectors) that encode the semantic meaning of the utterance.
As used herein, an âutteranceâ refers to a single natural language statement made by a user that may include one or more intents. As such, an utterance may be part of a previously captured corpus of source data, and an utterance may also be a new statement received from a user as part of an interaction with a virtual agent. As used herein, âmachine learningâ or âMLâ may be used to refer to any suitable statistical form of artificial intelligence capable of being trained using machine learning techniques, including supervised, unsupervised, and semi-supervised learning techniques. For example, in certain embodiments, ML-based techniques may be implemented using an artificial neural network (ANN) (e.g., a deep neural network (DNN), a recurrent neural network (RNN), a recursive neural network, a feedforward neural network). In contrast, ârules-basedâ methods and techniques refer to the use of rule-sets and ontologies (e.g., manually-crafted ontologies, statistically-derived ontologies) that enable precise adjudication of linguistic structure and semantic understanding to derive meaning representations from utterances. As used herein, a âvectorâ (e.g., a word vector, an intent vector, a subject vector, a subtree vector) refers to a linear algebra vector that is an ordered n-dimensional list (e.g., a 300 dimensional list) of floating point values (e.g., a 1ÃN or an NÃ1 matrix) that provides a mathematical representation of the semantic meaning of a portion (e.g., a word or phrase, an intent, an entity, a token) of an utterance. As used herein, âdomain specificityâ refers to how attuned a system is to correctly extracting intents and entities expressed in actual conversations in a given domain and/or conversational channel (e.g., a human resources domain, an information technology domain). As used herein, an âunderstandingâ of an utterance refers to an interpretation or a construction of the utterance by the NLU framework. As such, it may be appreciated that different understandings of an utterance may be associated with different meaning representations having different parse structures (e.g., different nodes, different relationships between nodes), different part-of-speech taggings, and so forth.
Agent Automation Framework
Present embodiments are directed to an agent automation framework that is capable of extracting meaning from user utterances, such as requests received by a virtual agent (e.g., a chat agent), and suitably responding to these user utterances. To do this, the agent automation framework includes a NLU framework and an intent/entity model having defined intents and entities that are associated with sample utterances. The NLU framework includes a meaning extraction subsystem that is designed to generate meaning representations for the sample utterances of the intent/entity model, as well as a meaning representation for a received user utterance. Additionally, the disclosed NLU framework includes a meaning search subsystem that is designed to search the meaning representations of the intent/entity model to locate matches for a meaning representation of a received user utterance. As such, present embodiments generally address the hard problem posed by NLU by transforming it into a manageable search problem.
In present embodiments, a meaning representation can be generated from an annotated utterance tree structure having a form or shape that represents the grammatical structures of the utterance, and having nodes that each represent words or phrases of the utterances as word vectors encoding the semantic meaning of the utterance. The meaning extraction subsystem includes a vocabulary subsystem, a structure subsystem, and a prosody subsystem that cooperate to parse utterances into the annotated utterance trees based on combinations of rule-based methods and machine learning (ML)-based (e.g., statistical) methods. Using one or more tree substructure vectorization algorithms and focus/attention/magnification (FAM) coefficients defined by a stored compilation model template, the meaning extraction subsystem subsequently generates subtree vectors for the annotated utterance tree structure, yielding the corresponding meaning representation for subsequent searching by the meaning search subsystem.
The disclosed NLU framework is also capable of detecting and addressing errors in an annotated utterance tree before the meaning representation is generated. For example, the meaning extraction subsystem can include a rule-based augmentation error detection subsystem that can cooperate with the vocabulary, structure subsystem, and prosody subsystems to iteratively parse and correct an utterance before meaning representations are generated for improved domain specificity. Additionally, present embodiments support entrenchment, whereby the NLU framework can continue to learn or infer meaning of new syntactic structures in new natural language utterance based on previous examples of similar syntactic structures. For example, components of the NLU framework (e.g., the structure subsystem or the vocabulary subsystem of the meaning extraction subsystem) may be continuously updated based on new utterances, such as exchanges between users and a virtual agent, to enhance the adaptability of the NLU framework to changes in the use of certain terms and phrases over time.
The meaning search subsystem of the disclosed NLU framework is designed to compare a meaning representation generated for a received user utterance to the set of meaning representations generated for the sample utterances of the intent/entity model based on the compilation model template. For example, the compilation model template defines one or more tree model comparison algorithms designed to determine a similarity score for two subtree vectors based on class compatibility rules and class-level scoring coefficients stored in the compilation model template. The class compatibility rules define which classes of subtree vectors can be compared to one another (e.g., verb subtree vectors are compared to one another, subject subtree vectors are compared to one another) to determine vector distances between the subtrees of the meaning representations. The class-level scoring coefficients define different relative weights that determine how much the different classes of subtree vectors contribute to an overall vector generated by the substructure vectorization algorithm for a given subtree (e.g., verb subtree vectors and/or direct object subtree vectors may be weighted higher and contribute more than subject subtree vectors or modifier subtree vectors). Using these algorithms, rules, and coefficients of the compilation model template, the meaning search subsystem determines similarity scores between portions of the meaning representation of the user utterance and portions of the meaning representations of the sample utterances of the intent/entity model. Based on these similarity scores, intents/entities defined within the intent/entity model are extracted from the user utterance and passed to a reasoning agent/behavior engine (RA/BE), such as a virtual agent, to take appropriate action based on the extracted intents/entities of the user utterance.
As mentioned, a computing platform may include a chat agent, or another similar virtual agent, that is designed to automatically respond to user requests to perform functions or address issues on the platform. There are two predominant technologies in NLU, namely traditional computational linguistics and newer machine learning (ML) methods. It is presently recognized that these two technologies demonstrate different strengths and weaknesses with respect to NLU. For example, traditional computational linguistic methods, also referred to herein as ârule-basedâ methods, include precision rule-sets and manually-crafted ontologies that enable precise adjudication of linguistic structure and semantic understanding to derive meaning representations. Traditional cognitive linguistic techniques also include the concept of construction grammars, in which an aspect of the meaning of a natural language utterance can be determined based on the form (e.g., syntactic structure) of the utterance. Therefore, rule-based methods offer results that are easily explainable and customizable. However, it is presently recognized that such rule-based methods are not particularly robust to natural language variation or adept at adapting to language evolution. As such, it is recognized that rule-based methods alone are unable to effectively react to (e.g., adjust to, learn from) data-driven trends, such as learning from chat logs and other data repositories. Furthermore, rule-based methods involve the creation of hand-crafted rules that can be cumbersome, wherein these rules usually are domain specific and are not easily transferable to other domains.
On the other hand, ML-based methods, perform well (e.g., better than rule-based methods) when a large corpus of natural language data is available for analysis and training. The ML-based methods have the ability to automatically âlearnâ from the data presented to recall over âsimilarâ input. Unlike rule-based methods, ML-based methods do not involve cumbersome hand-crafted features-engineering, and ML-based methods can support continued learning (e.g., entrenchment). However, it is recognized that ML-based methods struggle to be effective when the size of the corpus is insufficient. Additionally, ML-based methods are opaque (e.g., not easily explained) and are subject to biases in source data. Furthermore, while an exceedingly large corpus may be beneficial for ML training, source data may be subject to privacy considerations that run counter to the desired data aggregation.
Accordingly, present embodiments are generally directed toward an agent automation framework capable of applying a combination rule-based and ML-based cognitive linguistic techniques to leverage the strengths of both techniques in extracting meaning from natural language utterances. More specifically, present embodiments are directed to generating suitable meaning representations for utterances, including received user utterances and sample utterances of an intent/entity model. These meaning representations generally have a shape that captures the syntactic structure of an utterance, and include one or more subtree vectors that represent the semantic meanings of portions of the utterance. The meaning representation of the utterance can then be searched against a search space populated with the meaning representations of the sample utterances of the intent/entity model, and one or more matches may be identified. In this manner, present embodiments extract intents/entities from the user utterance, such that a virtual agent can suitably respond to these intent/entities. As such, present embodiments generally address the hard NLU problem by transforming it into a more manageable search problem.
With the preceding in mind, the following figures relate to various types of generalized system architectures or configurations that may be employed to provide services to an organization in a multi-instance framework and on which the present approaches may be employed. Correspondingly, these system and platform examples may also relate to systems and platforms on which the techniques discussed herein may be implemented or otherwise utilized. Turning now to FIG. 1 , a schematic diagram of an embodiment of a computing system 10 , such as a cloud computing system, where embodiments of the present disclosure may operate, is illustrated. Computing system 10 may include a client network 12 , network 18 (e.g., the Internet), and a cloud-based platform 20 . In some implementations, the cloud-based platform may host a management database (CMDB) system and/or other suitable systems. In one embodiment, the client network 12 may be a local private network, such as a local area network (LAN) having a variety of network devices that include, but are not limited to, switches, servers, and routers. In another embodiment, the client network 12 represents an enterprise network that could include one or more LANs, virtual networks, data centers 22 , and/or other remote networks. As shown in FIG. 1 , the client network 12 is able to connect to one or more client devices 14 A, 14 B, and 14 C so that the client devices are able to communicate with each other and/or with the network hosting the platform 20 . The client devices 14 A-C may be computing systems and/or other types of computing devices generally referred to as Internet of Things (IoT) devices that access cloud computing services, for example, via a web browser application or via an edge device 16 that may act as a gateway between the client devices and the platform 20 . FIG. 1 also illustrates that the client network 12 includes an administration or managerial device or server, such as a management, instrumentation, and discovery (MID) server 17 that facilitates communication of data between the network hosting the platform 20 , other external applications, data sources, and services, and the client network 12 . Although not specifically illustrated in FIG. 1 , the client network 12 may also include a connecting network device (e.g., a gateway or router) or a combination of devices that implement a customer firewall or intrusion protection system.
For the illustrated embodiment, FIG. 1 illustrates that client network 12 is coupled to a network 18 . The network 18 may include one or more computing networks, such as other LANs, wide area networks (WAN), the Internet, and/or other remote networks, to transfer data between the client devices 14 A-C and the network hosting the platform 20 . Each of the computing networks within network 18 may contain wired and/or wireless programmable devices that operate in the electrical and/or optical domain. For example, network 18 may include wireless networks, such as cellular networks (e.g., Global System for Mobile Communications (GSM) based cellular network), IEEE 802.11 networks, and/or other suitable radio-based networks. The network 18 may also employ any number of network communication protocols, such as Transmission Control Protocol (TCP) and Internet Protocol (IP). Although not explicitly shown in FIG. 1 , network 18 may include a variety of network devices, such as servers, routers, network switches, and/or other network hardware devices configured to transport data over the network 18 .
In FIG. 1 , the network hosting the platform 20 may be a remote network (e.g., a cloud network) that is able to communicate with the client devices 14 A-C via the client network 12 and network 18 . The network hosting the platform 20 provides additional computing resources to the client devices 14 A-C and/or client network 12 . For example, by utilizing the network hosting the platform 20 , users of client devices 14 A-C are able to build and execute applications for various enterprise, IT, and/or other organization-related functions. In one embodiment, the network hosting the platform 20 is implemented on one or more data centers 22 , where each data center could correspond to a different geographic location. Each of the data centers 22 includes a plurality of virtual servers 24 (also referred to herein as application nodes, application servers, virtual server instances, application instances, or application server instances), where each virtual server can be implemented on a physical computing system, such as a single electronic computing device (e.g., a single physical hardware server) or across multiple-computing devices (e.g., multiple physical hardware servers). Examples of virtual servers 24 include, but are not limited to a web server (e.g., a unitary web server installation), an application server (e.g., unitary JAVA Virtual Machine), and/or a database server, e.g., a unitary relational database management system (RDBMS) catalog.
To utilize computing resources within the platform 20 , network operators may choose to configure the data centers 22 using a variety of computing infrastructures. In one embodiment, one or more of the data centers 22 are configured using a multi-tenant cloud architecture, such that one of the server instances 24 handles requests from and serves multiple customers. Data centers with multi-tenant cloud architecture commingle and store data from multiple customers, where multiple customer instances are assigned to one of the virtual servers 24 . In a multi-tenant cloud architecture, the particular virtual server 24 distinguishes between and segregates data and other information of the various customers. For example, a multi-tenant cloud architecture could assign a particular identifier for each customer in order to identify and segregate the data from each customer. Generally, implementing a multi-tenant cloud architecture may suffer from various drawbacks, such as a failure of a particular one of the server instances 24 causing outages for all customers allocated to the particular server instance.
In another embodiment, one or more of the data centers 22 are configured using a multi-instance cloud architecture to provide every customer its own unique customer instance or instances. For example, a multi-instance cloud architecture could provide each customer instance with its own dedicated application server(s) and dedicated database server(s). In other examples, the multi-instance cloud architecture could deploy a single physical or virtual server and/or other combinations of physical and/or virtual servers 24 , such as one or more dedicated web servers, one or more dedicated application servers, and one or more database servers, for each customer instance. In a multi-instance cloud architecture, multiple customer instances could be installed on one or more respective hardware servers, where each customer instance is allocated certain portions of the physical server resources, such as computing memory, storage, and processing power. By doing so, each customer instance has its own unique software stack that provides the benefit of data isolation, relatively less downtime for customers to access the platform 20 , and customer-driven upgrade schedules. An example of implementing a customer instance within a multi-instance cloud architecture will be discussed in more detail below with reference to FIG. 2 .
FIG. 2 is a schematic diagram of an embodiment of a multi-instance cloud architecture 40 where embodiments of the present disclosure may operate. FIG. 2 illustrates that the multi-instance cloud architecture 40 includes the client network 12 and the network 18 that connect to two (e.g., paired) data centers 22 A and 22 B that may be geographically separated from one another. Using FIG. 2 as an example, network environment and service provider cloud infrastructure client instance 42 (also referred to herein as a simply client instance 42 ) is associated with (e.g., supported and enabled by) dedicated virtual servers (e.g., virtual servers 24 A, 24 B, 24 C, and 24 D) and dedicated database servers (e.g., virtual database servers 44 A and 44 B). Stated another way, the virtual servers 24 A- 24 D and virtual database servers 44 A and 44 B are not shared with other client instances and are specific to the respective client instance 42 . Other embodiments of the multi-instance cloud architecture 40 could include other types of dedicated virtual servers, such as a web server. For example, the client instance 42 could be associated with (e.g., supported and enabled by) the dedicated virtual servers 24 A- 24 D, dedicated virtual database servers 44 A and 44 B, and additional dedicated virtual web servers (not shown in FIG. 2 ).
In the depicted example, to facilitate availability of the client instance 42 , the virtual servers 24 A- 24 D and virtual database servers 44 A and 44 B are allocated to two different data centers 22 A and 22 B, where one of the data centers 22 acts as a backup data center. In reference to FIG. 2 , data center 22 A acts as a primary data center that includes a primary pair of virtual servers 24 A and 24 B and the primary virtual database server 44 A associated with the client instance 42 . Data center 22 B acts as a secondary data center 22 B to back up the primary data center 22 A for the client instance 42 . To back up the primary data center 22 A for the client instance 42 , the secondary data center 22 B includes a secondary pair of virtual servers 24 C and 24 D and a secondary virtual database server 44 B. The primary virtual database server 44 A is able to replicate data to the secondary virtual database server 44 B (e.g., via the network 18 ).
As shown in FIG. 2 , the primary virtual database server 44 A may back up data to the secondary virtual database server 44 B using a database replication operation. The replication of data between data centers could be implemented by performing full backups weekly and daily incremental backups in both data centers 22 A and 22 B. Having both a primary data center 22 A and secondary data center 22 B allows data traffic that typically travels to the primary data center 22 A for the client instance 42 to be diverted to the secondary data center 22 B during a failure and/or maintenance scenario. Using FIG. 2 as an example, if the virtual servers 24 A and 24 B and/or primary virtual database server instance 44 A fails and/or is under maintenance, data traffic for client instances 42 can be diverted to the secondary virtual servers 24 C and/or 24 D and the secondary virtual database server instance 44 B for processing.
Although FIGS. 1 and 2 illustrate specific embodiments of a cloud computing system 10 and a multi-instance cloud architecture 40 , respectively, the disclosure is not limited to the specific embodiments illustrated in FIGS. 1 and 2 . For instance, although FIG. 1 illustrates that the platform 20 is implemented using data centers, other embodiments of the platform 20 are not limited to data centers and can utilize other types of remote network infrastructures. Moreover, other embodiments of the present disclosure may combine one or more different virtual servers into a single virtual server or, conversely, perform operations attributed to a single virtual server using multiple virtual servers. For instance, using FIG. 2 as an example, the virtual servers 24 A-D and virtual database servers 44 A and 44 B may be combined into a single virtual server. Moreover, the present approaches may be implemented in other architectures or configurations, including, but not limited to, multi-tenant architectures, generalized client/server implementations, and/or even on a single physical processor-based device configured to perform some or all of the operations discussed herein. Similarly, though virtual servers or machines may be referenced to facilitate discussion of an implementation, physical servers may instead be employed as appropriate. The use and discussion of FIGS. 1 and 2 are only examples to facilitate ease of description and explanation and are not intended to limit the disclosure to the specific examples illustrated therein.
As may be appreciated, the respective architectures and frameworks discussed with respect to FIGS. 1 and 2 incorporate computing systems of various types (e.g., servers, workstations, client devices, laptops, tablet computers, cellular telephones, and so forth) throughout. For the sake of completeness, a brief, high level overview of components typically found in such systems is provided. As may be appreciated, the present overview is intended to merely provide a high-level, generalized view of components typical in such computing systems and should not be viewed as limiting in terms of components discussed or omitted from discussion.
With this in mind, and by way of background, it may be appreciated that the present approach may be implemented using one or more processor-based systems such as shown in FIG. 3 . Likewise, applications and/or databases utilized in the present approach may be stored, employed, and/or maintained on such processor-based systems. As may be appreciated, such systems as shown in FIG. 3 may be present in a distributed computing environment, a networked environment, or other multi-computer platform or architecture. Likewise, systems such as that shown in FIG. 3 , may be used in supporting or communicating with one or more virtual environments or computational instances on which the present approach may be implemented.
With this in mind, an example computer system may include some or all of the computer components depicted in FIG. 3 . FIG. 3 generally illustrates a block diagram of example components of a computing system 80 and their potential interconnections or communication paths, such as along one or more busses. As illustrated, the computing system 80 may include various hardware components such as, but not limited to, one or more processors 82 , one or more busses 84 , memory 86 , input devices 88 , a power source 90 , a network interface 92 , a user interface 94 , and/or other computer components use
CLAIMS
Claims ( 20 )
What is claimed is:
1. A natural language understanding (NLU) framework, comprising:
at least one memory configured to store a domain-aware vector encoding (DAVE) system that includes a plurality of domain-agnostic semantic (DAS) models and a plurality of vector translator (VT) models; and
at least one processor configured to execute stored instructions to cause the NLU framework to perform actions comprising:
selecting a DAS model from the plurality of DAS models that satisfies one or more constraints of the NLU framework;
selecting a VT model from the plurality of VT models that corresponds to the DAS model and satisfies the one or more constraints of the NLU framework;
providing, via the DAVE system, one or more portions of an utterance as input to the DAS model and, in response, receiving, as output from the DAS model, one or more domain-agnostic semantic vectors respectively representing the one or more portions of the utterance in a domain-agnostic vector space of the DAS model;
providing, via the DAVE system, the one or more domain-agnostic semantic vectors as input to the corresponding VT model and, in response, receiving, as output from the corresponding VT model, one or more domain-aware semantic vectors respectively representing the one or more portions of the utterance in a domain-aware vector space of the corresponding VT model; and
performing an utterance meaning search to extract one or more artifacts of the utterance based at least in part on the one or more domain-aware semantic vectors.
2. The NLU framework of claim 1 , wherein the DAS model comprises a Universal Sentence Encoder (USE) convolutional neural network (CNN) model, a Universal Sentence Encoder (USE) transformer model, a Bidirectional Encoder Representations from Transformers (BERT) model, a global vector (GloVe) model, a XLM-ROBERTa model, or a DistillUSE model, and wherein the corresponding VT model comprises a long short-term memory (LSTM) recurrent neural network (RNN) model, a CNN model, a transformer model, a deep adaptation network (DAN) model, or a feed-forward neural network (FFN) model.
3. The NLU framework of claim 1 , wherein, to provide the one or more portions of the utterance as input to the DAS model, the at least one processor is configured to execute the stored instructions to cause the NLU framework to perform actions comprising:
for each portion of the one or more portions of the utterance:
providing the portion of the utterance as input to the DAS model; and
in response to determining that the portion of the utterance is not orthogonal to the DAS model, determining a domain-agnostic semantic vector that represents the portion of the utterance in the domain-agnostic vector space of the DAS model.
4. The NLU framework of claim 1 , wherein the at least one memory is configured to store an intent-entity model that defines intents, and wherein the at least one processor is configured to execute the stored instructions to cause the NLU framework to perform actions comprising:
in response to providing the one or more domain-agnostic semantic vectors as input to the corresponding VT model, receiving one or more intent vectors respectively associated with the one or more portions of the utterance, wherein each intent vector comprises a respective probability score for each of the intents of the intent-entity model indicating a respective probability that the portion of the utterance associated with the intent vector corresponds to each of the intents, and wherein the NLU framework is configured to extract the one or more artifacts of the utterance based at least in part on the intent vector.
5. The NLU framework of claim 1 , wherein the at least one memory is configured to store a NLU system, and wherein the at least one processor is configured to execute the stored instructions to cause the NLU framework to perform actions comprising:
before providing the one or more portions of the utterance as input to the DAS model, performing NLU processing of the utterance to preprocess, syntactically parse, and linguistically process the utterance; and
after receiving the one or more domain-aware semantic vectors, performing, via the NLU system, a NLU meaning search based at least in part on the one or more domain-aware semantic vectors to extract the one or more artifacts of the utterance.
6. The NLU framework of claim 1 , wherein the one or more portions of the utterance comprises an entirety of the utterance, wherein the one or more domain-aware semantic vectors comprise a domain-aware semantic vector representing the entirety of the utterance, and wherein the NLU framework is configured to perform the utterance meaning search using the domain-aware semantic vector to extract the one or more artifacts of the utterance.
7. The NLU framework of claim 1 , wherein the domain-agnostic vector space of the DAS model and the domain-aware vector space of the corresponding VT model have different dimensionality, and wherein one or more domain-agnostic semantic vectors and the one or more domain-aware semantic vectors have a different number of dimensions.
8. The NLU framework of claim 1 , wherein the at least one memory is configured to store an intent-entity model that defines intents and includes sample utterances associated with each of the intents, and wherein the plurality of VT models comprises a plurality of untrained VT models, and wherein, before providing the one or more portions of the utterance as input to the DAS model, the at least one processor is configured to execute the stored instructions to cause the NLU framework to perform actions comprising:
selecting an untrained VT model that corresponds to the DAS model from the plurality of untrained VT models based on the one or more constraints of the NLU framework;
generating DAVE training data based on the sample utterances of the intent-entity model; and
training the untrained VT model using the DAVE training data to yield the corresponding VT model.
9. The NLU framework of claim 8 , wherein the one or more constraints of the NLU framework comprise prediction latency, precision, recall, operational explainability, amount and quality of available training data, training data complexity, processing time, memory usage, storage usage, or any combination thereof.
10. The NLU framework of claim 8 , wherein, to generate the DAVE training data, the at least one processor is configured to execute the stored instructions to cause the NLU framework to perform actions comprising:
performing NLU processing of the sample utterances to preprocess, syntactically parse, and linguistically process the sample utterances;
generating a map data structure from the sample utterances, wherein each entry in the map data structure stores a portion of a sample utterance and stores one or more intents that are associated with the portion of the sample utterance within the intent-entity model; and
generating the DAVE training data from the map data structure by replacing, within each entry of the map data structure, the portion of the sample utterance with a respective domain-agnostic semantic vector generated by the DAS model to represent the portion of the sample utterance, and by replacing, within each entry of the map data structure, the one or more intents with a respective intent vector representing the one or more intents.
11. The NLU framework of claim 8 , wherein, to train the untrained VT model, the at least one processor is configured to execute the stored instructions to cause the NLU framework to perform actions comprising:
training the untrained VT model over a plurality of epochs, wherein, within each epoch, the DAVE training data is randomly shuffled using a random shuffle seed that corresponds to an epoch number of the epoch.
12. A method of operating a natural language understanding (NLU) framework that comprises a domain-aware vector encoding (DAVE) system having a plurality of domain-agnostic semantic (DAS) models and a plurality of vector translator (VT) models, the method comprising:
selecting a DAS model from the plurality of DAS models that satisfies one or more constraints of the NLU framework;
selecting a VT model from the plurality of VT models that corresponds to the DAS model and satisfies the one or more constraints of the NLU framework;
providing one or more portions of an utterance as input to the DAS model and, in response, receiving, as output from the DAS model, one or more domain-agnostic semantic vectors respectively representing the one or more portions of the utterance in a domain-agnostic vector space of the DAS model;
providing the one or more domain-agnostic semantic vectors as input to the corresponding VT model and, in response, receiving, as output from the corresponding VT model, one or more domain-aware semantic vectors respectively representing the one or more portions of the utterance in a domain-aware vector space of the corresponding VT model; and
performing a NLU meaning search to extract one or more artifacts of the utterance based at least in part on the one or more domain-aware semantic vectors.
13. The method of claim 12 , comprising:
in response to providing the one or more domain-agnostic semantic vectors as input to the corresponding VT model, receiving one or more intent vectors respectively associated with the one or more portions of the utterance, wherein each intent vector comprises a respective probability score for each intent of an intent-entity model of the NLU framework indicating a respective probability that the portion of the utterance associated with the intent vector corresponds to each intent, and wherein the NLU framework is configured to extract the one or more artifacts of the utterance based at least in part on the intent vector.
14. The method of claim 12 , wherein the plurality of VT models comprises a plurality of untrained VT models, and wherein, before providing the one or more portions of the utterance as input to the DAS model, the method comprises:
selecting an untrained VT model that corresponds to the DAS model from a plurality of untrained VT models based on the one or more constraints of the NLU framework;
generating DAVE training data based on sample utterances of an intent-entity model of the NLU framework; and
training the untrained VT model using the DAVE training data over a plurality of epochs, wherein, within each epoch, the DAVE training data is randomly shuffled using a random shuffle seed that corresponds to an epoch number of the epoch.
15. The method of claim 14 , wherein generating the DAVE training data comprises:
performing NLU processing of the sample utterances to preprocess, syntactically parse, and linguistically process the sample utterances;
generating a map data structure from the sample utterances, wherein each entry in the map data structure stores a portion of a sample utterance and stores one or more intents that are associated with the portion of the sample utterance within the intent-entity model; and
generating the DAVE training data from the map data structure by replacing, within each entry of the map data structure, the portion of the sample utterance with a respective domain-agnostic semantic vector generated by the DAS model to represent the portion of the sample utterance, and by replacing, within each entry of the map data structure, the one or more intents with a respective intent vector representing the one or more intents.
16. A non-transitory, computer-readable medium storing instructions executable by a processor of a natural language understanding (NLU) framework that comprises a domain-aware vector encoding (DAVE) system having plurality of domain-agnostic semantic (DAS) models and a plurality of vector translator (VT) models, the instructions comprising instructions to:
select a DAS model from the plurality of DAS models that satisfies one or more constraints of the NLU framework;
select a VT model from the plurality of VT models that corresponds to the DAS model and satisfies the one or more constraints of the NLU framework;
provide a portion of an utterance as input to the DAS model and, in response, receiving, as output from the DAS model, a domain-agnostic semantic vector representing the portion of the utterance in a domain-agnostic vector space of the DAS model;
provide the domain-agnostic semantic vector as input to the corresponding VT model and, in response, receiving, as output from the corresponding VT model, a domain-aware semantic vector representing the portion of the utterance in a domain-aware vector space of the corresponding VT model; and
performing an utterance meaning search to extract one or more artifacts of the utterance based at least in part on the domain-aware semantic vector.
17. The medium of claim 16 , wherein the instructions comprise instructions to:
provide a second portion of the utterance as input to the DAS model and, in response, receiving, as output from the DAS model, a second domain-agnostic semantic vector representing the second portion of the utterance in the domain-agnostic vector space of the DAS model;
provide the second domain-agnostic semantic vector as input to the corresponding VT model and, in response, receiving, as output from the corresponding VT model, a second domain-aware semantic vector representing the second portion of the utterance in the domain-aware vector space of the corresponding VT model; and
extract the one or more artifacts of the utterance based at least in part on the second domain-aware semantic vector.
18. The medium of claim 16 , wherein the instructions comprise instructions to:
in response to providing the domain-agnostic semantic vector as input to the corresponding VT model, receive an intent vectors for the portion of the utterance, wherein the intent vector comprises a respective probability score for each intent of an intent-entity model of the NLU framework indicating a respective probability that the portion of the utterance corresponds to each intent, and wherein the NLU framework is configured to extract the one or more artifacts of the utterance based at least in part on the intent vector.
19. The medium of claim 18 , wherein the instructions to provide the domain-agnostic semantic vector as input to the corresponding VT model comprise instructions to:
provide the domain-agnostic semantic vector as input to a translation layer of the corresponding VT model, wherein the translation layer is configured to output the domain-aware semantic vector that represents the portion of the utterance;
provide the domain-aware semantic vector as input to a intent classification layer of the corresponding VT model, wherein the intent classification layer is configured to output a raw intent vector that includes a respective raw score for each intent of the intent-entity model;
provide the raw intent vector as input to a normalization layer of the corresponding VT model, wherein the normalization layer is configured to output a normalized intent vector that includes a respective normalized score for each intent of the intent-entity model; and
provide the normalized intent vector as input to a final layer of the corresponding VT model, wherein the final layer is configured to apply a softmax function or a sigmoid function to the normalized intent vector to generate the intent vector for the portion of the utterance.
20. The medium of claim 16 , wherein the plurality of VT models comprises a plurality of untrained VT models, wherein the instructions comprise instructions to:
before providing the one or more portions of the utterance as input to the DAS model:
select an untrained VT model that corresponds to the DAS model from a plurality of untrained VT models based on the one or more constraints of the NLU framework;
perform NLU processing of sample utterances of an intent-entity model of the NLU framework to preprocess, syntactically parse, and linguistically process the sample utterances;
generate a map data structure from the sample utterances, wherein each entry in the map data structure stores a portion of a sample utterance and stores one or more intents that are associated with the portion of the sample utterance within the intent-entity model; and
generate DAVE training data from the map data structure by replacing, within each entry of the map data structure, the portion of the sample utterance with a respective domain-agnostic semantic vector generated by the DAS model to represent the portion of the sample utterance, and by replacing, within each entry of the map data structure, the one or more intents with a respective intent vector representing the one or more intents; and
train the untrained VT model using the DAVE training data over a plurality of epochs to yield the corresponding VT model, wherein, within each epoch, the DAVE training data is randomly shuffled using a random shuffle seed that corresponds to an epoch number of the epoch.
US17/579,052
2021-01-21
2022-01-19
Domain-aware vector encoding (DAVE) system for a natural language understanding (NLU) framework
Active
2043-09-19
US12374325B2
( en )
Priority Applications (2)
Application Number
Priority Date
Filing Date
Title
US17/579,052
US12374325B2
( en )
2021-01-21
2022-01-19
Domain-aware vector encoding (DAVE) system for a natural language understanding (NLU) framework
US19/282,503
US20250356850A1
( en )
2021-01-21
2025-07-28
Domain-aware vector encoding (dave) system for a natural language understanding (nlu) framework
Applications Claiming Priority (2)
Application Number
Priority Date
Filing Date
Title
US202163140098P
2021-01-21
2021-01-21
US17/579,052
US12374325B2
( en )
2021-01-21
2022-01-19
Domain-aware vector encoding (DAVE) system for a natural language understanding (NLU) framework
Related Child Applications (1)
Application Number
Title
Priority Date
Filing Date
US19/282,503
Continuation
US20250356850A1
( en )
2021-01-21
2025-07-28
Domain-aware vector encoding (dave) system for a natural language understanding (nlu) framework
Publications (2)
Publication Number
Publication Date
US20220238103A1
US20220238103A1 ( en )
2022-07-28
US12374325B2
true
US12374325B2 ( en )
2025-07-29
Family
ID=82494883
Family Applications (2)
Application Number
Title
Priority Date
Filing Date
US17/579,052
Active
2043-09-19
US12374325B2
( en )
2021-01-21
2022-01-19
Domain-aware vector encoding (DAVE) system for a natural language understanding (NLU) framework
US19/282,503
Pending
US20250356850A1
( en )
2021-01-21
2025-07-28
Domain-aware vector encoding (dave) system for a natural language understanding (nlu) framework
Family Applications After (1)
Application Number
Title
Priority Date
Filing Date
US19/282,503
Pending
US20250356850A1
( en )
2021-01-21
2025-07-28
Domain-aware vector encoding (dave) system for a natural language understanding (nlu) framework
Country Status (1)
Country
Link
US
( 2 )
US12374325B2
( en )
Families Citing this family (14)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US10497366B2
( en )
*
2018-03-23
2019-12-03
Servicenow, Inc.
Hybrid learning system for natural language understanding
US12314675B2
( en )
*
2021-05-10
2025-05-27
Walden University, Llc
System and method for a cognitive conversation service
US12423371B2
( en )
*
2022-05-02
2025-09-23
Adobe Inc.
Utilizing machine learning models to process low-results web queries and generate web item deficiency predictions and corresponding user interfaces
US20230410682A1
( en )
*
2022-06-20
2023-12-21
International Business Machines Corporation
Task-specific language sets for multilingual learning
US12499879B1
( en )
*
2022-09-14
2025-12-16
Amazon Technologies, Inc.
Natural language understanding systems
US20240095751A1
( en )
*
2022-09-19
2024-03-21
Dell Products L.P.
Automatically predicting dispatch-related data using machine learning techniques
US12374324B2
( en )
*
2022-10-12
2025-07-29
Capital One Services, Llc
Transcript tagging and real-time whisper in interactive communications
US20240370764A1
( en )
*
2023-05-03
2024-11-07
Servicenow, Inc.
Generative machine learning framework
CN117251592A
( en )
*
2023-08-23
2023-12-19
å为ææ¯æéå ¬å¸
A data processing method and its device
TWI883580B
( en )
*
2023-10-04
2025-05-11
ç·¯åµé«å¸ç§æè¡ä»½æéå ¬å¸
Generation method and generation apparatus of medical report
US20250238612A1
( en )
*
2024-01-24
2025-07-24
Samsung Electronics Company, Ltd.
Systems and Methods for Domain-Agnostic Context Extraction in Natural Language Processing
US12430491B1
( en )
*
2024-12-19
2025-09-30
ConductorAI Corporation
Graphical user interface for syntax and policy compliance review
KR102896788B1
( en )
*
2024-12-24
2025-12-09
주ìíì¬ ì¸í°ìì¤
Method of custom manufacturing specific language model
US12505387B1
( en )
2025-02-06
2025-12-23
Eygs Llp
Systems and methods for facilitating data access based on cryptographic data generated using federated learning
Citations (42)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US6609122B1
( en )
2000-08-01
2003-08-19
Bmc Software, Inc.
Navigation of view relationships in database system
US7020706B2
( en )
2002-06-17
2006-03-28
Bmc Software, Inc.
Method and system for automatically updating multiple servers
US7028301B2
( en )
2000-12-08
2006-04-11
Bmc Software, Inc.
System and method for automatic workload characterization
US7062683B2
( en )
2003-04-22
2006-06-13
Bmc Software, Inc.
Two-phase root cause analysis
US7131037B1
( en )
2002-06-05
2006-10-31
Proactivenet, Inc.
Method and system to correlate a specific alarm to one or more events to identify a possible cause of the alarm
US7170864B2
( en )
2001-03-08
2007-01-30
Bmc Software, Inc.
System and method for WAP server management using a single console
US7350209B2
( en )
2001-06-29
2008-03-25
Bmc Software
System and method for application performance management
US7610512B2
( en )
2006-01-06
2009-10-27
Hewlett-Packard Development Company, L.P.
System and method for automated and assisted resolution of it incidents
US7617073B2
( en )
2002-03-01
2009-11-10
Bmc Software, Inc.
System and method for assessing and indicating the health of components
US7689628B2
( en )
2005-05-19
2010-03-30
Atul Garg
Monitoring several distributed resource elements as a resource pool
US7783744B2
( en )
2003-06-03
2010-08-24
Bmc Software, Inc.
Facilitating root cause analysis for abnormal behavior of systems in a networked environment
US7930396B2
( en )
2000-07-10
2011-04-19
Bmc Software, Inc.
System and method of enterprise systems and business impact management
US7945860B2
( en )
2003-05-14
2011-05-17
Hewlett-Packard Development Company, L.P.
Systems and methods for managing conversations between information technology resources
US7966398B2
( en )
2001-03-30
2011-06-21
Bmc Software, Inc.
Synthetic transaction monitor with replay capability
US8051164B2
( en )
2007-12-14
2011-11-01
Bmc Software, Inc.
Impact propagation in a directed acyclic graph having restricted views
US8224683B2
( en )
2003-07-08
2012-07-17
Hewlett-Packard Development Company, L.P.
Information technology service request level of service monitor
US8266096B2
( en )
2008-10-24
2012-09-11
Bmc Software, Inc.
Vendor portfolio management in support of vendor relationship management analysis, planning and evaluation
US8457928B2
( en )
2010-03-26
2013-06-04
Bmc Software, Inc.
Automatic determination of dynamic threshold for accurate detection of abnormalities
US8478569B2
( en )
2010-03-26
2013-07-02
Bmc Software, Inc.
Auto adjustment of baseline on configuration change
US8674992B2
( en )
2010-06-24
2014-03-18
Bmc Software, Inc.
Spotlight graphs
US8689241B2
( en )
2011-09-30
2014-04-01
Bmc Software, Inc.
Dynamic evocations for computer event management
US8743121B2
( en )
2009-12-23
2014-06-03
Bmc Software, Inc.
Smart impact views
US8887133B2
( en )
2006-04-28
2014-11-11
Bmc Software, Inc.
Bi-directional communication between change management tool and implementation tools
US9065683B2
( en )
2005-11-23
2015-06-23
Mediaclaw, Inc.
Content interactivity gateway
US9122552B2
( en )
2012-06-29
2015-09-01
Bmc Software, Inc.
Hybrid cloud infrastructures
US9239857B2
( en )
2004-12-21
2016-01-19
Bmc Software, Inc.
System and method for building business service model
US9535737B2
( en )
2015-01-30
2017-01-03
Bladelogic, Inc.
Dynamic virtual port provisioning
US9557969B2
( en )
2012-03-28
2017-01-31
Bmc Software, Inc.
Automated blueprint assembly for assembling an application
US9792387B2
( en )
2001-08-31
2017-10-17
Bmc Software, Inc.
Service desk data transfer interface
US20180341698A1
( en )
*
2017-05-27
2018-11-29
Beijing Baidu Netcom Science And Technology Co., Ltd.
Method and apparatus for parsing query based on artificial intelligence, and storage medium
US20190294676A1
( en )
2018-03-23
2019-09-26
Servicenow, Inc.
Written-modality prosody subsystem in a natural language understanding (nlu) framework
US10515625B1
( en )
*
2017-08-31
2019-12-24
Amazon Technologies, Inc.
Multi-modal natural language processing
US10740566B2
( en )
2018-03-23
2020-08-11
Servicenow, Inc.
Method and system for automated intent mining, classification and disposition
US20200302014A1
( en )
*
2019-03-18
2020-09-24
International Business Machines Corporation
Chat discourse convolution
US20200327284A1
( en )
2018-03-23
2020-10-15
Servicenow, Inc.
Hybrid learning system for natural language understanding
US20210004443A1
( en )
2019-07-02
2021-01-07
Servicenow, Inc.
Pinning artifacts for expansion of search keys and search spaces in a natural language understanding (nlu) framework
US11087090B2
( en )
2018-03-23
2021-08-10
Servicenow, Inc.
System for focused conversation context management in a reasoning agent/behavior engine of an agent automation system
US20210256966A1
( en )
*
2020-02-17
2021-08-19
Wipro Limited
Method and system for detecting intent as an ordered sequence from a user query
US11157693B2
( en )
*
2020-02-25
2021-10-26
Adobe Inc.
Stylistic text rewriting for a target author
US11196687B1
( en )
*
2018-12-10
2021-12-07
Target Brands, Inc.
Automated, extensible natural-language conversational system
US11205052B2
( en )
2019-07-02
2021-12-21
Servicenow, Inc.
Deriving multiple meaning representations for an utterance in a natural language understanding (NLU) framework
US11935539B1
( en )
*
2019-01-31
2024-03-19
Alan AI, Inc.
Integrating voice controls into applications
2022
2022-01-19
US
US17/579,052
patent/US12374325B2/en
active
Active
2025
2025-07-28
US
US19/282,503
patent/US20250356850A1/en
active
Pending
Patent Citations (53)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US7930396B2
( en )
2000-07-10
2011-04-19
Bmc Software, Inc.
System and method of enterprise systems and business impact management
US6609122B1
( en )
2000-08-01
2003-08-19
Bmc Software, Inc.
Navigation of view relationships in database system
US7028301B2
( en )
2000-12-08
2006-04-11
Bmc Software, Inc.
System and method for automatic workload characterization
US7170864B2
( en )
2001-03-08
2007-01-30
Bmc Software, Inc.
System and method for WAP server management using a single console
US7966398B2
( en )
2001-03-30
2011-06-21
Bmc Software, Inc.
Synthetic transaction monitor with replay capability
US7350209B2
( en )
2001-06-29
2008-03-25
Bmc Software
System and method for application performance management
US9792387B2
( en )
2001-08-31
2017-10-17
Bmc Software, Inc.
Service desk data transfer interface
US7617073B2
( en )
2002-03-01
2009-11-10
Bmc Software, Inc.
System and method for assessing and indicating the health of components
US7131037B1
( en )
2002-06-05
2006-10-31
Proactivenet, Inc.
Method and system to correlate a specific alarm to one or more events to identify a possible cause of the alarm
US7020706B2
( en )
2002-06-17
2006-03-28
Bmc Software, Inc.
Method and system for automatically updating multiple servers
US7062683B2
( en )
2003-04-22
2006-06-13
Bmc Software, Inc.
Two-phase root cause analysis
US7945860B2
( en )
2003-05-14
2011-05-17
Hewlett-Packard Development Company, L.P.
Systems and methods for managing conversations between information technology resources
US7783744B2
( en )
2003-06-03
2010-08-24
Bmc Software, Inc.
Facilitating root cause analysis for abnormal behavior of systems in a networked environment
US8224683B2
( en )
2003-07-08
2012-07-17
Hewlett-Packard Development Company, L.P.
Information technology service request level of service monitor
US9239857B2
( en )
2004-12-21
2016-01-19
Bmc Software, Inc.
System and method for building business service model
US7689628B2
( en )
2005-05-19
2010-03-30
Atul Garg
Monitoring several distributed resource elements as a resource pool
US9065683B2
( en )
2005-11-23
2015-06-23
Mediaclaw, Inc.
Content interactivity gateway
US7610512B2
( en )
2006-01-06
2009-10-27
Hewlett-Packard Development Company, L.P.
System and method for automated and assisted resolution of it incidents
US7890802B2
( en )
2006-01-06
2011-02-15
Hewlett-Packard Development Company, L.P.
System and method for automated and assisted resolution of IT incidents
US8887133B2
( en )
2006-04-28
2014-11-11
Bmc Software, Inc.
Bi-directional communication between change management tool and implementation tools
US8051164B2
( en )
2007-12-14
2011-11-01
Bmc Software, Inc.
Impact propagation in a directed acyclic graph having restricted views
US8266096B2
( en )
2008-10-24
2012-09-11
Bmc Software, Inc.
Vendor portfolio management in support of vendor relationship management analysis, planning and evaluation
US8743121B2
( en )
2009-12-23
2014-06-03
Bmc Software, Inc.
Smart impact views
US8457928B2
( en )
2010-03-26
2013-06-04
Bmc Software, Inc.
Automatic determination of dynamic threshold for accurate detection of abnormalities
US8478569B2
( en )
2010-03-26
2013-07-02
Bmc Software, Inc.
Auto adjustment of baseline on configuration change
US8674992B2
( en )
2010-06-24
2014-03-18
Bmc Software, Inc.
Spotlight graphs
US8689241B2
( en )
2011-09-30
2014-04-01
Bmc Software, Inc.
Dynamic evocations for computer event management
US9557969B2
( en )
2012-03-28
2017-01-31
Bmc Software, Inc.
Automated blueprint assembly for assembling an application
US9122552B2
( en )
2012-06-29
2015-09-01
Bmc Software, Inc.
Hybrid cloud infrastructures
US9535737B2
( en )
2015-01-30
2017-01-03
Bladelogic, Inc.
Dynamic virtual port provisioning
US20180341698A1
( en )
*
2017-05-27
2018-11-29
Beijing Baidu Netcom Science And Technology Co., Ltd.
Method and apparatus for parsing query based on artificial intelligence, and storage medium
US10515625B1
( en )
*
2017-08-31
2019-12-24
Amazon Technologies, Inc.
Multi-modal natural language processing
US20200349325A1
( en )
2018-03-23
2020-11-05
Servicenow, Inc.
Method and system for automated intent mining, classification and disposition
US20210224485A1
( en )
2018-03-23
2021-07-22
Servicenow, Inc.
Templated rule-based data augmentation for intent extraction
US10713441B2
( en )
2018-03-23
2020-07-14
Servicenow, Inc.
Hybrid learning system for natural language intent extraction from a dialog utterance
US10740566B2
( en )
2018-03-23
2020-08-11
Servicenow, Inc.
Method and system for automated intent mining, classification and disposition
US11087090B2
( en )
2018-03-23
2021-08-10
Servicenow, Inc.
System for focused conversation context management in a reasoning agent/behavior engine of an agent automation system
US20200327284A1
( en )
2018-03-23
2020-10-15
Servicenow, Inc.
Hybrid learning system for natural language understanding
US20210342547A1
( en )
2018-03-23
2021-11-04
Servicenow, Inc.
System for focused conversation context management in a reasoning agent/behavior engine of an agent automation system
US10956683B2
( en )
2018-03-23
2021-03-23
Servicenow, Inc.
Systems and method for vocabulary management in a natural learning framework
US10497366B2
( en )
2018-03-23
2019-12-03
Servicenow, Inc.
Hybrid learning system for natural language understanding
US20190294676A1
( en )
2018-03-23
2019-09-26
Servicenow, Inc.
Written-modality prosody subsystem in a natural language understanding (nlu) framework
US10970487B2
( en )
2018-03-23
2021-04-06
Servicenow, Inc.
Templated rule-based data augmentation for intent extraction
US20210200960A1
( en )
2018-03-23
2021-07-01
Servicenow, Inc.
Systems and method for vocabulary management in a natural learning framework
US11196687B1
( en )
*
2018-12-10
2021-12-07
Target Brands, Inc.
Automated, extensible natural-language conversational system
US11935539B1
( en )
*
2019-01-31
2024-03-19
Alan AI, Inc.
Integrating voice controls into applications
US20200302014A1
( en )
*
2019-03-18
<td itemprop="publication