Conceptio › Archive › Google Patents

Google Patentsopen access

Hybrid learning system for natural language understanding — Servicenow, Inc. (US11520992B2)

Servicenow, Inc. · Google Patents

Google Patents · Patents · License: Open Access

Open Source ↗

edwinsapugay servicenow

patent, google patents, intellectual property, US11520992B2, Servicenow, Inc., Edwin Sapugay, en, 2022

ABSTRACT

Abstract

An agent automation system includes a memory configured to store a natural language understanding (NLU) framework and a processor configured to execute instructions of the NLU framework to cause the agent automation system to perform actions. These actions comprise: generating an annotated utterance tree of an utterance using a combination of rules-based and machine-learning (ML)-based components, wherein a structure of the annotated utterance tree represents a syntactic structure of the utterance, and wherein nodes of the annotated utterance tree include word vectors that represent semantic meanings of words of the utterance; and using the annotated utterance tree as a basis for intent/entity extraction of the utterance.

Description

CROSS-REFERENCE

This application is a continuation of U.S. patent application Ser. No. 16/238,324, entitled, âHYBRID LEARNING SYSTEM FOR NATURAL LANGUAGE UNDERSTANDING,â filed Jan. 2, 2019, which claims priority from and the benefit of U.S. Provisional Application No. 62/646,915, entitled âHYBRID LEARNING SYSTEM FOR NATURAL LANGUAGE UNDERSTANDING,â filed Mar. 23, 2018; U.S. Provisional Application No. 62/646,916, entitled âVOCABULARY MANAGEMENT IN A NATURAL LEARNING FRAMEWORK,â filed Mar. 23, 2018; U.S. Provisional Application No. 62/646,917, entitled âMETHOD AND SYSTEM FOR AUTOMATED INTENT MINING, CLASSIFICATION AND DISPOSITION,â filed Mar. 23, 2018; U.S. Provisional Application No. 62/657,751, entitled âMETHOD AND SYSTEM FOR FOCUSED CONVERSATION CONTEXT MANAGEMENT IN A BEHAVIOR ENGINE,â filed Apr. 14, 2018; U.S. Provisional Application No. 62/652,903, entitled âTEMPLATED RULE-BASED DATA AUGMENTATION FOR INTENT EXTRACTION FROM SPARSE DATA,â filed Apr. 5, 2018; and U.S. Provisional Application No. 62/659,710, entitled âWRITTEN-MODALITY PROSODY SUBSYSTEM IN A NLU FRAMEWORK,â filed Apr. 19, 2018, which are incorporated by reference herein in their entirety for all purposes.

BACKGROUND

The present disclosure relates generally to the fields of natural language understanding (NLU) and artificial intelligence (AI), and more specifically, to a hybrid learning system for NLU.

This section is intended to introduce the reader to various aspects of art that may be related to various aspects of the present disclosure, which are described and/or claimed below. This discussion is believed to be helpful in providing the reader with background information to facilitate a better understanding of the various aspects of the present disclosure. Accordingly, it should be understood that these statements are to be read in this light, and not as admissions of prior art.

Cloud computing relates to the sharing of computing resources that are generally accessed via the Internet. In particular, a cloud computing infrastructure allows users, such as individuals and/or enterprises, to access a shared pool of computing resources, such as servers, storage devices, networks, applications, and/or other computing based services. By doing so, users are able to access computing resources on demand that are located at remote locations and these resources may be used to perform a variety computing functions (e.g., storing and/or processing large quantities of computing data). For enterprise and other organization users, cloud computing provides flexibility in accessing cloud computing resources without accruing large up-front costs, such as purchasing expensive network equipment or investing large amounts of time in establishing a private network infrastructure. Instead, by utilizing cloud computing resources, users are able redirect their resources to focus on their enterprise's core functions.

In modern communication networks, examples of cloud computing services a user may utilize include so-called infrastructure as a service (IaaS), software as a service (SaaS), and platform as a service (PaaS) technologies. IaaS is a model in which providers abstract away the complexity of hardware infrastructure and provide rapid, simplified provisioning of virtual servers and storage, giving enterprises access to computing capacity on demand. In such an approach, however, a user may be left to install and maintain platform components and applications. SaaS is a delivery model that provides software as a service rather than an end product. Instead of utilizing a local network or individual software installations, software is typically licensed on a subscription basis, hosted on a remote machine, and accessed by client customers as needed. For example, users are generally able to access a variety of enterprise and/or information technology (IT)-related software via a web browser. PaaS acts an extension of SaaS that goes beyond providing software services by offering customizability and expandability features to meet a user's needs. For example, PaaS can provide a cloud-based developmental platform for users to develop, modify, and/or customize applications and/or automating enterprise operations without maintaining network infrastructure and/or allocating computing resources normally associated with these functions.

Such a cloud computing service may host a virtual agent, such as a chat agent, that is designed to automatically respond to issues with the client instance based on natural language requests from a user of the client instance. For example, a user may provide a request to a virtual agent for assistance with a password issue, wherein the virtual agent is part of a Natural Language Processing (NLP) or Natural Language Understanding (NLU) system. NLP is a general area of computer science and AI that involves some form of processing of natural language input. Examples of areas addressed by NLP include language translation, speech generation, parse tree extraction, part-of-speech identification, and others. NLU is a sub-area of NLP that specifically focuses on understanding user utterances. Examples of areas addressed by NLU include question-answering (e.g., reading comprehension questions), article summarization, and others. For example, a NLU may use algorithms to reduce human language (e.g., spoken or written) into a set of known symbols for consumption by a downstream virtual agent. NLP is generally used to interpret free text for further analysis. Current approaches to NLP are typically based on deep learning, which is a type of AI that examines and uses patterns in data to improve the understanding of a program.

As such, it is presently recognized that there is a need to improve the ability of virtual agents to apply NLU techniques to properly derive meaning from complex natural language utterances. For example, it may be advantageous to create a virtual agent capable of comprehending complex language and executing contextually relevant requests, which could afford substantial advantages in terms of reduced operational cost and increased responsiveness to client issues. Additionally, it is recognized that it is advantageous for virtual agents to be customizable and adaptable to various communication channels and styles.

SUMMARY

A summary of certain embodiments disclosed herein is set forth below. It should be understood that these aspects are presented merely to provide the reader with a brief summary of these certain embodiments and that these aspects are not intended to limit the scope of this disclosure. Indeed, this disclosure may encompass a variety of aspects that may not be set forth below.

Present embodiments are directed to an agent automation framework that is capable of extracting meaning from user utterances, such as requests received by a virtual agent (e.g., a chat agent), and suitably responding to these user utterances. To do this, the agent automation framework includes a NLU framework and an intent/entity model having defined intents and entities that are associated with sample utterances. The NLU framework includes a meaning extraction subsystem that is designed to generate meaning representations for the sample utterances of the intent/entity model, as well as a meaning representation for a received user utterance. Additionally, the disclosed NLU framework includes a meaning search subsystem that is designed to search the meaning representations of the intent/entity model to locate matches for a meaning representation of a received user utterance. As such, present embodiments generally address the hard problem posed by NLU by transforming it into a manageable search problem.

In present embodiments, a meaning representation can be generated from an annotated utterance tree structure having a form or shape that represents the grammatical structures of the utterance, and having nodes that each represent words or phrases of the utterances as word vectors encoding the semantic meaning of the utterance. The meaning extraction subsystem includes a vocabulary subsystem, a structure subsystem, and a prosody subsystem that cooperate to parse utterances into the annotated utterance trees based on combinations of rule-based methods and machine learning (ML)-based (e.g., statistical) methods. Using one or more tree substructure vectorization algorithms and focus/attention/magnification (FAM) coefficients defined by a stored compilation model template, the meaning extraction subsystem subsequently generates subtree vectors for the annotated utterance tree structure, yielding the corresponding meaning representation for subsequent searching by the meaning search subsystem.

The disclosed NLU framework is also capable of detecting and addressing errors in an annotated utterance tree before the meaning representation is generated. For example, the meaning extraction subsystem can include a rule-based augmentation error detection subsystem that can cooperate with the vocabulary, structure subsystem, and prosody subsystems to iteratively parse and correct an utterance before meaning representations are generated for improved domain specificity. Additionally, present embodiments support entrenchment, whereby the NLU framework can continue to learn or infer meaning of new syntactic structures in new natural language utterance based on previous examples of similar syntactic structures. For example, components of the NLU framework (e.g., the structure subsystem or the vocabulary subsystem of the meaning extraction subsystem) may be continuously updated based on new utterances, such as exchanges between users and a virtual agent, to enhance the adaptability of the NLU framework to changes in the use of certain terms and phrases over time.

The meaning search subsystem of the disclosed NLU framework is designed to compare a meaning representation generated for a received user utterance to the set of meaning representations generated for the sample utterances of the intent/entity model based on the compilation model template. For example, the compilation model template defines one or more tree model comparison algorithms designed to determine a similarity score for two subtree vectors based on class compatibility rules and class-level scoring coefficients stored in the compilation model template. The class compatibility rules define which classes of subtree vectors can be compared to one another (e.g., verb subtree vectors are compared to one another, subject subtree vectors are compared to one another) to determine vector distances between the subtrees of the meaning representations. The class-level scoring coefficients define different relative weights that determine how much the different classes of subtree vectors contribute to an overall vector generated by the substructure vectorization algorithm for a given subtree (e.g., verb subtree vectors and/or direct object subtree vectors may be weighted higher and contribute more than subject subtree vectors or modifier subtree vectors). Using these algorithms, rules, and coefficients of the compilation model template, the meaning search subsystem determines similarity scores between portions of the meaning representation of the user utterance and portions of the meaning representations of the sample utterances of the intent/entity model. Based on these similarity scores, intents/entities defined within the intent/entity model are extracted from the user utterance and passed to a reasoning agent/behavior engine (RA/BE), such as a virtual agent, to take appropriate action based on the extracted intents/entities of the user utterance.

BRIEF DESCRIPTION OF THE DRAWINGS

Various aspects of this disclosure may be better understood upon reading the following detailed description and upon reference to the drawings in which:

FIG. 1 is a block diagram of an embodiment of a cloud computing system in which embodiments of the present technique may operate;

FIG. 2 is a block diagram of an embodiment of a multi-instance cloud architecture in which embodiments of the present technique may operate;

FIG. 3 is a block diagram of a computing device utilized in a computing system that may be present in FIG. 1 or 2 , in accordance with aspects of the present technique;

FIG. 4 A is a schematic diagram illustrating an embodiment of an agent automation framework including a NLU framework that is part of a client instance hosted by the cloud computing system, in accordance with aspects of the present technique;

FIG. 4 B is a schematic diagram illustrating an alternative embodiment of the agent automation framework in which portions of the NLU framework are part of an enterprise instance hosted by the cloud computing system, in accordance with aspects of the present technique;

FIG. 5 is a flow diagram illustrating an embodiment of a process by which an agent automation framework, including an NLU framework and a Reasoning Agent/Behavior Engine (RA/BE) framework, extracts intent/entities from and responds to a user utterance, in accordance with aspects of the present technique;

FIG. 6 is a block diagram illustrating an embodiment of the NLU framework including a meaning extraction subsystem and a meaning search subsystem, wherein the meaning extraction subsystem generates meaning representations from a received user utterance to yield an utterance meaning model and generates meaning representations from sample utterances of an intent/entity model to yield understanding model, and wherein the meaning search subsystem compares meaning representations of the utterance meaning model to meaning representations of the understanding model to extract intents and entities from the received user utterance, in accordance with aspects of the present technique;

FIG. 7 is a block diagram illustrating an embodiment of the meaning extraction subsystem using a combination of rules-based methods and machine-learning (ML)-based methods within a vocabulary subsystem, a structure subsystem, and a prosody subsystem, to generate an annotated utterance tree for an utterance, in accordance with aspects of the present technique;

FIG. 8 is a flow diagram illustrating an example process by which the meaning extraction subsystem performs error correction of an annotated utterance tree of an utterance before generating the corresponding meaning representation of the utterance, in accordance with aspects of the present technique;

FIG. 9 is a flow diagram illustrating an example process by which the meaning extraction subsystem generates a meaning representations of the understanding model or the utterance meaning model based on the annotated utterance trees and a compilation model template, in accordance with aspects of the present technique;

FIG. 10 is a block diagram illustrating an embodiment of the compilation model template, in accordance with aspects of the present technique;

FIG. 11 is a block diagram illustrating example operation of an embodiment of a tree substructure vectorization algorithm to generate a combined subtree vector for a subtree of an annotated utterance tree, in accordance with aspects of the present technique;

FIG. 12 is a flow diagram illustrating example process by which the meaning search subsystem searches the meaning representations of the understanding model for matches to the meaning representation of the user utterance, in accordance with aspects of the present technique;

FIG. 13 is a flow diagram illustrating an embodiment of a process by which a tree-model comparison algorithm compares an intent subtree of a first meaning representation to an intent subtree of a second meaning representation, based on the compilation model template, to generate an intent subtree similarity score, in accordance with aspects of the present technique;

FIG. 14 is a block diagram illustrating an embodiment of a process by which the agent automation system continuously improves a structure le

CROSS-REFERENCE

BACKGROUND

The present disclosure relates generally to the fields of natural language understanding (NLU) and artificial intelligence (AI), and more specifically, to a hybrid learning system for NLU.

SUMMARY

BRIEF DESCRIPTION OF THE DRAWINGS

Various aspects of this disclosure may be better understood upon reading the following detailed description and upon reference to the drawings in which:

FIG. 1 is a block diagram of an embodiment of a cloud computing system in which embodiments of the present technique may operate;

FIG. 2 is a block diagram of an embodiment of a multi-instance cloud architecture in which embodiments of the present technique may operate;

FIG. 3 is a block diagram of a computing device utilized in a computing system that may be present in FIG. 1 or 2 , in accordance with aspects of the present technique;

FIG. 10 is a block diagram illustrating an embodiment of the compilation model template, in accordance with aspects of the present technique;

FIG. 14 is a block diagram illustrating an embodiment of a process by which the agent automation system continuously improves a structure learning model, such as a recurrent neural network associated with a ML-based parser of the NLU framework, for improved domain specificity, based on a collection of utterances, in accordance with aspects of the present technique;

FIG. 15 is a block diagram illustrating an embodiment of a process by which the agent automation system continuously learns new words and/or refines word understanding for improved domain specificity based on a collection of utterances, in accordance with aspects of the present technique;

FIG. 16 is a diagram illustrating an embodiment of an annotated utterance tree, in accordance with aspects of the present technique; and

FIG. 17 is a diagram illustrating an embodiment of a meaning representation, in accordance with aspects of the present technique.

DETAILED DESCRIPTION OF SPECIFIC EMBODIMENTS

One or more specific embodiments will be described below. In an effort to provide a concise description of these embodiments, not all features of an actual implementation are described in the specification. It should be appreciated that in the development of any such actual implementation, as in any engineering or design project, numerous implementation-specific decisions must be made to achieve the developers' specific goals, such as compliance with system-related and business-related constraints, which may vary from one implementation to another. Moreover, it should be appreciated that such a development effort might be complex and time consuming, but would nevertheless be a routine undertaking of design, fabrication, and manufacture for those of ordinary skill having the benefit of this disclosure.

As used herein, the term âcomputing systemâ or âcomputing deviceâ refers to an electronic computing device such as, but not limited to, a single computer, virtual machine, virtual container, host, server, laptop, and/or mobile device, or to a plurality of electronic computing devices working together to perform the function described as being performed on or by the computing system. As used herein, the term âmachine-readable mediumâ may include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store one or more instructions or data structures. The term ânon-transitory machine-readable mediumâ shall also be taken to include any tangible medium that is capable of storing, encoding, or carrying instructions for execution by the computing system and that cause the computing system to perform any one or more of the methodologies of the present subject matter, or that is capable of storing, encoding, or carrying data structures utilized by or associated with such instructions. The term ânon-transitory machine-readable mediumâ shall accordingly be taken to include, but not be limited to, solid-state memories, and optical and magnetic media. Specific examples of non-transitory machine-readable media include, but are not limited to, non-volatile memory, including by way of example, semiconductor memory devices (e.g., Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), and flash memory devices), magnetic disks such as internal hard disks and removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

As used herein, the terms âapplicationâ and âengineâ refer to one or more sets of computer software instructions (e.g., computer programs and/or scripts) executable by one or more processors of a computing system to provide particular functionality. Computer software instructions can be written in any suitable programming languages, such as C, C++, C#, Pascal, Fortran, Perl, MATLAB, SAS, SPSS, JavaScript, AJAX, and JAVA. Such computer software instructions can comprise an independent application with data input and data display modules. Alternatively, the disclosed computer software instructions can be classes that are instantiated as distributed objects. The disclosed computer software instructions can also be component software, for example JAVABEANS or ENTERPRISE JAVABEANS. Additionally, the disclosed applications or engines can be implemented in computer software, computer hardware, or a combination thereof.

As used herein, the term âframeworkâ refers to a system of applications and/or engines, as well as any other supporting data structures, libraries, modules, and any other supporting functionality, that cooperate to perform one or more overall functions. In particular, a ânatural language understanding frameworkâ or âNLU frameworkâ comprises a collection of computer programs designed to process and derive meaning (e.g., intents, entities) from natural language utterances based on an intent/entity model. As used herein, a âreasoning agent/behavior engineâ or âRA/BEâ refers to a rule-based agent, such as a virtual agent, designed to interact with users based on a conversation model. For example, a âvirtual agentâ may refer to a particular example of a RA/BE that is designed to interact with users via natural language requests in a particular conversational or communication channel. With this in mind, the terms âvirtual agentâ and âRA/BEâ are used interchangeably herein. By way of specific example, a virtual agent may be or include a chat agent that interacts with users via natural language requests and responses in a chat room environment. Other examples of virtual agents may include an email agent, a forum agent, a ticketing agent, a telephone call agent, and so forth, which interact with users in the context of email, forum posts, and autoreplies to service tickets, phone calls, and so forth.

As used herein, an âintentâ refers to a desire or goal of an agent which may relate to an underlying purpose of a communication, such as an utterance. As used herein, an âentityâ refers to an object, subject, or some other parameterization of an intent. It is noted that, for present embodiments, entities are treated as parameters of a corresponding intent. More specifically, certain entities (e.g., time and location) may be globally recognized and extracted for all intents, while other entities are intent-specific (e.g., merchandise entities associated with purchase intents) and are generally extracted only when found within the intents that define them. As used herein, an âintent/entity modelâ refers to an intent model that associates particular intents with particular sample utterances, wherein certain entity data may be encoded as a parameter of the intent within the model. As used herein, the term âagentsâ may refer to computer-generated personas (e.g. chat agents or other virtual agents) that interact with one another within a conversational channel. As used herein, a âcorpusâ refers to a captured body of source data that includes interactions between various users and virtual agents, wherein the interactions include communications or conversations within one or more suitable types of media (e.g., a help line, a chat room or message string, an email string).

As used herein, âsource dataâ may include any suitable captured interactions between various agents, including but not limited to, chat logs, email strings, documents, help documentation, frequently asked questions (FAQs), forum entries, items in support ticketing, recordings of help line calls, and so forth. As used herein, an âutteranceâ refers to a single natural language statement made by a user or agent that may include one or more intents. As such, an utterance may be part of a previously captured corpus of source data, and an utterance may also be a new statement received from a user as part of an interaction with a virtual agent. As used herein, âmachine learningâ or âMLâ may be used to refer to any suitable statistical form of artificial intelligence capable of being trained using machine learning techniques, including supervised, unsupervised, and semi-supervised learning techniques. For example, in certain embodiments, ML techniques may be implemented using a neural network (NN) (e.g., a deep neural network (DNN), a recurrent neural network (RNN), a recursive neural network). As used herein, a âvectorâ (e.g., a word vector, an intent vector, a subject vector, a subtree vector) refers to a linear algebra vector that is an ordered n-dimensional list (e.g., a 300 dimensional list) of floating point values (e.g., a 1ÃN or an NÃ1 matrix) that provides a mathematical representation of the semantic meaning of a portion (e.g., a word or phrase, an intent, an entity) of an utterance.

As mentioned, a computing platform may include a chat agent, or another similar virtual agent, that is designed to automatically respond to user requests to perform functions or address issues on the platform. There are two predominant technologies in NLU, namely traditional computational linguistics and newer machine learning (ML) methods. It is presently recognized that these two technologies demonstrate different strengths and weaknesses with respect to NLU. For example, traditional computational linguistic methods, also referred to herein as ârule-basedâ methods, include precision rule-sets and manually-crafted ontologies that enable precise adjudication of linguistic structure and semantic understanding to derive meaning representations. Traditional cognitive linguistic techniques also include the concept of construction grammars, in which an aspect of the meaning of a natural language utterance can be determined based on the form (e.g., syntactic structure) of the utterance. Therefore, rule-based methods offer results that are easily explainable and customizable. However, it is presently recognized that such rule-based methods are not particularly robust to natural language variation or adept at adapting to language evolution. As such, it is recognized that rule-based methods alone are unable to effectively react to (e.g., adjust to, learn from) data-driven trends, such as learning from chat logs and other data repositories. Furthermore, rule-based methods involve the creation of hand-crafted rules that can be cumbersome, wherein these rules usually are domain specific and are not easily transferable to other domains.

On the other hand, ML-based methods, perform well (e.g., better than rule-based methods) when a large corpus of natural language data is available for analysis and training. The ML-based methods have the ability to automatically âlearnâ from the data presented to recall over âsimilarâ input. Unlike rule-based methods, ML-based methods do not involve cumbersome hand-crafted features-engineering, and ML-based methods can support continued learning (e.g., entrenchment). However, it is recognized that ML-based methods struggle to be effective when the size of the corpus is insufficient. Additionally, ML-based methods are opaque (e.g., not easily explained) and are subject to biases in source data. Furthermore, while an exceedingly large corpus may be beneficial for ML training, source data may be subject to privacy considerations that run counter to the desired data aggregation.

Accordingly, present embodiments are generally directed toward an agent automation framework capable of applying a combination rule-based and ML-based cognitive linguistic techniques to leverage the strengths of both techniques in extracting meaning from natural language utterances. More specifically, present embodiments are directed to generating suitable meaning representations for utterances, including received user utterances and sample utterances of an intent/entity model. These meaning representations generally have a shape that captures the syntactic structure of an utterance, and include one or more subtree vectors that represent the semantic meanings of portions of the utterance. The meaning representation of the utterance can then be searched against a search space populated with the meaning representations of the sample utterances of the intent/entity model, and one or more matches may be identified. In this manner, present embodiments extract intents/entities from the user utterance, such that a virtual agent can suitably respond to these intent/entities. As such, present embodiments generally address the hard NLU problem by transforming it into a more manageable search problem.

With the preceding in mind, the following figures relate to various types of generalized system architectures or configurations that may be employed to provide services to an organization in a multi-instance framework and on which the present approaches may be employed. Correspondingly, these system and platform examples may also relate to systems and platforms on which the techniques discussed herein may be implemented or otherwise utilized. Turning now to FIG. 1 , a schematic diagram of an embodiment of a computing system 10 , such as a cloud computing system, where embodiments of the present disclosure may operate, is illustrated. Computing system 10 may include a client network 12 , network 18 (e.g., the Internet), and a cloud-based platform 20 . In some implementations, the cloud-based platform may host a management database (CMDB) system and/or other suitable systems. In one embodiment, the client network 12 may be a local private network, such as local area network (LAN) having a variety of network devices that include, but are not limited to, switches, servers, and routers. In another embodiment, the client network 12 represents an enterprise network that could include one or more LANs, virtual networks, data centers 22 , and/or other remote networks. As shown in FIG. 1 , the client network 12 is able to connect to one or

more client devices

14 A, 14 B, and 14 C so that the client devices are able to communicate with each other and/or with the network hosting the platform 20 . The client devices 14 A-C may be computing systems and/or other types of computing devices generally referred to as Internet of Things (IoT) devices that access cloud computing services, for example, via a web browser application or via an edge device 16 that may act as a gateway between the client devices and the platform 20 . FIG. 1 also illustrates that the client network 12 includes an administration or managerial device or server, such as a management, instrumentation, and discovery (MID) server 17 that facilitates communication of data between the network hosting the platform 20 , other external applications, data sources, and services, and the client network 12 . Although not specifically illustrated in FIG. 1 , the client network 12 may also include a connecting network device (e.g., a gateway or router) or a combination of devices that implement a customer firewall or intrusion protection system.

For the illustrated embodiment, FIG. 1 illustrates that client network 12 is coupled to a network 18 . The network 18 may include one or more computing networks, such as other LANs, wide area networks (WAN), the Internet, and/or other remote networks, to transfer data between the client devices 14 A-C and the network hosting the platform 20 . Each of the computing networks within network 18 may contain wired and/or wireless programmable devices that operate in the electrical and/or optical domain. For example, network 18 may include wireless networks, such as cellular networks (e.g., Global System for Mobile Communications (GSM) based cellular network), IEEE 802.11 networks, and/or other suitable radio-based networks. The network 18 may also employ any number of network communication protocols, such as Transmission Control Protocol (TCP) and Internet Protocol (IP). Although not explicitly shown in FIG. 1 , network 18 may include a variety of network devices, such as servers, routers, network switches, and/or other network hardware devices configured to transport data over the network 18 .

In FIG. 1 , the network hosting the platform 20 may be a remote network (e.g., a cloud network) that is able to communicate with the client devices 14 A-C via the client network 12 and network 18 . The network hosting the platform 20 provides additional computing resources to the client devices 14 A-C and/or client network 12 . For example, by utilizing the network hosting the platform 20 , users of client devices 14 A-C are able to build and execute applications for various enterprise, IT, and/or other organization-related functions. In one embodiment, the network hosting the platform 20 is implemented on one or more data centers 22 , where each data center could correspond to a different geographic location. Each of the data centers 22 includes a plurality of virtual servers 24 (also referred to herein as application nodes, application servers, virtual server instances, application instances, or application server instances), where each virtual server can be implemented on a physical computing system, such as a single electronic computing device (e.g., a single physical hardware server) or across multiple-computing devices (e.g., multiple physical hardware servers). Examples of virtual servers 24 include, but are not limited to a web server (e.g., a unitary web server installation), an application server (e.g., unitary JAVA Virtual Machine), and/or a database server, e.g., a unitary relational database management system (RDBMS) catalog.

To utilize computing resources within the platform 20 , network operators may choose to configure the data centers 22 using a variety of computing infrastructures. In one embodiment, one or more of the data centers 22 are configured using a multi-tenant cloud architecture, such that one of the server instances 24 handles requests from and serves multiple customers. Data centers with multi-tenant cloud architecture commingle and store data from multiple customers, where multiple customer instances are assigned to one of the virtual servers 24 . In a multi-tenant cloud architecture, the particular virtual server 24 distinguishes between and segregates data and other information of the various customers. For example, a multi-tenant cloud architecture could assign a particular identifier for each customer in order to identify and segregate the data from each customer. Generally, implementing a multi-tenant cloud architecture may suffer from various drawbacks, such as a failure of a particular one of the server instances 24 causing outages for all customers allocated to the particular server instance.

In another embodiment, one or more of the data centers 22 are configured using a multi-instance cloud architecture to provide every customer its own unique customer instance or instances. For example, a multi-instance cloud architecture could provide each customer instance with its own dedicated application server(s) and dedicated database server(s). In other examples, the multi-instance cloud architecture could deploy a single physical or virtual server and/or other combinations of physical and/or virtual servers 24 , such as one or more dedicated web servers, one or more dedicated application servers, and one or more database servers, for each customer instance. In a multi-instance cloud architecture, multiple customer instances could be installed on one or more respective hardware servers, where each customer instance is allocated certain portions of the physical server resources, such as computing memory, storage, and processing power. By doing so, each customer instance has its own unique software stack that provides the benefit of data isolation, relatively less downtime for customers to access the platform 20 , and customer-driven upgrade schedules. An example of implementing a customer instance within a multi-instance cloud architecture will be discussed in more detail below with reference to FIG. 2 .

FIG. 2 is a schematic diagram of an embodiment of a multi-instance cloud architecture 40 where embodiments of the present disclosure may operate. FIG. 2 illustrates that the multi-instance cloud architecture 40 includes the client network 12 and the network 18 that connect to two (e.g., paired) data centers

22 A and 22 B that may be geographically separated from one another. Using FIG. 2 as an example, network environment and service provider cloud infrastructure client instance 42 (also referred to herein as a simply client instance 42 ) is associated with (e.g., supported and enabled by) dedicated virtual servers (e.g.,

virtual servers

24 A, 24 B, 24 C, and 24 D) and dedicated database servers (e.g., virtual database servers

44 A and 44 B). Stated another way, the virtual servers 24 A- 24 D and virtual database servers

44 A and 44 B are not shared with other client instances and are specific to the respective client instance 42 . Other embodiments of the multi-instance cloud architecture 40 could include other types of dedicated virtual servers, such as a web server. For example, the client instance 42 could be associated with (e.g., supported and enabled by) the dedicated virtual servers 24 A- 24 D, dedicated virtual database servers

44 A and 44 B, and additional dedicated virtual web servers (not shown in FIG. 2 ).

In the depicted example, to facilitate availability of the client instance 42 , the virtual servers 24 A- 24 D and virtual database servers

44 A and 44 B are allocated to two different data centers

22 A and 22 B, where one of the data centers 22 acts as a backup data center. In reference to FIG. 2 , data center 22 A acts as a primary data center that includes a primary pair of virtual servers

24 A and 24 B and the primary virtual database server 44 A associated with the client instance 42 . Data center 22 B acts as a secondary data center 22 B to back up the primary data center 22 A for the client instance 42 . To back up the primary data center 22 A for the client instance 42 , the secondary data center 22 B includes a secondary pair of virtual servers

24 C and 24 D and a secondary virtual database server 44 B. The primary virtual database server 44 A is able to replicate data to the secondary virtual database server 44 B (e.g., via the network 18 ).

As shown in FIG. 2 , the primary virtual database server 44 A may back up data to the secondary virtual database server 44 B using a database replication operation. The replication of data between data could be implemented by performing full backups weekly and daily incremental backups in both data centers

22 A and 22 B. Having both a primary data center 22 A and secondary data center 22 B allows data traffic that typically travels to the primary data center 22 A for the client instance 42 to be diverted to the second data center 22 B during a failure and/or maintenance scenario. Using FIG. 2 as an example, if the virtual servers

24 A and 24 B and/or primary virtual database server 44 A fails and/or is under maintenance, data traffic for <figure-callout id="42" label="client instances" filenames="US11520992-20221206-D00002.png,US11520992-20221206-D00004.png" state="{

CLAIMS

Claims ( 18 )

What is claimed is:

1. An agent automation system, comprising:

a memory configured to store a natural language understanding (NLU) framework and an intent/entity model, wherein the intent/entity model associates defined intents with a plurality of written sample utterances, and wherein the written sample utterances encode defined entities as parameters of the defined intents within the intent/entity model, wherein the NLU framework includes a vocabulary subsystem, a structure subsystem, and a prosody subsystem; and

a processor configured to execute instructions of the NLU framework to cause the agent automation system to perform actions comprising:

generating an annotated utterance tree for a written sample utterance of the plurality of written sample utterances by:

processing, via the prosody subsystem, the written sample utterance based on written prosody cues to divide the written sample utterance into a plurality of nodes that each represents a word or phrase of the written sample utterance, wherein the written prosody cues comprise a rhythm, an emphasis, or a focus of the written sample utterance;

processing, via the structure subsystem, the written sample utterance to organize the plurality of nodes into a dependency parse tree structure that encodes a syntactic structure of the written sample utterance; and

assigning, via the vocabulary subsystem, a respective word vector to each of the plurality of nodes, wherein each respective word vector encodes a semantic meaning of the word or phrase represented by each of the plurality of nodes.

2. The agent automation system of claim 1 , wherein the processor is configured to execute instructions of the NLU framework to cause the agent automation system to perform actions comprising:

generating a respective subtree vector for each subtree of the dependency parse tree structure based on the respective word vectors of the nodes of each subtree of the dependency parse tree structure.

3. The agent automation system of claim 2 , wherein the processor is configured to execute instructions of the NLU framework to cause the agent automation system to perform actions comprising:

receiving a written user utterance;

generating an annotated utterance tree for the written user utterance by:

processing, via the prosody subsystem, the written user utterance based on the written prosody cues to divide the written user utterance into a second plurality of nodes that each represents a word or phrase of the written user utterance;

processing, via the structure subsystem, the written user utterance to organize the second plurality of nodes into a second dependency parse tree structure that encodes a syntactic structure of the written user utterance; and

assigning, via the vocabulary subsystem, a respective word vector to each of the second plurality of nodes, wherein each respective word vector encodes a semantic meaning of the word or phrase represented by each of the second plurality of nodes.

4. The agent automation system of claim 3 , wherein the processor is configured to execute instructions of the NLU framework to cause the agent automation system to perform actions comprising:

performing, via the vocabulary subsystem, rules-based cleansing and augmentation to modify the written user utterance before generating the annotated utterance tree for the written user utterance, wherein the rules-based cleansing and augmentations comprises substituting synonyms, correcting misspellings, removing punctuation, addressing domain-specific syntax and terminology, combining words, and/or separating compounds words and contractions in the written user utterance.

5. The agent automation system of claim 3 , wherein the processor is configured to execute instructions of the NLU framework to cause the agent automation system to perform actions comprising:

generating a respective subtree vector for each subtree of the second dependency parse tree structure based on the respective word vectors of the nodes of each subtree of the second dependency parse tree structure; and

extracting an intent and/or entity from the written user utterance based on a comparison of the subtree vectors of the annotated utterance tree of the written user utterance to the subtree vectors of the annotated utterance tree of the written sample utterance.

6. The agent automation system of claim 5 , wherein the extracted intent and/or entity includes a corresponding confidence score that is based on subtree similarity scores determined during the comparison of the subtree vectors of the annotated utterance tree of the written user utterance to the subtree vectors of the annotated utterance tree of the written sample utterance.

7. The agent automation system of claim 3 , wherein the vocabulary subsystem includes a word vector distribution model, wherein vocabulary subsystem determines the respective word vector for each of the plurality of nodes of the annotated utterance tree based on the word vector distribution model.

8. The agent automation system of claim 7 , wherein the processor is configured to execute instructions of the NLU framework to cause the agent automation system to improve operation of the vocabulary subsystem by performing actions comprising:

performing rule-based unsupervised learning of the words or phrases of the written user utterance; and

modifying word vectors of the word vector distribution model based on the rule-based unsupervised learning.

9. The agent automation system of claim 3 , wherein the structure subsystem includes a plurality of rules-based parsers and a machine-learning (ML)-based parser, and wherein the processor is configured to execute instructions of the NLU framework to cause the agent automation system to improve operation of the ML-based parser by performing actions comprising:

parsing the written sample utterance using the plurality of rules-based parsers of the structure subsystem to generate a plurality of dependency parse tree structures; and

in response to the processor determining that a majority of the plurality of dependency parse tree structures has a common dependency parse tree structure, adjusting a neural network model associated with the ML-based parser such that the ML-based parser is configured to generate the common dependency parse tree structure from the written user utterance.

10. A method of operating a natural language understanding (NLU) framework, comprising:

generating an annotated utterance tree for each written sample utterance of a plurality of written sample utterances of an intent/entity model by:

processing the written sample utterance based on written prosody cues to segment the written sample utterance into a plurality of nodes that each represents a word or phrase of the written sample utterance, wherein the written prosody cues comprise a rhythm, an emphasis, or a focus of the written sample utterance;

organizing the plurality of nodes into a dependency parse tree structure that encodes a syntactic structure of the written sample utterance, wherein class annotations are assigned to each of the plurality of nodes in the dependency parse tree structure, and wherein the class annotations comprise: a verb annotation, a subject or entity annotation, a direct object annotation, a subject modifier annotation, an object modifier annotation, or a verb modifier annotation;

assigning a respective word vector to each of the plurality of nodes that encodes a semantic meaning of the word or phrase represented by each of the plurality of nodes; and

generating a respective subtree vector for each subtree of the dependency parse tree structure from the respective word vectors of the nodes of each subtree of the dependency parse tree structure.

11. The method of claim 10 , comprising:

receiving a written user utterance;

generating an annotated utterance tree for the written user utterance by:

processing the written user utterance based on the written prosody cues to segment the written user utterance into a second plurality of nodes that each represents a word or phrase of the written user utterance;

organizing the second plurality of nodes into a second dependency parse tree structure that encodes a syntactic structure of the written user utterance;

assigning a respective word vector to each of the second plurality of nodes that encodes a semantic meaning of the word or phrase represented by each of the second plurality of nodes;

generating a respective subtree vector for each subtree of the annotated utterance tree of the written user utterance from the respective word vectors the nodes of each subtree; and

12. The method of claim 11 , wherein the NLU framework includes a plurality of rules-based parsers and a machine-learning (ML)-based parser, wherein the method comprises:

parsing the written sample utterance using the plurality of rules-based parsers to generate a plurality of dependency parse tree structures; and

in response to determining that a majority of the plurality of dependency parse tree structures has a common dependency parse tree structure, adjusting a neural network model of the ML-based parser such that the ML-based parser is configured to generate the common dependency parse tree structure from the written user utterance.

13. A non-transitory, computer-readable medium storing instructions of a natural language understanding (NLU) framework executable by one or more processors of a computing system, the instructions comprising instructions to:

receive a written user utterance;

generate an annotated utterance tree for the written user utterance by:

processing the written user utterance based on written prosody cues to divide the written user utterance into a plurality of nodes that each represents a word or phrase of the written user utterance, wherein the written prosody cues comprise a rhythm, an emphasis, or a focus of the written user utterance;

organizing the plurality of nodes into a dependency parse tree structure that encodes a syntactic structure of the written user utterance;

assigning a word vector to each of the plurality of nodes that encodes a semantic meaning of the word or phrase represented by each of the plurality of nodes; and

assigning a subtree vector to each subtree of the dependency parse tree structure based on the word vectors of the nodes of each subtree of the dependency parse tree structure;

compare the subtree vectors of the annotated utterance tree of the written user utterance to subtree vectors of annotated utterance trees of written sample utterances of an intent/entity model to identify a matching written sample utterance; and

determine an intent and/or entity of the written user utterance to be a defined intent and/or entity of the intent/entity model that corresponds to the matching written sample utterance within the intent/entity model.

14. The medium of claim 13 , wherein the instructions comprise instructions to:

before assigning the subtree vectors to the subtrees of the dependency parse tree structure, performing rule-based error detection to detect, in the dependency parse tree structure, a misclassification or a misparse of the words or phrases of the written sample utterance;

performing a rule-based modification of the written user utterance to generate a modified written user utterance, wherein at least a portion of words and/or phrases of the modified written user utterance are different from the words and/or phrases of the written user utterance; and

regenerating the annotated utterance tree from the modified written utterance, wherein, after regeneration, the annotated utterance tree lacks the misclassification or the misparse.

15. The medium of claim 13 , wherein the instructions comprise instructions to:

generate annotated utterance trees for each written sample utterance of the written sample utterances of the intent/entity model by:

processing the written sample utterance based on the written prosody cues to divide the written sample utterance into a second plurality of nodes that each represents a word or phrase of the written sample utterance;

organizing the second plurality of nodes into a second dependency parse tree structure that encodes a syntactic structure of the written sample utterance;

assigning a word vector to each of the second plurality of nodes that encodes a semantic meaning of the word or phrase represented by each of the second plurality of nodes; and

assigning a subtree vector to each subtree of the second dependency parse tree structure based on the word vectors of the nodes of each subtree of the second dependency parse tree structure.

16. The medium of claim 13 , wherein the instructions to determine the intent and/or entity of the written user utterance comprise instructions to determine a confidence score for the intent and/or entity that is based on subtree similarity score calculated when comparing the subtree vectors of the annotated utterance tree of the written user utterance to the subtree vectors of the annotated utterance trees of the written sample utterances.

17. The medium of claim 13 , wherein the instructions comprise instructions to:

perform rule-based unsupervised learning of the words or phrases of the written user utterance; and

modify word vectors of a word vector distribution model of the NLU framework based on the rule-based unsupervised learning, wherein the NLU framework is configured to assign word vectors to each of the plurality of nodes of the dependency parse tree structure based on the word vector distribution model.

18. The medium of claim 13 , wherein the instructions comprise instructions to:

perform rules-based cleansing and augmentation to modify the written user utterance before generating the annotated utterance tree, wherein the rules-based cleansing and augmentation comprises substituting synonyms, correcting misspellings, removing punctuation, addressing domain-specific syntax and terminology, combining words, and/or separating compounds words and contractions in the written user utterance.

US16/909,731

2018-03-23

2020-06-23

Hybrid learning system for natural language understanding

Active

2039-09-06

US11520992B2

( en )

Priority Applications (1)

Application Number

Priority Date

Filing Date

Title

US16/909,731

US11520992B2

( en )

2018-03-23

2020-06-23

Hybrid learning system for natural language understanding

Applications Claiming Priority (8)

Application Number

Priority Date

Filing Date

Title

US201862646916P

2018-03-23

US201862646917P

2018-03-23

US201862646915P

2018-03-23

US201862647751P

2018-03-25

US201862652903P

2018-04-05

US201862659710P

2018-04-19

US16/238,324

US10713441B2

( en )

2018-03-23

2019-01-02

Hybrid learning system for natural language intent extraction from a dialog utterance

US16/909,731

US11520992B2

( en )

2018-03-23

2020-06-23

Hybrid learning system for natural language understanding

Related Parent Applications (1)

Application Number

Title

Priority Date

Filing Date

US16/238,324

Continuation

US10713441B2

( en )

2018-03-23

2019-01-02

Hybrid learning system for natural language intent extraction from a dialog utterance

Publications (2)

Publication Number

Publication Date

US20200327284A1

US20200327284A1 ( en )

2020-10-15

US11520992B2

true

US11520992B2 ( en )

2022-12-06

Family

ID=84272742

Family Applications (1)

Application Number

Title

Priority Date

Filing Date

US16/909,731

Active

2039-09-06

US11520992B2

( en )

2018-03-23

2020-06-23

Hybrid learning system for natural language understanding

Country Status (1)

Country

Link

( 1 )

US11520992B2

( en )

Cited By (1)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US12455779B2

( en )

2023-06-12

2025-10-28

Arista Networks, Inc.

Processing natural language network queries

Families Citing this family (24)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US11694025B2

( en )

2020-05-04

2023-07-04

Kyndryl Inc.

Cognitive issue description and multi-level category recommendation

US11907665B2

( en )

2020-05-11

2024-02-20

Rajiv Trehan

Method and system for processing user inputs using natural language processing

US11741317B2

( en )

2020-05-25

2023-08-29

Rajiv Trehan

Method and system for processing multilingual user inputs using single natural language processing model

CN112270533A

( en )

2020-11-12

2021-01-26

åäº¬ç¾åº¦ç½è®¯ç§ææéå¬å¸

Data processing method and device, electronic equipment and storage medium

AU2021204758A1

( en )

2020-11-20

2022-06-16

Soul Machines

Autonomous animation in embodied agents

CN112579792B

( en )

2020-12-22

2023-08-04

ä¸åå¤§å¦

PGAT and FTATT-based remote supervision relation extraction method

PT116967B

( en )

2020-12-23

2023-01-30

Altice Labs S A

METHOD FOR SEGMENT-BASED NATURAL LANGUAGE PROCESSING OF MULTIPLE MODELS USING A HYBRID PLN

US12511491B2

( en )

2021-01-21

2025-12-30

Servicenow, Inc.

System and method for managing and optimizing lookup source templates in a natural language understanding (NLU) framework

US12175196B2

( en )

2021-01-21

2024-12-24

Servicenow, Inc.

Operational modeling and optimization system for a natural language understanding (NLU) framework

US12499313B2

( en )

2021-01-21

2025-12-16

Servicenow, Inc.

Ensemble scoring system for a natural language understanding (NLU) framework

US12374325B2

( en )

2021-01-21

2025-07-29

Servicenow, Inc.

Domain-aware vector encoding (DAVE) system for a natural language understanding (NLU) framework

US12299391B2

( en )

2021-01-21

2025-05-13

Servicenow, Inc.

System and method for repository-aware natural language understanding (NLU) using a lookup source framework

US12197869B2

( en )

2021-01-21

2025-01-14

Servicenow, Inc.

Concept system for a natural language understanding (NLU) framework

US12175193B2

( en )

2021-01-21

2024-12-24

Servicenow, Inc.

System and method for lookup source segmentation scoring in a natural language understanding (NLU) framework

US12265796B2

( en )

2021-01-21

2025-04-01

Servicenow, Inc.

Lookup source framework for a natural language understanding (NLU) framework

US11880660B2

( en )

2021-02-22

2024-01-23

Microsoft Technology Licensing, Llc

Interpreting text classifier results with affiliation and exemplification

CN113012701B

( en )

2021-03-16

2024-03-22

èæ³(åäº¬)æéå¬å¸

Identification method, identification device, electronic equipment and storage medium

US20230083096A1

( en )

2021-05-11

2023-03-16

Argot Limited

Context based language training system, device, and method thereof

US11238849B1

( en )

2021-05-31

2022-02-01

Rockspoon, Inc.

System and method for federated, context-sensitive, adaptive language models

US11568860B2

( en )

2021-05-31

2023-01-31

Rockspoon, Inc.

System and method for federated, context-sensitive, acoustic model refinement

US12175966B1

( en )

2021-06-28

2024-12-24

Amazon Technologies, Inc.

Adaptations of task-oriented agents using user interactions

CN113408299B

( en )

2021-06-30

2022-03-25

åäº¬ç¾åº¦ç½è®¯ç§ææéå¬å¸

Training method, apparatus, device and storage medium for semantic representation model

CN113435863A

( en )

2021-07-22

2021-09-24

ä¸å½äººæ°å¤§å¦

Method and system for optimizing guided collaborative process, storage medium and computing device

US20240202518A1

( en )

2022-12-19

2024-06-20

Microsoft Technology Licensing, Llc

Privacy-preserving generation of synthesized training data

Citations (72)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US3006240A

( en )

1958-08-12

1961-10-31

Alois Handler

Synchronizing device for a film projector and a sound tape apparatus

US3650078A

( en )

1970-09-15

1972-03-21

Economy Forms Corp

Shore tower assembly

US20030009331A1

( en )

2001-07-05

2003-01-09

Johan Schalkwyk

Grammars for speech recognition

US6609122B1

( en )

2000-08-01

2003-08-19

Bmc Software, Inc.

Navigation of view relationships in database system

US20050033574A1

( en )

2003-08-06

2005-02-10

Samsung Electronics Co., Ltd.

Method and apparatus handling speech recognition errors in spoken dialogue systems

US7020706B2

( en )

2002-06-17

2006-03-28

Bmc Software, Inc.

Method and system for automatically updating multiple servers

US7028301B2

( en )

2000-12-08

2006-04-11

Bmc Software, Inc.

System and method for automatic workload characterization

US7062683B2

( en )

2003-04-22

2006-06-13

Bmc Software, Inc.

Two-phase root cause analysis

US20060130038A1

( en )

2004-12-15

2006-06-15

Claussen Christopher S

Apparatus, system, and method for facilitating dynamic modification of existing software objects defined in a strongly-typed programming language

US7131037B1

( en )

2002-06-05

2006-10-31

Proactivenet, Inc.

Method and system to correlate a specific alarm to one or more events to identify a possible cause of the alarm

US7170864B2

( en )

2001-03-08

2007-01-30

Bmc Software, Inc.

System and method for WAP server management using a single console

US20070261065A1

( en )

2006-04-20

2007-11-08

Astl Kenneth L

Framework for generating pre-packaged business integration component group pattern-based applications

US7350209B2

( en )

2001-06-29

2008-03-25

Bmc Software

System and method for application performance management

US20080075444A1

( en )

2006-09-25

2008-03-27

Murali Subbarao

Blur equalization for auto-focusing

US7509653B2

( en )

2004-01-22

2009-03-24

International Business Machines Corporation

Efficient and scalable event partitioning in business integration applications using multiple delivery queues

US20090119102A1

( en )

2007-11-01

2009-05-07

At&T Labs

System and method of exploiting prosodic features for dialog act tagging in a discriminative modeling framework

US20090259459A1

( en )

2002-07-12

2009-10-15

Werner Ceusters

Conceptual world representation natural language understanding system and method

US7610512B2

( en )

2006-01-06

2009-10-27

Hewlett-Packard Development Company, L.P.

System and method for automated and assisted resolution of it incidents

US7617073B2

( en )

2002-03-01

2009-11-10

Bmc Software, Inc.

System and method for assessing and indicating the health of components

US7617500B2

( en )

2005-05-20

2009-11-10

International Business Machines Corporation

Generic framework for integrating components with different interfaces in an enterprise application integration environment

US7630784B2

( en )

2005-07-22

2009-12-08

International Business Machines Corporation

Method and apparatus for independent deployment of roles

US7653650B2

( en )

2005-12-13

2010-01-26

International Business Machines Corporation

Apparatus, system, and method for synchronizing change histories in enterprise applications

US7689628B2

( en )

2005-05-19

2010-03-30

Atul Garg

Monitoring several distributed resource elements as a resource pool

US7716353B2

( en )

2005-12-21

2010-05-11

Bmc Software, Inc.

Web services availability cache

US7769718B2

( en )

2003-11-26

2010-08-03

Bmc Software, Inc.

Unobtrusive point-in-time consistent copies

US7783744B2

( en )

2003-06-03

2010-08-24

Bmc Software, Inc.

Facilitating root cause analysis for abnormal behavior of systems in a networked environment

US7849201B1

( en )

2005-06-14

2010-12-07

Billeo, Inc

Method and system for capturing, organizing, searching and sharing web pages

US7899670B1

( en )

2006-12-21

2011-03-01

Escription Inc.

Server-based speech recognition

US7925981B2

( en )

2003-05-14

2011-04-12

Hewlett-Packard Development Company, L.P.

Systems and methods for managing web services via a framework of interfaces

US7930396B2

( en )

2000-07-10

2011-04-19

Bmc Software, Inc.

System and method of enterprise systems and business impact management

US7945860B2

( en )

2003-05-14

2011-05-17

Hewlett-Packard Development Company, L.P.

Systems and methods for managing conversations between information technology resources

US7958031B2

( en )

2005-12-13

2011-06-07

International Business Machines Corporation

Apparatus, system, and method for automated identity relationship maintenance

US7966398B2

( en )

2001-03-30

2011-06-21

Bmc Software, Inc.

Synthetic transaction monitor with replay capability

US7971147B2

( en )

2005-07-25

2011-06-28

Billeo, Inc.

Methods and systems for automatically creating a site menu

US8051164B2

( en )

2007-12-14

2011-11-01

Bmc Software, Inc.

Impact propagation in a directed acyclic graph having restricted views

US8112354B2

( en )

2005-04-08

2012-02-07

Billeo, Inc

Method and system for virtual consolidation of biller direct web sites

US8156479B2

( en )

2007-12-07

2012-04-10

International Business Machines Corporation

System and method of monitoring dynamic scopes in synchronous and asynchronous calls

US20120158791A1

( en )

2010-12-21

2012-06-21

Microsoft Corporation

Feature vector construction

US8224683B2

( en )

2003-07-08

2012-07-17

Hewlett-Packard Development Company, L.P.

Information technology service request level of service monitor

US8266096B2

( en )

2008-10-24

2012-09-11

Bmc Software, Inc.

Vendor portfolio management in support of vendor relationship management analysis, planning and evaluation

US20120246073A1

( en )

2011-03-21

2012-09-27

Billeo, Inc.

Systems and methods for transferring transaction instructions from a remote repository to a merchant website using a transaction agent

US20120290938A1

( en )

2011-05-11

2012-11-15

Billeo, Inc.

Systems and Methods for Context Aware Interaction Across Websites and Apps

US20130055223A1

( en )

2011-08-25

2013-02-28

Myezapp Inc.

Compiler with Error Handling

US8433654B2

( en )

2010-05-10

2013-04-30

Billeo, Inc

Method and system for paying directly at biller websites from within a bill pay website

US8457928B2

( en )

2010-03-26

2013-06-04

Bmc Software, Inc.

Automatic determination of dynamic threshold for accurate detection of abnormalities

US8473361B2

( en )

2010-09-23

2013-06-25

Billeo, Inc.

Method and system for assisting users during online

US8478569B2

( en )

2010-03-26

2013-07-02

Bmc Software, Inc.

Auto adjustment of baseline on configuration change

US8612289B2

( en )

2011-03-04

2013-12-17

Billeo, Inc.

Methods and systems for paying with loyalty currency during online payment

US20140067379A1

( en )

2011-11-29

2014-03-06

Sk Telecom Co., Ltd.

Automatic sentence evaluation device using shallow parser to automatically evaluate sentence, and error detection apparatus and method of the same

US8674992B2

( en )

2010-06-24

2014-03-18

Bmc Software, Inc.

Spotlight graphs

US8689241B2

( en )

2011-09-30

2014-04-01

Bmc Software, Inc.

Dynamic evocations for computer event management

US8743121B2

( en )

2009-12-23

2014-06-03

Bmc Software, Inc.

Smart impact views

US20140163975A1

( en )

2012-12-07

2014-06-12

Postech Academy - Industry Foundation

Method and apparatus for correcting speech recognition error

US20140279274A1

( en )

2013-03-13

2014-09-18

Billeo, Inc.

Methods and systems for facilitating web-based purchases

US20140297458A1

( en )

2013-03-26

2014-10-02

Billeo, Inc.

Systems and Methods for Marketplace Catalogue Population

US20140297457A1

( en )

2013-03-26

2014-10-02

Billeo, Inc.

Systems and Methods for Product Recommendation Refinement in Topic-Based Virtual Storefronts

US20140303957A1

( en )

2013-04-08

2014-10-09

Electronics And Telecommunications Research Institute

Automatic translation and interpretation apparatus and method

US8887133B2

( en )

2006-04-28

2014-11-11

Bmc Software, Inc.

Bi-directional communication between change management tool and implementation tools

US20150154173A1

( en )

2012-08-10

2015-06-04

Sk Telecom Co., Ltd.

Method of detecting grammatical error, error detecting apparatus for the method, and computer-readable recording medium storing the method

US9065783B2

( en )

2006-08-31

2015-06-23

Bmc Software, Inc.

Automated capacity provisioning method using historical performance data

US9122552B2

( en )

2012-06-29

2015-09-01

Bmc Software, Inc.

Hybrid cloud infrastructures

US20150269573A1

( en )

2014-03-20

2015-09-24

Billeo, Inc.

Systems and methods for creating and accessing electronic wallet

US20150370778A1

( en )

2014-06-19

2015-12-24

Nuance Communications, Inc.

Syntactic Parser Assisted Semantic Rule Inference

US9239857B2

( en )

2004-12-21

2016-01-19

Bmc Software, Inc.

System and method for building business service model

US20160328388A1

( en )

2015-05-08

2016-11-10

International Business Machines Corporation

Semi-supervised learning of word embeddings

US20160350288A1

( en )

2015-05-29

2016-12-01

Oracle International Corporation

Multilingual embeddings for natural language processing

US9535737B2

( en )

2015-01-30

2017-01-03

Bladelogic, Inc.

Dynamic virtual port provisioning

US9557969B2

( en )

2012-03-28

2017-01-31

Bmc Software, Inc.

Automated blueprint assembly for assembling an application

US9626717B2

( en )

2013-03-26

2017-04-18

Billeo, Inc.

Methods and systems for dynamic and embeddable storefront widget

US9633004B2

( en )

2014-05-30

2017-04-25

Apple Inc.

Better resolution when referencing to concepts

US9792387B2

( en )

2001-08-31

2017-10-17

Bmc Software, Inc.

Service desk data transfer interface

US20180075844A1

( en )

2016-09-09

2018-03-15

Electronics And Telecommunications Research Institute

Speech recognition system and method

2020

2020-06-23

US16/909,731

patent/US11520992B2/en

active

Active

Patent Citations (73)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US3006240A

( en )

1958-08-12

1961-10-31

Alois Handler

Synchronizing device for a film projector and a sound tape apparatus

US3650078A

( en )

1970-09-15

1972-03-21

Economy Forms Corp

Shore tower assembly

US7930396B2

( en )

2000-07-10

2011-04-19

Bmc Software, Inc.

System and method of enterprise systems and business impact management

US6609122B1

( en )

2000-08-01

2003-08-19

Bmc Software, Inc.

Navigation of view relationships in database system

US7028301B2

( en )

2000-12-08

2006-04-11

Bmc Software, Inc.

System and method for automatic workload characterization

US7170864B2

( en )

2001-03-08

2007-01-30

Bmc Software, Inc.

System and method for WAP server management using a single console

US7966398B2

( en )

2001-03-30

2011-06-21

Bmc Software, Inc.

Synthetic transaction monitor with replay capability

US7350209B2

( en )

2001-06-29

2008-03-25

Bmc Software

System and method for application performance management

US20030009331A1

( en )

2001-07-05

2003-01-09

Johan Schalkwyk

Grammars for speech recognition

Hybrid learning system for natural language understanding — Servicenow, Inc. (US11520992B2)

Related documents