Conceptio › Archive › Google Patents

Google Patentsopen access

Systems and methods for intelligently curating machine learning training data … — Clinc, Inc. (US10679100B2)

Clinc, Inc. · Google Patents

Google Patents · Patents · License: Open Access

patent, google patents, intellectual property, US10679100B2, Clinc, Inc., Yiping Kang, en, 2020

ABSTRACT

Abstract

Systems and methods of intelligent formation and acquisition of machine learning training data for implementing an artificially intelligent dialogue system includes constructing a corpora of machine learning test corpus that comprise a plurality of historical queries and commands sampled from production logs of a deployed dialogue system; configuring training data sourcing parameters to source a corpora of raw machine learning training data from remote sources of machine learning training data; calculating efficacy metrics of the corpora of raw machine learning training data, wherein calculating the efficacy metrics includes calculating one or more of a coverage metric value and a diversity metric value of the corpora of raw machine learning training data; using the corpora of raw machine learning training data to train the at least one machine learning classifier if the calculated coverage metric value of the corpora of machine learning training data satisfies a minimum coverage metric threshold.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No. 16/143,773, filed 27 Sep. 2018, which claims the benefit of U.S. Provisional Application Ser. No. 62/648,318, filed 26 Mar. 2018, all of which are incorporated herein their entireties by this reference.

GOVERNMENT RIGHTS

The subject matter of the invention may be subject to U.S. Government Rights under National Science Foundation grants: NSF SBIR Phase 1 Grantâ1622049 and NSF SBIR Phase 2 Grantâ1738441.

TECHNICAL FIELD

The inventions herein relate generally to the machine learning field, and more specifically to a new and useful system and method for intelligently training machine learning models in the machine learning field.

BACKGROUND

Modern virtual assistants and/or online chatbots may typically be employed to perform various tasks or services based on an interaction with a user. Typically, a user interacting with a virtual assistant may pose a question or otherwise submit a command to the virtual assistant to which the virtual assistant may provide a response or a result. Many of these virtual assistants may be implemented using a rules-based approach, which typically requires coding or preprogramming many or hundreds of rules that may govern a manner in which the virtual assistant should operate to respond to a given query or command from a user.

While the rules-based approach for implementing a virtual assistant may be useful for addressing pointed or specific queries or commands made by a user, the rigid or finite nature of this approach severely limits a capability of a virtual assistant to address queries or commands from a user that exceed the scope of the finite realm of pointed and/or specific queries or commands that are addressable by the finite set of rules that drive the response operations of the virtual assistant.

That is, the modern virtual assistants implemented via a rules-based approach for generating responses to users may not fully satisfy queries and commands posed by a user for which there are no predetermined rules to provide a meaningful response or result to the user.

Additionally, while machine learning enhances capabilities of artificially intelligent conversational systems, inefficiencies continue to persist in training the underlying machine learning models performing classification and predictive functions of the artificially intelligent conversation systems.

Therefore, there is a need in the machine learning field for systems and methods that enable rapid and efficient training of machine learning models and for a flexible virtual assistant solution that is capable of evolving beyond a finite set of rules for effectively and conversantly interacting with a user. The embodiments of the present application described herein provide technical solutions that address, at least, the need described above, as well as the deficiencies of the state of the art described throughout the present application.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 illustrates a schematic representation of a system 100 in accordance with one or more embodiments of the present application;

FIG. 1A illustrates a schematic representation of a subsystem of system 100 in accordance with one or more embodiments of the present application;

FIG. 2 illustrates an example method in accordance with one or more embodiments of the present application;

FIG. 3 illustrates a schematic representation of a system and process for constructing a machine learning test corpus in accordance with one or more embodiments of the present application;

FIG. 4 illustrates a schematic representation of a system and process for determining a diversity of machine learning training data in accordance with one or more embodiments of the present application; and

FIG. 5 illustrates a schematic representation of a system and process for determining a coverage of machine learning training data in accordance with one or more embodiments of the present application.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

The following description of the preferred embodiments of the present application are not intended to limit the inventions to these preferred embodiments, but rather to enable any person skilled in the art to make and use these inventions.

Overview

As discussed above, existing virtual assistant implementations do not have the requisite flexibility to address unrecognized queries or commands from user in which there are no predetermined rules designed around narrowly-defined intents. This inflexible structure cannot reasonably and efficiently address the many variances in the manners in which a user may pose a query or command to the virtual assistant.

The embodiments of the present application, however, provide artificial intelligence virtual assistant platform (e.g., an artificially intelligent dialogue system) and natural language processing capabilities that function to process and comprehend structured and/or unstructured natural language input from a user or input from any other suitable source. Using one or more trained (deep) machine learning models, such as long short-term memory (LSTM) neural network, the embodiments of the present application may function to understand any variety of natural language utterance or textual input provided to the system. The one or more deep machine learning models post deployment can continue to train using unknown and previously incomprehensible queries or commands from users. As a result, the underlying system that implements the (deep) machine learning models may function to evolve with increasing interactions with users and training rather than being governed by a fixed set of predetermined rules for responding to narrowly-defined queries, as may be accomplished in the current state of the art.

Accordingly, the evolving nature of the artificial intelligence platform described herein therefore enables the artificially intelligent virtual assistant latitude to learn without a need for additional programming and the capabilities to ingest complex (or uncontemplated) utterances and text input to provide meaningful and accurate responses.

Additionally, a machine learning model configuration and management console of one or more embodiments of the present application enable a rapid and efficient training of machine learning models employed in the artificially intelligent virtual assistant.

Additionally, the one or more embodiments provide systems and techniques for intelligently curating large volumes of machine learning data for dialogue systems with limited access to sufficient training data. The one or more embodiments detail one or more training data sourcing techniques that may function to reduce training requirements of machine learning models of a deployed system. In some embodiments, the one or more techniques provide intelligence with respect to a quality of collected training data and whether the training data will function to improve one or more machine learning models without actually training the one or more machine learning models with the collected training data. Accordingly, a technical benefit of one or more of these embodiments include a significant reduction in the training of machine learning models with sub-optimal and/or poor quality training data and reduction in use of computing resources including memory and computer processing power because of the one or more embodiments enable a reduction in training requirements for machine learning models.

Intelligent Machine Learning Training Data Curation

Further, it may be additionally recognized in the artificially intelligent dialogue systems space that large volumes of training data are typically required for deploying high performance machine learning models in such dialogue systems. While some entities that implement and/or deploy these dialogue systems have access to large volumes of training data based data that may be accessible from related and well-developed systems and platforms, this is not the case for entities without such advantages. Thus, a lack of accessibility and/or availability of large volumes of training data for newly developing and/or relatively newly deployed artificially intelligent dialogue systems appears to be a significant hurdle.

The one or more embodiments of the present application, however, provide systems and techniques that enables an intelligent and efficient sourcing of large volumes of training data as well as one or more systems and techniques that enable an intelligent curation of training data for the purposes of implementing a highly performant artificially intelligent dialogue system.

As described in more detail below, one or more embodiments of the present application disclose intelligent training data sourcing methods (e.g., data collection methods). As detailed in one or more of the embodiments herein, the systems and methods function to configure one or more sourcing parameters including for constructing appropriate prompt compositions and mixtures for collecting a most optimal training data set from one or more training data sources including remote crowdsourcing platforms.

Additional embodiments of the present application provide systems and methods that enable intelligent measures of performance and measures of efficacy of the collected training data without having to train and measure an accuracy of a subject machine learning model. These intelligent measures may additionally function to inform an optimal curation of the training data.

1. System for Intelligently Curating Machine Learning Training Data for Improving Performance Metrics of a Machine Learning Model

As shown in FIG. 1 , a system 100 that automatically trains and/or configures machine learning models includes an artificial intelligence (AI) virtual assistant platform 110 (e.g., artificially intelligent dialogue platform), a machine learning configuration interface 120 , a training/ configuration data repository 130 , a configuration data queue 135 , and a plurality of external training/configuration data sources 140 .

As shown in FIG. 1A , a subsystem 170 for intelligently training and/or configuring a machine learning model includes a machine learning model configuration and management console 175 , a plurality of external training data sources 180 , a plurality of datastores 185 , and a training data processing engine 190 .

The machine learning model configuration and management console 175 preferably functions to provide a user interface that may be in operable communication and/or configurational control of one or more components of the subsystem 170 as well as the artificially intelligent conversational system 100 . The machine learning configuration and management console 175 preferably enables an administrator of a machine learning system or environment to perform configuration updates to one or more machine learning models of the machine learning system and/or configure new machine learning models into the machine learning system. The management console 175 may be implemented by one or more private or public (hardware) computing servers and/or computing servers of a distributed computing system (e.g., the cloud).

The plurality of external training data sources 180 preferably include several disparate sources of labeled training data that may be used for training machine learning models. For instance, the plurality of external training data sources 180 may include a crowdsourcing data platform, such as Amazon Mechanical Turk or the like, in which labeled data is sourced from a number of data sources or users into the crowdsourcing data platform.

The plurality of datastores 185 may function to collect and store machine learning training data from the plurality of external training data sources 180 .

The training data processing engine 190 may function to process the raw training data samples collected from the plurality of external training data sources 180 into a refined or finished composition or list of training data samples that may be deployed into an operational or live machine learning model of the system 100 .

CROSS-REFERENCE TO RELATED APPLICATIONS

GOVERNMENT RIGHTS

The subject matter of the invention may be subject to U.S. Government Rights under National Science Foundation grants: NSF SBIR Phase 1 Grantâ1622049 and NSF SBIR Phase 2 Grantâ1738441.

TECHNICAL FIELD

BACKGROUND

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 illustrates a schematic representation of a system 100 in accordance with one or more embodiments of the present application;

FIG. 1A illustrates a schematic representation of a subsystem of system 100 in accordance with one or more embodiments of the present application;

FIG. 2 illustrates an example method in accordance with one or more embodiments of the present application;

FIG. 3 illustrates a schematic representation of a system and process for constructing a machine learning test corpus in accordance with one or more embodiments of the present application;

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Overview

Intelligent Machine Learning Training Data Curation

1. System for Intelligently Curating Machine Learning Training Data for Improving Performance Metrics of a Machine Learning Model

The plurality of datastores 185 may function to collect and store machine learning training data from the plurality of external training data sources 180 .

Generally, the system 100 functions to implement the artificial intelligence virtual assistant platform no to enable intelligent and conversational responses by an artificially intelligent virtual assistant to a user query and/or user command input into the system 100 . Specifically, the system 100 functions to ingest user input in the form of text or speech into a user interface 160 . At natural language processing components of the system 100 that may include, at least, the competency classification engine 120 the slot identification engine 130 , and a slot value extractor 135 , the system 100 functions to identify a competency classification label for the user input data and parse the user input data into comprehensible slots or segments that may, in turn, be converted into program-comprehensible and/or useable features. Leveraging the outputs of the natural language processing components of the system 100 , the observables extractor 140 may function to generate handlers based on the outcomes of the natural language processing components and further, execute the generated handlers to thereby perform various operations that accesses one or more data sources relevant to the query or command and that also performs one or more operations (e.g., data filtering, data aggregation, and the like) to the data accessed from the one or more data sources.

The artificial intelligence virtual assistant platform 110 functions to implement an artificially intelligent virtual assistant capable of interacting and communication with a user. The artificial intelligence platform 110 may be implemented via one or more specifically configured web or private computing servers (or a distributed computing system; e.g., the cloud) or any suitable system for implementing the system 100 and/or the method 200 .

In some implementations, the artificial intelligence virtual assistant platform 110 may be a remote platform implemented over the web (e.g., using web servers) that is configured to interact with distinct and disparate service providers. In such implementation, an event such as a user attempting to access one or more services or data from one or more data sources of the service provider may trigger an implementation of the artificially intelligent virtual assistant of the AI platform 110 . Thus, the AI virtual assistant platform 110 may work in conjunction with the service provider to attend to the one or more queries and/or commands of the users of the service provider. In this implementation, the data sources 160 may be data sources of the service provider that are external data sources to the AI virtual assistant platform 110 .

The competency classification engine 120 together with the slot identification engine 130 and the slot value extractor 135 preferably function to define a natural language processing (NLP) component of the artificial intelligence platform 110 . In one implementation, the natural language processing component may additionally include the automatic speech recognition unit 105 .

The competency classification engine 120 functions to implement one or more competency classification machine learning models to label user input data comprising a user query or a user command. The one or more competency classification machine learning models may include one or more deep machine learning algorithms (e.g., a recurrent neural network, etc.) that have been specifically trained to identify and/or classify a competency label for utterance input and/or textual input. The training input used in training the one or more deep machine learning algorithms of the competency classification engine 120 may include crowdsourced data obtained from one or more disparate user query or user command data sources and/or platforms (e.g., messaging platforms, etc.). However, it shall be noted that the system 100 may obtain training data from any suitable external data sources. The one or more deep machine learning algorithms may additionally be continually trained using user queries and user commands that were miss-predicted or incorrectly analyzed by the system 100 including the competency classification engine 120 .

The competency classification engine 120 may additionally be configured to generate or identify one competency classification label for each user query and/or user command input into the engine 120 . The competency classification engine 120 may be configured to identify or select from a plurality of predetermined competency classification labels (e.g., Income, Balance, Spending, Investment, Location, etc.). Each competency classification label available to the competency classification engine 120 may define a universe of competency-specific functions available to the system 100 or the artificially intelligent assistant for handling a user query or user command. That is, once a competency classification label is identified for a user query or user command, the system 100 may use the competency classification label to restrict one or more computer-executable operations (e.g., handlers) and/or filters that may be used by system components when generating a response to the user query or user command. The one or more computer-executable operations and/or filters associated with each of the plurality of competency classifications may be different and distinct and thus, may be used to process user queries and/or user commands differently as well as used to process user data (e.g., transaction data obtained from external data sources 160 ).

Additionally, the competency classification machine learning model 120 may function to implement a single deep machine learning algorithm that has been trained to identify multiple competency classification labels. Alternatively, the competency classification machine learning model 120 may function to implement an ensemble of deep machine learning algorithms in which each deep machine learning algorithm of the ensemble functions to identify a single competency classification label for user input data. For example, if the competency classification model 120 is capable of identifying three distinct competency classification labels, such as Income, Balance, and Spending, then the ensemble of deep machine learning algorithms may include three distinct deep machine learning algorithms that classify user input data as Income, Balance, and Spending, respectively. While each of the deep machine learning algorithms that define the ensemble may individually be configured to identify a specific competency classification label, the combination of deep machine learning algorithms may additionally be configured to work together to generate individual competency classification labels. For example, if the system receives user input data that is determined to be highly complex (e.g., based on a value or computation of the user input data exceeding a complexity threshold), the system 100 may function to selectively implement a subset (e.g., three machine learning algorithms from a total of nine machine learning algorithms or the like) of the ensemble of machine learning algorithms to generate a competency classification label.

Additionally, the competency classification engine 120 may be implemented by the one or more computing servers, computer processors, and the like of the artificial intelligence virtual assistance platform 110 .

The slot identification engine 130 functions to implement one or more machine learning models to identify slots or meaningful segments of user queries or user commands and to assign a slot classification label for each identified slot. The one or more machine learning models implemented by the slot identification engine 130 may implement one or more trained deep machine learning algorithms (e.g., recurrent neural networks). The one or more deep machine learning algorithms of the slot identification engine 130 may be trained in any suitable manner including with sample data of user queries and user commands that have been slotted and assigned slot values and/or user system derived examples. Alternatively, the slot identification engine 130 may function to implement an ensemble of deep machine learning algorithms in which each deep machine learning algorithm of the ensemble functions to identify distinct slot labels or slot type labels for user input data. For example, slot identification engine 130 may be capable of identifying multiple distinct slot classification labels, such as Income, Account, and Date labels, then the ensemble of deep machine learning algorithms may include three distinct deep machine learning algorithms that function to classify segments or tokens of the user input data as Income, Account, and Date, respectively.

A slot, as referred to herein, generally relates to a defined segment of user input data (e.g., user query or user command) that may include one or more data elements (e.g., terms, values, characters, media, etc.). Accordingly, the slot identification engine 130 may function to decompose a query or command into defined, essential components that implicate meaningful information to be used when generating a response to the user query or command.

A slot label which may also be referred to herein as a slot classification label may be generated by the one or more slot classification deep machine learning models of the engine 130 . A slot label, as referred to herein, generally relates to one of a plurality of slot labels that generally describes a slot (or the data elements within the slot) of a user query or user command. The slot label may define a universe or set of machine or program-comprehensible objects that may be generated for the data elements within an identified slot.

Like the competency classification engine 120 , the slot identification engine 120 may implement a single deep machine learning algorithm or an ensemble of deep machine learning algorithms. Additionally, the slot identification engine 130 may be implemented by the one or more computing servers, computer processors, and the like of the artificial intelligence virtual assistance platform 110 .

The machine learning models and/or the ensemble of machine learning models may employ any suitable machine learning including one or more of: supervised learning (e.g., using logistic regression, using back propagation neural networks, using random forests, decision trees, etc.), unsupervised learning (e.g., using an Apriori algorithm, using K-means clustering), semi-supervised learning, reinforcement learning (e.g., using a Q-learning algorithm, using temporal difference learning), and any other suitable learning style. Each module of the plurality can implement any one or more of: a regression algorithm (e.g., ordinary least squares, logistic regression, stepwise regression, multivariate adaptive regression splines, locally estimated scatterplot smoothing, etc.), an instance-based method (e.g., k-nearest neighbor, learning vector quantization, self-organizing map, etc.), a regularization method (e.g., ridge regression, least absolute shrinkage and selection operator, elastic net, etc.), a decision tree learning method (e.g., classification and regression tree, iterative dichotomiser 3 , C 4.5 , chi-squared automatic interaction detection, decision stump, random forest, multivariate adaptive regression splines, gradient boosting machines, etc.), a Bayesian method (e.g., naÃ¯ve Bayes, averaged one-dependence estimators, Bayesian belief network, etc.), a kernel method (e.g., a support vector machine, a radial basis function, a linear discriminate analysis, etc.), a clustering method (e.g., k-means clustering, expectation maximization, etc.), an associated rule learning algorithm (e.g., an Apriori algorithm, an Eclat algorithm, etc.), an artificial neural network model (e.g., a Perceptron method, a back-propagation method, a Hopfield network method, a self-organizing map method, a learning vector quantization method, etc.), a deep learning algorithm (e.g., a restricted Boltzmann machine, a deep belief network method, a convolution network method, a stacked auto-encoder method, etc.), a dimensionality reduction method (e.g., principal component analysis, partial lest squares regression, Sammon mapping, multidimensional scaling, projection pursuit, etc.), an ensemble method (e.g., boosting, boostrapped aggregation, AdaBoost, stacked generalization, gradient boosting machine method, random forest method, etc.), and any suitable form of machine learning algorithm. Each processing portion of the system 100 can additionally or alternatively leverage: a probabilistic module, heuristic module, deterministic module, or any other suitable module leveraging any other suitable computation method, machine learning method or combination thereof. However, any suitable machine learning approach can otherwise be incorporated in the system 100 . Further, any suitable model (e.g., machine learning, non-machine learning, etc.) can be used in implementing the artificially intelligent virtual assistant and/or other components of the system 100 .

The slot value extraction unit 135 functions to generate slot values by extracting each identified slot and assigned slot label of the user query or user command and converting the data elements (i.e., slot data) within the slot to a machine or program-comprehensible object or instance (e.g., term or value); that is, the slot label is mapped to coding or data that a computer or program of the system 100 comprehends and is able to manipulate or execute processes on. Accordingly, using the slot label generated by the slot identification engine 130 , the slot extraction unit 135 identifies a set or group of machine or program-comprehensible objects or instances that may be applied to slot data of a slot assigned with the slot label. Thus, the slot extraction unit 135 may convert the slot data of a slot to a machine or program-comprehensible object (e.g., slot values) based on the slot label and specifically, based on the available objects, instances, or values mapped to or made available under the slot label.

The observables extractor 140 functions to use the slot values comprising the one or more program-comprehensible objects generated at slot extraction unit 135 to determine or generate one or more handlers or subroutines for handling the data of or responding to the user query or user command of user input data. The observables extractor 140 may function to use the slot values provided by the slot extraction unit 135 to determine one or more data sources relevant to and for addressing the user query or the user command and determine one or more filters and functions or operations to apply to data accessed or collected from the one or more identified data sources. Thus, the coding or mapping of the slot data, performed by slot extraction unit 135 , to program-comprehensible objects or values may be used to specifically identify the data sources and/or the one or more filters and operations for processing the data collected from the data sources.

The response generator 150 functions to use the competency classification label of the user input data to identify or select one predetermined response template or one of a plurality of predetermined response templates. For each competency classification label of the system 100 , the system 100 may have stored a plurality of response templates that may be selected by the response generator 150 based on an identified competency classification label for user input data. Additionally, or alternatively, the response template may be selected based on both the competency classification label and one or more generated slot values. In such instance, the one or more slot values may function to narrow the pool of response template selectable by the response generator to a subset of a larger pool of response templates to consider the variations in a query or user command identified in the slot values. The response templates may generally a combination of predetermined output language or text and one or more input slots for interleaving the handler outputs determined by the observables extractor 140 .

The user interface system 105 may include any type of device or combination of devices capable of receiving user input data and presenting a response to the user input data from the artificially intelligent virtual assistant. In some embodiments, the user interface system 105 receives user input data in the form of a verbal utterance and passes the utterance to the automatic speech recognition unit 115 to convert the utterance into text. The user interface system 105 may include, but are not limited to, mobile computing devices (e.g., mobile phones, tablets, etc.) having a client application of the system 100 , desktop computers or laptops implementing a web browser, an automated teller machine, virtual and/or personal assistant devices (e.g., Alexa, Google Home, Cortana, Jarvis, etc.), chatbots or workboats, etc. An intelligent personal assistant device (e.g., Alexa, etc.) may be any type of device capable of touchless interaction with a user to performing one or more tasks or operations including providing data or information and/or controlling one or more other devices (e.g., computers, other user interfaces, etc.). Thus, an intelligent personal assistant may be used by a user to perform any portions of the methods described herein, including the steps and processes of method 200 , described below. Additionally, a chatbot or a workbot may include any type of program (e.g., slack bot, etc.) implemented by one or more devices that may be used to interact with a user using any type of input method (e.g., verbally, textually, etc.). The chatbot or workbot may be embedded or otherwise placed in operable communication and/or control of a communication node and thus, capable of performing any process or task including, but not limited to, acquiring and providing information and performing one or more control operations.

2. Method for Intelligently Curating Machine Learning Training Data for Improving Performance Metrics of a Machine Learning Model

As shown in FIG. 2 , a method 200 for intelligently and automatically sourcing and curating machine learning training data includes constructing a machine learning test corpus S 210 , sourcing machine learning training data S 215 , measuring one or more efficacy metrics of a corpora of raw machine learning training data S 220 , measuring an accuracy of a set of machine learning models S 230 , tuning one or more parameters of the machine learning training data sourcing technique S 240 , processing the machine learning training data S 250 , and deploying the machine learning training data S 260 .

The method 200 functions to enable intelligent techniques and system for sourcing large volumes of machine learning training data and additional systems and techniques that enable intelligent curation for building a highly performant artificially intelligent dialogue system, as described in U.S. patent application Ser. No. 15/797,414 and U.S. patent application Ser. No. 15/821,010, which are both incorporated in their entireties in this application by this reference.

2.1 Constructing Machine Learning Test (Baseline) Sets

S 210 , which includes constructing a machine learning test corpus, functions to generate a machine learning test corpus for each of a plurality of distinct classification intents (e.g., classification categories, predefined categories, sub-categories, etc.) and/or for each of a plurality of machine learning classifiers. Preferably, a machine learning test corpus comprises a plurality of sentences (e.g., statements, queries, and/or commands with or without punctuation, etc.) defining a machine learning testing set for one or more classification intents of artificially intelligent dialogue system. That is, for a given dialogue system an intent classification task includes receiving user utterance as input and classifies the user utterance into one of a plurality of predefined categories.

In some embodiments, the machine learning test corpus may function as a baseline for evaluating one or more quality metrics or performance metrics of raw or unrefined machine learning training data. Additionally, or alternatively, the machine learning test corpus may function as machine learning training input for an initial training of a machine learning algorithm. In some embodiments, a plurality of distinct machine learning test corpus (i.e., a corpora of machine learning test corpus) may be defined for evaluating a plurality of distinct corpus of raw machine learning training data and/or for training a plurality of distinct machine learning algorithms.

S 210 , in some embodiments, may function to construct a machine learning test corpus using historical user queries or commands from a deployed (dialogue) system, as shown by way of example in FIG. 3 . For instance, S 210 may function to collect a plurality of user queries or user commands from a deployed artificially intelligent dialogue system by test sampling from the production logs of the deployed system. Accordingly, the production logs of such deployed system may include thousands or millions of historical user queries or user commands posed to or provided as input into the deployed system. It shall be noted that S 210 may function to construct the machine learning test corpus using sampled data from any suitable user logs, systems or repositories having real user query and/or user command data including, from recordings or transcripts between a user and a representative or agent of an entity (e.g., a customer service agent).

In some embodiments, S 210 may function to construct a machine learning test corpus based on artificial and/or engineered queries or commands. In such embodiments, a plurality of distinct sentences (test datum) may be engineered by one or more skilled artisans (e.g., developers, engineers, etc.) of an artificially intelligent deployed system or the like. Accordingly, in such embodiments, one or more ideal (or prototypical) user queries and/or one or more user commands may be manually-generated by an engineer or developer associated with the deployed system. The engineered user queries and/or user commands preferably include queries and/or commands that are distinct from the real user queries and/or real user commands of a deployed system. In some embodiments, the engineered user queries and/or user commands may be variations (or enhancements) of real user queries and/or real user commands.

Additionally, or alternatively, S 210 may function to configure a mixture of a machine learning test corpus to include a combination of historical user queries and/or user commands from a deployed system and engineered user queries and/or user commands. An initial composition of the mixture of the machine learning test corpus may be based on predetermined proportions or ratios for each of the historical queries and/or commands and the engineered queries and/or commands. For example, historical (real) user queries from a deployed system may form 70% of a mixture or composition of a machine learning test corpus and the engineered queries may form the remaining 30% of the mixture of the machine learning test corpus.

In some embodiments, a desired composition of a mixture of a machine learning test corpus may be set such that S 210 may function to automatically populate the machine learning test corpus with a number of historical queries and/or commands and engineered queries and/or commands according to pre-selected proportions (e.g., 60% historical, 40% engineered, etc.). In such embodiments, S 210 may function to test sample a historical database or repository of a deployed system to meet or satisfy the selected proportion for historical queries and/or commands and function to automatically engineer or pull from a database of engineered queries and/or commands to satisfy the predefined proportion for engineered queries and/or engineered commands.

Additionally, or alternatively, in a machine learning test corpora in which a plurality of distinct machine learning test corpus define the corpora, S 210 may function to augment each of the distinct corpus with intent or classification labels (or metadata) corresponding to one or more prescribed intents (i.e., classification intent labels).

2.2 Configuring Training Data Sourcing Parameters and Sourcing Training Data

S 215 , which includes sourcing machine learning training data, functions to enable a configuration and/or setting of one or more training data sourcing parameters for sourcing a corpora of raw machine learning training data from one or more sources (e.g., one or more crowdsourcing platforms, etc.) of training data. The one or more training data sourcing parameters for sourcing the corpora of training data preferably include classification intent-specific descriptions, prompts, or examples that define a scope for sourcing and/or generating suitable training data for a given intent classification task and/or an intent-specific machine learning model (classifier). Preferably, S 215 functions to source the corpora of raw machine learning training data for training and improving one or more machine learning algorithms used in implementing an artificially intelligent dialogue system (e.g., system 100 ) or the like from one or more remote crowdsourcing platforms. However, it shall be noted that S 215 may be implemented to configure training data sourcing parameters for sourcing machine learning training data for any suitable system or the like that implement machine learning algorithms for performing classification tasks and/or inference tasks based on any suitable input values.

In a preferred embodiment, a corpora of raw machine learning training data includes a plurality of distinct corpus of machine learning training data. In such embodiment, each of the plurality of distinct corpus of machine learning training data may be generated and/or collected for the purposes of training an ensemble of distinct machine learning classifiers used for classifying user utterances or user input in a deployed dialogue system. For instance, a corpora of raw machine learning training data may include forty-seven (47) distinct corpus of machine training data, for example, for training 47 distinct machine learning classifiers implemented within a deployed dialogue system or the like. In such example, each of the 47 distinct machine learning classifiers may be implemented for categorizing and/or classifying user input according to one of the 47 distinct classification intents of the deployed system.

In some embodiments, S 215 may additionally function to define a set of prompts (seed samples) for sourcing raw machine learning training data for each of a plurality of intent classification tasks and/or for each distinct machine learning classifier of an artificially intelligent dialogue system. In some embodiments, the set of prompts may be engineered based on a definition and/or specification of an intent classification task or the like. For instance, an engineer or a developer associated with a dialogue system may function to generate a set of prompts based on their understanding of the intent classification tasks of a deployed dialogue system or the like.

Additionally, or alternatively, S 215 may function to generate the set of prompts based on historical or real user queries and/or user commands. In one implementation, S 215 may function to randomly sample a predetermined number of user utterances from a deployed system and convert the random samples into prompts for constructing paraphrasing requests for sourcing training data. In another implementation, S 215 may function to randomly sample a predetermined number of user utterances from a machine learning test corpus and convert the random samples into prompts for constructing scenario-driven prompts and paraphrasing requests for sourcing training data.

In a first implementation, S 215 may function to define training data sourcing parameters comprising a set of prompts for sourcing raw machine learning training data for a classification intent. The set of prompts may define a set of scenarios that enable the generation of raw machine learning data responsive to the scenario. Accordingly, S 215 may function to define instructions for a scenario-driven request for raw machine learning training data from a remote crowdsourcing platform or the like. In the scenario-driven approach, the set of prompts describe or include real-world situations or circumstances that requires or prompts responses to the real-world circumstances or situations. The responses to the real-world situation of a prompt preferably comprises raw machine learning training data. Suitably, a scenario-driven prompt functions to simulate real world situations that enable the creation of natural user queries and/or commands (requests) resembling real user queries and/or commands.

In a first variation, S 215 may function to define a set of prompts for sourcing raw machine learning training data for an intent c

CLAIMS

Claims ( 17 )

What is claimed:

1. A system for intelligently identifying machine learning training data for implementing a machine learning-based dialogue service, the system comprising:

one or more sources of machine learning training data;

one or more hardware computing servers implementing a machine learning-based dialogue service that:

constructs a corpora of machine learning test corpus that comprise a plurality of historical queries and/or historical commands test-sampled from one or more production logs of a deployed dialogue system, the corpora of machine learning test corpus relates to a baseline set of user queries and/or user commands used in calculating efficacy metrics of raw machine learning data;

configures one or more training data sourcing parameters to source a corpora of raw machine learning training data from the one or more sources of machine learning training data;

obtains, from the one or more sources of machine learning training data, the corpora of raw machine learning training data based on the one or more training data sourcing parameters;

calculates, using the one or more hardware computing servers, efficacy metrics of the corpora of raw machine learning training data, wherein calculating the efficacy metrics includes:

using the corpora of machine learning test corpus to calculate a coverage metric value that indicates a degree to which the corpora of raw machine learning training data represents possibilities of expressing a target classification intent of the corpora of machine learning test corpus;

calculating the coverage metric value for each of a plurality of distinct corpus of machine learning training data within the corpora of raw machine learning training data, wherein calculating the coverage metric value for each of the plurality of distinct corpus of machine learning training data includes:

[i] selecting a subject test corpus datum from within a subject distinct machine learning test corpus of the corpora of machine learning test corpus;

[ii] constructing a plurality of diversity pairwise comprising the subject test corpus datum and each training data within a subject distinct corpus of machine learning training data of the corpora of raw machine learning training data;

[iii] calculating a semantic similarities value of each of the plurality of diversity pairwise involving the subject test corpus training datum;

[iv] identifying a minimum diversity metric value for the subject test corpus datum based on the semantic similarities value of each of the plurality of diversity pairwise involving the subject test corpus training datum;

[v] calculating a minimum diversity metric value for each remaining test corpus datum within the subject distinct machine learning test corpus; and

[vi] calculating the coverage metric value for the subject distinct corpus of machine learning training data based on the minimum diversity metric value for the subject test corpus datum and for each of the remaining test corpus datum of the subject distinct machine learning test corpus;

calculating the coverage metric value for the corpora of raw machine learning training data based on the coverage metric value for each of the plurality of distinct corpus of machine learning training data within the corpora;

identifies whether to train at least one machine learning classifier of the machine learning-based dialogue system using the corpora of raw machine learning training data based on whether the calculated coverage metric value satisfies a predetermined coverage metric threshold.

2. The system according to claim 1 , wherein

calculating the efficacy metrics includes calculating a diversity metric value of the corpora of raw machine learning training data.

3. The system according to claim 1 , wherein the machine learning-based dialogue service further:

uses the corpora of raw machine learning training data, as machine learning training input, to train the at least one machine learning classifier if a calculated coverage metric value of the corpora of machine learning training data satisfies a minimum coverage metric threshold.

4. The system according to claim 3 , wherein the machine learning-based dialogue service further:

responsive to training the at least one machine learning classifier using the corpora of raw machine learning training data, deploys the at least one machine learning classifier into a live implementation of the artificially intelligent dialogue system.

5. The system according to claim 1 , wherein

the machine learning-based dialogue service calculates the coverage metric value of the corpora of raw machine learning training data according to the following equations:

â¡

(

)

â¢

grams

â¢

grams

â¢

grams

â¢

grams

where: N is a maximum n-gram length, a is a first sentence in a diversity pairwise comparison, b is a second sentence in the diversity pairwise, D is the reverse of the mean Jaccard Index between the sentences a and b n-gram sets, and

CVG

â¡

(

)

â¢

max

â¢

(

â¡

(

)

where: I is a set of classification intents, Xi is a set of training datum labeled with the classification intent i within the corpora of raw machine learning training data X, and Yi is a test corpus datum labelled with the classification intent i within the corpora of machine learning test corpus, CVG is the coverage for the corpora of raw machine learning training data X and the corpora of machine learning test corpus Y.

6. The system according to claim 1 , wherein

analyzing the efficacy metrics of the corpora of raw machine learning training data includes:

calculating a diversity metric value for each of a plurality of distinct corpus of machine learning training data within the corpora of raw machine learning training data, wherein the diversity metric value relates to a measure indicating a level of heterogeneity among machine learning data within a distinct corpus of machine learning training data; and

calculating an aggregated diversity metric value for the corpora of raw machine learning training data based on the diversity metric value for each of the plurality of distinct corpus of machine learning training data within the corpora.

7. The system according to claim 6 , wherein

the machine learning-based dialogue service calculates the diversity metric value of the corpora of raw machine learning training data according to the following equations:

â¡

(

)

â¢

grams

â¢

grams

â¢

grams

â¢

grams

DIV

â¡

(

)

â¢

â¡

[

â¢

â¡

(

)

]

where: I is a set of classification intents, Xi is a set of training datum labeled the classification intent i within the corpora of raw machine learning training data X and DIV is the diversity of the corpora of raw machine learning training data X.

8. The system according to claim 1 , wherein:

the corpora of machine learning test corpus is defined by a plurality of distinct machine learning test corpus,

each of the plurality of distinct machine learning test corpus is associated with a distinct intent classification task of the machine learning-based dialogue service, and

each of the plurality of distinct machine learning test corpus includes at least one subset of the plurality of historical queries and/or historical commands obtained from a deployed dialogue system.

9. The system according to claim 1 , wherein

the machine learning-based dialogue service further constructs the corpora of machine learning test corpus using a plurality of engineered queries and/or engineered commands, and

each of the plurality of engineered queries and/or engineered commands is artificially generated for one or more identified intent classification tasks.

10. The system according to claim 1 , wherein

configuring the one or more training data sourcing parameters includes:

generating a plurality of distinct sets of prompts for sourcing raw machine learning training data for each of a plurality of intent classification tasks of the machine learning-based dialogue service.

11. The system according to claim 10 , wherein:

generating the plurality of distinct sets of prompts is based on a plurality of historical user queries and/or a plurality of historical user commands,

generating the plurality of distinct sets of prompts includes:

test sampling by the machine learning-based dialogue service the plurality of historical user queries and/or the plurality of historical user commands from one or more production logs of a deployed dialogue system, and

converting the plurality of historical user queries and/or the plurality of historical user commands into the set of prompts for sourcing raw machine learning training data.

12. The system according to claim 10 , wherein

the plurality of distinct sets of prompts comprises a combination of:

a plurality of scenario-driven prompts, wherein each of the plurality of scenario-driven prompts describes a real-world circumstance for which a response is required; and

a plurality of paraphrasing requests, wherein each of the plurality of paraphrasing requests includes an instruction to rephrase and/or paraphrase a given prompt or a given statement.

13. The system according to claim 12 , wherein:

the machine learning-based dialogue service constructs a composition of the plurality of distinct sets of prompts to include a predetermined ratio of scenario-driven prompts to paraphrasing requests, and

a number of the plurality of scenario-driven prompts in the predetermined ratio of scenario-driven prompts is greater than a number of the plurality of the paraphrasing requests.

14. The system according to claim 1 , further comprising:

calculating by the machine learning-based dialogue service:

an aggregated coverage metric value for the corpora of raw machine learning training data, wherein calculating the aggregated diversity metric includes:

calculating an average coverage metric value by calculating a sum of the coverage metric value for each of the plurality of distinct corpus of machine learning training data that defines the corpora and dividing the sum by a number of the distinct corpus of machine learning training data within the corpora.

15. The system according to claim 7 , wherein

calculating the diversity metric value for each of the plurality of distinct corpus of machine learning training data includes:

[i] selecting a subject training datum from training data within a subject distinct corpus of machine learning training data of the plurality of distinct corpus of machine learning training data;

[ii] constructing a plurality of diversity pairwise comprising the subject training datum and each of a remaining training data within the subject distinct corpus of machine learning training data;

[iii] calculating a semantic difference value of each of the plurality of diversity pairwise involving the subject training datum;

[iv] calculating a specific diversity metric value for the subject training datum based on an average of the semantic difference value of each of the plurality of diversity pairwise involving the subject training datum;

[v] calculating a specific diversity metric value for each of the remaining training data within the subject distinct corpus of machine learning training data; and

[vi] calculating the diversity metric value for the subject distinct corpus of machine learning training data based on the specific diversity metric value for the subject training datum and for each of the remaining training data of the subject distinct corpus of the machine learning training data.

16. A method for intelligently curating machine learning training data for implementing a machine learning-based dialogue service, the method comprising:

a machine learning-based dialogue service implemented by one or more hardware computing servers:

constructing a corpora of machine learning test corpus that comprise a plurality of historical queries and/or historical commands test-sampled from one or more production logs of a deployed dialogue system, the corpora of machine learning test corpus relates to a baseline set of user queries and/or user commands used in calculating efficacy metrics of raw machine learning data;

configuring one or more training data sourcing parameters to source a corpora of raw machine learning training data from the one or more sources of machine learning training data;

obtaining, from the one or more sources of machine learning training data, the corpora of raw machine learning training data based on the one or more training data sourcing parameters;

calculating, using the one or more hardware computing servers, efficacy metrics of the corpora of raw machine learning training data, wherein calculating the efficacy metrics includes:

calculating the coverage metric value for each of a plurality of distinct corpus of machine learning training data within the corpora of raw machine learning training data;

wherein the machine learning-based dialogue service calculates the coverage metric value of the corpora of raw machine learning training data according to:

â¡

(

)

â¢

grams

â¢

grams

â¢

grams

â¢

grams

CVG

â¡

(

)

â¢

max

â¢

(

â¡

(

)

identifying whether to train at least one machine learning classifier of the machine learning-based dialogue system using the corpora of raw machine learning training data based on whether the calculated coverage metric value satisfies a predetermined coverage metric threshold.

17. A method for intelligently curating machine learning training data for implementing a machine learning-based dialogue service, the method comprising:

a machine learning-based dialogue service implemented by distributed network of computers:

configuring one or more training data sourcing parameters to source a corpora of raw machine learning training data from the one or more sources of machine learning training data;

obtaining, from the one or more sources of machine learning training data, the corpora of raw machine learning training data based on the one or more training data sourcing parameters;

calculating, using the one or more hardware computing servers, efficacy metrics of the corpora of raw machine learning training data, wherein calculating the efficacy metrics includes using a corpora of machine learning test corpus to calculate a coverage metric value that indicates a degree to which the corpora of raw machine learning training data represents possibilities of expressing a target classification intent of the corpora of machine learning test corpus;

calculating the coverage metric value for each of a plurality of distinct corpus of machine learning training data within the corpora of raw machine learning training data;

wherein the machine learning-based dialogue service calculates the coverage metric value of the corpora of raw machine learning training data according to:

â¡

(

)

â¢

grams

â¢

grams

â¢

grams

â¢

grams

CVG

â¡

(

)

â¢

max

â¢

(

â¡

(

)

US16/379,978

2018-03-26

2019-04-10

Systems and methods for intelligently curating machine learning training data and improving machine learning model performance

Active

US10679100B2

( en )

Priority Applications (2)

Application Number

Priority Date

Filing Date

Title

US16/379,978

US10679100B2

( en )

2018-03-26

2019-04-10

Systems and methods for intelligently curating machine learning training data and improving machine learning model performance

US16/864,120

US20200272855A1

( en )

2018-03-26

2020-04-30

Systems and methods for intelligently curating machine learning training data and improving machine learning model performance

Applications Claiming Priority (3)

Application Number

Priority Date

Filing Date

Title

US201862648318P

2018-03-26

US16/143,773

US10303978B1

( en )

2018-03-26

2018-09-27

Systems and methods for intelligently curating machine learning training data and improving machine learning model performance

US16/379,978

US10679100B2

( en )

2018-03-26

2019-04-10

Systems and methods for intelligently curating machine learning training data and improving machine learning model performance

Related Parent Applications (1)

Application Number

Title

Priority Date

Filing Date

US16/143,773

Continuation

US10303978B1

( en )

2018-03-26

2018-09-27

Systems and methods for intelligently curating machine learning training data and improving machine learning model performance

Related Child Applications (1)

Application Number

Title

Priority Date

Filing Date

US16/864,120

Continuation

US20200272855A1

( en )

2018-03-26

2020-04-30

Systems and methods for intelligently curating machine learning training data and improving machine learning model performance

Publications (2)

Publication Number

Publication Date

US20190294925A1

US20190294925A1 ( en )

2019-09-26

US10679100B2

true

US10679100B2 ( en )

2020-06-09

Family

ID=66636245

Family Applications (3)

Application Number

Title

Priority Date

Filing Date

US16/143,773

Active

US10303978B1

( en )

2018-03-26

2018-09-27

Systems and methods for intelligently curating machine learning training data and improving machine learning model performance

US16/379,978

Active

US10679100B2

( en )

2018-03-26

2019-04-10

Systems and methods for intelligently curating machine learning training data and improving machine learning model performance

US16/864,120

Abandoned

US20200272855A1

( en )

2018-03-26

2020-04-30

Systems and methods for intelligently curating machine learning training data and improving machine learning model performance

Family Applications Before (1)

Application Number

Title

Priority Date

Filing Date

US16/143,773

Active

US10303978B1

( en )

2018-03-26

2018-09-27

Systems and methods for intelligently curating machine learning training data and improving machine learning model performance

Family Applications After (1)

Application Number

Title

Priority Date

Filing Date

US16/864,120

Abandoned

US20200272855A1

( en )

2018-03-26

2020-04-30

Systems and methods for intelligently curating machine learning training data and improving machine learning model performance

Country Status (1)

Country

Link

( 3 )

US10303978B1

( en )

Cited By (5)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20220283887A1

( en )

2020-05-14

2022-09-08

State Farm Mutual Automobile Insurance Company

System and method for automatically monitoring and diagnosing user experience problems

WO2022231970A1

( en )

2021-04-28

2022-11-03

GE Precision Healthcare LLC

Data diversity visualization and quantification for machine learning models

US11811585B2

( en )

2021-03-23

2023-11-07

International Business Machines Corporation

Measuring incident management process efficiency metrics utilizing real-time conversation analysis

US20250021550A1

( en )

2023-07-11

2025-01-16

Beijing Volcano Engine Technology Co., Ltd.

Database-based data query method, apparatus, electronic device and storage medium

US12287848B2

( en )

2021-06-11

2025-04-29

International Business Machines Corporation

Learning Mahalanobis distance metrics from data

Families Citing this family (52)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US10878144B2

( en )

2017-08-10

2020-12-29

Allstate Insurance Company

Multi-platform model processing and execution management engine

US11755949B2

( en )

2017-08-10

2023-09-12

Allstate Insurance Company

Multi-platform machine learning systems

US10909422B1

( en )

2017-08-22

2021-02-02

United Services Automobile Association (Usaa)

Customer service learning machine

US10970400B2

( en )

2018-08-14

2021-04-06

Kenna Security, Inc.

Multi-stage training of machine learning models

US11017028B2

( en )

2018-10-03

2021-05-25

The Toronto-Dominion Bank

Systems and methods for intelligent responses to queries based on trained processes

US10943185B1

( en )

2018-10-23

2021-03-09

Bank Of America Corporation

Supervised machine-learning training platform with isolated update testing and bi-directional update propogation

US10896034B2

( en )

2018-11-14

2021-01-19

Babu Vinod

Methods and systems for automated screen display generation and configuration

US11790245B2

( en )

2019-01-22

2023-10-17

International Business Machines Corporation

Cognitive machine learning for semantic network

US11354545B1

( en )

2019-02-28

2022-06-07

DefinedCrowd Corporation

Automated data generation, post-job validation and agreement voting using automatic results generation and humans-in-the-loop, such as for tasks distributed by a crowdsourcing system

US11288273B2

( en )

2019-07-02

2022-03-29

Sap Se

Machine learning based plug-in for providing access to cloud-based analytics engine

US10691897B1

( en )

2019-08-29

2020-06-23

Accenture Global Solutions Limited

Artificial intelligence based virtual agent trainer

US11645456B2

( en )

2020-01-28

2023-05-09

Microsoft Technology Licensing, Llc

Siamese neural networks for flagging training data in text-based machine learning

US11636389B2

( en )

2020-02-19

2023-04-25

Microsoft Technology Licensing, Llc

System and method for improving machine learning models by detecting and removing inaccurate training data

US11514364B2

( en )

2020-02-19

2022-11-29

Microsoft Technology Licensing, Llc

Iterative vectoring for constructing data driven machine learning models

US20210264283A1

( en )

2020-02-24

2021-08-26

International Business Machines Corporation

Dataset creation for deep-learning model

US11538457B2

( en )

2020-03-30

2022-12-27

Oracle International Corporation

Noise data augmentation for natural language processing

US11354595B2

( en )

2020-04-01

2022-06-07

International Business Machines Corporation

Similarity-based hierarchical data loading for machine learning training

US11481682B2

( en )

2020-05-07

2022-10-25

International Business Machines Corporation

Dataset management in machine learning

US11842256B2

( en )

2020-05-15

2023-12-12

International Business Machines Corporation Armonk

Ensemble training in a distributed marketplace

CN111680123B

( en )

2020-05-25

2024-01-26

è¾è®¯ç§æï¼æ·±å³ï¼æéå¬å¸

Training method, device, computer equipment and storage medium for dialogue model

US12033258B1

( en )

2020-06-05

2024-07-09

Meta Platforms Technologies, Llc

Automated conversation content items from natural language

US11550605B2

( en )

2020-06-30

2023-01-10

Kasisto, Inc.

Building and managing cohesive interaction for virtual assistants

CN111858888B

( en )

2020-07-13

2023-05-30

Multi-round dialogue system of check-in scene

CN111881251B

( en )

2020-07-23

2023-10-13

å¹³å®ç§æï¼æ·±å³ï¼æéå¬å¸

AI telephone sales testing method and device, electronic equipment and storage medium

US11756553B2

( en )

2020-09-17

2023-09-12

International Business Machines Corporation

Training data enhancement

CN112101570B

( en )

2020-09-22

2024-09-17

åäº¬ç¾åº¦ç½è®¯ç§ææéå¬å¸

A model training and use method, device, equipment and storage medium

CN112185359B

( en )

2020-09-28

2023-08-29

å¹¿å·ç§çç§ææéå¬å¸

Word coverage rate-based voice training set minimization method

JP2022065878A

( en )

2020-10-16

2022-04-28

å¯å£«éæ ªå¼ä¼ç¤¾

Information processing program, information processing method, and information processing apparatus

US11693878B2

( en )

2020-11-19

2023-07-04

International Business Machines Corporation

Generation of a dataset in the format of a machine learning framework

US12210891B2

( en )

2020-12-18

2025-01-28

Advanced Micro Devices (Shanghai) Co., Ltd.

Modifying device status in single virtual function mode

JP7679630B2

( en )

2021-01-28

2025-05-20

å¯å£«éæ ªå¼ä¼ç¤¾

Information processing program, information processing method, and information processing device

CN112785001B

( en )

2021-03-02

2024-01-05

ååå¸èå¤§å¦

Artificial intelligence educational back-province robot for overcoming discrimination and prejudice

CN112631892B

( en )

2021-03-10

2021-05-25

ä¸æºå³ç±é(åäº¬)ä¿¡æ¯ç§ææéå¬å¸

Method, computing device, and computer medium for predicting server health status

CN113064631B

( en )

2021-04-07

2025-10-31

æ³°è±ªè½¯ä»¶è¡ä»½æéå¬å¸

Robot version regression management method and device, robot and storage medium

US12079884B2

( en )

2021-04-19

2024-09-03

Meta Platforms Technologies, Llc

Automated memory creation and retrieval from moment content items

CA3172934A1

( en )

2021-04-29

2022-10-29

James Liu

Artificial intelligence assisted transfer tool

CN113314106A

( en )

2021-05-19

2021-08-27

å½ç½è¾½å®ççµåæéå¬å¸

Electric power information query and regulation function calling method based on voice and intention recognition

US20220398524A1

( en )

2021-06-11

2022-12-15

Aktify, Inc.

Consumer interaction agent management platform

US20230101734A1

( en )

2021-09-24

2023-03-30

Trueblue, Inc.

Machine learning model to fill gaps in adaptive rate shifting

US12130888B2

( en )

2021-11-05

2024-10-29

Raytheon Company

Feature extraction, labelling, and object feature map

DE102021214329A1

( en )

2021-12-14

2023-06-15

Robert Bosch Gesellschaft mit beschrÃ¤nkter Haftung

Method and device for determining coverage of a data set for a machine learning system with regard to trigger events

US20230267343A1

( en )

2022-02-23

2023-08-24

International Business Machines Corporation

Automated code-mixed natural language processing for artificial intelligence-based question answering techniques

CN114419399B

( en )

2022-03-22

2022-10-21

Automatic generation method of machine vision deep learning sample, computer and storage medium

US12536583B2

( en )

2022-05-02

2026-01-27

Truist Bank

Efficient processing of extreme inputs for real-time prediction of future events

US12547891B2

( en )

2022-05-02

2026-02-10

Truist Bank

Real-time prediction of future events using integrated input relevancy

US20230351491A1

( en )

2022-05-02

2023-11-02

Truist Bank

Accelerated model training for real-time prediction of future events

US20240285951A1

( en )

2023-02-24

2024-08-29

Boston Scientific Neuromodulation Corporation

Personalized neurostimulation closed loop programming training interface

US20250036636A1

( en )

2023-07-26

2025-01-30

Ford Global Technologies, Llc

Intelligent virtual assistant selection

CN117271718B

( en )

2023-08-17

2026-01-30

æ·±å³å¸å£°æ¬ç§ææéå¬å¸

Process control methods, devices, electronic equipment and storage media

WO2025072802A2

( en )

2023-09-27

2025-04-03

Black Knight Ip Holding Company, Llc

Methods and systems for implementing an intelligent digital assistant in a multi-application ne

Systems and methods for intelligently curating machine learning training data … — Clinc, Inc. (US10679100B2)

Related documents