ABSTRACT
Abstract
The present disclosure relates to systems and methods for an intelligent assistant (e.g., a chatbot) that can be used to enable a user to generate a machine learning system. Techniques can be used to automatically generate a machine learning system to assist a user. In some cases, the user may not be a software developer and may have little or no experience in either machine learning techniques or software programming. In some embodiments, a user can interact with an intelligent assistant. The interaction can be aural, textual, or through a graphical user interface. The chatbot can translate natural language inputs into a structural representation of a machine learning solution using an ontology. In this way, a user can work with artificial intelligence without being a data scientist to develop, train, refine, and compile machine learning models as stand-alone executable code.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
This application is a continuation of U.S. patent application Ser. No. 18/501,716, filed Nov. 3, 2023, which is a continuation of U.S. patent application Ser. No. 18/100,458 filed Jan. 23, 2023, now U.S. Pat. No. 11,847,578, which is a continuation of U.S. patent application Ser. No. 16/893,193 filed Jun. 4, 2020, now U.S. Pat. No. 11,562,267, which claims priority of U.S. Provisional Patent Application No. 62/900,537 filed Sep. 14, 2019, entitled âAUTOMATED MACHINE LEARNING SYSTEMS AND METHODSâ. Each of these applications is hereby incorporated by reference in their entireties and for all purposes.
FIELD
The present disclosure relates to systems and techniques for machine learning. More particularly, the present disclosure relates to systems and techniques for generating and managing a library of machine learning applications.
BACKGROUND
Machine learning has a wide range of applications, such as search engines, medical diagnosis, text and handwriting recognition, image processing and recognition, load forecasting, marketing and sales diagnosis, chatbots, autonomous driving, and the like. Various types and versions of machine learning models may be generated for similar applications using training data based on different technologies, languages, libraries, and the like, and thus may lack interoperability. In addition, different models may have different performances in different contexts and/or for different types of input data. Data scientists may not have the programming skills to generate the code necessary to build custom machine learning models. In addition, available machine learning tools do not store the various machine learning model components as part of a library to allow for efficient reuse of routines in other machine learning models.
Existing machine learning applications can require considerable programming knowledge by a data scientist to design and construct a machine learning application to solve specific problems. Intuitive interfaces can assist the data scientist construct a machine learning application through a series of queries.
Some organizations can store data from multiple clients or suppliers with customizable schemas. These customizable schemas may not match standardized data storage schemas used by existing machine learning models. Therefore, these other systems would need to perform a reconciliation process prior to using the stored data. The reconciliation process can be either a manual process or through a tedious extract, transform, load automated process prior to using the data for generating machine learning applications.
Machine learning applications based only on metrics (e.g., Quality of Service (QoS) or Key Performance Indicators) may not be sufficient to compose pipelines with minimal human intervention for a self-adaptive architecture. Pre-existing machine learning tools do not combine non-logical based and logic-based semantic services to generate a machine learning application.
Existing machine learning tools tend to maximize classification accuracy over new data from the same source. In doing so, these existing frameworks attempt to compose the best pipeline based on model performance. However, model performance on its own does not cover the cases where the pipeline must be adapted at run-time due to changes in online data metrics or QoS violations.
BRIEF SUMMARY
Certain aspects and features of the present disclosure relate to machine learning platform that generates a library of components to generate machine learning models and machine learning applications. The machine learning infrastructure system allows a user (i.e., a data scientist) to generate machine learning applications without having detailed knowledge of the cloud-based network infrastructure or knowledge of how to generate code for building the model. The machine learning platform can analyze the identified data and the user provided desired prediction and performance characteristics to select one or more library components and associated API to generate a machine learning application. The machine learning techniques can monitor and evaluate the outputs of the machine learning model to allow for feedback and adjustments to the model. The machine learning application can be trained, tested, and compiled for export as stand-alone executable code.
The machine learning platform can generate and store one or more library components that can be used for other machine learning applications. The machine learning platform can allow users to generate a profile which allows the platform to make recommendations based on a user's historical preferences. The model creation engine can detect the number and type of infrastructure resources necessary to achieve the desired results within the desired performance criteria.
A chatbot can provide an intuitive interface to allow the data scientist to generate a machine learning application without considerable programming experience. A chatbot is able to translate natural language into a structured representation of a machine learning solution using a conversational interface. A chatbot can be used to indicate the location of data, select a type of machine learning solution, display optimal solutions that best meet the constraints, and recommend the best environment to deploy the solution.
A self-adjusting corporation-wide discovery and integration feature can review a client's data store, review the labels for the various data schema, and effectively map the client's data schema to classifications used by the machine learning model. The various techniques can automatically select the features that are predictive for each individual use case (i.e., one client), effectively making a machine learning solution client-agnostic for the application developer. A weighted list of common representations of each feature for a particular machine learning solution can be generated and stored. When new data is added to the data store, a matching service can automatically detect which features should be fed into the machine-learning solution based at least in part on the weighted list. The weighted list can be updated as new data is made available to the model.
Existing data ontologies can be used for generating machine learning solutions for a high-precision search of relevant services to compose pipelines with minimal human intervention. Data ontologies can be used to create a combination of non-logic based and logic-based sematic services that can significantly outperform both kinds of selection in terms of precision. QoS and product KPI constraints can be used as part of architecture selection. For data sets without existing ontologies, one or more ontologies be generated.
The proposed system can use best available models at the time of construction to solve problems using the machine learning application. An adaptive pipelining composition service can identify and incorporate one or more new models into the machine learning application. The machine learning application with the new model can be tested off-line with the results being compared with ground truth data. If the machine learning application with the new model outperforms the previously used model, the machine learning application can be upgraded and auto-promoted to production. One or more parameters may also be discovered. The new parameters may be incorporated into the existing model in an off-line mode. The machine learning application with the new parameters can be tested off-line and the results can be compared with previous results with existing parameters. If the new parameters outperform the existing parameters as compared with ground-truth data, the machine learning application can be auto-promoted to production.
These and other embodiments are described in detail below. For example, other embodiments are directed to systems, devices, and computer readable media associated with methods described herein.
A better understanding of the nature and advantages of embodiments of the present disclosed may be gained with reference to the following detailed description and the accompanying drawings.
BRIEF DESCRIPTION OF THE DRAWINGS
The specification makes reference to the following appended figures, in which use of like reference numerals in different figures is intended to illustrate like or analogous components.
FIG. 1 is a block diagram illustrating an exemplary machine learning infrastructure system.
FIG. 2 is a flowchart illustrating an exemplary technique for generating a machine learning application.
FIG. 3 is a flowchart illustrating another exemplary technique for generating a machine learning application.
FIG. 4 depicts a distributed system that implements a bot system for communicating with end users using a messaging application in accordance with various embodiments
FIG. 5 depicts an integrated system including a bot system and a bot analytic system for monitoring, analyzing, visualizing, and improving the performance of a bot system in accordance with various embodiments.
FIG. 6 depicts a distributed system that implements a bot system for communicating with an end user using a messaging application according to certain embodiments.
FIG. 7 illustrates a flowchart for a technique for generating a machine learning application using a chatbot.
FIG. 8 illustrates a flowchart for protecting a serialized model and the storing private keys in a secure location.
FIGS. 9 A and 9 B illustrate exemplary user interface screens generated by a software development environment.
FIG. 10 illustrates an exemplary a distributed computing environment including a code integration request prediction server that may be used for receiving and processing input data, generating and using machine-learning or artificial intelligence models to generate outputs predictive of code integration requests.
FIG. 11 illustrates an exemplary flow chart for a real time code analysis plug-in.
FIG. 12 illustrates an exemplary flow chart for an automated on-demand library check.
FIGS. 13 A and 13 B illustrates exemplary user interface screens generated by a software development environment.
FIG. 14 illustrates a flow chart for an exemplary technique for self-adjusting corporation-wide feature discovery and integration.
FIG. 15 illustrates a flow chart for a second exemplary technique for self-adjusting corporation-wide feature discovery and integration.
FIG. 16 illustrates a block diagram illustrating an exemplary system for generating product graphs using ontologies.
FIG. 17 illustrates an exemplary flow chart for an adaptive and context-aware automated service composition for machine learning.
FIG. 18 illustrates a block diagram for generating one or more pipelines.
FIG. 19 illustrates an exemplary flow chart for an adaptive pipeline composition service.
FIG. 20 is a simplified diagram illustrating a distributed system for implementing one of the embodiments.
FIG. 21 is a simplified block diagram illustrating one or more components of a system environment.
FIG. 22 illustrates an exemplary computer system, in which various embodiments of the present invention may be implemented.
DETAILED DESCRIPTION
Certain embodiments of the present disclosure relate to systems, devices, computer-readable medium, and computer-implemented methods for implementing various techniques for machine learning. The machine learning techniques can allow a user (i.e., a data scientist) to generate machine learning applications without having detailed knowledge of the cloud-based network infrastructure or knowledge of how to generate code for building the model. The machine learning platform can analyze the identified data and the user provided desired prediction and performance characteristics to select one or more library components and associated API to generate a machine learning application.
The machine learning techniques can employ a chatbot to indicate the location of data, select a type of machine learning solution, display optimal solutions that best meet the constraints, and recommend the best environment to deploy the solution.
The techniques described herein can include a self-adjusting corporation-wide discovery and integration feature can review a client's data store, review the labels for the various data schema, and effectively map the client's data schema to classifications used by the machine learning model. The various techniques can automatically select the features that are predictive for each individual use case (i.e., one client), effectively making a machine learning solution client-agnostic for the application developer. A weighted list of common representations of each feature for a particular machine learning solution can be generated and stored.
The techniques can utilize existing data ontologies for generating machine learning solutions for a high-precision search of relevant services to compose pipelines with minimal human intervention. For data sets without existing ontologies, one or more ontologies be generated.
The techniques can employ an adaptive pipelining composition service to identify and incorporate or more new models into the machine learning application. The machine learning application with the new model can be tested off-line with the results being compared with ground truth data. If the machine learning application with the new model outperforms the previously used model, the machine learning application can be upgraded and auto-promoted to production.
I. Machine Learning Infrastructure Platform
FIG. 1 is a block diagram illustrating an exemplary machine learning platform 100 for generating a machine learning model. The machine learning platform 100 has various components that can be distributed between different networks and computing systems. A machine learning infrastructure library can store one or more components for generating machine learning applications 112 . All of the infrastructure required to productionize the machine learning applications 112 can be encapsulated and stored in the library.
Machine learning confi
CROSS-REFERENCE TO RELATED APPLICATIONS
This application is a continuation of U.S. patent application Ser. No. 18/501,716, filed Nov. 3, 2023, which is a continuation of U.S. patent application Ser. No. 18/100,458 filed Jan. 23, 2023, now U.S. Pat. No. 11,847,578, which is a continuation of U.S. patent application Ser. No. 16/893,193 filed Jun. 4, 2020, now U.S. Pat. No. 11,562,267, which claims priority of U.S. Provisional Patent Application No. 62/900,537 filed Sep. 14, 2019, entitled âAUTOMATED MACHINE LEARNING SYSTEMS AND METHODSâ. Each of these applications is hereby incorporated by reference in their entireties and for all purposes.
FIELD
The present disclosure relates to systems and techniques for machine learning. More particularly, the present disclosure relates to systems and techniques for generating and managing a library of machine learning applications.
BACKGROUND
Machine learning has a wide range of applications, such as search engines, medical diagnosis, text and handwriting recognition, image processing and recognition, load forecasting, marketing and sales diagnosis, chatbots, autonomous driving, and the like. Various types and versions of machine learning models may be generated for similar applications using training data based on different technologies, languages, libraries, and the like, and thus may lack interoperability. In addition, different models may have different performances in different contexts and/or for different types of input data. Data scientists may not have the programming skills to generate the code necessary to build custom machine learning models. In addition, available machine learning tools do not store the various machine learning model components as part of a library to allow for efficient reuse of routines in other machine learning models.
Existing machine learning applications can require considerable programming knowledge by a data scientist to design and construct a machine learning application to solve specific problems. Intuitive interfaces can assist the data scientist construct a machine learning application through a series of queries.
Some organizations can store data from multiple clients or suppliers with customizable schemas. These customizable schemas may not match standardized data storage schemas used by existing machine learning models. Therefore, these other systems would need to perform a reconciliation process prior to using the stored data. The reconciliation process can be either a manual process or through a tedious extract, transform, load automated process prior to using the data for generating machine learning applications.
Machine learning applications based only on metrics (e.g., Quality of Service (QoS) or Key Performance Indicators) may not be sufficient to compose pipelines with minimal human intervention for a self-adaptive architecture. Pre-existing machine learning tools do not combine non-logical based and logic-based semantic services to generate a machine learning application.
Existing machine learning tools tend to maximize classification accuracy over new data from the same source. In doing so, these existing frameworks attempt to compose the best pipeline based on model performance. However, model performance on its own does not cover the cases where the pipeline must be adapted at run-time due to changes in online data metrics or QoS violations.
BRIEF SUMMARY
Certain aspects and features of the present disclosure relate to machine learning platform that generates a library of components to generate machine learning models and machine learning applications. The machine learning infrastructure system allows a user (i.e., a data scientist) to generate machine learning applications without having detailed knowledge of the cloud-based network infrastructure or knowledge of how to generate code for building the model. The machine learning platform can analyze the identified data and the user provided desired prediction and performance characteristics to select one or more library components and associated API to generate a machine learning application. The machine learning techniques can monitor and evaluate the outputs of the machine learning model to allow for feedback and adjustments to the model. The machine learning application can be trained, tested, and compiled for export as stand-alone executable code.
The machine learning platform can generate and store one or more library components that can be used for other machine learning applications. The machine learning platform can allow users to generate a profile which allows the platform to make recommendations based on a user's historical preferences. The model creation engine can detect the number and type of infrastructure resources necessary to achieve the desired results within the desired performance criteria.
A chatbot can provide an intuitive interface to allow the data scientist to generate a machine learning application without considerable programming experience. A chatbot is able to translate natural language into a structured representation of a machine learning solution using a conversational interface. A chatbot can be used to indicate the location of data, select a type of machine learning solution, display optimal solutions that best meet the constraints, and recommend the best environment to deploy the solution.
A self-adjusting corporation-wide discovery and integration feature can review a client's data store, review the labels for the various data schema, and effectively map the client's data schema to classifications used by the machine learning model. The various techniques can automatically select the features that are predictive for each individual use case (i.e., one client), effectively making a machine learning solution client-agnostic for the application developer. A weighted list of common representations of each feature for a particular machine learning solution can be generated and stored. When new data is added to the data store, a matching service can automatically detect which features should be fed into the machine-learning solution based at least in part on the weighted list. The weighted list can be updated as new data is made available to the model.
Existing data ontologies can be used for generating machine learning solutions for a high-precision search of relevant services to compose pipelines with minimal human intervention. Data ontologies can be used to create a combination of non-logic based and logic-based sematic services that can significantly outperform both kinds of selection in terms of precision. QoS and product KPI constraints can be used as part of architecture selection. For data sets without existing ontologies, one or more ontologies be generated.
The proposed system can use best available models at the time of construction to solve problems using the machine learning application. An adaptive pipelining composition service can identify and incorporate one or more new models into the machine learning application. The machine learning application with the new model can be tested off-line with the results being compared with ground truth data. If the machine learning application with the new model outperforms the previously used model, the machine learning application can be upgraded and auto-promoted to production. One or more parameters may also be discovered. The new parameters may be incorporated into the existing model in an off-line mode. The machine learning application with the new parameters can be tested off-line and the results can be compared with previous results with existing parameters. If the new parameters outperform the existing parameters as compared with ground-truth data, the machine learning application can be auto-promoted to production.
These and other embodiments are described in detail below. For example, other embodiments are directed to systems, devices, and computer readable media associated with methods described herein.
A better understanding of the nature and advantages of embodiments of the present disclosed may be gained with reference to the following detailed description and the accompanying drawings.
BRIEF DESCRIPTION OF THE DRAWINGS
The specification makes reference to the following appended figures, in which use of like reference numerals in different figures is intended to illustrate like or analogous components.
FIG. 1 is a block diagram illustrating an exemplary machine learning infrastructure system.
FIG. 2 is a flowchart illustrating an exemplary technique for generating a machine learning application.
FIG. 3 is a flowchart illustrating another exemplary technique for generating a machine learning application.
FIG. 4 depicts a distributed system that implements a bot system for communicating with end users using a messaging application in accordance with various embodiments
FIG. 5 depicts an integrated system including a bot system and a bot analytic system for monitoring, analyzing, visualizing, and improving the performance of a bot system in accordance with various embodiments.
FIG. 6 depicts a distributed system that implements a bot system for communicating with an end user using a messaging application according to certain embodiments.
FIG. 7 illustrates a flowchart for a technique for generating a machine learning application using a chatbot.
FIG. 8 illustrates a flowchart for protecting a serialized model and the storing private keys in a secure location.
FIGS. 9 A and 9 B illustrate exemplary user interface screens generated by a software development environment.
FIG. 10 illustrates an exemplary a distributed computing environment including a code integration request prediction server that may be used for receiving and processing input data, generating and using machine-learning or artificial intelligence models to generate outputs predictive of code integration requests.
FIG. 11 illustrates an exemplary flow chart for a real time code analysis plug-in.
FIG. 12 illustrates an exemplary flow chart for an automated on-demand library check.
FIGS. 13 A and 13 B illustrates exemplary user interface screens generated by a software development environment.
FIG. 14 illustrates a flow chart for an exemplary technique for self-adjusting corporation-wide feature discovery and integration.
FIG. 15 illustrates a flow chart for a second exemplary technique for self-adjusting corporation-wide feature discovery and integration.
FIG. 16 illustrates a block diagram illustrating an exemplary system for generating product graphs using ontologies.
FIG. 17 illustrates an exemplary flow chart for an adaptive and context-aware automated service composition for machine learning.
FIG. 18 illustrates a block diagram for generating one or more pipelines.
FIG. 19 illustrates an exemplary flow chart for an adaptive pipeline composition service.
FIG. 20 is a simplified diagram illustrating a distributed system for implementing one of the embodiments.
FIG. 21 is a simplified block diagram illustrating one or more components of a system environment.
FIG. 22 illustrates an exemplary computer system, in which various embodiments of the present invention may be implemented.
DETAILED DESCRIPTION
Certain embodiments of the present disclosure relate to systems, devices, computer-readable medium, and computer-implemented methods for implementing various techniques for machine learning. The machine learning techniques can allow a user (i.e., a data scientist) to generate machine learning applications without having detailed knowledge of the cloud-based network infrastructure or knowledge of how to generate code for building the model. The machine learning platform can analyze the identified data and the user provided desired prediction and performance characteristics to select one or more library components and associated API to generate a machine learning application.
The machine learning techniques can employ a chatbot to indicate the location of data, select a type of machine learning solution, display optimal solutions that best meet the constraints, and recommend the best environment to deploy the solution.
The techniques described herein can include a self-adjusting corporation-wide discovery and integration feature can review a client's data store, review the labels for the various data schema, and effectively map the client's data schema to classifications used by the machine learning model. The various techniques can automatically select the features that are predictive for each individual use case (i.e., one client), effectively making a machine learning solution client-agnostic for the application developer. A weighted list of common representations of each feature for a particular machine learning solution can be generated and stored.
The techniques can utilize existing data ontologies for generating machine learning solutions for a high-precision search of relevant services to compose pipelines with minimal human intervention. For data sets without existing ontologies, one or more ontologies be generated.
The techniques can employ an adaptive pipelining composition service to identify and incorporate or more new models into the machine learning application. The machine learning application with the new model can be tested off-line with the results being compared with ground truth data. If the machine learning application with the new model outperforms the previously used model, the machine learning application can be upgraded and auto-promoted to production.
I. Machine Learning Infrastructure Platform
FIG. 1 is a block diagram illustrating an exemplary machine learning platform 100 for generating a machine learning model. The machine learning platform 100 has various components that can be distributed between different networks and computing systems. A machine learning infrastructure library can store one or more components for generating machine learning applications 112 . All of the infrastructure required to productionize the machine learning applications 112 can be encapsulated and stored in the library.
Machine learning configuration and interaction with the model composition engine 132 allows for selection of various library components 168 (e.g., pipelines 136 or workflows, micro services routines 140 , software modules 144 , and infrastructure modules 148 ) to define implementation of the logic of training and inference to build machine learning applications 112 . Different parameters, variables, scaling, settings, etc. for the library components 168 can be specified or determined by the model composition engine 132 . The complexity conventionally required to create the machine learning applications 112 can be performed largely automatically with the model composition engine 132 .
The library components 168 can be scalable to allows for the definition of multiple environments (e.g., different Kubernetes clusters) where the various portions of the application can be deployed to achieve any Quality of Service (QOS) or Key Performance Indicators (KPIs) specified. A Kubernetes cluster is a set of node machines for running containerized applications. The scalability can hide or abstract the complexity of the machine learning platform 100 from the application developer. A monitoring engine 156 can monitor operation of the machine learning applications 112 according to the KPI/QoS metrics 160 to assure the machine learning application 112 is performing according to requirements. In addition the monitoring engine 156 can seamlessly test end-to-end a new or evolving machine learning application at different scales, settings, loading, settings, etc. The monitoring engine 156 can recommend various adjustments to the machine learning application 112 by signaling needed changes to the model composition engine 132 .
To address scalability in some embodiments, the machine learning platform 100 creates infrastructure, which is based on a micro services architecture, making it robust and scalable. For example, various micro services routines 140 and infrastructure modules 148 can be configured and customized for embedding into the machine learning application 112 . The machine learning platform 100 can allow a developer to define the amount of resources (e.g. CPU, memory) needed for different library components 168 of the machine learning application 112 .
The machine learning platform 100 can generate highly customizable applications. The library components 168 contain a set of predefined, off-the-shelf workflows or pipelines 136 , which the application developer can incorporate into a new machine learning application 112 . A workflow specifies various micro services routines 140 , software modules 144 and/or infrastructure modules 148 configured in a particular way for a type or class of problem. In addition to this, it is also possible to define new workflows or pipelines 136 by re-using the library components or changing an existing workflow or pipeline 136 . The infrastructure modules 148 can also include services such as data gathering, process monitoring, and logging.
A model composition engine 132 can be executed on one or more computing systems (e.g., infrastructure 128 ). The model composition engine 132 can receive inputs from a user 116 through an interface 104 . The interface 104 can include various graphical user interfaces with various menus and user selectable elements. The interface 104 can include a chatbot (e.g., a text based or voice based interface). The user 116 can interact with the interface 104 to identify one or more of: a location of data, a desired prediction of machine learning application, and various performance metrics for the machine learning model. The model composition engine 132 can interface with library components 168 to identify various pipelines 136 , micro service routines 140 , software modules 144 , and infrastructure models 148 that can be used in the creation of the machine learning model 112 .
The model composition engine 132 can output one or more machine learning applications 112 . The machine learning applications 112 can be stored locally on a server or in a cloud-based network. The model composition engine 132 can output the machine learning application 112 as executable code that be run on various infrastructure 128 through the infrastructure interfaces 124 .
The model execution engine 108 can execute the machine learning application 112 on infrastructure 128 using one or more the infrastructure interfaces 124 . The infrastructure 128 can include one or more processors, one or more memories, and one or more network interfaces, one or more buses and control lines that can be used to generate, test, compile, and deploy a machine learning application 112 . In various embodiments, the infrastructure 128 can exit on a remote system 152 that is apart from the location of the user 116 . The infrastructure 128 can interact with the model execution engine 108 through the infrastructure interfaces 124 The model execution engine 108 can input the performance characteristics (e.g., KPI/QOS metrics storage 160 ) and the hosted input data 164 . The model execution engine 108 can generate one or more results from the machine learning application 112 .
The KPI/QoS metrics storage 160 can store one or more metrics that can be used for evaluating the machine learning application 112 . The metrics can include inference query metrics, performance metrics, sentiment metrics, and testing metrics. The metrics can be received from a user 116 through a user interface 104 .
The monitoring engine 156 can receive the results of the model execution engine 108 and compare the results with the performance characteristics (e.g., KPI/QoS metrics 160 ). The monitoring engine 156 can use ground truth data to test the machine learning application 112 to ensure the model can perform as intended. The monitoring engine 156 can provide feedback to the model composition engine 132 . The feedback can include adjustments to one or more variables or selected machine learning model used in the machine learning model 112 .
The library components 168 can include various pipelines 136 , micro service routines 140 , software modules 144 , and infrastructure modules 148 . Software pipelines 136 can consist of a sequence of computing processes (e.g., commands, program runs, tasks, threads, procedures, etc.).
Micro services routines 140 can be used in an architectural approach to building applications. As an architectural framework, micro services are distributed and loosely coupled, to allow for changes to one aspect of an application without destroying the entire application. The benefit to using micro services is that development teams can rapidly build new components of applications to meet changing development requirements. Micro service architecture breaks an application down into its core functions. Each function is called a service, and can be built and deployed independently, meaning individual services can function (and fail) without negatively affecting the others. A micro service can be a core function of an application that runs independent of other services. By storing various micro service routines 140 , the machine learning platform 100 can generate a machine learning application incrementally by identifying and selecting various different components from the library components 168 .
Software modules 144 can include batches of code that form part of a program that contains one or more routines. One or more independently developed modules make up a program. An enterprise-level software application can contain several different software modules 144 , and each module can serve unique and separate operations. A module interface can express the elements that are provided and required by the module. The elements defined in the interface can be detectable by other modules. The implementation can contain the working code that corresponds to the elements declared in the interface. Modular programming can be related to structured programming and object-oriented programming, all having the same goal of facilitating construction of large software programs and systems by decomposition into smaller pieces. While the historical usage of these terms has been inconsistent, âmodular programmingâ as used herein refers to high-level decomposition of the code of an entire program into pieces: structured programming to the low-level code use of structured control flow, and object-oriented programming to the data use of objects, a kind of data structure. In object-oriented programming, the use of interfaces as an architectural pattern to construct modules is known as interface-based programming.
Infrastructure modules 148 can include the technology stack necessary to get machine learning algorithms into production in a stable, scalable and reliable way. A technology stack can include set of software subsystems or components needed to create a complete platform such that no additional software is needed to support applications. For example, to develop a web application the architect defines the stack as the target operating system, web server, database, and programming language. Another version of a software stack is operating system, middleware, database, and applications. The components of a software stack can be developed by different developers independently from one another. The stack can extend from the data science tools used to select and train machine learning algorithms down to the hardware those algorithms run on and the databases and message queues from which they draw the datasets.
The machine learning platform 100 can include one or more data storage locations 170 . The user can identify the one or more data storage locations 170 . The data storage location 170 can be local (e.g., in a storage device electrically connected to the processing circuitry and interfaces used to generate, test, and execute the application). In various embodiments the data storage location 170 can be remote (e.g., accessible through a network such as a Local Area Network or the Internet). In some embodiments, the data storage location 170 can be a cloud-based server.
The data used for the machine learning model 112 often includes personally-identifiable information (PII), and thus, triggers certain safeguards provided by privacy laws. One way to protect the information contained in the data storage 170 can be to encrypt the data using one or more keys. Public-key cryptography, or asymmetric cryptography, is a cryptographic system that uses pairs of keys: public keys which may be disseminated widely, and private keys which are known only to the owner of the data. The private keys can be stored in the key storage 172 module to enable decrypting data for use by the machine learning platform 100 .
The model execution engine 108 can use hosted input data 164 to execute and test the machine learning application 112 . The hosted input data 164 can include a portion of the data stored at the data storage 170 . In various embodiments, a portion of the hosted input data 164 can be identified as testing data.
FIG. 2 illustrates a process 200 for generating a machine learning application using machine learning library infrastructure. Alternative embodiments may vary in function by combining, separating, or otherwise varying the functionality described in the blocks illustrated in FIG. 2 . Means for performing the functionality of one or more of the blocks illustrated in FIG. 2 may comprise hardware and/or software components of a distributed system including computing devices, storage devices, network infrastructure, and servers illustrated in FIGS. 20 , 21 , and 22 and as described below.
The functionality can include providing user access to a machine learning platform that develops production code bespoke to a hardware platform and user defined requirements. The interface for user access can include a graphical user interface on a touchscreen display, a keyboard, a keypad, a microphone to receive voice instructions, a chatbot, and/or a pointing device (e.g., a mouse, trackball for selecting one or more items in a graphical user interface).
At 202 , the functionality can include receiving a first user input identifying a location of data. In various embodiments, the user can use the interface to identify the one or more locations of data that will be used for generating the machine learning model. As described above, the data can be stored locally or remotely. In various embodiments, the user can enter a network location for the data (e.g., Internet Protocol (IP) address). In various embodiments, the user can select a folder from a plurality of folders on a storage device (e.g., a cloud-storage device). In various embodiments, the user can select a file on portable media device (e.g., a USB thumb drive, an optical disk, or solid state or magnetic disk drive). In various embodiments, the technique can suggest a location of data based at least upon a user's previous interactions with the machine learning interface.
At 204 , the functionality includes receiving a second user input identifies a problem for which a solution can be generated by the machine learning application. In various embodiments the second user input can specify a type of problem that the user would like to implement machine learning for. In various embodiments, the problem can be identified through input of text via a user interface. In various embodiments, the problems can be entered as native language speech or text (e.g., through the use of a chatbot). The technique can decipher the native language to understand the goals of the machine learning model. Some of types of problems that machine learning can solve can include classification, regression, product recommendations, medical diagnosis, financial analysis, predictive maintenance, image and sound recognition, text recognition, and tabular data analysis. The techniques can recognize one or more keywords in the native language to recommend or select a particular machine learning algorithm.
In various embodiments, a user can select the type of problem through a graphical user interface. The technique can display several generic models available to the user through a graphical user interface. The user can select one of the models by selecting, using a pointing device or touch screen display, one of the model descriptions. In addition to the generic models, additional custom models can be selected and modified to solve the problem received as the second input.
At 206 , the functionality includes receiving a third input of one or more performance requirements for the machine learning application. The third input can be entered as native language speech or text (e.g., through the use of a chatbot) or selected via an interface (e.g., a graphical user interface).
The performance requirements can include Quality of Service (QOS) metrics. QoS metrics refer to objective, system-related characteristics that provide insight into the performance of the delivery service at the network/transmission level. QoS metrics are parameters that reflect the quality of service on the sender side, rather than on the application side. Example QOS metrics can include system latency and reliability.
The performance requirements can include one or more Key Performance Indicators (KPI). Key Performance Indicators are measurable values that demonstrate how effectively the model is achieving its objectives. KPIs can be problem/solution specific and can include a measurement of the accuracy of the results of the machine learning application as compared with some ground truth test data.
The third user input can include inference indicators. Inference indicators are words commonly found at the beginning of premises and conclusions. Inference indicators alert the system of the premise or conclusion to follow.
The third user input can include sentiment indicators. A sentiment indicator refers to a graphical or numerical indicator designed to show how a group feels about a condition or state (e.g., market or economy). A sentiment indicator seeks to quantify how current beliefs and positions affect future behavior.
The third user input can include testing metrics. Testing metrics can include a quantitative measure that helps to estimate the progress, quality, and health of a software testing effort. A testing metric defines in quantitative terms the degree to which a system, system component, or process possesses a given attribute. Example testing metrics can include schedule variance, test design efficiency, number of defects per test hour, and fixed defects percentage.
The third user input can include training metrics. The training metrics help evaluate the performance of the model. Example training metrics can include classification accuracy, logarithmic loss, area under curve, F1 Score, mean absolute error, and mean squared error. The accuracy metric is a ratio of the number of correct predictions divided by the number of predictions made. The logarithmic loss metric works by penalizing false classifications. Area Under Curve (AUC) can be used for binary classification problem. AUC of a classifier is equal to the probability that the classifier will rank a randomly chosen positive example higher than a randomly chosen negative example. F1 Score is used to measure a test's accuracy. F1 Score is the Harmonic Mean between precision and recall. The range for F1 Score is [0, 1]. F1 Score can inform the user how precise a classifier is (how many instances it classifies correctly), as well as how robust it is (it does not miss a significant number of instances).
Mean Absolute Error is the average of the difference between the original values and the predicted values. Mean Absolute Error provides a measure of how far the predictions were from the actual output. However, Mean Absolute Error does not provide any idea of the direction of the error i.e., whether the model is under predicting the data or over predicting the data.
Mean Squared Error (MSE) is quite similar to Mean Absolute Error, the only difference being that MSE takes the average of the square of the difference between the original values and the predicted values. The advantage of MSE being that it is easier to compute the gradient, whereas Mean Absolute Error requires complicated linear programming tools to compute the gradient. As the system takes the square of the error, the effect of larger errors become more pronounced then smaller error, hence the model can now focus more on the larger errors.
The third user input can include Maintainability metrics. Maintainability metrics can be defined as the case with which a system can be maintained. It is an internal attribute of the system which cannot be measured directly. A solution to this problem is to build models to predict the maintenance process measures from software product measures like cohesion, coupling and complexity.
The third user input can include reusability metrics. Reusability metrics can be used to measure code reuse and reusability. Reusability metrics can be categorized into six types: reuse cost-benefits models, maturity assessment, amount of reuse, failure modes, reusability, and reuse library metrics.
At 208 , the functionality includes determining one or more library components to be selected for generating a machine learning model to prototype the machine learning application to comply with the performance requirements. The one or more library components can perform production functions. The library components can comprise at least one of pipelines 136 , workflows and micro services routines 140 , software modules 144 , and infrastructure modules 148 as described above for FIG. 1 .
The library components 168 can include metadata that identifies features and functions of each of the library components 168 . The technique can determine the one or more library components 168 to select based at least in part on the identified problem received via the second input to achieve the performance metrics of the third input. One or more variables of each of the library components can be adjusted to customize the machine learning model to achieve a solution to the identified problem.
At 210 , the functionality can include identifying one or more application programming interfaces stored in the library, wherein the application programming interfaces link the one or more components to form the machine learning application. Application Programming Interfaces (API) can be used to connect the one or more library components to generate a machine learning model.
In various embodiments, the production functions can include at least one of load balancing, fail-over caching, security, test capability, audit function, scalability, predicted performance, training models, predicted power, maintenance, debug function, and reusability. Load balancing refers to the process of distributing a set of tasks over a set of resources (computing units), with the aim of making their overall processing more efficient. Load balancing techniques can optimize the response time for each task, avoiding unevenly overloading compute nodes while other compute nodes are left idle. The primary mechanism for maintaining high system availability is called failover. Under this approach, a failed primary system is replaced by a backup system; that is, processing fails over to the backup system. Many high availability configurations also provide mechanisms for disaster recovery, which is the resumption of system availability when failover mechanisms have been unable to keep the system available. Test capability can address the case of testing the machine learning application to the desired performance characteristics. Audit function can address the ability the machine learning application can be evaluated against internal controls. Predicted power can refer the percentage of available processing power is required to execute the machine learning application.
After the machine learning model has been generated it can use the training data to training the machine learning model to the desired performance parameters. The training can adjust the one or more variables for the generated model.
At 212 , the functionality can include storing the machine learning application in a memory.
At 214 , the functionality includes testing the machine learning model according to the performance requirements. After the machine learning model has been generated and trained the technique can be tested by the model execution engine using unseen test data. Test data can be a selected percentage of the training data that is withheld from training the model. Test data can also be a separate set of data. The monitoring engine can determine if the machine learning model meets the Key Performance Indicators/Quality of Service metrics. Feedback from the monitoring engine can be sent to the model composition engine to provide recommendations to revise the machine learning model.
At 216 , the functionality includes compiling a machine learning application based upon the machine learning model depending on an outcome from the testing of the machine learning model. After the machine learning model has been training and tested, it can be compiled to output stand-alone executable code.
It should be appreciated that the specific steps illustrated in FIG. 2 provide particular techniques for generating a machine learning application according to various embodiments of the present disclosure. Other sequences of steps may also be performed according to alternative embodiments. For example, alternative embodiments of the present invention may perform the steps outlined above in a different order. Moreover, the individual steps illustrated in FIG. 2 may include multiple sub-steps that may be performed in various sequences as appropriate to the individual step. Furthermore, additional steps may be added or removed depending on the particular applications. One of ordinary skill in the art would recognize many variations, modifications, and alternatives.
FIG. 3 illustrates a process 300 for techniques to generate a machine learning application from a machine learning library infrastructure. Alternative embodiments may vary in function by combining, separating, or otherwise varying the functionality described in the blocks illustrated in FIG. 3 . Means for performing the functionality of one or more of the blocks illustrated in FIG. 3 may comprise hardware and/or software components of a distributed system including computing devices, storage devices, network infrastructure, and servers illustrated in FIGS. 20 , 21 , and 22 and described below.
At 302 , the functionality includes receiving a first user input identifying a location of data. In various embodiments, the user can use the interface to identify the location of data that will be used for generating the machine learning model. In various embodiments, the user can enter a network location for the data (e.g., Internet Protocol (IP) address). In various embodiments, the user can select a folder of a plurality of folders on a storage device (e.g., a cloud-storage device). In various embodiments, the user can select a file on portable media e.g., a USB thumb drive, an optical disk, or solid state or magnetic disk drive.
At 304 , the functionality includes receiving a second user input regarding a desired prediction for the machine learning (ML) application. In various embodiments the second user input can specify a type of problem that the user would like to implement machine learning to solve. In various embodiments, the problems can be entered as native language speech or text (e.g., through the use of a Chatbot). The technique can decipher the native language to understand the goals of the machine learning model. Some of types of problems that machine learning can solve can include classification, regression, product recommendations, medical diagnosis, financial analysis, predictive maintenance, image and sound recognition, text recognition, and tabular data analysis. The techniques will recognize one or more keywords in the native language speech to recommend or select a particular machine learning algorithm.
At 306 , the functionality includes gathering data in streams (with chunking) or in batches. Chunking is a term referring to the process of taking individual pieces of information (chunks) and grouping them into larger units. By grouping each piece into a large whole, you can improve the amount of information you can remember. The model composition engine can access the data storage to gather the data for generating the machine learning model. The data can be stored locally or in cloud-based network.
At 308 , the functionality includes validating data to check if the data characteristics (e.g., the data shape is within expectations). The center is the median and/or mean of the data. The spread is the range of the data. And, the shape describes the type of graph. The four ways to describe shape are whether it is symmetric, how many peaks it has, if it is skewed to the left or right, and whether it is uniform. The model composition engine can inform the user if the shape is not within the expectations and prompt the user to locate other/additional data.
At 310 , the functionality includes identifying issues before manifest downstream in pipeline. The model composition engine can review the machine learning query and data to ensure that the identified data have the characteristics for the query or problem. The model composition engine can alert the user if issues are identified and possibly recommend changes to settings or data to resolve the identified issues.
At 312 , the functionality includes building characteristics of problem (i.e., attributes/features). The model composition engine can interpret the user input to identify the type of problem that the user would like to use machine learning to solve. After the problem has been identified, the problem can be deconstructed into attributed and features that will be required to resolve that problem. For example, if a user desires to determine the most productive workers at a facility, the system will need to understand the features or attributes that define a productive worker. In some embodiments, the productivity can be determined by number of units produced per unit time worked. The attributes can then include data on number of hours worked, number of units produced, and number of defective units produced.
At 314 , the functionality includes determining outcomes to be predicted. The technique can leverage the second user identified information, the data, and the library components to determine the one or more output metrics that can be predicted by the machine learning application. In various embodiments, the machine leaning platform can present the output metrics to the user. In various embodiments, the user can select one or more output metrics from a plurality of available output metrics.
At 316 , the functionality includes monitoring values on ongoing basis for QoS/KPI to validate model. In various embodiments, the monitoring engine can evaluate one or more QoS or KPI metrics to determine if the model meets the performance specifications. In various embodiments, the machine learning platform can inform the user of the monitored values, and alert the user if the QoS/KPI metrics fall outside prescribed thresholds.
At 318 , the functionality includes training the machine learning model with predictions judged against QoS/KPIs. In various embodiments, the data can be used to train the machine learning model. The variables of the model can be adjusted based on the output values based on the QoS/KPI metrics.
At 320 , the functionality includes providing controls to adjust model. The controls can be executed through a Chatbot, a graphical user interface, or one or more user selectable menus. Controls allow a user to adjust the outcome of the model by adjusting the variables used for the selected algorithm. In various embodiments, the control display the outcome values as the model is adjusted.
At 322 , the functionality includes auto-adjusting the model as needed. In various embodiments, the values of the algorithm can be automatically adjusted to achieve a desired QoS/KPI outcome. In various embodiments, the values can be adjusted within a defined range of values. The adjustments can include selecting different pipelines 136 , microservices routines 140 , software modules 144 , and infrastructure modules 148 .
At 324 , the functionality includes compiling model into a machine learning application. In various embodiments, the model can be compiled into stand-alone executable code. In various embodiments, the model can be compiled to form part of another program.
At 326 , the functionality includes monitoring predictions to evaluate whether the results are within expectations. In various embodiments, the monitoring engine can provide feedback to model execution engine to inform the user if the model is providing results within an expected range.
At 328 , the functionality optionally provides interpretation and explanation of model to make the model drivers so they can be understood by the user. In various embodiments, the machine learning platform can provide results of the model to the user. The results can be detailed to explain the effect of various drivers on the results of the model. In this way, the user can make informed decisions on how the model needs to be adjusted to achieve the desired results.
At 330 , the functionality includes capturing anomalies before they manifest into inaccurate predictions. In various embodiments, the monitoring engine can receive information regarding output values that fall outside certain thresholds.
At 332 , the functionality includes fixing inaccurate data automatically or semi-autonomously. In various embodiments, the monitoring engine can determine that received data may be inaccurate. In various embodiments, the monitoring engine can notify a user that the data may be inaccurate. The monitoring engine can solicit input from the user to verify or correct the suspect data. In various embodiments, the monitoring engine can automatically correct the suspect data.
It should be appreciated that the specific steps illustrated in FIG. 3 provide particular techniques for generating a machine learning application according to various embodiments of the present disclosure. Other sequences of steps may also be performed according to alternative embodiments. For example, alternative embodiments of the present invention may perform the steps outlined above in a different order. Moreover, the individual steps illustrated in FIG. 3 may include multiple sub-steps that may be performed in various sequences as appropriate to the individual step. Furthermore, additional steps may be added or removed depending on the particular applications. One of ordinary skill in the art would recognize many variations, modifications, and alternatives.
II. A Chatbot for Defining a Machine Learning Solution
Machine learning models are trained for generating predictive outcomes for code integration requests. In one aspect, techniques can be used for defining a machine learning solution, including receiving a first input (e.g., aural, textual, or GUI) describing a problem for the machine learning solution. A model composition engine 132 , as shown in FIG. 1 , can transcribe the first input into one or more text fragments. The model composition engine 132 can determine an intent of a user to create a machine learning architecture based at least in part on the one or more text fragments. The techniques can include correlating the one or more text fragments to one or more machine learning frameworks of a plurality of models. The techniques can include presenting (e.g., interface or audio) the one or more machine learning model to the user. The model composition engine 132 can receive a selection of one or more machine learning model (e.g., classification, recommender, reinforcement learning). The model composition engine 132 can receive several other user inputs including a second input identifying a data source for the machine learning architecture and a third input of one or more constraints (e.g., resources, location, security, or privacy) for the machine learning architecture. The model composition engine 132 can generate a plurality of code for the machine learning architecture based at least in part on the selected model, the second input identifying the data source, and the third input identifying the one or more constraints. The generated code can be stored in a memory.
An intelligent assistant (e.g., a chatbot) can be used to enable a user to generate a machine learning system. Techniques can be used to automatically generate a machine learning system to assist a user. In some cases, the user may not be a software developer and may have little or no experience in either machine learning techniques or software programming. In some embodiments, a user can interact with an intelligent assistant. The interaction can be aural, textual, or through a graphical user interface. The chatbot can translate natural language inputs into a structural representation of a machine learning solution using an ontology. In this way, a user can work with artificial intelligence without being a data scientist.
<div id="p-0107" nu
CLAIMS
Claims ( 20 )
What is claimed is:
1 . A computer-implemented method comprising:
accessing, from a user device in a chat session with a master bot, one or more inputs identifying a user-requested task to be performed in the chat session and one or more other inputs identifying a data source; correlating one or more text fragments of the one or more inputs to metadata that identifies different skill bots to determine confidence scores for different intents associated with the different skill bots, of a plurality of skill bots, for carrying out the user-requested task; based at least in part on the confidence scores, routing the user-requested task to a particular skill bot of the plurality of skill bots, wherein the particular skill bot is configured for machine learning model generation; consuming, by the particular skill bot, the user-requested task and the one or more other inputs identifying the data source to train a machine learning model as part of a machine learning architecture; generating code for the machine learning architecture based at least in part on the user-requested task and the machine learning model as trained by the training data from the data source; and storing the generated code in cloud infrastructure storage accessible via an application programming interface.
2 . The computer-implemented method of claim 1 , wherein the application programming interface is accessible via the chat session.
3 . The computer-implemented method of claim 1 , wherein generating the code for the machine learning model is further based at least in part on additional input selecting a particular machine-learning-model architecture of one or more available machine-learning-model architectures.
4 . The computer-implemented method of claim 1 , wherein generating the code for the machine learning model comprises constraining the machine learning model based on one or more constraints determined from an architecture for completing the user-requested task.
5 . The computer-implemented method of claim 1 , further comprising training the machine learning model based at least in part on the training data from the data source.
6 . The computer-implemented method of claim 1 , wherein generating the code for the machine learning model comprises identifying one or more services to be used by the machine learning model determined from an architecture for completing the user-requested task.
7 . The computer-implemented method of claim 1 , further comprising:
accessing, from the user device in the chat session with the master bot, one or more other inputs identifying one or more other user-requested tasks to be performed in the chat session; determining other confidence scores for the different intents associated with the different skill bots for carrying out the one or more other user-requested tasks; based at least in part on the other confidence scores, routing the one or more other user-requested tasks to one or more other skill bots of the plurality of skill bots, wherein the one or more other skill bots are configured to perform functionality other than machine learning model generation.
8 . A system comprising:
one or more data processors; and a non-transitory computer readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform a set of actions including: accessing, from a user device in a chat session with a master bot, one or more inputs identifying a user-requested task to be performed in the chat session and one or more other inputs identifying a data source; correlating one or more text fragments of the one or more inputs to metadata that identifies different skill bots to determine confidence scores for different intents associated with the different skill bots, of a plurality of skill bots, for carrying out the user-requested task; based at least in part on the confidence scores, routing the user-requested task to a particular skill bot of the plurality of skill bots, wherein the particular skill bot is configured for machine learning model generation; consuming, by the particular skill bot, the user-requested task and the one or more other inputs identifying the data source to train a machine learning model as part of a machine learning architecture; generating code for the machine learning architecture based at least in part on the user-requested task and the machine learning model as trained by the training data from the data source; and storing the generated code in cloud infrastructure storage accessible via an application programming interface.
9 . The system of claim 8 , wherein the application programming interface is accessible via the chat session.
10 . The system of claim 8 , wherein generating the code for the machine learning model is further based at least in part on additional input selecting a particular machine-learning-model architecture of one or more available machine-learning-model architectures.
11 . The system of claim 8 , wherein generating the code for the machine learning model comprises constraining the machine learning model based on one or more constraints determined from an architecture for completing the user-requested task.
12 . The system of claim 8 , further comprising training the machine learning model based at least in part on the training data from the data source.
13 . The system of claim 8 , wherein generating the code for the machine learning model comprises identifying one or more services to be used by the machine learning model determined from an architecture for completing the user-requested task.
14 . The system of claim 8 , wherein the set of actions further includes:
accessing, from the user device in the chat session with the master bot, one or more other inputs identifying one or more other user-requested tasks to be performed in the chat session; determining other confidence scores for the different intents associated with the different skill bots for carrying out the one or more other user-requested tasks; based at least in part on the other confidence scores, routing the one or more other user-requested tasks to one or more other skill bots of the plurality of skill bots, wherein the one or more other skill bots are configured to perform functionality other than machine learning model generation.
15 . A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, including instructions configured to cause one or more data processors to perform a set of actions including:
accessing, from a user device in a chat session with a master bot, one or more inputs identifying a user-requested task to be performed in the chat session and one or more other inputs identifying a data source; correlating one or more text fragments of the one or more inputs to metadata that identifies different skill bots to determine confidence scores for different intents associated with the different skill bots, of a plurality of skill bots, for carrying out the user-requested task; based at least in part on the confidence scores, routing the user-requested task to a particular skill bot of the plurality of skill bots, wherein the particular skill bot is configured for machine learning model generation; consuming, by the particular skill bot, the user-requested task and the one or more other inputs identifying the data source to train a machine learning model as part of a machine learning architecture; generating code for the machine learning architecture based at least in part on the user-requested task and the machine learning model as trained by the training data from the data source; and storing the generated code in cloud infrastructure storage accessible via an application programming interface.
16 . The computer-program product of claim 15 , wherein the application programming interface is accessible via the chat session.
17 . The computer-program product of claim 15 , wherein generating the code for the machine learning model is further based at least in part on additional input selecting a particular machine-learning-model architecture of one or more available machine-learning-model architectures.
18 . The computer-program product of claim 15 , wherein generating the code for the machine learning model comprises constraining the machine learning model based on one or more constraints determined from an architecture for completing the user-requested task.
19 . The computer-program product of claim 15 , further comprising training the machine learning model based at least in part on the training data from the data source.
20 . The computer-program product of claim 15 , wherein generating the code for the machine learning model comprises identifying one or more services to be used by the machine learning model determined from an architecture for completing the user-requested task.
US18/954,220
2019-09-14
2024-11-20
Chatbot for defining a machine learning (ML) solution
Active
US12566982B2
( en )
Priority Applications (1)
Application Number
Priority Date
Filing Date
Title
US18/954,220
US12566982B2
( en )
2019-09-14
2024-11-20
Chatbot for defining a machine learning (ML) solution
Applications Claiming Priority (5)
Application Number
Priority Date
Filing Date
Title
US201962900537P
2019-09-14
2019-09-14
US16/893,193
US11562267B2
( en )
2019-09-14
2020-06-04
Chatbot for defining a machine learning (ML) solution
US18/100,458
US11847578B2
( en )
2019-09-14
2023-01-23
Chatbot for defining a machine learning (ML) solution
US18/501,716
US12190254B2
( en )
2019-09-14
2023-11-03
Chatbot for defining a machine learning (ML) solution
US18/954,220
US12566982B2
( en )
2019-09-14
2024-11-20
Chatbot for defining a machine learning (ML) solution
Related Parent Applications (1)
Application Number
Title
Priority Date
Filing Date
US18/501,716
Continuation
US12190254B2
( en )
2019-09-14
2023-11-03
Chatbot for defining a machine learning (ML) solution
Publications (2)
Publication Number
Publication Date
US20250077915A1
US20250077915A1 ( en )
2025-03-06
US12566982B2
true
US12566982B2 ( en )
2026-03-03
Family
ID=72560938
Family Applications (4)
Application Number
Title
Priority Date
Filing Date
US16/893,193
Active
2041-04-08
US11562267B2
( en )
2019-09-14
2020-06-04
Chatbot for defining a machine learning (ML) solution
US18/100,458
Active
US11847578B2
( en )
2019-09-14
2023-01-23
Chatbot for defining a machine learning (ML) solution
US18/501,716
Active
US12190254B2
( en )
2019-09-14
2023-11-03
Chatbot for defining a machine learning (ML) solution
US18/954,220
Active
US12566982B2
( en )
2019-09-14
2024-11-20
Chatbot for defining a machine learning (ML) solution
Family Applications Before (3)
Application Number
Title
Priority Date
Filing Date
US16/893,193
Active
2041-04-08
US11562267B2
( en )
2019-09-14
2020-06-04
Chatbot for defining a machine learning (ML) solution
US18/100,458
Active
US11847578B2
( en )
2019-09-14
2023-01-23
Chatbot for defining a machine learning (ML) solution
US18/501,716
Active
US12190254B2
( en )
2019-09-14
2023-11-03
Chatbot for defining a machine learning (ML) solution
Country Status (4)
Country
Link
US
( 4 )
US11562267B2
( en )
EP
( 1 )
EP4028903A1
( en )
CN
( 1 )
CN114556322A
( en )
WO
( 1 )
WO2021050382A1
( en )
Families Citing this family (127)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
GB2569335B
( en )
*
2017-12-13
2022-07-27
Sage Global Services Ltd
Chatbot system
US20190311807A1
( en )
*
2018-04-06
2019-10-10
Curai, Inc.
Systems and methods for responding to healthcare inquiries
US11637792B2
( en )
*
2019-04-19
2023-04-25
Oracle International Corporation
Systems and methods for a metadata driven integration of chatbot systems into back-end application services
US20200356866A1
( en )
*
2019-05-08
2020-11-12
International Business Machines Corporation
Operative enterprise application recommendation generated by cognitive services from unstructured requirements
US11580131B2
( en )
*
2019-09-01
2023-02-14
Honeywell International Inc.
Methods and apparatus for monitoring configurable performance indicators
US12118474B2
( en )
2019-09-14
2024-10-15
Oracle International Corporation
Techniques for adaptive pipelining composition for machine learning (ML)
US12386918B2
( en )
2019-09-14
2025-08-12
Oracle International Corporation
Techniques for service execution and monitoring for run-time service composition
US11562267B2
( en )
2019-09-14
2023-01-24
Oracle International Corporation
Chatbot for defining a machine learning (ML) solution
US11556862B2
( en )
2019-09-14
2023-01-17
Oracle International Corporation
Techniques for adaptive and context-aware automated service composition for machine learning (ML)
US11468238B2
( en )
2019-11-06
2022-10-11
ServiceNow Inc.
Data processing systems and methods
US11481417B2
( en )
*
2019-11-06
2022-10-25
Servicenow, Inc.
Generation and utilization of vector indexes for data processing systems and methods
US11455357B2
( en )
2019-11-06
2022-09-27
Servicenow, Inc.
Data processing systems and methods
CN114586047B
( en )
*
2019-12-13
2025-10-28
è°·ææéè´£ä»»å ¬å¸
Automatically execute computer actions in response to meeting machine learning-based conditions
KR102746947B1
( en )
*
2019-12-30
2024-12-27
ì¼ì±ì ì주ìíì¬
Electronic device transmitting and receiving data with server device
WO2021134432A1
( en )
*
2019-12-31
2021-07-08
Paypal, Inc.
Framework for managing natural language processing tools
CN113128686A
( en )
*
2020-01-16
2021-07-16
å为ææ¯æéå ¬å¸
Model training method and device
WO2021144750A1
( en )
*
2020-01-17
2021-07-22
Bitonic Technology Labs Private Limited
Method and system for identifying ideal virtual assitant bots for providing response to user queries
WO2021162681A1
( en )
*
2020-02-11
2021-08-19
Siemens Aktiengesellschaft
Method and system for imposing constraints in a skill-based autonomous system
US11997125B2
( en )
*
2020-02-24
2024-05-28
Strikeready Inc.
Automated actions in a security platform
US11462220B2
( en )
*
2020-03-04
2022-10-04
Accenture Global Solutions Limited
Infrastructure automation platform to assist in performing actions in response to tasks
JP7463560B2
( en )
*
2020-06-25
2024-04-08
ãã¿ã ã´ã¡ã³ã¿ã© ã¨ã«ã¨ã«ã·ã¼
Automated Machine Learning: An Integrated, Customizable, and Extensible System
KR102445519B1
( en )
*
2020-06-25
2022-09-21
(주)ìí¬ë¦´
Conversational intelligence service provision chatbot production system and method
US11694289B2
( en )
2020-06-30
2023-07-04
Cerner Innovation, Inc.
System and method for conversion achievement
US11196548B1
( en )
*
2020-07-02
2021-12-07
International Business Machines Corporation
Hypervisor protected key
US11875362B1
( en )
*
2020-07-14
2024-01-16
Cisco Technology, Inc.
Humanoid system for automated customer support
US20220051160A1
( en )
*
2020-08-15
2022-02-17
Yevgeniy Ivanovich Shadrin
Applied artificial intelligence system and method for constraints discovery and compliance using conversations
US11551674B2
( en )
*
2020-08-18
2023-01-10
Bank Of America Corporation
Multi-pipeline language processing platform
US12086048B2
( en )
*
2020-09-09
2024-09-10
Jpmorgan Chase Bank, N.A.
Method and apparatus for generating an architecture weighted scoring model
US11568135B1
( en )
*
2020-09-23
2023-01-31
Amazon Technologies, Inc.
Identifying chat correction pairs for training models to automatically correct chat inputs
US12626065B2
( en )
*
2020-09-30
2026-05-12
Amazon Technologies, Inc.
Lifecycle management for customized natural language processing
US11336507B2
( en )
*
2020-09-30
2022-05-17
Cisco Technology, Inc.
Anomaly detection and filtering based on system logs
US12174915B1
( en )
*
2020-10-01
2024-12-24
Wells Fargo Bank, N.A.
Progressive machine learning using a qualifier
US11948003B2
( en )
*
2020-11-04
2024-04-02
RazorThink, Inc.
System and method for automated production and deployment of packaged AI solutions
US11620473B1
( en )
*
2020-12-03
2023-04-04
Amazon Technologies, Inc.
Pre-processing raw data in user networks to create training data for service provider networks
US12260638B1
( en )
*
2020-12-31
2025-03-25
Squirrel Ai Learning Inc.
Adaptive learning systems utilizing machine learning techniques
US11861918B2
( en )
*
2021-01-13
2024-01-02
Dell Products L.P.
Image analysis for problem resolution
US11373131B1
( en )
*
2021-01-21
2022-06-28
Dell Products L.P.
Automatically identifying and correcting erroneous process actions using artificial intelligence techniques
US12498908B2
( en )
*
2021-01-26
2025-12-16
Akkio Inc.
Methods and systems for automatically generating and executing computer code using a natural language description of a data manipulation to be performed on a data set
US12118568B2
( en )
*
2021-01-27
2024-10-15
Cisco Technology, Inc.
Self-provisioning humanoid for automated customer support
US11729202B2
( en )
*
2021-03-17
2023-08-15
Butchershop Creative, LLC
Reducing project failure probability through generation, evaluation, and/or dependency structuring of a critical event object
CN112906907B
( en )
*
2021-03-24
2024-02-23
æé½å·¥ä¸å¦é¢
Method and system for layering management and distribution of machine learning pipeline model
US20220318679A1
( en )
*
2021-03-31
2022-10-06
Jio Platforms Limited
Multi-faceted bot system and method thereof
US11657112B2
( en )
*
2021-04-21
2023-05-23
Dell Products L.P.
Artificial intelligence-based cache distribution
US11451496B1
( en )
*
2021-04-30
2022-09-20
Microsoft Technology Licensing, Llc
Intelligent, personalized, and dynamic chatbot conversation
US12086553B2
( en )
*
2021-06-18
2024-09-10
International Business Machines Corporation
Anticipatory dialog design for question answering with a relational database
US20220413821A1
( en )
*
2021-06-28
2022-12-29
International Business Machines Corporation
Deploying a machine learning model
US12080324B2
( en )
*
2021-06-30
2024-09-03
Oracle International Corporation
Data tape media quality validation and action recommendation
EP4364006A4
( en )
*
2021-06-30
2025-04-02
PWC Product Sales LLC
Ai-augmented auditing platform including techniques for automated adjudication of commercial substance, related parties, and collectability
US11935570B2
( en )
2021-06-30
2024-03-19
Oracle International Corporation
Data tape quality analysis
US12271908B2
( en )
*
2021-07-08
2025-04-08
International Business Machines Corporation
Automated system for customer support
US12321428B2
( en )
*
2021-07-08
2025-06-03
Nippon Telegraph And Telephone Corporation
User authentication device, user authentication method, and user authentication computer program
US11836120B2
( en )
2021-07-23
2023-12-05
Oracle International Corporation
Machine learning techniques for schema mapping
US11645069B2
( en )
*
2021-08-26
2023-05-09
International Business Machines Corporation
Synchronizing access controls for audited code development
US11397808B1
( en )
*
2021-09-02
2022-07-26
Confluera, Inc.
Attack detection based on graph edge context
EP4399592A4
( en )
*
2021-09-07
2025-06-11
Yohana LLC
Systems and methods for integrating calendar applications with task facilitation services
US11989517B2
( en )
*
2021-09-13
2024-05-21
Sap Se
Conversational automated machine learning
US20230101955A1
( en )
*
2021-09-27
2023-03-30
At&T Intellectual Property I, L.P.
Reuse of machine learning models
US11782964B2
( en )
*
2021-10-08
2023-10-10
Adp, Inc.
Method to recommend intents based on a weighted ranked hierarchical graph
US11531943B1
( en )
2021-11-18
2022-12-20
Slate Technologies Inc.
Intelligence driven method and system for multi-factor optimization of schedules and resource recommendations for smart construction
CA3239204A1
( en )
*
2021-11-21
2023-05-25
Schlumberger Canada Limited
Machine learning model deployment, management and monitoring at scale
WO2023095078A1
( en )
*
2021-11-26
2023-06-01
Jio Platforms Limited
System and method for providing a customized solution for a predefined problem
US11711287B2
( en )
*
2021-12-08
2023-07-25
Ayla Networks, Inc.
Unified recommendation engine
US12405976B2
( en )
2021-12-16
2025-09-02
Hitachi, Ltd.
Method and system for generating training data for classifying intents in a conversational system
US20230196063A1
( en )
*
2021-12-21
2023-06-22
Cisco Technology, Inc.
Artificial intelligence development and upgrading using a neuro-symbolic metamodel
US12387160B2
( en )
*
2021-12-22
2025-08-12
Fidelity Information Services, Llc
Systems and methods for improving quality of artificial intelligence model
WO2023150673A2
( en )
*
2022-02-04
2023-08-10
Mezo Inc.
Diagnosing and troubleshooting maintenance repair requests using an artificial intelligence-driven chatbot
US11868686B2
( en )
2022-03-04
2024-01-09
Slate Technologies Inc.
System and method for manufacture and customization of construction assemblies in a computing environment
US20230289650A1
( en )
*
2022-03-09
2023-09-14
International Business Machines Corporation
Continuous machine learning system for containerized environment with limited resources
US20250209225A1
( en )
*
2022-03-29
2025-06-26
Slate Technologies Inc.
System and method for intent-based computational simulation in a construction environment
US20230316100A1
( en )
*
2022-03-29
2023-10-05
Fujitsu Limited
Machine learning pipeline augmented with explanation
US11907885B1
( en )
*
2022-03-29
2024-02-20
Slate Technologies Inc.
System and method for computational simulation and augmented/virtual reality in a construction environment
US12192235B2
( en )
*
2022-04-04
2025-01-07
Proofpoint, Inc.
Machine learning uniform resource locator (URL) classifier
US12541402B2
( en )
*
2022-04-19
2026-02-03
Salesforce, Inc.
Machine learning model layer
US20230393832A1
( en )
*
2022-06-06
2023-12-07
Incentive Marketing Group, Inc.
Automated translation of computer languages to extract and deploy computer systems and software
US20230419162A1
( en )
*
2022-06-22
2023-12-28
International Business Machines Corporation
Composing a machine learning model for complex data sources
US11811681B1
( en )
2022-07-12
2023-11-07
T-Mobile Usa, Inc.
Generating and deploying software architectures using telecommunication resources
US12356207B2
( en )
2022-07-12
2025-07-08
T-Mobile Usa, Inc.
Telecommunication resource deployment using machine learning systems and methods
US12164887B2
( en )
2022-07-12
2024-12-10
T-Mobile Usa, Inc.
Identifying standards-related requirements for software architectures using telecommunication resources
US11909737B2
( en )
*
2022-07-20
2024-02-20
Citizens Financial Group, Inc.
AI-driven integration platform and user-adaptive interface for business relationship orchestration
US12621227B2
( en )
*
2022-09-30
2026-05-05
Microsoft Technology Licensing, Llc
Service assurance in 5G networks using key performance indicator navigation tool
US20240119348A1
( en )
*
2022-10-11
2024-04-11
Sap Se
Bot transport process across tenants
US20240232606A9
( en )
*
2022-10-19
2024-07-11
Dell Products, L.P.
Computing services architect
US20240184991A1
( en )
*
2022-12-02
2024-06-06
Nvidia Corporation
Generating variational dialogue responses from structured data for conversational ai systems and applications
US20240185369A1
( en )
*
2022-12-05
2024-06-06
Capital One Services, Llc
Biasing machine learning model outputs
US12100088B2
( en )
*
2022-12-30
2024-09-24
Theai, Inc.
Recognition of intent of artificial intelligence characters
US11973839B1
( en )
2022-12-30
2024-04-30
Nutanix, Inc.
Microservice throttling based on learned demand predictions
US12536000B2
( en )
*
2023-01-05
2026-01-27
Hcl Technologies Limited
Method and system for automatically generating codes to operate ml models
US12099435B2
( en )
*
2023-01-06
2024-09-24
Bank Of America Corporation
Self-correcting bot
US12341733B2
( en )
2023-02-23
2025-06-24
State Farm Mutual Automobile Insurance Company
AI/ML chatbot for negotiations
US20240296315A1
( en )
*
2023-03-03
2024-09-05
Microsoft Technology Licensing, Llc
Artificial intelligence prompt processing and storage system
US12541785B2
( en )
2023-03-03
2026-02-03
State Farm Mutual Automobile Insurance Company
Chatbot to assist in vehicle shopping
US12321708B2
( en )
2023-03-03
2025-06-03
International Business Machines Corporation
Generating an AI conversation model using API test data
US12537779B2
( en )
*
2023-03-10
2026-01-27
Microsoft Technology Licensing, Llc
Computer-implemented multi-user messaging application
US20240330151A1
( en )
2023-04-03
2024-10-03
State Farm Mutual Automobile Insurance Company
Generative Artificial Intelligence for Source Code Security Vulnerability Inspection and Remediation
US12457519B2
( en )
*
2023-04-07
2025-10-28
Adobe Inc.
Message generation based on communication loss correlation
CN116347356B
( en )
*
2023-04-16
2026-02-10
å京佰æé¦ææ¯è¡ä»½æéå ¬å¸
Communication method, device and system
US12175341B2
( en )
*
2023-05-01
2024-12-24
The Strategic Coach Inc.
Apparatus and a method for higher-order growth modeling
US12562163B2
( en )
*
2023-05-12
2026-02-24
Servicenow, Inc.
Bidirectional assistant for development platforms
US20240394503A1
( en )
2023-05-25
2024-11-28
State Farm Mutual Automobile Insurance Company
Providing information via a machine learning chatbot emulating traits of a person
US12412030B2
( en )
2023-06-09
2025-09-09
Bank Of America Corporation
Utterance building to convey user input to conversational agents
US12456019B2
( en )
2023-06-09
2025-10-28
Bank Of America Corporation
Response generation from multi-intent input
US12332771B2
( en )
2023-06-28
2025-06-17
Bank Of America Corporation
System for automated intelligent testing of desktop applications through machine learning
US12541650B2
( en )
2023-06-28
2026-02-03
Quantiphi, Inc.
Method and system for training a virtual agent using optimal utterances
CN116821305B
( en )
*
2023-07-07
2026-01-16
ä¸å½å¹³å®äººå¯¿ä¿é©è¡ä»½æéå ¬å¸
Multi-round dialogue management method, device, equipment and storage medium
US20250061374A1
( en )
*
2023-08-15
2025-02-20
International Business Machines Corporation
Intelligent event prediction and visualization
US12328232B2
( en )
2023-08-16
2025-06-10
Bank Of America Corporation
System for automated self-discoverable generation of networked computing application flows
US20250103908A1
( en )
*
2023-09-21
2025-03-27
International Business Machines Corporation
Dynamic Selection of AI Computer Models to Reduce Costs and Maximize User Experience
US12159313B1
( en )
2023-10-18
2024-12-03
Slate Technologies Inc.
System and method for computational prediction and recommendation of real-world entities and their value assertion