ABSTRACT
Abstract
The present disclosure relates to systems and methods for an adaptive pipelining composition service that can identify and incorporate one or more new models into the machine learning application. The machine learning application with the new model can be tested off-line with the results being compared with ground truth data. If the machine learning application with the new model outperforms the previously used model, the machine learning application can be upgraded and auto-promoted to production. One or more parameters may also be discovered. The new parameters may be incorporated into the existing model in an off-line mode. The machine learning application with the new parameters can be tested off-line and the results can be compared with previous results with existing parameters. If the new parameters outperform the existing parameters as compared with ground-truth data, the machine learning application can be auto-promoted to production.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
This application claims priority of U.S. Provisional Patent Application No. 62/900,537 filed Sep. 14, 2019, entitled âAUTOMATED MACHINE LEARNING SYSTEMS AND METHODSâ, which is hereby incorporated by reference in its entirety and for all purposes.
FIELD
The present disclosure relates to systems and techniques for machine learning. More particularly, the present disclosure relates to systems and techniques for generating and managing a library of machine learning applications.
BACKGROUND
Machine learning has a wide range of applications, such as search engines, medical diagnosis, text and handwriting recognition, image processing and recognition, load forecasting, marketing and sales diagnosis, chatbots, autonomous driving, and the like. Various types and versions of machine learning models may be generated for similar applications using training data based on different technologies, languages, libraries, and the like, and thus may lack interoperability. In addition, different models may have different performances in different contexts and/or for different types of input data. Data scientists may not have the programming skills to generate the code necessary to build custom machine learning models. In addition, available machine learning tools do not store the various machine learning model components as part of a library to allow for efficient reuse of routines in other machine learning models.
Existing machine learning applications can require considerable programming knowledge by a data scientist to design and construct a machine learning application to solve specific problems. Intuitive interfaces can assist the data scientist construct a machine learning application through a series of queries.
Some organizations can store data from multiple clients or suppliers with customizable schemas. These customizable schemas may not match standardized data storage schemas used by existing machine learning models. Therefore, these other systems would need to perform a reconciliation process prior to using the stored data. The reconciliation process can be either a manual process or through a tedious extract, transform, load automated process prior to using the data for generating machine learning applications.
Machine learning applications based only on metrics (e.g., Quality of Service (QoS) or Key Performance Indicators) may not be sufficient to compose pipelines with minimal human intervention for a self-adaptive architecture. Pre-existing machine learning tools do not combine non-logical based and logic-based semantic services to generate a machine learning application.
Existing machine learning tools tend to maximize classification accuracy over new data from the same source. In doing so, these existing frameworks attempt to compose the best pipeline based on model performance. However, model performance on its own does not cover the cases where the pipeline must be adapted at run-time due to changes in online data metrics or QoS violations.
BRIEF SUMMARY
Certain aspects and features of the present disclosure relate to machine learning platform that generates a library of components to generate machine learning models and machine learning applications. The machine learning infrastructure system allows a user (i.e., a data scientist) to generate machine learning applications without having detailed knowledge of the cloud-based network infrastructure or knowledge of how to generate code for building the model. The machine learning platform can analyze the identified data and the user provided desired prediction and performance characteristics to select one or more library components and associated API to generate a machine learning application. The machine learning techniques can monitor and evaluate the outputs of the machine learning model to allow for feedback and adjustments to the model. The machine learning application can be trained, tested, and compiled for export as stand-alone executable code.
The machine learning platform can generate and store one or more library components that can be used for other machine learning applications. The machine learning platform can allow users to generate a profile which allows the platform to make recommendations based on a user's historical preferences. The model creation engine can detect the number and type of infrastructure resources necessary to achieve the desired results within the desired performance criteria.
A chatbot can provide an intuitive interface to allow the data scientist to generate a machine learning application without considerable programming experience. A chatbot is able to translate natural language into a structured representation of a machine learning solution using a conversational interface. A chatbot can be used to indicate the location of data, select a type of machine learning solution, display optimal solutions that best meet the constraints, and recommend the best environment to deploy the solution.
A self-adjusting corporation-wide discovery and integration feature can review a client's data store, review the labels for the various data schema, and effectively map the client's data schema to classifications used by the machine learning model. The various techniques can automatically select the features that are predictive for each individual use case (i.e., one client), effectively making a machine learning solution client-agnostic for the application developer. A weighted list of common representations of each feature for a particular machine learning solution can be generated and stored. When new data is added to the data store, a matching service can automatically detect which features should be fed into the machine-learning solution based at least in part on the weighted list. The weighted list can be updated as new data is made available to the model.
Existing data ontologies can be used for generating machine learning solutions for a high-precision search of relevant services to compose pipelines with minimal human intervention. Data ontologies can be used to create a combination of non-logic based and logic-based sematic services that can significantly outperform both kinds of selection in terms of precision. QoS and product KPI constraints can be used as part of architecture selection. For data sets without existing ontologies, one or more ontologies be generated.
The proposed system can use best available models at the time of construction to solve problems using the machine learning application. An adaptive pipelining composition service can identify and incorporate one or more new models into the machine learning application. The machine learning application with the new model can be tested off-line with the results being compared with ground truth data. If the machine learning application with the new model outperforms the previously used model, the machine learning application can be upgraded and auto-promoted to production. One or more parameters may also be discovered. The new parameters may be incorporated into the existing model in an off-line mode. The machine learning application with the new parameters can be tested off-line and the results can be compared with previous results with existing parameters. If the new parameters outperform the existing parameters as compared with ground-truth data, the machine learning application can be auto-promoted to production.
These and other embodiments are described in detail below. For example, other embodiments are directed to systems, devices, and computer readable media associated with methods described herein.
A better understanding of the nature and advantages of embodiments of the present disclosed may be gained with reference to the following detailed description and the accompanying drawings.
BRIEF DESCRIPTION OF THE DRAWINGS
The specification makes reference to the following appended figures, in which use of like reference numerals in different figures is intended to illustrate like or analogous components.
FIG. 1 is a block diagram illustrating an exemplary machine learning infrastructure system.
FIG. 2 is a flowchart illustrating an exemplary technique for generating a machine learning application.
FIG. 3 is a flowchart illustrating another exemplary technique for generating a machine learning application.
FIG. 4 depicts a distributed system that implements a bot system for communicating with end users using a messaging application in accordance with various embodiments
FIG. 5 depicts an integrated system including a bot system and a bot analytic system for monitoring, analyzing, visualizing, and improving the performance of a bot system in accordance with various embodiments.
FIG. 6 depicts a distributed system that implements a bot system for communicating with an end user using a messaging application according to certain embodiments.
FIG. 7 illustrates a flowchart for a technique for generating a machine learning application using a chatbot.
FIG. 8 illustrates a flowchart for protecting a serialized model and the storing private keys in a secure location.
FIGS. 9 A and 9 B illustrate exemplary user interface screens generated by a software development environment.
FIG. 10 illustrates an exemplary a distributed computing environment including a code integration request prediction server that may be used for receiving and processing input data, generating and using machine-learning or artificial intelligence models to generate outputs predictive of code integration requests.
FIG. 11 illustrates an exemplary flow chart for a real time code analysis plug-in.
FIG. 12 illustrates an exemplary flow chart for an automated on-demand library check.
FIGS. 13 A and 13 B illustrates exemplary user interface screens generated by a software development environment.
FIG. 14 illustrates a flow chart for an exemplary technique for self-adjusting corporation-wide feature discovery and integration.
FIG. 15 illustrates a flow chart for a second exemplary technique for self-adjusting corporation-wide feature discovery and integration.
FIG. 16 illustrates a block diagram illustrating an exemplary system for generating product graphs using ontologies.
FIG. 17 illustrates an exemplary flow chart for an adaptive and context-aware automated service composition for machine learning.
FIG. 18 illustrates a block diagram for generating one or more pipelines.
FIG. 19 illustrates an exemplary flow chart for an adaptive pipeline composition service.
FIG. 20 is a simplified diagram illustrating a distributed system for implementing one of the embodiments.
FIG. 21 is a simplified block diagram illustrating one or more components of a system environment.
FIG. 22 illustrates an exemplary computer system, in which various embodiments of the present invention may be implemented.
DETAILED DESCRIPTION
Certain embodiments of the present disclosure relate to systems, devices, computer-readable medium, and computer-implemented methods for implementing various techniques for machine learning. The machine learning techniques can allow a user (i.e., a data scientist) to generate machine learning applications without having detailed knowledge of the cloud-based network infrastructure or knowledge of how to generate code for building the model. The machine learning platform can analyze the identified data and the user provided desired prediction and performance characteristics to select one or more library components and associated API to generate a machine learning application.
The machine learning techniques can employ a chatbot to indicate the location of data, select a type of machine learning solution, display optimal solutions that best meet the constraints, and recommend the best environment to deploy the solution.
The techniques described herein can include a self-adjusting corporation-wide discovery and integration feature can review a client's data store, review the labels for the various data schema, and effectively map the client's data schema to classifications used by the machine learning model. The various techniques can automatically select the features that are predictive for each individual use case (i.e., one client), effectively making a machine learning solution client-agnostic for the application developer. A weighted list of common representations of each feature for a particular machine learning solution can be generated and stored.
The techniques can utilize existing data ontologies for generating machine learning solutions for a high-precision search of relevant services to compose pipelines with minimal human intervention. For data sets without existing ontologies, one or more ontologies be generated.
The techniques can employ an adaptive pipelining composition service to identify and incorporate or more new models into the machine learning application. The machine learning application with the new model can be tested off-line with the results being compared with ground truth data. If the machine learning application with the new model outperforms the previously used model, the machine learning application can be upgraded and auto-promoted to production.
I. Machine Learning Infrastructure Platform
FIG. 1 is a block diagram illustrating an exemplary machine learning platform 100 for generating a machine learning model. The machine learning platform 100 has various components that can be distributed between different networks and computing systems. A machine learning infrastructure library can store one or more components for generating machine learning applications 112 . All of the infrastructure required to productionize the <figure-callout id="112" label="machine learning applica
CROSS-REFERENCE TO RELATED APPLICATIONS
This application claims priority of U.S. Provisional Patent Application No. 62/900,537 filed Sep. 14, 2019, entitled âAUTOMATED MACHINE LEARNING SYSTEMS AND METHODSâ, which is hereby incorporated by reference in its entirety and for all purposes.
FIELD
The present disclosure relates to systems and techniques for machine learning. More particularly, the present disclosure relates to systems and techniques for generating and managing a library of machine learning applications.
BACKGROUND
Machine learning has a wide range of applications, such as search engines, medical diagnosis, text and handwriting recognition, image processing and recognition, load forecasting, marketing and sales diagnosis, chatbots, autonomous driving, and the like. Various types and versions of machine learning models may be generated for similar applications using training data based on different technologies, languages, libraries, and the like, and thus may lack interoperability. In addition, different models may have different performances in different contexts and/or for different types of input data. Data scientists may not have the programming skills to generate the code necessary to build custom machine learning models. In addition, available machine learning tools do not store the various machine learning model components as part of a library to allow for efficient reuse of routines in other machine learning models.
Existing machine learning applications can require considerable programming knowledge by a data scientist to design and construct a machine learning application to solve specific problems. Intuitive interfaces can assist the data scientist construct a machine learning application through a series of queries.
Some organizations can store data from multiple clients or suppliers with customizable schemas. These customizable schemas may not match standardized data storage schemas used by existing machine learning models. Therefore, these other systems would need to perform a reconciliation process prior to using the stored data. The reconciliation process can be either a manual process or through a tedious extract, transform, load automated process prior to using the data for generating machine learning applications.
Machine learning applications based only on metrics (e.g., Quality of Service (QoS) or Key Performance Indicators) may not be sufficient to compose pipelines with minimal human intervention for a self-adaptive architecture. Pre-existing machine learning tools do not combine non-logical based and logic-based semantic services to generate a machine learning application.
Existing machine learning tools tend to maximize classification accuracy over new data from the same source. In doing so, these existing frameworks attempt to compose the best pipeline based on model performance. However, model performance on its own does not cover the cases where the pipeline must be adapted at run-time due to changes in online data metrics or QoS violations.
BRIEF SUMMARY
Certain aspects and features of the present disclosure relate to machine learning platform that generates a library of components to generate machine learning models and machine learning applications. The machine learning infrastructure system allows a user (i.e., a data scientist) to generate machine learning applications without having detailed knowledge of the cloud-based network infrastructure or knowledge of how to generate code for building the model. The machine learning platform can analyze the identified data and the user provided desired prediction and performance characteristics to select one or more library components and associated API to generate a machine learning application. The machine learning techniques can monitor and evaluate the outputs of the machine learning model to allow for feedback and adjustments to the model. The machine learning application can be trained, tested, and compiled for export as stand-alone executable code.
The machine learning platform can generate and store one or more library components that can be used for other machine learning applications. The machine learning platform can allow users to generate a profile which allows the platform to make recommendations based on a user's historical preferences. The model creation engine can detect the number and type of infrastructure resources necessary to achieve the desired results within the desired performance criteria.
A chatbot can provide an intuitive interface to allow the data scientist to generate a machine learning application without considerable programming experience. A chatbot is able to translate natural language into a structured representation of a machine learning solution using a conversational interface. A chatbot can be used to indicate the location of data, select a type of machine learning solution, display optimal solutions that best meet the constraints, and recommend the best environment to deploy the solution.
A self-adjusting corporation-wide discovery and integration feature can review a client's data store, review the labels for the various data schema, and effectively map the client's data schema to classifications used by the machine learning model. The various techniques can automatically select the features that are predictive for each individual use case (i.e., one client), effectively making a machine learning solution client-agnostic for the application developer. A weighted list of common representations of each feature for a particular machine learning solution can be generated and stored. When new data is added to the data store, a matching service can automatically detect which features should be fed into the machine-learning solution based at least in part on the weighted list. The weighted list can be updated as new data is made available to the model.
Existing data ontologies can be used for generating machine learning solutions for a high-precision search of relevant services to compose pipelines with minimal human intervention. Data ontologies can be used to create a combination of non-logic based and logic-based sematic services that can significantly outperform both kinds of selection in terms of precision. QoS and product KPI constraints can be used as part of architecture selection. For data sets without existing ontologies, one or more ontologies be generated.
The proposed system can use best available models at the time of construction to solve problems using the machine learning application. An adaptive pipelining composition service can identify and incorporate one or more new models into the machine learning application. The machine learning application with the new model can be tested off-line with the results being compared with ground truth data. If the machine learning application with the new model outperforms the previously used model, the machine learning application can be upgraded and auto-promoted to production. One or more parameters may also be discovered. The new parameters may be incorporated into the existing model in an off-line mode. The machine learning application with the new parameters can be tested off-line and the results can be compared with previous results with existing parameters. If the new parameters outperform the existing parameters as compared with ground-truth data, the machine learning application can be auto-promoted to production.
These and other embodiments are described in detail below. For example, other embodiments are directed to systems, devices, and computer readable media associated with methods described herein.
A better understanding of the nature and advantages of embodiments of the present disclosed may be gained with reference to the following detailed description and the accompanying drawings.
BRIEF DESCRIPTION OF THE DRAWINGS
The specification makes reference to the following appended figures, in which use of like reference numerals in different figures is intended to illustrate like or analogous components.
FIG. 1 is a block diagram illustrating an exemplary machine learning infrastructure system.
FIG. 2 is a flowchart illustrating an exemplary technique for generating a machine learning application.
FIG. 3 is a flowchart illustrating another exemplary technique for generating a machine learning application.
FIG. 4 depicts a distributed system that implements a bot system for communicating with end users using a messaging application in accordance with various embodiments
FIG. 5 depicts an integrated system including a bot system and a bot analytic system for monitoring, analyzing, visualizing, and improving the performance of a bot system in accordance with various embodiments.
FIG. 6 depicts a distributed system that implements a bot system for communicating with an end user using a messaging application according to certain embodiments.
FIG. 7 illustrates a flowchart for a technique for generating a machine learning application using a chatbot.
FIG. 8 illustrates a flowchart for protecting a serialized model and the storing private keys in a secure location.
FIGS. 9 A and 9 B illustrate exemplary user interface screens generated by a software development environment.
FIG. 10 illustrates an exemplary a distributed computing environment including a code integration request prediction server that may be used for receiving and processing input data, generating and using machine-learning or artificial intelligence models to generate outputs predictive of code integration requests.
FIG. 11 illustrates an exemplary flow chart for a real time code analysis plug-in.
FIG. 12 illustrates an exemplary flow chart for an automated on-demand library check.
FIGS. 13 A and 13 B illustrates exemplary user interface screens generated by a software development environment.
FIG. 14 illustrates a flow chart for an exemplary technique for self-adjusting corporation-wide feature discovery and integration.
FIG. 15 illustrates a flow chart for a second exemplary technique for self-adjusting corporation-wide feature discovery and integration.
FIG. 16 illustrates a block diagram illustrating an exemplary system for generating product graphs using ontologies.
FIG. 17 illustrates an exemplary flow chart for an adaptive and context-aware automated service composition for machine learning.
FIG. 18 illustrates a block diagram for generating one or more pipelines.
FIG. 19 illustrates an exemplary flow chart for an adaptive pipeline composition service.
FIG. 20 is a simplified diagram illustrating a distributed system for implementing one of the embodiments.
FIG. 21 is a simplified block diagram illustrating one or more components of a system environment.
FIG. 22 illustrates an exemplary computer system, in which various embodiments of the present invention may be implemented.
DETAILED DESCRIPTION
Certain embodiments of the present disclosure relate to systems, devices, computer-readable medium, and computer-implemented methods for implementing various techniques for machine learning. The machine learning techniques can allow a user (i.e., a data scientist) to generate machine learning applications without having detailed knowledge of the cloud-based network infrastructure or knowledge of how to generate code for building the model. The machine learning platform can analyze the identified data and the user provided desired prediction and performance characteristics to select one or more library components and associated API to generate a machine learning application.
The machine learning techniques can employ a chatbot to indicate the location of data, select a type of machine learning solution, display optimal solutions that best meet the constraints, and recommend the best environment to deploy the solution.
The techniques described herein can include a self-adjusting corporation-wide discovery and integration feature can review a client's data store, review the labels for the various data schema, and effectively map the client's data schema to classifications used by the machine learning model. The various techniques can automatically select the features that are predictive for each individual use case (i.e., one client), effectively making a machine learning solution client-agnostic for the application developer. A weighted list of common representations of each feature for a particular machine learning solution can be generated and stored.
The techniques can utilize existing data ontologies for generating machine learning solutions for a high-precision search of relevant services to compose pipelines with minimal human intervention. For data sets without existing ontologies, one or more ontologies be generated.
The techniques can employ an adaptive pipelining composition service to identify and incorporate or more new models into the machine learning application. The machine learning application with the new model can be tested off-line with the results being compared with ground truth data. If the machine learning application with the new model outperforms the previously used model, the machine learning application can be upgraded and auto-promoted to production.
I. Machine Learning Infrastructure Platform
FIG. 1 is a block diagram illustrating an exemplary machine learning platform 100 for generating a machine learning model. The machine learning platform 100 has various components that can be distributed between different networks and computing systems. A machine learning infrastructure library can store one or more components for generating machine learning applications 112 . All of the infrastructure required to productionize the machine learning applications 112 can be encapsulated and stored in the library.
Machine learning configuration and interaction with the model composition engine 132 allows for selection of various library components 168 (e.g., pipelines 136 or workflows, micro services routines 140 , software modules 144 , and infrastructure modules 148 ) to define implementation of the logic of training and inference to build machine learning applications 112 . Different parameters, variables, scaling, settings, etc. for the library components 168 can be specified or determined by the model composition engine 132 . The complexity conventionally required to create the machine learning applications 112 can be performed largely automatically with the model composition engine 132 .
The library components 168 can be scalable to allows for the definition of multiple environments (e.g., different Kubernetes clusters) where the various portions of the application can be deployed to achieve any Quality of Service (QoS) or Key Performance Indicators (KPIs) specified. A Kubernetes cluster is a set of node machines for running containerized applications. The scalability can hide or abstract the complexity of the machine learning platform 100 from the application developer. A monitoring engine 156 can monitor operation of the machine learning applications 112 according to the KPI/ QoS metrics 160 to assure the machine learning application 112 is performing according to requirements. In addition the monitoring engine 156 can seamlessly test end-to-end a new or evolving machine learning application at different scales, settings, loading, settings, etc. The monitoring engine 156 can recommend various adjustments to the machine learning application 112 by signaling needed changes to the model composition engine 132 .
To address scalability in some embodiments, the machine learning platform 100 creates infrastructure, which is based on a micro services architecture, making it robust and scalable. For example, various micro services routines 140 and infrastructure modules 148 can be configured and customized for embedding into the machine learning application 112 . The machine learning platform 100 can allow a developer to define the amount of resources (e.g. CPU, memory) needed for different library components 168 of the machine learning application 112 .
The machine learning platform 100 can generate highly customizable applications. The library components 168 contain a set of predefined, off-the-shelf workflows or pipelines 136 , which the application developer can incorporate into a new machine learning application 112 . A workflow specifies various micro services routines 140 , software modules 144 and/or infrastructure modules 148 configured in a particular way for a type or class of problem. In addition to this, it is also possible to define new workflows or pipelines 136 by re-using the library components or changing an existing workflow or pipeline 136 . The infrastructure modules 148 can also include services such as data gathering, process monitoring, and logging.
A model composition engine 132 can be executed on one or more computing systems (e.g., infrastructure 128 ). The model composition engine 132 can receive inputs from a user 116 through an interface 104 . The interface 104 can include various graphical user interfaces with various menus and user selectable elements. The interface 104 can include a chatbot (e.g., a text based or voice based interface). The user 116 can interact with the interface 104 to identify one or more of: a location of data, a desired prediction of machine learning application, and various performance metrics for the machine learning model. The model composition engine 132 can interface with library components 168 to identify various pipelines 136 , micro service routines 140 , software modules 144 , and infrastructure models 148 that can be used in the creation of the machine learning model 112 .
The model composition engine 132 can output one or more machine learning applications 112 . The machine learning applications 112 can be stored locally on a server or in a cloud-based network. The model composition engine 132 can output the machine learning application 112 as executable code that be run on various infrastructure 128 through the infrastructure interfaces 124 .
The model execution engine 108 can execute the machine learning application 112 on infrastructure 128 using one or more the infrastructure interfaces 124 . The infrastructure 128 can include one or more processors, one or more memories, and one or more network interfaces, one or more buses and control lines that can be used to generate, test, compile, and deploy a machine learning application 112 . In various embodiments, the infrastructure 128 can exit on a remote system 152 that is apart from the location of the user 116 . The infrastructure 128 can interact with the model execution engine 108 through the infrastructure interfaces 124 The model execution engine 108 can input the performance characteristics (e.g., KPI/QoS metrics storage 160 ) and the hosted input data 164 . The model execution engine 108 can generate one or more results from the machine learning application 112 .
The KPI/ QoS metrics storage 160 can store one or more metrics that can be used for evaluating the machine learning application 112 . The metrics can include inference query metrics, performance metrics, sentiment metrics, and testing metrics. The metrics can be received from a user 116 through a user interface 104 .
The monitoring engine 156 can receive the results of the model execution engine 108 and compare the results with the performance characteristics (e.g., KPI/QoS metrics 160 ). The monitoring engine 156 can use ground truth data to test the machine learning application 112 to ensure the model can perform as intended. The monitoring engine 156 can provide feedback to the model composition engine 132 . The feedback can include adjustments to one or more variables or selected machine learning model used in the machine learning model 112 .
The library components 168 can include various pipelines 136 , micro service routines 140 , software modules 144 , and infrastructure modules 148 . Software pipelines 136 can consist of a sequence of computing processes (e.g., commands, program runs, tasks, threads, procedures, etc.).
Micro services routines 140 can be used in an architectural approach to building applications. As an architectural framework, micro services are distributed and loosely coupled, to allow for changes to one aspect of an application without destroying the entire application. The benefit to using micro services is that development teams can rapidly build new components of applications to meet changing development requirements. Micro service architecture breaks an application down into its core functions. Each function is called a service, and can be built and deployed independently, meaning individual services can function (and fail) without negatively affecting the others. A micro service can be a core function of an application that runs independent of other services. By storing various micro service routines 140 , the machine learning platform 100 can generate a machine learning application incrementally by identifying and selecting various different components from the library components 168 .
Software modules 144 can include batches of code that form part of a program that contains one or more routines. One or more independently developed modules make up a program. An enterprise-level software application can contain several different software modules 144 , and each module can serve unique and separate operations. A module interface can express the elements that are provided and required by the module. The elements defined in the interface can be detectable by other modules. The implementation can contain the working code that corresponds to the elements declared in the interface. Modular programming can be related to structured programming and object-oriented programming, all having the same goal of facilitating construction of large software programs and systems by decomposition into smaller pieces. While the historical usage of these terms has been inconsistent, âmodular programmingâ as used herein refers to high-level decomposition of the code of an entire program into pieces: structured programming to the low-level code use of structured control flow, and object-oriented programming to the data use of objects, a kind of data structure. In object-oriented programming, the use of interfaces as an architectural pattern to construct modules is known as interface-based programming.
Infrastructure modules 148 can include the technology stack necessary to get machine learning algorithms into production in a stable, scalable and reliable way. A technology stack can include set of software subsystems or components needed to create a complete platform such that no additional software is needed to support applications. For example, to develop a web application the architect defines the stack as the target operating system, web server, database, and programming language. Another version of a software stack is operating system, middleware, database, and applications. The components of a software stack can be developed by different developers independently from one another. The stack can extend from the data science tools used to select and train machine learning algorithms down to the hardware those algorithms run on and the databases and message queues from which they draw the datasets.
The machine learning platform 100 can include one or more data storage locations 170 . The user can identify the one or more data storage locations 170 . The data storage location 170 can be local (e.g., in a storage device electrically connected to the processing circuitry and interfaces used to generate, test, and execute the application). In various embodiments the data storage location 170 can be remote (e.g., accessible through a network such as a Local Area Network or the Internet). In some embodiments, the data storage location 170 can be a cloud-based server.
The data used for the machine learning model 112 often includes personally-identifiable information (PII), and thus, triggers certain safeguards provided by privacy laws. One way to protect the information contained in the data storage 170 can be to encrypt the data using one or more keys. Public-key cryptography, or asymmetric cryptography, is a cryptographic system that uses pairs of keys: public keys which may be disseminated widely, and private keys which are known only to the owner of the data. The private keys can be stored in the key storage 172 module to enable decrypting data for use by the machine learning platform 100 .
The model execution engine 108 can use hosted input data 164 to execute and test the machine learning application 112 . The hosted input data 164 can include a portion of the data stored at the data storage 170 . In various embodiments, a portion of the hosted input data 164 can be identified as testing data.
FIG. 2 illustrates a process 200 for generating a machine learning application using machine learning library infrastructure. Alternative embodiments may vary in function by combining, separating, or otherwise varying the functionality described in the blocks illustrated in FIG. 2 . Means for performing the functionality of one or more of the blocks illustrated in FIG. 2 may comprise hardware and/or software components of a distributed system including computing devices, storage devices, network infrastructure, and servers illustrated in FIGS. 20 , 21 , and 22 and as described below.
The functionality can include providing user access to a machine learning platform that develops production code bespoke to a hardware platform and user defined requirements. The interface for user access can include a graphical user interface on a touchscreen display, a keyboard, a keypad, a microphone to receive voice instructions, a chatbot, and/or a pointing device (e.g., a mouse, trackball for selecting one or more items in a graphical user interface).
At 202 , the functionality can include receiving a first user input identifying a location of data. In various embodiments, the user can use the interface to identify the one or more locations of data that will be used for generating the machine learning model. As described above, the data can be stored locally or remotely. In various embodiments, the user can enter a network location for the data (e.g., Internet Protocol (IP) address). In various embodiments, the user can select a folder from a plurality of folders on a storage device (e.g., a cloud-storage device). In various embodiments, the user can select a file on portable media device (e.g., a USB thumb drive, an optical disk, or solid state or magnetic disk drive). In various embodiments, the technique can suggest a location of data based at least upon a user's previous interactions with the machine learning interface.
At 204 , the functionality includes receiving a second user input identifies a problem for which a solution can be generated by the machine learning application. In various embodiments the second user input can specify a type of problem that the user would like to implement machine learning for. In various embodiments, the problem can be identified through input of text via a user interface. In various embodiments, the problems can be entered as native language speech or text (e.g., through the use of a chatbot). The technique can decipher the native language to understand the goals of the machine learning model. Some of types of problems that machine learning can solve can include classification, regression, product recommendations, medical diagnosis, financial analysis, predictive maintenance, image and sound recognition, text recognition, and tabular data analysis. The techniques can recognize one or more keywords in the native language to recommend or select a particular machine learning algorithm.
In various embodiments, a user can select the type of problem through a graphical user interface. The technique can display several generic models available to the user through a graphical user interface. The user can select one of the models by selecting, using a pointing device or touch screen display, one of the model descriptions. In addition to the generic models, additional custom models can be selected and modified to solve the problem received as the second input.
At 206 , the functionality includes receiving a third input of one or more performance requirements for the machine learning application. The third input can be entered as native language speech or text (e.g., through the use of a chatbot) or selected via an interface (e.g., a graphical user interface).
The performance requirements can include Quality of Service (QoS) metrics. QoS metrics refer to objective, system-related characteristics that provide insight into the performance of the delivery service at the network/transmission level. QoS metrics are parameters that reflect the quality of service on the sender side, rather than on the application side. Example QoS metrics can include system latency and reliability.
The performance requirements can include one or more Key Performance Indicators (KPI). Key Performance Indicators are measurable values that demonstrate how effectively the model is achieving its objectives. KPIs can be problem/solution specific and can include a measurement of the accuracy of the results of the machine learning application as compared with some ground truth test data.
The third user input can include inference indicators. Inference indicators are words commonly found at the beginning of premises and conclusions. Inference indicators alert the system of the premise or conclusion to follow.
The third user input can include sentiment indicators. A sentiment indicator refers to a graphical or numerical indicator designed to show how a group feels about a condition or state (e.g., market or economy). A sentiment indicator seeks to quantify how current beliefs and positions affect future behavior.
The third user input can include testing metrics. Testing metrics can include a quantitative measure that helps to estimate the progress, quality, and health of a software testing effort. A testing metric defines in quantitative terms the degree to which a system, system component, or process possesses a given attribute. Example testing metrics can include schedule variance, test design efficiency, number of defects per test hour, and fixed defects percentage.
The third user input can include training metrics. The training metrics help evaluate the performance of the model. Example training metrics can include classification accuracy, logarithmic loss, area under curve, F1 Score, mean absolute error, and mean squared error. The accuracy metric is a ratio of the number of correct predictions divided by the number of predictions made. The logarithmic loss metric works by penalizing false classifications. Area Under Curve (AUC) can be used for binary classification problem. AUC of a classifier is equal to the probability that the classifier will rank a randomly chosen positive example higher than a randomly chosen negative example. F1 Score is used to measure a test's accuracy. F1 Score is the Harmonic Mean between precision and recall. The range for F1 Score is [0, 1]. F1 Score can inform the user how precise a classifier is (how many instances it classifies correctly), as well as how robust it is (it does not miss a significant number of instances).
Mean Absolute Error is the average of the difference between the original values and the predicted values. Mean Absolute Error provides a measure of how far the predictions were from the actual output. However, Mean Absolute Error does not provide any idea of the direction of the error i.e., whether the model is under predicting the data or over predicting the data.
Mean Squared Error (MSE) is quite similar to Mean Absolute Error, the only difference being that MSE takes the average of the square of the difference between the original values and the predicted values. The advantage of MSE being that it is easier to compute the gradient, whereas Mean Absolute Error requires complicated linear programming tools to compute the gradient. As the system takes the square of the error, the effect of larger errors become more pronounced then smaller error, hence the model can now focus more on the larger errors.
The third user input can include Maintainability metrics. Maintainability metrics can be defined as the ease with which a system can be maintained. It is an internal attribute of the system which cannot be measured directly. A solution to this problem is to build models to predict the maintenance process measures from software product measures like cohesion, coupling and complexity.
The third user input can include reusability metrics. Reusability metrics can be used to measure code reuse and reusability. Reusability metrics can be categorized into six types: reuse cost-benefits models, maturity assessment, amount of reuse, failure modes, reusability, and reuse library metrics.
At 208 , the functionality includes determining one or more library components to be selected for generating a machine learning model to prototype the machine learning application to comply with the performance requirements. The one or more library components can perform production functions. The library components can comprise at least one of pipelines 136 , workflows and micro services routines 140 , software modules 144 , and infrastructure modules 148 as described above for FIG. 1 .
The library components 168 can include metadata that identifies features and functions of each of the library components 168 . The technique can determine the one or more library components 168 to select based at least in part on the identified problem received via the second input to achieve the performance metrics of the third input. One or more variables of each of the library components can be adjusted to customize the machine learning model to achieve a solution to the identified problem.
At 210 , the functionality can include identifying one or more application programming interfaces stored in the library, wherein the application programming interfaces link the one or more components to form the machine learning application. Application Programming Interfaces (API) can be used to connect the one or more library components to generate a machine learning model.
In various embodiments, the production functions can include at least one of load balancing, fail-over caching, security, test capability, audit function, scalability, predicted performance, training models, predicted power, maintenance, debug function, and reusability. Load balancing refers to the process of distributing a set of tasks over a set of resources (computing units), with the aim of making their overall processing more efficient. Load balancing techniques can optimize the response time for each task, avoiding unevenly overloading compute nodes while other compute nodes are left idle. The primary mechanism for maintaining high system availability is called failover. Under this approach, a failed primary system is replaced by a backup system; that is, processing fails over to the backup system. Many high availability configurations also provide mechanisms for disaster recovery, which is the resumption of system availability when failover mechanisms have been unable to keep the system available. Test capability can address the ease of testing the machine learning application to the desired performance characteristics. Audit function can address the ability the machine learning application can be evaluated against internal controls. Predicted power can refer the percentage of available processing power is required to execute the machine learning application.<
CLAIMS
Claims ( 20 )
What is claimed is:
1. A method for automatically composing a machine learning pipeline for use in a production environment executed as a background process of a machine learning application, the method comprising:
identifying a first machine learning model, wherein the first machine learning model utilizes one or more first parameters, wherein the one or more first parameters identify and define properties of the first machine learning model;
identifying the one or more first parameters that affect a speed or a quality of a learning process of the first machine learning model;
generating one or more first results for a first data set as an input to the first machine learning model and a subset of the one or more first parameters;
identifying one or more metrics for the first machine learning model, wherein the metrics define how a performance of the first machine learning model is measured;
comparing the one or more first results with a second data set to calculate a first scoring based at least in part on a difference in the one or more first results with the second data set, wherein the second data set comprises ground truth data;
analyzing the first data set to identify an ontology;
using the ontology of the first data set to identify a second machine learning model based at least in part on comparing first metadata of the second machine learning model with second metadata for the first machine learning model;
testing the second machine learning model on the first data set to produce one or more second results;
comparing the one or more second results with the second data set to generate a second scoring; and
based on the second scoring being less than the first scoring, replacing the first machine learning model with the second machine learning model for the machine learning application.
2. The method of claim 1 , further comprising:
analyzing the ontology of the first data set to identify one or more second parameters based at least in part on comparing metadata of the first parameters with the metrics for the first machine learning model;
testing the second machine learning model using the one or more second parameters to produce one or more third results;
comparing the one or more third results with the second data set to generate a third scoring; and
based on the third scoring being less than the second scoring, replacing the one or more first parameters with the one or more second parameters for the machine learning application.
3. The method of claim 1 , wherein the replacing the first machine learning model is replaced with the second machine learning model in a shadow mode until it the second machine learning model satisfies one or more conditions for auto-promoting the second machine learning model to production.
4. The method of claim 1 , further comprising:
generating a log comprising the one or more first parameters, the first machine learning model, the second machine learning model, the one or more first results, and the one or more second results; and
storing the log in a memory.
5. The method of claim 4 , further comprising analyzing the log to determine one or more patterns.
6. The method of claim 4 , further comprising saving supplemental metadata concerning the second machine learning model, wherein the supplemental metadata includes at least the one or more first parameters the one or more second results.
7. The method of claim 1 , wherein the first metadata comprises at least one of a number of levels for a decision tree and a number of parameters of an algorithm for the second machine learning model.
8. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, including instructions configured to cause a data processing apparatus to perform operations for automatically composing a machine learning pipeline for use in a production environment executed as a background process of a machine learning application, the operations comprising:
identifying a first machine learning model, wherein the first machine learning model utilizes one or more first parameters, wherein the one or more first parameters identify and define properties of the first machine learning model;
identifying the one or more first parameters that affect a speed or a quality of a learning process of the first machine learning model;
generating one or more first results for a first data set as an input to the first machine learning model and a subset of the one or more first parameters;
identifying one or more metrics for the first machine learning model, wherein the metrics define how a performance of the first machine learning model is measured;
comparing the one or more first results with a second data set to calculate a first scoring based at least in part on a difference in the one or more first results with the second data set, wherein the second data set comprises ground truth data;
analyzing the first data set to identify an ontology;
using the ontology of the first data set to identify a second machine learning model based at least in part on comparing first metadata of the second machine learning model with second metadata for the first machine learning model;
testing the second machine learning model on the first data set to produce one or more second results;
comparing the one or more second results with the second data set to generate a second scoring; and
based on the second scoring being less than the first scoring, replacing the first machine learning model with the second machine learning model for the machine learning application.
9. The computer-program product of claim 8 , including instructions configured to cause a data processing apparatus to perform further operations comprising:
analyzing the ontology of the first data set to identify one or more second parameters based at least in part on comparing metadata of the first parameters with the metrics for the first machine learning model;
testing the second machine learning model using the one or more second parameters to produce one or more third results;
comparing the one or more third results with the second data set to generate a third scoring; and
based on the third scoring being less than the second scoring, replacing the one or more first parameters with the one or more second parameters for the machine learning application.
10. The computer-program product of claim 8 , wherein the replacing the first machine learning model is replaced by the second machine learning model in a shadow mode until the second machine learning model satisfies one or more conditions for auto-promoting the second machine learning model to production.
11. The computer-program product of claim 8 , including instructions configured to cause a data processing apparatus to perform further operations comprising:
generating a log comprising the one or more first parameters, the first machine learning model, the second machine learning model, the one or more first results, and the one or more second results; and
storing the log in a memory.
12. The computer-program product of claim 11 , including instructions configured to cause a data processing apparatus to perform further operations comprising analyzing the log to determine one or more patterns.
13. The computer-program product of claim 11 , including instructions configured to cause a data processing apparatus to perform further operations comprising saving supplemental metadata concerning the second machine learning model based at least in part on the one or more first parameters the one or more second results.
14. The computer-program product of claim 8 , wherein the first metadata comprises at least one of a number of levels for a decision tree and a number of parameters of an algorithm for the second machine learning model.
15. A system for automatically composing a machine learning pipeline for use in a production environment executed as a background process of a machine learning application, comprising:
one or more data processors; and
a non-transitory computer-readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform operations comprising:
identifying a first machine learning model, wherein the first machine learning model utilizes one or more first parameters, wherein the one or more first parameters identify and define properties of the first machine learning model;
identifying the one or more first parameters that affect a speed or a quality of a learning process of the first machine learning model;
generating one or more first results for a first data set as an input to the first machine learning model and a subset of the one or more first parameters;
identifying one or more metrics for the first machine learning model, wherein the metrics define how a performance of the first machine learning model is measured;
comparing the one or more first results with a second data set to calculate a first scoring based at least in part on a difference in the one or more first results with the second data set, wherein the second data set comprises ground truth data;
analyzing the first data set to identify an ontology;
using the ontology of the first data set to identify a second machine learning model based at least in part on comparing first metadata of the second machine learning model with second metadata for the first machine learning model;
testing the second machine learning model on the first data set to produce one or more second results;
comparing the one or more second results with the second data set to generate a second scoring; and
based on the second scoring being less than the first scoring, replacing the first machine learning model with the second machine learning model for the machine learning application.
16. The system of claim 15 , wherein the non-transitory computer-readable storage medium includes further instructions which, when executed on the one or more data processors, cause the one or more data processors to perform further operations comprising:
analyzing the ontology of the first data set to identify one or more second parameters based at least in part on comparing metadata of the first parameters with the metrics for the first machine learning model;
testing the second machine learning model using the one or more second parameters to produce one or more third results;
comparing the one or more third results with the second data set to generate a third scoring; and
based on the third scoring being less than the second scoring, replacing the one or more first parameters with the one or more second parameters for the machine learning application.
17. The system of claim 15 , wherein the replacing the first machine learning model is replaced by the second machine learning model in a shadow mode until the second machine learning model satisfies one or more conditions for auto-promoting the second machine learning model to production.
18. The system of claim 15 , wherein the non-transitory computer-readable storage medium includes further instructions which, when executed on the one or more data processors, cause the one or more data processors to perform further operations comprising:
generating a log comprising the one or more first parameters, the first machine learning model, the second machine learning model, the one or more first results, and the one or more second results; and
storing the log in a memory.
19. The system of claim 18 , wherein the a non-transitory computer-readable storage medium includes further instructions which, when executed on the one or more data processors, cause the one or more data processors to perform further operations comprising saving supplemental metadata concerning the second machine learning model based at least in part on the one or more first parameters the one or more second results.
20. The system of claim 15 , wherein the first metadata comprises at least one of a number of levels for a decision tree and a number of parameters of an algorithm for the second machine learning model.
US16/892,935
2019-09-14
2020-06-04
Techniques for adaptive pipelining composition for machine learning (ML)
Active
2041-06-12
US11625648B2
( en )
Priority Applications (6)
Application Number
Priority Date
Filing Date
Title
US16/892,935
US11625648B2
( en )
2019-09-14
2020-06-04
Techniques for adaptive pipelining composition for machine learning (ML)
EP20780530.0A
EP4028874B1
( en )
2019-09-14
2020-09-13
Techniques for adaptive and context-aware automated service composition for machine learning (ml)
CN202080072159.8A
CN114616560B
( en )
2019-09-14
2020-09-13
Technology for adaptive and context-aware automated service composition for machine learning (ML)
PCT/US2020/050600
WO2021051031A1
( en )
2019-09-14
2020-09-13
Techniques for adaptive and context-aware automated service composition for machine learning (ml)
US18/132,859
US12118474B2
( en )
2019-09-14
2023-04-10
Techniques for adaptive pipelining composition for machine learning (ML)
US18/885,502
US20250013884A1
( en )
2019-09-14
2024-09-13
Techniques for adaptive pipelining composition for machine learning (ml)
Applications Claiming Priority (2)
Application Number
Priority Date
Filing Date
Title
US201962900537P
2019-09-14
2019-09-14
US16/892,935
US11625648B2
( en )
2019-09-14
2020-06-04
Techniques for adaptive pipelining composition for machine learning (ML)
Related Child Applications (1)
Application Number
Title
Priority Date
Filing Date
US18/132,859
Continuation
US12118474B2
( en )
2019-09-14
2023-04-10
Techniques for adaptive pipelining composition for machine learning (ML)
Publications (2)
Publication Number
Publication Date
US20210081848A1
US20210081848A1 ( en )
2021-03-18
US11625648B2
true
US11625648B2 ( en )
2023-04-11
Family
ID=74868570
Family Applications (7)
Application Number
Title
Priority Date
Filing Date
US16/892,724
Active
2041-03-05
US11556862B2
( en )
2019-09-14
2020-06-04
Techniques for adaptive and context-aware automated service composition for machine learning (ML)
US16/893,073
Active
2040-10-20
US11475374B2
( en )
2019-09-14
2020-06-04
Techniques for automated self-adjusting corporation-wide feature discovery and integration
US16/892,935
Active
2041-06-12
US11625648B2
( en )
2019-09-14
2020-06-04
Techniques for adaptive pipelining composition for machine learning (ML)
US17/019,254
Active
2043-02-10
US12039004B2
( en )
2019-09-14
2020-09-12
Techniques for service execution and monitoring for run-time service composition
US17/019,256
Active
2042-01-14
US11811925B2
( en )
2019-09-14
2020-09-12
Techniques for the safe serialization of the prediction pipeline
US17/019,255
Active
US11238377B2
( en )
2019-09-14
2020-09-12
Techniques for integrating segments of code into machine-learning model
US17/019,258
Active
2042-11-12
US11921815B2
( en )
2019-09-14
2020-09-13
Techniques for the automated customization and deployment of a machine learning application
Family Applications Before (2)
Application Number
Title
Priority Date
Filing Date
US16/892,724
Active
2041-03-05
US11556862B2
( en )
2019-09-14
2020-06-04
Techniques for adaptive and context-aware automated service composition for machine learning (ML)
US16/893,073
Active
2040-10-20
US11475374B2
( en )
2019-09-14
2020-06-04
Techniques for automated self-adjusting corporation-wide feature discovery and integration
Family Applications After (4)
Application Number
Title
Priority Date
Filing Date
US17/019,254
Active
2043-02-10
US12039004B2
( en )
2019-09-14
2020-09-12
Techniques for service execution and monitoring for run-time service composition
US17/019,256
Active
2042-01-14
US11811925B2
( en )
2019-09-14
2020-09-12
Techniques for the safe serialization of the prediction pipeline
US17/019,255
Active
US11238377B2
( en )
2019-09-14
2020-09-12
Techniques for integrating segments of code into machine-learning model
US17/019,258
Active
2042-11-12
US11921815B2
( en )
2019-09-14
2020-09-13
Techniques for the automated customization and deployment of a machine learning application
Country Status (1)
Country
Link
US
( 7 )
US11556862B2
( en )
Cited By (3)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US20220391176A1
( en )
*
2021-06-08
2022-12-08
Nvidia Corporation
Configuring machine learning models for training and deployment using graphical components
US20230214190A1
( en )
*
2021-12-30
2023-07-06
Kyndryl, Inc.
Intelligent user centric design platform
WO2025211775A1
( en )
*
2024-04-05
2025-10-09
Samsung Electronics Co., Ltd.
Method and apparatus for managing pipeline for split operations in communication network system
Families Citing this family (186)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
EP3475885B1
( en )
*
2016-08-22
2023-10-11
Oracle International Corporation
System and method for dynamic, incremental recommendations within real-time visual simulation
JP2020077300A
( en )
*
2018-11-09
2020-05-21
æ¥æ¬é»ä¿¡é»è©±æ ªå¼ä¼ç¤¾
Distributed deep learning system and data transfer method
US11637792B2
( en )
*
2019-04-19
2023-04-25
Oracle International Corporation
Systems and methods for a metadata driven integration of chatbot systems into back-end application services
US11580131B2
( en )
*
2019-09-01
2023-02-14
Honeywell International Inc.
Methods and apparatus for monitoring configurable performance indicators
US12118474B2
( en )
2019-09-14
2024-10-15
Oracle International Corporation
Techniques for adaptive pipelining composition for machine learning (ML)
US12386918B2
( en )
2019-09-14
2025-08-12
Oracle International Corporation
Techniques for service execution and monitoring for run-time service composition
US11663523B2
( en )
2019-09-14
2023-05-30
Oracle International Corporation
Machine learning (ML) infrastructure techniques
US11562267B2
( en )
2019-09-14
2023-01-24
Oracle International Corporation
Chatbot for defining a machine learning (ML) solution
US11556862B2
( en )
2019-09-14
2023-01-17
Oracle International Corporation
Techniques for adaptive and context-aware automated service composition for machine learning (ML)
US11783266B2
( en )
2019-09-18
2023-10-10
Tableau Software, LLC
Surfacing visualization mirages
US11593709B2
( en )
*
2019-10-15
2023-02-28
UiPath, Inc.
Inserting and/or replacing machine learning models in a pipeline for robotic process automation workflows
US12373498B2
( en )
2019-11-01
2025-07-29
Tableau Software, LLC
Providing data visualizations based on personalized recommendations
US12572150B2
( en )
2019-11-05
2026-03-10
Strong Force Vcn Portfolio 2019, Llc
Robot fleet management for value chain networks
US20210174453A1
( en )
*
2019-12-07
2021-06-10
Cerity Services, Inc.
Managing risk assessment and services through modeling
US11727284B2
( en )
2019-12-12
2023-08-15
Business Objects Software Ltd
Interpretation of machine learning results using feature analysis
CN115004171B
( en )
*
2019-12-16
2025-05-16
è°·ææéè´£ä»»å ¬å¸
System and method for generation and application of schema-independent query templates
US20210192376A1
( en )
*
2019-12-23
2021-06-24
Sap Se
Automated, progressive explanations of machine learning results
KR102746947B1
( en )
*
2019-12-30
2024-12-27
ì¼ì±ì ì주ìíì¬
Electronic device transmitting and receiving data with server device
WO2021162681A1
( en )
*
2020-02-11
2021-08-19
Siemens Aktiengesellschaft
Method and system for imposing constraints in a skill-based autonomous system
US11462220B2
( en )
*
2020-03-04
2022-10-04
Accenture Global Solutions Limited
Infrastructure automation platform to assist in performing actions in response to tasks
US11580455B2
( en )
2020-04-01
2023-02-14
Sap Se
Facilitating machine learning configuration
WO2021207269A1
( en )
*
2020-04-07
2021-10-14
Assia Spe, Llc
Systems and methods for remote collaboration
US11418381B2
( en )
*
2020-06-05
2022-08-16
Accenture Global Solutions Limited
Hybrid cloud integration deployment and management
US11222075B2
( en )
2020-06-12
2022-01-11
Earley Information Science, Inc.
Digital data processing systems and methods for digital content retrieval and generation
US11354302B2
( en )
*
2020-06-16
2022-06-07
Sap Se
Automatic creation and synchronization of graph database objects
US11861469B2
( en )
*
2020-07-02
2024-01-02
International Business Machines Corporation
Code generation for Auto-AI
US11550815B2
( en )
2020-07-30
2023-01-10
Tableau Software, LLC
Providing and surfacing metrics for visualizations
US11397746B2
( en )
2020-07-30
2022-07-26
Tableau Software, LLC
Interactive interface for data analysis and report generation
US20220051160A1
( en )
*
2020-08-15
2022-02-17
Yevgeniy Ivanovich Shadrin
Applied artificial intelligence system and method for constraints discovery and compliance using conversations
US11551674B2
( en )
*
2020-08-18
2023-01-10
Bank Of America Corporation
Multi-pipeline language processing platform
US11403340B2
( en )
*
2020-08-18
2022-08-02
Bank Of America Corporation
Hierarchical multi-tier language processing platform
US12198016B2
( en )
*
2020-08-19
2025-01-14
Bank Of America Corporation
Machine learning model training system
US12033037B2
( en )
2020-08-24
2024-07-09
International Business Machines Corporation
Open feature library management
US20220058015A1
( en )
*
2020-08-24
2022-02-24
International Business Machines Corporation
Optimization for open feature library management
US20220067458A1
( en )
*
2020-09-01
2022-03-03
International Business Machines Corporation
Smart metadata mapping for sensor data
US12039419B2
( en )
*
2020-09-02
2024-07-16
Fujitsu Limited
Automatically labeling functional blocks in pipelines of existing machine learning projects in a corpus adaptable for use in new machine learning projects
US11579760B2
( en )
2020-09-08
2023-02-14
Tableau Software, LLC
Automatic data model generation
US12242980B2
( en )
*
2020-09-09
2025-03-04
International Business Machines Corporation
Machine learning with multiple constraints
US11836268B2
( en )
*
2020-10-02
2023-12-05
Servicenow, Inc.
Machine learning platform with model storage
US11595212B2
( en )
*
2020-10-13
2023-02-28
EMC IP Holding Company LLC
Secure approval chain for runtime protection
US12326851B1
( en )
*
2020-10-23
2025-06-10
State Farm Mutual Automobile Insurance Company
Systems and methods for over-time event monitoring with enhanced rules conflict checking with data validation
US12038903B1
( en )
2020-10-23
2024-07-16
State Farm Mutual Automobile Insurance Company
Systems and methods for enhanced cloud-based rules conflict checking with data validation
US11797541B1
( en )
2020-10-23
2023-10-24
State Farm Mutual Automobile Insurance Company
Systems and methods for enhanced rules conflict checking with data validation
US11348035B2
( en )
*
2020-10-27
2022-05-31
Paypal, Inc.
Shared prediction engine for machine learning model deployment
US12045698B1
( en )
*
2020-11-03
2024-07-23
Amdocs Development Limited
System, method, and computer program for a machine learning model-agnostic cloud-agnostic no-code onboarding platform
US11948003B2
( en )
*
2020-11-04
2024-04-02
RazorThink, Inc.
System and method for automated production and deployment of packaged AI solutions
US11775641B2
( en )
*
2020-11-06
2023-10-03
Vmware, Inc.
Systems and methods for classifying malware based on feature reuse
US11620473B1
( en )
*
2020-12-03
2023-04-04
Amazon Technologies, Inc.
Pre-processing raw data in user networks to create training data for service provider networks
CN116783590A
( en )
*
2020-12-08
2023-09-19
äºé©¬éææ¯æéå ¬å¸
Get results from machine learning models using graph queries
US11307847B1
( en )
*
2020-12-10
2022-04-19
International Business Machines Corporation
Contextual application programming interfaces in a development environment
US12488279B2
( en )
*
2020-12-28
2025-12-02
International Business Machines Corporation
Domain-specific constraints for predictive modeling
US11568319B2
( en )
*
2020-12-30
2023-01-31
Hyland Uk Operations Limited
Techniques for dynamic machine learning integration
US11373131B1
( en )
*
2021-01-21
2022-06-28
Dell Products L.P.
Automatically identifying and correcting erroneous process actions using artificial intelligence techniques
US20220237229A1
( en )
*
2021-01-26
2022-07-28
Tableau Software, LLC
Content based related view recommendations
US12626155B2
( en )
*
2021-02-10
2026-05-12
Capital One Services, Llc
Machine learning model for entity resolution
WO2021119678A2
( en )
*
2021-02-23
2021-06-17
Futurewei Technologies, Inc.
Adaptive service level accounting system
US11847417B2
( en )
*
2021-03-12
2023-12-19
Accenture Global Solutions Limited
Data-driven social media analytics application synthesis
US20220309379A1
( en )
*
2021-03-18
2022-09-29
International Business Machines Corporation
Automatic Identification of Improved Machine Learning Models
US11748559B2
( en )
*
2021-03-24
2023-09-05
International Business Machines Corporation
Auto generation of conversational artifacts from specifications
US12450518B2
( en )
*
2021-03-29
2025-10-21
Google Llc
Evaluating on-device machine learning model(s) based on performance measures of client device(s) and/or the on-device machine learning model(s)
US20220318679A1
( en )
*
2021-03-31
2022-10-06
Jio Platforms Limited
Multi-faceted bot system and method thereof
CN115271087A
( en )
*
2021-04-29
2022-11-01
å为äºè®¡ç®ææ¯æéå ¬å¸
Method and device for acquiring knowledge
US12400437B2
( en )
2021-05-12
2025-08-26
Arterys Inc.
Model combining and interaction for medical imaging
US11599242B2
( en )
2021-06-04
2023-03-07
Bank Of America Corporation
Context-sensitive user interface shortcuts for a wearable device
US12013748B2
( en )
*
2021-06-09
2024-06-18
Bank Of America Corporation
Intelligent quality accelerator with root mapping overlay
US11221937B1
( en )
*
2021-06-11
2022-01-11
App Annie Inc.
Using machine learning model to make action recommendation to improve performance of client application
US11656855B2
( en )
*
2021-06-14
2023-05-23
International Business Machines Corporation
Identify, suggest, and auto-implement best practices for requirement specific software development
CN117677889A
( en )
*
2021-06-16
2024-03-08
æ´å å©å åæéå ¬å¸
Optoelectronic devices and arrays
US20220114495A1
( en )
*
2021-08-10
2022-04-14
Intel Corporation
Apparatus, articles of manufacture, and methods for composable machine learning compute nodes
US11681505B2
( en )
*
2021-06-29
2023-06-20
Opentrons LabWorks Inc.
Systems and methods for facilitating generation and deployment of machine learning software applications
US12242932B2
( en )
2021-06-29
2025-03-04
Paypal, Inc.
Execution of machine learning models at client devices
US20230004948A1
( en )
*
2021-06-30
2023-01-05
Earley Information Science, Inc.
Digital data processing systems and methods for commerce-related digital content retrieval and generation
US12430400B2
( en )
2021-07-15
2025-09-30
International Business Machines Corporation
Multi-class classification using a dual model
US12572827B2
( en )
*
2021-07-28
2026-03-10
International Business Machines Corporation
Artificial intelligence (AI) model deployment
US11868166B2
( en )
2021-08-05
2024-01-09
International Business Machines Corporation
Repairing machine learning pipelines
CN117813590B
( en )
*
2021-08-11
2024-12-13
é¹°å¾å ¬å¸
A cloud-based system for optimized multi-domain processing of input problems using multiple solver types
US11934882B2
( en )
2021-08-11
2024-03-19
Intergraph Corporation
Cloud-based systems for optimized multi-domain processing of input problems using a serverless request management engine native to a server cloud infrastructure
WO2023018599A1
( en )
*
2021-08-11
2023-02-16
Intergraph Corporation
Cloud-based systems for optimized multi-domain processing of input problems using multiple solver types
CN117859144A
( en )
*
2021-08-13
2024-04-09
西é¨åè¡ä»½å ¬å¸
Computer-based assistants to facilitate automated product and service upgrades
CN113741865B
( en )
*
2021-08-19
2023-10-27
æå·ä¼å¿ å¦ç§ææéå ¬å¸
Thinking assessment method based on programming machine data
US20230059741A1
( en )
*
2021-08-23
2023-02-23
X Development Llc
Design and implementation of machine learning state machines
US11755592B2
( en )
*
2021-08-25
2023-09-12
International Business Machines Corporation
Data search with automated selection of artificial intelligence inference models and inference label indexing
US12033001B2
( en )
*
2021-09-02
2024-07-09
Hitachi, Ltd.
Optimizing machine learning as-a-service performance for cellular communication systems
US11397808B1
( en )
*
2021-09-02
2022-07-26
Confluera, Inc.
Attack detection based on graph edge context
US20230073719A1
( en )
*
2021-09-07
2023-03-09
Early Warning Services, Llc
Target factory
US20230086609A1
( en )
*
2021-09-22
2023-03-23
Accenture Global Solutions Limited
Securely designing and executing an automation workflow based on validating the automation workflow
US12242233B2
( en )
*
2021-09-24
2025-03-04
Rockwell Automation Technologies, Inc.
Machine learning models for asset optimization within industrial automation environments
US20230099519A1
( en )
*
2021-09-24
2023-03-30
Fierce, Inc.
Systems and methods for managing stress experienced by users during events
US12314039B2
( en )
2021-09-24
2025-05-27
Rockwell Automation Technologies, Inc.
Providing a model as an industrial automation object
US20230101955A1
( en )
*
2021-09-27
2023-03-30
At&T Intellectual Property I, L.P.
Reuse of machine learning models
US20230102063A1
( en )
*
2021-09-28
2023-03-30
Advanced Micro Devices, Inc.
Providing an optimized service-based pipeline
<tr itemprop="for