ABSTRACT
Abstract
The present disclosure relates to systems and methods for an adaptive pipelining composition service that can identify and incorporate one or more new models into the machine learning application. The machine learning application with the new model can be tested off-line with the results being compared with ground truth data. If the machine learning application with the new model outperforms the previously used model, the machine learning application can be upgraded and auto-promoted to production. One or more parameters may also be discovered. The new parameters may be incorporated into the existing model in an off-line mode. The machine learning application with the new parameters can be tested off-line and the results can be compared with previous results with existing parameters. If the new parameters outperform the existing parameters as compared with ground-truth data, the machine learning application can be auto-promoted to production.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
The present application is a continuation of and claims the benefit of and priority to U.S. Non-Provisional application Ser. No. 16/892,935, filed Jun. 4, 2020, entitled âTECHNIQUES FOR ADAPTIVE PIPELINING COMPOSITION FOR MACHINE LEARNING (ML),â the entire contents of which are incorporated herein by reference for all purposes. The present application claims the benefit of and priority to U.S. Provisional Patent Application No. 62/900,537 filed Sep. 14, 2019, entitled âAUTOMATED MACHINE LEARNING SYSTEMS AND METHODS,â the entire contents of which are incorporated herein by reference for all purposes.
FIELD
The present disclosure relates to systems and techniques for machine learning. More particularly, the present disclosure relates to systems and techniques for generating and managing a library of machine learning applications.
BACKGROUND
Machine learning has a wide range of applications, such as search engines, medical diagnosis, text and handwriting recognition, image processing and recognition, load forecasting, marketing and sales diagnosis, chatbots, autonomous driving, and the like. Various types and versions of machine learning models may be generated for similar applications using training data based on different technologies, languages, libraries, and the like, and thus may lack interoperability. In addition, different models may have different performances in different contexts and/or for different types of input data. Data scientists may not have the programming skills to generate the code necessary to build custom machine learning models. In addition, available machine learning tools do not store the various machine learning model components as part of a library to allow for efficient reuse of routines in other machine learning models.
Existing machine learning applications can require considerable programming knowledge by a data scientist to design and construct a machine learning application to solve specific problems. Intuitive interfaces can assist the data scientist construct a machine learning application through a series of queries.
Some organizations can store data from multiple clients or suppliers with customizable schemas. These customizable schemas may not match standardized data storage schemas used by existing machine learning models. Therefore, these other systems would need to perform a reconciliation process prior to using the stored data. The reconciliation process can be either a manual process or through a tedious extract, transform, load automated process prior to using the data for generating machine learning applications.
Machine learning applications based only on metrics (e.g., Quality of Service (QoS) or Key Performance Indicators) may not be sufficient to compose pipelines with minimal human intervention for a self-adaptive architecture. Pre-existing machine learning tools do not combine non-logical based and logic-based semantic services to generate a machine learning application.
Existing machine learning tools tend to maximize classification accuracy over new data from the same source. In doing so, these existing frameworks attempt to compose the best pipeline based on model performance. However, model performance on its own does not cover the cases where the pipeline must be adapted at run-time due to changes in online data metrics or QoS violations.
BRIEF SUMMARY
Certain aspects and features of the present disclosure relate to machine learning platform that generates a library of components to generate machine learning models and machine learning applications. The machine learning infrastructure system allows a user (i.e., a data scientist) to generate machine learning applications without having detailed knowledge of the cloud-based network infrastructure or knowledge of how to generate code for building the model. The machine learning platform can analyze the identified data and the user provided desired prediction and performance characteristics to select one or more library components and associated API to generate a machine learning application. The machine learning techniques can monitor and evaluate the outputs of the machine learning model to allow for feedback and adjustments to the model. The machine learning application can be trained, tested, and compiled for export as stand-alone executable code.
The machine learning platform can generate and store one or more library components that can be used for other machine learning applications. The machine learning platform can allow users to generate a profile which allows the platform to make recommendations based on a user's historical preferences. The model creation engine can detect the number and type of infrastructure resources necessary to achieve the desired results within the desired performance criteria.
A chatbot can provide an intuitive interface to allow the data scientist to generate a machine learning application without considerable programming experience. A chatbot is able to translate natural language into a structured representation of a machine learning solution using a conversational interface. A chatbot can be used to indicate the location of data, select a type of machine learning solution, display optimal solutions that best meet the constraints, and recommend the best environment to deploy the solution.
A self-adjusting corporation-wide discovery and integration feature can review a client's data store, review the labels for the various data schema, and effectively map the client's data schema to classifications used by the machine learning model. The various techniques can automatically select the features that are predictive for each individual use case (i.e., one client), effectively making a machine learning solution client-agnostic for the application developer. A weighted list of common representations of each feature for a particular machine learning solution can be generated and stored. When new data is added to the data store, a matching service can automatically detect which features should be fed into the machine-learning solution based at least in part on the weighted list. The weighted list can be updated as new data is made available to the model.
Existing data ontologies can be used for generating machine learning solutions for a high-precision search of relevant services to compose pipelines with minimal human intervention. Data ontologies can be used to create a combination of non-logic based and logic-based sematic services that can significantly outperform both kinds of selection in terms of precision. QoS and product KPI constraints can be used as part of architecture selection. For data sets without existing ontologies, one or more ontologies be generated.
The proposed system can use best available models at the time of construction to solve problems using the machine learning application. An adaptive pipelining composition service can identify and incorporate one or more new models into the machine learning application. The machine learning application with the new model can be tested off-line with the results being compared with ground truth data. If the machine learning application with the new model outperforms the previously used model, the machine learning application can be upgraded and auto-promoted to production. One or more parameters may also be discovered. The new parameters may be incorporated into the existing model in an off-line mode. The machine learning application with the new parameters can be tested off-line and the results can be compared with previous results with existing parameters. If the new parameters outperform the existing parameters as compared with ground-truth data, the machine learning application can be auto-promoted to production.
These and other embodiments are described in detail below. For example, other embodiments are directed to systems, devices, and computer readable media associated with methods described herein.
A better understanding of the nature and advantages of embodiments of the present disclosed may be gained with reference to the following detailed description and the accompanying drawings.
BRIEF DESCRIPTION OF THE DRAWINGS
The specification makes reference to the following appended figures, in which use of like reference numerals in different figures is intended to illustrate like or analogous components.
FIG. 1 is a block diagram illustrating an exemplary machine learning infrastructure system.
FIG. 2 is a flowchart illustrating an exemplary technique for generating a machine learning application.
FIG. 3 is a flowchart illustrating another exemplary technique for generating a machine learning application.
FIG. 4 depicts a distributed system that implements a bot system for communicating with end users using a messaging application in accordance with various embodiments
FIG. 5 depicts an integrated system including a bot system and a bot analytic system for monitoring, analyzing, visualizing, and improving the performance of a bot system in accordance with various embodiments.
FIG. 6 depicts a distributed system that implements a bot system for communicating with an end user using a messaging application according to certain embodiments.
FIG. 7 illustrates a flowchart for a technique for generating a machine learning application using a chatbot.
FIG. 8 illustrates a flowchart for protecting a serialized model and the storing private keys in a secure location.
FIGS. 9 A and 9 B illustrate exemplary user interface screens generated by a software development environment.
FIG. 10 illustrates an exemplary a distributed computing environment including a code integration request prediction server that may be used for receiving and processing input data, generating and using machine-learning or artificial intelligence models to generate outputs predictive of code integration requests.
FIG. 11 illustrates an exemplary flow chart for a real time code analysis plug-in.
FIG. 12 illustrates an exemplary flow chart for an automated on-demand library check.
FIGS. 13 A and 13 B illustrates exemplary user interface screens generated by a software development environment.
FIG. 14 illustrates a flow chart for an exemplary technique for self-adjusting corporation-wide feature discovery and integration.
FIG. 15 illustrates a flow chart for a second exemplary technique for self-adjusting corporation-wide feature discovery and integration.
FIG. 16 illustrates a block diagram illustrating an exemplary system for generating product graphs using ontologies.
FIG. 17 illustrates an exemplary flow chart for an adaptive and context-aware automated service composition for machine learning.
FIG. 18 illustrates a block diagram for generating one or more pipelines.
FIG. 19 illustrates an exemplary flow chart for an adaptive pipeline composition service.
FIG. 20 is a simplified diagram illustrating a distributed system for implementing one of the embodiments.
FIG. 21 is a simplified block diagram illustrating one or more components of a system environment.
FIG. 22 illustrates an exemplary computer system, in which various embodiments of the present invention may be implemented.
DETAILED DESCRIPTION
Certain embodiments of the present disclosure relate to systems, devices, computer-readable medium, and computer-implemented methods for implementing various techniques for machine learning. The machine learning techniques can allow a user (i.e., a data scientist) to generate machine learning applications without having detailed knowledge of the cloud-based network infrastructure or knowledge of how to generate code for building the model. The machine learning platform can analyze the identified data and the user provided desired prediction and performance characteristics to select one or more library components and associated API to generate a machine learning application.
The machine learning techniques can employ a chatbot to indicate the location of data, select a type of machine learning solution, display optimal solutions that best meet the constraints, and recommend the best environment to deploy the solution.
The techniques described herein can include a self-adjusting corporation-wide discovery and integration feature can review a client's data store, review the labels for the various data schema, and effectively map the client's data schema to classifications used by the machine learning model. The various techniques can automatically select the features that are predictive for each individual use case (i.e., one client), effectively making a machine learning solution client-agnostic for the application developer. A weighted list of common representations of each feature for a particular machine learning solution can be generated and stored.
The techniques can utilize existing data ontologies for generating machine learning solutions for a high-precision search of relevant services to compose pipelines with minimal human intervention. For data sets without existing ontologies, one or more ontologies be generated.
The techniques can employ an adaptive pipelining composition service to identify and incorporate or more new models into the machine learning application. The machine learning application with the new model can be tested off-line with the results being compared with ground truth data. If the machine learning application with the new model outperforms the previously used model, the machine learning application can be upgraded and auto promoted to production.
I. Machine Learning Infrastructure Platform
FIG. 1 is a block diagram illustrating an exemplary machine learning platform 100 for generating a machine learning model. The machine learning platform 100 has various components that can be distributed between different networks and computing systems. A machine learning in
CROSS-REFERENCE TO RELATED APPLICATIONS
The present application is a continuation of and claims the benefit of and priority to U.S. Non-Provisional application Ser. No. 16/892,935, filed Jun. 4, 2020, entitled âTECHNIQUES FOR ADAPTIVE PIPELINING COMPOSITION FOR MACHINE LEARNING (ML),â the entire contents of which are incorporated herein by reference for all purposes. The present application claims the benefit of and priority to U.S. Provisional Patent Application No. 62/900,537 filed Sep. 14, 2019, entitled âAUTOMATED MACHINE LEARNING SYSTEMS AND METHODS,â the entire contents of which are incorporated herein by reference for all purposes.
FIELD
The present disclosure relates to systems and techniques for machine learning. More particularly, the present disclosure relates to systems and techniques for generating and managing a library of machine learning applications.
BACKGROUND
Machine learning has a wide range of applications, such as search engines, medical diagnosis, text and handwriting recognition, image processing and recognition, load forecasting, marketing and sales diagnosis, chatbots, autonomous driving, and the like. Various types and versions of machine learning models may be generated for similar applications using training data based on different technologies, languages, libraries, and the like, and thus may lack interoperability. In addition, different models may have different performances in different contexts and/or for different types of input data. Data scientists may not have the programming skills to generate the code necessary to build custom machine learning models. In addition, available machine learning tools do not store the various machine learning model components as part of a library to allow for efficient reuse of routines in other machine learning models.
Existing machine learning applications can require considerable programming knowledge by a data scientist to design and construct a machine learning application to solve specific problems. Intuitive interfaces can assist the data scientist construct a machine learning application through a series of queries.
Some organizations can store data from multiple clients or suppliers with customizable schemas. These customizable schemas may not match standardized data storage schemas used by existing machine learning models. Therefore, these other systems would need to perform a reconciliation process prior to using the stored data. The reconciliation process can be either a manual process or through a tedious extract, transform, load automated process prior to using the data for generating machine learning applications.
Machine learning applications based only on metrics (e.g., Quality of Service (QoS) or Key Performance Indicators) may not be sufficient to compose pipelines with minimal human intervention for a self-adaptive architecture. Pre-existing machine learning tools do not combine non-logical based and logic-based semantic services to generate a machine learning application.
Existing machine learning tools tend to maximize classification accuracy over new data from the same source. In doing so, these existing frameworks attempt to compose the best pipeline based on model performance. However, model performance on its own does not cover the cases where the pipeline must be adapted at run-time due to changes in online data metrics or QoS violations.
BRIEF SUMMARY
Certain aspects and features of the present disclosure relate to machine learning platform that generates a library of components to generate machine learning models and machine learning applications. The machine learning infrastructure system allows a user (i.e., a data scientist) to generate machine learning applications without having detailed knowledge of the cloud-based network infrastructure or knowledge of how to generate code for building the model. The machine learning platform can analyze the identified data and the user provided desired prediction and performance characteristics to select one or more library components and associated API to generate a machine learning application. The machine learning techniques can monitor and evaluate the outputs of the machine learning model to allow for feedback and adjustments to the model. The machine learning application can be trained, tested, and compiled for export as stand-alone executable code.
The machine learning platform can generate and store one or more library components that can be used for other machine learning applications. The machine learning platform can allow users to generate a profile which allows the platform to make recommendations based on a user's historical preferences. The model creation engine can detect the number and type of infrastructure resources necessary to achieve the desired results within the desired performance criteria.
A chatbot can provide an intuitive interface to allow the data scientist to generate a machine learning application without considerable programming experience. A chatbot is able to translate natural language into a structured representation of a machine learning solution using a conversational interface. A chatbot can be used to indicate the location of data, select a type of machine learning solution, display optimal solutions that best meet the constraints, and recommend the best environment to deploy the solution.
A self-adjusting corporation-wide discovery and integration feature can review a client's data store, review the labels for the various data schema, and effectively map the client's data schema to classifications used by the machine learning model. The various techniques can automatically select the features that are predictive for each individual use case (i.e., one client), effectively making a machine learning solution client-agnostic for the application developer. A weighted list of common representations of each feature for a particular machine learning solution can be generated and stored. When new data is added to the data store, a matching service can automatically detect which features should be fed into the machine-learning solution based at least in part on the weighted list. The weighted list can be updated as new data is made available to the model.
Existing data ontologies can be used for generating machine learning solutions for a high-precision search of relevant services to compose pipelines with minimal human intervention. Data ontologies can be used to create a combination of non-logic based and logic-based sematic services that can significantly outperform both kinds of selection in terms of precision. QoS and product KPI constraints can be used as part of architecture selection. For data sets without existing ontologies, one or more ontologies be generated.
The proposed system can use best available models at the time of construction to solve problems using the machine learning application. An adaptive pipelining composition service can identify and incorporate one or more new models into the machine learning application. The machine learning application with the new model can be tested off-line with the results being compared with ground truth data. If the machine learning application with the new model outperforms the previously used model, the machine learning application can be upgraded and auto-promoted to production. One or more parameters may also be discovered. The new parameters may be incorporated into the existing model in an off-line mode. The machine learning application with the new parameters can be tested off-line and the results can be compared with previous results with existing parameters. If the new parameters outperform the existing parameters as compared with ground-truth data, the machine learning application can be auto-promoted to production.
These and other embodiments are described in detail below. For example, other embodiments are directed to systems, devices, and computer readable media associated with methods described herein.
A better understanding of the nature and advantages of embodiments of the present disclosed may be gained with reference to the following detailed description and the accompanying drawings.
BRIEF DESCRIPTION OF THE DRAWINGS
The specification makes reference to the following appended figures, in which use of like reference numerals in different figures is intended to illustrate like or analogous components.
FIG. 1 is a block diagram illustrating an exemplary machine learning infrastructure system.
FIG. 2 is a flowchart illustrating an exemplary technique for generating a machine learning application.
FIG. 3 is a flowchart illustrating another exemplary technique for generating a machine learning application.
FIG. 4 depicts a distributed system that implements a bot system for communicating with end users using a messaging application in accordance with various embodiments
FIG. 5 depicts an integrated system including a bot system and a bot analytic system for monitoring, analyzing, visualizing, and improving the performance of a bot system in accordance with various embodiments.
FIG. 6 depicts a distributed system that implements a bot system for communicating with an end user using a messaging application according to certain embodiments.
FIG. 7 illustrates a flowchart for a technique for generating a machine learning application using a chatbot.
FIG. 8 illustrates a flowchart for protecting a serialized model and the storing private keys in a secure location.
FIGS. 9 A and 9 B illustrate exemplary user interface screens generated by a software development environment.
FIG. 10 illustrates an exemplary a distributed computing environment including a code integration request prediction server that may be used for receiving and processing input data, generating and using machine-learning or artificial intelligence models to generate outputs predictive of code integration requests.
FIG. 11 illustrates an exemplary flow chart for a real time code analysis plug-in.
FIG. 12 illustrates an exemplary flow chart for an automated on-demand library check.
FIGS. 13 A and 13 B illustrates exemplary user interface screens generated by a software development environment.
FIG. 14 illustrates a flow chart for an exemplary technique for self-adjusting corporation-wide feature discovery and integration.
FIG. 15 illustrates a flow chart for a second exemplary technique for self-adjusting corporation-wide feature discovery and integration.
FIG. 16 illustrates a block diagram illustrating an exemplary system for generating product graphs using ontologies.
FIG. 17 illustrates an exemplary flow chart for an adaptive and context-aware automated service composition for machine learning.
FIG. 18 illustrates a block diagram for generating one or more pipelines.
FIG. 19 illustrates an exemplary flow chart for an adaptive pipeline composition service.
FIG. 20 is a simplified diagram illustrating a distributed system for implementing one of the embodiments.
FIG. 21 is a simplified block diagram illustrating one or more components of a system environment.
FIG. 22 illustrates an exemplary computer system, in which various embodiments of the present invention may be implemented.
DETAILED DESCRIPTION
Certain embodiments of the present disclosure relate to systems, devices, computer-readable medium, and computer-implemented methods for implementing various techniques for machine learning. The machine learning techniques can allow a user (i.e., a data scientist) to generate machine learning applications without having detailed knowledge of the cloud-based network infrastructure or knowledge of how to generate code for building the model. The machine learning platform can analyze the identified data and the user provided desired prediction and performance characteristics to select one or more library components and associated API to generate a machine learning application.
The machine learning techniques can employ a chatbot to indicate the location of data, select a type of machine learning solution, display optimal solutions that best meet the constraints, and recommend the best environment to deploy the solution.
The techniques described herein can include a self-adjusting corporation-wide discovery and integration feature can review a client's data store, review the labels for the various data schema, and effectively map the client's data schema to classifications used by the machine learning model. The various techniques can automatically select the features that are predictive for each individual use case (i.e., one client), effectively making a machine learning solution client-agnostic for the application developer. A weighted list of common representations of each feature for a particular machine learning solution can be generated and stored.
The techniques can utilize existing data ontologies for generating machine learning solutions for a high-precision search of relevant services to compose pipelines with minimal human intervention. For data sets without existing ontologies, one or more ontologies be generated.
The techniques can employ an adaptive pipelining composition service to identify and incorporate or more new models into the machine learning application. The machine learning application with the new model can be tested off-line with the results being compared with ground truth data. If the machine learning application with the new model outperforms the previously used model, the machine learning application can be upgraded and auto promoted to production.
I. Machine Learning Infrastructure Platform
FIG. 1 is a block diagram illustrating an exemplary machine learning platform 100 for generating a machine learning model. The machine learning platform 100 has various components that can be distributed between different networks and computing systems. A machine learning infrastructure library can store one or more components for generating machine learning applications 112 . All of the infrastructure required to productionize the machine learning applications 112 can be encapsulated and stored in the library.
Machine learning configuration and interaction with the model composition engine 132 allows for selection of various library components 168 (e.g., pipelines 136 or workflows, micro services routines 140 , software modules 144 , and infrastructure modules 148 ) to define implementation of the logic of training and inference to build machine learning applications 112 . Different parameters, variables, scaling, settings, etc. for the library components 168 can be specified or determined by the model composition engine 132 . The complexity conventionally required to create the machine learning applications 112 can be performed largely automatically with the model composition engine 132 .
The library components 168 can be scalable to allows for the definition of multiple environments (e.g., different Kubernetes clusters) where the various portions of the application can be deployed to achieve any Quality of Service (QoS) or Key Performance Indicators (KPIs) specified. A Kubernetes cluster is a set of node machines for running containerized applications. The scalability can hide or abstract the complexity of the machine learning platform 100 from the application developer. A monitoring engine 156 can monitor operation of the machine learning applications 112 according to the KPI/ QoS metrics 160 to assure the machine learning application 112 is performing according to requirements. In addition the monitoring engine 156 can seamlessly test end-to-end a new or evolving machine learning application at different scales, settings, loading, settings, etc. The monitoring engine 156 can recommend various adjustments to the machine learning application 112 by signaling needed changes to the model composition engine 132 .
To address scalability in some embodiments, the machine learning platform 100 creates infrastructure, which is based on a micro services architecture, making it robust and scalable. For example, various micro services routines 140 and infrastructure modules 148 can be configured and customized for embedding into the machine learning application 112 . The machine learning platform 100 can allow a developer to define the amount of resources (e.g. CPU, memory) needed for different library components 168 of the machine learning application 112 .
The machine learning platform 100 can generate highly customizable applications. The library components 168 contain a set of predefined, off-the-shelf workflows or pipelines 136 , which the application developer can incorporate into a new machine learning application 112 . A workflow specifies various micro services routines 140 , software modules 144 and/or infrastructure modules 148 configured in a particular way for a type or class of problem. In addition to this, it is also possible to define new workflows or pipelines 136 by re-using the library components or changing an existing workflow or pipeline 136 . The infrastructure modules 148 can also include services such as data gathering, process monitoring, and logging.
A model composition engine 132 can be executed on one or more computing systems (e.g., infrastructure 128 ). The model composition engine 132 can receive inputs from a user 116 through an interface 104 . The interface 104 can include various graphical user interfaces with various menus and user selectable elements. The interface 104 can include a chatbot (e.g., a text based or voice based interface). The user 116 can interact with the interface 104 to identify one or more of: a location of data, a desired prediction of machine learning application, and various performance metrics for the machine learning model. The model composition engine 132 can interface with library components 168 to identify various pipelines 136 , micro service routines 140 , software modules 144 , and infrastructure models 148 that can be used in the creation of the machine learning model 112 .
The model composition engine 132 can output one or more machine learning applications 112 . The machine learning applications 112 can be stored locally on a server or in a cloud-based network. The model composition engine 132 can output the machine learning application 112 as executable code that be run on various infrastructure 128 through the infrastructure interfaces 124 .
The model execution engine 108 can execute the machine learning application 112 on infrastructure 128 using one or more the infrastructure interfaces 124 . The infrastructure 128 can include one or more processors, one or more memories, and one or more network interfaces, one or more buses and control lines that can be used to generate, test, compile, and deploy a machine learning application 112 . In various embodiments, the infrastructure 128 can exit on a remote system 152 that is apart from the location of the user 116 . The infrastructure 128 can interact with the model execution engine 108 through the infrastructure interfaces 124 The model execution engine 108 can input the performance characteristics (e.g., KPI/QoS metrics storage 160 ) and the hosted input data 164 . The model execution engine 108 can generate one or more results from the machine learning application 112 .
The KPI/ QoS metrics storage 160 can store one or more metrics that can be used for evaluating the machine learning application 112 . The metrics can include inference query metrics, performance metrics, sentiment metrics, and testing metrics. The metrics can be received from a user 116 through a user interface 104 .
The monitoring engine 156 can receive the results of the model execution engine 108 and compare the results with the performance characteristics (e.g., KPI/QoS metrics 160 ). The monitoring engine 156 can use ground truth data to test the machine learning application 112 to ensure the model can perform as intended. The monitoring engine 156 can provide feedback to the model composition engine 132 . The feedback can include adjustments to one or more variables or selected machine learning model used in the machine learning model 112 .
The library components 168 can include various pipelines 136 , micro service routines 140 , software modules 144 , and infrastructure modules 148 . Software pipelines 136 can consist of a sequence of computing processes (e.g., commands, program runs, tasks, threads, procedures, etc.).
Micro services routines 140 can be used in an architectural approach to building applications. As an architectural framework, micro services are distributed and loosely coupled, to allow for changes to one aspect of an application without destroying the entire application. The benefit to using micro services is that development teams can rapidly build new components of applications to meet changing development requirements. Micro service architecture breaks an application down into its core functions. Each function is called a service, and can be built and deployed independently, meaning individual services can function (and fail) without negatively affecting the others. A micro service can be a core function of an application that runs independent of other services. By storing various micro service routines 140 , the machine learning platform 100 can generate a machine learning application incrementally by identifying and selecting various different components from the library components 168 .
Software modules 144 can include batches of code that form part of a program that contains one or more routines. One or more independently developed modules makeup a program. An enterprise-level software application can contain several different software modules 144 , and each module can serve unique and separate operations. A module interface can express the elements that are provided and required by the module. The elements defined in the interface can be detectable by other modules. The implementation can contain the working code that corresponds to the elements declared in the interface. Modular programming can be related to structured programming and object-oriented programming, all having the same goal of facilitating construction of large software programs and systems by decomposition into smaller pieces. While the historical usage of these terms has been inconsistent, âmodular programmingâ as used herein refers to high-level decomposition of the code of an entire program into pieces: structured programming to the low-level code use of structured control flow, and object-oriented programming to the data use of objects, a kind of data structure. In object-oriented programming, the use of interfaces as an architectural pattern to construct modules is known as interface-based programming.
Infrastructure modules 148 can include the technology stack necessary to get machine learning algorithms into production in a stable, scalable and reliable way. A technology stack can include set of software subsystems or components needed to create a complete platform such that no additional software is needed to support applications. For example, to develop a web application the architect defines the stack as the target operating system, web server, database, and programming language. Another version of a software stack is operating system, middleware, database, and applications. The components of a software stack can be developed by different developers independently from one another. The stack can extend from the data science tools used to select and train machine learning algorithms down to the hardware those algorithms run on and the databases and message queues from which they draw the datasets.
The machine learning platform 100 can include one or more data storage locations 170 . The user can identify the one or more data storage locations 170 . The data storage location 170 can be local (e.g., in a storage device electrically connected to the processing circuitry and interfaces used to generate, test, and execute the application). In various embodiments the data storage location 170 can be remote (e.g., accessible through a network such as a Local Area Network or the Internet). In some embodiments, the data storage location 170 can be a cloud-based server.
The data used for the machine learning model 112 often includes personally-identifiable information (PII), and thus, triggers certain safeguards provided by privacy laws. One way to protect the information contained in the data storage 170 can be to encrypt the data using one or more keys. Public-key cryptography, or asymmetric cryptography, is a cryptographic system that uses pairs of keys: public keys which may be disseminated widely, and private keys which are known only to the owner of the data. The private keys can be stored in the key storage 172 module to enable decrypting data for use by the machine learning platform 100 .
The model execution engine 108 can use hosted input data 164 to execute and test the machine learning application 112 . The hosted input data 164 can include a portion of the data stored at the data storage 170 . In various embodiments, a portion of the hosted input data 164 can be identified as testing data.
FIG. 2 illustrates a process 200 for generating a machine learning application using machine learning library infrastructure. Alternative embodiments may vary in function by combining, separating, or otherwise varying the functionality described in the blocks illustrated in FIG. 2 . Means for performing the functionality of one or more of the blocks illustrated in FIG. 2 may comprise hardware and/or software components of a distributed system including computing devices, storage devices, network infrastructure, and servers illustrated in FIGS. 20 , 21 , and 22 and as described below.
The functionality can include providing user access to a machine learning platform that develops production code bespoke to a hardware platform and user defined requirements. The interface for user access can include a graphical user interface on a touchscreen display, a keyboard, a keypad, a microphone to receive voice instructions, a chatbot, and/or a pointing device (e.g., a mouse, trackball for selecting one or more items in a graphical user interface).
At 202 , the functionality can include receiving a first user input identifying a location of data. In various embodiments, the user can use the interface to identify the one or more locations of data that will be used for generating the machine learning model. As described above, the data can be stored locally or remotely. In various embodiments, the user can enter a network location for the data (e.g., Internet Protocol (IP) address). In various embodiments, the user can select a folder from a plurality of folders on a storage device (e.g., a cloud-storage device). In various embodiments, the user can select a file on portable media device (e.g., a USB thumb drive, an optical disk, or solid state or magnetic disk drive). In various embodiments, the technique can suggest a location of data based at least upon a user's previous interactions with the machine learning interface.
At 204 , the functionality includes receiving a second user input identifies a problem for which a solution can be generated by the machine learning application. In various embodiments the second user input can specify a type of problem that the user would like to implement machine learning for. In various embodiments, the problem can be identified through input of text via a user interface. In various embodiments, the problems can be entered as native language speech or text (e.g., through the use of a chatbot). The technique can decipher the native language to understand the goals of the machine learning model. Some of types of problems that machine learning can solve can include classification, regression, product recommendations, medical diagnosis, financial analysis, predictive maintenance, image and sound recognition, text recognition, and tabular data analysis. The techniques can recognize one or more keywords in the native language to recommend or select a particular machine learning algorithm.
In various embodiments, a user can select the type of problem through a graphical user interface. The technique can display several generic models available to the user through a graphical user interface. The user can select one of the models by selecting, using a pointing device or touch screen display, one of the model descriptions. In addition to the generic models, additional custom models can be selected and modified to solve the problem received as the second input.
At 206 , the functionality includes receiving a third input of one or more performance requirements for the machine learning application. The third input can be entered as native language speech or text (e.g., through the use of a chatbot) or selected via an interface (e.g., a graphical user interface).
The performance requirements can include Quality of Service (QoS) metrics. QoS metrics refer to objective, system-related characteristics that provide insight into the performance of the delivery service at the network/transmission level. QoS metrics are parameters that reflect the quality of service on the sender side, rather than on the application side. Example QoS metrics can include system latency and reliability.
The performance requirements can include one or more Key Performance Indicators (KPI). Key Performance Indicators are measurable values that demonstrate how effectively the model is achieving its objectives. KPIs can be problem/solution specific and can include a measurement of the accuracy of the results of the machine learning application as compared with some ground truth test data.
The third user input can include inference indicators. Inference indicators are words commonly found at the beginning of premises and conclusions. Inference indicators alert the system of the premise or conclusion to follow.
The third user input can include sentiment indicators. A sentiment indicator refers to a graphical or numerical indicator designed to show how a group feels about a condition or state (e.g., market or economy). A sentiment indicator seeks to quantify how current beliefs and positions affect future behavior.
The third user input can include testing metrics. Testing metrics can include a quantitative measure that helps to estimate the progress, quality, and health of a software testing effort. A testing metric defines in quantitative terms the degree to which a system, system component, or process possesses a given attribute. Example testing metrics can include schedule variance, test design efficiency, number of defects per test hour, and fixed defects percentage.
The third user input can include training metrics. The training metrics help evaluate the performance of the model. Example training metrics can include classification accuracy, logarithmic loss, area under curve, F1 Score, mean absolute error, and mean squared error. The accuracy metric is a ratio of the number of correct predictions divided by the number of predictions made. The logarithmic loss metric works by penalizing false classifications. Area Under Curve (AUC) can be used for binary classification problem. AUC of a classifier is equal to the probability that the classifier will rank a randomly chosen positive example higher than a randomly chosen negative example. F1 Score is used to measure a test's accuracy. F1 Score is the Harmonic Mean between precision and recall. The range for F1 Score is [0, 1]. F1 Score can inform the user how precise a classifier is (how many instances it classifies correctly), as well as how robust it is (it does not miss a significant number of instances).
Mean Absolute Error is the average of the difference between the original values and the predicted values. Mean Absolute Error provides a measure of how far the predictions were from the actual output. However, Mean Absolute Error does not provide any idea of the direction of the error i.e., whether the model is under predicting the data or over predicting the data.
Mean Squared Error (MSE) is quite similar to Mean Absolute Error, the only difference being that MSE takes the average of the square of the difference between the original values and the predicted values. The advantage of MSE being that it is easier to compute the gradient, whereas Mean Absolute Error requires complicated linear programming tools to compute the gradient. As the system takes the square of the error, the effect of larger errors become more pronounced then smaller error, hence the model can now focus more on the larger errors.
The third user input can include Maintainability metrics. Maintainability metrics can be defined as the ease with which a system can be maintained. It is an internal attribute of the system which cannot be measured directly. A solution to this problem is to build models to predict the maintenance process measures from software product measures like cohesion, coupling and complexity.
The third user input can include reusability metrics. Reusability metrics can be used to measure code reuse and reusability. Reusability metrics can be categorized into six types: reuse cost-benefits models, maturity assessment, amount of reuse, failure modes, reusability, and reuse library metrics.
At 208 , the functionality includes determining one or more library components to be selected for generating a machine learning model to prototype the machine learning application to comply with the performance requirements. The one or more library components can perform production functions. The library components can comprise at least one of pipelines 136 , workflows and micro services routines 140 , software modules 144 , and infrastructure modules 148 as described above for FIG. 1 .
The library components 168 can include metadata that identifies features and functions of each of the library components 168 . The technique can determine the one or more library components 168 to select based at least in part on the identified problem received via the second input to achieve the performance metrics of the third input. One or more variables of each of the library components can be adjusted to customize the machine learning model to achieve a solution to the identified problem.
At 210 , the functionality can include identifying one or more application programming interfaces stored in the library, wherein the application programming interfaces link the one or more components to form the machine learning application. Application Programming Interfaces (API) can be used to connect the one or more library components to generate a machine learning model.
In various embodiments, the production functions can include at least one of load balancing, fail-over caching, security, test capability, audit function, scalability, predicted performance, training models, predicted power, maintenance, debug function, and reusability. Load balancing refers to the process of distributing a set of tasks over a set of resources (computing units), with the aim of making their overall processing more efficient. Load balancing techniques can optimize the response time for each task, avoiding unevenly overloading compute nodes while other compute nodes are left idle. The primary mechanism for maintaining high system availability is called failover. Under this approach, a failed primary system is replaced by a backup system; that is, processing fails over to the backup system. Many high availability configurations also provide mechanisms for disaster recovery, which is the resumption of system availability when failover mechanisms have been unable to keep the system available. Test c
CLAIMS
Claims ( 20 )
What is claimed is:
1. A computer-implemented method, the method comprising:
identifying a first machine learning model, wherein the first machine learning model utilizes one or more first parameters to identify and define properties of the first machine learning model;
generating a first result for the first machine learning model using a first data set as an input to the first machine learning model;
identifying one or more metrics for the first machine learning model, wherein the metrics define how a performance of the first machine learning model is measured;
generating a second result for the first machine learning model using a second data set as the input to the first machine learning model, wherein the second data set comprises a labeled data set;
comparing the first result with the second result to calculate a first scoring;
analyzing the one or more first parameters to identify an ontology for the first machine learning model;
using the ontology to identify a second machine learning model based at least in part on comparing first metadata of the second machine learning model with second metadata for the first machine learning model;
testing the second machine learning model on the first data set to produce a third result;
generating a fourth result for the second machine learning model using the second data set as the input to the second machine learning model;
comparing the third result to the fourth result to generate a second scoring; and
based on the second scoring being less than the first scoring, replacing the first machine learning model with the second machine learning model in a machine learning application.
2. The method of claim 1 , further comprising:
storing the second machine learning model in a memory.
3. The method of claim 1 , wherein the replacing the first machine learning model with the second machine learning model is performed in a shadow mode until it the second machine learning model satisfies one or more conditions for auto-promoting the second machine learning model to production.
4. The method of claim 1 , further comprising:
generating a log comprising the one or more first parameters, the first machine learning model, the second machine learning model, the first result, and the second result; and
storing the log in a memory.
5. The method of claim 4 , further comprising analyzing the log to determine one or more patterns.
6. The method of claim 4 , further comprising saving supplemental metadata concerning the second machine learning model, wherein the supplemental metadata includes at least the one or more first parameters the second result.
7. The method of claim 1 , wherein the first metadata comprises at least one of a number of levels for a decision tree and a number of parameters of an algorithm for the second machine learning model.
8. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, including instructions configured to cause a data processing apparatus to perform operations executed as a background process of a machine learning application, the operations comprising:
identifying a first machine learning model, wherein the first machine learning model utilizes one or more first parameters to identify and define properties of the first machine learning model;
generating a first result for the first machine learning model using a first data set as an input to the first machine learning model;
identifying one or more metrics for the first machine learning model, wherein the metrics determine a performance of the first machine learning model;
generating a second result for the first machine learning model using a second data set as the input to the first machine learning model, wherein the second data set comprises ground truth data;
comparing the first result with the second result to calculate a first scoring;
analyzing one or more first parameters to identify an ontology for the first machine learning model;
using the ontology to identify a second machine learning model based at least in part on comparing first metadata of the second machine learning model with second metadata for the first machine learning model;
testing the second machine learning model on the first data set to produce a third result;
generating a fourth result for the second machine learning model using the second data set as the input to the second machine learning model;
comparing the third result to the fourth result to generate a second scoring; and
based on the second scoring being less than the first scoring, replacing the first machine learning model with the second machine learning model for the machine learning application.
9. The computer-program product of claim 8 , including instructions configured to cause a data processing apparatus to perform further operations comprising storing the second machine learning model in a memory.
10. The computer-program product of claim 8 , wherein the replacing the first machine learning model is replaced by the second machine learning model in a shadow mode until the second machine learning model satisfies one or more conditions for auto-promoting the second machine learning model to production.
11. The computer-program product of claim 8 , including instructions configured to cause a data processing apparatus to perform further operations comprising:
generating a log comprising the one or more first parameters, the first machine learning model, the second machine learning model, the first result, and the second result; and
storing the log in a memory.
12. The computer-program product of claim 11 , including instructions configured to cause a data processing apparatus to perform further operations comprising analyzing the log to determine one or more patterns.
13. The computer-program product of claim 11 , including instructions configured to cause a data processing apparatus to perform further operations comprising saving supplemental metadata concerning the second machine learning model based at least in part on the one or more first parameters the second result.
14. The computer-program product of claim 8 , wherein the first metadata comprises at least one of a number of levels for a decision tree and a number of parameters of an algorithm for the second machine learning model.
15. A system for executed as a background process of a machine learning application, comprising:
one or more data processors; and
a non-transitory computer-readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform operations comprising:
identifying a first machine learning model, wherein the first machine learning model utilizes one or more first parameters to identify and define properties of the first machine learning model;
generating a first result for the machine learning model using a first data set as an input to the first machine learning model;
identifying one or more metrics for the first machine learning model, wherein the metrics determine a performance of the first machine learning model;
generating a second result for the first machine learning model using a second data set as the input to the first machine learning model, wherein the second data set comprises ground truth data;
comparing the first result to the second result to calculate a first scoring;
analyzing the one or more first parameters to identify an ontology for the first machine learning model;
using the ontology to identify a second machine learning model based at least in part on comparing first metadata of the second machine learning model with second metadata for the first machine learning model;
testing the second machine learning model on the first data set to produce a third result;
generating a fourth result for the second machine learning model using the second data set as the input to the second machine learning model;
comparing the third result to the fourth result to generate a second scoring; and
based on the second scoring being less than the first scoring, replacing the first machine learning model with the second machine learning model for the machine learning application.
16. The system of claim 15 , wherein operations further comprise storing the second machine learning model in a memory.
17. The system of claim 15 , wherein the replacing the first machine learning model is replaced by the second machine learning model in a shadow mode until the second machine learning model satisfies one or more conditions for auto-promoting the second machine learning model to production.
18. The system of claim 15 , wherein the operations further comprise:
generating a log comprising the one or more first parameters, the first machine learning model, the second machine learning model, the first result, and the second result; and
storing the log in a memory.
19. The system of claim 18 , wherein the a non-transitory computer-readable storage medium includes further instructions which, when executed on the one or more data processors, cause the one or more data processors to perform further operations comprising saving supplemental metadata concerning the second machine learning model based at least in part on the one or more first parameters the second result.
20. The system of claim 15 , wherein the first metadata comprises at least one of a number of levels for a decision tree and a number of parameters of an algorithm for the second machine learning model.
US18/132,859
2019-09-14
2023-04-10
Techniques for adaptive pipelining composition for machine learning (ML)
Active
US12118474B2
( en )
Priority Applications (2)
Application Number
Priority Date
Filing Date
Title
US18/132,859
US12118474B2
( en )
2019-09-14
2023-04-10
Techniques for adaptive pipelining composition for machine learning (ML)
US18/885,502
US20250013884A1
( en )
2019-09-14
2024-09-13
Techniques for adaptive pipelining composition for machine learning (ml)
Applications Claiming Priority (3)
Application Number
Priority Date
Filing Date
Title
US201962900537P
2019-09-14
2019-09-14
US16/892,935
US11625648B2
( en )
2019-09-14
2020-06-04
Techniques for adaptive pipelining composition for machine learning (ML)
US18/132,859
US12118474B2
( en )
2019-09-14
2023-04-10
Techniques for adaptive pipelining composition for machine learning (ML)
Related Parent Applications (1)
Application Number
Title
Priority Date
Filing Date
US16/892,935
Continuation
US11625648B2
( en )
2019-09-14
2020-06-04
Techniques for adaptive pipelining composition for machine learning (ML)
Related Child Applications (1)
Application Number
Title
Priority Date
Filing Date
US18/885,502
Continuation
US20250013884A1
( en )
2019-09-14
2024-09-13
Techniques for adaptive pipelining composition for machine learning (ml)
Publications (2)
Publication Number
Publication Date
US20230336340A1
US20230336340A1 ( en )
2023-10-19
US12118474B2
true
US12118474B2 ( en )
2024-10-15
Family
ID=88307271
Family Applications (2)
Application Number
Title
Priority Date
Filing Date
US18/132,859
Active
US12118474B2
( en )
2019-09-14
2023-04-10
Techniques for adaptive pipelining composition for machine learning (ML)
US18/885,502
Pending
US20250013884A1
( en )
2019-09-14
2024-09-13
Techniques for adaptive pipelining composition for machine learning (ml)
Family Applications After (1)
Application Number
Title
Priority Date
Filing Date
US18/885,502
Pending
US20250013884A1
( en )
2019-09-14
2024-09-13
Techniques for adaptive pipelining composition for machine learning (ml)
Country Status (1)
Country
Link
US
( 2 )
US12118474B2
( en )
Cited By (1)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US20240370670A1
( en )
*
2023-05-02
2024-11-07
Microsoft Technology Licensing, Llc
Conditioning of productivity application file content for ingestion by an artificial intelligence model
Families Citing this family (31)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US12386918B2
( en )
2019-09-14
2025-08-12
Oracle International Corporation
Techniques for service execution and monitoring for run-time service composition
US11556862B2
( en )
2019-09-14
2023-01-17
Oracle International Corporation
Techniques for adaptive and context-aware automated service composition for machine learning (ML)
US11562267B2
( en )
2019-09-14
2023-01-24
Oracle International Corporation
Chatbot for defining a machine learning (ML) solution
US20240049003A1
( en )
*
2020-07-13
2024-02-08
Telefonaktiebolaget Lm Ericsson (Publ)
Managing a wireless device that is operable to connect to a communication network
US20230262448A1
( en )
*
2020-07-14
2023-08-17
Telefonaktiebolaget Lm Ericsson (Publ)
Managing a wireless device that is operable to connect to a communication network
US12216738B2
( en )
*
2020-10-12
2025-02-04
International Business Machines Corporation
Predicting performance of machine learning models
US11948003B2
( en )
*
2020-11-04
2024-04-02
RazorThink, Inc.
System and method for automated production and deployment of packaged AI solutions
US20220190990A1
( en )
*
2020-12-16
2022-06-16
Qualcomm Incorporated
Network-configured training procedure
US12626155B2
( en )
*
2021-02-10
2026-05-12
Capital One Services, Llc
Machine learning model for entity resolution
US11934531B2
( en )
*
2021-02-25
2024-03-19
Bank Of America Corporation
System and method for automatically identifying software vulnerabilities using named entity recognition
US20220318679A1
( en )
*
2021-03-31
2022-10-06
Jio Platforms Limited
Multi-faceted bot system and method thereof
US12626183B2
( en )
*
2021-06-14
2026-05-12
International Business Machines Corporation
Training machine learning models
US20230101955A1
( en )
*
2021-09-27
2023-03-30
At&T Intellectual Property I, L.P.
Reuse of machine learning models
US20230315862A1
( en )
*
2022-03-30
2023-10-05
WhiteSource Ltd.
Method and apparatus for identifying dynamically invoked computer code using literal values
EP4544388A4
( en )
2022-06-24
2026-01-14
Celligence Int Llc
USE OF A CONVERSATION INTERFACE FOR TRANSMITTING DIGITAL SYSTEMS
US12393719B2
( en )
*
2022-09-02
2025-08-19
Dymium Inc.
Artificially intelligent systems and methods for managing data security
US12513199B2
( en )
2023-01-21
2025-12-30
Concentric Software, Inc.
Method and electronic device for managing sensitive data based on semantic categorization
US20240333612A1
( en )
*
2023-03-31
2024-10-03
Atlassian Pty Ltd
Apparatuses, methods, and computer program products for processing multi-channel service data objects to initiate automated resolution actions via an intent engine
CN116089800B
( en )
2023-04-10
2023-06-27
æ¦æ±å·¥ç¨å¤§å¦
Method and system for extracting and correcting ringing component of dynamic pressure measurement signal of shock wave flow field
US20250036782A1
( en )
*
2023-07-28
2025-01-30
Fantagic Holdings LLC
User device with topology builder and client side personalized trusted output with data security
US20250126128A1
( en )
*
2023-10-14
2025-04-17
VMware LLC
Methods and apparatus to determine templates for use with cloud accounts
US20250138976A1
( en )
*
2023-10-30
2025-05-01
Dell Products L.P.
System and method for managing operation of a deployment based on compatability mapping
US12118117B1
( en )
*
2023-12-04
2024-10-15
Validate Me LLC
Counting machine for manufacturing and validating event-relevant identities via an ensemble network
US20250209549A1
( en )
*
2023-12-22
2025-06-26
Powerflex Systems, Llc
End-to-end model learning for distributed energy resource management systems
US20250225054A1
( en )
*
2024-01-05
2025-07-10
INVISV, Inc.
Systems and methods for securely identifying deficiencies in software code
US20250306881A1
( en )
*
2024-04-01
2025-10-02
Microsoft Technology Licensing, Llc
Ai-based software pipeline transformation
US20250328317A1
( en )
*
2024-04-18
2025-10-23
Microsoft Technology Licensing, Llc
Navigation from external code snippet symbols
GB2640743A
( en )
*
2024-05-03
2025-11-05
Dataops Software Ltd
Improvements in data product development
US20260004932A1
( en )
*
2024-06-27
2026-01-01
Siemens Healthineers Ag
Combining generalist and specialist medical ai for optimizing performance
US12602498B1
( en )
2024-09-30
2026-04-14
Dell Products Lp
System and method for securing audio and video data for artificial intelligence operations on an information handling system
CN120892923B
( en )
*
2025-07-30
2026-05-01
西å®äº¤é大å¦
Rolling bearing fault diagnosis method, device and equipment
Citations (69)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US5343527A
( en )
1993-10-27
1994-08-30
International Business Machines Corporation
Hybrid encryption method and system for protecting reusable software components
US5699507A
( en )
1995-01-17
1997-12-16
Lucent Technologies Inc.
Method of identifying similarities in code segments
US20040006761A1
( en )
2002-07-05
2004-01-08
Anand Minakshisundaran B.
System and method for automating generation of an automated sensor network
US20050102227A1
( en )
2003-11-09
2005-05-12
Aleksey Solonchev
Electronic commerce method and system utilizing integration server
US20070043734A1
( en )
2005-08-17
2007-02-22
Cannon David M
Elimination of redundant objects in storage systems
US20070239630A1
( en )
2006-02-28
2007-10-11
International Business Machines Corporation
Method and system for allowing multiple applications to utilize customized feedback with a shared machine learning engine
US20080133435A1
( en )
2005-06-12
2008-06-05
Infosys Technologies, Ltd.
System for performance and scalability analysis and methods thereof
US20090144698A1
( en )
2007-11-29
2009-06-04
Microsoft Corporation
Prioritizing quality improvements to source code
US20090276389A1
( en )
2008-05-02
2009-11-05
Paul Constantine
Systems and methods for ranking nodes of a graph using random parameters
CN101782976A
( en )
2010-01-15
2010-07-21
å京é®çµå¤§å¦
Automatic selection method for machine learning in cloud computing environment
US20110099532A1
( en )
2009-10-23
2011-04-28
International Business Machines Corporation
Automation of Software Application Engineering Using Machine Learning and Reasoning
US20130204809A1
( en )
2012-02-08
2013-08-08
Microsoft Corporation
Estimation of predictive accuracy gains from added features
US20140180738A1
( en )
2012-12-21
2014-06-26
Cloudvu, Inc.
Machine learning for systems management
US20150170053A1
( en )
2013-12-13
2015-06-18
Microsoft Corporation
Personalized machine learning models
US20160055426A1
( en )
2014-08-25
2016-02-25
Sunstone Analytics
Customizable machine learning models
US9306738B2
( en )
2012-12-21
2016-04-05
Microsoft Technology Licensing, Llc
Managed secure computations on encrypted data
US20160110657A1
( en )
2014-10-14
2016-04-21
Skytree, Inc.
Configurable Machine Learning Method Selection and Parameter Optimization System and Method
US20160179063A1
( en )
2014-12-17
2016-06-23
Microsoft Technology Licensing, Llc
Pipeline generation for data stream actuated control
US9384450B1
( en )
2015-01-22
2016-07-05
International Business Machines Corporation
Training machine learning models for open-domain question answering system
US20160275964A1
( en )
2015-03-20
2016-09-22
Electronics And Telecommunications Research Institute
Feature compensation apparatus and method for speech recogntion in noisy environment
US20160358099A1
( en )
2015-06-04
2016-12-08
The Boeing Company
Advanced analytical infrastructure for machine learning
US20170061021A1
( en )
2015-08-28
2017-03-02
Yandex Europe Ag
Method and apparatus for generating a recommended content list
US20170277693A1
( en )
2016-03-28
2017-09-28
International Business Machines Corporation
Image and data capture and machine learning for automatic generation of search queries
US20180052824A1
( en )
2016-08-19
2018-02-22
Microsoft Technology Licensing, Llc
Task identification and completion based on natural language query
US20180060744A1
( en )
2014-05-23
2018-03-01
DataRobot, Inc.
Systems for second-order predictive data analytics, and related methods and apparatus
US20180060738A1
( en )
2014-05-23
2018-03-01
DataRobot, Inc.
Systems and techniques for determining the predictive value of a feature
US20180089593A1
( en )
2016-09-26
2018-03-29
Acusense Technologies, Inc.
Method and system for an end-to-end artificial intelligence workflow
US20180136617A1
( en )
*
2016-11-11
2018-05-17
General Electric Company
Systems and methods for continuously modeling industrial asset performance
WO2018111270A1
( en )
2016-12-15
2018-06-21
Schlumberger Technology Corporation
Systems and methods for generating, deploying, discovering, and managing machine learning model packages
US20180222776A1
( en )
2013-03-15
2018-08-09
Southern Methodist University
Method and Apparatus for Electrochemical Bromide Removal
US20180314926A1
( en )
2017-04-28
2018-11-01
Intel Corporation
Smart memory handling and data management for machine learning networks
US20180314936A1
( en )
2017-04-28
2018-11-01
Intel Corporation
Tool for facilitating efficiency in machine learning
US20180314250A1
( en )
2017-04-28
2018-11-01
Intel Corporation
Autonomous machines through cloud, error corrections, and predictions
US20180322365A1
( en )
2017-05-05
2018-11-08
Intel Corporation
On the fly deep learning in machine learning for autonomous machines
US20180322387A1
( en )
2017-05-05
2018-11-08
Intel Corporation
Hardware implemented point to point communication primitives for machine learning
US20180322403A1
( en )
2017-05-05
2018-11-08
Liveperson, Inc.
Dynamic response prediction for improved bot task processing
WO2018217635A1
( en )
2017-05-20
2018-11-29
Google Llc
Application development platform and software development kits that provide comprehensive machine learning services
US20180349447A1
( en )
2017-06-02
2018-12-06
Apple Inc.
Methods and systems for customizing suggestions using user-specific information
US20180349499A1
( en )
2017-06-01
2018-12-06
Facebook Inc.
Real-time Counters for Search Results on Online Social Networks
US10198399B1
( en )
2018-03-06
2019-02-05
KenSci Inc.
Cryptographically secure machine learning
US20190108417A1
( en )
2017-10-05
2019-04-11
Parallel Machines, Inc.
Machine learning abstraction
US20190163758A1
( en )
2017-11-24
2019-05-30
Yandex Europe Ag
Method and server for presenting a recommended content item to a user
US20190228261A1
( en )
2018-01-24
2019-07-25
WeR.AI, Inc.
Systems and methods for generating machine learning applications
US20190236894A1
( en )
2012-08-08
2019-08-01
Skillz Inc.
Peer-to-Peer Wagering Platform
US20190279114A1
( en )
2018-03-08
2019-09-12
Capital One Services, Llc
System and Method for Deploying and Versioning Machine Learning Models
US10417577B2
( en )
2015-06-05
2019-09-17
Facebook, Inc.
Machine learning system interface
US20190317805A1
( en )
2019-06-25
2019-10-17
Intel Corporation
Methods and apparatus to optimize workflows
US20190334716A1
( en )
2018-04-27
2019-10-31
The University Of Akron
Blockchain-empowered crowdsourced computing system
US20190354765A1
( en )
2014-02-28
2019-11-21
Second Spectrum, Inc.
Methods, systems, and user interface navigation of video content based spatiotemporal pattern recognition
WO2019236894A1
( en )
2018-06-06
2019-12-12
Parallel Machines, Inc.
Detecting suitability of machine learning models for datasets
US20200012900A1
( en )
*
2018-07-06
2020-01-09
Capital One Services, Llc
Systems and methods for detecting data drift for data used in machine learning models
US20200081899A1
( en )
2018-09-12
2020-03-12
Pricewaterhousecoopers Llp
Automated database schema matching
US20200151619A1
( en )
*
2018-11-09
2020-05-14
Hewlett Packard Enterprise Development Lp
Systems and methods for determining machine learning training approaches based on identified impacts of one or more types of concept drift
US10657447B1
( en )
2018-11-29
2020-05-19
SparkCognition, Inc.
Automated model building search space reduction
US20200210769A1
( en )
2018-12-27
2020-07-02
Didi Research America, Llc
Using image pre-processing to generate a machine learning model
US20200242510A1
( en )
2019-01-30
2020-07-30
International Business Machines Corporation
System for constructing effective machine-learning pipelines with optimized outcomes
US20200333772A1
( en )
2019-04-18
2020-10-22
Siemens Industry Software Ltd.
Semantic modeling and machine learning-based generation of conceptual plans for manufacturing assemblies
US20200394044A1
( en )
2019-06-14
2020-12-17
Here Global B.V.
Method and apparatus for creating shared pipelines for data processing
US20200410011A1
( en )
2019-06-28
2020-12-31
Microsoft Technology Licensing, Llc
Vertical processing of natural language searches
US20210065048A1
( en )
2019-08-30
2021-03-04
International Business Machines Corporation
Automated artificial intelligence radial visualization
WO2021050382A1
( en )
2019-09-14
2021-03-18
Oracle International Corporation
Chatbot for defining a machine learning (ml) solution
US20210081837A1
( en )
2019-09-14
2021-03-18
Oracle International Corporation
Machine learning (ml) infrastructure techniques
WO2021051031A1
( en )
2019-09-14
2021-03-18
Oracle International Corporation
Techniques for adaptive and context-aware automated service composition for machine learning (ml)
US20210083855A1
( en )
2019-09-14
2021-03-18
Oracle International Corporation
Techniques for the safe serialization of the prediction pipeline
US20210133670A1
( en )
2019-11-05
2021-05-06
Strong Force Vcn Portfolio 2019, Llc
Control tower and enterprise management platform with a machine learning/artificial intelligence managing sensor and the camera feeds into digital twin
US20210174217A1
( en )
2019-12-06
2021-06-10
Accenture Global Solutions Limited
Systems and methods driven by link-specific numeric information for predicting associations based on predicate types
US20210250305A1
( en )
2020-02-06
2021-08-12
Camila do Espirito Santo
System and method to manage and exchange resources between enterprises in a cloud computing environment
US20210358601A1
( en )
2020-05-13
2021-11-18
Optum Technology, Inc.
Artificial intelligence system for clinical data semantic interoperability
US11182691B1
( en )
2014-08-14
2021-11-23
Amazon Technologies, Inc.
Category-based sampling of machine learning data
2023
2023-04-10
US
US18/132,859
patent/US12118474B2/en
active
Active
2024
2024-09-13
US
US18/885,502
patent/US20250013884A1/en
active
Pending
Patent Citations (80)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US5343527A
( en )
1993-10-27
1994-08-30
International Business Machines Corporation
Hybrid encryption method and system for protecting reusable software components
US5699507A
( en )
1995-01-17
1997-12-16
Lucent Technologies Inc.
Method of identifying similarities in code segments
US20040006761A1
( en )
2002-07-05
2004-01-08
Anand Minakshisundaran B.
System and method for automating generation of an automated sensor network
US20050102227A1
( en )
2003-11-09
2005-05-12
Aleksey Solonchev
Electronic commerce method and system utilizing integration server
US20080133435A1
( en )
2005-06-12
2008-06-05
Infosys Technologies, Ltd.
System for performance and scalability analysis and methods thereof
US20070043734A1
( en )
2005-08-17
2007-02-22
Cannon David M
Elimination of redundant objects in storage systems
US20070239630A1
( en )
2006-02-28
2007-10-11
International Business Machines Corporation
Method and system for allowing multiple applications to utilize customized feedback with a shared machine learning engine
US20090144698A1