Efficient duplicate detection for machine learning data sets — Amazon Technologies, Inc. (US12229642B2)

ABSTRACT

Abstract

At a machine learning service, a determination is made that an analysis to detect whether at least a portion of contents of one or more observation records of a first data set are duplicated in a second set of observation records is to be performed. A duplication metric is obtained, indicative of a non-zero probability that one or more observation records of the second set are duplicates of respective observation records of the first set. In response to determining that the duplication metric meets a threshold criterion, one or more responsive actions are initiated, such as the transmission of a notification to a client of the service.

Description

This application is a continuation of U.S. patent application Ser. No. 14/569,458, filed Dec. 12, 2014, which is a Continuation-In-Part of U.S. patent application Ser. No. 14/319,902, filed Jun. 30, 2014, now U.S. Pat. No. 10,102,480, which are herein incorporated by reference in their entirety.

BACKGROUND

Machine learning combines techniques from statistics and artificial intelligence to create algorithms that can learn from empirical data and generalize to solve problems in various domains such as natural language processing, financial fraud detection, terrorism threat level detection, human health diagnosis and the like. In recent years, more and more raw data that can potentially be utilized for machine learning models is being collected from a large variety of sources, such as sensors of various kinds, web server logs, social media services, financial transaction records, security cameras, and the like.

Traditionally, expertise in statistics and in artificial intelligence has been a prerequisite for developing and using machine learning models. For many business analysts and even for highly qualified subject matter experts, the difficulty of acquiring such expertise is sometimes too high a barrier to be able to take full advantage of the large amounts of data potentially available to make improved business predictions and decisions. Furthermore, many machine learning techniques can be computationally intensive, and in at least some cases it can be hard to predict exactly how much computing power may be required for various phases of the techniques. Given such unpredictability, it may not always be advisable or viable for business organizations to build out their own machine learning computational facilities.

The quality of the results obtained from machine learning algorithms may depend on how well the empirical data used for training the models captures key relationships among different variables represented in the data, and on how effectively and efficiently these relationships can be identified. Depending on the nature of the problem that is to be solved using machine learning, very large data sets may have to be analyzed in order to be able to make accurate predictions, especially predictions of relatively infrequent but significant events. For example, in financial fraud detection applications, where the number of fraudulent transactions is typically a very small fraction of the total number of transactions, identifying factors that can be used to label a transaction as fraudulent may potentially require analysis of millions of transaction records, each representing dozens or even hundreds of variables. Constraints on raw input data set size, cleansing or normalizing large numbers of potentially incomplete or error-containing records, and/or on the ability to extract representative subsets of the raw data also represent barriers that are not easy to overcome for many potential beneficiaries of machine learning techniques. For many machine learning problems, transformations may have to be applied on various input data variables before the data can be used effectively to train models. In some traditional machine learning environments, the mechanisms available to apply such transformations may be less than optimal—e.g., similar transformations may sometimes have to be applied one by one to many different variables of a data set, potentially requiring a lot of tedious and error-prone work.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 illustrates an example system environment in which various components of a machine learning service may be implemented, according to at least some embodiments.

FIG. 2 illustrates an example of a machine learning service implemented using a plurality of network-accessible services of a provider network, according to at least some embodiments.

FIG. 3 illustrates an example of the use of a plurality of availability containers and security containers of a provider network for a machine learning service, according to at least some embodiments.

FIG. 4 illustrates examples of a plurality of processing plans and corresponding resource sets that may be generated at a machine learning service, according to at least some embodiments.

FIG. 5 illustrates an example of asynchronous scheduling of jobs at a machine learning service, according to at least some embodiments.

FIG. 6 illustrates example artifacts that may be generated and stored using a machine learning service, according to at least some embodiments.

FIG. 7 illustrates an example of automated generation of statistics in response to a client request to instantiate a data source, according to at least some embodiments.

FIG. 8 illustrates several model usage modes that may be supported at a machine learning service, according to at least some embodiments.

FIGS. 9 a and 9 b are flow diagrams illustrating aspects of operations that may be performed at a machine learning service that supports asynchronous scheduling of machine learning jobs, according to at least some embodiments.

FIG. 10 a is a flow diagram illustrating aspects of operations that may be performed at a machine learning service at which a set of idempotent programmatic interfaces are supported, according to at least some embodiments.

FIG. 10 b is a flow diagram illustrating aspects of operations that may be performed at a machine learning service to collect and disseminate information about best practices related to different problem domains, according to at least some embodiments.

FIG. 11 illustrates examples interactions associated with the use of recipes for data transformations at a machine learning service, according to at least some embodiments.

FIG. 12 illustrates example sections of a recipe, according to at least some embodiments.

FIG. 13 illustrates an example grammar that may be used to define recipe syntax, according to at least some embodiments.

FIG. 14 illustrates an example of an abstract syntax tree that may be generated for a portion of a recipe, according to at least some embodiments.

FIG. 15 illustrates an example of a programmatic interface that may be used to search for domain-specific recipes available from a machine learning service, according to at least some embodiments.

FIG. 16 illustrates an example of a machine learning service that automatically explores a range of parameter settings for recipe transformations on behalf of a client, and selects acceptable or recommended parameter settings based on results of such explorations, according to at least some embodiments.

FIG. 17 is a flow diagram illustrating aspects of operations that may be performed at a machine learning service that supports re-usable recipes for data set transformations, according to at least some embodiments.

FIG. 18 illustrates an example procedure for performing efficient in-memory filtering operations on a large input data set by a machine learning service, according to at least some embodiments.

FIG. 19 illustrates tradeoffs associated with varying the chunk size used for filtering operation sequences on machine learning data sets, according to at least some embodiments.

FIG. 20 a illustrates an example sequence of chunk-level filtering operations, including a shuffle followed by a split, according to at least some embodiments.

FIG. 20 b illustrates an example sequence of in-memory filtering operations that includes chunk-level filtering as well as intra-chunk filtering, according to at least some embodiments.

FIG. 21 illustrates examples of alternative approaches to in-memory sampling of a data set, according to at least some embodiments.

FIG. 22 illustrates examples of determining chunk boundaries based on the location of observation record boundaries, according to at least some embodiments

FIG. 23 illustrates examples of jobs that may be scheduled at a machine learning service in response to a request for extraction of data records from any of a variety of data source types, according to at least some embodiments.

FIG. 24 illustrates examples constituent elements of a record retrieval request that may be submitted by a client using a programmatic interface of an I/O (input-output) library implemented by a machine learning service, according to at least some embodiments.

FIG. 25 is a flow diagram illustrating aspects of operations that may be performed at a machine learning service that implements an I/O library for in-memory filtering operation sequences on large input data sets, according to at least some embodiments.

FIG. 26 illustrates an example of an iterative procedure that may be used to improve the quality of predictions made by a machine learning model, according to at least some embodiments.

FIG. 27 illustrates an example of data set splits that may be used for cross-validation of a machine learning model, according to at least some embodiments.

FIG. 28 illustrates examples of consistent chunk-level splits of input data sets for cross validation that may be performed using a sequence of pseudo-random numbers, according to at least some embodiments.

FIG. 29 illustrates an example of an inconsistent chunk-level split of an input data set that may occur as a result of inappropriately resetting a pseudo-random number generator, according to at least some embodiments.

FIG. 30 illustrates an example timeline of scheduling related pairs of training and evaluation jobs, according to at least some embodiments.

FIG. 31 illustrates an example of a system in which consistency metadata is generated at a machine learning service in response to a client request, according to at least some embodiments.

FIG. 32 is a flow diagram illustrating aspects of operations that may be performed at a machine learning service in response to a request for training and evaluation iterations of a machine learning model, according to at least some embodiments.

FIG. 33 illustrates an example of a decision tree that may be generated for predictions at a machine learning service, according to at least some embodiments.

FIG. 34 illustrates an example of storing representations of decision tree nodes in a depth-first order at persistent storage devices during a tree-construction pass of a training phase for a machine learning model, according to at least some embodiments.

FIG. 35 illustrates an example of predictive utility distribution information that may be generated for the nodes of a decision tree, according to at least some embodiments.

FIG. 36 illustrates an example of pruning a decision tree based at least in part on a combination of a run-time memory footprint goal and cumulative predictive utility, according to at least some embodiments.

FIG. 37 illustrates an example of pruning a decision tree based at least in part on a prediction time variation goal, according to at least some embodiments.

FIG. 38 illustrates examples of a plurality of jobs that may be generated for training a model that uses an ensemble of decision trees at a machine learning service, according to at least some embodiments.

FIG. 39 is a flow diagram illustrating aspects of operations that may be performed at a machine learning service to generate and prune decision trees stored to persistent storage in depth-first order, according to at least some embodiments.

FIG. 40 illustrates an example of a machine learning service configured to generate feature processing proposals for clients based on an analysis of costs and benefits of candidate feature processing transformations, according to at least some embodiments.

FIG. 41 illustrates an example of selecting a feature processing set form several alternatives based on measured prediction speed and prediction quality, according to at least some embodiments.

FIG. 42 illustrates example interactions between a client and a feature processing manager of a machine learning service, according to at least some embodiments.

FIG. 43 illustrates an example of pruning candidate feature processing transformations using random selection, according to at least some embodiments.

FIG. 44 illustrates an example of a greedy technique for identifying recommended sets of candidate feature processing transformations, according to at least some embodiments.

FIG. 45 illustrates an example of a first phase of a feature processing optimization technique, in which a model is trained using a first set of candidate processed variables and evaluated, according to at least some embodiments.

FIG. 46 illustrates an example of a subsequent phase of the feature processing optimization technique, in which a model is re-evaluated using modified evaluation data sets to determine the impact on prediction quality of using various processed variables, according to at least some embodiments.

FIG. 47 illustrates another example phase of the feature processing optimization technique, in which a model is re-trained using a modified set of

This application is a continuation of U.S. patent application Ser. No. 14/569,458, filed Dec. 12, 2014, which is a Continuation-In-Part of U.S. patent application Ser. No. 14/319,902, filed Jun. 30, 2014, now U.S. Pat. No. 10,102,480, which are herein incorporated by reference in their entirety.

BACKGROUND

Machine learning combines techniques from statistics and artificial intelligence to create algorithms that can learn from empirical data and generalize to solve problems in various domains such as natural language processing, financial fraud detection, terrorism threat level detection, human health diagnosis and the like. In recent years, more and more raw data that can potentially be utilized for machine learning models is being collected from a large variety of sources, such as sensors of various kinds, web server logs, social media services, financial transaction records, security cameras, and the like.

Traditionally, expertise in statistics and in artificial intelligence has been a prerequisite for developing and using machine learning models. For many business analysts and even for highly qualified subject matter experts, the difficulty of acquiring such expertise is sometimes too high a barrier to be able to take full advantage of the large amounts of data potentially available to make improved business predictions and decisions. Furthermore, many machine learning techniques can be computationally intensive, and in at least some cases it can be hard to predict exactly how much computing power may be required for various phases of the techniques. Given such unpredictability, it may not always be advisable or viable for business organizations to build out their own machine learning computational facilities.

The quality of the results obtained from machine learning algorithms may depend on how well the empirical data used for training the models captures key relationships among different variables represented in the data, and on how effectively and efficiently these relationships can be identified. Depending on the nature of the problem that is to be solved using machine learning, very large data sets may have to be analyzed in order to be able to make accurate predictions, especially predictions of relatively infrequent but significant events. For example, in financial fraud detection applications, where the number of fraudulent transactions is typically a very small fraction of the total number of transactions, identifying factors that can be used to label a transaction as fraudulent may potentially require analysis of millions of transaction records, each representing dozens or even hundreds of variables. Constraints on raw input data set size, cleansing or normalizing large numbers of potentially incomplete or error-containing records, and/or on the ability to extract representative subsets of the raw data also represent barriers that are not easy to overcome for many potential beneficiaries of machine learning techniques. For many machine learning problems, transformations may have to be applied on various input data variables before the data can be used effectively to train models. In some traditional machine learning environments, the mechanisms available to apply such transformations may be less than optimal—e.g., similar transformations may sometimes have to be applied one by one to many different variables of a data set, potentially requiring a lot of tedious and error-prone work.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 illustrates an example system environment in which various components of a machine learning service may be implemented, according to at least some embodiments.

FIG. 2 illustrates an example of a machine learning service implemented using a plurality of network-accessible services of a provider network, according to at least some embodiments.

FIG. 3 illustrates an example of the use of a plurality of availability containers and security containers of a provider network for a machine learning service, according to at least some embodiments.

FIG. 4 illustrates examples of a plurality of processing plans and corresponding resource sets that may be generated at a machine learning service, according to at least some embodiments.

FIG. 5 illustrates an example of asynchronous scheduling of jobs at a machine learning service, according to at least some embodiments.

FIG. 6 illustrates example artifacts that may be generated and stored using a machine learning service, according to at least some embodiments.

FIG. 7 illustrates an example of automated generation of statistics in response to a client request to instantiate a data source, according to at least some embodiments.

FIG. 8 illustrates several model usage modes that may be supported at a machine learning service, according to at least some embodiments.

FIGS. 9 a and 9 b are flow diagrams illustrating aspects of operations that may be performed at a machine learning service that supports asynchronous scheduling of machine learning jobs, according to at least some embodiments.

FIG. 10 a is a flow diagram illustrating aspects of operations that may be performed at a machine learning service at which a set of idempotent programmatic interfaces are supported, according to at least some embodiments.

FIG. 10 b is a flow diagram illustrating aspects of operations that may be performed at a machine learning service to collect and disseminate information about best practices related to different problem domains, according to at least some embodiments.

FIG. 11 illustrates examples interactions associated with the use of recipes for data transformations at a machine learning service, according to at least some embodiments.

FIG. 12 illustrates example sections of a recipe, according to at least some embodiments.

FIG. 13 illustrates an example grammar that may be used to define recipe syntax, according to at least some embodiments.

FIG. 14 illustrates an example of an abstract syntax tree that may be generated for a portion of a recipe, according to at least some embodiments.

FIG. 15 illustrates an example of a programmatic interface that may be used to search for domain-specific recipes available from a machine learning service, according to at least some embodiments.

FIG. 16 illustrates an example of a machine learning service that automatically explores a range of parameter settings for recipe transformations on behalf of a client, and selects acceptable or recommended parameter settings based on results of such explorations, according to at least some embodiments.

FIG. 17 is a flow diagram illustrating aspects of operations that may be performed at a machine learning service that supports re-usable recipes for data set transformations, according to at least some embodiments.

FIG. 18 illustrates an example procedure for performing efficient in-memory filtering operations on a large input data set by a machine learning service, according to at least some embodiments.

FIG. 19 illustrates tradeoffs associated with varying the chunk size used for filtering operation sequences on machine learning data sets, according to at least some embodiments.

FIG. 20 a illustrates an example sequence of chunk-level filtering operations, including a shuffle followed by a split, according to at least some embodiments.

FIG. 20 b illustrates an example sequence of in-memory filtering operations that includes chunk-level filtering as well as intra-chunk filtering, according to at least some embodiments.

FIG. 21 illustrates examples of alternative approaches to in-memory sampling of a data set, according to at least some embodiments.

FIG. 22 illustrates examples of determining chunk boundaries based on the location of observation record boundaries, according to at least some embodiments

FIG. 23 illustrates examples of jobs that may be scheduled at a machine learning service in response to a request for extraction of data records from any of a variety of data source types, according to at least some embodiments.

FIG. 24 illustrates examples constituent elements of a record retrieval request that may be submitted by a client using a programmatic interface of an I/O (input-output) library implemented by a machine learning service, according to at least some embodiments.

FIG. 25 is a flow diagram illustrating aspects of operations that may be performed at a machine learning service that implements an I/O library for in-memory filtering operation sequences on large input data sets, according to at least some embodiments.

FIG. 26 illustrates an example of an iterative procedure that may be used to improve the quality of predictions made by a machine learning model, according to at least some embodiments.

FIG. 27 illustrates an example of data set splits that may be used for cross-validation of a machine learning model, according to at least some embodiments.

FIG. 28 illustrates examples of consistent chunk-level splits of input data sets for cross validation that may be performed using a sequence of pseudo-random numbers, according to at least some embodiments.

FIG. 29 illustrates an example of an inconsistent chunk-level split of an input data set that may occur as a result of inappropriately resetting a pseudo-random number generator, according to at least some embodiments.

FIG. 30 illustrates an example timeline of scheduling related pairs of training and evaluation jobs, according to at least some embodiments.

FIG. 31 illustrates an example of a system in which consistency metadata is generated at a machine learning service in response to a client request, according to at least some embodiments.

FIG. 32 is a flow diagram illustrating aspects of operations that may be performed at a machine learning service in response to a request for training and evaluation iterations of a machine learning model, according to at least some embodiments.

FIG. 33 illustrates an example of a decision tree that may be generated for predictions at a machine learning service, according to at least some embodiments.

FIG. 34 illustrates an example of storing representations of decision tree nodes in a depth-first order at persistent storage devices during a tree-construction pass of a training phase for a machine learning model, according to at least some embodiments.

FIG. 35 illustrates an example of predictive utility distribution information that may be generated for the nodes of a decision tree, according to at least some embodiments.

FIG. 36 illustrates an example of pruning a decision tree based at least in part on a combination of a run-time memory footprint goal and cumulative predictive utility, according to at least some embodiments.

FIG. 37 illustrates an example of pruning a decision tree based at least in part on a prediction time variation goal, according to at least some embodiments.

FIG. 38 illustrates examples of a plurality of jobs that may be generated for training a model that uses an ensemble of decision trees at a machine learning service, according to at least some embodiments.

FIG. 39 is a flow diagram illustrating aspects of operations that may be performed at a machine learning service to generate and prune decision trees stored to persistent storage in depth-first order, according to at least some embodiments.

FIG. 40 illustrates an example of a machine learning service configured to generate feature processing proposals for clients based on an analysis of costs and benefits of candidate feature processing transformations, according to at least some embodiments.

FIG. 41 illustrates an example of selecting a feature processing set form several alternatives based on measured prediction speed and prediction quality, according to at least some embodiments.

FIG. 42 illustrates example interactions between a client and a feature processing manager of a machine learning service, according to at least some embodiments.

FIG. 43 illustrates an example of pruning candidate feature processing transformations using random selection, according to at least some embodiments.

FIG. 44 illustrates an example of a greedy technique for identifying recommended sets of candidate feature processing transformations, according to at least some embodiments.

FIG. 45 illustrates an example of a first phase of a feature processing optimization technique, in which a model is trained using a first set of candidate processed variables and evaluated, according to at least some embodiments.

FIG. 46 illustrates an example of a subsequent phase of the feature processing optimization technique, in which a model is re-evaluated using modified evaluation data sets to determine the impact on prediction quality of using various processed variables, according to at least some embodiments.

FIG. 47 illustrates another example phase of the feature processing optimization technique, in which a model is re-trained using a modified set of processed variables to determine the impact on prediction run-time cost of using a processed variable, according to at least some embodiments.

FIG. 48 is a flow diagram illustrating aspects of operations that may be performed at a machine learning service that recommends feature processing transformations based on quality vs. run-time cost tradeoffs, according to at least some embodiments.

FIG. 49 is an example of a programmatic dashboard interface that may enable clients to view the status of a variety of machine learning model runs, according to at least some embodiments.

FIG. 50 illustrates an example procedure for generating and using linear prediction models, according to at least some embodiments.

FIG. 51 illustrates an example scenario in which the memory capacity of a machine learning server that is used for training a model may become a constraint on feature set size, according to at least some embodiments.

FIG. 52 illustrates a technique in which a subset of features of a feature set generated during training may be selected as pruning victims, according to at least some embodiments.

FIG. 53 illustrates a system in which observation records to be used for learning iterations of a linear model's training phase may be streamed to a machine learning service, according to at least some embodiments.

FIG. 54 is a flow diagram illustrating aspects of operations that may be performed at a machine learning service at which, in response to a detection of a triggering condition, parameters corresponding to one or more features may be pruned from a feature set to reduce memory consumption during training, according to at least some embodiments.

FIG. 55 illustrates a single-pass technique that may be used to obtain quantile boundary estimates of weights assigned to features, according to at least some embodiments.

FIG. 56 illustrates examples of using quantile binning transformations to capture non-linear relationships between raw input variables and prediction target variables of a machine learning model, according to at least some embodiments

FIG. 57 illustrates examples of concurrent binning plans that may be generated during a training phase of a model at a machine learning service, according to at least some embodiments.

FIG. 58 illustrates examples of concurrent multi-variable quantile binning transformations that may be implemented at a machine learning service, according to at least some embodiments.

FIG. 59 illustrates examples of recipes that may be used for representing concurrent binning operations at a machine learning service, according to at least some embodiments.

FIG. 60 illustrates an example of a system in which clients may utilize programmatic interfaces of a machine learning service to indicate their preferences regarding the use of concurrent quantile binning, according to at least some embodiments.

FIG. 61 is a flow diagram illustrating aspects of operations that may be performed at a machine learning service at which concurrent quantile binning transformations are implemented, according to at least some embodiments.

FIG. 62 illustrates an example system environment in which a machine learning service implements an interactive graphical interface enabling clients to explore tradeoffs between various prediction quality metric goals, and to modify settings that can be used for interpreting model execution results, according to at least some embodiments.

FIG. 63 illustrates an example view of results of an evaluation run of a binary classification model that may be provided via an interactive graphical interface, according to at least some embodiments.

FIGS. 64 a and 64 b collectively illustrate an impact of a change to a prediction interpretation threshold value, indicated by a client via a particular control of an interactive graphical interface, on a set of model quality metrics, according to at least some embodiments.

FIG. 65 illustrates examples of advanced metrics pertaining to an evaluation run of a machine learning model for which respective controls may be included in an interactive graphical interface, according to at least some embodiments.

FIG. 66 illustrates examples of elements of an interactive graphical interface that may be used to modify classification labels and to view details of observation records selected based on output variable values, according to at least some embodiments.

FIG. 67 illustrates an example view of results of an evaluation run of a multi-way classification model that may be provided via an interactive graphical interface, according to at least some embodiments.

FIG. 68 illustrates an example view of results of an evaluation run of a regression model that may be provided via an interactive graphical interface, according to at least some embodiments.

FIG. 69 is a flow diagram illustrating aspects of operations that may be performed at a machine learning service that implements interactive graphical interfaces enabling clients to modify prediction interpretation settings based on exploring evaluation results, according to at least some embodiments.

FIG. 70 illustrates an example duplicate detector that may utilize space-efficient representations of machine learning data sets to determine whether one data set is likely to include duplicate observation records of another data set at a machine learning service, according to at least some embodiments.

FIGS. 71 a and 71 b collectively illustrate an example of a use of a Bloom filter for probabilistic detection of duplicate observation records at a machine learning service, according to at least some embodiments.

FIG. 72 illustrates examples of alternative duplicate definitions that may be used at a duplicate detector of a machine learning service, according to at least some embodiments

FIG. 73 illustrates an example of a parallelized approach towards duplicate detection for large data sets at a machine learning service, according to at least some embodiments.

FIG. 74 illustrates an example of probabilistic duplicate detection within a given machine learning data set, according to at least some embodiments

FIG. 75 is a flow diagram illustrating aspects of operations that may be performed at a machine learning service that implements duplicate detection of observation records, according to at least some embodiments.

FIG. 76 is a block diagram illustrating an example computing device that may be used in at least some embodiments.

While embodiments are described herein by way of example for several embodiments and illustrative drawings, those skilled in the art will recognize that embodiments are not limited to the embodiments or drawings described. It should be understood, that the drawings and detailed description thereto are not intended to limit embodiments to the particular form disclosed, but on the contrary, the intention is to cover all modifications, equivalents and alternatives falling within the spirit and scope as defined by the appended claims. The headings used herein are for organizational purposes only and are not meant to be used to limit the scope of the description or the claims. As used throughout this application, the word “may” is used in a permissive sense (i.e., meaning having the potential to), rather than the mandatory sense (i.e., meaning must). Similarly, the words “include,” “including,” and “includes” mean including, but not limited to.

DETAILED DESCRIPTION

Various embodiments of methods and apparatus for a customizable, easy-to-use machine learning service (MLS) designed to support large numbers of users and a wide variety of algorithms and problem sizes are described. In one embodiment, a number of MLS programmatic interfaces (such as application programming interfaces (APIs)) may be defined by the service, which guide non-expert users to start using machine learning best practices relatively quickly, without the users having to expend a lot of time and effort on tuning models, or on learning advanced statistics or artificial intelligence techniques. The interfaces may, for example, allow non-experts to rely on default settings or parameters for various aspects of the procedures used for building, training and using machine learning models, where the defaults are derived from the accumulated experience of other practitioners addressing similar types of machine learning problems. At the same time, expert users may customize the parameters or settings they wish to use for various types of machine learning tasks, such as input record handling, feature processing, model building, execution and evaluation. In at least some embodiments, in addition to or instead of using pre-defined libraries implementing various types of machine learning tasks, MLS clients may be able to extend the built-in capabilities of the service, e.g., by registering their own customized functions with the service. Depending on the business needs or goals of the clients that implement such customized modules or functions, the modules may in some cases be shared with other users of the service, while in other cases the use of the customized modules may be restricted to their implementers/owners.

In some embodiments, a relatively straightforward recipe language may be supported, allowing MLS users to indicate various feature processing steps that they wish to have applied on data sets. Such recipes may be specified in text format, and then compiled into executable formats that can be re-used with different data sets on different resource sets as needed. In at least some embodiments, the MLS may be implemented at a provider network that comprises numerous data centers with hundreds of thousands of computing and storage devices distributed around the world, allowing machine learning problems with terabyte-scale or petabyte-scale data sets and correspondingly large compute requirements to be addressed in a relatively transparent fashion while still ensuring high levels of isolation and security for sensitive data. Pre-existing services of the provider network, such as storage services that support arbitrarily large data objects accessible via web service interfaces, database services, virtual computing services, parallel-computing services, high-performance computing services, load-balancing services, and the like may be used for various machine learning tasks in at least some embodiments. For MLS clients that have high availability and data durability requirements, machine learning data (e.g., raw input data, transformed/manipulated input data, intermediate results, or final results) and/or models may be replicated across different geographical locations or availability containers as described below. To meet an MLS client's data security needs, selected data sets, models or code implementing user-defined functions or third-party functions may be restricted to security containers defined by the provider network in some embodiments, in which for example the client's machine learning tasks are executed in an isolated, single-tenant fashion instead of the multi-tenant approach that may typically be used for some of the provider network's services. The term “MLS control plane” may be used herein to refer to a collection of hardware and/or software entities that are responsible for implementing various types of machine learning functionality on behalf of clients of the MLS, and for administrative tasks not necessarily visible to external MLS clients, such as ensuring that an adequate set of resources is provisioned to meet client demands, detecting and recovering from failures, generating bills, and so on. The term “MLS data plane” may refer to the pathways and resources used for the processing, transfer, and storage of the input data used for client-requested operations, as well as the processing, transfer and storage of output data produced as a result of client-requested operations.

According to some embodiments, a number of different types of entities related to machine learning tasks may be generated, modified, read, executed, and/or queried/searched via MLS programmatic interfaces. Supported entity types in one embodiment may include, among others, data sources (e.g., descriptors of locations or objects from which input records for machine learning can be obtained), sets of statistics generated by analyzing the input data, recipes (e.g., descriptors of feature processing transformations to be applied to input data for training models), processing plans (e.g., templates for executing various machine learning tasks), models (which may also be referred to as predictors), parameter sets to be used for recipes and/or models, model execution results such as predictions or evaluations, online access points for models that are to be used on streaming or real-time data, and/or aliases (e.g., pointers to model versions that have been “published” for use as described below). Instances of these entity types may be referred to as machine learning artifacts herein—for example, a specific recipe or a specific model may each be considered an artifact. Each of the entity types is discussed in further detail below.

The MLS programmatic interfaces may enable users to submit respective requests for several related tasks of a given machine learning workflow, such as tasks for extracting records from data sources, generating statistics on the records, feature processing, model training, prediction, and so on. A given invocation of a programmatic interface (such as an API) may correspond to a request for one or more operations or tasks on one or more instances of a supported type of entity. Some tasks (and the corresponding APIs) may involve multiple different entity types—e.g., an API requesting a creation of a data source may result in the generation of a data source entity instance as well as a statistics entity instance. Some of the tasks of a given workflow may be dependent on the results of other tasks. Depending on the amount of data, and/or on the nature of the processing to be performed, some tasks may take hours or even days to complete. In at least some embodiments, an asynchronous approach may be taken to scheduling the tasks, in which MLS clients can submit additional tasks that depend on the output of earlier-submitted tasks without waiting for the earlier-submitted tasks to complete. For example, a client may submit respective requests for tasks T2 and T3 before an earlier-submitted task T1 completes, even though the execution of T2 depends at least partly on the results of T1, and the execution of T3 depends at least partly on the results of T2. In such embodiments, the MLS may take care of ensuring that a given task is scheduled for execution only when its dependencies (if any dependencies exist) have been met.

A queue or collection of job objects may be used for storing internal representations of requested tasks in some implementations. The term “task”, as used herein, refers to a set of logical operations corresponding to a given request from a client, while the term “job” refers to the internal representation of a task within the MLS. In some embodiments, a given job object may represent the operations to be performed as a result of a client's invocation of a particular programmatic interface, as well as dependencies on other jobs. The MLS may be responsible for ensuring that the dependencies of a given job have been met before the corresponding operations are initiated. The MLS may also be responsible in such embodiments for generating a processing plan for each job, identifying the appropriate set of resources (e.g., CPUs/cores, storage or memory) for the plan, scheduling the execution of the plan, gathering results, providing/saving the results in an appropriate destination, and at least in some cases for providing status updates or responses to the requesting clients. The MLS may also be responsible in some embodiments for ensuring that the execution of one client's jobs do not affect or interfere with the execution of other clients' jobs. In some embodiments, partial dependencies among tasks may be supported—e.g., in a sequence of tasks (T1, T2, T3), T2 may depend on partial completion of T1, and T2 may therefore be scheduled before T1 completes. For example, T1 may comprise two phases or passes P1 and P2 of statistics calculations, and T2 may be able to proceed as soon as phase P1 is completed, without waiting for phase P2 to complete. Partial results of T1 (e.g., at least some statistics computed during phase P1) may be provided to the requesting client as soon as they become available in some cases, instead of waiting for the entire task to be completed. A single shared queue that includes jobs corresponding to requests from a plurality of clients of the MLS may be used in some implementations, while in other implementations respective queues may be used for different clients. In some implementations, lists or other data structures that can be used to model object collections may be used as containers of to-be-scheduled jobs instead of or in addition to queues. In some embodiments, a single API request from a client may lead to the generation of several different job objects by the MLS. In at least one embodiment, not all client API requests may be implemented using jobs—e.g., a relatively short or lightweight task may be performed synchronously with respect to the corresponding request, without incurring the overhead of job creation and asynchronous job scheduling.

The APIs implemented by the MLS may in some embodiments allow clients to submit requests to create, query the attributes of, read, update/modify, search, or delete an instance of at least some of the various entity types supported. For example, for the entity type “DataSource”, respective APIs similar to “createDataSource”, “describeDataSource” (to obtain the values of attributes of the data source), “updateDataSource”, “searchForDataSource”, and “deleteDataSource” may be supported by the MLS. A similar set of APIs may be supported for recipes, models, and so on. Some entity types may also have APIs for executing or running the entities, such as “executeModel” or “executeRecipe” in various embodiments. The APIs may be designed to be largely easy to learn and self-documenting (e.g., such that the correct way to use a given API is obvious to non-experts), with an emphasis on making it simple to perform the most common tasks without making it too hard to perform more complex tasks. In at least some embodiments multiple versions of the APIs may be supported: e.g., one version for a wire protocol (at the application level of a networking stack), another version as a Java™ library or SDK (software development kit), another version as a Python library, and so on. API requests may be submitted by clients using HTTP (Hypertext Transfer Protocol), HTTPS (secure HTTP), Javascript, XML, or the like in various implementations.

In some embodiments, some machine learning models may be created and trained, e.g., by a group of model developers or data scientists using the MLS APIs, and then published for use by another community of users. In order to facilitate publishing of models for use by a wider audience than just the creators of the model, while preventing potentially unsuitable modifications to the models by unskilled members of the wider audience, the “alias” entity type may be supported in such embodiments. In one embodiment, an alias may comprise an immutable name (e.g., “SentimentAnalysisModel1”) and a pointer to a model that has already been created and stored in an MLS artifact repository (e.g., “samModel-23adf-2013-12-13-08-06-01”, an internal identifier generated for the model by the MLS). Different sets of permissions on aliases may be granted to model developers than are granted to the users to whom the aliases are being made available for execution. For example, in one implementation, members of a business analyst group may be allowed to run the model using its alias name, but may not be allowed to change the pointer, while model developers may be allowed to modify the pointer and/or modify the underlying model. For the business analysts, the machine learning model exposed via the alias may represent a “black box” tool, already validated by experts, which is expected to provide useful predictions for various input data sets. The business analysts may not be particularly concerned about the internal working of such a model. The model developers may continue to experiment with various algorithms, parameters and/or input data sets to obtain improved versions of the underlying model, and may be able to change the pointer to point to an enhanced version to improve the quality of predictions obtained by the business analysts. In at least some embodiments, to isolate alias users from changes to the underlying models, the MLS may guarantee that (a) an alias can only point to a model that has been successfully trained and (b) when an alias pointer is changed, both the original model and the new model (i.e., the respective models being pointed to by the old pointer and the new pointer) consume the same type of input and provide the same type of prediction (e.g., binary classification, multi-class classification or regression). In some implementations, a given model may itself be designated as un-modifiable if an alias is created for it—e.g., the model referred to by the pointer “samModel-23adf-2013-12-13-08-06-01” may no longer be modified even by its developers after the alias is created in such an implementation. Such clean separation of roles and capabilities with respect to model development and use may allow larger audiences within a business organization to benefit from machine learning models than simply those skilled enough to develop the models.

A number of choices may be available with respect to the manner in which the operations corresponding to a given job are mapped to MLS servers. For example, it may be possible to partition the work required for a given job among many different servers to achieve better performance. As part of developing the processing plan for a job, the MLS may select a workload distribution strategy for the job in some embodiments. The parameters determined for workload distribution in various embodiments may differ based on the nature of the job. Such factors may include, for example, (a) determining a number of passes of processing, (b) determining a parallelization level (e.g., the number of “mappers” and “reducers” in the case of a job that is to be implemented using the Map-Reduce technique), (c) determining a convergence criterion to be used to terminate the job, (d) determining a target durability level for intermediate data produced during the job, or (e) determining a resource capacity limit for the job (e.g., a maximum number of servers that can be assigned to the job based on the number of servers available in MLS server pools, or on the client's budget limit). After the workload strategy is selected, the actual set of resources to be used may be identified in accordance with the strategy, and the job's operations may be scheduled on the identified resources. In some embodiments, a pool of compute servers and/or storage servers may be pre-configured for the MLS, and the resources for a given job may be selected from such a pool. In other embodiments, the resources may be selected from a pool assigned to the client on whose behalf the job is to be executed—e.g., the client may acquire resources from a computing service of the provider network prior to submitting API requests, and may provide an indication of the acquired resources to the MLS for job scheduling. If client-provided code (e.g., code that has not necessarily been thoroughly tested by the MLS, and/or is not included in the MLS's libraries) is being used for a given job, in some embodiments the client may be required to acquire the resources to be used for the job, so that any side effects of running the client-provided code may be restricted to the client's own resources instead of potentially affecting other clients.

Example System Environments

FIG. 1 illustrates an example system environment in which various components of a machine learning service (MLS) may be implemented, according to at least some embodiments. In system 100 , the MLS may implement a set of programmatic interfaces 161 (e.g., APIs, command-line tools, web pages, or standalone GUIs) that can be used by clients 164 (e.g., hardware or software entities owned by or assigned to customers of the MLS) to submit requests 111 for a variety of machine learning tasks or operations. The administrative or control plane portion of the MLS may include MLS request handler 180 , which accepts the client requests 111 and inserts corresponding job objects into MLS job queue 142 , as indicated by arrow 112 . In general, the control plane of the MLS may comprise a plurality of components (including the request handler, workload distribution strategy selectors, one or more job schedulers, metrics collectors, and modules that act as interfaces with other services) which may also be referred to collectively as the MLS manager. The data plane of the MLS may include, for example, at least a subset of the servers of pool(s) 185 , storage devices that are used to store input data sets, intermediate results or final results (some of which may be part of the MLS artifact repository), and the network pathways used for transferring client input data and results.

As mentioned earlier, each job object may indicate one or more operations that are to be performed as a result of the invocation of a programmatic interface 161 , and the scheduling of a given job may in some cases depend upon the successful completion of at least a subset of the operations of an earlier-generated job. In at least some implementations, job queue 142 may be managed as a first-in-first-out (FIFO) queue, with the further constraint that the dependency requirements of a given job must have been met in order for that job to be removed from the queue. In some embodiments, jobs created on behalf of several different clients may be placed in a single queue, while in other embodiments multiple queues may be maintained (e.g., one queue in each data center of the provider network being used, or one queue per MLS customer). Asynchronously with respect to the submission of the requests 111 , the next job whose dependency requirements have been met may be removed from job queue 142 in the depicted embodiment, as indicated by arrow 113 , and a processing plan comprising a workload distribution strategy may be identified for it. The workload distribution strategy layer 175 , which may also be a component of the MLS control plane as mentioned earlier, may determine the manner in which the lower level operations of the job are to be distributed among one or more compute servers (e.g., servers selected from pool 185 ), and/or the manner in which the data analyzed or manipulated for the job is to be distributed among one or more storage devices or servers. After the processing plan has been generated and the appropriate set of resources to be utilized for the job has been identified, the job's operations may be scheduled on the resources. Results of some jobs may be stored as MLS artifacts within repository 120 in some embodiments, as indicated by arrow 142 .

In at least one embodiment, some relatively simple types of client requests 111 may result in the immediate generation, retrieval, storage, or modification of corresponding artifacts within MLS artifact repository 120 by the MLS request handler 180 (as indicated by arrow 141 ). Thus, the insertion of a job object in job queue 142 may not be required for all types of client requests. For example, a creation or removal of an alias for an existing model may not require the creation of a new job in such embodiments. In the embodiment shown in FIG. 1 , clients 164 may be able to view at least a subset of the artifacts stored in repository 120 , e.g., by issuing read requests 118 via programmatic interfaces 161 .

A client request 111 may indicate one or more parameters that may be used by the MLS to perform the operations, such as a data source definition 150 , a feature processing transformation recipe 152 , or parameters 154 to be used for a particular machine learning algorithm. In some embodiments, artifacts respectively representing the parameters may also be stored in repository 120 . Some machine learning workflows, which may correspond to a sequence of API requests from a client 164 , may include the extraction and cleansing of input data records from raw data repositories 130 (e.g., repositories indicated in data source definitions 150 ) by input record handlers 160 of the MLS, as indicated by arrow 114 . This first portion of the workflow may be initiated in response to a particular API invocation from a client 164 , and may be executed using a first set of resources from pool 185 . The input record handlers may, for example, perform such tasks as splitting the data records, sampling the data records, and so on, in accordance with a set of functions defined in an I/O (input/output) library of the MLS. The input data may comprise data records that include variables of any of a variety of data types, such as, for example text, a numeric data type (e.g., real or integer), Boolean, a binary data type, a categorical data type, an image processing data type, an audio processing data type, a bioinformatics data type, a structured data type such as a data type compliant with the Unstructured Information Management Architecture (UIMA), and so on. In at least some embodiments, the input data reaching the MLS may be encrypted or compressed, and the MLS input data handling machinery may have to perform decryption or decompression before the input data records can be used for machine learning tasks. In some embodiments in which encryption is used, MLS clients may have to provide decryption metadata (e.g., keys, passwords, or other credentials) to the MLS to allow the MLS to decrypt data records. Similarly, an indication of the compression technique used may be provided by the clients in some implementations to enable the MLS to decompress the input data records appropriately. The output produced by the input record handlers may be fed to feature processors 162 (as indicated by arrow 115 ), where a set of transformation operations may be performed 162 in accordance with recipes 152 using another set of resources from pool 185 . Any of a variety of feature processing approaches may be used depending on the problem domain: e.g., the recipes typically used for computer vision problems may differ from those used for voice recognition problems, natural language processing, and so on. The output 116 of the feature processing transformations may in turn be used as input for a selected machine learning algorithm 166 , which may be executed in accordance with algorithm parameters 154 using yet another set of resources from pool 185 . A wide variety of machine learning algorithms may be supported natively by the MLS libraries, including for example random forest algorithms, neural network algorithms, stochastic gradient descent algori

CLAIMS

Claims ( 20 )

What is claimed is:

1. A computer-implemented method, comprising:

obtaining, via one or more programmatic interfaces at a service of a cloud computing environment, a request to tune one or more hyper-parameters of a machine learning model which performs one or more machine learning tasks, wherein tuning the one or more hyper-parameters of the machine learning model affect the result of the one or more machine learning tasks;

based on the one or more hyper-parameters, selecting by the service, respective ranges of values applicable for individual ones of the one or more hyper-parameters to determine respective candidate values for the one or more hyper-parameters from the respective ranges of values that optimize the one or more hyper-parameters, wherein the respective ranges of values are selected by the service based, at least in part, on:

one or more respective ranges of values of the service for one or more previously obtained requests to tune one or more hyper-parameters of one or more machine learning models;

performing one or more tuning operations in which the respective ranges of values for individual ones of the one or more hyper-parameters are utilized for the machine learning model;

evaluating respective results of the one or more tuning operations to generate the respective candidate values for the one or more hyper-parameters according to the evaluation based, at least in part, on a comparison of results for the one or more machine learning tasks that selects the respective candidate values out of the ranges of values for the individual ones of the one or more hyper-parameters according to result similarity; and

providing, via the one or more programmatic interfaces, the respective candidate values for the one or more hyper-parameters.

2. The computer-implemented method as recited in claim 1 , wherein the request comprises an indication that the service is to identify an acceptance criterion for terminating tuning of the one or more hyper-parameters, and wherein the one or more tuning operations are terminated after determining that an acceptance criterion identified by the service has been satisfied.

3. The computer-implemented method as recited in claim 1 , wherein the request comprises an indication of a resource limit for tuning the one or more hyper-parameters, and wherein the one or more tuning operations are terminated based at least in part on the resource limit.

4. The computer-implemented method as recited in claim 1 , wherein the respective candidate values for the one or more hyper-parameters are a set of candidate values for at least one of the one or more hyper-parameters.

5. The computer-implemented method as recited in claim 1 , wherein the machine learning model one or more tuning operations comprises a feature processing transformation.

6. The computer-implemented method as recited in claim 1 , wherein the one or more hyper-parameters comprise an image transformation parameter.

7. The computer-implemented method as recited in claim 1 , wherein the one or more hyper-parameters comprise a boundary to be used for grouping data.

8. A system, comprising:

one or more computing devices;

wherein the one or more computing devices include instructions that upon execution on or across the one or more computing devices cause the one or more computing devices to:

obtain, via one or more programmatic interfaces at a service of a cloud computing environment, a request to tune one or more hyper-parameters of a machine learning model which performs one or more machine learning tasks, wherein tuning the one or more hyper-parameters of the machine learning model affect the results of the one or more machine learning tasks;

based on the one or more hyper-parameters, select by the service, respective ranges of values applicable for individual ones of the one or more hyper-parameters to determine respective candidate values for the one or more hyper-parameters from the respective ranges of values that optimize the one or more hyper-parameters, wherein the respective ranges of values are selected by the service based, at least in part, on:

one or more respective ranges of values of the service for one or more previously obtained requests to tune one or more hyper-parameters of one or more machine learning models;

perform one or more tuning operations in which the respective ranges of values for individual ones of the one or more hyper-parameters are utilized for the machine learning model;

evaluate respective results of the one or more tuning operations for the ranges of values to generate the respective candidate values for the one or more hyper-parameters according to the evaluation based, at least in part, on a comparison of results for the one or more machine learning tasks that selects the respective candidate values out of the ranges of values for the individual ones of the one or more hyper-parameters according to result similarity; and

provide, via the one or more programmatic interfaces, the respective candidate values for the one or more hyper-parameters.

9. The system as recited in claim 8 , wherein the request comprises an indication that the service is to identify an acceptance criterion for terminating tuning of the one or more hyper-parameters, and wherein the one or more tuning operations are terminated after determining that an acceptance criterion identified by the service has been satisfied.

10. The system as recited in claim 8 , wherein the request comprises an indication of a resource limit for tuning the one or more hyper-parameters, and wherein the one or more tuning operations are terminated based at least in part on the resource limit.

11. The system as recited in claim 8 , wherein the respective candidate values for the one or more hyper-parameters are a set of candidate values for at least one of the one or more hyper-parameters.

12. The system as recited in claim 8 , wherein the machine learning model one or more tuning operations comprises a feature processing transformation.

13. The system as recited in claim 8 , wherein the one or more hyper-parameters comprise a number of words to be included in an n-gram.

14. The system as recited in claim 8 , wherein the one or more hyper-parameters comprise a group boundary to be used for grouping data.

15. One or more non-transitory computer-accessible storage media storing program instructions that when executed on or across one or more processors cause the one or more processors to:

obtain, via one or more programmatic interfaces at a service of a cloud computing environment, a request to tune one or more hyper-parameters of a machine learning model which performs one or more machine learning tasks, wherein tuning the one or more hyper-parameters of the machine learning model affect the result of the one or more machine learning tasks;

based on the one or more hyper-parameters, select respective ranges of values applicable for individual ones of the one or more hyper-parameters to determine respective candidate values for the one or more hyper-parameters from the respective ranges of values that optimize the one or more hyper-parameters, wherein the respective ranges of values are selected by the service based at least in part on:

one or more respective ranges of values of the service for one or more previously obtained requests to tune one or more hyper-parameters of one or more machine learning models;

perform one or more tuning operations in which the respective ranges of values for individual ones of the one or more hyper-parameters are utilized for the machine learning;

evaluate respective results of the one or more tuning operations for the ranges of values to generate the respective candidate values for the one or more hyper-parameters according to the evaluation based, at least in part, on a comparison of results for the one or more machine learning tasks that selects the respective candidate values out of the ranges of values for the individual ones of the one or more hyper-parameters according to result summary; and

provide, via the one or more programmatic interfaces, the respective candidate values for the one or more hyper-parameters.

16. The one or more non-transitory computer-accessible storage media as recited in claim 15 , wherein the request comprises an indication that the service is to identify an acceptance criterion for terminating tuning of the one or more hyper-parameters, and wherein the one or more tuning operations are terminated after determining that an acceptance criterion identified by the service has been satisfied.

17. The one or more non-transitory computer-accessible storage media as recited in claim 15 , wherein the request comprises an indication of a resource limit for tuning the one or more hyper-parameters, and wherein the one or more tuning operations are terminated based at least in part on the resource limit.

18. The one or more non-transitory computer-accessible storage media as recited in claim 15 , wherein the respective candidate values for the one or more hyper-parameters are a set of candidate values for at least one of the one or more hyper-parameters.

19. The one or more non-transitory computer-accessible storage media as recited in claim 15 , wherein the machine learning model tuning operations comprises a feature processing transformation.

20. The one or more non-transitory computer-accessible storage media as recited in claim 15 , wherein the one or more hyper-parameters comprise a bin size to be used for quantile binning.

US17/214,047

2014-06-30

2021-03-26

Efficient duplicate detection for machine learning data sets

Active

2035-06-16

US12229642B2

( en )

Priority Applications (1)

Application Number

Priority Date

Filing Date

Title

US17/214,047

US12229642B2

( en )

2014-06-30

2021-03-26

Efficient duplicate detection for machine learning data sets

Applications Claiming Priority (3)

Application Number

Priority Date

Filing Date

Title

US14/319,902

US10102480B2

( en )

2014-06-30

2014-06-30

Machine learning service

US14/569,458

US10963810B2

( en )

2014-06-30

2014-12-12

Efficient duplicate detection for machine learning data sets

US17/214,047

US12229642B2

( en )

2014-06-30

2021-03-26

Efficient duplicate detection for machine learning data sets

Related Parent Applications (1)

Application Number

Title

Priority Date

Filing Date

US14/569,458

Continuation

US10963810B2

( en )

2014-06-30

2014-12-12

Efficient duplicate detection for machine learning data sets

Publications (2)

Publication Number

Publication Date

US20210374610A1

US20210374610A1 ( en )

2021-12-02

US12229642B2

true

US12229642B2 ( en )

2025-02-18

Family

ID=54930937

Family Applications (2)

Application Number

Title

Priority Date

Filing Date

US14/569,458

Active

2037-11-18

US10963810B2

( en )

2014-06-30

2014-12-12

Efficient duplicate detection for machine learning data sets

US17/214,047

Active

2035-06-16

US12229642B2

( en )

2014-06-30

2021-03-26

Efficient duplicate detection for machine learning data sets

Family Applications Before (1)

Application Number

Title

Priority Date

Filing Date

US14/569,458

Active

2037-11-18

US10963810B2

( en )

2014-06-30

2014-12-12

Efficient duplicate detection for machine learning data sets

Country Status (1)

Country

Link

US

( 2 )

US10963810B2

( en )

Cited By (3)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20230153395A1

( en )

*

2021-11-18

2023-05-18

International Business Machines Corporation

Class prediction based on class accuracy of multiple models

US20240126902A1

( en )

*

2022-10-18

2024-04-18

Bank Of America Corporation

Ingestion layer for improved security of distributed information

US12585647B2

( en )

*

2024-06-18

2026-03-24

Sap Se

Query execution system with plan reuse

Families Citing this family (523)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US10318970B2

( en )

2013-10-04

2019-06-11

International Business Machines Corporation

Generating a succinct approximate representation of a time series

US10395198B2

( en )

2013-10-04

2019-08-27

International Business Machines Corporation

Forecasting a time series based on actuals and a plan

US9917727B2

( en )

*

2014-06-03

2018-03-13

Nicira, Inc.

Consistent hashing for network traffic dispatching

US10169715B2

( en )

*

2014-06-30

2019-01-01

Amazon Technologies, Inc.

Feature processing tradeoff management

US9672474B2

( en )

*

2014-06-30

2017-06-06

Amazon Technologies, Inc.

Concurrent binning of machine learning data

US10452992B2

( en )

2014-06-30

2019-10-22

Amazon Technologies, Inc.

Interactive interfaces for machine learning model evaluations

US10963810B2

( en )

2014-06-30

2021-03-30

Amazon Technologies, Inc.

Efficient duplicate detection for machine learning data sets

US10102480B2

( en )

2014-06-30

2018-10-16

Amazon Technologies, Inc.

Machine learning service

US9886670B2

( en )

2014-06-30

2018-02-06

Amazon Technologies, Inc.

Feature processing recipes for machine learning

US11100420B2

( en )

2014-06-30

2021-08-24

Amazon Technologies, Inc.

Input processing for machine learning

US10339465B2

( en )

2014-06-30

2019-07-02

Amazon Technologies, Inc.

Optimized decision tree based models

US10540606B2

( en )

2014-06-30

2020-01-21

Amazon Technologies, Inc.

Consistent filtering of machine learning data

US10318882B2

( en )

2014-09-11

2019-06-11

Amazon Technologies, Inc.

Optimized training of linear machine learning models

US10289867B2

( en )

2014-07-27

2019-05-14

OneTrust, LLC

Data processing systems for webform crawling to map processing activities and related methods

US9729583B1

( en )

2016-06-10

2017-08-08

OneTrust, LLC

Data processing systems and methods for performing privacy assessments and monitoring of new versions of computer code for privacy compliance

US10181051B2

( en )

2016-06-10

2019-01-15

OneTrust, LLC

Data processing systems for generating and populating a data inventory for processing data access requests

US9986378B2

( en )

*

2014-07-29

2018-05-29

GeoFrenzy, Inc.

Systems and methods for defining and implementing rules for three dimensional geofences

US11182691B1

( en )

2014-08-14

2021-11-23

Amazon Technologies, Inc.

Category-based sampling of machine learning data

US11093982B1

( en )

2014-10-02

2021-08-17

Zillow, Inc.

Determine regional rate of return on home improvements

US9805099B2

( en )

*

2014-10-30

2017-10-31

The Johns Hopkins University

Apparatus and method for efficient identification of code similarity

US11783927B2

( en )

*

2014-11-11

2023-10-10

Healthsparq, Inc.

Methods and systems for calculating health care treatment statistics

US10325220B2

( en )

*

2014-11-17

2019-06-18

Oath Inc.

System and method for large-scale multi-label learning using incomplete label assignments

US9679024B2

( en )

*

2014-12-01

2017-06-13

Facebook, Inc.

Social-based spelling correction for online social networks

US9922037B2

( en )

2015-01-30

2018-03-20

Splunk Inc.

Index time, delimiter based extractions and previewing for use in indexing

US11562286B2

( en )

2015-02-06

2023-01-24

Box, Inc.

Method and system for implementing machine learning analysis of documents for classifying documents by associating label values to the documents

US11005950B1

( en )

*

2015-02-10

2021-05-11

Marvell Asia Pte, Ltd.

Optimized bloom filter

GB201502371D0

( en )

*

2015-02-13

2015-04-01

Bae Systems Plc

Provision, configuration and use of a telecommunications network

JP6617771B2

( en )

*

2015-06-02

2019-12-11

日本電気株式会社

Linear parameter variation model estimation system, method and program

US10339467B2

( en )

*

2015-06-02

2019-07-02

International Business Machines Corporation

Quantitative discovery of name changes

US11068827B1

( en )

*

2015-06-22

2021-07-20

Wells Fargo Bank, N.A.

Master performance indicator

WO2017003499A1

( en )

*

2015-06-29

2017-01-05

Wepay, Inc.

System and methods for generating reason codes for ensemble computer models

US10191966B2

( en )

*

2015-07-08

2019-01-29

Business Objects Software Ltd.

Enabling advanced analytics with large data sets

US10437945B2

( en )

*

2015-08-05

2019-10-08

Arizona Board Of Regents On Behalf Of Arizona State University

Systems and methods for order-of-magnitude viral cascade prediction in social networks

US10628801B2

( en )

*

2015-08-07

2020-04-21

Tata Consultancy Services Limited

System and method for smart alerts

US10972332B2

( en )

*

2015-08-31

2021-04-06

Adobe Inc.

Identifying factors that contribute to a metric anomaly

US9699205B2

( en )

*

2015-08-31

2017-07-04

Splunk Inc.

Network security system

US10664777B2

( en )

*

2015-09-11

2020-05-26

Workfusion, Inc.

Automated recommendations for task automation

US10089481B2

( en )

*

2015-09-23

2018-10-02

International Business Machines Corporation

Securing recorded data

US10373069B2

( en )

*

2015-09-26

2019-08-06

Intel Corporation

Technologies for platform-targeted machine learning

US20170091673A1

( en )

*

2015-09-29

2017-03-30

Skytree, Inc.

Exporting a Transformation Chain Including Endpoint of Model for Prediction

US10169719B2

( en )

*

2015-10-20

2019-01-01

International Business Machines Corporation

User configurable message anomaly scoring to identify unusual activity in information technology systems

US10257275B1

( en )

2015-10-26

2019-04-09

Amazon Technologies, Inc.

Tuning software execution environments using Bayesian models

US10127266B2

( en )

*

2015-11-03

2018-11-13

International Business Machines Corporation

Scalable virtual namespace and semantic policy management for object storage

US10628826B2

( en )

*

2015-11-24

2020-04-21

Vesta Corporation

Training and selection of multiple fraud detection models

US20170168832A1

( en )

*

2015-12-11

2017-06-15

International Business Machines Corporation

Instruction weighting for performance profiling in a group dispatch processor

CA2952174A1

( en )

2015-12-18

2017-06-18

Chandra Dhandapani

Automated systems for reducing computational loads in the mass execution of analytical models using scale-out computing

US10235780B2

( en )

*

2015-12-18

2019-03-19

Dropbox, Inc.

Integrated data exploration, modeling and visualization

EP3188038B1

( en )

*

2015-12-31

2020-11-04

Dassault Systèmes

Evaluation of a training set

US11232085B2

( en )

*

2016-01-07

2022-01-25

Amazon Technologies, Inc.

Outlier detection for streaming data

CN105678587B

( en )

*

2016-01-12

2020-11-24

腾讯科技（深圳）有限公司

A recommendation feature determination method, information recommendation method and device

US10162867B2

( en )

*

2016-01-13

2018-12-25

International Business Machines Corporation

Low memory sampling-based estimation of distinct elements and deduplication

US10417346B2

( en )

*

2016-01-23

2019-09-17

Microsoft Technology Licensing, Llc

Tool for facilitating the development of new language understanding scenarios

US10789549B1

( en )

*

2016-02-25

2020-09-29

Zillow, Inc.

Enforcing, with respect to changes in one or more distinguished independent variable values, monotonicity in the predictions produced by a statistical model

US10713589B1

( en )

*

2016-03-03

2020-07-14

Amazon Technologies, Inc.

Consistent sort-based record-level shuffling of machine learning data

US10789547B2

( en )

*

2016-03-14

2020-09-29

Business Objects Software Ltd.

Predictive modeling optimization

US10949852B1

( en )

2016-03-25

2021-03-16

State Farm Mutual Automobile Insurance Company

Document-based fraud detection

US12125039B2

( en )

2016-03-25

2024-10-22

State Farm Mutual Automobile Insurance Company

Reducing false positives using customer data and machine learning

US10176503B2

( en )

2016-04-01

2019-01-08

OneTrust, LLC

Data processing systems and methods for efficiently assessing the risk of privacy campaigns

US11244367B2

( en )

2016-04-01

2022-02-08

OneTrust, LLC

Data processing systems and methods for integrating privacy information management systems with data loss prevention tools or other tools for privacy design

US10176502B2

( en )

2016-04-01

2019-01-08

OneTrust, LLC

Data processing systems and methods for integrating privacy information management systems with data loss prevention tools or other tools for privacy design

US9892444B2

( en )

2016-04-01

2018-02-13

OneTrust, LLC

Data processing systems and communication systems and methods for the efficient generation of privacy risk assessments

US10423996B2

( en )

2016-04-01

2019-09-24

OneTrust, LLC

Data processing systems and communication systems and methods for the efficient generation of privacy risk assessments

US9898769B2

( en )

2016-04-01

2018-02-20

OneTrust, LLC

Data processing systems and methods for operationalizing privacy compliance via integrated mobile applications

US10706447B2

( en )

2016-04-01

2020-07-07

OneTrust, LLC

Data processing systems and communication systems and methods for the efficient generation of privacy risk assessments

US11004125B2

( en )

2016-04-01

2021-05-11

OneTrust, LLC

Data processing systems and methods for integrating privacy information management systems with data loss prevention tools or other tools for privacy design

US9892443B2

( en )

2016-04-01

2018-02-13

OneTrust, LLC

Data processing systems for modifying privacy campaign data via electronic messaging systems

US12288233B2

( en )

2016-04-01

2025-04-29

OneTrust, LLC

Data processing systems and methods for integrating privacy information management systems with data loss prevention tools or other tools for privacy design

US10853661B2

( en )

2016-04-06

2020-12-01

Intellective Ai, Inc.

Methods and systems for optimized selection of data features for a neuro-linguistic cognitive artificial intelligence system

US20170293606A1

( en )

*

2016-04-06

2017-10-12

Gang Xu

Methods and systems for optimized selection of data features for a neuro-linguistic cognitive artifical intelligence system

US10827185B2

( en )

*

2016-04-07

2020-11-03

Netflix, Inc.

Techniques for robustly predicting perceptual video quality

US10229100B1

( en )

2016-04-22

2019-03-12

Intuit Inc.

Augmented reality form fill

AU2017260842A1

( en )

2016-05-03

2018-11-01

Yembo, Inc.

Systems and methods for providing ai-based cost estimates for services

US10867328B2

( en )

2016-05-03

2020-12-15

Yembo, Inc.

Systems and methods for providing AI-based cost estimates for services

US10592917B2

( en )

*

2016-05-03

2020-03-17

Cox Automotive, Inc.

Method and systems for determining programmatically expected performances

US11544783B1

( en )

2016-05-12

2023-01-03

State Farm Mutual Automobile Insurance Company

Heuristic credit risk assessment engine

US10769722B1

( en )

2016-05-12

2020-09-08

State Farm Mutual Automobile Insurance Company

Heuristic credit risk assessment engine

US10740409B2

( en )

*

2016-05-20

2020-08-11

Magnet Forensics Inc.

Systems and methods for graphical exploration of forensic data

WO2017197526A1

( en )

2016-05-20

2017-11-23

Roman Czeslaw Kordasiewicz

Systems and methods for graphical exploration of forensic data

US10108328B2

( en )

*

2016-05-20

2018-10-23

Vmware, Inc.

Method for linking selectable parameters within a graphical user interface

US11514304B2

( en )

*

2016-05-26

2022-11-29

Samsung Sds America, Inc.

Continuously provisioning large-scale machine learning models

US10984385B2

( en )

*

2016-05-31

2021-04-20

Microsoft Technology Licensing, Llc

Query building for search by ideal candidates

CN106911592B

( en )

*

2016-06-01

2020-06-12

创新先进技术有限公司

Self-adaptive resource allocation method and device

US11438386B2

( en )

2016-06-10

2022-09-06

OneTrust, LLC

Data processing systems for data-transfer risk identification, cross-border visualization generation, and related methods

US10438017B2

( en )

2016-06-10

2019-10-08

OneTrust, LLC

Data processing systems for processing data subject access requests

US10949565B2

( en )

2016-06-10

2021-03-16

OneTrust, LLC

Data processing systems for generating and populating a data inventory

US11416589B2

( en )

2016-06-10

2022-08-16

OneTrust, LLC

Data processing and scanning systems for assessing vendor risk

US12136055B2

( en )

2016-06-10

2024-11-05

OneTrust, LLC

Data processing systems for identifying, assessing, and remediating data processing risks using data modeling techniques

US10181019B2

( en )

2016-06-10

2019-01-15

OneTrust, LLC

Data processing systems and communications systems and methods for integrating privacy compliance systems with software development and agile tools for privacy design

US11134086B2

( en )

2016-06-10

2021-09-28

OneTrust, LLC

Consent conversion optimization systems and related methods

US10282559B2

( en )

2016-06-10

2019-05-07

OneTrust, LLC

Data processing systems for identifying, assessing, and remediating data processing risks using data modeling techniques

US10565397B1

( en )

2016-06-10

2020-02-18

OneTrust, LLC

Data processing systems for fulfilling data subject access requests and related methods

US11675929B2

( en )

2016-06-10

2023-06-13

OneTrust, LLC

Data processing consent sharing systems and related methods

US10706379B2

( en )

2016-06-10

2020-07-07

OneTrust, LLC

Data processing systems for automatic preparation for remediation and related methods

US10282700B2

( en )

2016-06-10

2019-05-07

OneTrust, LLC

Data processing systems for generating and populating a data inventory

US11227247B2

( en )

2016-06-10

2022-01-18

OneTrust, LLC

Data processing systems and methods for bundled privacy policies

US10769301B2

( en )

2016-06-10

2020-09-08

OneTrust, LLC

Data processing systems for webform crawling to map processing activities and related methods

US10346638B2

( en )

2016-06-10

2019-07-09

OneTrust, LLC

Data processing systems for identifying and modifying processes that are subject to data subject access requests

US10496803B2

( en )

2016-06-10

2019-12-03

OneTrust, LLC

Data processing systems and methods for efficiently assessing the risk of privacy campaigns

US11138242B2

( en )

2016-06-10

2021-10-05

OneTrust, LLC

Data processing systems and methods for automatically detecting and documenting privacy-related aspects of computer software

US11392720B2

( en )

2016-06-10

2022-07-19

OneTrust, LLC

Data processing systems for verification of consent and notice processing and related methods

US11144622B2

( en )

2016-06-10

2021-10-12

OneTrust, LLC

Privacy management systems and methods

US10997318B2

( en )

2016-06-10

2021-05-04

OneTrust, LLC

Data processing systems for generating and populating a data inventory for processing data access requests

US11188615B2

( en )

2016-06-10

2021-11-30

OneTrust, LLC

Data processing consent capture systems and related methods

US10585968B2

( en )

2016-06-10

2020-03-10

OneTrust, LLC

Data processing systems for fulfilling data subject access requests and related methods

US11416590B2

( en )

2016-06-10

2022-08-16

OneTrust, LLC

Data processing and scanning systems for assessing vendor risk

US10873606B2

( en )

2016-06-10

2020-12-22

OneTrust, LLC

Data processing systems for data-transfer risk identification, cross-border visualization generation, and related methods

US10318761B2

( en )

2016-06-10

2019-06-11

OneTrust, LLC

Data processing systems and methods for auditing data request compliance

US11138299B2

( en )

2016-06-10

2021-10-05

OneTrust, LLC

Data processing and scanning systems for assessing vendor risk

US10496846B1

( en )

2016-06-10

2019-12-03

OneTrust, LLC

Data processing and communications systems and methods for the efficient implementation of privacy by design

US11100444B2

( en )

2016-06-10

2021-08-24

OneTrust, LLC

Data processing systems and methods for providing training in a vendor procurement process

US12299065B2

( en )

2016-06-10

2025-05-13

OneTrust, LLC

Data processing systems and methods for dynamically determining data processing consent configurations

US10685140B2

( en )

2016-06-10

2020-06-16

OneTrust, LLC

Consent receipt management systems and related methods

US10416966B2

( en )

2016-06-10

2019-09-17

OneTrust, LLC

Data processing systems for identity validation of data subject access requests and related methods

US10503926B2

( en )

Related documents

• Machine unlearning using forgetting neural networks#405948
• Finding Duplicates in 1.1M BDD Steps: cukereuse, a Paraphrase-Robust Static Detector for Cucumber and Gherkin#124115
• Automated cloud data and technology solution delivery using machine learning … — Mckinsey & Company, Inc. (US11645548B1)#607050
• Automated machine learning#132297
• SAGE: Similarity-Based Cleaning of Poisoned Training Data from Verified Examples#1160929