ABSTRACT
Abstract
Systems and methods for transforming legacy models and transforming a model into a neural network model are disclosed. In an embodiment, a method may include receiving input data comprising an input model, an input dataset, and an input command. The method may include applying the input model to the input dataset to generate model output and storing model output and at least one of input model features or a map of the input model. The method may include generating a candidate neural network models with parameters. The method may include tuning the candidate neural network models to the input model. The method may include receiving model output from the candidate neural network models and selecting a neural network model from the candidate neural network models based on the candidate model output and the model selection criteria. In some aspects, the method may include returning the selected neural network model.
Description
CROSS REFERENCE TO RELATED APPLICATIONS
This application claims the benefit of U.S. Provisional Application No. 62/694,968, filed Jul. 6, 2018, and incorporated herein by reference in its entirety.
This application also relates to U.S. patent application Ser. No. 16/151,385 filed on Oct. 4, 2018, and titled Data Model Generation Using Generative Adversarial Networks, the disclosure of which is also incorporated herein by reference in its entirety.
TECHNICAL FIELD
The disclosed embodiments concern a platform for management of artificial intelligence systems. In particular, the disclosed embodiments concern using the disclosed platform to create neural network models of data based on previously existing models, including legacy models. A legacy model is a model that runs in a programming environment that has been or is being replaced by a different programming environment. These data models can be used, for example, to generate synthetic data for testing or training artificial intelligence systems. The disclosed embodiments also concern improvements transforming any model, including a legacy model, into a neural network model.
BACKGROUND
Neural network models provide advantages over conventional modeling approaches. Neural network models model data efficiently, adaptably, and accurately than conventional models. Neural network models may be recurrent neural network models, deep learning models, long-short-term memory (LSTM) models, convolutional neural network (CNN) models, generative adversarial networks (GANs), and the like. Neural network models meet different modeling needs than conventional models and can analyze a wide variety of data as data analysis objectives evolve.
A neural network model may be designed according to an underlying physical or relational understanding of the data structure. Further, to assist user understanding of a model, it may be desirable to base the neural network model on an original model, e.g., a conventional model, that captures physical or statistical relationships between data elements. Further, a neural network model designed based on the features of the original model may behave more like the original model when encountering new datasets, and thereby yield similar insights that are easier for the user to understand. Therefore, it is often desirable to transform original models into neural network models instead of, for instance, replacing original models with completely new neural network models. For example, a neural network model may train more efficiently and more accurately predict an outcome if certain nodes are trained to replicate a linear regression model that had already identified certain statistically significant relationships between input variables. Further, a previously trained predictive neural network model may be an efficient seed model for an updated, more accurate predictive neural network model because the previously trained model may have learned to make predictions with some accuracy.
Conventional approaches to transform a model into a neural network model involve time consuming efforts to oversee the development of the neural network model. Many organizations lack the resources or capacity to engage in conventional approaches to transforming a model into a neural network model.
In addition, there is a need to transform models of a given type (e.g., a random forest model, a gradient boosting machine, a regression model, a linear regression model, a symbolic model, a neural network model, or other model), into a new model of the same type. This may happen, for instance, with a legacy model. For example, an organization may be in the process of phasing out SAS and implementing PYTHON across a platform, so the organization may need to transform legacy SAS models to PYTHON models.
Conventionally, to transform a legacy model into a model running in a new environment, developers manually performed many steps of the transformation, which is a time consuming, costly, and error-prone process. For example, to transform a random forest model built with SCIKIT into a gradient boosting model built for XGBOOST, the developer must configure the system, install dependencies, import libraries, modify code to accept new output types of XGBOOST, and remove outdated dependencies. Any of these steps could lead to errors and extensive time debugging the new model code. Many organizations lack the resources or capacity to engage in conventional approaches to transforming a legacy model.
Therefore, in light of the shortcomings and deficiencies of conventional methods of modeling and of transforming models, systems and methods that reduce development time, reduce cost, improve modeling accuracy, and increase flexibility are desirable. There is a need for systems and methods that transform a model of any type, e.g., a legacy model, into a new neural network model. There is also a need for systems and methods that transform a legacy model running in one type of environment into a model (e.g., a conventional model or a neural network model) that can run in a different environment. Further, for organizations or users that lack the resources or capacity to transform models independently, there is a need for model transformation to be provided as a service.
SUMMARY
The disclosed embodiments provide unconventional systems and methods for transforming models that are more efficient, less costly, more flexible, and more accurate than conventional approaches. The disclosed embodiments provide unconventional systems and methods to transform any input model into a neural network model. The transformed neural network model is designed based on the features of the input model and is designed to overfit the input model. By overfitting the input model, the transformed neural network model accurately reproduces the modeling results of the input model on training datasets and is likely to behave like the input model when analyzing other, non-training datasets.
Further, the disclosed embodiments provide systems and methods to transform legacy models into new models of the same type using machine learning. By using neural network models to create a new model that is the same type as the legacy model but will run in a different environment than the legacy model, the unconventional systems and methods of disclosed embodiments save time, reduce costs, and reduce errors.
The disclosed embodiments include a system for transforming a model into a neural network model. The system may include one or more memory units for storing instructions, and one or more processors configured to execute the instructions to perform operations. The operations may include receiving input data comprising an input model, an input dataset, and an input command specifying model selection criteria. The operations may include applying the input model to the input dataset to generate model output. The operations may include storing model output and at least one of input model features or a map of the input model and generating a plurality of candidate neural network models. The parameters of the candidate neural network models may be based on the input model features. The operations may include tuning the plurality of candidate neural network models to the input model. The operations may include receiving model output from the plurality of candidate neural network models and selecting a neural network model from the plurality of the candidate neural network models based on the candidate model output and the model selection criteria. In some aspects, the operations may include returning the selected neural network model.
Consistent with disclosed embodiments, a method for transforming a model into a neural network model is disclosed. The method may include receiving input data comprising an input model, an input dataset, and an input command specifying model selection criteria. The method may include applying the input model to the input dataset to generate model output. The method may include storing model output and at least one of input model features or a map of the input model and generating a plurality of candidate neural network models. The parameters of the candidate neural network models may be based on the input model features. The method may include tuning the plurality of candidate neural network models to the input model. The method may include receiving model output from the plurality of candidate neural network models and selecting a neural network model from the plurality of the candidate neural network models based on the candidate model output and the model selection criteria. In some aspects, the method may include returning the selected neural network model.
Consistent with other disclosed embodiments, non-transitory computer readable storage media may store program instructions, which are executed by at least one processor device and perform any of the methods described herein.
The foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the claims.
BRIEF DESCRIPTION OF THE DRAWINGS
The drawings are not necessarily to scale or exhaustive. Instead, emphasis is generally placed upon illustrating the principles of the embodiments described herein. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate several embodiments consistent with the disclosure and, together with the description, serve to explain the principles of the disclosure. In the drawings:
FIG. 1 depicts an exemplary cloud-computing environment for generating data models, consistent with disclosed embodiments.
FIG. 2 depicts an exemplary process for generating data models, consistent with disclosed embodiments.
FIG. 3 depicts an exemplary process for generating synthetic data using existing data models, consistent with disclosed embodiments.
FIG. 4 depicts an exemplary implementation of the cloud-computing environment of FIG. 1 , consistent with disclosed embodiments.
FIG. 5A depicts an exemplary process for generating synthetic data using class-specific models, consistent with disclosed embodiments.
FIG. 5B depicts an exemplary process for generating synthetic data using class and subclass-specific models, consistent with disclosed embodiments.
FIG. 6 depicts an exemplary process for training a classifier for generation of synthetic data, consistent with disclosed embodiments.
FIG. 7 depicts an exemplary process for training a classifier for generation of synthetic data, consistent with disclosed embodiments.
FIG. 8 depicts an exemplary process for training a generative adversarial using a normalized reference dataset, consistent with disclosed embodiments.
FIG. 9 depicts an exemplary process for training a generative adversarial network using a loss function configured to ensure a predetermined degree of similarity, consistent with disclosed embodiments.
FIG. 10 depicts an exemplary process for supplementing or transform datasets using code-space operations, consistent with disclosed embodiments.
FIGS. 11A and 11B depict an exemplary illustration of points in code-space, consistent with disclosed embodiments.
FIG. 12A depicts an exemplary illustration of supplementing datasets using code-space operations, consistent with disclosed embodiments.
FIG. 12B depicts an exemplary illustration of transforming datasets using code-space operations, consistent with disclosed embodiments.
FIG. 13 depicts an exemplary cloud computing system for generating a synthetic data stream that tracks a reference data stream, consistent with disclosed embodiments.
FIG. 14 depicts a process for generating synthetic JSON log data using the cloud computing system of FIG. 13 , consistent with disclosed embodiments.
FIG. 15 depicts a system for secure generation and insecure use of models of sensitive data, consistent with disclosed embodiments.
FIG. 16 depicts a process for transforming any model into a neural network model, consistent with disclosed embodiments.
FIG. 17 depicts a process for transforming a legacy model, consistent with disclosed embodiments.
DETAILED DESCRIPTION
Reference will now be made in detail to exemplary embodiments, discussed with regards to the accompanying drawings. In some instances, the same reference numbers will be used throughout the drawings and the following description to refer to the same or like parts. Unless otherwise defined, technical and/or scientific terms have the meaning commonly understood by one of ordinary skill in the art. The disclosed embodiments are described in sufficient detail to enable those skilled in the art to practice the disclosed embodiments. It is to be understood that other embodiments may be utilized and that changes may be made without departing from the scope of the disclosed embodiments. Thus, the materials, methods, and examples are illustrative only and are not intended to be necessarily limiting.
The disclosed embodiments can be used to create models of datasets, which may include sensitive datasets (e.g., customer financial information, patient healthcare information, and the like). Using these models, the disclosed embodiments can produce fully synthetic datasets with similar structure and statistics as the original sensitive or non-sensitive datasets. The disclosed embodiments also provide tools for desensitizing datasets and tokenizing sensitive values. In some embodiments, the disclosed systems can include a secure environment for training a model of sensitive data, and a non-secure environment for generating synthetic data with similar structure and statistics as the original sensitive data. In various embodiments, the disclosed systems can be used to tokenize the sensitive portions of a dataset (e.g., mailing addresses, social security numbers, email addresses, account numbers, demographic information, and the like). In some embodiments, the disclosed systems can be used to replace parts of sensitive portions of the dataset (e.g., preserve the first or last 3 digits of an account number, social security number, or the like; change a name to a first and last initial). In some aspects, the dataset can include one or more JSON (JavaScript Object Notation) or delimited files (e.g., comma-separated value, or CSV, files). In various embodiments, the disclosed systems can automatically detect sensitive portions of structured and unstructured datasets and automatically replace them with similar but synthetic values.
FIG. 1 depicts a cloud- computing environment 100 for generating data models. Environment 100 can be configured to suppo
CROSS REFERENCE TO RELATED APPLICATIONS
This application claims the benefit of U.S. Provisional Application No. 62/694,968, filed Jul. 6, 2018, and incorporated herein by reference in its entirety.
This application also relates to U.S. patent application Ser. No. 16/151,385 filed on Oct. 4, 2018, and titled Data Model Generation Using Generative Adversarial Networks, the disclosure of which is also incorporated herein by reference in its entirety.
TECHNICAL FIELD
The disclosed embodiments concern a platform for management of artificial intelligence systems. In particular, the disclosed embodiments concern using the disclosed platform to create neural network models of data based on previously existing models, including legacy models. A legacy model is a model that runs in a programming environment that has been or is being replaced by a different programming environment. These data models can be used, for example, to generate synthetic data for testing or training artificial intelligence systems. The disclosed embodiments also concern improvements transforming any model, including a legacy model, into a neural network model.
BACKGROUND
Neural network models provide advantages over conventional modeling approaches. Neural network models model data efficiently, adaptably, and accurately than conventional models. Neural network models may be recurrent neural network models, deep learning models, long-short-term memory (LSTM) models, convolutional neural network (CNN) models, generative adversarial networks (GANs), and the like. Neural network models meet different modeling needs than conventional models and can analyze a wide variety of data as data analysis objectives evolve.
A neural network model may be designed according to an underlying physical or relational understanding of the data structure. Further, to assist user understanding of a model, it may be desirable to base the neural network model on an original model, e.g., a conventional model, that captures physical or statistical relationships between data elements. Further, a neural network model designed based on the features of the original model may behave more like the original model when encountering new datasets, and thereby yield similar insights that are easier for the user to understand. Therefore, it is often desirable to transform original models into neural network models instead of, for instance, replacing original models with completely new neural network models. For example, a neural network model may train more efficiently and more accurately predict an outcome if certain nodes are trained to replicate a linear regression model that had already identified certain statistically significant relationships between input variables. Further, a previously trained predictive neural network model may be an efficient seed model for an updated, more accurate predictive neural network model because the previously trained model may have learned to make predictions with some accuracy.
Conventional approaches to transform a model into a neural network model involve time consuming efforts to oversee the development of the neural network model. Many organizations lack the resources or capacity to engage in conventional approaches to transforming a model into a neural network model.
In addition, there is a need to transform models of a given type (e.g., a random forest model, a gradient boosting machine, a regression model, a linear regression model, a symbolic model, a neural network model, or other model), into a new model of the same type. This may happen, for instance, with a legacy model. For example, an organization may be in the process of phasing out SAS and implementing PYTHON across a platform, so the organization may need to transform legacy SAS models to PYTHON models.
Conventionally, to transform a legacy model into a model running in a new environment, developers manually performed many steps of the transformation, which is a time consuming, costly, and error-prone process. For example, to transform a random forest model built with SCIKIT into a gradient boosting model built for XGBOOST, the developer must configure the system, install dependencies, import libraries, modify code to accept new output types of XGBOOST, and remove outdated dependencies. Any of these steps could lead to errors and extensive time debugging the new model code. Many organizations lack the resources or capacity to engage in conventional approaches to transforming a legacy model.
Therefore, in light of the shortcomings and deficiencies of conventional methods of modeling and of transforming models, systems and methods that reduce development time, reduce cost, improve modeling accuracy, and increase flexibility are desirable. There is a need for systems and methods that transform a model of any type, e.g., a legacy model, into a new neural network model. There is also a need for systems and methods that transform a legacy model running in one type of environment into a model (e.g., a conventional model or a neural network model) that can run in a different environment. Further, for organizations or users that lack the resources or capacity to transform models independently, there is a need for model transformation to be provided as a service.
SUMMARY
The disclosed embodiments provide unconventional systems and methods for transforming models that are more efficient, less costly, more flexible, and more accurate than conventional approaches. The disclosed embodiments provide unconventional systems and methods to transform any input model into a neural network model. The transformed neural network model is designed based on the features of the input model and is designed to overfit the input model. By overfitting the input model, the transformed neural network model accurately reproduces the modeling results of the input model on training datasets and is likely to behave like the input model when analyzing other, non-training datasets.
Further, the disclosed embodiments provide systems and methods to transform legacy models into new models of the same type using machine learning. By using neural network models to create a new model that is the same type as the legacy model but will run in a different environment than the legacy model, the unconventional systems and methods of disclosed embodiments save time, reduce costs, and reduce errors.
The disclosed embodiments include a system for transforming a model into a neural network model. The system may include one or more memory units for storing instructions, and one or more processors configured to execute the instructions to perform operations. The operations may include receiving input data comprising an input model, an input dataset, and an input command specifying model selection criteria. The operations may include applying the input model to the input dataset to generate model output. The operations may include storing model output and at least one of input model features or a map of the input model and generating a plurality of candidate neural network models. The parameters of the candidate neural network models may be based on the input model features. The operations may include tuning the plurality of candidate neural network models to the input model. The operations may include receiving model output from the plurality of candidate neural network models and selecting a neural network model from the plurality of the candidate neural network models based on the candidate model output and the model selection criteria. In some aspects, the operations may include returning the selected neural network model.
Consistent with disclosed embodiments, a method for transforming a model into a neural network model is disclosed. The method may include receiving input data comprising an input model, an input dataset, and an input command specifying model selection criteria. The method may include applying the input model to the input dataset to generate model output. The method may include storing model output and at least one of input model features or a map of the input model and generating a plurality of candidate neural network models. The parameters of the candidate neural network models may be based on the input model features. The method may include tuning the plurality of candidate neural network models to the input model. The method may include receiving model output from the plurality of candidate neural network models and selecting a neural network model from the plurality of the candidate neural network models based on the candidate model output and the model selection criteria. In some aspects, the method may include returning the selected neural network model.
Consistent with other disclosed embodiments, non-transitory computer readable storage media may store program instructions, which are executed by at least one processor device and perform any of the methods described herein.
The foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the claims.
BRIEF DESCRIPTION OF THE DRAWINGS
The drawings are not necessarily to scale or exhaustive. Instead, emphasis is generally placed upon illustrating the principles of the embodiments described herein. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate several embodiments consistent with the disclosure and, together with the description, serve to explain the principles of the disclosure. In the drawings:
FIG. 1 depicts an exemplary cloud-computing environment for generating data models, consistent with disclosed embodiments.
FIG. 2 depicts an exemplary process for generating data models, consistent with disclosed embodiments.
FIG. 3 depicts an exemplary process for generating synthetic data using existing data models, consistent with disclosed embodiments.
FIG. 4 depicts an exemplary implementation of the cloud-computing environment of FIG. 1 , consistent with disclosed embodiments.
FIG. 5A depicts an exemplary process for generating synthetic data using class-specific models, consistent with disclosed embodiments.
FIG. 5B depicts an exemplary process for generating synthetic data using class and subclass-specific models, consistent with disclosed embodiments.
FIG. 6 depicts an exemplary process for training a classifier for generation of synthetic data, consistent with disclosed embodiments.
FIG. 7 depicts an exemplary process for training a classifier for generation of synthetic data, consistent with disclosed embodiments.
FIG. 8 depicts an exemplary process for training a generative adversarial using a normalized reference dataset, consistent with disclosed embodiments.
FIG. 9 depicts an exemplary process for training a generative adversarial network using a loss function configured to ensure a predetermined degree of similarity, consistent with disclosed embodiments.
FIG. 10 depicts an exemplary process for supplementing or transform datasets using code-space operations, consistent with disclosed embodiments.
FIGS. 11A and 11B depict an exemplary illustration of points in code-space, consistent with disclosed embodiments.
FIG. 12A depicts an exemplary illustration of supplementing datasets using code-space operations, consistent with disclosed embodiments.
FIG. 12B depicts an exemplary illustration of transforming datasets using code-space operations, consistent with disclosed embodiments.
FIG. 13 depicts an exemplary cloud computing system for generating a synthetic data stream that tracks a reference data stream, consistent with disclosed embodiments.
FIG. 14 depicts a process for generating synthetic JSON log data using the cloud computing system of FIG. 13 , consistent with disclosed embodiments.
FIG. 15 depicts a system for secure generation and insecure use of models of sensitive data, consistent with disclosed embodiments.
FIG. 16 depicts a process for transforming any model into a neural network model, consistent with disclosed embodiments.
FIG. 17 depicts a process for transforming a legacy model, consistent with disclosed embodiments.
DETAILED DESCRIPTION
Reference will now be made in detail to exemplary embodiments, discussed with regards to the accompanying drawings. In some instances, the same reference numbers will be used throughout the drawings and the following description to refer to the same or like parts. Unless otherwise defined, technical and/or scientific terms have the meaning commonly understood by one of ordinary skill in the art. The disclosed embodiments are described in sufficient detail to enable those skilled in the art to practice the disclosed embodiments. It is to be understood that other embodiments may be utilized and that changes may be made without departing from the scope of the disclosed embodiments. Thus, the materials, methods, and examples are illustrative only and are not intended to be necessarily limiting.
The disclosed embodiments can be used to create models of datasets, which may include sensitive datasets (e.g., customer financial information, patient healthcare information, and the like). Using these models, the disclosed embodiments can produce fully synthetic datasets with similar structure and statistics as the original sensitive or non-sensitive datasets. The disclosed embodiments also provide tools for desensitizing datasets and tokenizing sensitive values. In some embodiments, the disclosed systems can include a secure environment for training a model of sensitive data, and a non-secure environment for generating synthetic data with similar structure and statistics as the original sensitive data. In various embodiments, the disclosed systems can be used to tokenize the sensitive portions of a dataset (e.g., mailing addresses, social security numbers, email addresses, account numbers, demographic information, and the like). In some embodiments, the disclosed systems can be used to replace parts of sensitive portions of the dataset (e.g., preserve the first or last 3 digits of an account number, social security number, or the like; change a name to a first and last initial). In some aspects, the dataset can include one or more JSON (JavaScript Object Notation) or delimited files (e.g., comma-separated value, or CSV, files). In various embodiments, the disclosed systems can automatically detect sensitive portions of structured and unstructured datasets and automatically replace them with similar but synthetic values.
FIG. 1 depicts a cloud- computing environment 100 for generating data models. Environment 100 can be configured to support generation and storage of synthetic data, generation and storage of data models, optimized choice of parameters for machine learning, and imposition of rules on synthetic data and data models. Environment 100 can be configured to expose an interface for communication with other systems. Environment 100 can include computing resources 101 , dataset generator 103 , database 105 , model optimizer 107 , model storage 109 , model curator 111 , and interface 113 . These components of environment 100 can be configured to communicate with each other, or with external components of environment 100 , using network 115 . The particular arrangement of components depicted in FIG. 1 is not intended to be limiting. System 100 can include additional components, or fewer components. Multiple components of system 100 can be implemented using the same physical computing device or different physical computing devices.
Computing resources 101 can include one or more computing devices configurable to train data models. The computing devices can be special-purpose computing devices, such as graphical processing units (GPUs) or application-specific integrated circuits. The cloud computing instances can be general-purpose computing devices. The computing devices can be configured to host an environment for training data models. For example, the computing devices can host virtual machines, pods, or containers. The computing devices can be configured to run applications for generating data models. For example, the computing devices can be configured to run SAGEMAKER, or similar machine learning training applications. Computing resources 101 can be configured to receive models for training from model optimizer 107 , model storage 109 , or another component of system 100 . Computing resources 101 can be configured provide training results, including trained models and model information, such as the type and/or purpose of the model and any measures of classification error.
Dataset generator 103 can include one or more computing devices configured to generate data. Dataset generator 103 can be configured to provide data to computing resources 101 , database 105 , to another component of system 100 (e.g., interface 113 ), or another system (e.g., an APACHE KAFKA cluster or other publication service). Dataset generator 103 can be configured to receive data from database 105 or another component of system 100 . Dataset generator 103 can be configured to receive data models from model storage 109 or another component of system 100 . Dataset generator 103 can be configured to generate synthetic data. For example, dataset generator 103 can be configured to generate synthetic data by identifying and replacing sensitive information in data received from database 103 or interface 113 . As an additional example, dataset generator 103 can be configured to generate synthetic data using a data model without reliance on input data. For example, the data model can be configured to generate data matching statistical and content characteristics of a training dataset. In some aspects, the data model can be configured to map from a random or pseudorandom vector to elements in the training data space.
Database 105 can include one or more databases configured to store data for use by system 100 . The databases can include cloud-based databases (e.g., AMAZON WEB SERVICES S3 buckets) or on-premises databases.
Model optimizer 107 can include one or more computing systems configured to manage training of data models for system 100 . Model optimizer 107 can be configured to generate models for export to computing resources 101 . Model optimizer 107 can be configured to generate models based on instructions received from a user or another system. These instructions can be received through interface 113 . For example, model optimizer 107 can be configured to receive a graphical depiction of a machine learning model and parse that graphical depiction into instructions for creating and training a corresponding neural network on computing resources 101 . Model optimizer 107 can be configured to select model training parameters. This selection can be based on model performance feedback received from computing resources 101 . Model optimizer 107 can be configured to provide trained models and descriptive information concerning the trained models to model storage 109 .
Model storage 109 can include one or more databases configured to store data models and descriptive information for the data models. Model storage 109 can be configured to provide information regarding available data models to a user or another system. This information can be provided using interface 113 . The databases can include cloud-based databases (e.g., AMAZON WEB SERVICES S3 buckets) or on-premises databases. The information can include model information, such as the type and/or purpose of the model and any measures of classification error.
Model curator 111 can be configured to impose governance criteria on the use of data models. For example, model curator 111 can be configured to delete or control access to models that fail to meet accuracy criteria. As a further example, model curator 111 can be configured to limit the use of a model to a particular purpose, or by a particular entity or individual. In some aspects, model curator 11 can be configured to ensure that data model satisfies governance criteria before system 100 can process data using the data model.
Interface 113 can be configured to manage interactions between system 100 and other systems using network 115 . In some aspects, interface 113 can be configured to publish data received from other components of system 100 (e.g., dataset generator 103 , computing resources 101 , database 105 , or the like). This data can be published in a publication and subscription framework (e.g., using APACHE KAFKA), through a network socket, in response to queries from other systems, or using other known methods. The data can be synthetic data, as described herein. As an additional example, interface 113 can be configured to provide information received from model storage 109 regarding available datasets. In various aspects, interface 113 can be configured to provide data or instructions received from other systems to components of system 100 . For example, interface 113 can be configured to receive instructions for generating data models (e.g., type of data model, data model parameters, training data indicators, training parameters, or the like) from another system and provide this information to model optimizer 107 . As an additional example, interface 113 can be configured to receive data including sensitive portions from another system (e.g. in a file, a message in a publication and subscription framework, a network socket, or the like) and provide that data to dataset generator 103 or database 105 .
Network 115 can include any combination of electronics communications networks enabling communication between components of system 100 . For example, network 115 may include the Internet and/or any type of wide area network, an intranet, a metropolitan area network, a local area network (LAN), a wireless network, a cellular communications network, a Bluetooth network, a radio network, a device bus, or any other type of electronics communications network known to one of skill in the art.
FIG. 2 depicts a process 200 for generating data models. Process 200 can be used to generate a data model for a machine learning application, consistent with disclosed embodiments. The data model can be generated using synthetic data in some aspects. This synthetic data can be generated using a synthetic dataset model, which can in turn be generated using actual data. The synthetic data may be similar to the actual data in terms of values, value distributions (e.g., univariate and multivariate statistics of the synthetic data may be similar to that of the actual data), structure and ordering, or the like. In this manner, the data model for the machine learning application can be generated without directly using the actual data. As the actual data may include sensitive information, and generating the data model may require distribution and/or review of training data, the use of the synthetic data can protect the privacy and security of the entities and/or individuals whose activities are recorded by the actual data.
Process 200 can then proceed to step 201 . In step 201 , interface 113 can provide a data model generation request to model optimizer 107 . The data model generation request can include data and/or instructions describing the type of data model to be generated. For example, the data model generation request can specify a general type of data model (e.g., neural network, recurrent neural network, generative adversarial network, kernel density estimator, random data generator, or the like) and parameters specific to the particular type of model (e.g., the number of features and number of layers in a generative adversarial network or recurrent neural network). In some embodiments, a recurrent neural network can include long short term memory modules (LSTM units), or the like.
Process 200 can then proceed to step 203 . In step 203 , one or more components of system 100 can interoperate to generate a data model. For example, as described in greater detail with regard to FIG. 3 , a data model can be trained using computing resources 101 using data provided by dataset generator 103 . In some aspects, this data can be generated using dataset generator 103 from data stored in database 105 . In various aspects, the data used to train dataset generator 103 can be actual or synthetic data retrieved from database 105 . This training can be supervised by model optimizer 107 , which can be configured to select model parameters (e.g., number of layers for a neural network, kernel function for a kernel density estimator, or the like), update training parameters, and evaluate model characteristics (e.g., the similarity of the synthetic data generated by the model to the actual data). In some embodiments, model optimizer 107 can be configured to provision computing resources 101 with an initialized data model for training. The initialized data model can be, or can be based upon, a model retrieved from model storage 109 .
Process 200 can then proceed to step 205 . In step 205 , model optimizer 107 can evaluate the performance of the trained synthetic data model. When the performance of the trained synthetic data model satisfies performance criteria, model optimizer 107 can be configured to store the trained synthetic data model in model storage 109 . For example, model optimizer 107 can be configured to determine one or more values for similarity and/or predictive accuracy metrics, as described herein. In some embodiments, based on values for similarity metrics, model optimizer 107 can be configured to assign a category to the synthetic data model.
According to a first category, the synthetic data model generates data maintaining a moderate level of correlation or similarity with the original data, matches well with the original schema, and does not generate too many row or value duplicates. According to a second category, the synthetic data model may generate data maintaining a high level of correlation or similarity of the original level, and therefore could potentially cause the original data to be discernable from the original data (e.g., a data leak). A synthetic data model generating data failing to match the schema with the original data or providing many duplicated rows and values may also be placed in this category. According to a third category, the synthetic data model may likely generate data maintaining a high level of correlation or similarity with the original data, likely allowing a data leak. A synthetic data model generating data badly failing to match the schema with the original data or providing far too many duplicated rows and values may also be placed in this category.
In some embodiments, system 100 can be configured to provide instructions for improving the quality of the synthetic data model. If a user requires synthetic data reflecting less correlation or similarity with the original data, the use can change the models' parameters to make them perform worse (e.g., by decreasing number of layers in GAN models, or reducing the number of training iterations). If the users want the synthetic data to have better quality, they can change the models' parameters to make them perform better (e.g., by increasing number of layers in GAN models, or increasing the number of training iterations).
Process 200 can then proceed to step 207 , in step 207 , model curator 111 can evaluate the trained synthetic data model for compliance with governance criteria.
FIG. 3 depicts a process 300 for generating a data model using an existing synthetic data model, consistent with disclosed embodiments. Process 300 can include the steps of retrieving a synthetic dataset model from model storage 109 , retrieving data from database 105 , providing synthetic data to computing resources 101 , providing an initialized data model to computing resources 101 , and providing a trained data model to model optimizer 107 . In this manner, process 300 can allow system 100 to generate a model using synthetic data.
Process 300 can then proceed to step 301 . In step 301 , dataset generator 103 can retrieve a training dataset from database 105 . The training dataset can include actual training data, in some aspects. The training dataset can include synthetic training data, in some aspects. In some embodiments, dataset generator 103 can be configured to generate synthetic data from sample values. For example, dataset generator 103 can be configured to use the generative network of a generative adversarial network to generate data samples from random-valued vectors. In such embodiments, process 300 may forgo step 301 .
Process 300 can then proceed to step 303 . In step 303 , dataset generator 103 can be configured to receive a synthetic data model from model storage 109 . In some embodiments, model storage 109 can be configured to provide the synthetic data model to dataset generator 103 in response to a request from dataset generator 103 . In various embodiments, model storage 109 can be configured to provide the synthetic data model to dataset generator 103 in response to a request from model optimizer 107 , or another component of system 100 . As a non-limiting example, the synthetic data model can be a neural network, recurrent neural network (which may include LSTM units), generative adversarial network, kernel density estimator, random value generator, or the like.
Process 300 can then proceed to step 305 . In step 305 , in some embodiments, dataset generator 103 can generate synthetic data. Dataset generator 103 can be configured, in some embodiments, to identify sensitive data items (e.g., account numbers, social security numbers, names, addresses, API keys, network or IP addresses, or the like) in the data received from model storage 109 . In some embodiments, dataset generator 103 can be configured to identify sensitive data items using a recurrent neural network. Dataset generator 103 can be configured to use the data model retrieved from model storage 109 to generate a synthetic dataset by replacing the sensitive data items with synthetic data items.
Dataset generator 103 can be configured to provide the synthetic dataset to computing resources 101 . In some embodiments, dataset generator 103 can be configured to provide the synthetic dataset to computing resources 101 in response to a request from computing resources 101 , model optimizer 107 , or another component of system 100 . In various embodiments, dataset generator 103 can be configured to provide the synthetic dataset to database 105 for storage. In such embodiments, computing resources 101 can be configured to subsequently retrieve the synthetic dataset from database 105 directly, or indirectly through model optimizer 107 or dataset generator 103 .
Process 300 can then proceed to
CLAIMS
Claims ( 19 )
What is claimed is:
1. A system for transforming an input model into a neural network model, the system comprising:
one or more memory units for storing instructions; and
one or more processors configured to execute the instructions to perform operations comprising:
receiving input data comprising the input model, an input dataset, and an input command specifying one or more model selection criteria, the input model comprising a regression model having input model features based on a set of regression coefficients, the input dataset comprising synthetic data, and the synthetic data being a representation of original data;
generating a development instance;
sending a command to the development instance to run the input model;
generating dataset model output by applying the input model to the input dataset;
storing the generated dataset model output and at least one of:
the input model features; or
a map of the input model features;
sending a first command to the development instance to generate a plurality of candidate neural network models;
in response to the first command, generating the plurality of candidate neural network models comprising parameters based on the input model features, the candidate neural network model generating comprising sending a command to the development instance to generate the candidate neural network models;
training the candidate neural network models;
sending a second command to the development instance to tune the candidate neural network models;
in response to the second command, tuning the candidate neural network models by adjusting at least one of a plurality of hidden layers, a plurality of inputs, or a type of layer during training such that the input model features are reproduced, the tuning comprising sending a command to the development instance to tune the candidate neural network models;
receiving, from the development instance, candidate model output from the candidate neural network models;
selecting a neural network model from the candidate neural network models based on the candidate model output and the one or more model selection criteria; and
returning the selected neural network model.
2. The system of claim 1 , wherein the command further specifies one or more model parameters, and generating the plurality of neural network models is based on the model parameters.
3. The system of claim 1 , wherein the input model output comprises a modeling result and a log.
4. The system of claim 1 , wherein:
the input command specifies a number of candidate neural network models of a type of neural network model, and
generating the candidate neural network models comprises generating the specified number of candidate neural network models of the specified type.
5. The system of claim 1 , wherein generating the candidate neural network models comprises retrieving a candidate neural network model from a model storage.
6. The system of claim 1 , wherein generating the candidate neural network models comprises spinning up a development instance for each candidate neural network model.
7. The system of claim 1 , wherein generating the candidate neural network models comprises overfitting each of the candidate neural network models to the input model.
8. The system of claim 1 , wherein the parameters of the candidate neural network models comprise at least one of a number of hidden layers, a number of nodes, or a dropout rate.
9. The system of claim 1 , wherein:
tuning the candidate neural network models comprises training the candidate neural network models; and
training terminates when one or more training conditions are satisfied.
10. The system of claim 9 , wherein the one or more training conditions comprise at least one of a run time, a number of epochs, or an accuracy score.
11. The system of claim 1 , wherein:
the generated model output comprises a plurality of accuracy scores corresponding to the input model features; and
selecting a neural network comprises selecting a neural network based on the accuracy scores.
12. The system of claim 1 , wherein:
the respective candidate model output comprises a model run time; and
selecting a neural network comprises selecting a neural network based on the model run time.
13. The system of claim 1 , wherein:
generating the candidate neural network models comprises generating a plurality of development instances corresponding to the candidate neural network models;
tuning the plurality of neural network models comprises sending a command to the plurality of development instances to tune the candidate neural network models; and
receiving the candidate model output comprises receiving the candidate model output from the corresponding development instances.
14. The system of claim 1 , wherein tuning the plurality of neural network models comprises adjusting at least one of a number of hidden layers, a number of inputs, or a type of layer.
15. The system of claim 1 , wherein tuning the neural network models comprises:
selecting first model training parameters for one of the candidate neural network models;
generating first candidate model output based on the one candidate neural network model and the first model training parameters;
selecting second model training parameters for the one candidate neural network model; and
generating second candidate model output based on the one candidate neural network model and the second model training parameters.
16. A method for transforming an input model into a neural network model, the method comprising:
receiving input data comprising the input model, an input dataset, and an input command specifying one or more model selection criteria, the input model comprising a regression model having input model features based on a set of regression coefficients, the input dataset comprising synthetic data, and the synthetic data being a representation of original data;
generating a development instance;
sending a command to the development instance to run the input model;
generating dataset model output by applying the input model to the input dataset to generate model output;
storing the generated dataset model output and at least one of:
the input model features; or
a map of the input model features;
generating a candidate neural network models comprising parameters based on the input model features, the candidate neural network generating comprising sending a command to the development instance to generate the candidate neural network models;
training the candidate neural network models;
tuning the candidate neural network models to by adjusting at least one of a plurality of hidden layers, a plurality of inputs, or a type of layer during training, thereby reproducing the input model features, the tuning comprising sending a command to the development instance to tune the candidate neural network models;
receiving, from the development instance, candidate model output from the candidate neural network models;
selecting a neural network model from the plurality of the candidate neural network models based on the candidate model output and the one or more model selection criteria; and
returning the selected neural network model.
17. A system for transforming an input model into a neural network model, the system comprising:
one or more memory units for storing instructions; and
one or more processors configured to execute the instructions to perform operations comprising:
receiving input data comprising the input model, an input dataset, and an input command specifying model selection criteria, the input model comprising a regression model having input model features based on a set of regression coefficients, the input dataset comprising synthetic data, and the synthetic data being a representation of original data;
generating a plurality of development instances; and
sending instructions to each of the plurality of development instances to perform operations comprising:
identifying a candidate model from a model storage based on the model selection criteria and at least one of the input model features, the candidate model being a neural network model;
training the candidate model;
tuning the candidate model by adjusting at least one of a number of hidden layers, a number of inputs, or a type of layer during training, such that the input model features are reproduced;
generating candidate model parameters based on the training;
generating a candidate modeling result; and
returning the candidate model, the candidate model parameters, and the candidate modeling result;
receiving a plurality of candidate models, candidate model parameters, and candidate modeling results corresponding to the development instances;
selecting a candidate model from among the plurality of candidate models based on the candidate modeling results and the model selection criterion; and
returning the selected candidate model.
18. The system of claim 17 , wherein training the candidate model comprises iteratively performing operations comprising:
selecting a first model training parameter for the candidate model;
generating first candidate model output based on the first model training parameters and the input dataset;
determine whether the first candidate model output satisfies a training condition; and
based on the determination, perform one of:
selecting second model training parameters and generating second candidate model output based on the second model training parameters; or
terminating model training and returning the candidate model and the candidate model output.
19. The system of claim 1 , wherein the synthetic data comprises a desensitized dataset and the original data comprises a sensitive dataset.
US16/172,480
2018-07-06
2018-10-26
Systems and methods to use neural networks to transform a model into a neural network model
Active
US11126475B2
( en )
Priority Applications (2)
Application Number
Priority Date
Filing Date
Title
US16/172,480
US11126475B2
( en )
2018-07-06
2018-10-26
Systems and methods to use neural networks to transform a model into a neural network model
US17/464,796
US20220092419A1
( en )
2018-07-06
2021-09-02
Systems and methods to use neural networks for model transformations
Applications Claiming Priority (2)
Application Number
Priority Date
Filing Date
Title
US201862694968P
2018-07-06
2018-07-06
US16/172,480
US11126475B2
( en )
2018-07-06
2018-10-26
Systems and methods to use neural networks to transform a model into a neural network model
Related Child Applications (1)
Application Number
Title
Priority Date
Filing Date
US17/464,796
Continuation
US20220092419A1
( en )
2018-07-06
2021-09-02
Systems and methods to use neural networks for model transformations
Publications (2)
Publication Number
Publication Date
US20200012917A1
US20200012917A1 ( en )
2020-01-09
US11126475B2
true
US11126475B2 ( en )
2021-09-21
Family
ID=67543579
Family Applications (62)
Application Number
Title
Priority Date
Filing Date
US16/151,431
Active
US12455778B2
( en )
2018-07-06
2018-10-04
Systems and methods for data stream simulation
US16/152,072
Active
US10635939B2
( en )
2018-07-06
2018-10-04
System, method, and computer-accessible medium for evaluating multi-dimensional synthetic data using integrated variants analysis
US16/151,385
Active
US10460235B1
( en )
2018-07-06
2018-10-04
Data model generation using generative adversarial networks
US16/151,407
Active
2039-01-29
US11615208B2
( en )
2018-07-06
2018-10-04
Systems and methods for synthetic data generation
US16/172,344
Active
US10599957B2
( en )
2018-07-06
2018-10-26
Systems and methods for detecting data drift for data used in machine learning models
US16/172,480
Active
US11126475B2
( en )
2018-07-06
2018-10-26
Systems and methods to use neural networks to transform a model into a neural network model
US16/172,430
Active
2038-12-06
US11210144B2
( en )
2018-07-06
2018-10-26
Systems and methods for hyperparameter tuning
US16/172,223
Active
2038-12-12
US11256555B2
( en )
2018-07-06
2018-10-26
Automatically scalable system for serverless hyperparameter tuning
US16/172,508
Active
US10592386B2
( en )
2018-07-06
2018-10-26
Fully automated machine learning system which generates and optimizes solutions given a dataset and a desired outcome
US16/173,374
Active
US10382799B1
( en )
2018-07-06
2018-10-29
Real-time synthetically generated video from still frames
US16/181,568
Active
2039-01-28
US11385942B2
( en )
2018-07-06
2018-11-06
Systems and methods for censoring text inline
US16/181,673
Active
2039-01-08
US10983841B2
( en )
2018-07-06
2018-11-06
Systems and methods for removing identifiable information
US16/251,867
Active
US10459954B1
( en )
2018-07-06
2019-01-18
Dataset connector and crawler to identify data lineage and segment data
US16/263,141
Active
US10521719B1
( en )
2018-07-06
2019-01-31
Systems and methods to identify neural network brittleness based on sample data and seed generation
US16/263,839
Active
US10482607B1
( en )
2018-07-06
2019-01-31
Systems and methods for motion correction in synthetic images
US16/298,463
Active
US11513869B2
( en )
2018-07-06
2019-03-11
Systems and methods for synthetic database query generation
US16/362,568
Active
US10452455B1
( en )
2018-07-06
2019-03-22
Systems and methods to manage application program interface communications
US16/362,537
Active
US10860460B2
( en )
2018-07-06
2019-03-22
Automated honeypot creation within a network
US16/362,466
Active
US10379995B1
( en )
2018-07-06
2019-03-22
Systems and methods to identify breaking application program interface changes
US16/405,989
Active
US10884894B2
( en )
2018-07-06
2019-05-07
Systems and methods for synthetic data generation for time-series data using data segments
US16/409,745
Active
US11113124B2
( en )
2018-07-06
2019-05-10
Systems and methods for quickly searching datasets by indexing synthetic data generating models
US16/454,041
Active
US10664381B2
( en )
2018-07-06
2019-06-26
Method and system for synthetic generation of time series data
US16/457,670
Active
US11032585B2
( en )
2018-07-06
2019-06-28
Real-time synthetically generated video from still frames
US16/457,548
Active
US10599550B2
( en )
2018-07-06
2019-06-28
Systems and methods to identify breaking application program interface changes
US16/503,428
Active
US10671884B2
( en )
2018-07-06
2019-07-03
Systems and methods to improve data clustering using a meta-clustering model
US16/565,565
Active
US11210145B2
( en )
2018-07-06
2019-09-10
Systems and methods to manage application program interface communications
US16/577,010
Active
US11182223B2
( en )
2018-07-06
2019-09-20
Dataset connector and crawler to identify data lineage and segment data
US16/658,858
Active
US10896072B2
( en )
2018-07-06
2019-10-21
Systems and methods for motion correction in synthetic images
US16/666,316
Active
2040-11-16
US11704169B2
( en )
2018-07-06
2019-10-28
Data model generation using generative adversarial networks
US16/715,924
Active
2040-12-04
US11836537B2
( en )
2018-07-06
2019-12-16
Systems and methods to identify neural network brittleness based on sample data and seed generation
US16/748,917
Active
US10970137B2
( en )
2018-07-06
2020-01-22
Systems and methods to identify breaking application program interface changes
US16/825,040
Active
2039-02-17
US11385943B2
( en )
2018-07-06
2020-03-20
System, method, and computer-accessible medium for evaluating multi-dimensional synthetic data using integrated variants analysis
US16/852,795
Active
US12093753B2
( en )
2018-07-06
2020-04-20
Method and system for synthetic generation of time series data
US16/889,363
Active
2039-11-07
US11604896B2
( en )
2018-07-06
2020-06-01
Systems and methods to improve data clustering using a meta-clustering model
US17/084,203
Active
US11237884B2
( en )
2018-07-06
2020-10-29
Automated honeypot creation within a network
US17/102,526
Active
2040-03-02
US11822975B2
( en )
2018-07-06
2020-11-24
Systems and methods for synthetic data generation for time-series data using data segments
US17/139,203
Active
2039-06-22
US11687382B2
( en )
2018-07-06
2020-12-31
Systems and methods for motion correction in synthetic images
US17/189,193
Active
US11372694B2
( en )
2018-07-06
2021-03-01
Systems and methods to identify breaking application program interface changes
US17/220,409
Active
2039-03-11
US11574077B2
( en )
2018-07-06
2021-04-01
Systems and methods for removing identifiable information
US17/307,361
Active
US11687384B2
( en )
2018-07-06
2021-05-04
Real-time synthetically generated video from still frames
US17/395,899
Active
2040-06-06
US12210917B2
( en )
2018-07-06
2021-08-06
Systems and methods for quickly searching datasets by indexing synthetic data generating models
US17/464,796
Pending
US20220092419A1
( en )
2018-07-06
2021-09-02
Systems and methods to use neural networks for model transformations
US17/505,840
Active
2039-02-19
US11989597B2
( en )
2018-07-06
2021-10-20
Dataset connector and crawler to identify data lineage and segment data
US17/526,073
Pending
US20220075670A1
( en )
2018-07-06
2021-11-15
Systems and methods for replacing sensitive data
US17/553,023
Active
US11580261B2
( en )
2018-07-06
2021-12-16
Automated honeypot creation within a network
US17/585,698
Pending
US20220147405A1
( en )
2018-07-06
2022-01-27
Automatically scalable system for serverless hyperparameter tuning
US17/836,614
Active
2039-05-06
US12379975B2
( en )
2018-07-06
2022-06-09
Systems and methods for censoring text inline
US17/845,786
Active
US11900178B2
( en )
2018-07-06
2022-06-21
System, method, and computer-accessible medium for evaluating multi-dimensional synthetic data using integrated variants analysis
US18/050,694
Active
US12405844B2
( en )
2018-07-06
2022-10-28
Systems and methods for synthetic database query generation
US18/091,638
Active
US12271768B2
( en )
2018-07-06
2022-12-30
Systems and methods for removing identifiable information
US18/155,529
Active
2039-07-05
US11861418B2
( en )
2018-07-06
2023-01-17
Systems and methods to improve data clustering using a meta-clustering model
US18/165,725
Pending
US20230195541A1
( en )
2018-07-06
2023-02-07
Systems and methods for synthetic data generation
US18/312,481
Pending
US20230273841A1
( en )
2018-07-06
2023-05-04
Real-time synthetically generated video from still frames
US18/316,868
Active
US12056541B2
( en )
2018-07-06
2023-05-12
Systems and methods for motion correction in synthetic images
US18/321,370
Pending
US20230297446A1
( en )
2018-07-06
2023-05-22
Data model generation using generative adversarial networks
US18/360,482
Active
US12379977B2
( en )
2018-07-06
2023-07-27
Systems and methods for synthetic data generation for time-series data using data segments
US18/383,946
Active
US12277455B2
( en )
2018-07-06
2023-10-26
Systems and methods to identify neural network brittleness based on sample data and seed generation
US18/402,937
Active
US12175308B2
( en )
2018-07-06
2024-01-03
System, method, and computer-accessible medium for evaluating multi-dimensional synthetic data using integrated variants analysis
US18/763,367
Pending
US20240411619A1
( en )
2018-07-06
2024-07-03
Systems and methods for motion correction in synthetic images
US19/083,376
Pending
US20250245076A1
( en )
2018-07-06
2025-03-18
Systems and methods for removing identifiable information
US19/174,894
Pending
US20250238288A1
( en )
2018-07-06
2025-04-09
Systems and methods to identify neural network brittleness based on sample data and seed generation
US19/290,260
Pending
US20250362980A1
( en )
2018-07-06
2025-08-04
Systems and methods for censoring text inline
Family Applications Before (5)
Application Number
Title
Priority Date
Filing Date
US16/151,431
Active
US12455778B2
( en )
2018-07-06
2018-10-04
Systems and methods for data stream simulation
US16/152,072
Active
US10635939B2
( en )
2018-07-06
2018-10-04
System, method, and computer-accessible medium for evaluating multi-dimensional synthetic data using integrated variants analysis
US16/151,385
Active
US10460235B1
( en )
2018-07-06
2018-10-04
Data model generation using generative adversarial networks
US16/151,407
Active
2039-01-29
US11615208B2
( en )
2018-07-06
2018-10-04
Systems and methods for synthetic data generation
US16/172,344
Active
US10599957B2
( en )
2018-07-06
2018-10-26
Systems and methods for detecting data drift for data used in machine learning models
Family Applications After (56)
Application Number
Title
Priority Date
Filing Date
US16/172,430
Active
2038-12-06
US11210144B2
( en )
2018-07-06
2018-10-26
Systems and methods for hyperparameter tuning
US16/172,223
Active
2038-12-12
US11256555B2
( en )
2018-07-06
2018-10-26
Automatically scalable system for serverless hyperparameter tuning
US16/172,508
Active
US10592386B2
( en )
2018-07-06
2018-10-26
Fully automated machine learning system which generates and optimizes solutions given a dataset and a desired outcome
US16/173,374
Active
US10382799B1
( en )
2018-07-06
2018-10-29
Real-time synthetically generated video from still frames
US16/181,568
Active
2039-01-28
US11385942B2
( en )
2018-07-06
2018-11-06
Systems and methods for censoring text inline
US16/181,673
Active
2039-01-08
US10983841B2
( en )
2018-07-06
2018-11-06
Systems and methods for removing identifiable information
US16/251,867
Active
US10459954B1
( en )
2018-07-06
2019-01-18
Dataset connector and crawler to identify data lineage and segment data
US16/263,141
Active
US10521719B1
( en )
2018-07-06
2019-01-31
Systems and methods to identify neural network brittleness based on sample data and seed generation
US16/263,839
Active
US10482607B1
( en )
2018-07-06
2019-01-31
Systems and methods for motion correction in synthetic images
US16/298,463
Active
US11513869B2
( en )
2018-07-06
2019-03-11
Systems and methods for synthetic database query generation
US16/362,568
Active
US10452455B1
( en )
2018-07-06
2019-03-22
Systems and methods to manage application program interface communications
US16/362,537
Active
US10860460B2
( en )
2018-07-06
2019-03-22
Automated honeypot creation within a network
US16/362,466
Active
US10379995B1
( en )
2018-07-06
2019-03-22
Systems and methods to identify breaking application program interface changes
US16/405,989
Active
US10884894B2
( en )
2018-07-06
2019-05-07
Systems and methods for synthetic data generation for time-series data using data segments
US16/409,745
Active
US11113124B2
( en )
2018-07-06
2019-05-10
Systems and methods for quickly searching datasets by indexing synthetic data generating models
US16/454,041
Active
US10664381B2
( en )
2018-07-06
2019-06-26
Method and system for synthetic generation of time series data
US16/457,670
Active
US11032585B2
( en )
2018-07-06
2019-06-28
Real-time synthetically generated video from still frames
US16/457,548
Active
US10599550B2
( en )
2018-07-06
2019-06-28
Systems and methods to identify breaking application program interface changes
US16/503,428
Active
US10671884B2
( en )
2018-07-06
2019-07-03
Systems and methods to improve data clustering using a meta-clustering model
US16/565,565
Active
US11210145B2
( en )
2018-07-06
2019-09-10
Systems and methods to manage application program interface communications
US16/577,010
Active
US11182223B2
( en )
2018-07-06
2019-09-20
Dataset connector and crawler to identify data lineage and segment data
US16/658,858
Active
US10896072B2
( en )
2018-07-06
2019-10-21
Systems and methods for motion correction in synthetic images
US16/666,316
Active
2040-11-16
US11704169B2
( en )
2018-07-06
2019-10-28
Data model generation using generative adversarial networks
US16/715,924
Active
2040-12-04
US11836537B2
( en )
2018-07-06
2019-12-16
Systems and methods to identify neural network brittleness based on sample data and seed generation
US16/748,917
Active
US10970137B2
( en )
2018-07-06
2020-01-22
Systems and methods to identify breaking application program interface changes
US16/825,040
Active
2039-02-17
US11385943B2
( en )
2018-07-06
2020-03-20
System, method, and computer-accessible medium for evaluating multi-dimensional synthetic data using integrated variants analysis
US16/852,795
Active
US12093753B2
( en )
2018-07-06
2020-04-20
Method and system for synthetic generation of time series data
US16/889,363
Active
2039-11-07
US11604896B2
( en )
2018-07-06
2020-06-01
Systems and methods to improve data clustering using a meta-clustering model
US17/084,203
Active
US11237884B2
( en )
2018-07-06
2020-10-29
Automated honeypot creation within a network
US17/102,526
Active
2040-03-02
US11822975B2
( en )
2018-07-06
2020-11-24
Systems and methods for synthetic data generation for time-series data using data segments
US17/139,203
Active
2039-06-22
US11687382B2
( en )
2018-07-06
2020-12-31
Systems and methods for motion correction in synthetic images
US17/189,193
Active
US11372694B2
( en )
2018-07-06
2021-03-01
Systems and methods to identify breaking application program interface changes
US17/220,409
Active
2039-03-11
US11574077B2
( en )
2018-07-06
2021-04-01
Systems and methods for removing identifiable information
US17/307,361
Active
US11687384B2
( en )
2018-07-06
2021-05-04
Real-time synthetically generated video from still frames
US17/395,899
Active
2040-06-06
US12210917B2
( en )
2018-07-06
2021-08-06
Systems and methods for quickly searching datasets by indexing synthetic data generating models
US17/464,796
Pending
US20220092419A1
( en )
2018-07-06
2021-09-02
Systems and methods to use neural networks for model transformations
US17/505,840
Active
2039-02-19
US11989597B2
( en )
2018-07-06
2021-10-20
Dataset connector and crawler to identify data lineage and segment data
US17/526,073
Pending
US20220075670A1
( en )
2018-07-06
2021-11-15
Systems and methods for replacing sensitive data
US17/553,023
Active
US11580261B2
( en )
2018-07-06
2021-12-16
Automated honeypot creation within a network
US17/585,698
Pending
US20220147405A1
( en )
2018-07-06
2022-01-27
Automatically scalable system for serverless hyperparameter tuning
US17/836,614
Active
2039-05-06
US12379975B2
( en )
2018-07-06
2022-06-09
Systems and methods for censoring text inline
US17/845,786
Active
US11900178B2
( en )
2018-07-06
2022-06-21
System, method, and computer-accessible medium for evaluating multi-dimensional synthetic data using integrated variants analysis
US18/050,694
Active
US12405844B2
( en )
2018-07-06
2022-10-28
Systems and methods for synthetic database query generation
US18/091,638
Active
US12271768B2
( en )
2018-07-06
2022-12-30
Systems and methods for removing identifiable information
US18/155,529
Active
2039-07-05
US11861418B2
( en )
2018-07-06
2023-01-17
Systems and methods to improve data clustering using a meta-clustering model
US18/165,725
Pending
US20230195541A1
( en )
2018-07-06
2023-02-07
Systems and methods for synthetic data generation
US18/312,481
Pending
US20230273841A1
( en )
2018-07-06
2023-05-04
Real-time synthetically generated video from still frames
US18/316,868
Active
US12056541B2
( en )
2018-07-06
2023-05-12
Systems and methods for motion correction in synthetic images
US18/321,370
Pending
US20230297446A1
( en )
2018-07-06
2023-05-22
Data model generation using generative adversarial networks
US18/360,482
Active