Interactive digital dashboards for trained machine learning or artificial … — The Toronto-Dominion Bank (US20220188705A1)

ABSTRACT

Abstract

The disclosed embodiments include computer-implemented processes that generate and maintain interactive digital dashboards for machine learning or artificial intelligence processes. For example, an apparatus may obtain process data associated with an execution of a plurality of machine learning or artificial intelligence processes. Based on the process data, the apparatus may determine, for each of the plurality of machine learning or artificial intelligence processes, value of one or more metrics characterizing a status of one or more operations that support the execution of the corresponding machine learning or artificial intelligence process. Further, the apparatus may transmit status data that includes the one or more metric values and corresponding process identifiers to a device, which presents, for each of the machine learning or artificial intelligence processes, a graphical representation of at least one of the determined one or more metric values within a digital interface.

Description

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of priority under 35 U.S.C. § 119(e) to prior U.S. Provisional Application No. 63/126,392, filed Dec. 16, 2020, the disclosure of which is incorporated by reference herein to its entirety.

TECHNICAL FIELD

The disclosed embodiments generally relate to computer-implemented processes for generate and maintain interactive digital dashboards for machine learning or artificial intelligence processes.

BACKGROUND

Machine learning and artificial intelligence processes are widely adopted throughout the financial services industry, and an output of these machine learning and artificial intelligence process may inform decisions associated with a targeted marketing of products and services to existing and prospective customers, a credit risk associated with these existing and prospective customers, or a suspiciousness of certain activities performed by these existing and prospective customers. By way of example, a financial institution may include multiple business units, and each of the business units may adaptively train, and then deploy, one or more trained artificial intelligence or machine learning processes that generate elements of output data consistent with the needs of that business unit and on a schedule appropriate to those needs.

SUMMARY

In some examples, an apparatus includes a communications interface, a memory storing instructions, and at least one processor coupled to the communications interface and to the memory. The at least one processor is configured to execute the instructions to obtain process data associated with an execution of a plurality of machine learning or artificial intelligence processes. The at least one processor is further configured to execute the instructions to, based on the process data, determine, for each of the plurality of machine learning or artificial intelligence processes, value of one or more metrics characterizing a status of one or more operations that support the execution of the corresponding machine learning or artificial intelligence process. The at least one processor is further configured to generate status data for each of the machine learning or artificial intelligence processes, and transmit one or more elements of the status data to a device via the communications interface. The status data includes, for each of the machine learning or artificial intelligence processes, the determined one or more metric values and a corresponding process identifier, and the status data causes the device to present, for each of the machine learning or artificial intelligence processes, a graphical representation of at least one of the determined one or more metric values within a digital interface.

In other examples, a computer-implemented method includes obtaining, using at least one processor, process data associated with an execution of a plurality of machine learning or artificial intelligence processes. The computer-implemented method also includes, based on the process data, determining, using the at least one processor, and for each of the machine learning or artificial intelligence processes, a value of one or more metrics characterizing a status of one or more operations that support the execution of the corresponding machine learning or artificial intelligence process. The computer-implemented method includes using the at least one processor, generating status data for each of the plurality of machine learning or artificial intelligence processes, and transmitting one or more elements of the status data to a device. The status data includes, for each of the machine learning or artificial intelligence processes, the determined one or more metric values and a corresponding process identifier, and the status data causes the device to present, for each of the machine learning or artificial intelligence processes, a graphical representation of at least one of the determined one or more metric values within a digital interface.

Additionally, in some examples, a device includes a communications interface, a display unit, an input unit, a memory storing instructions and at least one processor coupled to the communications interface, the display unit, the input unit, and the memory. The at least one processor is configured to execute the instructions to receive, via the communications interface, status data associated with an execution of a plurality of machine learning processes. The status data includes, for each of the plurality of machine learning processes, a process identifier and a value of one or more metrics characterizing a status of one or more operations that support the execution of the corresponding machine learning process. The at least one processor is further configured to execute the instructions to, based on the status data, generate and present, via the display unit, and for each of the plurality of machine learning processes, a first graphical representation of a first subset of the one or more metric values within a portion of a digital interface. The at least one processor is further configured to execute the instructions to receive, via the input unit, input data indicative of a selection of one or the first graphical representation associated with a corresponding one of the plurality of machine learning processes. The at least one processor is further configured to, based on the input data, generate and present, via the display unit, a second graphical representation of a second subset of the one or more metric values associated with the corresponding machine learning process within an additional portion of the digital interface.

The details of one or more exemplary embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other potential features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1, 2A, 2B, and 3 are block diagrams illustrating portions of an exemplary computing environment, in accordance with some exemplary embodiments.

FIGS. 4A, 4B, and 4C are diagrams illustrating portions of an exemplary, interactive digital dashboard, in accordance with some exemplary embodiments; and

FIG. 5 is a flowchart of an exemplary process for generating and maintaining an interactive digital dashboard for trained machine learning and artificial intelligence processes, in accordance with some embodiments

Like reference numbers and designations in the various drawings indicate like elements.

DETAILED DESCRIPTION

Today, many organizations, such as financial institutions, rely on a predicted output of machine learning and artificial intelligence processes to inform a variety of decisions and strategies. By way of example, a financial institution may obtain elements of predicted, customer-specific output generated through an application of one or more trained machine learning or artificial intelligence processes, and the obtained elements of predicted, customer-specific output may inform, among other things, customer-specific strategies for mitigating or managing risk, decisions related to a suspiciousness of certain activities performed by these existing and prospective customers, or collection strategies involving one or more existing customers of the financial institution. Further, in some examples, the financial institution may also rely on the elements of customer-specific predicted output to inform decisions associated with the provisioning of financial products or services to existing or prospective customers of the financial institution, decisions associated with a requested modification to a term of condition of a provisioned financial product or service, or decisions associated with a targeted marketing of products and services to existing and prospective customers.

Although the machine learning and artificial intelligence processes may be trained adaptively to predicted future occurrences of events involving customers of the financial institution during future temporal intervals, each of the trained machine learning and artificial intelligence processes may be associated with, and may predict an occurrence of a corresponding, distinct, target event during a corresponding, and distinct future temporal interval. By way of example, and to inform a customer-specific decision regarding a requested modification to a term or condition of a provisioned financial product (e.g., an unsecured credit product, etc.), a machine learning and artificial intelligence process may be trained to predict an occurrence of a particular default event of a predetermined duration during a future, twelve-month interval. In other examples, to inform a customer-specific decision regarding a targeted marketing of financial products and services, an additional machine learning and artificial intelligence process may be trained to predict an occurrence of a particular engagement event involving a customer and a corresponding financial product or service during an additional, or alternate, future temporal interval, such as a three-month interval.

In some instances, each of the trained machine learning and artificial intelligence processes may be associated with a corresponding, process-specific input datasets, each of which may include a plurality of sequentially ordered, process-specific input feature and corresponding, customer-specific feature values, and with corresponding process-specific schedules for a delivery of corresponding elements of predicted output to a computing system associated with corresponding business units of the financial institution. Further, each of the trained machine learning and artificial intelligence processes may also be associated with a plurality of sequentially implemented data-pipelining operations that, when implemented by one or more computing systems associated with, or operated by, the financial institution, facilitate a generation and ingestion of corresponding process-specific input datasets, and generation, validation, or transaction of the elements of predictive output, in accordance with the underlying, process-specific delivery schedule for the corresponding elements of the predicted output (for example, at an expected delivery time on a daily basis, a weekly basis, a bi-monthly basis, or on a monthly basis).

To facilitate a delivery of the process-specific elements of the predicted output in accordance with corresponding ones of the process-specific delivery schedules, the one or more computing systems associated with, or operated by, the financial institution may collectively execute one, or more, of the process-specific data pipelining operations described herein on a simultaneous, or overlapping basis, which may facilitate a simultaneous, or overlapping, execution of one or more of the trained machine learning or artificial intelligence processes. In some instances, one or more of the executed process-specific data pipelining operations may experience a failure or a delay (e.g., due to absence of elements of customer-specific data associated with a corresponding input dataset, due to a failure in a communications network interconnecting one or more distributed computing components, etc.), and the one or more computing systems of the financial institution may detect the failure or delay, and generate and transmit a notification to a device operable by a representative of the financial institution, such as an analyst device operable by an analyst. Although the analyst device may generate and provision programmatic instructions (e.g., based on corresponding input from the analyst) that the one or more computing systems of the financial institution to reinitiate the failed or delayed, process-specific data pipelining operation, the re-initiation of the process-specific data pipelining operation may result in a temporal delay in the delivery of the corresponding, customer-specific elements of the predicted output to the computing system of the corresponding business unit.

In some examples, absent a real-time detection of the failure or a delay in the process-specific data pipelining operation, and a real-time provisioning of a notification of the detected failure or delay to an analyst device, the resulting temporal delay may reduce any ability of the one or more computing systems of the financial institution to re-initiate failed or delayed data pipelining operation based on programmatic instructions received from the analyst device and while delivering the customer-specific elements of the predicted output to the computing system of the corresponding business unit in accordance with the delivery schedule and at, or before, an agreed-upon delivery time. Further, although notification processes may enable the one or more computing systems of the financial institution (or an application program, code engine or module, or other elements of code executable by the one or more computing systems) to perform operations that generate and transmit notifications (e.g., email messages, text messages, etc.) to a computing device or system of a predetermined set of recipients (including the analyst described herein), these notification processes often cause the computing systems or devices to presents these received notification sequentially within a digital interface (e.g., a digital interface of an email or messaging application, as banners on a home screen, etc.).

Certain of these existing notification processes may fail to provide any centralized platform that enables the analyst to visualize a current status of each of the multiple, executed trained machine learning or artificial intelligence processes, and corresponding ones of the process-specific data pipelining operations, much less that enables the analyst to determine an impact of a particular delay or failure (e.g., associated with a corresponding one of the executed, machine learning or artificial intelligence processes and the corresponding business unit) on the delivery schedule associated with the corresponding, process-specific elements of predicted output, or on delivery schedules associated with other simultaneously or contemporaneously executed machine learning or artificial intelligence processes. Further, the absence of a centralized platform may also render impractical any process for debugging a data pipelining or execution of one or more of the executed, machine learning or artificial intelligence processes, and for comparing an actual data-pipelining or execution time associated with one or more of the machine learning or artificial intelligence processes against corresponding “expected” times, and additionally, or alternatively, for identifying or mediating potential “choke points” that trigger a delay, or failure, of the data pipelining associated with or the execution of one or more discrete machine learning or artificial intelligence processes, or classes or types of machine learning or artificial intelligence processes.

In some examples, one or more computing systems associated with the financial institution (or one or more distributed computing components) may perform operations, described herein, to generate, and render for presentation at a computing system or device of an analyst, a digital interface, such as an interactive, digital dashboard, having interface elements that identify each of a machine learning or artificial intelligence processes executed by the one or more computing systems associated with the financial institution. The exemplary processes described herein may enable one or more computing systems associated with the financial institution to provide a graphical representation of not only a current status of each of machine learning or artificial intelligence processes, as well as historical data characterizing a success or failure in the data pipelining operations associated with each of the executed machine learning or artificial intelligence processes over one or more prior temporal intervals. Further, the generated dashboard may, u

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of priority under 35 U.S.C. § 119(e) to prior U.S. Provisional Application No. 63/126,392, filed Dec. 16, 2020, the disclosure of which is incorporated by reference herein to its entirety.

TECHNICAL FIELD

The disclosed embodiments generally relate to computer-implemented processes for generate and maintain interactive digital dashboards for machine learning or artificial intelligence processes.

BACKGROUND

Machine learning and artificial intelligence processes are widely adopted throughout the financial services industry, and an output of these machine learning and artificial intelligence process may inform decisions associated with a targeted marketing of products and services to existing and prospective customers, a credit risk associated with these existing and prospective customers, or a suspiciousness of certain activities performed by these existing and prospective customers. By way of example, a financial institution may include multiple business units, and each of the business units may adaptively train, and then deploy, one or more trained artificial intelligence or machine learning processes that generate elements of output data consistent with the needs of that business unit and on a schedule appropriate to those needs.

SUMMARY

In some examples, an apparatus includes a communications interface, a memory storing instructions, and at least one processor coupled to the communications interface and to the memory. The at least one processor is configured to execute the instructions to obtain process data associated with an execution of a plurality of machine learning or artificial intelligence processes. The at least one processor is further configured to execute the instructions to, based on the process data, determine, for each of the plurality of machine learning or artificial intelligence processes, value of one or more metrics characterizing a status of one or more operations that support the execution of the corresponding machine learning or artificial intelligence process. The at least one processor is further configured to generate status data for each of the machine learning or artificial intelligence processes, and transmit one or more elements of the status data to a device via the communications interface. The status data includes, for each of the machine learning or artificial intelligence processes, the determined one or more metric values and a corresponding process identifier, and the status data causes the device to present, for each of the machine learning or artificial intelligence processes, a graphical representation of at least one of the determined one or more metric values within a digital interface.

In other examples, a computer-implemented method includes obtaining, using at least one processor, process data associated with an execution of a plurality of machine learning or artificial intelligence processes. The computer-implemented method also includes, based on the process data, determining, using the at least one processor, and for each of the machine learning or artificial intelligence processes, a value of one or more metrics characterizing a status of one or more operations that support the execution of the corresponding machine learning or artificial intelligence process. The computer-implemented method includes using the at least one processor, generating status data for each of the plurality of machine learning or artificial intelligence processes, and transmitting one or more elements of the status data to a device. The status data includes, for each of the machine learning or artificial intelligence processes, the determined one or more metric values and a corresponding process identifier, and the status data causes the device to present, for each of the machine learning or artificial intelligence processes, a graphical representation of at least one of the determined one or more metric values within a digital interface.

Additionally, in some examples, a device includes a communications interface, a display unit, an input unit, a memory storing instructions and at least one processor coupled to the communications interface, the display unit, the input unit, and the memory. The at least one processor is configured to execute the instructions to receive, via the communications interface, status data associated with an execution of a plurality of machine learning processes. The status data includes, for each of the plurality of machine learning processes, a process identifier and a value of one or more metrics characterizing a status of one or more operations that support the execution of the corresponding machine learning process. The at least one processor is further configured to execute the instructions to, based on the status data, generate and present, via the display unit, and for each of the plurality of machine learning processes, a first graphical representation of a first subset of the one or more metric values within a portion of a digital interface. The at least one processor is further configured to execute the instructions to receive, via the input unit, input data indicative of a selection of one or the first graphical representation associated with a corresponding one of the plurality of machine learning processes. The at least one processor is further configured to, based on the input data, generate and present, via the display unit, a second graphical representation of a second subset of the one or more metric values associated with the corresponding machine learning process within an additional portion of the digital interface.

The details of one or more exemplary embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other potential features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1, 2A, 2B, and 3 are block diagrams illustrating portions of an exemplary computing environment, in accordance with some exemplary embodiments.

FIGS. 4A, 4B, and 4C are diagrams illustrating portions of an exemplary, interactive digital dashboard, in accordance with some exemplary embodiments; and

FIG. 5 is a flowchart of an exemplary process for generating and maintaining an interactive digital dashboard for trained machine learning and artificial intelligence processes, in accordance with some embodiments

Like reference numbers and designations in the various drawings indicate like elements.

DETAILED DESCRIPTION

Today, many organizations, such as financial institutions, rely on a predicted output of machine learning and artificial intelligence processes to inform a variety of decisions and strategies. By way of example, a financial institution may obtain elements of predicted, customer-specific output generated through an application of one or more trained machine learning or artificial intelligence processes, and the obtained elements of predicted, customer-specific output may inform, among other things, customer-specific strategies for mitigating or managing risk, decisions related to a suspiciousness of certain activities performed by these existing and prospective customers, or collection strategies involving one or more existing customers of the financial institution. Further, in some examples, the financial institution may also rely on the elements of customer-specific predicted output to inform decisions associated with the provisioning of financial products or services to existing or prospective customers of the financial institution, decisions associated with a requested modification to a term of condition of a provisioned financial product or service, or decisions associated with a targeted marketing of products and services to existing and prospective customers.

Although the machine learning and artificial intelligence processes may be trained adaptively to predicted future occurrences of events involving customers of the financial institution during future temporal intervals, each of the trained machine learning and artificial intelligence processes may be associated with, and may predict an occurrence of a corresponding, distinct, target event during a corresponding, and distinct future temporal interval. By way of example, and to inform a customer-specific decision regarding a requested modification to a term or condition of a provisioned financial product (e.g., an unsecured credit product, etc.), a machine learning and artificial intelligence process may be trained to predict an occurrence of a particular default event of a predetermined duration during a future, twelve-month interval. In other examples, to inform a customer-specific decision regarding a targeted marketing of financial products and services, an additional machine learning and artificial intelligence process may be trained to predict an occurrence of a particular engagement event involving a customer and a corresponding financial product or service during an additional, or alternate, future temporal interval, such as a three-month interval.

In some instances, each of the trained machine learning and artificial intelligence processes may be associated with a corresponding, process-specific input datasets, each of which may include a plurality of sequentially ordered, process-specific input feature and corresponding, customer-specific feature values, and with corresponding process-specific schedules for a delivery of corresponding elements of predicted output to a computing system associated with corresponding business units of the financial institution. Further, each of the trained machine learning and artificial intelligence processes may also be associated with a plurality of sequentially implemented data-pipelining operations that, when implemented by one or more computing systems associated with, or operated by, the financial institution, facilitate a generation and ingestion of corresponding process-specific input datasets, and generation, validation, or transaction of the elements of predictive output, in accordance with the underlying, process-specific delivery schedule for the corresponding elements of the predicted output (for example, at an expected delivery time on a daily basis, a weekly basis, a bi-monthly basis, or on a monthly basis).

To facilitate a delivery of the process-specific elements of the predicted output in accordance with corresponding ones of the process-specific delivery schedules, the one or more computing systems associated with, or operated by, the financial institution may collectively execute one, or more, of the process-specific data pipelining operations described herein on a simultaneous, or overlapping basis, which may facilitate a simultaneous, or overlapping, execution of one or more of the trained machine learning or artificial intelligence processes. In some instances, one or more of the executed process-specific data pipelining operations may experience a failure or a delay (e.g., due to absence of elements of customer-specific data associated with a corresponding input dataset, due to a failure in a communications network interconnecting one or more distributed computing components, etc.), and the one or more computing systems of the financial institution may detect the failure or delay, and generate and transmit a notification to a device operable by a representative of the financial institution, such as an analyst device operable by an analyst. Although the analyst device may generate and provision programmatic instructions (e.g., based on corresponding input from the analyst) that the one or more computing systems of the financial institution to reinitiate the failed or delayed, process-specific data pipelining operation, the re-initiation of the process-specific data pipelining operation may result in a temporal delay in the delivery of the corresponding, customer-specific elements of the predicted output to the computing system of the corresponding business unit.

In some examples, absent a real-time detection of the failure or a delay in the process-specific data pipelining operation, and a real-time provisioning of a notification of the detected failure or delay to an analyst device, the resulting temporal delay may reduce any ability of the one or more computing systems of the financial institution to re-initiate failed or delayed data pipelining operation based on programmatic instructions received from the analyst device and while delivering the customer-specific elements of the predicted output to the computing system of the corresponding business unit in accordance with the delivery schedule and at, or before, an agreed-upon delivery time. Further, although notification processes may enable the one or more computing systems of the financial institution (or an application program, code engine or module, or other elements of code executable by the one or more computing systems) to perform operations that generate and transmit notifications (e.g., email messages, text messages, etc.) to a computing device or system of a predetermined set of recipients (including the analyst described herein), these notification processes often cause the computing systems or devices to presents these received notification sequentially within a digital interface (e.g., a digital interface of an email or messaging application, as banners on a home screen, etc.).

Certain of these existing notification processes may fail to provide any centralized platform that enables the analyst to visualize a current status of each of the multiple, executed trained machine learning or artificial intelligence processes, and corresponding ones of the process-specific data pipelining operations, much less that enables the analyst to determine an impact of a particular delay or failure (e.g., associated with a corresponding one of the executed, machine learning or artificial intelligence processes and the corresponding business unit) on the delivery schedule associated with the corresponding, process-specific elements of predicted output, or on delivery schedules associated with other simultaneously or contemporaneously executed machine learning or artificial intelligence processes. Further, the absence of a centralized platform may also render impractical any process for debugging a data pipelining or execution of one or more of the executed, machine learning or artificial intelligence processes, and for comparing an actual data-pipelining or execution time associated with one or more of the machine learning or artificial intelligence processes against corresponding “expected” times, and additionally, or alternatively, for identifying or mediating potential “choke points” that trigger a delay, or failure, of the data pipelining associated with or the execution of one or more discrete machine learning or artificial intelligence processes, or classes or types of machine learning or artificial intelligence processes.

In some examples, one or more computing systems associated with the financial institution (or one or more distributed computing components) may perform operations, described herein, to generate, and render for presentation at a computing system or device of an analyst, a digital interface, such as an interactive, digital dashboard, having interface elements that identify each of a machine learning or artificial intelligence processes executed by the one or more computing systems associated with the financial institution. The exemplary processes described herein may enable one or more computing systems associated with the financial institution to provide a graphical representation of not only a current status of each of machine learning or artificial intelligence processes, as well as historical data characterizing a success or failure in the data pipelining operations associated with each of the executed machine learning or artificial intelligence processes over one or more prior temporal intervals. Further, the generated dashboard may, upon presentation by the computing system or device of the analyst, also include one or more interface elements that provide a graphical representation of a relative frequency of one or more data-pipelining or process-execution errors across all executed machine learning or artificial intelligence processes during one or more temporal intervals, and further, that provide a graphical representation of an aggregate number of successful, or failed, process executions during a current and one or more prior temporal intervals.

Through these exemplary processes, the presentation of the aggregate and process-specific outcome data within the dashboard may enable an analyst to not only identify and mediate, in real-time, the potential choke points in the data pipelining associated with, and the execution of, one or more of the machine learning or artificial intelligence processes, but also enable a characterizing and improvement (e.g., via load-balancing) of an efficiency of the data-pipelining and execution processes performed by the one or more computing systems of the financial institution, and a determination of a predicted consumption of computational resources during data pipelining associated with, and an execution of, a new machine learning or artificial intelligence processes. Certain of these exemplary processes, which provide a centralized and interactive digital interface characterizing a current status of each of the multiple, executed trained machine learning or artificial intelligence processes, and corresponding ones of the process-specific data pipelining operations, and which identifies, in real-time on a time-evolving basis, delays or failures associated with one or more of the executed trained machine learning or artificial intelligence processes, or corresponding ones of the process-specific data pipelining operations, may be implemented in addition to, or as an alternate to, existing notification processes that generate and provision process- or operation-specific notifications of detected delays or failures to an analyst device for sequential presentation within one or more digital interfaces.

A. Exemplary Computing Environments

FIG. 1 illustrates an exemplary computing environment 100 that includes, among other things, one or more computing devices, such as an analyst device 102 , and one or more computing systems, such as source systems 110 , a financial institution (FI) system 130 , distributed modeling system 150 , and one or more business-unit systems 160 , such as business- unit computing system 162 . Each of the one or more computing devices and one or more computing systems may be operatively connected to, and interconnected across, one or more communications networks, such as communications network 120 . Examples of communications network 120 include, but are not limited to, a wireless local area network (LAN) (e.g., a “Wi-Fi” network), a network utilizing radio-frequency (RF) communication protocols, a Near Field Communication (NFC) network, a wireless Metropolitan Area Network (MAN) connecting multiple wireless LANs, and a wide area network (WAN) (e.g., the Internet).

Analyst device 102 may include a computing device having one or more tangible, non-transitory memories, such as memory 105 , that store data and/or software instructions, and one or more processors, such as, processor 104 , configured to execute the software instructions. The one or more tangible, non-transitory memories may, in some aspects, store software applications, application modules, and other elements of code executable by the one or more processors, such as, but not limited to, an executable web browser 106 (e.g., Google Chrome™, Apple Safari™, etc.), and additionally or alternatively, an executable application associated with FI computing system 130 (e.g., dashboard application 108 ). In some instances, not illustrated in FIG. 1 , memory 105 may also include one or more structured or unstructured data repositories or databases, and analyst device 102 may maintain one or more elements of device data and location data within the one or more structured or unstructured data repositories or databases. For example, the elements of device data may uniquely identify analyst device 102 within computing environment 100 , and may include, but are not limited to, an Internet Protocol (IP) address assigned to analyst device 102 or a media access control (MAC) layer assigned to analyst device 102 .

Analyst device 102 may also include a display unit 109 A configured to present interface elements to a corresponding user and an input unit 109 B configured to receive input from the user. For example, input unit 109 B configured to receive input from the user in response to the interface elements presented through display unit 109 A. By way of example, display unit 109 A may include, but is not limited to, an LCD display unit or other appropriate type of display unit, and input unit 109 B may include, but is not limited to, a keypad, keyboard, touchscreen, voice activated control technologies, or appropriate type of input unit. Further, in additional aspects (not illustrated in FIG. 1 ), the functionalities of display unit 109 A and input unit 109 B may be combined into a single device, such as, a pressure-sensitive touchscreen display unit that presents interface elements and receives input from the user of analyst device 102 . Analyst device 102 may also include a communications interface 109 C, such as a wireless transceiver device, coupled to processor 104 and configured by processor 104 to establish and maintain communications with communications network 120 via one or more communication protocols, such as WiFi®, Bluetooth®, NFC, a cellular communications protocol (e.g., LTE®, CDMA®, GSM®, etc.), or any other suitable communications protocol.

Examples of analyst device 102 may include, but not limited to, a personal computer, a laptop computer, a tablet computer, a notebook computer, a hand-held computer, a personal digital assistant, a portable navigation device, a mobile phone, a smart phone, a wearable computing device (e.g., a smart watch, a wearable activity monitor, wearable smart jewelry, and glasses and other optical devices that include optical head-mounted displays (OHMDs), an embedded computing device (e.g., in communication with a smart textile or electronic fabric), and any other type of computing device that may be configured to store data and software instructions, execute software instructions to perform operations, and/or display information on an interface device or unit, such as display unit 109 A. In some instances, analyst device 102 may also establish communications with one or more additional computing systems or devices operating within computing environment 100 across a wired or wireless communications channel (via the communications interface 109 C using any appropriate communications protocol). Further, a user, such as an analyst 101 , may operate analyst device 102 and may do so to cause analyst device 102 to perform one or more exemplary processes described herein.

In some examples, source systems 110 (including internal source system 110 A, and external source system 110 B), FI computing system 130 , distributed modeling system 150 , and business-unit systems 160 (including computing system 162 ) may represent a computing system that includes one or more servers and tangible, non-transitory memories storing executable code and application modules. Further, the one or more servers may each include one or more processors, which may be configured to execute portions of the stored code or application modules to perform operations consistent with the disclosed embodiments. For example, the one or more processors may include a central processing unit (CPU) capable of processing a single operation (e.g., a scalar operations) in a single clock cycle. Further, each of source systems 110 (including internal source system 110 A, and external source system 110 B), FI computing system 130 , distributed modeling system 150 , and business-unit systems 160 (including computing system 162 ) may also include a communications interface, such as one or more wireless transceivers, coupled to the one or more processors for accommodating wired or wireless internet communication with other computing systems and devices operating within computing environment 100 .

Further, in some instances, source systems 110 (including internal source system 110 A, and external source system 110 B), FI computing system 130 , distributed modeling system 150 , and business-unit systems 160 may each be incorporated into a respective, discrete computing system. In additional, or alternate, instances, one or more of source systems 110 (including internal source system 110 A, and external source system 110 B), FI computing system 130 , distributed modeling system 150 , and business-unit systems 160 , may correspond to a distributed computing system having a plurality of interconnected, computing components distributed across an appropriate computing network, such as communications network 120 of FIG. 1 . For example, FI computing system 130 and additional, or alternatively, distributed modeling system 150 , may correspond, respectively, to a distributed or cloud-based computing cluster associated with, and maintained by, the financial institution, although in other examples, FI computing system 130 or distributed modeling system 150 may correspond, respectively, to a publicly accessible, distributed or cloud-based computing cluster, such as a computing cluster maintained by Microsoft Azure™, Amazon Web Services™, Google Cloud™, or another third-party provider.

For example, each of FI computing system 130 and distributed modeling system 150 may include a corresponding plurality of interconnected, distributed computing components, such as those described herein (not illustrated in FIG. 1A ), which may be configured to implement one or more parallelized, fault-tolerant distributed computing and analytical processes (e.g., an Apache Spark™ distributed, cluster-computing framework, a Databricks™ analytical platform, etc.). Further, and in addition to the CPUs described herein, the distributed computing components of FI computing system 130 , and additionally, or alternatively, the distributed computing components of distributed modeling system 150 , may also include one or more graphics processing units (GPUs) capable of processing thousands of operations (e.g., vector operations) in a single clock cycle, and additionally, or alternatively, one or more tensor processing units (TPUs) capable of processing hundreds of thousands of operations (e.g., matrix operations) in a single clock cycle. Through an implementation of the parallelized, fault-tolerant distributed computing and analytical protocols described herein, the distributed computing components of FI computing system 130 may perform any of the exemplary processes described herein to, among other things, ingest elements of data associated with the customers of the financial institution, preprocess and store the ingested data elements within an accessible data repository (e.g., within a portion of a distributed file system, such as a Hadoop distributed file system (HDFS)), generate customer-specific input datasets suitable for ingestion by corresponding trained, machine learning or artificial intelligence processes based on an application of one of more extract, transform, and load (ETL) to the ingested data elements, and post-process and transform customer- specific elements of predicted output data into formats suitable for provisioning to corresponding ones of business-unit systems 160 .

Further, and through an implementation of the parallelized, fault-tolerant distributed computing and analytical protocols described herein, the distributed components of distributed modeling system 150 may perform operations in parallel that not only train adaptively one or more machine learning or artificial intelligence processes using corresponding training and validation datasets, but also apply the trained machine learning or artificial intelligence processes to customer-specific input datasets and generate, in real time, elements of predicted output data. The implementation of the parallelized, fault-tolerant distributed computing and analytical protocols described herein across the one or more GPUs or TPUs included within the distributed components of FI computing system 130 and/or distributed modeling system 150 may, in some instances, accelerate the training, and the post-training deployment, of the machine-learning and artificial-intelligence process when compared to a training and deployment of the machine-learning and artificial-intelligence process across comparable clusters of CPUs capable of processing a single operation per clock cycle.

Referring back to FIG. 1 , each of source systems 110 may maintain, within corresponding tangible, non-transitory memories, a data repository that includes confidential data associated with customers of the financial institution. For example, internal source system 110 A may be associated with, or operated by, the financial institution, and may maintain, within the corresponding one or more tangible, non-transitory memories, a source data repository that includes one or more elements of customer profile data, account data, and transaction data that identify or characterize customers of the financial institution of FI computing system 130 . Further, as illustrated in FIG. 1 , external source system 1106 may also be associated with, or operated by, a third-party vendor, and maintain a source data repository that includes elements of third-party agency data. In some examples, the data records of third-party agency data may include reporting data associated with the customers of the financial institution of FI computing system 130 , such as, but not limited to, elements of credit-bureau data associated with, and generated by, one or more credit bureaus. Further, each of the source systems 110 (e.g., internal source system 110 A and external source system 1106 ) may perform operations that generate of obtain all, or a selected portion of the data records of customer profile data, account data, and transaction data, and reporting data associated with the customers, and that transmit the obtained portions as source data across communications network 120 to FI computing system 130 in real-time data and on a continuous streaming basis, on in batch form in accordance with a predetermined temporal schedule (e.g., on a daily basis, a monthly basis, etc.).

In some instances, one or more distributed components of FI computing system 130 may perform any of the exemplary processes described herein to execute one or more process-specific data pipelining operations associated with corresponding trained machine learning or artificial intelligence processes, which may facilitate a generation of customer- and process-specific input datasets. The one or more distributed components of FI computing system 130 may also perform any of the exemplary processes described herein that, in conjunction with distributed modeling system 150 , apply each of the trained machine learning or artificial intelligence processes and generate corresponding, customer-specific elements of predicted output data, and that execute additional data pipelining operations to validate, post-process, and deliver the customer- and process-specific input datasets to corresponding ones of the business-unit systems 160 , including computing system 162 , in accordance with a delivery schedule established by corresponding ones of business-unit systems 160 .

Further, in some instances, one or more distributed components of FI computing system 130 may perform any of the exemplary processes described herein to monitor an execution of each of the executed, process-specific data pipelining operations, and the application of the corresponding trained machine learning or artificial intelligence processes to the customer- and process-specific input data, and generate corresponding elements of log data that identify and characterize a successful initiation and completion of each of the executed data pipelining operations (including the application of the corresponding trained machine learning or artificial intelligence processes to the customer- and process-specific input) and any detected delays or failures. As described herein, and based on elements of the generated log data, the one or more distributed components of FI computing system 130 may perform operations that generate, and render for presentation at a computing system or device of an analyst, such as analyst device 102 , an interactive, digital dashboard, having interface elements that identify each of the trained machine learning or artificial intelligence processes and provide a graphical representation of not only a current status of each of trained machine learning or artificial intelligence processes, and the corresponding data pipelining operations, but also of a relative frequency of one or more data-pipelining or process-execution errors, and of one or more successful process executions, during a current temporal interval and one or more prior temporal intervals.

To facilitate a performance of one or more of these exemplary processes, FI computing system 130 may maintain, within the one or more tangible, non-transitory memories, a data repository 132 that includes a pipelining data store 134 maintaining, among other things, elements of process data 136 , delivery data 138 , log data 140 , and output data 142 . By way of example, the elements of process data 136 may include, for each of the trained machine learning or artificial intelligence processes, a corresponding process identifier (e.g., an alphanumeric character string, such a process name, assigned by FI computing system 130 ). Further, the elements of process data 136 may also include, for each of the trained machine learning or artificial intelligence processes, corresponding elements of process parameter data, which specify value of one or more process parameters associated with the trained machine learning or artificial intelligence process, and corresponding elements of process input data, which characterize a composition of an input dataset for the trained machine learning or artificial intelligence process and identifies each of the discrete input features within the input data set, along with a sequence or position of the input feature values within the input data set.

Further, in some example, the elements of delivery data 138 may include, for each of the trained machine learning or artificial intelligence processes, an expected time or an expected interval associated with the delivery of customer- and process-specific output data to the corresponding business unit (e.g., a corresponding one of business-unit systems 160 , etc.) and scheduling data characterizing, among other things a scheduled initiation time for data pipelining processes that support the execution of the machine learning or artificial intelligence processes (e.g., a time at which data pipelining engine 144 accesses source data, etc.). The elements of delivery data 138 may also include, for each of the trained machine learning or artificial intelligence processes, expected processing times for each of the exemplary data pipelining operations described herein, and each of the elements of delivery data, including the expected delivery time or delivery interval, the elements of scheduling data, and the expected processing time may be associated with a process identifier associated with the corresponding, trained machine learning or artificial intelligence process (e.g., the process name described herein).

In some instances, the elements of log data 140 may include, but are not limited to: (i) elements of a data-ingestion log that characterize a success, or failure, of a data-ingestion operation for each of the trained machine learning or artificial intelligence processes; (ii) elements of data-preparation log that characterize a success, or failure, of a data preparation operation for each of the trained machine learning or artificial intelligence processes; (iii) elements of execution log that characterize the success, or the failure, of an application of each of the trained machine learning or artificial intelligence processes to corresponding input datasets (e.g., a corresponding “execution” operation); (iv) elements of validation log that characterize the success or failure of a validation operation for each of the trained machine learning or artificial intelligence processes; and (v) elements of post-processing log that characterize, for each of the trained machine learning or artificial intelligence processes, a success, or failure, of not only a post-processing of the predictive output, but also of the provisioning of the transformed output to corresponding one of business-unit systems 160 , such as computing system 162 .

By way of example, each of the elements of data-ingestion log may be associated with an application of the data-ingestion operations described herein to a corresponding one of the Machine learning or artificial intelligence processes. In some instances, each of the elements of data-ingestion log may include a corresponding process identifier (e.g., as specified within process data 136 ) a starting time stamp (e.g., T START ) and an ending time stamp (T END ) for the data-ingestion operations, and additional data indicative of a successful outcome of the data-ingestion operations, or alternatively, a failure of the data-ingestion operations and a reason for that failure. Further, and by way of example, each of the elements of data-ingestion log may also include additional data indicative of either a programmatic initiation of the data-ingestion process or alternatively, or a manual ini

CLAIMS

Claims ( 20 )

What is claimed is:

1 . An apparatus, comprising:

a communications interface; a memory storing instructions; and at least one processor coupled to the communications interface and to the memory, the at least one processor being configured to execute the instructions to:

obtain process data associated with an execution of a plurality of machine learning or artificial intelligence processes;

based on the process data, determine, for each of the plurality of machine learning or artificial intelligence processes, value of one or more metrics characterizing a status of one or more operations that support the execution of the corresponding machine learning or artificial intelligence process; and

generate status data for each of the machine learning or artificial intelligence processes, and transmit one or more elements of the status data to a device via the communications interface, the status data comprising, for each of the machine learning or artificial intelligence processes, the determined one or more metric values and a corresponding process identifier, and the status data causing the device to present, for each of the machine learning or artificial intelligence processes, a graphical representation of at least one of the determined one or more metric values within a digital interface.

2 . The apparatus of claim 1 , wherein the process data for each of the machine learning or artificial intelligence process comprises a plurality of elements, each of the process data elements being associated with a corresponding one of the operations, and each of the process data elements being indicative of a success or failure of an execution of the corresponding one of the operations.

3 . The apparatus of claim 2 , wherein:

the at least one processor is further configured to execute the instructions to:

determine, based on the process data elements associated with each of the machine learning or artificial intelligence processes, a first aggregate value characterizing a number of failed executions of the operations during one or more temporal intervals and a second aggregate value characterizing a number of successful executions of the operations during the one or more temporal intervals; and

compute at least one of an aggregate failure rate or an aggregate success rate based on the first and second aggregate values; and

the status data comprises the first aggregate value, the second aggregate value, and the at least one of the aggregate failure rate or the aggregate success rate.

4 . The apparatus of claim 3 , wherein the at least one process is further configured to execute the instructions to determine the first aggregate value and the second aggregate value during each of the one or more temporal intervals.

5 . The apparatus of claim 2 , wherein the at least one processor is further configured to execute the instructions to:

based on the process data elements, determine a first value characterizing a number of failed executions of the operations associated with each of the machine learning or artificial intelligence processes during one or more temporal intervals, and determine a second value characterizing a number of successful executions of the operations associated with each of the machine learning or artificial intelligence processes during the one or more temporal intervals; compute a rate of successful or failed execution of the operations associated with each of the machine learning or artificial intelligence processes based on corresponding ones of the first and second values; and generate, for each of the machine learning or artificial intelligence processes, an element of the status data that includes the corresponding process identifier, the corresponding ones of the first and second values, and corresponding ones of the success and failure rates.

6 . The apparatus of claim 5 , wherein the at least one process is further configured to execute the instructions to determine, each of the machine learning or artificial intelligence processes, the corresponding ones of the first and second values during each of the one or more temporal intervals.

7 . The apparatus of claim 1 , wherein the at least one processor is further configured to execute the instructions to:

obtain scheduling data associated with a corresponding one of the machine learning or artificial intelligence processes, the scheduling data comprising first temporal data and second temporal data, the first temporal data characterizing a scheduled initiation of a first one of the operations associated with the corresponding machine learning or artificial intelligence processes, and the second temporal data characterizing a scheduled provisioning of predicted output associated with the corresponding machine learning or artificial intelligence processes to a computing system; and generate an element of the status data that includes the corresponding process identifier, the first temporal data, and the second temporal data.

8 . The apparatus of claim 7 , wherein the at least one processor is further configured to execute the instructions to:

obtain third temporal data associated with the corresponding machine learning or artificial intelligence process, the third temporal data characterizing a re-initiation of the first one of the operations based on a prior failure in an execution of the first one of the operations; and generate an element of the status data that includes the corresponding process identifier and the third temporal data.

9 . The apparatus of claim 1 , wherein the plurality of machine learning or artificial intelligence processes comprises a trained, gradient-boosted, decision-tree process.

10 . The apparatus of claim 1 , wherein the at least one processor is further configured to execute the instructions to:

obtain a process identifier associated with at least one of the machine learning or artificial intelligence processes, and based on the process identifier, obtain elements of scheduling data and process input data associated with the at least one machine learning or artificial intelligence process; perform operations that execute the one or more operations associated with the at least one machine learning or artificial intelligence process based on the process input data and the scheduling data; and generate, for the at least one machine learning or artificial intelligence process, an element of the process data associated with each of the executed operations, the elements of the process data characterizing a success or a failure of corresponding ones of the executed operations.

11 . The apparatus of claim 1 , wherein the status data further causes the device to:

generate and present, and for each of the machine learning or artificial intelligence processes, a first graphical representation of a first subset of the one or more metric values within a portion of a digital interface; receive input data indicative of a selection of one or the first graphical representation associated with a corresponding one of the machine learning or artificial intelligence processes; and based on the input data, generate and present a second graphical representation of a second subset of the one or more metric values associated with the corresponding machine learning or artificial intelligence process within an additional portion of the digital interface.

12 . A computer-implemented method, comprising:

obtaining, using at least one processor, process data associated with an execution of a plurality of machine learning or artificial intelligence processes; based on the process data, determining, using the at least one processor, and for each of the machine learning or artificial intelligence processes, a value of one or more metrics characterizing a status of one or more operations that support the execution of the corresponding machine learning or artificial intelligence process; and using the at least one processor, generating status data for each of the plurality of machine learning or artificial intelligence processes, and transmitting one or more elements of the status data to a device, the status data comprising, for each of the machine learning or artificial intelligence processes, the determined one or more metric values and a corresponding process identifier, and the status data causing the device to present, for each of the machine learning or artificial intelligence processes, a graphical representation of at least one of the determined one or more metric values within a digital interface.

13 . The computer-implemented method of claim 12 , wherein the process data for each of the machine learning or artificial intelligence process comprises a plurality of elements, each of the process data elements being associated with a corresponding one of the operations, and each of the process data elements being indicative of a success or failure of an execution of the corresponding one of the operations.

14 . The computer-implemented method of claim 13 , wherein:

the computer-implemented method further comprises:

using the at least one processor, determining, based on the process data elements associated with each of the machine learning or artificial intelligence processes, a first aggregate value characterizing a number of failed executions of the operations during one or more temporal intervals and a second aggregate value characterizing a number of successful executions of the operations during the one or more temporal intervals; and

computing, using the at least one processor, at least one of an aggregate failure rate or an aggregate success rate based on the first and second aggregate values; and

the status data comprises the first aggregate value, the second aggregate value, and the at least one of the aggregate failure rate or the aggregate success rate.

15 . The computer-implemented method of claim 13 , wherein:

the computer-implemented method further comprises:

using the at least one processor, and based on the process data elements, determining a first value characterizing a number of failed executions of the operations associated with each of the machine learning or artificial intelligence processes during one or more temporal intervals, and determining a second value characterizing a number of successful executions of the operations associated with each of the machine learning or artificial intelligence processes during the one or more temporal intervals;

computing, using the at least one processor, a rate of successful or failed execution of the operations associated with each of the machine learning or artificial intelligence processes based on corresponding ones of the first and second values; and

generating the status data comprises generating, for each of the machine learning or artificial intelligence processes, an element of the status data that includes the corresponding process identifier, the corresponding ones of the first and second values, and corresponding ones of the success and failure rates.

16 . The computer-implemented method of claim 12 , wherein:

the computer-implemented method further comprises obtaining, using the at least one processor, scheduling data associated with a corresponding one of the machine learning or artificial intelligence processes, the scheduling data comprising first temporal data and second temporal data, the first temporal data characterizing a scheduled initiation of a first one of the operations associated with the corresponding machine learning or artificial intelligence processes, and the second temporal data characterizing a scheduled provisioning of predicted output associated with the corresponding machine learning or artificial intelligence processes to a computing system; and generating the status data comprises generating an element of the status data that includes the corresponding process identifier, the first temporal data, and the second temporal data.

17 . The computer-implemented method of claim 16 , wherein:

the computer-implemented method further comprises, obtaining, using the at least one processor, third temporal data associated with the corresponding machine learning or artificial intelligence process, the third temporal data characterizing a re-initiation of the first one of the operations based on a prior failure in an execution of the first one of the operations; and generating the status data further comprises generating an additional element of the status data that includes the corresponding process identifier and the third temporal data.

18 . The computer-implemented method of claim 12 , further comprising:

using the at least one processor, obtaining a process identifier associated with at least one of the machine learning or artificial intelligence processes, and based on the process identifier, obtaining elements of scheduling data and process input data associated with the at least one machine learning or artificial intelligence process; performing, using the at least one processor, operations that execute the one or more operations associated with the at least one machine learning or artificial intelligence process based on the process input data and the scheduling data; and using the at least one processor, generating, for the at least one machine learning or artificial intelligence process, an element of the process data associated with each of the executed operations, the elements of the process data characterizing a success or a failure of corresponding ones of the executed operations.

19 . The computer-implemented method of claim 12 , wherein the status data further causes the device to:

generate and present, and for each of the machine learning or artificial intelligence processes, a first graphical representation of a first subset of the one or more metric values within a portion of a digital interface; receive input data indicative of a selection of one or the first graphical representation associated with a corresponding one of the machine learning or artificial intelligence processes; and based on the input data, generate and present a second graphical representation of a second subset of the one or more metric values associated with the corresponding machine learning or artificial intelligence process within an additional portion of the digital interface.

20 . A device, comprising:

a communications interface; a display unit; an input unit; a memory storing instructions; and at least one processor coupled to the communications interface, the display unit, the input unit, and the memory, the at least one processor being configured to execute the instructions to:

receive, via the communications interface, status data associated with an execution of a plurality of machine learning processes, the status data comprising, for each of the plurality of machine learning processes, a process identifier and a value of one or more metrics characterizing a status of one or more operations that support the execution of the corresponding machine learning process;

based on the status data, generate and present, via the display unit, and for each of the plurality of machine learning processes, a first graphical representation of a first subset of the one or more metric values within a portion of a digital interface;

receive, via the input unit, input data indicative of a selection of one or the first graphical representation associated with a corresponding one of the plurality of machine learning processes; and

based on the input data, generate and present, via the display unit, a second graphical representation of a second subset of the one or more metric values associated with the corresponding machine learning process within an additional portion of the digital interface.

US17/542,178

2020-12-16

2021-12-03

Interactive digital dashboards for trained machine learning or artificial intelligence processes

Pending

US20220188705A1

( en )

Priority Applications (2)

Application Number

Priority Date

Filing Date

Title

US17/542,178

US20220188705A1

( en )

2020-12-16

2021-12-03

Interactive digital dashboards for trained machine learning or artificial intelligence processes

CA3141107A

CA3141107A1

( en )

2020-12-16

2021-12-06

Interactive digital dashboards for trained machine learning or artificial intelligence processes

Applications Claiming Priority (2)

Application Number

Priority Date

Filing Date

Title

US202063126392P

2020-12-16

2020-12-16

US17/542,178

US20220188705A1

( en )

2020-12-16

2021-12-03

Interactive digital dashboards for trained machine learning or artificial intelligence processes

Publications (1)

Publication Number

Publication Date

US20220188705A1

true

US20220188705A1 ( en )

2022-06-16

Family

ID=81942623

Family Applications (1)

Application Number

Title

Priority Date

Filing Date

US17/542,178

Pending

US20220188705A1

( en )

2020-12-16

2021-12-03

Interactive digital dashboards for trained machine learning or artificial intelligence processes

Country Status (2)

Country

Link

US

( 1 )

US20220188705A1

( en )

CA

( 1 )

CA3141107A1

( en )

Cited By (31)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20210326644A1

( en )

*

2020-04-17

2021-10-21

Applied Engineering Concepts, Inc.

Physical Layer Authentication of Electronic Communication Networks

US20230146414A1

( en )

*

2021-11-08

2023-05-11

Servicenow, Inc.

Root Cause Analysis Based on Process Optimization Data

US20230297875A1

( en )

*

2022-03-16

2023-09-21

Qualcomm Incorporated

Federated learning in a disaggregated radio access network

US20240303237A1

( en )

*

2023-03-06

2024-09-12

Plaid Inc.

Predicting data availability and scheduling data pulls

US20250117595A1

( en )

*

2023-10-05

2025-04-10

The Toronto-Dominion Bank

Dynamic prompting based on conversation context

US12316715B2

( en )

2023-10-05

2025-05-27

The Toronto-Dominion Bank

Dynamic push notifications

US12399687B2

( en )

2023-08-30

2025-08-26

The Toronto-Dominion Bank

Generating software architecture from conversation

US12499241B2

( en )

2023-09-06

2025-12-16

The Toronto-Dominion Bank

Correcting security vulnerabilities with generative artificial intelligence

US12517812B2

( en )

2023-09-06

2026-01-06

The Toronto-Dominion Bank

Security testing based on generative artificial intelligence

US12536264B2

( en )

2024-07-19

2026-01-27

The Toronto-Dominion Bank

Parallel artificial intelligence driven identity checking with biometric prompting

US12541544B2

( en )

2024-03-28

2026-02-03

The Toronto-Dominion Bank

Generating a response for a communication session based on previous conversation content using a large language model

US12541894B2

( en )

2023-08-30

2026-02-03

The Toronto-Dominion Bank

Image modification based on goal progression

US12566541B2

( en )

2023-09-07

2026-03-03

The Toronto-Dominion Bank

Reconfigurable dashboard with moveable modules

US12585435B2

( en )

2023-08-30

2026-03-24

The Toronto-Dominion Bank

Real-time visualization of complex software architecture

US12592301B2

( en )

2023-08-30

2026-03-31

The Toronto-Dominion Bank

Prompt engineering and generative AI for goal-based imagery

US12591559B2

( en )

2024-03-28

2026-03-31

The Toronto-Dominion Back

Contextualized attributes for vectorized data

US12625680B2

( en )

2023-08-30

2026-05-12

The Toronto-Dominion Bank

Creating a model of software architecture

US12641178B2

( en )

2023-10-04

2026-05-26

The Toronto-Dominion Bank

Real-time dynamic visualization of content

US12645689B2

( en )

2024-06-14

2026-06-02

The Toronto-Dominion Bank

Weighted retrieval for in-context tabular model

US12646051B2

( en )

2023-09-21

2026-06-02

The Toronto-Dominion Bank

Dynamically evolving image based on pursuit of goals

US12645838B2

( en )

2024-07-18

2026-06-02

The Toronto-Dominion Bank

Limiting activity based on a profile

US12645429B2

( en )

2023-08-30

2026-06-02

The Toronto-Dominion Bank

Service design coordination among developers

US12650836B2

( en )

2023-08-31

2026-06-09

The Toronto-Dominion Bank

Gap identification and solution recommendation for complex software architecture

US12657566B2

( en )

2023-10-05

2026-06-16

The Toronto-Dominion Bank

GenAI letter of confirmation of benefits

US12670334B2

( en )

2024-03-25

2026-06-30

The Toronto-Dominion Bank

Chatbot with dynamic conversational tone

US12670640B2

( en )

2023-09-21

2026-06-30

The Toronto-Dominion Bank

Partial exchange of NFT via NFT interaction

US12688620B2

( en )

2023-08-30

2026-07-21

The Toronto-Dominion Bank

Dynamic generation of goals and images

US12693842B2

( en )

2024-01-12

2026-07-28

The Toronto-Dominion Bank

Feature activation based on alternative feature behavior

US12699556B2

( en )

2023-08-30

2026-08-04

The Toronto-Dominion Bank

Software library usage optimization

US12705398B2

( en )

2024-03-25

2026-08-11

The Toronto-Dominion Bank

Youth application with parental guidance role

US12711683B2

( en )

2024-01-12

2026-08-18

The Toronto-Dominion Bank

Dynamic animation based on waiting period

Citations (13)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20070094168A1

( en )

*

2005-07-29

2007-04-26

The Florida International University Board Of Trustees

Artificial neural network design and evaluation tool

US20120158623A1

( en )

*

2010-12-21

2012-06-21

Microsoft Corporation

Visualizing machine learning accuracy

US20180203754A1

( en )

*

2017-01-17

2018-07-19

Bank Of America Corporation

Individualized Channel Error Detection and Resolution

US20190050756A1

( en )

*

2014-06-30

2019-02-14

Amazon Technologies, Inc.

Machine learning service

US20190138903A1

( en )

*

2017-11-08

2019-05-09

International Business Machines Corporation

Reducing the cost of n modular redundancy for neural networks

US20190286538A1

( en )

*

2016-06-10

2019-09-19

Palantir Technologies, Inc.

Data pipeline monitoring

US20200226490A1

( en )

*

2019-01-15

2020-07-16

Ebay Inc.

Machine learning-based infrastructure anomaly and incident detection using multi-dimensional machine metrics

US20200234162A1

( en )

*

2019-01-22

2020-07-23

Servicenow, Inc.

Machine learning pipeline failure prediction

US20200272947A1

( en )

*

2019-02-25

2020-08-27

Sap Se

Orchestrator for machine learning pipeline

US20200334570A1

( en )

*

2019-04-16

2020-10-22

Apple Inc.

Data visualization for machine learning model performance

US20200349466A1

( en )

*

2019-05-03

2020-11-05

Microsoft Technology Licensing, Llc

Providing performance views associated with performance of a machine learning system

US20200349395A1

( en )

*

2019-05-03

2020-11-05

Microsoft Technology Licensing, Llc

Characterizing failures of a machine learning model based on instance features

US11605012B2

( en )

*

2020-02-28

2023-03-14

Intuit Inc.

Framework for processing machine learning model metrics

2021

2021-12-03

US

US17/542,178

patent/US20220188705A1/en

active

Pending

2021-12-06

CA

CA3141107A

patent/CA3141107A1/en

active

Pending

Patent Citations (13)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20070094168A1

( en )

*

2005-07-29

2007-04-26

The Florida International University Board Of Trustees

Artificial neural network design and evaluation tool

US20120158623A1

( en )

*

2010-12-21

2012-06-21

Microsoft Corporation

Visualizing machine learning accuracy

US20190050756A1

( en )

*

2014-06-30

2019-02-14

Amazon Technologies, Inc.

Machine learning service

US20190286538A1

( en )

*

2016-06-10

2019-09-19

Palantir Technologies, Inc.

Data pipeline monitoring

US20180203754A1

( en )

*

2017-01-17

2018-07-19

Bank Of America Corporation

Individualized Channel Error Detection and Resolution

US20190138903A1

( en )

*

2017-11-08

2019-05-09

International Business Machines Corporation

Reducing the cost of n modular redundancy for neural networks

US20200226490A1

( en )

*

2019-01-15

2020-07-16

Ebay Inc.

Machine learning-based infrastructure anomaly and incident detection using multi-dimensional machine metrics

US20200234162A1

( en )

*

2019-01-22

2020-07-23

Servicenow, Inc.

Machine learning pipeline failure prediction

US20200272947A1

( en )

*

2019-02-25

2020-08-27

Sap Se

Orchestrator for machine learning pipeline

US20200334570A1

( en )

*

2019-04-16

2020-10-22

Apple Inc.

Data visualization for machine learning model performance

US20200349466A1

( en )

*

2019-05-03

2020-11-05

Microsoft Technology Licensing, Llc

Providing performance views associated with performance of a machine learning system

US20200349395A1

( en )

*

2019-05-03

2020-11-05

Microsoft Technology Licensing, Llc

Characterizing failures of a machine learning model based on instance features

US11605012B2

( en )

*

2020-02-28

2023-03-14

Intuit Inc.

Framework for processing machine learning model metrics

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party

Title

D. Pérez-Berenguer, M. Kessler and J. García-Molina, " A Customizable and Incremental Processing Approach for Learning Analytics, " in IEEE Access, vol. 8, pp. 36350-36362, 2020, doi: 10.1109/ACCESS.2020.2975384 (Year: 2020)

*

Elshawi, Radwa, Mohamed Maher, and Sherif Sakr. " Automated machine learning: State-of-the-art and open challenges. " arXiv preprint arXiv:1906.02287 (2019) (Year: 2019)

*

Wu, Tongshuang, et al. " Errudite: Scalable, reproducible, and testable error analysis. " Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019. (Year: 2019)

*

Cited By (36)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US12014277B2

( en )

2020-04-17

2024-06-18

Applied Engineering Concepts, Inc.

Physical layer authentication of electronic communication networks

US11645538B2

( en )

*

2020-04-17

2023-05-09

Applied Engineering Concepts, Inc.

Physical layer authentication of electronic communication networks

US20210326644A1

( en )

*

2020-04-17

2021-10-21

Applied Engineering Concepts, Inc.

Physical Layer Authentication of Electronic Communication Networks

US12282385B2

( en )

2021-11-08

2025-04-22

Service Now, Inc.

Root cause analysis based on process optimization data

US20230146414A1

( en )

*

2021-11-08

2023-05-11

Servicenow, Inc.

Root Cause Analysis Based on Process Optimization Data

US11960353B2

( en )

*

2021-11-08

2024-04-16

Servicenow, Inc.

Root cause analysis based on process optimization data

US20230297875A1

( en )

*

2022-03-16

2023-09-21

Qualcomm Incorporated

Federated learning in a disaggregated radio access network

US20240303237A1

( en )

*

2023-03-06

2024-09-12

Plaid Inc.

Predicting data availability and scheduling data pulls

US12517903B2

( en )

*

2023-03-06

2026-01-06

Plaid Inc.

Predicting data availability and scheduling data pulls

US12699556B2

( en )

2023-08-30

2026-08-04

The Toronto-Dominion Bank

Software library usage optimization

US12399687B2

( en )

2023-08-30

2025-08-26

The Toronto-Dominion Bank

Generating software architecture from conversation

US12688620B2

( en )

2023-08-30

2026-07-21

The Toronto-Dominion Bank

Dynamic generation of goals and images

US12645429B2

( en )

2023-08-30

2026-06-02

The Toronto-Dominion Bank

Service design coordination among developers

US12585435B2

( en )

2023-08-30

2026-03-24

The Toronto-Dominion Bank

Real-time visualization of complex software architecture

US12625680B2

( en )

2023-08-30

2026-05-12

The Toronto-Dominion Bank

Creating a model of software architecture

US12592301B2

( en )

2023-08-30

2026-03-31

The Toronto-Dominion Bank

Prompt engineering and generative AI for goal-based imagery

US12541894B2

( en )

2023-08-30

2026-02-03

The Toronto-Dominion Bank

Image modification based on goal progression

US12650836B2

( en )

2023-08-31

2026-06-09

The Toronto-Dominion Bank

Gap identification and solution recommendation for complex software architecture

US12517812B2

( en )

2023-09-06

2026-01-06

The Toronto-Dominion Bank

Security testing based on generative artificial intelligence

US12499241B2

( en )

2023-09-06

2025-12-16

The Toronto-Dominion Bank

Correcting security vulnerabilities with generative artificial intelligence

US12566541B2

( en )

2023-09-07

2026-03-03

The Toronto-Dominion Bank

Reconfigurable dashboard with moveable modules

US12670640B2

( en )

2023-09-21

2026-06-30

The Toronto-Dominion Bank

Partial exchange of NFT via NFT interaction

US12646051B2

( en )

2023-09-21

2026-06-02

The Toronto-Dominion Bank

Dynamically evolving image based on pursuit of goals

US12641178B2

( en )

2023-10-04

2026-05-26

The Toronto-Dominion Bank

Real-time dynamic visualization of content

US12316715B2

( en )

2023-10-05

2025-05-27

The Toronto-Dominion Bank

Dynamic push notifications

US12657566B2

( en )

2023-10-05

2026-06-16

The Toronto-Dominion Bank

GenAI letter of confirmation of benefits

US20250117595A1

( en )

*

2023-10-05

2025-04-10

The Toronto-Dominion Bank

Dynamic prompting based on conversation context

US12693842B2

( en )

2024-01-12

2026-07-28

The Toronto-Dominion Bank

Feature activation based on alternative feature behavior

US12711683B2

( en )

2024-01-12

2026-08-18

The Toronto-Dominion Bank

Dynamic animation based on waiting period

US12670334B2

( en )

2024-03-25

2026-06-30

The Toronto-Dominion Bank

Chatbot with dynamic conversational tone

US12705398B2

( en )

2024-03-25

2026-08-11

The Toronto-Dominion Bank

Youth application with parental guidance role

US12591559B2

( en )

2024-03-28

2026-03-31

The Toronto-Dominion Back

Contextualized attributes for vectorized data

US12541544B2

( en )

2024-03-28

2026-02-03

The Toronto-Dominion Bank

Generating a response for a communication session based on previous conversation content using a large language model

US12645689B2

( en )

2024-06-14

2026-06-02

The Toronto-Dominion Bank

Weighted retrieval for in-context tabular model

US12645838B2

( en )

2024-07-18

2026-06-02

The Toronto-Dominion Bank

Limiting activity based on a profile

US12536264B2

( en )

2024-07-19

2026-01-27

The Toronto-Dominion Bank

Parallel artificial intelligence driven identity checking with biometric prompting

Also Published As

Publication number

Publication date

CA3141107A1

( en )

2022-06-16

Similar Documents

Publication

Publication Date

Title

CA3141107A1

( en )

2022-06-16

Interactive digital dashboards for trained machine learning or artificial intelligence processes

US12455743B2

( en )

2025-10-28

Configurable pipelines for training and deploying machine learning processes in distributed computing environments

US20240386295A1

( en )

2024-11-21

Monitoring and versioning of pipelined machine learning processes in distributed computing environments

US20240386326A1

( en )

2024-11-21

Programmatic analysis and monitoring of pipelined machine learning processes in distributed computing environments

US20240370881A1

( en )

2024-11-07

Management of programmatic and compliance workflows using robotic process automation

US20240386325A1

( en )

2024-11-21

Modular framework for training forward-in-time machine learning processes in distributed computing environments

US20240412083A1

( en )

2024-12-12

Modular and configurable computational frameworks for real-time inferencing in distributed computing environments

US20220108069A1

( en )

2022-04-07

Dynamic management of compliance workflow using trained machine-learning and artificial-intelligence processes

US12307396B2

( en )

2025-05-20

Optimizing user task schedules in a customer relationship management platform

US20220207606A1

( en )

2022-06-30

Prediction of future occurrences of events using adaptively trained artificial-intelligence processes

US12223352B2

( en )

2025-02-11

Systems and methods for maintaining data objects to manage asynchronous workflows

US20180239615A1

( en )

2018-08-23

Dynamic execution of parameterized applications for the processing of keyed network data streams

US20180143891A1

( en )

2018-05-24

Related documents

• TokenBank: Financial Infrastructure for AI Services#1248613
• Dashboard visualizations: Supporting real-time throughput decision-making.#269437
• Human in the loop in AI-enabled clinical decision support: a systematic scoping review and reporting checklist for lifecycle governance.#1221477
• Empowering Financial Institutions with GPT Powered Frameworks for Market Intelligence and Decision-Making#82207
• Artificial Intelligence for Early Detection of Peripheral Artery Disease#968019