ConceptioArchiveGoogle Patents
Google Patentsopen access

Systems and methods of large language model driven orchestration of task- … — Broadridge Financial Solutions, Inc. (US12307349B2)

Broadridge Financial Solutions, Inc. · Google Patents
Google Patents · Patents · License: Open Access
Open Source ↗
patent, google patents, intellectual property, US12307349B2, Broadridge Financial Solutions, Inc., Joseph Lo, en, 2025

ABSTRACT

Abstract

Systems and methods of the present disclosure may receive, from a user computing device, a user-provided data record query including a natural language request for information associated with one or more data sources. User persona attributes of the user may be determined, such as a user role or security parameters or both. Based on the user persona attributes a context query may be generated to obtain context attributes associated with the user-provided query. The natural language request and the context attributes are input into the model orchestration large language model (LLM) to output instructions to machine learning (ML) agents based on the context attributes. The ML agents output responses associated with the user-provided data record query based on the instructions, and the responses are input into the model orchestration LLM to output to the user computing device a natural language response based on the context attributes.

Description

FIELD OF TECHNOLOGY

The present disclosure generally relates to systems and methods of large language model (LLM) driven orchestration of task-specific machine learning agents, including user interfacing via the LLM to initiate and coordinate task-specific machine learning agents in response to a user-provided query for data.

BACKGROUND OF TECHNOLOGY

Electronic information search typically employs one or more machine learning and/or heuristic search algorithms. These algorithms are often discrete and task-specific, requiring complex resources to coordinate the algorithms and reconcile the results. Moreover, some domains require compliance with guidelines, rules, regulations and/or other standards for accuracy and conformance.

SUMMARY OF DESCRIBED SUBJECT MATTER

In some aspects, the techniques described herein relate to a method including: receiving, by at least one processor from at least one user computing device associated with a user, a user-provided data record query including a natural language request to perform at least one action with at least one data record; determining, by the at least one processor, a user profile associated with the user; wherein the user profile includes user persona attributes; wherein the user persona attributes include at least one of: a user role, or at least one security parameter associated with the user; generating, by the at least one processor, based on the user persona attributes of the user profile, at least one context query to at least one data source so as to obtain at least one context attribute associated with the data record query; inputting, by the at least one processor, the at least one context attribute into a model orchestration large language model to inject context into a model orchestration large language model runtime of the model orchestration large language model; inputting, by the at least one processor, the natural language request of the data record query as a data record query prompt into the model orchestration large language model to output at least one instruction to at least one data record processing machine learning agent of a plurality of data record processing machine learning agents based at least in part on trained parameters of the model orchestration large language model and the at least one context attribute; inputting, by the at least one processor, the at least one instruction into the at least one data record processing machine learning agent to output at least one response; inputting, by the at least one processor, the at least one response as a response prompt into the model orchestration large language model to output to the user computing device at least one natural language response representative of the at least one action based at least in part on trained parameters of the model orchestration large language model and the at least one context attribute; and causing to display, by the at least one processor, the at least one natural language response in a graphical user interface (GUI) rendered on the user computing device.

In some aspects, the techniques described herein relate to a method, wherein the plurality of data record processing machine learning agents are configured to be instantiated in parallel.

In some aspects, the techniques described herein relate to a method, wherein the at least one data record processing machine learning agent is configured to utilize at least one modular shared data processing component to output the at least one response.

In some aspects, the techniques described herein relate to a method, wherein the at least one data record processing machine learning agent is at least two data record processing machine learning agents; wherein at least one first data record processing machine learning agent is adversarial to at least one second data record processing machine learning agent.

In some aspects, the techniques described herein relate to a method, wherein the at least one first data record processing machine learning agent is configured to output the at least one response; and wherein the at least one second data record processing machine learning agent is configured to: ingest the at least one response from the at least one first data record processing machine learning agent; determine a correctness assessment based at least in part on correctness assessment machine learning parameters; and refine the at least one response based at least in part on the correctness assessment.

In some aspects, the techniques described herein relate to a method, further including: inputting, by the at least one processor, at least one compliance rule into at least one compliance verification machine learning agent to output at least one compliance verification prompt based at least in part on a plurality of compliance verification parameters; wherein the at least one compliance verification prompt is configured to cause the model orchestration large language model to verify compliance of the at least one natural language response with the at least one compliance rule; and inputting, by the at least one processor, the at least one compliance verification prompt into the model orchestration large language model to output at least one compliance verification of the at least one natural language model based at least in part on the trained parameters of the model orchestration large language model and the at least one context attribute.

In some aspects, the techniques described herein relate to a method, further including: inputting, by the at least one processor, based on the at least one compliance verification being representative of the at least one natural language response being non-compliant, the natural language response into at least one compliance machine learning agent to output a variation to the natural language response; inputting, by the at least one processor, the at least one compliance verification prompt and the variation to the natural language response into the model orchestration large language model to output at least one new compliance verification of the variation to the at least one natural language model based at least in part on the trained parameters of the model orchestration large language model and the at least one context attribute.

In some aspects, the techniques described herein relate to a method, wherein the at least one compliance verification includes at least one of: a pass indicative of the at least one natural language response being compliant, or a fail indicative of the at least one natural language response being non-compliant.

In some aspects, the techniques described herein relate to a method, wherein the at least one instruction includes at least one programmatic step including at least one of: at least one database query, at least one application programming interface (API) call, or at least one internet search query.

In some aspects, the techniques described herein relate to a method, further including: tracking, by the at least one processor, the at least one instruction; generating, by the at least one processor, at least one model explainability prompt representative of the at least one instruction; wherein the at least one model explainability prompt is configured to cause the model orchestration large language model to output a natural language explanation of at least one of: the at least one data record processing machine learning agent, or the at least one programmatic step; and inputting, by the at least one processor, the at least one model explainability prompt into the model orchestration large language model to output the natural language explanation based at least in part on the trained parameters of the model orchestration large language model and the at least one context attribute.

In some aspects, the techniques described herein relate to a system including: at least one processor in communication with at least one non-transitory computer readable medium having software instructions stored thereon, wherein the at least one processor, upon execution of the software instructions, is configured to: receive, from at least one user computing device associated with a user, a user-provided data record query including a natural language request to perform at least one action with at least one data record; determine a user profile associated with the user; wherein the user profile includes user persona attributes; wherein the user persona attributes include at least one of: a user role, or at least one security parameter associated with the user; generate based on the user persona attributes of the user profile, at least one context query to at least one data source so as to obtain at least one context attribute associated with the data record query; input the at least one context attribute into a model orchestration large language model to inject context into a model orchestration large language model runtime of the model orchestration large language model; input the natural language request of the data record query as a data record query prompt into the model orchestration large language model to output at least one instruction to at least one data record processing machine learning agent of a plurality of data record processing machine learning agents based at least in part on trained parameters of the model orchestration large language model and the at least one context attribute; input the at least one instruction into the at least one data record processing machine learning agent to output at least one response; input the at least one response as a response prompt into the model orchestration large language model to output to the user computing device at least one natural language response representative of the at least one action based at least in part on trained parameters of the model orchestration large language model and the at least one context attribute; and cause to display the at least one natural language response in a graphical user interface (GUI) rendered on the user computing device.

In some aspects, the techniques described herein relate to a system, wherein the plurality of data record processing machine learning agents are configured to be instantiated in parallel.

In some aspects, the techniques described herein relate to a system, wherein the at least one data record processing machine learning agent is configured to utilize at least one modular shared data processing component to output the at least one response.

In some aspects, the techniques described herein relate to a system, wherein the at least one data record processing machine learning agent is at least two data record processing machine learning agents; wherein at least one first data record processing machine learning agent is adversarial to at least one second data record processing machine learning agent.

In some aspects, the techniques described herein relate to a system, wherein the at least one first data record processing machine learning agent is configured to output the at least one response; and wherein the at least one second data record processing machine learning agent is configured to: ingest the at least one response from the at least one first data record processing machine learning agent; determine a correctness assessment based at least in part on correctness assessment machine learning parameters; and refine the at least one response based at least in part on the correctness assessment.

In some aspects, the techniques described herein relate to a system, wherein the at least one processor is further configured to: input at least one compliance rule into at least one compliance verification machine learning agent to output at least one compliance verification prompt based at least in part on a plurality of compliance verification parameters; wherein the at least one compliance verification prompt is configured to cause the model orchestration large language model to verify compliance of the at least one natural language response with the at least one compliance rule; and input the at least one compliance verification prompt into the model orchestration large language model to output at least one compliance verification of the at least one natural language model based at least in part on the trained parameters of the model orchestration large language model and the at least one context attribute.

In some aspects, the techniques described herein relate to a system, wherein the at least one processor is further configured to: input based on the at least one compliance verification being representative of the at least one natural language response being non-compliant, the natural language response into at least one compliance machine learning agent to output a variation to the natural language response; input the at least one compliance verification prompt and the variation to the natural language response into the model orchestration large language model to output at least one new compliance verification of the variation to the at least one natural language model based at least in part on the trained parameters of the model orchestration large language model and the at least one context attribute.

In some aspects, the techniques described herein relate to a system, wherein the at least one compliance verification includes at least one of: a pass indicative of the at least one natural language response being compliant, or a fail indicative of the at least one natural language response being non-compliant.

In some aspects, the techniques described herein relate to a system, wherein the at least one instruction includes at least one programmatic step including at least one of: at least one database query, at least one application programming interface (API) call, or at least one internet search query.

In some aspects, the techniques described herein relate to a system, wherein the at least one processor is further configured to: track the at least one instruction; generate at least one model explainability prompt representative of the at least one instruction; wherein the at least one model explainability prompt is configured to cause the model orchestration large language model to output a natural language explanation of at least one of: the at least one data record processing machine learning agent, or the at least one programmatic step; and input the at least one model explainability prompt into the model orchestration large language model to output the natural language explanation based at least in part on the trained parameters of the model orchestration large language model and the at least one context attribute.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments of the present disclosure can be further explained with reference to the attached drawings, wherein like structures are referred to by like numerals throughout the several views. The drawings shown are not necessarily to scale, with emphasis instead generally being placed upon illustrating the principles of the present disclosure. Therefore, specific structural and functional details disclosed herein are not to be interpreted as limiting, but merely as a representative basis for teaching one skilled in the art to variously employ one or more illustrative embodiments.

FIG. 1 depicts a system for large language model (LLM) orchestrated data search across public and private data sources in accordance with one or more embodiments of the present disclosure.

FIG. 2 depicts a context engine for injecting and configuring context in a run-time of the large language model (LLM) orchestrated data search across public and private data sources in accordance with one or more embodiments of the present disclosure.

FIG. 3 depicts a compliance agent for verifying and ensuring compliance to a ruleset for large language model (LLM) orchestrated data search across public and private data sources in accordance with one or more embodiments of the present disclosure.

FIG. 4 depicts a block diagram of an exemplary computer-based system and platform for large language model (LLM) orchestrated data search across public and private data sources in accordance with one or more embodiments of the present disclosure.

FIG. 5 depicts a block diagram of another exemplary computer-based system

FIELD OF TECHNOLOGY

The present disclosure generally relates to systems and methods of large language model (LLM) driven orchestration of task-specific machine learning agents, including user interfacing via the LLM to initiate and coordinate task-specific machine learning agents in response to a user-provided query for data.

BACKGROUND OF TECHNOLOGY

Electronic information search typically employs one or more machine learning and/or heuristic search algorithms. These algorithms are often discrete and task-specific, requiring complex resources to coordinate the algorithms and reconcile the results. Moreover, some domains require compliance with guidelines, rules, regulations and/or other standards for accuracy and conformance.

SUMMARY OF DESCRIBED SUBJECT MATTER

In some aspects, the techniques described herein relate to a method including: receiving, by at least one processor from at least one user computing device associated with a user, a user-provided data record query including a natural language request to perform at least one action with at least one data record; determining, by the at least one processor, a user profile associated with the user; wherein the user profile includes user persona attributes; wherein the user persona attributes include at least one of: a user role, or at least one security parameter associated with the user; generating, by the at least one processor, based on the user persona attributes of the user profile, at least one context query to at least one data source so as to obtain at least one context attribute associated with the data record query; inputting, by the at least one processor, the at least one context attribute into a model orchestration large language model to inject context into a model orchestration large language model runtime of the model orchestration large language model; inputting, by the at least one processor, the natural language request of the data record query as a data record query prompt into the model orchestration large language model to output at least one instruction to at least one data record processing machine learning agent of a plurality of data record processing machine learning agents based at least in part on trained parameters of the model orchestration large language model and the at least one context attribute; inputting, by the at least one processor, the at least one instruction into the at least one data record processing machine learning agent to output at least one response; inputting, by the at least one processor, the at least one response as a response prompt into the model orchestration large language model to output to the user computing device at least one natural language response representative of the at least one action based at least in part on trained parameters of the model orchestration large language model and the at least one context attribute; and causing to display, by the at least one processor, the at least one natural language response in a graphical user interface (GUI) rendered on the user computing device.

In some aspects, the techniques described herein relate to a method, wherein the plurality of data record processing machine learning agents are configured to be instantiated in parallel.

In some aspects, the techniques described herein relate to a method, wherein the at least one data record processing machine learning agent is configured to utilize at least one modular shared data processing component to output the at least one response.

In some aspects, the techniques described herein relate to a method, wherein the at least one data record processing machine learning agent is at least two data record processing machine learning agents; wherein at least one first data record processing machine learning agent is adversarial to at least one second data record processing machine learning agent.

In some aspects, the techniques described herein relate to a method, wherein the at least one first data record processing machine learning agent is configured to output the at least one response; and wherein the at least one second data record processing machine learning agent is configured to: ingest the at least one response from the at least one first data record processing machine learning agent; determine a correctness assessment based at least in part on correctness assessment machine learning parameters; and refine the at least one response based at least in part on the correctness assessment.

In some aspects, the techniques described herein relate to a method, further including: inputting, by the at least one processor, at least one compliance rule into at least one compliance verification machine learning agent to output at least one compliance verification prompt based at least in part on a plurality of compliance verification parameters; wherein the at least one compliance verification prompt is configured to cause the model orchestration large language model to verify compliance of the at least one natural language response with the at least one compliance rule; and inputting, by the at least one processor, the at least one compliance verification prompt into the model orchestration large language model to output at least one compliance verification of the at least one natural language model based at least in part on the trained parameters of the model orchestration large language model and the at least one context attribute.

In some aspects, the techniques described herein relate to a method, further including: inputting, by the at least one processor, based on the at least one compliance verification being representative of the at least one natural language response being non-compliant, the natural language response into at least one compliance machine learning agent to output a variation to the natural language response; inputting, by the at least one processor, the at least one compliance verification prompt and the variation to the natural language response into the model orchestration large language model to output at least one new compliance verification of the variation to the at least one natural language model based at least in part on the trained parameters of the model orchestration large language model and the at least one context attribute.

In some aspects, the techniques described herein relate to a method, wherein the at least one compliance verification includes at least one of: a pass indicative of the at least one natural language response being compliant, or a fail indicative of the at least one natural language response being non-compliant.

In some aspects, the techniques described herein relate to a method, wherein the at least one instruction includes at least one programmatic step including at least one of: at least one database query, at least one application programming interface (API) call, or at least one internet search query.

In some aspects, the techniques described herein relate to a method, further including: tracking, by the at least one processor, the at least one instruction; generating, by the at least one processor, at least one model explainability prompt representative of the at least one instruction; wherein the at least one model explainability prompt is configured to cause the model orchestration large language model to output a natural language explanation of at least one of: the at least one data record processing machine learning agent, or the at least one programmatic step; and inputting, by the at least one processor, the at least one model explainability prompt into the model orchestration large language model to output the natural language explanation based at least in part on the trained parameters of the model orchestration large language model and the at least one context attribute.

In some aspects, the techniques described herein relate to a system including: at least one processor in communication with at least one non-transitory computer readable medium having software instructions stored thereon, wherein the at least one processor, upon execution of the software instructions, is configured to: receive, from at least one user computing device associated with a user, a user-provided data record query including a natural language request to perform at least one action with at least one data record; determine a user profile associated with the user; wherein the user profile includes user persona attributes; wherein the user persona attributes include at least one of: a user role, or at least one security parameter associated with the user; generate based on the user persona attributes of the user profile, at least one context query to at least one data source so as to obtain at least one context attribute associated with the data record query; input the at least one context attribute into a model orchestration large language model to inject context into a model orchestration large language model runtime of the model orchestration large language model; input the natural language request of the data record query as a data record query prompt into the model orchestration large language model to output at least one instruction to at least one data record processing machine learning agent of a plurality of data record processing machine learning agents based at least in part on trained parameters of the model orchestration large language model and the at least one context attribute; input the at least one instruction into the at least one data record processing machine learning agent to output at least one response; input the at least one response as a response prompt into the model orchestration large language model to output to the user computing device at least one natural language response representative of the at least one action based at least in part on trained parameters of the model orchestration large language model and the at least one context attribute; and cause to display the at least one natural language response in a graphical user interface (GUI) rendered on the user computing device.

In some aspects, the techniques described herein relate to a system, wherein the plurality of data record processing machine learning agents are configured to be instantiated in parallel.

In some aspects, the techniques described herein relate to a system, wherein the at least one data record processing machine learning agent is configured to utilize at least one modular shared data processing component to output the at least one response.

In some aspects, the techniques described herein relate to a system, wherein the at least one data record processing machine learning agent is at least two data record processing machine learning agents; wherein at least one first data record processing machine learning agent is adversarial to at least one second data record processing machine learning agent.

In some aspects, the techniques described herein relate to a system, wherein the at least one first data record processing machine learning agent is configured to output the at least one response; and wherein the at least one second data record processing machine learning agent is configured to: ingest the at least one response from the at least one first data record processing machine learning agent; determine a correctness assessment based at least in part on correctness assessment machine learning parameters; and refine the at least one response based at least in part on the correctness assessment.

In some aspects, the techniques described herein relate to a system, wherein the at least one processor is further configured to: input at least one compliance rule into at least one compliance verification machine learning agent to output at least one compliance verification prompt based at least in part on a plurality of compliance verification parameters; wherein the at least one compliance verification prompt is configured to cause the model orchestration large language model to verify compliance of the at least one natural language response with the at least one compliance rule; and input the at least one compliance verification prompt into the model orchestration large language model to output at least one compliance verification of the at least one natural language model based at least in part on the trained parameters of the model orchestration large language model and the at least one context attribute.

In some aspects, the techniques described herein relate to a system, wherein the at least one processor is further configured to: input based on the at least one compliance verification being representative of the at least one natural language response being non-compliant, the natural language response into at least one compliance machine learning agent to output a variation to the natural language response; input the at least one compliance verification prompt and the variation to the natural language response into the model orchestration large language model to output at least one new compliance verification of the variation to the at least one natural language model based at least in part on the trained parameters of the model orchestration large language model and the at least one context attribute.

In some aspects, the techniques described herein relate to a system, wherein the at least one compliance verification includes at least one of: a pass indicative of the at least one natural language response being compliant, or a fail indicative of the at least one natural language response being non-compliant.

In some aspects, the techniques described herein relate to a system, wherein the at least one instruction includes at least one programmatic step including at least one of: at least one database query, at least one application programming interface (API) call, or at least one internet search query.

In some aspects, the techniques described herein relate to a system, wherein the at least one processor is further configured to: track the at least one instruction; generate at least one model explainability prompt representative of the at least one instruction; wherein the at least one model explainability prompt is configured to cause the model orchestration large language model to output a natural language explanation of at least one of: the at least one data record processing machine learning agent, or the at least one programmatic step; and input the at least one model explainability prompt into the model orchestration large language model to output the natural language explanation based at least in part on the trained parameters of the model orchestration large language model and the at least one context attribute.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments of the present disclosure can be further explained with reference to the attached drawings, wherein like structures are referred to by like numerals throughout the several views. The drawings shown are not necessarily to scale, with emphasis instead generally being placed upon illustrating the principles of the present disclosure. Therefore, specific structural and functional details disclosed herein are not to be interpreted as limiting, but merely as a representative basis for teaching one skilled in the art to variously employ one or more illustrative embodiments.

FIG. 1 depicts a system for large language model (LLM) orchestrated data search across public and private data sources in accordance with one or more embodiments of the present disclosure.

FIG. 2 depicts a context engine for injecting and configuring context in a run-time of the large language model (LLM) orchestrated data search across public and private data sources in accordance with one or more embodiments of the present disclosure.

FIG. 3 depicts a compliance agent for verifying and ensuring compliance to a ruleset for large language model (LLM) orchestrated data search across public and private data sources in accordance with one or more embodiments of the present disclosure.

FIG. 4 depicts a block diagram of an exemplary computer-based system and platform for large language model (LLM) orchestrated data search across public and private data sources in accordance with one or more embodiments of the present disclosure.

FIG. 5 depicts a block diagram of another exemplary computer-based system and platform for large language model (LLM) orchestrated data search across public and private data sources in accordance with one or more embodiments of the present disclosure.

FIG. 6 depicts illustrative schematics of an exemplary implementation of the cloud computing/architecture(s) in which embodiments of a system for large language model (LLM) orchestrated data search across public and private data sources may be specifically configured to operate in accordance with some embodiments of the present disclosure.

FIG. 7 depicts illustrative schematics of another exemplary implementation of the cloud computing/architecture(s) in which embodiments of a system for large language model (LLM) orchestrated data search across public and private data sources may be specifically configured to operate in accordance with some embodiments of the present disclosure.

DETAILED DESCRIPTION

FIGS. 1 through 7 illustrate systems and methods of large language model (LLM) driven data querying for response to a user-provided natural language query. The following embodiments provide technical solutions and technical improvements that overcome technical problems, drawbacks and/or deficiencies in the technical fields involving data search scalability when searching across multiple private and/or public data sources, data source integration where each data source typical requires a customized and particular set of tools for interaction, LLM answers that often result in false information delivered as if it were true (commonly referred to as “hallucination”) and/or in violation of rules, standards and/or guidelines. As explained in more detail, below, technical solutions and technical improvements herein include aspects of improved integration of machine learning (ML)-based software agents with one or more LLMs such that the LLM(s) provide orchestration of the ML-based software agents enabling improved scalability of data sources, search and analytics, while the ML-based software agents provide parallel checks and verifications of each other and the LLM to reduce hallucination and non-compliant information. Based on such technical features, further technical benefits become available to users and operators of these systems and methods. Moreover, various practical applications of the disclosed technology are also described, which provide further practical benefits to users and operators that are also new and useful improvements in the art.

In at least some embodiments, the terms “agent” and “software agent” are used interchangeably and refer to a program that may perform at least one task at a particular schedule and/or triggering event with at least some degree of autonomy on behalf of its host and flexibility.

In embodiments, the systems and methods of present disclosure may use an LLM in conjunction with a range of publicly available data, privately available data, and task-specific models to answer user queries questions and assist users in identifying information, trends, themes, insights and/or analytics among other information or any combination thereof. In some embodiments, the described systems and methods may be adapted to one or more different domains.

One such example is financial instrument trading. In embodiments of such an example, the systems and methods may use an LLM in conjunction with a range of publicly available data, privately available data, and task-specific models to answer bond-related questions and assist users in identifying bonds, themes, and trends. In embodiments of such an example, the systems and methods may inform and expedite vital pricing decisions, facilitates counterparty selection, broadens liquidity access, enhances the often-complex bond selection and portfolio construction processes, and/or provide other improvements to the financial instrument trading domain.

In some embodiments, this architecture leverages the ability of the LLM to interpret and understand a user-provided question and identify the types of information to be queried. This ability can be leveraged to use the LLM to generate instructions to one or more different task-specific models based on the user-provided question in order to orchestrate the task-specific models that are associated with the information being sought, and in turn resulting in a scalable platform of task-specific models whose results can be translated by the LLM into a natural language response to the user for improved data search, data source integration, data analytics and user interfacing. As a result, the systems and methods of the present disclosure enable more complicated user questions that are answered in reduced time and with greater insight, thus providing improved data timeliness and accuracy, and reduced infrastructure costs.

Various detailed embodiments of the present disclosure, taken in conjunction with the accompanying FIGs., are disclosed herein; however, it is to be understood that the disclosed embodiments are merely illustrative. In addition, each of the examples given in connection with the various embodiments of the present disclosure is intended to be illustrative, and not restrictive.

Throughout the specification, the following terms take the meanings explicitly associated herein, unless the context clearly dictates otherwise. The phrases “in one embodiment” and “in some embodiments” as used herein do not necessarily refer to the same embodiment(s), though it may. Furthermore, the phrases “in another embodiment” and “in some other embodiments” as used herein do not necessarily refer to a different embodiment, although it may. Thus, as described below, various embodiments may be readily combined, without departing from the scope or spirit of the present disclosure.

In addition, the term “based on” is not exclusive and allows for being based on additional factors not described, unless the context clearly dictates otherwise. In addition, throughout the specification, the meaning of “a,” “an,” and “the” include plural references. The meaning of “in” includes “in” and “on.”

As used herein, the terms “and” and “or” may be used interchangeably to refer to a set of items in both the conjunctive and disjunctive in order to encompass the full description of combinations and alternatives of the items. By way of example, a set of items may be listed with the disjunctive “or”, or with the conjunction “and.” In either case, the set is to be interpreted as meaning each of the items singularly as alternatives, as well as any combination of the listed items.

Referring now to FIG. 1 , a system for large language model (LLM) orchestrated data search across public and private data sources is depicted in accordance with one or more embodiments of the present disclosure.

In some embodiments, a user may interact with an LLM orchestrated data search platform 110 via a graphical user interface (GUI) of a user computing device 140 . In some embodiments, the LLM orchestrated data search platform 110 may leverage a model orchestration LLM 114 to orchestrate data retrieval and data processing of one or more data sources 120 and/or data record processing machine learning (ML) agents 130 . In some embodiments, the user may query the LLM orchestrated data search platform 110 for information associated with one or more domains, such as financial instruments, medical information, patient data, scientific information, personal data, business data, among other information and/or data associated with one or more domains or any combination thereof. In some embodiments, to improve accuracy of the data as well as presentation in the GUI, a context engine 112 may inject context data into the model orchestration LLM 114 based on the identify of the user, the user's query, among other factors or any combination thereof. The model orchestration LLM 114 may then task the data sources 120 and/or agent(s) 130 to search, generate, transform or otherwise act on the query to obtain a response 102 to the user's query. A compliance agent 116 may verify the response 102 for compliance to one or more rules defining one or more standards, regulations, guidelines or other rules or any combination thereof. The verified response may then be returned to the user via the GUI of the user computing device 140 to provide the user with compliant, context-dependent information.

In some embodiments, the LLM orchestrated data search platform 110 may include hardware components such as a processor 118 , which may include local or remote processing components. In some embodiments, the processor 118 may include any type of data processing capacity, such as a hardware logic circuit, for example an application specific integrated circuit (ASIC) and a programmable logic, or such as a computing device, for example, a microcomputer or microcontroller that include a programmable microprocessor. In some embodiments, the processor 118 may include data-processing capacity provided by the microprocessor. In some embodiments, the microprocessor may include memory, processing, interface resources, controllers, and counters. In some embodiments, the microprocessor may also include one or more programs stored in memory.

In some embodiments, the LLM orchestrated data search platform 110 may include data sources 120 including privately and/or publicly accessible information retrieval tools. In some embodiments, the data sources 120 may include one or more local and/or remote data storage solutions such as, e.g., local hard-drive, solid-state drive, flash drive, database or other local data storage solutions or any combination thereof, and/or remote data storage solutions such as a server, mainframe, database or cloud services, distributed database or other suitable data storage solutions or any combination thereof. In some embodiments, the data sources 120 may include, e.g., a suitable non-transient computer readable medium such as, e.g., random access memory (RAM), read only memory (ROM), one or more buffers and/or caches, among other memory devices or any combination thereof.

For example, the data sources 120 may include at least one database 122 . The database may include a database model formed by one or more formal design and modeling techniques. The database model may include, e.g., a navigational database, a hierarchical database, a network database, a graph database, an object database, a relational database, an object-relational database, an entity-relationship database, an enhanced entity-relationship database, a document database, an entity-attribute-value database, a star schema database, or any other suitable database model and combinations thereof. For example, the database may include database technology such as, e.g., a centralized or distributed database, cloud storage platform, decentralized system, server or server system, among other storage systems. In some embodiments, the database may, additionally or alternatively, include one or more data storage devices such as, e.g., a hard drive, solid-state drive, flash drive, or other suitable storage device. In some embodiments, the database may, additionally or alternatively, include one or more temporary storage devices such as, e.g., a random-access memory, cache, buffer, or other suitable memory device, or any other data storage solution and combinations thereof.

Depending on the database model, one or more database query languages may be employed to retrieve data from the database. Examples of database query languages may include: JSONiq, LDAP, Object Query Language (OQL), Object Constraint Language (OCL), PTXL, QUEL, SPARQL, SQL, XQuery, Cypher, DMX, FQL, Contextual Query Language (CQL), AQL, among suitable database query languages.

In some embodiments, the data sources 120 may include one or more local and/or remote data integrations, such as a software service including, e.g., a security service 124 , a cloud service 126 , an internet-based content and/or information service via HTTP 128 , among others or any combination thereof. In some embodiments, the LLM orchestrated data search platform 110 may interface with the data sources 120 via one or more computer interfaces. In some embodiments, the computer interfaces may utilize one or more software computing interface technologies, such as, e.g., an application programming interface (API) and/or application binary interface (ABI), among others or any combination thereof. In some embodiments, an API and/or ABI defines the kinds of calls or requests that can be made, how to make the calls, the data formats that should be used, the conventions to follow, among other requirements and constraints. An “application programming interface” or “API” can be entirely custom, specific to a component, or designed based on an industry-standard to ensure interoperability to enable modular programming through information hiding, allowing users to use the interface independently of the implementation.

In some embodiments, the LLM orchestrated data search platform 110 may be implemented as a centralized computing system, a computing device, a distributed computing system, a cloud hosted service and/or platform, a server-hosted platform, a hybrid cloud and local computing system, or any combination thereof. As used herein, terms “cloud,” “Internet cloud,” “cloud computing,” “cloud architecture,” and similar terms correspond to at least one of the following: (1) a large number of computers connected through a real-time communication network (e.g., Internet); (2) providing the ability to run a program or application on many connected computers (e.g., physical machines, virtual machines (VMs)) at the same time; (3) network-based services, which appear to be provided by real server hardware, and are in fact served up by virtual hardware (e.g., virtual servers), simulated by software running on one or more real machines (e.g., allowing to be moved around and scaled up (or down) on the fly without affecting the end user). The aforementioned examples are, of course, illustrative and not restrictive.

In some embodiments, the LLM orchestrated data search platform 110 may implement computer engines for the context engine 112 , the model orchestration LLM 114 , the compliance agent 116 and/or the agent(s) 130 . In some embodiments, the terms “computer engine” and “engine” identify at least one software component and/or a combination of at least one software component and at least one hardware component which are designed/programmed/configured to manage/control other software and/or hardware components (such as the libraries, software development kits (SDKs), objects, etc.).

Examples of hardware elements may include processors, microprocessors, circuits, circuit elements (e.g., transistors, resistors, capacitors, inductors, and so forth), integrated circuits, application specific integrated circuits (ASIC), programmable logic devices (PLD), digital signal processors (DSP), field programmable gate array (FPGA), logic gates, registers, semiconductor device, chips, microchips, chip sets, and so forth. In some embodiments, the one or more processors may be implemented as a Complex Instruction Set Computer (CISC) or Reduced Instruction Set Computer (RISC) processors; x86 instruction set compatible processors, multi-core, or any other microprocessor or central processing unit (CPU). In various implementations, the one or more processors may be dual-core processor(s), dual-core mobile processor(s), and so forth. In some embodiments, the computer engine(s) may include dedicated and/or shared software components, hardware components, or a combination thereof.

Examples of software may include software components, programs, applications, computer programs, application programs, system programs, machine programs, operating system software, middleware, firmware, software modules, routines, subroutines, functions, methods, procedures, software interfaces, application program interfaces (API), instruction sets, computing code, computer code, code segments, computer code segments, words, values, symbols, or any combination thereof. Determining whether an embodiment is implemented using hardware elements and/or software elements may vary in accordance with any number of factors, such as desired computational rate, power levels, heat tolerances, processing cycle budget, input data rates, output data rates, memory resources, data bus speeds and other design or performance constraints.

In some embodiments, the context engine 112 may dynamically contextually inject data at runtime into the model orchestration LLM 114 , the agent(s) 130 , or a combination thereof. The contextual data may be based on the user, security parameters or other data or any combination thereof. For example, the contextual data may include the user's role in an organization, a user type, a persona representative of a style of communication, a skill-level in one or more domains, domain-specific data, inferences from others of the model orchestration LLM 114 and/or agent(s) 130 , among other data or any combination thereof. For example, the user may be querying financial instrument information associated with a financial instrument, a market, or other information, thus the contextual data may include bond data, market data, security and exchange commission (SEC) filing data, user activity, user persona/role information, among other data or any combination thereof.

In some embodiments, the contextual data may be stored in a user profile associated with the user, e.g., in the database 122 or other data store. In some embodiments, the contextual data may be retrieved via a query to a private and/or public data source 120 , such as the Internet via HTTP 128 , or any other suitable private and/or public data source or any combination thereof. For example, the security parameters may include, e.g., user location, user department, user firm, user entitlements, user job function, time of day, among others or any combination thereof, which may be used to determine an amount and type of information that the user may access.

Thus, in some embodiments, based on the user's role and persona, the context engine 112 retrieves data (over the internet, over local, over cloud) from multiple sources (API, flat files, databases) and inserts it into the model orchestration LLM 114 runtime context. In some embodiments, the injection may be done in memory and specific to the user's session. As a result, the context engine 112 prevents leakage of data to other users of the LLM orchestrated data search platform 110 since it is only dynamically retrieved at run time and specific to the user's commands.

In some embodiments, the context engine 112 may dynamically control the context kept in the runtime. Thus, in some embodiments, the context engine 112 may systematically reduce the context input into the model orchestration LLM 114 while preserving the meaning of context. In some embodiments, the model orchestration LLM 114 may have a limited context size, including both hard limits (there is a hard ceiling), and also semantic cognitive limits (e.g., the more in the context, the less attention is direct to the context). In some embodiments, to optimize the resource use and attention of the model orchestration LLM 114 , the context engine 112 may dynamically reduce context size to balance continuity for the user with attention in the model orchestration LLM 114 . To do so, the context engine 112 may review the conversation occurring via the GUI and determine how much conversation history is required to maintain the conversation. The context engine 112 performs this review dynamically and in conjunction with the injection of context data described above to enable seamless conversation across hundreds of messages or more.

In some embodiments, the balancing is configured to optimize based on one or more parameters, such as, e.g., recency (memory depth), stated importance, model capability, cost, attribute(s) of the conversation (e.g., industry, sector, investment grade versus high yield, or other attributes of the financial investment related conversations or any combination thereof) among others or any combination thereof. In some embodiments, recency refers to how recent the data is to be kept in the runtime. For example if the user asked for high yield bonds, and then next question ‘just above 5% yield’, it should know high yield bonds just above 5%. But if the user asked high yield bonds last week, it should not maintain that query in memory. In some embodiments, stated Importance refers to where the user inputs something that is clear in intent, the input is prioritized by applying a weighting based on the stated importance. In some embodiments, model capability refers to the limit of the context size available to the model orchestration LLM 114 , such as, e.g., 4 kB, 8 kB, 16 kB or other limit to the amount that the memory can maintain. In some embodiments, the cost refers to the cost profiles for maintaining and running the model orchestration LLM 114 and/or the LLM orchestrated data search platform 110 .

In some embodiments, the model orchestration LLM 114 may receive the natural language query from the user and the context from the context engine 112 . In some embodiments, the GUI may also include structured commands (e.g., via user interface elements including buttons, toggles, switches, multiple choice selections, filters, etc.). The structured commands may be provided with the natural language query.

In some embodiments, based on the input, the model orchestration LLM 114 may load what the user and process conversation has been to the point in time into context, and compress what the user and process conversation has been to the point in time to optimize what is put into context as necessary. In some embodiments, given the above, the model orchestration LLM 114 may determine the type of response that is required, such as what data sources 120 and/or agent(s) 130 may fulfill the response 102 . In some embodiments, given the type or response, the model orchestration LLM 114 may generate one or more retrieval instructions for each data source 120 and/or agent(s) 130 that may provide all or a portion of the response 102 . In some embodiments, the model orchestration LLM 114 may collate the retrieved data and generate a natural language response 102 answering the user's query.

In some embodiments, to do so, the model orchestration LLM 114 may dynamically determine which sub-process (e.g., which agent(s) 130 ) to use to fulfill a user's inferred request. In some embodiments, to improve for scalability (in runtime performance, and also in expandability of the underlying model), the LLM orchestrated data search platform 110 may be configured to run the agent(s) 130 in parallel, enabling more agents to be created but also for agents to cooperate and work with each other, and agents that are adversarial and work against each other for the best result. In some embodiment, the agents 130 may be machine learning (ML)-based agents, an API call to an external service, a function call within a process, or a query, or other machine learning, statistical, programmatic and/or rules-based process or any combination thereof.

For example, in some embodiments, the model orchestration LLM 114 may generate instructions for a “sector search” agent that works with a ‘cognition” agent to refine the user's request to search for the right market sectors, industries, and groupings most appropriate to the user's request for financial instrument information. Another example may include an “internet” agent that is able to retrieve real time data from internet sources such as via an API, and in conjunction with the above method, copy, move, download or other otherwise obtain the real-time data for use by the LLM orchestrated data search platform 110 .

In some embodiments, the agents 130 and/or the model orchestration LLM 114 may include functionality for explainability (e.g., “Show Your Work”) based on explainability programming. Machine learning models, such as neural networks and LLM's, among others, struggle with explainability, including explaining why a model inferred a specific output. In some embodiments, the agents 130 and/or the model orchestration LLM 114 may include dynamic, systematically created explainability suitable for the end user, e.g., an employee at a financial institution. In some embodiments, the explainability programming may include a method to create text explaining inner workings of the model orchestration LLM 114 . Given a set of input text, the explainability functionality explains in text in language specific to the user's persona and security status the specific steps taken by the model orchestration LLM 114 (across multiple agents 130 when applicable) to fulfill the user's answers.

In some embodiments, the model orchestration LLM 114 may include explainability capabilities according to explainability programming. To do so, the model orchestration LLM 114 may track the instructions output in response to the user's query and dynamically and systematicall

CLAIMS

Claims ( 20 )

What is claimed is:

1. A method comprising:

receiving, by at least one processor from at least one user computing device associated with a user, a user-provided data record query comprising a natural language request to perform at least one action with at least one data record;

wherein the at least one data record comprises at least one attribute associated with at least one financial instrument;

inputting, by the at least one processor, the natural language request of the data record query as a data record query prompt into a model orchestration large language model to output at least one instruction to at least one data record processing machine learning agent of a plurality of data record processing machine learning agents based at least in part on trained parameters of the model orchestration large language model;

wherein the at least one data record processing machine learning agent comprises at least one similarity-based data record identification machine learning model configured to determine a metric between at least two financial instruments based at least in part on at least one attribute of each financial instrument;

inputting, by the at least one processor, the at least one instruction, including the at least one parameter of the at least one financial instrument, into the at least one data record processing machine learning agent to output at least one response;

wherein the at least one response comprises at least one similar data record of at least one similar financial instrument based at least in part on at least one metric-based data record identification machine learning agent determining a metric between the at least one data record and the at least one similar data record;

determining, by the at least one processor, at least one trigger characteristic associated with the at least one data record based at least in part on a metric associated with the at least one financial instrument and at least one value associated with the at least one similar financial instrument;

inputting, by the at least one processor, the at least one response, the at least one trigger characteristic-, or both, as at least one response prompt into the model orchestration large language model to output at least one natural language response representative of the at least one trigger characteristic-based at least in part on trained parameters of the model orchestration large language model;

inputting, by the at least one processor, into an adversarial machine learning agent, the at least one natural language response representative of the at least one trigger characteristic to output at least one auto-corrected natural language response, by recursively modifying the at least one natural language response until at least one predetermined criteria is met;

wherein the adversarial machine learning agent comprises:

at least one adversarial machine learning model trained to determine when the at least one natural language response meets the at least one predetermined criteria, and

at least one natural language response modification component configured to modify the at least one natural language response when the at least one adversarial machine learning model determines that the at least one natural language response fails the at least one predetermined criteria; and

causing to display, by the at least one processor, the at least one natural language response in a graphical user interface (GUI) rendered on the user computing device.

2. The method of claim 1 , wherein the plurality of data record processing machine learning agents are configured to be instantiated in parallel.

3. The method of claim 1 , further comprising:

storing, by the at least one processor, at least one record of the at least one instruction output by the model orchestration large language model;

determining, by the at least one processor, the at least one data record processing machine learning agent associated with the at least one instruction;

determining, by the at least one processor, the at least one response associated with the at least one data record processing machine learning agent;

generating, by the at least one processor, at least one explainability prompt configured to elicit a natural language explanation from the model orchestration large language model;

inputting, by the at least one processor, the at least one explainability prompt, the at least one record of the at least one instruction, the at least one data record processing machine learning agent and the at least one response into the model orchestration large language model to output the natural language explanation based at least in part on trained parameters of the model orchestration large language model; and

causing to display, by the at least one processor, the natural language explanation in the GUI rendered on the user computing device.

4. The method of claim 1 , wherein the at least one data record processing machine learning agent is at least two data record processing machine learning agents;

wherein at least one first data record processing machine learning agent is adversarial to at least one second data record processing machine learning agent.

5. The method of claim 4 , wherein the at least one first data record processing machine learning agent is configured to output the at least one response; and

wherein the at least one second data record processing machine learning agent is configured to:

ingest the at least one response from the at least one first data record processing machine learning agent;

determine a correctness assessment based at least in part on correctness assessment machine learning parameters; and

refine the at least one response based at least in part on the correctness assessment.

6. The method of claim 1 , further comprising:

inputting, by the at least one processor, at least one compliance rule into at least one compliance verification machine learning agent to output at least one compliance verification prompt based at least in part on a plurality of compliance verification parameters;

wherein the at least one compliance verification prompt is configured to cause the model orchestration large language model to verify compliance of the at least one natural language response with the at least one compliance rule; and

inputting, by the at least one processor, the at least one compliance verification prompt into the model orchestration large language model to output at least one compliance verification of the at least one natural language model based at least in part on the trained parameters of the model orchestration large language model.

7. The method of claim 6 , further comprising:

inputting, by the at least one processor, based on the at least one compliance verification being representative of the at least one natural language response being non-compliant, the natural language response into at least one compliance machine learning agent to output a variation to the natural language response; and

inputting, by the at least one processor, the at least one compliance verification prompt and the variation to the natural language response into the model orchestration large language model to output at least one new compliance verification of the variation to the at least one natural language model based at least in part on the trained parameters of the model orchestration large language model.

8. The method of claim 6 , wherein the at least one compliance verification comprises at least one of:

a pass indicative of the at least one natural language response being compliant, or

a fail indicative of the at least one natural language response being non-compliant.

9. The method of claim 1 , wherein the at least one instruction comprises at least one programmatic step comprising at least one of:

at least one database query,

at least one application programming interface (API) call, or

at least one internet search query.

10. The method of claim 9 , further comprising:

tracking, by the at least one processor, the at least one instruction;

generating, by the at least one processor, at least one model explainability prompt representative of the at least one instruction;

wherein the at least one model explainability prompt is configured to cause the model orchestration large language model to output a natural language explanation of at least one of:

the at least one data record processing machine learning agent, or

the at least one programmatic step; and

inputting, by the at least one processor, the at least one model explainability prompt into the model orchestration large language model to output the natural language explanation based at least in part on the trained parameters of the model orchestration large language model.

11. A method comprising:

receiving, by at least one processor from at least one user computing device associated with a user, a user-provided data record query comprising a natural language request to perform at least one action with at least one data record;

wherein the at least one data record comprises at least one attribute associated with at least one financial instrument;

inputting, by the at least one processor, the natural language request of the data record query as a data record query prompt into a model orchestration large language model to output at least one instruction to at least one data record processing machine learning agent of a plurality of data record processing machine learning agents based at least in part on trained parameters of the model orchestration large language model;

wherein the at least one data record processing machine learning agent comprises at least one similarity-based data record identification machine learning model configured to determine a similarity between a plurality of financial instruments based at least in part on at least one attribute of each financial instrument;

inputting, by the at least one processor, the at least one instruction, including the at least one parameter of the at least one financial instrument, into the at least one data record processing machine learning agent to output at least one response;

wherein the at least one response comprises at least one similar data record of at least one similar financial instrument based at least in part on the at least one similarity-based data record identification machine learning model determining a similarity between the at least one data record and the at least one similar data record;

determining, by the at least one processor, at least one error associated with the at least one financial instrument and the at least one similar financial instrument;

inputting, by the at least one processor, the at least one response, the at least one error, or both, as at least one response prompt into the model orchestration large language model to output at least one natural language response representative of the at least one error based at least in part on trained parameters of the model orchestration large language model;

inputting, by the at least one processor, into an adversarial machine learning agent, the at least one natural language response representative of the at least one error to output at least one auto-corrected natural language response, by recursively modifying the at least one natural language response until at least one predetermined criteria is met;

wherein the adversarial machine learning agent comprises;

at least one adversarial machine learning model trained to determine when the at least one natural language response meets the at least one predetermined criteria, and

at least one natural language response modification component configured to modify the at least one natural language response when the at least one adversarial machine learning model determines that the at least one natural language response fails the at least one predetermined criteria; and

facilitating, by the at least one processor, at least one action in response to the at least one error based at least in part on the at least one natural language response in a graphical user interface (GUI) rendered on the user computing device.

12. The method of claim 11 , wherein the plurality of data record processing machine learning agents are configured to be instantiated in parallel.

13. The method of claim 11 , further comprising:

storing, by the at least one processor, at least one record of the at least one instruction output by the model orchestration large language model;

determining, by the at least one processor, the at least one data record processing machine learning agent associated with the at least one instruction;

determining, by the at least one processor, the at least one response associated with the at least one data record processing machine learning agent;

generating, by the at least one processor, at least one explainability prompt configured to elicit a natural language explanation from the model orchestration large language model;

inputting, by the at least one processor, the at least one explainability prompt, the at least one record of the at least one instruction, the at least one data record processing machine learning agent and the at least one response into the model orchestration large language model to output the natural language explanation based at least in part on trained parameters of the model orchestration large language model; and

causing to display, by the at least one processor, the natural language explanation in the GUI rendered on the user computing device.

14. The method of claim 11 , wherein the at least one data record processing machine learning agent is at least two data record processing machine learning agents;

wherein at least one first data record processing machine learning agent is adversarial to at least one second data record processing machine learning agent.

15. The method of claim 14 , wherein the at least one first data record processing machine learning agent is configured to output the at least one response; and

wherein the at least one second data record processing machine learning agent is configured to:

ingest the at least one response from the at least one first data record processing machine learning agent;

determine a correctness assessment based at least in part on correctness assessment machine learning parameters; and

refine the at least one response based at least in part on the correctness assessment.

16. The method of claim 11 , further comprising:

inputting, by the at least one processor, at least one compliance rule into at least one compliance verification machine learning agent to output at least one compliance verification prompt based at least in part on a plurality of compliance verification parameters;

wherein the at least one compliance verification prompt is configured to cause the model orchestration large language model to verify compliance of the at least one natural language response with the at least one compliance rule; and

inputting, by the at least one processor, the at least one compliance verification prompt into the model orchestration large language model to output at least one compliance verification of the at least one natural language model based at least in part on the trained parameters of the model orchestration large language model.

17. The method of claim 16 , further comprising:

inputting, by the at least one processor, based on the at least one compliance verification being representative of the at least one natural language response being non-compliant, the natural language response into at least one compliance machine learning agent to output a variation to the natural language response; and

inputting, by the at least one processor, the at least one compliance verification prompt and the variation to the natural language response into the model orchestration large language model to output at least one new compliance verification of the variation to the at least one natural language model based at least in part on the trained parameters of the model orchestration large language model.

18. The method of claim 11 , wherein the at least one instruction comprises at least one programmatic step comprising at least one of:

at least one database query,

at least one application programming interface (API) call, or

at least one internet search query.

19. The method of claim 18 , further comprising:

tracking, by the at least one processor, the at least one instruction;

generating, by the at least one processor, at least one model explainability prompt representative of the at least one instruction;

wherein the at least one model explainability prompt is configured to cause the model orchestration large language model to output a natural language explanation of at least one of:

the at least one data record processing machine learning agent, or

the at least one programmatic step; and

inputting, by the at least one processor, the at least one model explainability prompt into the model orchestration large language model to output the natural language explanation based at least in part on the trained parameters of the model orchestration large language model.

20. A system comprising:

at least one processor in communication with at least one non-transitory computer-readable medium having computer instructions stored thereon, wherein, upon execution of the computer instructions, the at least one processor is configured to perform steps comprising:

receiving, from at least one user computing device associated with a user, a user-provided data record query comprising a natural language request to perform at least one action with at least one data record;

wherein the at least one data record comprises at least one attribute associated with at least one financial instrument;

inputting the natural language request of the data record query as a data record query prompt into a model orchestration large language model to output at least one instruction to at least one data record processing machine learning agent of a plurality of data record processing machine learning agents based at least in part on trained parameters of the model orchestration large language model;

wherein the at least one data record processing machine learning agent comprises at least one similarity-based data record identification machine learning model configured to determine a metric between at least two financial instruments based at least in part on at least one attribute of each financial instrument;

inputting the at least one instruction, including the at least one parameter of the at least one financial instrument, into the at least one data record processing machine learning agent to output at least one response;

wherein the at least one response comprises at least one similar data record of at least one similar financial instrument based at least in part on at least one metric-based data record identification machine learning agent determining a metric between the at least one data record and the at least one similar data record;

determining at least one trigger characteristic associated with the at least one data record based at least in part on a metric associated with the at least one financial instrument and at least one value associated with the at least one similar financial instrument;

inputting the at least one response, the at least one trigger characteristic, or both, as at least one response prompt into the model orchestration large language model to output at least one natural language response representative of the at least one trigger characteristic based at least in part on trained parameters of the model orchestration large language model;

inputting by the at least one processor into an adversarial machine learning agent the at least one natural language response representative of the at least one trigger characteristic to output at least one auto-corrected natural language response, by recursively modifying the at least one natural language response until at least one predetermined criteria is met;

wherein the adversarial machine learning agent comprises:

at least one adversarial machine learning model trained to determine when the at least one natural language response meets the at least one predetermined criteria, and

at least one natural language response modification component configured to modify the at least one natural language response when the at least one adversarial machine learning model determines that the at least one natural language response fails the at least one predetermined criteria; and

causing to display the at least one natural language response in a graphical user interface (GUI) rendered on the user computing device.

US18/792,157

2023-10-06

2024-08-01

Systems and methods of large language model driven orchestration of task-specific machine learning software agents

Active

US12307349B2

( en )

Priority Applications (2)

Application Number

Priority Date

Filing Date

Title

US18/792,157

US12307349B2

( en )

2023-10-06

2024-08-01

Systems and methods of large language model driven orchestration of task-specific machine learning software agents

US19/184,459

US20250252293A1

( en )

2023-10-06

2025-04-21

Systems and methods of large language model driven orchestration of task-specific machine learning software agents

Applications Claiming Priority (2)

Application Number

Priority Date

Filing Date

Title

US18/482,731

US12061970B1

( en )

2023-10-06

2023-10-06

Systems and methods of large language model driven orchestration of task-specific machine learning software agents

US18/792,157

US12307349B2

( en )

2023-10-06

2024-08-01

Systems and methods of large language model driven orchestration of task-specific machine learning software agents

Related Parent Applications (1)

Application Number

Title

Priority Date

Filing Date

US18/482,731

Continuation

US12061970B1

( en )

2023-10-06

2023-10-06

Systems and methods of large language model driven orchestration of task-specific machine learning software agents

Related Child Applications (1)

Application Number

Title

Priority Date

Filing Date

US19/184,459

Continuation

US20250252293A1

( en )

2023-10-06

2025-04-21

Systems and methods of large language model driven orchestration of task-specific machine learning software agents

Publications (2)

Publication Number

Publication Date

US20250117627A1

US20250117627A1 ( en )

2025-04-10

US12307349B2

true

US12307349B2 ( en )

2025-05-20

Family

ID=92217361

Family Applications (3)

Application Number

Title

Priority Date

Filing Date

US18/482,731

Active

US12061970B1

( en )

2023-10-06

2023-10-06

Systems and methods of large language model driven orchestration of task-specific machine learning software agents

US18/792,157

Active

US12307349B2

( en )

2023-10-06

2024-08-01

Systems and methods of large language model driven orchestration of task-specific machine learning software agents

US19/184,459

Pending

US20250252293A1

( en )

2023-10-06

2025-04-21

Systems and methods of large language model driven orchestration of task-specific machine learning software agents

Family Applications Before (1)

Application Number

Title

Priority Date

Filing Date

US18/482,731

Active

US12061970B1

( en )

2023-10-06

2023-10-06

Systems and methods of large language model driven orchestration of task-specific machine learning software agents

Family Applications After (1)

Application Number

Title

Priority Date

Filing Date

US19/184,459

Pending

US20250252293A1

( en )

2023-10-06

2025-04-21

Systems and methods of large language model driven orchestration of task-specific machine learning software agents

Country Status (3)

Country

Link

US

( 3 )

US12061970B1

( en )

AU

( 1 )

AU2024355938A1

( en )

WO

( 1 )

WO2025076107A1

( en )

Families Citing this family (39)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US12596813B2

( en )

2023-01-19

2026-04-07

Citibank, N.A

Autonomous agent observation and control

US12511282B1

( en )

2023-05-02

2025-12-30

Microstrategy Incorporated

Generating structured query language using machine learning

US20240380653A1

( en )

*

2023-05-10

2024-11-14

Edgio, Inc.

Network configuration using natural language

EP4690015A1

( en )

*

2023-07-05

2026-02-11

Neoptio Health Incorporated

Methods, systems, apparatuses, and devices for facilitating customizable communications with users

US20250123894A1

( en )

*

2023-10-17

2025-04-17

Salesforce, Inc.

Autonomous execution of unstructured compute processes

US20250131190A1

( en )

*

2023-10-23

2025-04-24

Qualcomm Incorporated

Contextualized Language Model Prompt With Sensing Hub

US12585889B2

( en )

*

2023-12-14

2026-03-24

Amazon Technologies, Inc.

Natural language generation

US20250217403A1

( en )

*

2024-01-03

2025-07-03

Optum, Inc.

Machine learning techniques for question resolution

US20250238305A1

( en )

*

2024-01-19

2025-07-24

Dell Products L.P.

Managing data processing system failures using visualizations of hidden knowledge from predictive models

US20250238306A1

( en )

*

2024-01-19

2025-07-24

Dell Products L.P.

Interactive data processing system failure management using hidden knowledge from predictive models

US12613764B2

( en )

2024-01-19

2026-04-28

Dell Products L.P.

Managing data processing system failures using a predictive model as a controller and hidden knowledge from predictive models

US12493512B2

( en )

2024-01-19

2025-12-09

Dell Products L.P.

Managing data processing system failures using hidden knowledge from predictive models for failure response generation

US12602277B2

( en )

2024-01-19

2026-04-14

Dell Products L.P.

Managing data processing system failures using hidden knowledge from predictive models for failure response generation

US12332755B1

( en )

2024-01-19

2025-06-17

Dell Products L.P.

Managing data processing system failures using citations generated based on hidden knowledge from predictive models

US20250238307A1

( en )

*

2024-01-19

2025-07-24

Dell Products L.P.

Interactive data processing system failure management using hidden knowledge from predictive models

US12254053B1

( en )

*

2024-03-12

2025-03-18

Capital One Services, Llc

Computer-based systems configured for generating search queries for a search engine and methods of use thereof

US12489766B2

( en )

*

2024-03-14

2025-12-02

SecurityScorecard, Inc.

Cybersecurity event detection, analysis, and integration from multiple sources

US20250298792A1

( en )

*

2024-03-22

2025-09-25

Palo Alto Networks, Inc.

Grammar powered retrieval augmented generation for domain specific languages

US20250342181A1

( en )

*

2024-05-02

2025-11-06

International Business Machines Corporation

Ranking-augmented generation for long documents

US20250370846A1

( en )

*

2024-06-04

2025-12-04

Insight Direct Usa, Inc.

Automated diagnostic plan generation for technical support

US12135949B1

( en )

*

2024-06-07

2024-11-05

Citibank, N.A.

Layered measurement, grading and evaluation of pretrained artificial intelligence models

US12493772B1

( en )

2024-06-07

2025-12-09

Citibank, N.A.

Layered multi-prompt engineering for pre-trained large language models

US12154019B1

( en )

2024-06-07

2024-11-26

Citibank, N.A.

System and method for constructing a layered artificial intelligence model

US20250390515A1

( en )

*

2024-06-21

2025-12-25

Intuit Inc.

Query augmentation

US20260004115A1

( en )

*

2024-06-27

2026-01-01

Meltwater News Us Inc.

Spike analysis with contextually informed spike explanations generated by use of large language models

US12436995B1

( en )

2024-06-28

2025-10-07

Dell Products L.P.

Interactive user assistance system with rich multimodal content experience

CN118656385B

( en )

*

2024-08-20

2024-11-26

广州思迈特软件有限公司

A data query method based on large model

CN118673102B

( en )

*

2024-08-23

2024-11-05

清华大学

Human movement behavior generation method and system based on large language model

EP4703916A1

( en )

*

2024-08-30

2026-03-04

DocuSign, Inc.

Multi-agent reports for electronic documents

US20260065395A1

( en )

*

2024-08-30

2026-03-05

Docusign, Inc.

Multi-agent reports for electronic documents

KR20260031831A

( en )

*

2024-08-30

2026-03-09

라인플러스 주식회사

Method and system for configuring multi-agent based on retrieving agent

CN118897198B

( en )

*

2024-10-09

2025-02-11

山东大学

Battery health state estimation method and system based on pre-trained large language model

CN119045928B

( en )

*

2024-10-29

2025-03-25

中科先进(深圳)集成技术有限公司

Data question answering system and calling method thereof

CN119476492B

( en )

*

2024-11-13

2025-06-10

华东师范大学

Multi-agent reasoning framework construction method based on meta-cognition thought

US12579594B1

( en )

*

2024-11-22

2026-03-17

BH Operations, LLC

Systems and methods for dynamically updating data for course generation

US12476993B1

( en )

*

2024-12-31

2025-11-18

Cyberark Software Ltd.

Detection and prevention of adversarial attacks against large language models

US12481835B1

( en )

*

2025-03-19

2025-11-25

Character Technologies Inc.

Character turn summaries

US12417250B1

( en )

2025-03-27

2025-09-16

Morgan Stanley Services Group Inc.

Processing user input to a computing environment using artificial intelligence

US12536212B1

( en )

*

2025-05-29

2026-01-27

Aj Press, Llc

Systems and methods for error correction in large language model-based database querying

Citations (4)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20220067604A1

( en )

2020-08-26

2022-03-03

Accenture Global Solutions Limited

Utilizing machine learning models to aggregate applications and users with events associated with the applications

US20230114826A1

( en )

2021-10-12

2023-04-13

Virtuous AI, Inc.

Ai platform with customizable content analysis control panel and methods for use therewith

US20230141408A1

( en )

*

2021-11-08

2023-05-11

Accenture Global Solutions Limited

Utilizing machine learning and natural language generation models to generate a digitized dynamic client solution

US20230360388A1

( en )

2020-10-14

2023-11-09

UiPath, Inc.

Training a generative artificial intelligence / machine learning model to recognize applications, screens, and user interface elements using computer vision

2023

2023-10-06

US

US18/482,731

patent/US12061970B1/en

active

Active

2024

2024-08-01

US

US18/792,157

patent/US12307349B2/en

active

Active

2024-10-02

AU

AU2024355938A

patent/AU2024355938A1/en

active

Pending

2024-10-02

WO

PCT/US2024/049620

patent/WO2025076107A1/en

active

Pending

2025

2025-04-21

US

US19/184,459

patent/US20250252293A1/en

active

Pending

Patent Citations (4)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20220067604A1

( en )

2020-08-26

2022-03-03

Accenture Global Solutions Limited

Utilizing machine learning models to aggregate applications and users with events associated with the applications

US20230360388A1

( en )

2020-10-14

2023-11-09

UiPath, Inc.

Training a generative artificial intelligence / machine learning model to recognize applications, screens, and user interface elements using computer vision

US20230114826A1

( en )

2021-10-12

2023-04-13

Virtuous AI, Inc.

Ai platform with customizable content analysis control panel and methods for use therewith

US20230141408A1

( en )

*

2021-11-08

2023-05-11

Accenture Global Solutions Limited

Utilizing machine learning and natural language generation models to generate a digitized dynamic client solution

Non-Patent Citations (5)

* Cited by examiner, † Cited by third party

Title

Chen et al., " NetGPT: An Al-Native Network Architecture for Provisioning Beyond Personalized Generative Services, " arXiv preprint arXiv:2311/05842 (2023) pp. 1-7.

Chen et al., " NewGPT: An AI-Native Network Architecture for Provisioning Beyond Personalized Generative Services ", IEEE Network, Jul. 12, 2023.

Chen, Yuxuan, et al. " Netgpt: A native-ai network architecture beyond provisioning personalized generative services. " arXiv preprint arXiv:2307.06148 (2023). (Year: 2023).

*

International Search Report and Written Opinion from International Application No. PCT/US2024/049620 dated Oct. 25, 2024.

Tarkoma et al., " AI-native Interconnect Framework for Integration of Large Language Model Technologies in 6G Systems, " arXiv preprint arXiv:2311.05842 (2023).

Also Published As

Publication number

Publication date

AU2024355938A1

( en )

2026-04-23

US20250117627A1

( en )

2025-04-10

WO2025076107A1

( en )

2025-04-10

US12061970B1

( en )

2024-08-13

US20250252293A1

( en )

2025-08-07

Similar Documents

Publication

Publication Date

Title

US12061970B1

( en )

2024-08-13

Systems and methods of large language model driven orchestration of task-specific machine learning software agents

US20240232843A1

( en )

2024-07-11

Machine-learning based electronic activity accuracy verification and detection of anomalous attributes and methods thereof

US11995054B2

( en )

2024-05-28

Machine-learning based data entry duplication detection and mitigation and methods thereof

US20210241040A1

( en )

2021-08-05

Systems and Methods for Ground Truth Dataset Curation

US12361209B2

( en )

2025-07-15

Systems for database searching and database schemas management and methods of use thereof

US11783272B2

( en )

2023-10-10

Systems for implementing a permission-based automatic update of physical addresses in a database and methods of use thereof

US20220343288A1

( en )

2022-10-27

Computer systems for database management based on data pattern recognition, prediction and recommendation and methods of use thereof

US20210334707A1

( en )

2021-10-28

System and method for managing classification outcomes of data inputs classified into bias categories

US12536266B2

( en )

2026-01-27

Systems and methods for content selections for securing communications between agent devices and user devices

US12493505B2

( en )

2025-12-09

Systems and methods for data lineage-based system anomaly detection

US20260127643A1

( en )

2026-05-07

Computer-based systems configured for identifying restricted reversals of operations in data entry and methods of use thereof

US20250181914A1

( en )

2025-06-05

Computer-based systems configured for detecting and splitting data types in a data file and methods of use thereof

WO2022226270A2

( en )

2022-10-27

Systems for automatic detection, rating and recommendation of entity records and methods of use thereof

US20230156123A1

( en )

2023-05-18

Computer-based systems configured for automated extraction of data from inputs and methods of use thereof

US20260050975A1

( en )

2026-02-19

Systems and methods for artificial intelligence-based anomaly search in electronic records

US12287890B2

( en )

2025-04-29

Computer-based systems configured for efficient and secure multi-user electronic activities and methods of use thereof

US12339774B2

( en )

2025-06-24

Automated predictive caching of cloud-sourced data and methods of use thereof

US20260064633A1

( en )

2026-03-05

Computer-based platforms/systems/devices/components and/or objects configured for generating values for data structures and methods of use thereof

US12505104B2

( en )

2025-12-23

Computer-based systems configured for novel information processing and storage utilizing elastic search environment of a local cache and method and use thereof

US20240338696A1

( en )

2024-10-10

Computer-based systems configured to automatically program a plurality of controls to modify a communication session associated with a transaction and methods of use thereof

US20250291649A1

( en )

2025-09-18

Computer-based systems configured for a cloud-first multifunction api cluster providing microservices capable of both batch and real-time processing and method and use thereof

US20220277327A1

( en )

2022-09-01

Computer-based systems for data distribution allocation utilizing machine learning models and methods of use thereof

US12056011B2

( en )

2024-08-06

Computer-based systems configured for dynamic performance scoring of software agents and methods of use thereof

Legal Events

Date

Code

Title

Description

2024-08-01

FEPP

Fee payment procedure

Free format text : ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

2025-05-07

STCF

Information on status: patent grant

Free format text : PATENTED CASE

Related documents

Record · ID 607519
Conceptio Open Knowledge Archive — every document is proof-bundled with source, license, and retrieval metadata.