Google Patentsopen access

Choosing a large language model interfacing mechanism based on sample question … — Box, Inc. (US20240362497A1)

ABSTRACT

Abstract

Methods, systems, and computer program products for managing interactions between a content management system (CMS) and a large language model (LLM) system. The semantics of user questions can be considered before prompting an LLM, or alternatively, before querying datasets that are local to the CMS. Given a user question to be answered, the embedding of the user question can be matched against preconfigured sample question embeddings to determine a best match. A prompt corresponding to the determined best match is then configured based on identification of the class or classes that correspond to the matched question. Prompts for provision to LLMs can be synthesized based on a particular user's identity and/or based on the particular user's historical collaboration activities over objects of the CMS. The LLM can be hosted by a third-party provider. Alternatively all or portions of a large language model system can be hosted within the CMS.

Description

RELATED APPLICATIONS

The present application is related to co-pending U.S. Patent Application Ser. No. ______ titled “GENERATING A LARGE LANGUAGE MODEL PROMPT BASED ON COLLABORATION ACTIVITIES OF A USER” (Attorney Docket No. BOX-2023-0002-US20), filed on even date herewith, which is hereby incorporated by reference in its entirety; and The present application is related to co-pending U.S. Patent Application Ser. No. ______ titled “USING SAMPLE QUESTION EMBEDDINGS TO CHOOSE BETWEEN AN LLM INTERFACING MODEL AND A NON-LLM INTERFACING MODEL” (Attorney Docket No. BOX-2023-0002-US30), filed on even date herewith, which is hereby incorporated by reference in its entirety; and the present application claims the benefit of priority to U.S. Provisional Patent Application Ser. No. 63/543,503 titled “METHOD AND SYSTEM TO IMPLEMENT ARTIFICIAL INTELLIGENCE INTEGRATED WITH A CONTENT MANAGEMENT SYSTEM” filed on Oct. 10, 2023, and the present application claims the benefit of priority to U.S. Provisional Patent Application Ser. No. 63/527,534 titled “GENERATIVE ARTIFICIAL INTELLIGENCE PROMPT GENERATION USING EXAMPLE QUESTION EMBEDDINGS” filed on Jul. 18, 2023, and the present application claims the benefit of priority to U.S. Provisional Patent Application Ser. No. 63/463,049 titled “ARTIFICIAL INTELLIGENCE AGENTS INTEGRATED WITH A CONTENT MANAGEMENT SYSTEM” filed on Apr. 30, 2023, all of which are hereby incorporated by reference in their entirety.

TECHNICAL FIELD

This disclosure relates to determining how to interface with one or more generative artificial intelligence entities, and more particularly, this disclosure relates to techniques for interfacing with AI entities based on information drawn from a content management system.

BACKGROUND

The emergence of generative artificial intelligence has changed the way we interact with computers. By using generative artificial intelligence, it is now possible to pose questions to an artificial intelligence (AI) entity and receive a conversational response back from the AI entity that is often indistinguishable from a response generated by a human had the question been posed to a human in the course of a human-to-human conversation. Generative AI works on the basis of a language model, sometimes termed a large language model (LLM), that is trained on a large corpus of input materials. Usually such input materials are drawn from publicly available documents (e.g., books, public records, public databases, etc.), any of which publicly available documents might include opposing discourse, or at least discourse that is biased toward one or another position on a topic. The diversity of such discourse naturally includes exposition of different fact sets that were relied on by the original authors of the discourse. As such, the training corpus (possibly involving many exabytes of training data) almost necessarily comprises a panoply of answers that could be presented to an inquirer.

This leads to the problem of choosing how to interface with selected one or more generative AI entities, which further leads to the need to prompt the selected AI entities in a manner that leads to generative AI answers that are responsive to the information that the user is seeking. For example, the prompt, “What are shopping bags made from?” might garner the answer, “paper.” Or, the prompt, “What are shopping bags made from?” might garner the answer, “plastic.” Going further, and strictly as an example, the prompt, “What are shopping bags made from?” might garner the answer, “organic material.” As can be seen from these simple examples, one question or prompt might have many correct, but different, answers. In this hypothetical example inquiry, and given that the inquirer was seeking to know the sustainability of use of shopping bags, the inquirer might do well to present the AI entity with a prompt more like, “How sustainable is the use of disposable shopping bags?” And the answer might come back, “Given that plastic shopping bags are a result of a simple manufacturing process involving petroleum and that only a tiny bit of such petroleum is used in each shopping bag, and given that paper shopping bags involve the complex energy-intensive and water-wasteful process of destroying forests in order to make paper pulp from the wood, plastic bags are far more sustainable.”

The foregoing question and answer sessions are presented to highlight the fact that the nature of the prompt greatly influences the generative AI response/answer that emerges. This then leads us to an understanding that what is often needed is a way to generate generative AI prompts that are more likely to generate answers useful to the inquirer. Moreover what is needed are ways to do so on an ongoing basis in a computer-aided manner that does not require the inquirer to participate in any manner other than to provide a question (e.g., via a user interface). Still further, what is needed are ways to choose an AI LLM that has been trained using training data that contains information likely to be useful for generating answers to a particular question.

The problem to be solved is therefore rooted in various technological limitations of legacy approaches. Improved technologies are needed. In particular, improved applications of technologies are needed to address various technological limitations of legacy approaches.

SUMMARY

This summary is provided to introduce a selection of concepts that are further described elsewhere in the written description and in the figures. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Moreover, the individual embodiments of this disclosure each have several innovative aspects, no single one of which is solely responsible for any particular desirable attribute or end result.

The present disclosure describes techniques used in systems, methods, and computer program products for AI entity prompt generation using example question embeddings, which techniques advance the relevant technologies to address technological issues with legacy approaches. More specifically, the present disclosure describes techniques used in systems, methods, and in computer program products for selecting a prompt generation technique based on a corpus of example question embeddings. Certain embodiments are directed to technological solutions for selecting a purpose-specific prompt generation agent based on characteristics of a user question.

The disclosed embodiments modify and improve beyond legacy approaches. In particular, the herein-disclosed techniques provide technical solutions that address the technical problems that arise when choosing a large language model interfacing mechanism based on a set of candidate sample question embedding vectors. Moreover, the herein-disclosed techniques provide technical solutions that address how to use data of a content management system to synthesize generative artificial intelligence prompts. Such technical solutions involve specific implementations (e.g., data organization, data communication paths, module-to-module interrelationships, etc.) that relate to the software arts for improving computer functionality.

Various applications of the herein-disclosed improvements in computer functionality serve to reduce demand for computer memory, reduce demand for computer processing power, reduce network bandwidth usage, and reduce demand for intercomponent communication. For example, when performing computer operations that address the various technical problems underlying how to use data of a content management system to synthesize generative artificial intelligence prompts, both memory usage and CPU cycles demanded are significantly reduced as compared to the memory usage and CPU cycles that would be needed but for practice of the herein-disclosed techniques. This is because the foregoing synthesized generative artificial intelligence prompts are more likely to generate answers useful to the seeker, thus avoiding retries and retries and further retries.

Some of the ordered combination of steps of the embodiments serve in the context of practical applications that perform purpose-specific prompt generation agent based on characteristics of a user question. As such, the herein-disclosed techniques pertaining to purpose-specific prompt generation techniques and/or deployment of their corresponding agents overcome heretofore unsolved technological problems associated with how to use data of a content management system to synthesize generative artificial intelligence prompts.

The herein-disclosed embodiments pertain to technological problems that arise in the hardware and software arts that underlie, for instance, content management systems. Aspects of the present disclosure achieve performance and other improvements in peripheral technical fields including, but not limited to, prompt compilation as well as selection and optimization of corpora that are used for training a large language model that is situated in a customer-sequestered security perimeter.

Some embodiments include a sequence of instructions that are stored on a non-transitory computer readable medium. Such a sequence of instructions, when stored in memory and executed by one or more processors, causes the one or more processors to perform a set of acts for interfacing a content management system with a large language model system.

Some embodiments include the aforementioned sequence of instructions that are stored in a memory, which memory is interfaced to one or more processors such that the one or more processors can execute the sequence of instructions to cause the one or more processors to implement acts for interfacing a content management system with a large language model system.

In various embodiments, any combinations of any of the above can be organized to perform any variation of acts for selecting a prompt generation technique based on a corpus of example question embeddings. Many further combinations of aspects of the above elements are contemplated. For example, in addition to the forgoing prompt generation techniques, combinations of any of the above can be organized to implement (1) choosing a large language model interfacing mechanism based on sample question embeddings, (2) generating a large language model prompt based on collaboration activities of a user, and (3) using sample question embeddings when choosing between an LLM interfacing model and a non-LLM interfacing model.

Further details of aspects, objectives and advantages of the technological embodiments are described herein and in the figures and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings described below are for illustration purposes only. The drawings are not intended to limit the scope of the present disclosure.

FIG. 1 A presents an illustrative flow depicting initial population and ongoing use of example question embeddings as used in content management systems that perform different types of user question processing based on the example question embeddings, according to an embodiment.

FIG. 1 B depicts how different agents perform different processing based on the classification of a particular example question embedding, according to an embodiment.

FIG. 2 A is a first sample conversation that is carried out in systems that perform prompt generation based on a first type of user question, according to an embodiment.

FIG. 2 B presents an illustrative flow depicting selected operations of a first type of LLM interfacing agent as deployed in systems that perform prompt generation based on a first type of user question, according to an embodiment.

FIG. 2 C 1 is a second sample conversation that is carried out in systems that perform prompt generation based on a first type of user question, according to an embodiment.

FIG. 2 C 2 is a third sample conversation that is carried out in systems that perform prompt generation based on a first type of user question, according to an embodiment.

FIG. 2 D presents an illustrative flow depicting selected operations of a non-LLM agent as deployed in systems that generate answers to user questions, according to an embodiment.

FIG. 2 E presents an illustrative flow depicting selected operations of a non-LLM agent as deployed in systems that use templates when generating answers to user questions, according to an embodiment.

FIG. 3 A 1 presents an illustrative flow depicting selected operations of a second type of LLM interfacing agent as deployed in systems that perform prompt generation based on a user question, according to an embodiment.

FIG. 3 A 2 presents an illustrative flow depicting selected operations observing a generative LLM prompt budget, according to an embodiment.

FIG. 3 A 3 presents an illustrative flow depicting selected operations for remediating an over budget condition, according to an embodiment.

FIG. 3 A 4 shows a plurality of chunk rejection techniques, according to an embodiment.

FIG. 3 A 5 shows a chunk selection technique, according to an embodiment.

FIG. 3 A 6 is a diagram that shows an example chunk type assignment technique where individual portions of a document or documents are scored for relevance with respect to a provided user question, according to an embodiment.

FIG. 3 A 7 depicts a system for capturing historical interactions by users of a content management system, according to an embodiment.

FIG. 3 A 8 depicts a content management system that captures and stores a history of interaction activities, according to an embodiment.

FIG. 3 B shows use of historical collaboration activities when performing AI entity prompt generation, according to an embodiment.

FIG. 3 C shows a user interface for specifying aspects of a prompt template, according to an embodiment.

<div id="p-0037" num="0036" class="description-li

RELATED APPLICATIONS

The present application is related to co-pending U.S. Patent Application Ser. No. ______ titled “GENERATING A LARGE LANGUAGE MODEL PROMPT BASED ON COLLABORATION ACTIVITIES OF A USER” (Attorney Docket No. BOX-2023-0002-US20), filed on even date herewith, which is hereby incorporated by reference in its entirety; and The present application is related to co-pending U.S. Patent Application Ser. No. ______ titled “USING SAMPLE QUESTION EMBEDDINGS TO CHOOSE BETWEEN AN LLM INTERFACING MODEL AND A NON-LLM INTERFACING MODEL” (Attorney Docket No. BOX-2023-0002-US30), filed on even date herewith, which is hereby incorporated by reference in its entirety; and the present application claims the benefit of priority to U.S. Provisional Patent Application Ser. No. 63/543,503 titled “METHOD AND SYSTEM TO IMPLEMENT ARTIFICIAL INTELLIGENCE INTEGRATED WITH A CONTENT MANAGEMENT SYSTEM” filed on Oct. 10, 2023, and the present application claims the benefit of priority to U.S. Provisional Patent Application Ser. No. 63/527,534 titled “GENERATIVE ARTIFICIAL INTELLIGENCE PROMPT GENERATION USING EXAMPLE QUESTION EMBEDDINGS” filed on Jul. 18, 2023, and the present application claims the benefit of priority to U.S. Provisional Patent Application Ser. No. 63/463,049 titled “ARTIFICIAL INTELLIGENCE AGENTS INTEGRATED WITH A CONTENT MANAGEMENT SYSTEM” filed on Apr. 30, 2023, all of which are hereby incorporated by reference in their entirety.

TECHNICAL FIELD

This disclosure relates to determining how to interface with one or more generative artificial intelligence entities, and more particularly, this disclosure relates to techniques for interfacing with AI entities based on information drawn from a content management system.

BACKGROUND

The emergence of generative artificial intelligence has changed the way we interact with computers. By using generative artificial intelligence, it is now possible to pose questions to an artificial intelligence (AI) entity and receive a conversational response back from the AI entity that is often indistinguishable from a response generated by a human had the question been posed to a human in the course of a human-to-human conversation. Generative AI works on the basis of a language model, sometimes termed a large language model (LLM), that is trained on a large corpus of input materials. Usually such input materials are drawn from publicly available documents (e.g., books, public records, public databases, etc.), any of which publicly available documents might include opposing discourse, or at least discourse that is biased toward one or another position on a topic. The diversity of such discourse naturally includes exposition of different fact sets that were relied on by the original authors of the discourse. As such, the training corpus (possibly involving many exabytes of training data) almost necessarily comprises a panoply of answers that could be presented to an inquirer.

This leads to the problem of choosing how to interface with selected one or more generative AI entities, which further leads to the need to prompt the selected AI entities in a manner that leads to generative AI answers that are responsive to the information that the user is seeking. For example, the prompt, “What are shopping bags made from?” might garner the answer, “paper.” Or, the prompt, “What are shopping bags made from?” might garner the answer, “plastic.” Going further, and strictly as an example, the prompt, “What are shopping bags made from?” might garner the answer, “organic material.” As can be seen from these simple examples, one question or prompt might have many correct, but different, answers. In this hypothetical example inquiry, and given that the inquirer was seeking to know the sustainability of use of shopping bags, the inquirer might do well to present the AI entity with a prompt more like, “How sustainable is the use of disposable shopping bags?” And the answer might come back, “Given that plastic shopping bags are a result of a simple manufacturing process involving petroleum and that only a tiny bit of such petroleum is used in each shopping bag, and given that paper shopping bags involve the complex energy-intensive and water-wasteful process of destroying forests in order to make paper pulp from the wood, plastic bags are far more sustainable.”

The foregoing question and answer sessions are presented to highlight the fact that the nature of the prompt greatly influences the generative AI response/answer that emerges. This then leads us to an understanding that what is often needed is a way to generate generative AI prompts that are more likely to generate answers useful to the inquirer. Moreover what is needed are ways to do so on an ongoing basis in a computer-aided manner that does not require the inquirer to participate in any manner other than to provide a question (e.g., via a user interface). Still further, what is needed are ways to choose an AI LLM that has been trained using training data that contains information likely to be useful for generating answers to a particular question.

The problem to be solved is therefore rooted in various technological limitations of legacy approaches. Improved technologies are needed. In particular, improved applications of technologies are needed to address various technological limitations of legacy approaches.

SUMMARY

This summary is provided to introduce a selection of concepts that are further described elsewhere in the written description and in the figures. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Moreover, the individual embodiments of this disclosure each have several innovative aspects, no single one of which is solely responsible for any particular desirable attribute or end result.

The present disclosure describes techniques used in systems, methods, and computer program products for AI entity prompt generation using example question embeddings, which techniques advance the relevant technologies to address technological issues with legacy approaches. More specifically, the present disclosure describes techniques used in systems, methods, and in computer program products for selecting a prompt generation technique based on a corpus of example question embeddings. Certain embodiments are directed to technological solutions for selecting a purpose-specific prompt generation agent based on characteristics of a user question.

The disclosed embodiments modify and improve beyond legacy approaches. In particular, the herein-disclosed techniques provide technical solutions that address the technical problems that arise when choosing a large language model interfacing mechanism based on a set of candidate sample question embedding vectors. Moreover, the herein-disclosed techniques provide technical solutions that address how to use data of a content management system to synthesize generative artificial intelligence prompts. Such technical solutions involve specific implementations (e.g., data organization, data communication paths, module-to-module interrelationships, etc.) that relate to the software arts for improving computer functionality.

Various applications of the herein-disclosed improvements in computer functionality serve to reduce demand for computer memory, reduce demand for computer processing power, reduce network bandwidth usage, and reduce demand for intercomponent communication. For example, when performing computer operations that address the various technical problems underlying how to use data of a content management system to synthesize generative artificial intelligence prompts, both memory usage and CPU cycles demanded are significantly reduced as compared to the memory usage and CPU cycles that would be needed but for practice of the herein-disclosed techniques. This is because the foregoing synthesized generative artificial intelligence prompts are more likely to generate answers useful to the seeker, thus avoiding retries and retries and further retries.

Some of the ordered combination of steps of the embodiments serve in the context of practical applications that perform purpose-specific prompt generation agent based on characteristics of a user question. As such, the herein-disclosed techniques pertaining to purpose-specific prompt generation techniques and/or deployment of their corresponding agents overcome heretofore unsolved technological problems associated with how to use data of a content management system to synthesize generative artificial intelligence prompts.

The herein-disclosed embodiments pertain to technological problems that arise in the hardware and software arts that underlie, for instance, content management systems. Aspects of the present disclosure achieve performance and other improvements in peripheral technical fields including, but not limited to, prompt compilation as well as selection and optimization of corpora that are used for training a large language model that is situated in a customer-sequestered security perimeter.

Some embodiments include a sequence of instructions that are stored on a non-transitory computer readable medium. Such a sequence of instructions, when stored in memory and executed by one or more processors, causes the one or more processors to perform a set of acts for interfacing a content management system with a large language model system.

Some embodiments include the aforementioned sequence of instructions that are stored in a memory, which memory is interfaced to one or more processors such that the one or more processors can execute the sequence of instructions to cause the one or more processors to implement acts for interfacing a content management system with a large language model system.

In various embodiments, any combinations of any of the above can be organized to perform any variation of acts for selecting a prompt generation technique based on a corpus of example question embeddings. Many further combinations of aspects of the above elements are contemplated. For example, in addition to the forgoing prompt generation techniques, combinations of any of the above can be organized to implement (1) choosing a large language model interfacing mechanism based on sample question embeddings, (2) generating a large language model prompt based on collaboration activities of a user, and (3) using sample question embeddings when choosing between an LLM interfacing model and a non-LLM interfacing model.

Further details of aspects, objectives and advantages of the technological embodiments are described herein and in the figures and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings described below are for illustration purposes only. The drawings are not intended to limit the scope of the present disclosure.

FIG. 1 A presents an illustrative flow depicting initial population and ongoing use of example question embeddings as used in content management systems that perform different types of user question processing based on the example question embeddings, according to an embodiment.

FIG. 1 B depicts how different agents perform different processing based on the classification of a particular example question embedding, according to an embodiment.

FIG. 2 A is a first sample conversation that is carried out in systems that perform prompt generation based on a first type of user question, according to an embodiment.

FIG. 2 B presents an illustrative flow depicting selected operations of a first type of LLM interfacing agent as deployed in systems that perform prompt generation based on a first type of user question, according to an embodiment.

FIG. 2 C 1 is a second sample conversation that is carried out in systems that perform prompt generation based on a first type of user question, according to an embodiment.

FIG. 2 C 2 is a third sample conversation that is carried out in systems that perform prompt generation based on a first type of user question, according to an embodiment.

FIG. 2 D presents an illustrative flow depicting selected operations of a non-LLM agent as deployed in systems that generate answers to user questions, according to an embodiment.

FIG. 2 E presents an illustrative flow depicting selected operations of a non-LLM agent as deployed in systems that use templates when generating answers to user questions, according to an embodiment.

FIG. 3 A 1 presents an illustrative flow depicting selected operations of a second type of LLM interfacing agent as deployed in systems that perform prompt generation based on a user question, according to an embodiment.

FIG. 3 A 2 presents an illustrative flow depicting selected operations observing a generative LLM prompt budget, according to an embodiment.

FIG. 3 A 3 presents an illustrative flow depicting selected operations for remediating an over budget condition, according to an embodiment.

FIG. 3 A 4 shows a plurality of chunk rejection techniques, according to an embodiment.

FIG. 3 A 5 shows a chunk selection technique, according to an embodiment.

FIG. 3 A 6 is a diagram that shows an example chunk type assignment technique where individual portions of a document or documents are scored for relevance with respect to a provided user question, according to an embodiment.

FIG. 3 A 7 depicts a system for capturing historical interactions by users of a content management system, according to an embodiment.

FIG. 3 A 8 depicts a content management system that captures and stores a history of interaction activities, according to an embodiment.

FIG. 3 B shows use of historical collaboration activities when performing AI entity prompt generation, according to an embodiment.

FIG. 3 C shows a user interface for specifying aspects of a prompt template, according to an embodiment.

FIG. 3 D depicts a system for performing prompt engineering in a content management system, according to an embodiment.

FIG. 4 A is a system diagram depicting an interdomain interfacing technique as used in various cloud-based environments, according to an embodiment.

FIG. 4 B presents an illustrative flow depicting an embedding generation technique as used in systems that perform prompt generation based on a user question, according to an embodiment.

FIG. 4 C is a system diagram depicting techniques for isolation of different content object corpora when generating embeddings, according to an embodiment.

FIG. 4 D is a system diagram depicting techniques for training a local LLM using one or more content object repositories of a content management system, according to an embodiment.

FIG. 5 A , FIG. 5 B and FIG. 5 C present block diagrams of computing architectures having components suitable for implementing embodiments of the present disclosure and/or for use in the herein-described environments.

DETAILED DESCRIPTION

Aspects of the present disclosure solve problems associated with using computer systems for how to use data of a content management system to synthesize generative artificial intelligence prompts. These problems are unique to, and may have been created by, various computer-implemented methods for how to use data of a content management system to synthesize generative artificial intelligence prompts in the context of content management systems. Some embodiments are directed to approaches for selecting a purpose-specific prompt generation agent based on characteristics of a user question. The accompanying figures and discussions herein present example environments, systems, methods, and computer program products for selecting a prompt generation technique based on a corpus of example question embeddings.

Overview

As heretofore mentioned, one question or prompt to an artificial intelligence entity might have many ‘correct’ (but different) answers. Although it is not necessarily ‘wrong’ to have different answers to the same question, it does make it more difficult for the seeker to get to the sought-after information. Without a technical solution (e.g., based on some sort of computer-aided prompt engineering), the seeker must somehow converge to a prompt that gets to the information being sought. In many cases, the information being sought is based on analysis (e.g., an analysis that generates the probabilities that drive the generative AI processes) that derives directly from the contents of the training set, such as the content of a book or article. In other cases, the information being sought is based not on the content itself, but rather on analysis of the nature of the constituents of a training set.

For example, consider the user question, “What are the main terms and conditions of federally-offered oil drilling lease contracts?” This question, if posed to an AI entity, would return an answer that that derives directly from the contents of the federally-offered oil drilling lease contracts that were included in a training set of the prompted LLM. On the other hand, the user question, “What is the jurisdictional breakdown of federally-offered oil drilling lease contracts?” would be a question that can only be addressed based on either (1) the occurrence of a published answer to that or similar question that was used in the training set, or (2) an analysis of knowledge (e.g., metadata) of the constituents of the training set.

As used herein, a large language model (LLM) is a collection of information taken from training data that comprises representations of information drawn from books, magazines, posts, and/or from any other source of text, including text pertaining to computer code. In the embodiments contemplated herein, the discussed LLMs ingest massive amounts of data (e.g., word-by-word parameters) to learn billions of parameters that are calculated and/or stored during training. Some LLMs are composed of artificial neural networks that are trained using self-supervised learning and/or semi-supervised learning. Some LLMs are composed of machine learning vectors that are constructed during training using self-supervised learning and/or semi-supervised learning. An LLM takes as an input a prompt (e.g., a word or a sequence of words) and produces an output (e.g., an LLM answer) that is a word or a sequence of words that probabilistically follow, given the particular prompt. In some deployments an LLM is implemented within or as a generative large language model AI entity.

The foregoing scenario is similar to those scenarios that frequently emerge in business settings. Unfortunately, AI entities are often unable to discern whether a user question pertains to characteristics of the contents of a corpus of documents, or whether the user question pertains to the corpus as a whole. Absent some sort of mind reader, and for the purpose of generating a prompt that is likely to generate answers that are useful to the seeker, there needs to be some technology that behaves in the same fashion as a mind reader. To address this need, there needs to be a technology that is able to classify what is the thrust of the inquiry, and thereafter further technology that generates prompts that are likely to generate answers that are useful to the seeker (e.g., answers that correspond to the thrust of the inquiry).

Another problem that arises in business settings is the need or requirement (e.g., possibly due to an applicable policy or even a law) that no part of any conversation with the AI entity is to contain proprietary information. Unfortunately, such requirements are often impractical, and in some cases, might even be impossible to satisfy. To explain, it is often impossible to objectively bound what is proprietary and what is not. Consider a seeker&#39;s inquiry into a focus group&#39;s results that have been published to look at “Combinations of features in proposed introductions of Product ‘X’.” So now, we ask, is it a leak of proprietary information to refer to “Product ‘X’” in a prompt to a generative AI large language model (LLM)? One argument says, “Yes,” at least from the perspective that even asking the question to a public forum (e.g., forming a question or prompt to an AI entity) improperly disseminates proprietary information, at least because the submission of that question signals interest in “Product ‘X’.” Hence, what is needed is a way to bring the power of the AI LLM into a proprietary sandbox, and/or to obfuscate the question before dissemination to a publicly-accessible LLM.

The foregoing technical problems and corresponding solutions are further described as pertains to the figures.

Definitions and Use of Figures

Some of the terms used in this description are defined below for easy reference. The presented terms and their respective definitions are not rigidly restricted to these definitions-a term may be further defined by the term&#39;s use within this disclosure. The term “exemplary” is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects or designs. Rather, use of the word exemplary is intended to present concepts in a concrete fashion. As used in this application and the appended claims, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless specified otherwise, or is clear from the context, “X employs A or B” is intended to mean any of the natural inclusive permutations. That is, if X employs A, X employs B, or X employs both A and B, then “X employs A or B” is satisfied under any of the foregoing instances. As used herein, at least one of A or B means at least one of A, or at least one of B, or at least one of both A and B. In other words, this phrase is disjunctive. The articles “a” and “an” as used in this application and the appended claims should generally be construed to mean “one or more” unless specified otherwise or is clear from the context to be directed to a singular form.

Various embodiments are described herein with reference to the figures. It should be noted that the figures are not necessarily drawn to scale, and that elements of similar structures or functions are sometimes represented by like reference characters throughout the figures. It should also be noted that the figures are only intended to facilitate the description of the disclosed embodiments-they are not representative of an exhaustive treatment of all possible embodiments, and they are not intended to impute any limitation as to the scope of the claims. In addition, an illustrated embodiment need not portray all aspects or advantages of usage in any particular environment.

An aspect or an advantage described in conjunction with a particular embodiment is not necessarily limited to that embodiment and can be practiced in any other embodiment even if not so illustrated. References throughout this specification to “some embodiments” or “other embodiments” refer to a particular feature, structure, material, or characteristic described in connection with the embodiments as being included in at least one embodiment. Thus, the appearance of the phrases “in some embodiments” or “in other embodiments” in various places throughout this specification are not necessarily referring to the same embodiment or embodiments. The disclosed embodiments are not intended to be limiting of the claims.

Descriptions of Example Embodiments

FIG. 1 A presents an illustrative flow depicting initial population and ongoing use of example question embeddings as used in content management systems that perform different types of user question processing based on the example question embeddings. As an option, one or more variations of example question embeddings or any aspect thereof may be implemented in the context of the architecture and functionality of the shown content management system setting 1 A 00 and/or in any of the embodiments described herein and/or in any environment.

The figure is being presented to illustrate how and why example question embeddings might be configured to operate in a cloud-based content management system (CCM) environment that hosts all or portions of a content management system (CMS).

As used herein, a “content management system” is a collection of executable code that facilitates performance of a set of coordinated functions, workflows, tasks or other activities on behalf of a plurality of collaborating users that operate over shared content objects. More specifically, a content management system facilitates collaboration activities such as creating and sharing a shared content object and establishing a set of users who can access the shared content concurrently. In some embodiments as contemplated herein, a “content management system” is implemented as a set of computer-implemented modules that interoperate to capture, store, and provision access to electronically-stored data that is associated with a history of access/sharing events taken over shared content objects. As used herein, the term “collaboration system” is used interchangeably with the term “content management system.”

As shown, a database of example question embeddings 108 includes pairings between an example question embedding and a corresponding class. Many embeddings can refer to the same class. In some cases (not show) a particular embedding may be paired with multiple classes. This pairing is used in the flow of FIG. 1 A at switch 116 , where a particular agent is selected based on a determined class.

The class can be determined algorithmically as follows:

Step 1: Receive a question (e.g., user question 102 ) from a user (e.g., CMS user 101 ) and generate an embedding for the question (step 104 ). Generation of the embedding can be done in a local domain, or the embedding generation can be done by accessing an LLM over a network (e.g., via a predefined network-based protocol). Step 2: Compare the user question embedding 106 to the example question embeddings (step 110 ) to determine the best match (or matches). Step 3: Find the class of the closest match (e.g., closest match class 114 ) by observing the pairings (step 112 ).

The shown example question embeddings 108 include a plurality of embeddings (e.g., E 1 , E 2 , . . . , E 99 ) where each individual embedding has a pairwise association to one or more class designations (e.g., embedding E 1 is pairwise associated with class C 1 , embedding E 4 is pairwise associated with class C 3 , etc.).

FIG. 1 A shows three agents: agent 1 118 (that processes a class C 1 user question 123 ), agent 2 120 (that processes a class C 2 user question 125 ), . . . , and agentN 122 (that processes a class C N user question 127 ). These types of agents are presented here merely as examples for illustration; other types of agents are possible, in fact systems having more, or fewer, or other/different agents are contemplated. For example, there might be an agent that is configured to gather documents of the CCM and run comparisons. Or, there might be an agent that is configured to generate metadata over a sub-corpus of documents of the CCM and to then interact with the LLM on the basis of the generated metadata. Or, there might be an agent that is configured to analyze various metadata corresponding to a sub-corpus of documents of the CCM and to then either interact with the LLM on the basis of the generated metadata, or to respond to the user question with or without interaction with the LLM. In some settings, there might be an agent that is configured to analyze a sub-corpus of documents of the CCM and to then synthesize a summary—with or without interaction with the LLM.

In some cases, multiple agents may be invoked based on a single user question. Results from such multiple agents can be analyzed with respect to each other and an amalgamated answer can be provided to the user (e.g., as depicted by step 130 ).

Generating Embeddings to Class-Pair Mappings for the Example Question Embeddings Database

Initial seeding is done by manually creating questions such as, “What is this document about?” (call this Q 1 ) or “What is our parental leave policy?” (call this Q 2 ), then calculating the embedding for the created question. A user or administrator can manually assign class designations (e.g., C 1 , C 2 , C 3 , . . . , C N ) to the questions. To explain, assume embedding E 1 is an embedding for Q 1 and its class is “C 1 : Summarization.” Further assume that there is an embedding E 2 that is an embedding for Q 2 , and that embedding E 2 is designated to correspond with the class of “C 2 : Policy.” In such a case, the data in Table 1 is generated:

TABLE 1

Embedding-to-classification mapping

Embedding

Classification

E 1

Summarization

E 2

Policy

In some embodiments, it is possible to calculate an embedding Ex for each manually-posed question by asking an LLM model (e.g., ChatGPT4) to classify the question. This can be accomplished by prompting the LLM model with a request in the form of, “Please classify the question: “{question}” into one of the following categories: “{a sample category list}.” The sample category list might include names such as “Summarization.” Other possible sample categories may include “Policy”, “Legal”, “Health”, “Architecture”, etc. The LLM will return a classification selected from the provided sample category list. This can be repeated for each manually-posed question until such time as all manually-posed questions have a corresponding embedding as well as a corresponding designated class.

Note that the cardinality of the set of classifications is relatively low as compared to the possibly much larger set of questions and their respective question-specific embeddings. This is shown in example question embeddings 108 by the depiction of embeddings (e.g., embedding E 1 through embedding E 99 ), whereas there are far fewer class designations (e.g., class C 1 through class C 3 ).

Different Agent Types Support Different LLM Interfaces

Different agents interact with the CMS and one or more LLMs in different ways. The specifics of how each agent interacts with the CMS and the one or more LLMs are shown and described in FIG. 1 B , FIG. 2 A , FIG. 2 B , FIG. 2 C 1 , FIG. 2 C 2 , FIG. 2 D , FIG. 2 E , FIG. 3 A 1 , FIG. 3 A 2 , FIG. 3 A 3 , FIG. 3 A 4 , FIG. 3 A 5 , FIG. 3 A 6 , FIG. 3 A 7 , and FIG. 3 A 8 as well as in other disclosures herein.

FIG. 1 B depicts how different agents perform different processing based on the classification of a particular example question embedding.

Specifically, this embodiment shows how characteristics of a user question can influence how the answer to the user&#39;s question is sought. In this illustrative embodiment, the flow 1 B 00 shows cases where characteristics of a user question determine whether the answer to the user&#39;s question is sought (1) via deployment of LLM agent type processing 131 or (2) via non-LLM agent type processing 132 .

To explain, there are many scenarios where a user question can be answered using solely local resources (e.g., computing and storage resources of the CMS). In those scenarios, it is often most efficient to have an agent perform the local processing using solely CMS-local resources. However, there are also many situations where a user question can be best answered using external LLM resources (e.g., computing and storage resources of a selected LLM). In those situations where a user question can be best answered using external LLM resources, an LLM agent is deployed to interact with the external LLM resources.

In addition to the aforementioned efficiency considerations, there are many reasons why local processing (e.g., local processing using solely specifically-allocated local resources of the CMS) is preferred. Strictly as example reasons, consider that a CMS might serve many customers, and each customer has an expectation that their data is not shared outside of that customer&#39;s security perimeter. Therefore, processing within the customer&#39;s security perimeter (and not involving public LLMs) is strongly preferred. Further consider that processing using solely specifically-allocated local resources of the CMS prevents the possibility that a particular customer&#39;s data can be used with an LLM (e.g., for training or for inferencing). As such, a second customer cannot access any form of the first customer&#39;s data, even if both the first customer and the second customer use the same LLM.

Now, returning to the top-to-bottom discussion of flow 1 B 00 , it can be seen that any known technique can be used to analyze a user question 102 so as to determine (e.g., via module 113 ) what class the user question belongs to. In many cases such a determination is facilitated by accessing an embedding-to-class mapping, as shown. In other cases, the class into which the user question belongs is determined by inspection (e.g., presence of keywords) or analysis (e.g., natural language analysis). In any of the aforementioned cases, a determined class 115 is made available to switch 116 . In this embodiment, switch 116 implements two levels of consideration that result in determination of downstream processing. In a first level of consideration, the question of whether to use an LLM is answered. This first level of consideration determines coarsely whether to use LLM agent type processing 131 or whether to use non-LLM agent type processing 132 . In a second level of consideration, switch 116 considers the determined class 115 so as to choose what specific agent should be deployed. In the example shown, both a class C 1 user question 123 and a class C 2 user question 125 are handled by LLM agent type processing 131 (e.g., using either agent 1

118 or agent 2 120 ), whereas in the event that the determined class 115 of the matched user question is class C N , then downstream processing of the user question is handled by non-LLM agent type processing 132 (e.g., using agentN 122 ).

In one particular case of this example, the shown non-LLM agent type processing is responsive to receipt of a user question by performing downstream processing using solely local CMS resources (e.g., via module 121 ), whereas the shown LLM agent type processing performs downstream processing by invoking prompt engineering 119 before sending the engineered prompt to an LLM system.

Regardless of which arm (e.g., class C 1 , class C 2 , . . . , class C N ) of switch 116 is taken, an answer to the user question is developed (e.g., an answer from LLM processing 124 or an answer from CMS processing 126 ), and such an answer is provided to the requesting user (e.g., as depicted by step 130 ).

Now, as suggested above, since there are many different types of user questions and possibly a similar number of distinct user question classes, it follows that there might be many different ways to process said different user questions. The example of FIG. 2 A that follows hereunder covers the case where answering a user question involves converting from a natural language inquiry (e.g., “Which of the files in this folder are contracts?”) to an SQL query that can in turn be executed over a dataset of the CMS. More specifically, the example of FIG. 2 A that follows hereunder covers the case where an agent-to-LLM conversation (e.g., between an agent and an LLM) is carried out so as to generate a prompt to the LLM that will result in the sought-after SQL query.

FIG. 2 A is a first sample conversation that is carried out in systems that perform prompt generation based on a first type of user question. As an option, one or more variations of sample conversation 2 A 00 or any aspect thereof may be implemented in the context of the architecture and functionality of the embodiments described herein and/or in any environment.

The figure is being presented to illustrate how a sample agent-to-LLM conversation can be carried out. More specifically, the figure is being presented to illustrate how an LLM can be used to convert data from a first representation to a second representation. In this case, the first representation is a natural language question and the second representation (i.e., the requested second representation) is an SQL query. As can be seen, the LLM responds to LLM prompt 219 with the natural language prompt, “Which of the files in this folder are contracts?” with an SQL “Select” query. The dataset search query (e.g., the SQL query) is stored at operation 215 for subsequent downstream processing. Of course, embodiments involving SQL as the dataset search query language are shown and described here merely for ease of understanding. Other dataset search query languages are known in the art (e.g., Microsoft “iqy” language, XML XQuery language, etc.), and any one or more additional or alternative dataset search query languages can be used.

One of skill in the art will recognize that some context was provided to the LLM prior to the “Convert” prompt. The specific context provided here is abridged for ease of understanding. Additional context (e.g., the designation of “this folder” that might be present in a conversation between agent 230 and an instance of large language model 232 ) is omitted so as not to occlude the essence of the conversation.

The inner workings of some types of agents such as the first type of LLM interfacing agent of FIG. 1 A (e.g., agent 1 118 of FIG. 1 A ) is shown and described as pertains to FIG. 2 B .

FIG. 2 B presents an illustrative flow depicting selected operations of a first type of LLM interfacing agent as deployed in systems that perform prompt generation based on a first type of user question. As an option, one or more variations of user question processing agent 2 B 00 or any aspect thereof may be implemented in the context of the architecture and functionality of the embodiments described herein and/or in any environment.

The figure is being presented to illustrate how a user question processing agent might be configured to operate in a CCM environment. To explain, suppose that CMS user 101 poses a user question, and further suppose that the posed user question is of class C 1 . Now, further suppose that the user question is, “How many of my contracts are valued at &gt;$100,000?” It would be unreasonable to expect the LLM to know the answer to that question unless the LLM had been trained on the corpus of contracts. However, providing the LLM access to the corpus of contracts would almost certainly involve leakage of proprietary information.

An alternative way is as follows: Rather than providing the LLM access to the corpus of contracts, instead, provide the LLM with some information about the individual contracts in the corpus of contracts. This can be accomplished by converting the user question into a dataset search query (step 210 ) and then providing the results of executing the query to the LLM. In this embodiment, the LLM reformats the query results (step 222 ) into language that is akin to a human-to-human conversation. To accomplish this, and as shown, the user question is converted into a query language (step 204 ), which might involve use of a natural language processor 206 to identify the subject of the conversation, qualifications, limitations, verbs, etc., that correspond to syntax and semantics of a query language.

It should be noted that a user question might be a compound request or question (e.g., a formulation that has two or more clauses embedded or implied). Accordingly, decision 208 serves to iteratively process individual constituent embedded or implied questions of a compound request in a loop, where multiple respective dataset search queries are generated based on each identified embedded or implied question. For example, when the “Yes” branch of decision 208 is taken, then for each individual embedded or implied question of the compound request, processing passes through step 210 such that an instance of large language model 212 is requested to convert the individual embedded or implied question from natural language into a domain-specific language.

At step 216 , the dataset search query 214 , either as a single query or as multiple queries, are executed over content objects and/or over corresponding content object metadata. This results in query results 218 . In some cases, the query results might be complex or compound so, accordingly, step 222 might be accomplished by performing several iterations of asking the LLM to reformat the query results (e.g., reformatted query results 227 ) into a large language model system answer (e.g., LLM answer 226 ) that comports with the requested reformatting (e.g., into the natural language of CMS user 101 ). It should be noted that the large language model might have been trained on materials that are in a particular language or dialect (e.g., French, German, etc.). In such a case, when step 228 provides the LLM answer to the requesting user, the LLM answer will be in, or at least include, that particular language or dialect.

The foregoing written description pertains to merely one possible embodiment and/or way to implement a user question processing agent. Many variations are possible. Moreover, the user question processing agent as comprehended in the foregoing can be implemented in any environment and/or using any technique for interaction with an LLM. Example interaction techniques (e.g., agent-to-LLM conversations) are shown and described as pertains to FIG. 2 C 1 . Moreover, the foregoing written description pertains to merely one possible embodiment and/or one way to implement an agent-to-LLM conversation. Many variations are possible. For example, the sample conversation as comprehended in the foregoing can be implemented in any environment and/or involving any types of data in the conversation. One example of a sample agent-to-LLM conversation is shown and described as pertains to the second sample conversation of FIG. 2 C 1 and to the third sample conversation FIG. 2 C 2 .

FIG. 2 C 1 is a second sample conversation that is carried out in systems that perform prompt generation based on a first type of user question. As an option, one or more variations of second sample conversation 2 C 100 or any aspect thereof may be implemented in the context of the architecture and functionality of the embodiments described herein and/or in any environment.

As shown, the conversation inclu

CLAIMS

Claims ( 20 )

What is claimed is:

1 . A method for selecting a large language model interfacing technique, the method comprising:

responsive to an occurrence of a user question, selecting a large language model interfacing technique by, calculating a subject embedding vector based on at least a portion of the user question; comparing the subject embedding vector to one or more sample question embedding vectors to select a candidate one of the sample question embedding vectors that is similar to the subject embedding vector; determining a classification of the subject embedding vector; and invoking at least one computing agent based at least in part on the classification.

2 . The method of claim 1 , further comprising: populating a dataset of sample question embedding vectors, wherein a particular one from a set of candidate sample question embedding vectors is associated with a corresponding classification.

3 . The method of claim 1 , further comprising: gathering an output from the at least one computing agent and providing at least a portion of the output from the at least one computing agent to a large language model system.

4 . The method of claim 1 , further comprising: selecting a particular instance of a large language model system taken from a plurality of large language model system instances.

5 . The method of claim 4 , further comprising selecting the particular instance of the large language model system based at least in part on interfacing requirements of the particular instance of the large language model system.

6 . The method of claim 1 , wherein the at least one computing agent of a content management system interfaces with an application programming interface of a large language model system.

7 . The method of claim 6 , further comprising storing at least one embedding vector in a local embedding storage of a content management system.

8 . The method of claim 7 , wherein the large language model system is in a first domain having a first security perimeter and wherein the content management system is in a second domain having a second security perimeter.

9 . A non-transitory computer readable medium having stored thereon a sequence of instructions which, when stored in memory and executed by one or more processors causes the one or more processors to perform a set of acts for selecting a large language model interfacing technique, the set of acts comprising:

responsive to an occurrence of a user question, selecting a large language model interfacing technique by, calculating a subject embedding vector based on at least a portion of the user question; comparing the subject embedding vector to one or more sample question embedding vectors to select a candidate one of the sample question embedding vectors that is similar to the subject embedding vector; determining a classification of the subject embedding vector; and invoking at least one computing agent based at least in part on the classification.

10 . The non-transitory computer readable medium of claim 9 , further comprising instructions which, when stored in memory and executed by the one or more processors causes the one or more processors to perform acts of: populating a dataset of sample question embedding vectors, wherein a particular one from a set of candidate sample question embedding vectors is associated with a corresponding classification.

11 . The non-transitory computer readable medium of claim 9 , further comprising instructions which, when stored in memory and executed by the one or more processors causes the one or more processors to perform acts of: gathering an output from the at least one computing agent and providing at least a portion of the output from the at least one computing agent to a large language model system.

12 . The non-transitory computer readable medium of claim 9 , further comprising instructions which, when stored in memory and executed by the one or more processors causes the one or more processors to perform acts of: selecting a particular instance of a large language model system taken from a plurality of large language model system instances.

13 . The non-transitory computer readable medium of claim 12 , further comprising instructions which, when stored in memory and executed by the one or more processors causes the one or more processors to perform acts of selecting the particular instance of the large language model system based at least in part on interfacing requirements of the particular instance of the large language model system.

14 . The non-transitory computer readable medium of claim 9 , wherein the at least one computing agent of a content management system interfaces with an application programming interface of a large language model system.

15 . The non-transitory computer readable medium of claim 14 , further comprising instructions which, when stored in memory and executed by the one or more processors causes the one or more processors to perform acts of storing at least one embedding vector in a local embedding storage of a content management system.

16 . The non-transitory computer readable medium of claim 15 , wherein the large language model system is in a first domain having a first security perimeter and wherein the content management system is in a second domain having a second security perimeter.

17 . A system for selecting a large language model interfacing technique, the system comprising:

a storage medium having stored thereon a sequence of instructions; and one or more processors that execute the sequence of instructions to cause the one or more processors to perform a set of acts, the set of acts comprising,

responsive to an occurrence of a user question, selecting a large language model interfacing technique by,

calculating a subject embedding vector based on at least a portion of the user question;

comparing the subject embedding vector to one or more sample question embedding vectors to select a candidate one of the sample question embedding vectors that is similar to the subject embedding vector;

determining a classification of the subject embedding vector; and

invoking at least one computing agent based at least in part on the classification.

18 . The system of claim 17 , further comprising: populating a dataset of sample question embedding vectors, wherein a particular one from a set of candidate sample question embedding vectors is associated with a corresponding classification.

19 . The system of claim 17 , further comprising: gathering an output from the at least one computing agent and providing at least a portion of the output from the at least one computing agent to a large language model system.

20 . The system of claim 17 , further comprising: selecting a particular instance of a large language model system taken from a plurality of large language model system instances.

US18/398,039

2023-04-30

2023-12-27

Choosing a large language model interfacing mechanism based on sample question embeddings

Pending

US20240362497A1

( en )

Priority Applications (3)

Application Number

Priority Date

Filing Date

Title

US18/398,039

US20240362497A1

( en )

2023-04-30

2023-12-27

Choosing a large language model interfacing mechanism based on sample question embeddings

PCT/US2024/026906

WO2024228967A2

( en )

2023-04-30

2024-04-29

Generating a large language model prompt based on collaboration activities of a user

EP24800417.8A

EP4705893A2

( en )

2023-04-30

2024-04-29

Generating a large language model prompt based on collaboration activities of a user

Applications Claiming Priority (4)

Application Number

Priority Date

Filing Date

Title

US202363463049P

2023-04-30

2023-04-30

US202363527534P

2023-07-18

2023-07-18

US202363543503P

2023-10-10

2023-10-10

US18/398,039

US20240362497A1

( en )

2023-04-30

2023-12-27

Choosing a large language model interfacing mechanism based on sample question embeddings

Publications (1)

Publication Number

Publication Date

US20240362497A1

true

US20240362497A1 ( en )

2024-10-31

Family

ID=93215572

Family Applications (3)

Application Number

Title

Priority Date

Filing Date

US18/398,058

Active

2044-03-18

US12614080B2

( en )

2023-04-30

2023-12-27

Using sample question embeddings to choose between an LLM interfacing model and a non-LLM interfacing model

US18/398,039

Pending

US20240362497A1

( en )

2023-04-30

2023-12-27

Choosing a large language model interfacing mechanism based on sample question embeddings

US18/398,050

Pending

US20240362476A1

( en )

2023-04-30

2023-12-27

Generating a large language model prompt based on collaboration activities of a user

Family Applications Before (1)

Application Number

Title

Priority Date

Filing Date

US18/398,058

Active

2044-03-18

US12614080B2

( en )

2023-04-30

2023-12-27

Using sample question embeddings to choose between an LLM interfacing model and a non-LLM interfacing model

Family Applications After (1)

Application Number

Title

Priority Date

Filing Date

US18/398,050

Pending

US20240362476A1

( en )

2023-04-30

2023-12-27

Generating a large language model prompt based on collaboration activities of a user

Country Status (3)

Country

Link

US

( 3 )

US12614080B2

( en )

EP

( 1 )

EP4705893A2

( en )

WO

( 1 )

WO2024228967A2

( en )

Cited By (13)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20240403564A1

( en )

*

2023-05-30

2024-12-05

Google Llc

Large-Scale, Privacy Preserving Personalized Large Language Models (LLMs)

US20250103826A1

( en )

*

2023-09-21

2025-03-27

Google Llc

Processing documents in cloud storage using query embeddings

US12271707B1

( en )

*

2023-11-29

2025-04-08

Hebbia Inc.

Chat integration with grid-based data structure

US20250232123A1

( en )

*

2024-01-11

2025-07-17

Oracle International Corporation

Large language model aggregator

US20250252164A1

( en )

*

2024-02-02

2025-08-07

Gen Digital Inc.

Multidimensional local large language model user authentication

US20250298856A1

( en )

*

2024-03-19

2025-09-25

Ly Corporation

Information processing apparatus, information processing method, and non-transitory computer readable storage medium

US12443595B1

( en )

*

2024-06-26

2025-10-14

Jpmorgan Chase Bank, N.A.

Method and system for verification with large language models for data querying

US12566923B1

( en )

*

2025-07-11

2026-03-03

Geotab Inc.

Systems and methods for training and using custom vector embedding models

US12596710B1

( en )

*

2024-12-16

2026-04-07

Sap Se

AI-assisted natural language request transformation with reduced cognitive load

US20260100921A1

( en )

*

2024-10-07

2026-04-09

Zoom Video Communications, Inc.

Artificial intelligence (ai)-based user interfaces

US12608562B2

( en )

2023-09-21

2026-04-21

Google Llc

Providing personalized prompts to users based on documents in cloud storage

WO2026097079A1

( en )

*

2024-11-04

2026-05-07

Soroco India Private Limited

Techniques for process discovery and user guidance

US12632480B2

( en )

*

2024-05-31

2026-05-19

Dell Products L.P.

System and method for enhancing generative artificial intelligence (AI) model-based matching of queries and contents

Families Citing this family (11)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20240394600A1

( en )

*

2023-05-22

2024-11-28

Sage Global Services Limited

Hallucination Detection

US20250053835A1

( en )

*

2023-08-07

2025-02-13

Trunk Tools, Inc.

Methods and systems for generative question answering for construction project data

TWI867936B

( en )

*

2024-01-11

2024-12-21

玉山商業銀行股份有限公司

System and method for managing token usage

US12572756B2

( en )

*

2024-02-22

2026-03-10

Microsoft Technology Licensing, Llc

Discovery and selection of content based on language model token restrictions

US20250322328A1

( en )

*

2024-04-12

2025-10-16

International Business Machines Corporation

Next best agent selection in an adaptive workflow

US12572830B1

( en )

*

2024-05-23

2026-03-10

AtomBeam Technologies Inc.

System and method for enterprise hierarchical persistent cognitive machines with organizational hierarchy awareness and compliance integration

KR102849665B1

( en )

*

2024-11-13

2025-08-22

주식회사 로그프레소

Method for Extending LLM Prompt by Use of App Prompt

JP7751715B1

( en )

*

2024-11-19

2025-10-08

株式会社ソフトクリエイト

Prompt engineering computer, prompt engineering method and program

CN119149575B

( en )

*

2024-11-20

2025-03-28

图灵人工智能研究院（南京）有限公司

A Text2SQL intelligent query and answer method and system based on LLM and vector model

CN119807233B

( en )

*

2024-12-30

2025-10-24

中交第一公路勘察设计研究院有限公司

Data query method, device and medium based on large language model

CN121301547A

( en )

*

2025-10-31

2026-01-09

北京天星医疗股份有限公司

A question-answering interaction method, apparatus, device, and storage medium based on a large model.

Family Cites Families (18)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

JP2005196728A

( en )

2003-12-11

2005-07-21

Nec Corp

Security verification system, device, method, and program for security verification

US8484140B2

( en )

2009-06-09

2013-07-09

Microsoft Corporation

Feature vector clustering

US9443007B2

( en )

*

2011-11-02

2016-09-13

Salesforce.Com, Inc.

Tools and techniques for extracting knowledge from unstructured data retrieved from personal data sources

US10296520B1

( en )

2013-07-24

2019-05-21

Veritas Technologies Llc

Social network analysis of file access information

US9978362B2

( en )

2014-09-02

2018-05-22

Microsoft Technology Licensing, Llc

Facet recommendations from sentiment-bearing content

US10326768B2

( en )

*

2015-05-28

2019-06-18

Google Llc

Access control for enterprise knowledge

US10382440B2

( en )

*

2016-09-22

2019-08-13

International Business Machines Corporation

Method to allow for question and answer system to dynamically return different responses based on roles

US11200027B2

( en )

2017-09-19

2021-12-14

Google Llc

Virtual assistant configured to automatically customize groups of actions

US10936586B2

( en )

*

2017-11-29

2021-03-02

Wipro Limited

Method and system for providing domain-specific response to a user query

CN111625635B

( en )

2020-05-27

2023-09-29

北京百度网讯科技有限公司

Question and answer processing method, device, equipment and storage medium

CN111951805B

( en )

2020-07-10

2024-09-20

华为技术有限公司

A text data processing method and device

US20220036153A1

( en )

2020-07-29

2022-02-03

Thayermahan, Inc.

Ultra large language models as ai agent controllers for improved ai agent performance in an environment

US20220121694A1

( en )

2020-10-20

2022-04-21

Unisys Corporation

Semantic search and response

US11790017B2

( en )

2021-04-30

2023-10-17

CS Disco, Inc.

Systems and methods for searching related documents and associated search operators

US12050712B2

( en )

*

2021-09-30

2024-07-30

Glean Technologies, Inc.

Enterprise knowledge assistant with permissions-aware automated responses

CN120615194A

( en )

*

2022-12-16

2025-09-09

思睿人工智能公司

Iterative, context-based generative AI

US12339917B2

( en )

*

2023-09-06

2025-06-24

Infosys Limited

Method and system for generating user role-specific responses through Large Language Models

US12481790B2

( en )

*

2023-09-29

2025-11-25

Intuit Inc.

Leakage detection for large language models

2023

2023-12-27

US

US18/398,058

patent/US12614080B2/en

active

Active

2023-12-27

US

US18/398,039

patent/US20240362497A1/en

active

Pending

2023-12-27

US

US18/398,050

patent/US20240362476A1/en

active

Pending

2024

2024-04-29

EP

EP24800417.8A

patent/EP4705893A2/en

active

Pending

2024-04-29

WO

PCT/US2024/026906

patent/WO2024228967A2/en

not_active

Ceased

Cited By (17)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20240403564A1

( en )

*

2023-05-30

2024-12-05

Google Llc

Large-Scale, Privacy Preserving Personalized Large Language Models (LLMs)

US12417356B2

( en )

*

2023-05-30

2025-09-16

Google Llc

Large-scale, privacy preserving personalized large language models (LLMs)

US12608562B2

( en )

2023-09-21

2026-04-21

Google Llc

Providing personalized prompts to users based on documents in cloud storage

US20250103826A1

( en )

*

2023-09-21

2025-03-27

Google Llc

Processing documents in cloud storage using query embeddings

US12271707B1

( en )

*

2023-11-29

2025-04-08

Hebbia Inc.

Chat integration with grid-based data structure

US12393788B2

( en )

2023-11-29

2025-08-19

Hebbia Inc.

Matrix user interface for LLM-powered data analysis and generation

US12626067B2

( en )

2023-11-29

2026-05-12

Hebbia Inc.

Grid-cell highlighting based on LLM attention scores

US20250232123A1

( en )

*

2024-01-11

2025-07-17

Oracle International Corporation

Large language model aggregator

US20250252164A1

( en )

*

2024-02-02

2025-08-07

Gen Digital Inc.

Multidimensional local large language model user authentication

US20250298856A1

( en )

*

2024-03-19

2025-09-25

Ly Corporation

Information processing apparatus, information processing method, and non-transitory computer readable storage medium

US12632480B2

( en )

*

2024-05-31

2026-05-19

Dell Products L.P.

System and method for enhancing generative artificial intelligence (AI) model-based matching of queries and contents

US12443595B1

( en )

*

2024-06-26

2025-10-14

Jpmorgan Chase Bank, N.A.

Method and system for verification with large language models for data querying

US20260100921A1

( en )

*

2024-10-07

2026-04-09

Zoom Video Communications, Inc.

Artificial intelligence (ai)-based user interfaces

WO2026097079A1

( en )

*

2024-11-04

2026-05-07

Soroco India Private Limited

Techniques for process discovery and user guidance

US12596710B1

( en )

*

2024-12-16

2026-04-07

Sap Se

AI-assisted natural language request transformation with reduced cognitive load

US12632489B1

( en )

*

2025-01-02

2026-05-19

International Business Machines Corporation

Generation of context for a question submitted to a generative language machine learning model

US12566923B1

( en )

*

2025-07-11

2026-03-03

Geotab Inc.

Systems and methods for training and using custom vector embedding models

Also Published As

Publication number

Publication date

WO2024228967A2

( en )

2024-11-07

US20240362476A1

( en )

2024-10-31

EP4705893A2

( en )

2026-03-11

US20240362213A1

( en )

2024-10-31

US12614080B2

( en )

2026-04-28

WO2024228967A3

( en )

2025-04-17

Similar Documents

Publication

Publication Date

Title

US12614080B2

( en )

2026-04-28

Using sample question embeddings to choose between an LLM interfacing model and a non-LLM interfacing model

US12236288B2

( en )

2025-02-25

Flexible and scalable artificial intelligence and analytics platform with advanced content analytics and data ingestion

US12314666B2

( en )

2025-05-27

Stable identification of entity mentions

US11941034B2

( en )

2024-03-26

Conversational database analysis

JP2025526284A

( en )

2025-08-13

Controlled summarization and structuring of unstructured documents

JP2023532669A

( en )

2023-07-31

Document processing and response generation system

JP2021529385A5

( en )

2022-06-29

US12072878B2

( en )

2024-08-27

Search architecture for hierarchical data using metadata defined relationships

JP2017513134A

( en )

2017-05-25

Ontology mapping method and apparatus

CN116414961A

( en )

2023-07-11

Question answering method and system based on knowledge graph in military field

US20250077237A1

( en )

2025-03-06

Gai to app interface engine

US20240061831A1

( en )

2024-02-22

Generating Object Morphisms During Object Search

US11803543B2

( en )

2023-10-31

Lossless switching between search grammars

CN119903159A

( en )

2025-04-29

A knowledge question-answering fast processing system based on artificial intelligence

Xiao et al.

2022

Datalab: A platform for data analysis and intervention

WO2025194913A1

( en )

2025-09-25

Data query method and apparatus, computer device, and storage medium

US12072890B2

( en )

2024-08-27

Visualization data reuse in a data analysis system

CN120911441B

( en )

2026-01-13

Marketing System Based on Intelligent Analysis of Multimodal Document Data

Beniwal et al.

2018

Data mining with linked data: Past, present, and future

US20260072898A1

( en )

2026-03-12

Ephemeral views for database query generator

Khan

2013

Processing big data with natural semantics and natural language understanding using brain-like approach

US12591586B1

( en )

2026-03-31

Efficient processing and display of data insights

US12626158B1

( en )

2026-05-12

Automated contribution analysis for question answering

US20260064567A1

( en )

2026-03-05

System and method for graph-augmented test case generation using artificial intelligence (ai)

US20250355894A1

( en )

2025-11-20

Universal data representation for heterogeneous data

Legal Events

Date

Code

Title

Description

2023-12-28

AS

Assignment

Owner name : BOX, INC., CALIFORNIA

Free format text : ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:GRENADER, DENIS;KUS, BENJAMIN JOHN;SIGNING DATES FROM 20231218 TO 20231221;REEL/FRAME:065972/0494

2024-03-05

STPP

Information on status: patent application and granting procedure in general

Free format text : DOCKETED NEW CASE - READY FOR EXAMINATION

Related documents

• Virtual reality system using multiple force arrays for a solver — DeepMotion, Inc. (US10657696B2)#653961
• Augmented reality system with spatialized audio tied to user manipulated … — Magic Leap, Inc. (US11231770B2)#653963
• High-efficiency fuel cell system with carbon dioxide capture assembly and … — Fuelcell Energy, Inc. (US10170782B2)#653965
• Integrated electrochemical cell power generation and electrolysis systems — Czero, Inc. (US20250087711A1)#653966
• Generating augmented reality experiences utilizing physical objects to … — Meta Platforms Technologies, Llc (US20260037214A1)#653960