Google Patentsopen access

Enhanced searching using fine-tuned machine learning models — Snowflake Inc. (US12314318B2)

ABSTRACT

Abstract

An advanced search system leverages a pre-trained large language model to enhance user query responses. The system, equipped with hardware processors, a search query via an interface and accesses a pre-trained large language model designed to respond to the search query. The system fine-tunes the model to generate a task-specific generative model. The system employs the task-specific generative model to generate a search result to the search query and analyzes the search result based on a performance metric associated with the task-specific generative model. The system refines the task-specific generative model based on the analyzing of the search result.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims benefit of earlier filing date and right of priority to U.S. Provisional Patent Application Ser. No. 63/446,750, filed on Feb. 17, 2023, entitled, “SYSTEM, METHOD, AND APPARATUS FOR SEARCH ENHANCEMENT,” all of the contents of which are hereby incorporated by reference herein in its entirety.

TECHNICAL FIELD

The present disclosure generally relates to special-purpose machines that use large language models and generative artificial intelligence for summarization, more specifically, to provide enhanced search capabilities using fine-tuned machine learning models.

BACKGROUND

The current state of the art in search technologies encompasses systems capable of indexing and searching through extensive collections of digital information. These systems utilize complex algorithms to analyze and rank web pages, documents, and other data sources based on their relevance to user queries. The ranking mechanisms often consider factors such as keyword frequency, site authority, and user engagement metrics.

Machine learning models, including various forms of deep learning architectures like neural networks, have been increasingly integrated into search technologies. These models are trained on large datasets to predict the relevance of content, personalize search experiences, and automate the summarization of information. Natural language processing (NLP) plays a crucial role in enhancing search capabilities, allowing for a more nuanced understanding of both the user's query and the content within the indexed data. NLP techniques enable the extraction of meaningful patterns, sentiment, and entities from text, which can improve the accuracy and contextuality of search results.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The present disclosure will be apparent from the following more particular description of examples of embodiments of the technology, as illustrated in the accompanying drawings. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating embodiments of the present disclosure. In the drawings, like numerals may describe similar components in different views. Like numerals having different letter suffixes may represent different instances of similar components. Various ones of the appended drawings merely illustrate example embodiments of the present disclosure and should not be considered as limiting its scope.

FIG. 1 is a system architecture diagram illustrating an example of a machine learning model fine-tuning system, according to some example embodiments.

FIG. 2 is a data flow diagram illustrating data movement through a fine-tuning system and interactive user interface, according to some example embodiments.

FIG. 3 is a block diagram illustrating a large language model and task-specific generative model paradigm, according to some example embodiments.

FIG. 4 is a block diagram illustrating a fine-tuning system to fine-tune a pre-trained model that is further trained on a smaller, task-specific dataset, according to some example embodiments.

FIG. 5 is a query processing pipeline illustrating selection and use of one or more machine-learning programs, according to some example embodiments.

FIG. 6 illustrates a model architecture illustrating generative artificial intelligence model selection and use of one or more machine-learning programs, according to some example embodiments.

FIG. 7 illustrates a general model architecture for generating models according to reward modeling, according to some example embodiments.

FIG. 8 illustrates a method for generating a task-specific generative model, according to some example embodiments.

FIG. 9 illustrates a method for improving interference latency of a neural network, according to some example embodiments.

FIG. 10 illustrates a user interface implementation of the machine learning model fine-tuning system presented on a browser of a user's user device, according to some example embodiments.

FIG. 11 is a user interface diagram illustrating an implementation of the multi-document summarization system output, according to some example embodiments.

FIG. 12 illustrates a method for receiving an initial search query from a browser-based search interface, according to some example embodiments.

FIG. 13 is a block diagram illustrating a machine-learning pipeline, according to some example embodiments.

FIG. 14 is a data flow diagram illustrating training and use of a machine-learning program, according to some example embodiments.

FIG. 15 is a data flow diagram illustrating content generation with generative artificial intelligence, according to some example embodiments.

FIG. 16 is an example diagrammatic representation illustrating a machine in the form of a computer system within which a set of instructions may be executed for causing the machine to perform any one or more of the methodologies discussed herein, according to some example embodiments.

DETAILED DESCRIPTION

The description that follows includes systems, methods, techniques, instruction sequences, and computing machine program products that embody illustrative embodiments of the disclosure. In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide an understanding of various embodiments of the inventive subject matter. It will be evident, however, to those skilled in the art, that embodiments of the inventive subject matter may be practiced without these specific details. In general, well-known instruction instances, protocols, structures, and techniques are not necessarily shown in detail. For purposes of this description, the phrase “enhanced summarization system” may be referred to as and used interchangeably with the phrases “a multi-document summarization system,” or merely “summarization system.”

Disclosed herein are various examples of systems, methods, and machine-readable mediums for using an interactive interface combined with generative artificial intelligence to generate an answer to a user's query or set of queries utilizing large language models (LLMs) that have been fine-tuned into task-specific models optimized for different types of content. Examples provide techniques to create training data using LLMs, to model compression methods to reduce model size for fast inference, and to provide reward modeling to improve summarization quality. Example embodiments include machine learning models used for enhanced search capabilities using fine-tuned large language models, including the process of training and/or fine-tuning the models, the roles of large models (opposed to task-oriented models) in generating training data, and techniques for reducing model size (e.g., reducing large models to smaller, task-oriented models) for faster performance. In some examples, the user interface is a browser-based interface, but other examples can be used, such as an application-based interface or other interface; for exemplary purposes, a browser-based interface is used but a person have ordinary skill in the art will understand that any interface may similarly be implemented and used.

Existing search engine(s) fail to provide succinct, accurate, and comprehensive answers to user queries within a browser-based interface. Current search systems often require users to sift through multiple results and/or multiple pages of results and perform additional research (e.g., clicking on hyperlinks for each document) to find the information the user needs. The conventional wisdom is to prompt the one largest LLM possible to receive the best output; however, the use of one LLM causes scalability, resource, and speed problems.

To address these and other issues and shortcomings of existing search engines, example embodiments of the fine-tuning system eliminate prior inefficiencies by summarizing information from various sources into a coherent and interactive response, tailored to the user's specific query using fine-tuned machine learning models to enhance search capabilities. Task-specific generative models can leverage problem structure to run smaller, faster, and cheaper at web scale.

Disclosed herein are various examples of systems and methods for a generative artificial intelligence (GenAI) based search system that leverages machine learning models to optimize automated data generation, model compression, and reward modeling enhancements that provide for summarization, as well as applicability beyond summarization. Example embodiments provide a system for model training, model tuning, and fine-tuning processing underpinned by sophisticated machine learning models that have undergone extensive training and fine-tuning to provide increased efficacy. For example, models are adept at interpreting the nuances of natural language, enabling them to extract and synthesize information from a multitude of documents. Examples of the training process involves leveraging large, pre-existing, and new models to generate high-quality training data, which can then be used to fine-tune smaller, more efficient models tailored to specific tasks, such as summarization tasks, citation tasks, web-interface building tasks, or the like.

Examples of the model training and fine-tuning system architecture are designed to be scalable, for example, using a combination of open-source models fine-tuned with proprietary data, the system is more cost-effective and efficient than relying solely on large, expensive LLMs. The advantages of the model training and fine-tuning system disclosed herein provide a more advanced, user-friendly, and efficient solution compared to traditional search engines, driven by machine learning models that have been trained on large datasets. Examples of these models are capable of natural language understanding, allowing the models to parse and summarize complex documents. Examples of the training process include using larger models to generate training data, which is then used to fine-tune smaller, domain-specific models for efficiency.

Example embodiments of the model training and fine-tuning system provide a search system that utilizes advanced machine learning techniques to generate interactive, high-quality search results based on domain-specific models. For example, a method for improving search engine responses using machine learning models that are fine-tuned on specific datasets to provide accurate, fluent, and/or comprehensive answers derived from multiple web sources or other documents. Examples of the system employ large models for generating training data and implement techniques to reduce the size of operational models, to ensure faster performance without compromising the quality of search results.

The system further innovates model training, where it fine-tunes smaller, domain-specific models using, for example, labels generated from larger, more comprehensive models. This allows for efficient scaling and deployment of the system to handle real-time user queries with lower latency.

Additionally, the system incorporates a reward modeling component that uses heuristics and user feedback to continuously improve the quality of the generated answers, ensuring that the system adapts and evolves with use. Examples of the model training and fine-tuning system provide for enhanced multi-document summarization advancements in search technology by providing a system that not only simplifies the user's search experience but also delivers high-quality, synthesized answers that are both informative and easily digestible.

Example embodiments of the model training and fine-tuning system encompasses a system that leverages Large Language Models (LLMs) to enhance the user experience in search query responses. The LLMs employed within the system are based on sophisticated encoder-decoder architectures, which are instrumental in processing and generating natural language. These models are particularly adept at creating personalized, digestible, and comprehensive responses to search queries, thereby significantly improving the relevance and utility of the information presented to the user. In the process of training and fine-tuning these LLMs, the system utilizes supervised fine-tuning (SFT) techniques that involve the use of curated input-output pairs. This fine-tuning is further refined through human annotation and reward modeling, which aligns the models'outputs with human preferences. The training process is designed to optimize the models to maximize a mean reward, which is indicative of the models'alignment with human judgments and preferences.

The role of large models in the generation of training data is pivotal within the system. These pre-trained models are utilized to generate an initial corpus of training data, which serves as the foundation for fine-tuning smaller, more specialized models. The system employs strategies to ensure that the datasets created are diverse and of high quality, which is crucial for the effective fine-tuning of the models. Large models also serve as a benchmark for performance, providing a baseline against which the fine-tuned models can be evaluated. To enhance the system's efficiency, examples of the fine-tuning system incorporate techniques for reducing the size of the models without compromising their performance. For example, asymmetric compression techniques are implemented to decrease the model size, which in turn positively impacts inference efficiency and reduces search result latency. The performance of these reduced-size models is meticulously compared with their larger counterparts, ensuring that the reduction in size does not detrimentally affect the models' accuracy, fluency, and/or the overall user experience. This balance between size and performance is useful in the deployment of the system in real-world applications where speed and resource utilization are of paramount importance, such as in a browser-based interface.

Example embodiments of the model training and fine-tuning system can use the enhanced summarization system that provides a novel approach to search technology that significantly enhances the user experience by generating a single, comprehensive answer from multiple sources using advanced natural language processing and machine learning techniques to interpret user queries, summarize relevant information, and synthesize this information into a cohesive response including citation of the original sources within the browser-based interface, without a new webpage being generated. Example embodiments of the enhanced summarization system can stand alone or be incorporated in a technical architecture that is multi-layered framework that integrates several components, such as a search engine to retrieve relevant documents from a vast index of web pages, a summarization engine configured to summarize the content of retrieved and/or selected documents, a query processor to analyze and interpret the user's search intent, a response generator that compiles the summaries into a human-coherent answer to the user's query, cross-document summarization algorithms, and the like. The algorithms underpinning the cross-document summarization system are engineered to navigate the intricacies of discrepancies, contradictions, and the multitude of perspectives that may emerge across the document corpus. For example, the system is adept at managing information redundancy, employing advanced mechanisms to ensure that duplicate content from various documents is effectively synthesized into a

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims benefit of earlier filing date and right of priority to U.S. Provisional Patent Application Ser. No. 63/446,750, filed on Feb. 17, 2023, entitled, “SYSTEM, METHOD, AND APPARATUS FOR SEARCH ENHANCEMENT,” all of the contents of which are hereby incorporated by reference herein in its entirety.

TECHNICAL FIELD

The present disclosure generally relates to special-purpose machines that use large language models and generative artificial intelligence for summarization, more specifically, to provide enhanced search capabilities using fine-tuned machine learning models.

BACKGROUND

The current state of the art in search technologies encompasses systems capable of indexing and searching through extensive collections of digital information. These systems utilize complex algorithms to analyze and rank web pages, documents, and other data sources based on their relevance to user queries. The ranking mechanisms often consider factors such as keyword frequency, site authority, and user engagement metrics.

Machine learning models, including various forms of deep learning architectures like neural networks, have been increasingly integrated into search technologies. These models are trained on large datasets to predict the relevance of content, personalize search experiences, and automate the summarization of information. Natural language processing (NLP) plays a crucial role in enhancing search capabilities, allowing for a more nuanced understanding of both the user's query and the content within the indexed data. NLP techniques enable the extraction of meaningful patterns, sentiment, and entities from text, which can improve the accuracy and contextuality of search results.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The present disclosure will be apparent from the following more particular description of examples of embodiments of the technology, as illustrated in the accompanying drawings. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating embodiments of the present disclosure. In the drawings, like numerals may describe similar components in different views. Like numerals having different letter suffixes may represent different instances of similar components. Various ones of the appended drawings merely illustrate example embodiments of the present disclosure and should not be considered as limiting its scope.

FIG. 1 is a system architecture diagram illustrating an example of a machine learning model fine-tuning system, according to some example embodiments.

FIG. 2 is a data flow diagram illustrating data movement through a fine-tuning system and interactive user interface, according to some example embodiments.

FIG. 3 is a block diagram illustrating a large language model and task-specific generative model paradigm, according to some example embodiments.

FIG. 4 is a block diagram illustrating a fine-tuning system to fine-tune a pre-trained model that is further trained on a smaller, task-specific dataset, according to some example embodiments.

FIG. 5 is a query processing pipeline illustrating selection and use of one or more machine-learning programs, according to some example embodiments.

FIG. 6 illustrates a model architecture illustrating generative artificial intelligence model selection and use of one or more machine-learning programs, according to some example embodiments.

FIG. 7 illustrates a general model architecture for generating models according to reward modeling, according to some example embodiments.

FIG. 8 illustrates a method for generating a task-specific generative model, according to some example embodiments.

FIG. 9 illustrates a method for improving interference latency of a neural network, according to some example embodiments.

FIG. 10 illustrates a user interface implementation of the machine learning model fine-tuning system presented on a browser of a user's user device, according to some example embodiments.

FIG. 11 is a user interface diagram illustrating an implementation of the multi-document summarization system output, according to some example embodiments.

FIG. 12 illustrates a method for receiving an initial search query from a browser-based search interface, according to some example embodiments.

FIG. 13 is a block diagram illustrating a machine-learning pipeline, according to some example embodiments.

FIG. 14 is a data flow diagram illustrating training and use of a machine-learning program, according to some example embodiments.

FIG. 15 is a data flow diagram illustrating content generation with generative artificial intelligence, according to some example embodiments.

FIG. 16 is an example diagrammatic representation illustrating a machine in the form of a computer system within which a set of instructions may be executed for causing the machine to perform any one or more of the methodologies discussed herein, according to some example embodiments.

DETAILED DESCRIPTION

The description that follows includes systems, methods, techniques, instruction sequences, and computing machine program products that embody illustrative embodiments of the disclosure. In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide an understanding of various embodiments of the inventive subject matter. It will be evident, however, to those skilled in the art, that embodiments of the inventive subject matter may be practiced without these specific details. In general, well-known instruction instances, protocols, structures, and techniques are not necessarily shown in detail. For purposes of this description, the phrase “enhanced summarization system” may be referred to as and used interchangeably with the phrases “a multi-document summarization system,” or merely “summarization system.”

Disclosed herein are various examples of systems, methods, and machine-readable mediums for using an interactive interface combined with generative artificial intelligence to generate an answer to a user's query or set of queries utilizing large language models (LLMs) that have been fine-tuned into task-specific models optimized for different types of content. Examples provide techniques to create training data using LLMs, to model compression methods to reduce model size for fast inference, and to provide reward modeling to improve summarization quality. Example embodiments include machine learning models used for enhanced search capabilities using fine-tuned large language models, including the process of training and/or fine-tuning the models, the roles of large models (opposed to task-oriented models) in generating training data, and techniques for reducing model size (e.g., reducing large models to smaller, task-oriented models) for faster performance. In some examples, the user interface is a browser-based interface, but other examples can be used, such as an application-based interface or other interface; for exemplary purposes, a browser-based interface is used but a person have ordinary skill in the art will understand that any interface may similarly be implemented and used.

Existing search engine(s) fail to provide succinct, accurate, and comprehensive answers to user queries within a browser-based interface. Current search systems often require users to sift through multiple results and/or multiple pages of results and perform additional research (e.g., clicking on hyperlinks for each document) to find the information the user needs. The conventional wisdom is to prompt the one largest LLM possible to receive the best output; however, the use of one LLM causes scalability, resource, and speed problems.

To address these and other issues and shortcomings of existing search engines, example embodiments of the fine-tuning system eliminate prior inefficiencies by summarizing information from various sources into a coherent and interactive response, tailored to the user's specific query using fine-tuned machine learning models to enhance search capabilities. Task-specific generative models can leverage problem structure to run smaller, faster, and cheaper at web scale.

Disclosed herein are various examples of systems and methods for a generative artificial intelligence (GenAI) based search system that leverages machine learning models to optimize automated data generation, model compression, and reward modeling enhancements that provide for summarization, as well as applicability beyond summarization. Example embodiments provide a system for model training, model tuning, and fine-tuning processing underpinned by sophisticated machine learning models that have undergone extensive training and fine-tuning to provide increased efficacy. For example, models are adept at interpreting the nuances of natural language, enabling them to extract and synthesize information from a multitude of documents. Examples of the training process involves leveraging large, pre-existing, and new models to generate high-quality training data, which can then be used to fine-tune smaller, more efficient models tailored to specific tasks, such as summarization tasks, citation tasks, web-interface building tasks, or the like.

Examples of the model training and fine-tuning system architecture are designed to be scalable, for example, using a combination of open-source models fine-tuned with proprietary data, the system is more cost-effective and efficient than relying solely on large, expensive LLMs. The advantages of the model training and fine-tuning system disclosed herein provide a more advanced, user-friendly, and efficient solution compared to traditional search engines, driven by machine learning models that have been trained on large datasets. Examples of these models are capable of natural language understanding, allowing the models to parse and summarize complex documents. Examples of the training process include using larger models to generate training data, which is then used to fine-tune smaller, domain-specific models for efficiency.

Example embodiments of the model training and fine-tuning system provide a search system that utilizes advanced machine learning techniques to generate interactive, high-quality search results based on domain-specific models. For example, a method for improving search engine responses using machine learning models that are fine-tuned on specific datasets to provide accurate, fluent, and/or comprehensive answers derived from multiple web sources or other documents. Examples of the system employ large models for generating training data and implement techniques to reduce the size of operational models, to ensure faster performance without compromising the quality of search results.

The system further innovates model training, where it fine-tunes smaller, domain-specific models using, for example, labels generated from larger, more comprehensive models. This allows for efficient scaling and deployment of the system to handle real-time user queries with lower latency.

Additionally, the system incorporates a reward modeling component that uses heuristics and user feedback to continuously improve the quality of the generated answers, ensuring that the system adapts and evolves with use. Examples of the model training and fine-tuning system provide for enhanced multi-document summarization advancements in search technology by providing a system that not only simplifies the user's search experience but also delivers high-quality, synthesized answers that are both informative and easily digestible.

Example embodiments of the model training and fine-tuning system encompasses a system that leverages Large Language Models (LLMs) to enhance the user experience in search query responses. The LLMs employed within the system are based on sophisticated encoder-decoder architectures, which are instrumental in processing and generating natural language. These models are particularly adept at creating personalized, digestible, and comprehensive responses to search queries, thereby significantly improving the relevance and utility of the information presented to the user. In the process of training and fine-tuning these LLMs, the system utilizes supervised fine-tuning (SFT) techniques that involve the use of curated input-output pairs. This fine-tuning is further refined through human annotation and reward modeling, which aligns the models'outputs with human preferences. The training process is designed to optimize the models to maximize a mean reward, which is indicative of the models'alignment with human judgments and preferences.

The role of large models in the generation of training data is pivotal within the system. These pre-trained models are utilized to generate an initial corpus of training data, which serves as the foundation for fine-tuning smaller, more specialized models. The system employs strategies to ensure that the datasets created are diverse and of high quality, which is crucial for the effective fine-tuning of the models. Large models also serve as a benchmark for performance, providing a baseline against which the fine-tuned models can be evaluated. To enhance the system's efficiency, examples of the fine-tuning system incorporate techniques for reducing the size of the models without compromising their performance. For example, asymmetric compression techniques are implemented to decrease the model size, which in turn positively impacts inference efficiency and reduces search result latency. The performance of these reduced-size models is meticulously compared with their larger counterparts, ensuring that the reduction in size does not detrimentally affect the models' accuracy, fluency, and/or the overall user experience. This balance between size and performance is useful in the deployment of the system in real-world applications where speed and resource utilization are of paramount importance, such as in a browser-based interface.

Example embodiments of the model training and fine-tuning system can use the enhanced summarization system that provides a novel approach to search technology that significantly enhances the user experience by generating a single, comprehensive answer from multiple sources using advanced natural language processing and machine learning techniques to interpret user queries, summarize relevant information, and synthesize this information into a cohesive response including citation of the original sources within the browser-based interface, without a new webpage being generated. Example embodiments of the enhanced summarization system can stand alone or be incorporated in a technical architecture that is multi-layered framework that integrates several components, such as a search engine to retrieve relevant documents from a vast index of web pages, a summarization engine configured to summarize the content of retrieved and/or selected documents, a query processor to analyze and interpret the user's search intent, a response generator that compiles the summaries into a human-coherent answer to the user's query, cross-document summarization algorithms, and the like. The algorithms underpinning the cross-document summarization system are engineered to navigate the intricacies of discrepancies, contradictions, and the multitude of perspectives that may emerge across the document corpus. For example, the system is adept at managing information redundancy, employing advanced mechanisms to ensure that duplicate content from various documents is effectively synthesized into a cohesive summary. A salient feature of the cross-document summarization system is its robust citation and attribution framework. Unlike single document summarization, which does not necessitate the management of multiple citations for identical information, the cross-document summarization system meticulously attributes each piece of synthesized content to its respective source documents. The two-phase combination of summarization techniques according to examples of the present disclosure not only enhances the integrity and traceability of the summarized information but also enriches the user's understanding by providing a clear lineage of the content's origins.

In a second example embodiment of the present disclosure disclosed herein are various examples of systems, methods, and machine-readable mediums for using a browser-based interface combined with generative artificial intelligence (GenAI or GAI) to provide an interactive search component. Examples of the interactive search component of a user interface provide for real-time or near real-time adjustment of parameters (e.g., source type, trust level, etc.), interactive formatting preferences for user query results (e.g., final summarized answer to a user query), and interactive clarifying questions to refine search intent, all within the browser-based interface. For example, examples provide user interface functionality including real-time control over query results (e.g., search answers), clarification dialogs to improve search results within the browser-based interface, and user-feedback system outputs to provide the user for providing continuous user feedback on system status and outputs all without leaving the browser-based interface.

Current web search engine(s) operate through a systematic process that involves crawling, indexing, and retrieving information from the World Wide Web to present relevant results in response to user queries. The search engine deploys automated programs (commonly referred to as “crawlers” or “spiders”) that traverse the web by following hyperlinks from one web page to another. These crawlers systematically browse the web to discover and access publicly available web pages. The crawling process is guided by algorithms that determine the frequency, path, and number of pages to crawl. Once a web page is accessed, the search engine processes and analyzes the content of the page to understand its subject matter. Key elements such as text, images, and video content are extracted, and the information is organized in a database known as an “index.” This index is designed to efficiently store and retrieve data, with the content being categorized based on keywords, tags, and other relevant metadata. Common search engines employ complex ranking algorithm(s) to evaluate the relevance and authority of web pages in relation to specific search queries. Factors that may influence ranking include keyword density, the quality and quantity of inbound links, user engagement metrics, website speed, mobile friendliness, the freshness of content, and the like. The algorithm(s) assign a rank to each web page within the index, which determines the order in which pages are presented in search results.

When a user enters a search query into a search query input field (commonly referred to as a “search box,” an “input box,” or “search bar,” the search engine processes the query by parsing and understanding the user's intent. The query processing component may include natural language processing (NLP) techniques to handle complex queries, synonyms, and context. Based on the processed query, the search engine retrieves the most relevant web pages from the index. The retrieval mechanism uses the ranking algorithm(s) to select and order the pages that best match the user's query. The search engine presents the retrieved results to the user through a user interface, typically a web page that displays a list of search results. Each result is simply a snippet of information about the web page, generally including a title, a URL, and a brief description or excerpt from the web page. The interface may also offer advanced search options and filters to refine the results. Modern search engines incorporate machine learning algorithms that analyze user interaction with search results (e.g., click-through rates, time spent on a page, etc.) to continuously improve the relevance and accuracy of the search results.

Previously known search tools and systems suffered from a number of challenges that result in users getting sub-optimal search results, getting partial returns for search results, and spending excessive time getting effective search results, as well as parsing through multiple search results to get all of the information they are looking for. Search technologies have evolved to index and retrieve vast amounts of data from the web, offering users access to a wide range of information. Existing technologies typically employ algorithms to rank search results based on relevance to the user's query. The indexed information may come from diverse sources, including web pages, databases, and specialized documents; however, the output of a web search using these existing technologies requires a user to individually open and review tens to hundreds of hyperlinks and web pages. Over time, advancements in natural language processing have enabled more sophisticated interpretations of user queries and the content of potential search results. The primary challenge in existing search technologies includes the inadequacy of existing search engines to provide succinct, accurate, and comprehensive answers to user queries using machine learning optimizations.

The present web search technologies encounter difficulties when processing user queries that are vague or possess multiple interpretations, leading to challenges in discerning the true intent of the user. Current search engines may yield search results that lack relevance or contain low-quality information, such as outdated or incorrect data. The algorithms governing search engine operations may unintentionally exhibit bias towards certain websites or sources. Search engines that rely heavily on keywords may overlook the broader context of user queries. While personalization in search results can enhance user experience, it can also lead to the creation of “filter bubbles,” limiting exposure to diverse information. The indexing of non-textual content such as images, videos, and audio by search engines is often less effective than textual content. Search engines may exhibit suboptimal performance in non-English languages or in regions with lower levels of internet access. Further, search results are often polluted with spam or low-quality content that manipulates search engine ranking and/or filtering.

Examples of the present disclosure overcome the existing technological difficulties by introducing an interactive user interface in a browser-based interface, a novel methodology that represents a significant departure from the existing web search technologies. This methodology incorporates unique processes, algorithms, or systems that address existing challenges including interactive user interface functionality in the browser-based interface, without a new webpage being generated. The examples of the interactive user interface system provide advanced technical solutions that enhance performance, efficiency, and/or user experience beyond the current state of the art. The example embodiments further feature an unprecedented integration of components or systems that synergistically produce enhanced results to user queries in the browser-based interface, without a new webpage being generated. The interconnectivity and interaction between these components provide technical advantages and benefits, including improvements in speed, accuracy, reliability, scalability, user engagement, cost-effectiveness, or other measurable metrics related to user query and result output in the browser-based interface.

For example, the system's ability to handle summarization is a standout feature that leverages advanced machine learning techniques to process and condense information from various documents into concise summaries. While example embodiments of the present disclosure are provided with reference to an enhanced summarization system, it will be understood by those having ordinary skill in the art that the model training and fine-tuning system and/or the interactive user interface system described herein can be used on other systems outside the realm of summarization.

FIG. 1 is an example high-level system architecture illustrating an example of a machine learning model fine- tuning system 106 including a controller 102 embodying circuits, controllers, computing devices, data stores, communication infrastructure (e.g., network connections, protocols, etc.), or the like that implement operations described herein, in accordance with some embodiments of the present disclosure.

As utilized herein, circuits, controllers, computing devices, components, modules, or other similar aspects set forth herein should be understood broadly. Such terminology is utilized to highlight that the related hardware devices may be configured in a number of arrangements, and include any hardware configured to perform the operations herein. Any such devices may be a single device, a distributed device, and/or implemented as any hardware configuration to perform the described operations. In certain embodiments, hardware devices may include computing devices of any type, logic circuits, input/output devices, processors, sensors, actuators, web-based servers, LAN servers, WLAN servers, cloud computing devices, memory storage of any type, and/or aspects embodied as instructions stored on a computer readable medium and configured to cause a processor to perform recited operations. Communication between devices, whether inter-communication (e.g., a user device 104 communicating with the controller 102 ) or intra-device communication (e.g., one circuit or component of the controller 102 communicating with another circuit or component of the controller 102 ) may be performed in any manner, for example using internet-based communication, LAN/WLAN communication, direct networking communication, Wi-Fi communication, or the like.

The example controller 102 is configured to provide enhanced search results to the user device 104 , based on a knowledge corpus 112 of information available to the controller 102 . The example knowledge corpus 112 may be any type of knowledge base of documents, for example the entire public internet (e.g., based on a web index built by a web crawler). An example knowledge corpus 112 includes one or more aspects such as web accessible documents, proprietary database information (e.g., a database for a company, engineering documents, a subscription-based information data set, etc.), specific knowledge bases (e.g., journals, white papers, or the like), and/or additional data sources. The operations of the system architecture 100 may be performed on any corpus of documents and may be utilized for general search purposes (e.g., a user searching the internet) and/or for specific search purposes (e.g., a user searching a specific corpus of documents to determine responsive information that may be present therein).

The example controller 102 interfaces with a user device 104 , for example associated with a user 108 , to receive search queries, user preferences, user response to search results (e.g., selection of certain returns, pursuing links, further search queries, etc.) of any type as set forth herein, and interfaces with the machine learning model fine- tuning system 106 . The user device 104 , controller 102 , and machine learning model fine- tuning system 106 are shown as separate devices for clarity of the present description, but these may be on distinct devices, on the same device in whole or part (e.g., a part, or all, of the machine learning model fine- tuning system 106 stored on the controller 102 , for example where a proprietary database is stored on a same device, web server, or the like as the controller 102 ; where a proprietary database is stored on a same device, web server, or the like as the user device 104 ; and/or where one or more circuits, components, or other aspects of the controller 102 are positioned, in whole or part, on the user device 104 ).

The machine learning model fine- tuning system 106 includes a model quality optimization engine 132 , a model performance optimization engine 134 , a training data generator 136 , a model compression engine 138 (described in detail in connection with FIG. 6 ), and a reward modeling engine 140 (described in detail in connection with FIG. 7 ) to help refine large language models into task-specific (e.g., domain-specific) generative models. The model quality optimization engine 132 improves the overall quality of the LLM's and/or task-specific generative model (e.g., smaller model) outputs. ‘Quality’can encompass various attributes, for example and not limitation, such as coherence, relevance, factual accuracy, and fluency of the generated text. The optimization engine uses techniques like hyperparameter tuning, architecture search, or advanced training strategies to refine the model's ability to produce high-quality results. The model performance optimization engine 134 improves the performance of the systems and models presented throughout, such as improving the computational aspects related to model performance, for example, speed, efficiency, and resource usage of the model. A model performance optimization engine enhances how well the model runs, focusing on reducing latency, optimizing memory usage, and improving throughput. Techniques include model pruning, quantization, and knowledge distillation, which help make the model smaller and faster without significantly compromising its predictive capabilities. The training data generator 136 is responsible for creating the datasets used to train or fine-tune the LLMs. A training data generator might automate the collection, cleaning, and labeling of data, ensuring that the model has a diverse and representative set of examples to learn from. It can involve techniques like data scraping, synthetic data generation, or semi-supervised learning approaches where the model itself helps to generate new training examples.

The example controller 102 finds responsive results in the knowledge corpus 112 , constructs a search result via a result builder 130 , provides the search result to the user device via a user interface 144 of a browser 142 of the user device 104 , and/or receives feedback from the user to perform further searching or the like. According to example embodiments, the controller 102 includes a search interface component 126 that implements a search interface on the user device 104 , for example providing a search query window, providing the user interface 144 for the user to indicate preferences, to select aspects of the search result(s), to receive suggestions from the controller 102 , or the like.

In certain embodiments, the search interface component 126 interprets responsive information from the user (e.g., receiving and/or processing search terms for queries), interprets user inputs related to preferences for searching (e.g., interface display and/or arrangement, priorities to be applied to certain aspects herein such as trust determinations for sources, treatment of certain content types such as machine generated content; and/or configuration options for returned search results such as number of results to be returned, citation options, thresholds for classification or other determinations, or the like), interprets other data related to the user interface experience (e.g., where the user focus is, for example, based on cursor locations and/or user eye positioning determined from a camera, and/or time the user spends on aspects such as time to enter a query, time spent reading or following source elements), and/or stores user interaction information for further use by the system architecture 100 within the same or a subsequent search session, or the like).

In certain embodiments, the search interface component 126 creates and/or operates within a single session, for example, a user searching within a single window of a web browser 142 , and/or can operate across multiple sessions sequentially (e.g., using search history after the browser is closed and reopened) and/or simultaneously (e.g., the user is performing two searches in separate windows, tabs, and/or on separate user devices). In certain embodiments, the search interface component 126 provides search results to the user, for example, providing the constructed search results to the window being accessed by the user. The example controller 102 interprets search queries 114 , historical data 122 , and/or provides constructed search results 120 to the user via the browser 142 . Interpreted parameters may be explicit (e.g., as entered by the user), inferred (e.g., adjusted spelling of terms, based on observed data from the user rather than explicitly entered information, etc.), and/or combinations thereof.

Examples of the search interface component 126 further include a search interface implementation 146 component and a search intent and classification 148 component that perform additional operations related to search queries. For example, the search interface implementation 146 performs operations to provide the search interface to the user 108 , for the user to indicate preferences and respond to search results, and/or to provide constructed search results to the user. The example search interface component 126 further includes a search intent and classification 148 component, that parses the search query for specific terms, and/or that provides one or more classifications for the query, which may be utilized to determine which sources in the knowledge corpus 112 should be utilized, what the purpose of the search is, which type of information is most responsive to the search, etc. The search intent and classification 148 component may utilize any type of classifier and/or intent determiner known in the art. In certain embodiments, multiple intents and/or classifications of the search query may be determined and/or ranked and used to determine which answer or answers to construct.

The example controller 102 includes a result construction component 128 that queries the knowledge corpus 112 for responsive documents to a search, parses the documents for responsive portions (e.g., sentences, paragraphs, phrases, tables, graphical information, etc.), constructs a single best answer that is responsive to the search, and provides the single best answer to the user responsive to the search. As utilized herein, a single best answer (referred to as “exactly one answer”) includes a search result that is constructed to be responsive to the user search, and may include elements from multiple source documents, with citations within the single best answer to the multiple source documents utilized to construct the answer. In example embodiments, aspects from the multiple source documents may be processed to determine the responsive information, for example, including paraphrasing, summarizing, aggregating, or otherwise including derived information from the multiple source documents to create a responsive result. In example embodiments, the result construction component 128 may include more than one, or many, “single best answers,” for example, where a classification of the search query indicates that multiple intents may be present, an example result construction component 128 may construct a single best answer for more than one (or all) of the intents. In example embodiments, for example where more than one single best answer is provided, information may be provided to the user about each of the answers (e.g., including language related to the intent determination for each answer). In certain embodiments, information may be provided to the user for a single best answer where only one is provided, for example, a described intent that is determined for the query may be provided to the user, for example to enhance the user confidence that the query was understood and that the search results are likely to be the best results.

In example embodiments, determined intents may be provided to the user, and/or accessible to the user for confirmation, modification, or the like; for example, allowing the user to explicitly adjust the intent and repeat the search. In certain embodiments, the result construction component 128 further combines the search result elements into a cohesive answer, for example as a paragraph, page, sentence, graph, or the like. In certain embodiments, the controller 102 includes a Natural Language Processor 150 , which may be available to any other aspect or component of the controller 102 , for example to be utilized by the result construction component 128 to adjust the single best answer into a naturally readable answer, to parse the search query into specific terms, indicated intents, classification of the query, or the like. The example single best answer as set forth herein is an abstracted answer, for example, an answer including derivative information from a number of references, where one or more elements of the answer may not appear individually within any single reference, as opposed to an extractive answer where the best available single source is provided to the user. In certain embodiments, search results may additionally include one or more extractive answers, for example provided as additional search results below the single best answer (or single best answers).

The example controller 102 includes a user information processing search interface component 126 that interprets and/or stores user information utilized by aspects of the system architecture 100 , for example determining user intents, user behavior, user search history, user preferences, or the like. In certain embodiments, the user information processing search interface component 126 is further able to integrate user information from offset users (e.g., other users that have a similarity to the current user, for example based on a classification of the user type, similarity in search terminology and/or search logic, similarity in search responses such as the type of sources trusted, language in responses that the user tends to favor as responsive, etc.). The user type may be a classification related to the user that can be utilized to inform responsive search results, and may be based upon characteristics of the user (e.g., user education level, user history favoring certain document types such as news, academic papers, articles from particular industry, etc.) and/or the context of the user search operations (e.g., professional user, academic user, casual user, traveling user, etc.), including aspects such as the device being used by the user, the time of day, the geographic location of the user, etc. While the user type, user characteristics, and/or user context may be utilized to enhance search servicing herein, aspects of the present disclosure are nevertheless beneficial to user interactions where no information is available about the specific user, for example basing operations to parse and classify queries and/or construct answers based on only information available about the user through interactions within a single searching session.

Returning to the controller 102 , the controller includes the result construction component 128 , which includes a result parsing component (not shown), which performs operations to parse potentially responsive documents for relevant data, text, tables, figures, or the like. Operations to parse potentially responsive documents may include providing a vector description of elements of the potentially responsive documents, allowing for identification of relevant portions, as well as determining which elements to utilize in constructing an answer. The example results construction component 128 includes a result builder 130 component, which determines which elements to include in the answer, ordering of the elements, combination of elements into single sentences, paragraphs, and/or visual elements, or the like. In example embodiments, the result builder 130 component accesses the Natural Language Processor 150 , which may be utilized to finalize the answer into a natural reading information packet: for example, as sentences, paragraphs, illustrations, and/or as a web page or other document.

In example embodiments, the result construction component 128 can paraphrase, summarize, aggregate, or otherwise derive information from the sources, for example to focus on aspects of the sources that are responsive to the search query, and/or to construct an answer from a number of sources that, taken together, are responsive to the search query even where none of the individual references are responsive to the specific query. For example, a first source document may have a capacity description for a container, and a second source document may have a volume description for a commodity, where the result construction component 128 is capable of answering a query about how much of the commodity can be stored in the container. The example results construction component 128 includes a citation processor (not shown) component that incorporates citations into the constructed answer, for example using linked text (e.g., text elements from the source that are hyperlinked to the source), footnotes, or the like. The utilization of citations ensures that proper attribution is given to the source documents, enhances user confidence in the answer, and/or provides the user with additional original sources for further research.

The example user information processing component 110 operates on any type of user information as set forth throughout the present disclosure. An example user information processing component 110 includes user preferences 118 and/or search session management 116 . Any or all of this information may be explicit (e.g., provided by the user), inferred (e.g., based on user interactions and/or other information available about the user such as in a profile of the user stored by the system architecture 100 , available from public information such as public records, social media information, or the like, inferred from correlations with other information about the user (e.g., a user that is an engineer searching about an engineering topic may be more likely to be a “professional” user rather than a “casual” user in that context), and/or combinations thereof. This user information search session management 116 and/or user preferences 118 may additionally or alternatively be based on the context of the user, such as time of day, which browser is being utilized, which user device is being utilized, etc. Inferred information and/or context may be further modified by pattern recognition operations, for example when the user performs searches at a certain time of day, day of the week, using certain search terms, or the like, patterns may be recognized indicating intents or classifications for searches, the types of documents the user is looking for, and the like.

Additional example embodiments utilize user activity data (not shown) to adjust search query answers and/or other determinations throughout the present dis

CLAIMS

Claims ( 20 )

What is claimed is:

1. A system comprising:

one or more hardware processors of a machine; and

at least one memory storing instructions that, when executed by the one or more hardware processors, cause the system to perform operations comprising:

receiving, by the one or more hardware processors, a search query via an interface;

accessing a pre-trained large language model designed to respond to the search query; and

performing a plurality of iterations, using the pre-trained large language model, to generate a task-specific generative model, each iteration of the plurality of iterations comprising:

performing domain-specific pre-training on an index to fine tune the pre-trained large language model;

employing the task-specific generative model to generate a search result to the search query;

analyzing the search result based on a performance metric associated with the task-specific generative model; and

refining the task-specific generative model based on the analyzing of the search result.

2. The system of claim 1 , wherein performing the plurality of iterations further comprises:

generating one or more outputs from the task-specific generative model by applying the plurality of iterations on a new search query, the one or more outputs including the search result; and

providing the one or more outputs to a user via the interface, wherein the interface is a browser-based interface.

3. The system of claim 2 , wherein performing the plurality of iterations further comprises:

receiving user feedback based on the one or more outputs; and

utilizing the user feedback to improve accuracy and fluency of the search result generated by the task-specific generative model.

4. The system of claim 1 , wherein performing the plurality of iterations further comprises:

evaluating a quality of the task-specific generative model upon conclusion of each iteration, wherein the quality includes a percentage of correct search results; and

stopping the plurality of iterations after the quality of the task-specific generative model is satisfactory to a user.

5. The system of claim 1 , the operations further comprising:

reducing a size of the task-specific generative model using asymmetric compression techniques including selective pruning of task-specific generative model parameters without identified loss of model performance.

6. The system of claim 1 , wherein performing the domain-specific pre-training on the index to fine tune the pre-trained large language model further comprises:

tailoring the pre-trained large language model using proprietary data, the proprietary data including a curated dataset representative of a plurality of types of queries and content associated with a search system.

7. The system of claim 1 , the operations further comprising:

applying a reward modeling process to the task-specific generative model to align the search result with human preferences; and

improving a quality of the search result based on the reward modeling process, wherein the reward modeling process includes collecting human annotations to define a reward function that approximates human judgments of fluency and relevance associated with the search result.

8. A method comprising:

receiving, by one or more hardware processors, a search query via an interface;

accessing a pre-trained large language model designed to respond to the search query; and

performing a plurality of iterations, using the pre-trained large language model, to generate a task-specific generative model, each iteration comprising:

performing domain-specific pre-training on an index to fine tune the pre-trained large language model;

employing the task-specific generative model to generate a search result to the search query;

analyzing the search result based on a performance metric associated with the task-specific generative model; and

refining the task-specific generative model based on the analyzing of the search result.

9. The method of claim 8 , wherein performing the plurality of iterations further comprises:

generating one or more outputs from the task-specific generative model by applying the plurality of iterations on a new search query, the one or more outputs including the search result; and

providing the one or more outputs to the user.

10. The method of claim 9 , wherein performing the plurality of iterations further comprises:

receiving, from the user, user feedback based on the one or more outputs; and

utilizing the user feedback to literately improve accuracy and fluency of the search result generated by the task-specific generative model.

11. The method of claim 8 , wherein performing the plurality of iterations further comprises:

evaluating a quality of the task-specific generative model at an end of each iteration, wherein the quality includes a percentage of correct search results; and

stopping the plurality of iterations after the quality of the task-specific generative model is satisfactory to the user.

12. The method of claim 8 , further comprising:

reducing a size of the task-specific generative model using asymmetric compression techniques including selective pruning of task-specific generative model parameters without identified loss of model performance.

13. The method of claim 8 , wherein performing the domain-specific pre-training on the index to fine tune the pre-trained large language model further comprises:

tailoring the pre-trained large language model using proprietary data, the proprietary data including a curated dataset representative of a plurality of types of queries and content associated with a search system.

14. The method of claim 8 , further comprising:

applying a reward modeling process to the task-specific generative model to align the search result with human preferences; and

improving a quality of the search result based on the reward modeling process, wherein the reward modeling process includes collecting human annotations to define a reward function that approximates human judgments of fluency and relevance associated with the search result.

15. A machine-storage medium embodying instructions that, when executed by a machine, cause the machine to perform operations comprising:

receiving, by one or more hardware processors, a search query via an interface;

accessing a pre-trained large language model designed to respond to the search query; and

performing a plurality of iterations, using the pre-trained large language model, to generate a task-specific generative model, each iteration comprising:

performing domain-specific pre-training on an index to fine tune the pre-trained large language model;

employing the task-specific generative model to generate a search result to the search query;

analyzing the search result based on a performance metric associated with the task-specific generative model; and

refining the task-specific generative model based on the analyzing of the search result.

16. The machine-storage medium of claim 15 , wherein performing the plurality of iterations further comprises:

generating one or more outputs from the task-specific generative model by applying the plurality of iterations on a new search query, the one or more outputs including the search result; and

providing the one or more outputs to a user device.

17. The machine-storage medium of claim 16 , wherein performing the plurality of iterations further comprises:

receiving user feedback based on the one or more outputs; and

utilizing the user feedback to literately improve accuracy and fluency of the search result generated by the task-specific generative model.

18. The machine-storage medium of claim 15 , wherein performing the plurality of iterations further comprises:

evaluating a quality of the task-specific generative model at an end of each iteration, wherein the quality includes a percentage of correct search results; and

stopping the plurality of iterations after the quality of the task-specific generative model is satisfactory to a user.

19. The machine-storage medium of claim 15 , wherein performing the domain-specific pre-training on the index to fine tune the pre-trained large language model further comprises:

tailoring the pre-trained large language model using proprietary data, the proprietary data including a curated dataset representative of a plurality of types of queries and content associated with a search system.

20. The machine-storage medium of claim 15 , wherein the operations comprise:

applying a reward modeling process to the task-specific generative model to align the search result with human preferences; and

improving a quality of the search result based on the reward modeling process, wherein the reward modeling process includes collecting human annotations to define a reward function that approximates human judgments of fluency and relevance associated with the search result.

US18/444,078

2023-02-17

2024-02-16

Enhanced searching using fine-tuned machine learning models

Active

US12314318B2

( en )

Priority Applications (2)

Application Number

Priority Date

Filing Date

Title

US18/444,078

US12314318B2

( en )

2023-02-17

2024-02-16

Enhanced searching using fine-tuned machine learning models

US19/082,454

US20250217418A1

( en )

2023-02-17

2025-03-18

Enhanced searching using fine-tuned machine learning models

Applications Claiming Priority (2)

Application Number

Priority Date

Filing Date

Title

US202363446750P

2023-02-17

2023-02-17

US18/444,078

US12314318B2

( en )

2023-02-17

2024-02-16

Enhanced searching using fine-tuned machine learning models

Related Child Applications (1)

Application Number

Title

Priority Date

Filing Date

US19/082,454

Continuation

US20250217418A1

( en )

2023-02-17

2025-03-18

Enhanced searching using fine-tuned machine learning models

Publications (2)

Publication Number

Publication Date

US20240281446A1

US20240281446A1 ( en )

2024-08-22

US12314318B2

true

US12314318B2 ( en )

2025-05-27

Family

ID=92304192

Family Applications (4)

Application Number

Title

Priority Date

Filing Date

US18/443,903

Pending

US20240281472A1

( en )

2023-02-17

2024-02-16

Interactive interface with generative artificial intelligence

US18/444,078

Active

US12314318B2

( en )

2023-02-17

2024-02-16

Enhanced searching using fine-tuned machine learning models

US18/443,838

Active

US12561375B2

( en )

2023-02-17

2024-02-16

Enhanced search result generation using multi-document summarization

US19/082,454

Pending

US20250217418A1

( en )

2023-02-17

2025-03-18

Enhanced searching using fine-tuned machine learning models

Family Applications Before (1)

Application Number

Title

Priority Date

Filing Date

US18/443,903

Pending

US20240281472A1

( en )

2023-02-17

2024-02-16

Interactive interface with generative artificial intelligence

Family Applications After (2)

Application Number

Title

Priority Date

Filing Date

US18/443,838

Active

US12561375B2

( en )

2023-02-17

2024-02-16

Enhanced search result generation using multi-document summarization

US19/082,454

Pending

US20250217418A1

( en )

2023-02-17

2025-03-18

Enhanced searching using fine-tuned machine learning models

Country Status (1)

Country

Link

US

( 4 )

US20240281472A1

( en )

Cited By (1)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US12406028B1

( en )

*

2024-08-22

2025-09-02

Accenture Global Solutions Limited

System and method for analyzing and tuning input dataset to a foundation model

Families Citing this family (108)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US12277735B2

( en )

2017-07-03

2025-04-15

StyleRiser Inc.

Style profile engine

US12001462B1

( en )

*

2023-05-04

2024-06-04

Vijay Madisetti

Method and system for multi-level artificial intelligence supercomputer design

US12561391B2

( en )

*

2021-04-02

2026-02-24

Relativity Oda Llc

Methods and systems for presenting user interfaces to render multiple documents

US20240340301A1

( en )

2022-05-31

2024-10-10

As0001, Inc.

Adaptive security architecture based on state of posture

US12333612B2

( en )

2022-05-31

2025-06-17

As0001, Inc.

Systems and methods for dynamic valuation of protection products

US12177242B2

( en )

2022-05-31

2024-12-24

As0001, Inc.

Systems and methods for dynamic valuation of protection products

US12244703B2

( en )

2022-05-31

2025-03-04

As0001, Inc.

Systems and methods for configuration locking

US12047400B2

( en )

2022-05-31

2024-07-23

As0001, Inc.

Adaptive security architecture based on state of posture

US12236491B2

( en )

2022-05-31

2025-02-25

As0001, Inc.

Systems and methods for synchronizing and protecting data

US12189787B2

( en )

*

2022-05-31

2025-01-07

As0001, Inc.

Systems and methods for protection modeling

US20240281472A1

( en )

*

2023-02-17

2024-08-22

Snowflake Inc.

Interactive interface with generative artificial intelligence

US20240296295A1

( en )

*

2023-03-03

2024-09-05

Microsoft Technology Licensing, Llc

Attribution verification for answers and summaries generated from large language models (llms)

US12505307B2

( en )

*

2023-04-12

2025-12-23

Microsoft Technology Licensing, Llc

Natural language training and/or augmentation with large language models

FR3148110B3

( en )

*

2023-04-21

2025-04-11

Amadeus Sas

Data enrichment system

US12572575B2

( en )

*

2023-05-31

2026-03-10

Highspot, Inc.

Using large language models to generate search query answers

US12456013B2

( en )

*

2023-06-06

2025-10-28

Salesforce, Inc.

Systems and methods for training a neural network model using knowledge from pre-trained large language models

US20240420391A1

( en )

*

2023-06-16

2024-12-19

The Toronto-Dominion Bank

Intelligent dashboard search engine

US12585879B2

( en )

*

2023-06-20

2026-03-24

T-Mobile Innovations Llc

Artificial intelligence assisted network operations reporting and management

US12505308B2

( en )

*

2023-06-22

2025-12-23

Microsoft Technology Licensing, Llc

Research activities through conversational user experience using multi-modal large pretrained models

US12353458B2

( en )

*

2023-07-21

2025-07-08

Google Llc

Deep linking using generative artificial intelligence

US20250036636A1

( en )

*

2023-07-26

2025-01-30

Ford Global Technologies, Llc

Intelligent virtual assistant selection

US11916767B1

( en )

*

2023-08-03

2024-02-27

Dropzone.ai, Inc.

Security analysis agents

US20250045316A1

( en )

*

2023-08-04

2025-02-06

Google Llc

Systems and Methods for Generating Instruction Fine-tuning Dataset for a General Purpose Embedding Model

US20250053835A1

( en )

*

2023-08-07

2025-02-13

Trunk Tools, Inc.

Methods and systems for generative question answering for construction project data

US20250061140A1

( en )

*

2023-08-17

2025-02-20

CS Disco, Inc.

Systems and methods for enhancing search using semantic search results

US12450505B2

( en )

*

2023-08-24

2025-10-21

Tiny Fish Inc.

Utilizing large language model responses to train an inference pattern engine

US12236216B1

( en )

2023-08-24

2025-02-25

Tiny Fish Inc.

Generate a script to automate a task associated with a webpage

US20250124235A1

( en )

*

2023-10-11

2025-04-17

Adobe Inc.

Using generative artificial intelligence to evaluate fine-tuned language models

US20250124215A1

( en )

*

2023-10-11

2025-04-17

Concord Iii, Llc

Summarization processor for unstructured health care documentation received from over a communications network

US12585643B2

( en )

*

2023-10-17

2026-03-24

Salesforce, Inc.

Generated content source attribution

US20250165480A1

( en )

*

2023-11-20

2025-05-22

Elsevier Inc.

Systems and methods for retrieval augmented generation

US20250166762A1

( en )

*

2023-11-21

2025-05-22

GE Precision Healthcare LLC

Clinical workflows utilizing patient report summarization and q&a technologies

US12437016B2

( en )

*

2023-12-07

2025-10-07

Google Llc

Fine-tuning large language model(s) using reinforcement learning with search engine feedback

US20250209127A1

( en )

*

2023-12-20

2025-06-26

Google Llc

Search System Having Task-Based Machined-Learned Models

US12602436B2

( en )

*

2024-01-17

2026-04-14

Ai Seer Pte. Ltd.

System for real-time fact-checking of multimedia content

US12608545B2

( en )

*

2024-01-19

2026-04-21

Salesforce, Inc.

Validating generative artificial intelligence output

US12511105B2

( en )

*

2024-02-02

2025-12-30

Driver AI, Inc.

Transpiler to extract and use intermediate representations of a code base

US12468770B2

( en )

*

2024-02-15

2025-11-11

Qomplx Llc

Dynamic experience curation platform

US12293272B1

( en )

*

2024-03-08

2025-05-06

Seekr Technologies, Inc.

Agentic workflow system and method for generating synthetic data for training or post training artificial intelligence models to be aligned with domain-specific principles

US20240273120A1

( en )

*

2024-03-14

2024-08-15

Intel Corporation

Automatic data source marking using autonomous systems

US12437238B1

( en )

2024-03-20

2025-10-07

Anthropic, Pbc

Generation of agentic trajectories for training artificial intelligence agents to automate multimodal interface task workflows

US20250298816A1

( en )

*

2024-03-20

2025-09-25

Counsel AI Corporation

Document question answering system using layered language models

US12531788B2

( en )

*

2024-03-27

2026-01-20

Infosys Limited

Method and system for automatically generating network design documents using generative artificial intelligence (AI) models

US20250307775A1

( en )

*

2024-03-27

2025-10-02

Procore Technologies, Inc.

Computing System and Method for Creating and Executing Attribute-Specific Predictive Analytics Pipelines for a Construction Project

US12254005B1

( en )

*

2024-03-29

2025-03-18

nference, inc.

Systems and methods for retrieving patient information using large language models

US12579174B1

( en )

*

2024-03-29

2026-03-17

Amazon Technologies, Inc.

Evaluating retrieval system for language model processing

US12517945B2

( en )

*

2024-04-05

2026-01-06

Dell Products L.P.

System and method for enhancing generative artificial intelligence (AI) model-based document search with image retrieval

US12602410B2

( en )

*

2024-04-10

2026-04-14

Dell Products L.P.

Multimodal context selection for large language model based resolutions addressing technical issues

US20250335521A1

( en )

*

2024-04-30

2025-10-30

Maplebear Inc.

Supplementing a search query using a large language model

US12105746B1

( en )

2024-05-01

2024-10-01

Dropzone.ai, Inc.

System for surveying security environments

KR20250161877A

( en )

*

2024-05-09

2025-11-18

라인플러스 주식회사

Method and system for advanced retrieval-augmented generation through metadata extraction and retrieval engine mixing

US20250348497A1

( en )

*

2024-05-13

2025-11-13

Apple Inc.

Techniques for providing relevant search results for search queries

US20250363676A1

( en )

*

2024-05-24

2025-11-27

Microsoft Technology Licensing, Llc

Electronic sticker packs generated by artificial intelligence based on user prompt

US12332858B1

( en )

*

2024-06-03

2025-06-17

Bank Of America Corporation

Systems and methods for integrated analysis of foreground and background communication data

US12505137B1

( en )

*

2024-06-21

2025-12-23

Microsoft Technology Licensing, Llc

Digital content generation with in-prompt hallucination management for conversational agent

US20260010573A1

( en )

*

2024-07-03

2026-01-08

Sas Institute Inc.

System and method for compressing prompts to language models for document processing

WO2026015471A1

( en )

*

2024-07-08

2026-01-15

Bria Artificial Intelligence Ltd

Controllable and attributable generation of machine-generated content

CN118467845B

( en )

*

2024-07-09

2024-09-17

卡奥斯工业智能研究院(青岛)有限公司

Method for constructing intelligent interactive service system, website intelligent interactive method and device

US20260017302A1

( en )

*

2024-07-11

2026-01-15

Shopify Inc.

Methods and systems for updating a retrieval-augmented generation framework

US12561335B2

( en )

*

2024-07-12

2026-02-24

Glean Technologies, Inc.

Automated expert detection

WO2026019927A1

( en )

*

2024-07-17

2026-01-22

Reliaquest Holdings, Llc

Threat mitigation system and method

US12235856B1

( en )

2024-08-26

2025-02-25

Dropbox, Inc.

Performing unified search using a hybrid search index

WO2026049834A1

( en )

*

2024-08-26

2026-03-05

Dropbox, Inc.

Performing unified search using a hybrid search index

US12332949B1

( en )

2024-08-26

2025-06-17

Dropbox, Inc.

Generating a hybrid search index for unified search

WO2026049857A1

( en )

*

2024-08-27

2026-03-05

Microsoft Technology Licensing, Llc

Aligning large language models with in-situ user interactions and feedback

WO2026044544A1

( en )

*

2024-08-28

2026-03-05

北京字跳网络技术有限公司

Method for optimizing model, device, medium and program product

CN119025655A

( en )

*

2024-08-28

2024-11-26

无限光年(上海)技术有限公司

Language model training method and device, information processing method and device

WO2026047914A1

( en )

*

2024-08-28

2026-03-05

株式会社Ｎｔｔドコモ

Device and method

US20260064739A1

( en )

*

2024-08-29

2026-03-05

Adp, Inc.

Large language model architecture for delivering digital content

US12493620B1

( en )

*

2024-08-30

2025-12-09

Snap Inc.

Hybrid client and server ranking system

KR102840101B1

( en )

*

2024-09-04

2025-08-01

사이오닉에이아이 주식회사

Method and electronic device for generating language model

US12287837B1

( en )

*

2024-09-10

2025-04-29

Oxylabs, Uab

Generating a path to a document element using machine learning

WO2026057163A1

( en )

*

2024-09-12

2026-03-19

Huawei Cloud Computing Technologies Co., Ltd.

Method and controller configured to execute agent for adapting llm response based on user preferences

CN118779437A

( en )

*

2024-09-12

2024-10-15

北京市大数据中心

A large model RAG method based on hierarchical indexing and hybrid retrieval

US12585661B1

( en )

*

2024-09-20

2026-03-24

Notion Labs, Inc.

User engagement quality estimation

US12423265B1

( en )

2024-09-23

2025-09-23

Aravo Solutions, Inc.

Prompting a large language model for vector embeddings and metadata to generate an indexed computing file

US12399889B1

( en )

2024-09-23

2025-08-26

Aravo Solutions, Inc.

Modifying a database using a large language model

US20260086986A1

( en )

*

2024-09-23

2026-03-26

Aravo Solutions, Inc.

Utilizing a large language model to generate a computing structure

US12436966B1

( en )

*

2024-09-23

2025-10-07

Aravo Solutions, Inc.

Utilizing a large language model to generate a computing structure

US12367213B1

( en )

2024-09-23

2025-07-22

Aravo Solutions, Inc.

Generating structured data from unstructured data using metadata from a large language model

WO2026071536A1

( en )

*

2024-09-24

2026-04-02

삼성전자 주식회사

Electronic device for providing result to user utterance, operating method thereof, and non-transitory storage medium

CN118839021B

( en )

*

2024-09-24

2025-01-21

山东浪潮科学研究院有限公司

A dynamic relevance enhanced retrieval generation system and method driven by intelligent knowledge graph

KR20260045368A

( en )

*

2024-09-27

2026-04-03

(주)아이와즈

VFX generation intention analysis system using LLM and Method thereof

US12499115B1

( en )

*

2024-09-27

2025-12-16

Salesforce, Inc.

Systems and methods for a reasoning-intensive reranking based artificial intelligence conversation agent

US12481665B1

( en )

*

2024-09-27

2025-11-25

Amazon Technologies, Inc.

Configurable interactive components for structured query results

WO2026089798A1

( en )

*

2024-10-21

2026-04-30

Microsoft Technology Licensing, Llc

Continuous learning for multi-agent system

US20260111431A1

( en )

*

2024-10-21

2026-04-23

Microsoft Technology Licensing, Llc

Continuous learning for multi-agent system

CN119381009B

( en )

*

2024-10-23

2025-10-28

北京航空航天大学

Retrieval enhancement method and system based on structural entropy hierarchical knowledge tree

US20260119470A1

( en )

*

2024-10-25

2026-04-30

Delve Deep Learning Inc.

Artificially intelligent content surfacing of relevant content from heterogeneous content sources

US20260119138A1

( en )

*

2024-10-31

2026-04-30

Microsoft Technology Licensing, Llc

Rendering gui components for refining queries to generative machine learning models

US12619649B1

( en )

*

2024-11-01

2026-05-05

Intuit Inc.

Large language model input preprocessing and refinement

CN119476219B

( en )

*

2024-11-14

2025-10-14

哈尔滨工业大学

Public opinion report generation method based on retrieval-enhanced generation

US12536177B1

( en )

2024-11-22

2026-01-27

Anomali Incorporated

Incremental search results for sequential partial data queries

CN119202206A

( en )

*

2024-11-26

2024-12-27

北京百车宝科技有限公司

An automobile intelligent question answering system and method based on large model and retrieval enhancement

CN119204025B

( en )

*

2024-11-27

2025-05-13

山东浪潮科学研究院有限公司

Method, system, equipment and medium for verifying facts of large language model based on retrieval enhancement

CN119621911A

( en )

*

2024-12-06

2025-03-14

广州华微明天软件技术有限公司

Iterative retrieval-enhanced question answer generation method

CN119917965B

( en )

*

2024-12-09

2026-04-14

复旦大学

Steel production equipment fault diagnosis system based on large language model

US20250139108A1

( en )

*

2024-12-28

2025-05-01

Miao Cui

Design Strategies to Prevent Data Hallucination in Large Language Models for the Medical Field

CN119494410B

( en )

*

2025-01-20

2025-05-23

湖南农业大学

Retrieval generation optimization method, medium and system based on long-term enhancement

US12511557B1

( en )

2025-01-21

2025-12-30

Seekr Technologies Inc.

System and method for explaining and contesting outcomes of generative AI models with desired explanation properties

CN119938858B

( en )

*

2025-01-21

2025-10-31

中山大学

Multimodal emergency information generation methods, devices, electronic equipment, and media

CN120296110B

( en )

*

2025-06-11

2025-08-08

浙江师范大学

Intelligent question-answering method and device based on domain knowledge lightweight fine tuning and cross-domain dynamic knowledge base and readable storage medium thereof

US12489778B1

( en )

2025-08-11

2025-12-02

Dropzone.ai, Inc.

Context repository management

US12499243B1

( en )

2025-08-11

2025-12-16

Dropzone.ai, Inc.

Automated threat hunting

CN121257699A

( en )

*

2025-08-13

2026-01-02

粤港澳大湾区(广东)国创中心

Industrial material conversational recommendation method and system based on large model and elastic search

CN121009200A

( en )

*

2025-08-20

2025-11-25

北京字跳网络技术有限公司

Content display methods, devices, equipment, computer-readable storage media and products

US12609968B1

( en )

2025-11-19

2026-04-21

Dropzone.ai, Inc.

Dynamic playbooks for security investigations

CN121256067B

( en )

*

2025-12-08

2026-04-10

三峡高科信息技术有限责任公司

Methods, devices, equipment, and storage media for enhancing enterprise multi-source knowledge tracing

Citations (22)

Related documents

• Generating augmented reality experiences utilizing physical objects to … — Meta Platforms Technologies, Llc (US20260037214A1)#653960
• Virtual reality system using multiple force arrays for a solver — DeepMotion, Inc. (US10657696B2)#653961
• Tracking position of device inside-out for virtual reality interactivity — Sony Interactive Entertainment America Llc (US10535153B2)#653962
• Augmented reality system with spatialized audio tied to user manipulated … — Magic Leap, Inc. (US11231770B2)#653963
• Big data-based autonomous flight drone system and autonomous flight method … — Rgblab Co., Ltd. (US20210390867A1)#653964