ABSTRACT
Abstract
Relevance scores may be determined based on text included in a document. The text may be divided into a text portions, with the relevance scores being determined based on a comparison of a text portion of the plurality of text portions with a criterion specified in natural language. A subset of the plurality of text portions may be selected based on the plurality of relevance scores, with each of the subset of the plurality of text portions having a relevance score surpassing a threshold. A criteria evaluation prompt may be sent to a remote text generation modeling system via a communication interface. The criteria evaluation prompts may include an instruction to evaluate one or more of the subset of text portions against the criterion.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
This application claims priority to U.S. patent application Ser. No. 18/329,039, filed Jun. 5, 2023 by Qadrud-Din et al., titled âLARGE LANGUAGE MODEL ARTIFICIAL INTELLIGENCE TEXT EVALUATION SYSTEMâ, which claims priority to provisional U.S. Patent App. No. 63/487,330, filed Feb. 28, 2023 by Qadrud-Din et al., titled âLARGE LANGUAGE MODEL ARTIFICIAL INTELLIGENCE TEXT EVALUATION SYSTEMâ, which is hereby incorporated by reference in its entirety and for all purposes.
FIELD OF TECHNOLOGY
This patent document relates generally to natural language processing and more specifically to interactions with text generation modeling systems.
BACKGROUND
Companies often need to evaluate documents against one or more criteria. For example, documents of a given type may be required to include particular legal disclaimers under company policy. As another example, a company may require that all contracts of a given type contain a particular term. The evaluation and enforcement of such policies using conventional techniques is a largely manual process. For large companies and large pools of documents, such manual evaluation and enforcement of policies as applied to text-based documents can be prohibitively time consuming and expensive. Accordingly, improved techniques for policy evaluation are desired.
BRIEF DESCRIPTION OF THE DRAWINGS
The included drawings are for illustrative purposes and serve only to provide examples of possible structures and operations for the disclosed inventive systems, apparatus, methods and computer program products for text analysis and text generation. These drawings in no way limit any changes in form and detail that may be made by one skilled in the art without departing from the spirit and scope of the disclosed implementations.
FIG. 1 illustrates a document policy evaluation overview method, performed in accordance with one or more embodiments.
FIG. 2 illustrates a text generation system, configured in accordance with one or more embodiments.
FIG. 3 illustrates a document parsing method, performed in accordance with one or more embodiments.
FIG. 4 illustrates a text generation method, performed in accordance with one or more embodiments.
FIG. 5 illustrates a method of sharding text, performed in accordance with one or more embodiments.
FIG. 6 illustrates a text chunk determination method, performed in accordance with one or more embodiments.
FIG. 7 illustrates one example of a computing device, configured in accordance with one or more embodiments.
FIG. 8 illustrates an example of a method for conducting a chat session, performed in accordance with one or more embodiments.
FIG. 9 illustrates an example of a method for generating a document summary, performed in accordance with one or more embodiments.
FIG. 10 illustrates an example of a method for generating a timeline, performed in accordance with one or more embodiments.
FIG. 11 illustrates a flow diagram for generating correspondence, configured in accordance with one or more embodiments.
FIG. 12 illustrates a hallucination detection method, performed in accordance with one or more embodiments.
FIG. 13 illustrates a policy evaluation pre-processing method, performed in accordance with one or more embodiments.
FIG. 14 illustrates a text portion selection first stage method, performed in accordance with one or more embodiments.
FIG. 15 illustrates a cross-encoder modeling system, configured in accordance with one or more embodiments.
FIG. 16 illustrates a text portion selection second stage method, performed in accordance with one or more embodiments.
FIG. 17 illustrates a policy evaluation method, performed in accordance with one or more embodiments.
DETAILED DESCRIPTION
Techniques and mechanisms described herein provide for the automated evaluation of text against criteria specified in natural language. According to various embodiments, one or more operations may be performed via a large language model capable of evaluating and generating natural language text. A policy including one or more criteria may be specified in natural language. A document may be divided into a set of clauses. These clauses may then be evaluated for relevance against the one or more criteria. Clauses deemed relevant may then be individually evaluated for compliance against the relevant criteria. Information about clause-level compliance may then be aggregated to the document level to determine whether the document as a whole complies with the policy. When the document is deemed non-compliant, one or more remedial actions may be performed.
In many contexts, a large number of documents need to be evaluated against a policy. For example, a company may identify terms that the company prefers to include in all of their contracts of a given type. As another example, a company may require that all documents of a particular type include a legal disclaimer, notice, or other language. However, evaluating documents using simple text-searching approaches is inadequate because the language may vary between documents. Accordingly, conventional techniques for evaluating documents against a policy involve considerable manual review.
In contrast to conventional techniques, techniques and mechanisms described herein provide for automated processes for evaluating documents against a policy. According to various embodiments, a user may specify a policy that includes one or more terms described in natural language. The system may then identify areas of documents that implicate those terms and determine whether the documents comply with the terms. If a document is identified as non-compliant, the system may perform one or more remedial actions such as summarizing the differences and/or suggesting changes to the document to bring it into compliance.
According to various embodiments, techniques and mechanisms described herein provide for novel text generation in domain-specific contexts. A text generation interface system may take as input one or more arbitrary documents, process them via optical text recognition, segment them into portions, and process the segmented text via various tasks based on need. Different workflows are provided for different tasks, and this application describes a number of examples of such workflows. In many workflows, an input document is divided into chunks via a chunking technique. Then, chunks are inserted into prompt templates for processing by a large language model such as the GPT-3 or GPT-4 available from OpenAI. The large language model's response is then parsed and potentially used to trigger additional analysis, such as one or more database searches, one or more additional prompts sent back to the large language model, and/or a response returned to a client machine.
According to various embodiments, techniques and mechanisms described herein provide for retrieval augmented generation. A search is conducted base on a search query. Then, the search results are provided to an artificial intelligence system. The artificial intelligence system then further processes the search results to produce an answer based on those search results. In this context, a large language model may be used to determine the search query, apply one or more filters and/or tags, and/or synthesize potentially many different types of search.
According to various embodiments, techniques and mechanisms described herein provide for a sophisticated document processing pipeline. The pipeline receives one or more input documents, identifies text that should be kept together, identifies extraneous text such as headers, footers, and line numbers, and segments the text accordingly. In this way, the quality of the text provided to the rest of the system is improved.
According to various embodiments, techniques and mechanisms described herein provide for new approaches to text segmentation. Large language models often receive as input a portion of input text and generate in response a portion of output text. In many systems, the large language model imposes a limit on the input text size. Accordingly, in the event that the large language model is asked to summarize a length document, the document may need to be segmented into portions in order to achieve the desired summarization.
Conventional text segmentation techniques frequently create divisions in text that negatively affect the performance of the model, particularly in domains-specific contexts such as law. For example, consider a caption page of a legal brief, which includes text in a column on the left that encompasses the parties, text in a column on the right that includes the case number, a title that follows lower on the page, and line numbering on the left. In such a configuration, the text in the different columns should not be mixed and should be treated separately from the line numbers, while both columns should precede the document title, when converting the document to an input query for a large language model. However, conventional techniques would result in these semantically different elements of text being jumbled together, resulting in an uninformative query provided to the large language model and hence a low quality response. In contrast to these conventional techniques, techniques and mechanisms described herein provide for a pipeline that cleans such raw text so that it can be provided to a large language model.
According to various embodiments, techniques and mechanisms described herein provide for the division of text into chunks, and the incorporation of those chunks into prompts that can be provided to a large language model. For instance, a large language model may impose a limit of, for instance, 8,193 tokens on a task, including text input, text output, and task instructions. In order to process longer documents, the system may split them. However, splitting a document can easily destroy meaning depending on where and how the document is split. Techniques and mechanisms described herein provide for evenly splitting a document or documents into chunks, and incorporating those chunks into prompts, in ways that retain the semantic content associated with the raw input document or documents.
In some embodiments, techniques and mechanisms described herein may be applied to generate novel text in domain-specific contexts, such as legal analysis. Large language models, while powerful, have a number of drawbacks when used for technical, domain-specific tasks. When using conventional techniques, large language models often invent âfactsâ that are actually not true. For instance, if asked to summarize the law related to non-obviousness in the patent context, a large language model might easily invent a court case, complete with caption and ruling, that in fact did not occur. In contrast to conventional techniques, techniques and mechanisms described herein provide for the generation of novel text in domain-specific contexts while avoiding such drawbacks.
According to various embodiments, techniques and mechanisms described herein may be used to automate complex, domain-specific tasks that were previously the sole domain of well-trained humans. Moreover, such tasks may be executed in ways that are significantly faster, less expensive, and more auditable than the equivalent tasks performed by humans. For example, a large language model may be employed to produce accurate summaries of legal texts, to perform legal research tasks, to generate legal documents, to generate questions for legal depositions, and the like.
In some embodiments, techniques and mechanisms described herein may be used to divide text into portions while respecting semantic boundaries and simultaneously reducing calls to the large language model. The cost of using many large language models depends on the amount of input and/or output text. Accordingly, techniques and mechanisms described herein provide for reduced overhead associated with prompt instructions while at the same time providing for improved model context to yield an improved response.
In some embodiments, techniques and mechanisms described herein may be used to process an arbitrary number of unique documents (e.g., legal documents) that cannot be accurately parsed and processed via existing optical character recognition and text segmentation solutions.
In some embodiments, techniques and mechanisms described herein may be used to link a large language model with a legal research database, allowing the large language model to automatically determine appropriate searches to perform and then ground its responses to a source of truth (e.g., in actual law) so that it does not âhallucinateâ a response that is inaccurate.
In some embodiments, techniques and mechanisms described herein provide for specific improvements in the legal domain. For example, tasks that were previously too laborious for attorneys with smaller staffs may now be more easily accomplished. As another example, attorneys may automatically analyze large volumes of documents rather than needing to perform such tasks manually. As another example, text chunking may reduce token overhead and hence cost expended on large language model prompts. As yet another example, text chunking may reduce calls to a large language model, increasing response speed. As still another example, text chunking may increase and preserve context provided to a large language model by dividing text into chunks in semantically meaningful ways.
According to various embodiments, techniques and mechanisms described herein may provide for automated solutions for generated text in accordance with a number of specialized applications. Such applications may include, but are not limited to: simplifying language, generating correspondence, generating a timeline, reviewing documents, editing a contract clause, drafting a contract, performing legal research, preparing for a depositions, drafting legal interrogatories, drafting requests for admission, drafting requests for production, briefing a litigation case, responding to requests for admission, responding to interrogatories, responding to requests for production, analyzing cited authorities, and answering a complaint.
FIG. 1 illustrates a document policy evaluation overview method 100 , performed in accordance with one or more embodiments. In some implementations, the method 100 may be performed at a text generation interface system such as the system 200 shown in FIG. 2 . For instance, the <figure-callout id="100" label="method" filenames="US12299406-20250513-D00000.png,US12299406-20250513-D
CROSS-REFERENCE TO RELATED APPLICATIONS
This application claims priority to U.S. patent application Ser. No. 18/329,039, filed Jun. 5, 2023 by Qadrud-Din et al., titled âLARGE LANGUAGE MODEL ARTIFICIAL INTELLIGENCE TEXT EVALUATION SYSTEMâ, which claims priority to provisional U.S. Patent App. No. 63/487,330, filed Feb. 28, 2023 by Qadrud-Din et al., titled âLARGE LANGUAGE MODEL ARTIFICIAL INTELLIGENCE TEXT EVALUATION SYSTEMâ, which is hereby incorporated by reference in its entirety and for all purposes.
FIELD OF TECHNOLOGY
This patent document relates generally to natural language processing and more specifically to interactions with text generation modeling systems.
BACKGROUND
Companies often need to evaluate documents against one or more criteria. For example, documents of a given type may be required to include particular legal disclaimers under company policy. As another example, a company may require that all contracts of a given type contain a particular term. The evaluation and enforcement of such policies using conventional techniques is a largely manual process. For large companies and large pools of documents, such manual evaluation and enforcement of policies as applied to text-based documents can be prohibitively time consuming and expensive. Accordingly, improved techniques for policy evaluation are desired.
BRIEF DESCRIPTION OF THE DRAWINGS
The included drawings are for illustrative purposes and serve only to provide examples of possible structures and operations for the disclosed inventive systems, apparatus, methods and computer program products for text analysis and text generation. These drawings in no way limit any changes in form and detail that may be made by one skilled in the art without departing from the spirit and scope of the disclosed implementations.
FIG. 1 illustrates a document policy evaluation overview method, performed in accordance with one or more embodiments.
FIG. 2 illustrates a text generation system, configured in accordance with one or more embodiments.
FIG. 3 illustrates a document parsing method, performed in accordance with one or more embodiments.
FIG. 4 illustrates a text generation method, performed in accordance with one or more embodiments.
FIG. 5 illustrates a method of sharding text, performed in accordance with one or more embodiments.
FIG. 6 illustrates a text chunk determination method, performed in accordance with one or more embodiments.
FIG. 7 illustrates one example of a computing device, configured in accordance with one or more embodiments.
FIG. 8 illustrates an example of a method for conducting a chat session, performed in accordance with one or more embodiments.
FIG. 9 illustrates an example of a method for generating a document summary, performed in accordance with one or more embodiments.
FIG. 10 illustrates an example of a method for generating a timeline, performed in accordance with one or more embodiments.
FIG. 11 illustrates a flow diagram for generating correspondence, configured in accordance with one or more embodiments.
FIG. 12 illustrates a hallucination detection method, performed in accordance with one or more embodiments.
FIG. 13 illustrates a policy evaluation pre-processing method, performed in accordance with one or more embodiments.
FIG. 14 illustrates a text portion selection first stage method, performed in accordance with one or more embodiments.
FIG. 15 illustrates a cross-encoder modeling system, configured in accordance with one or more embodiments.
FIG. 16 illustrates a text portion selection second stage method, performed in accordance with one or more embodiments.
FIG. 17 illustrates a policy evaluation method, performed in accordance with one or more embodiments.
DETAILED DESCRIPTION
Techniques and mechanisms described herein provide for the automated evaluation of text against criteria specified in natural language. According to various embodiments, one or more operations may be performed via a large language model capable of evaluating and generating natural language text. A policy including one or more criteria may be specified in natural language. A document may be divided into a set of clauses. These clauses may then be evaluated for relevance against the one or more criteria. Clauses deemed relevant may then be individually evaluated for compliance against the relevant criteria. Information about clause-level compliance may then be aggregated to the document level to determine whether the document as a whole complies with the policy. When the document is deemed non-compliant, one or more remedial actions may be performed.
In many contexts, a large number of documents need to be evaluated against a policy. For example, a company may identify terms that the company prefers to include in all of their contracts of a given type. As another example, a company may require that all documents of a particular type include a legal disclaimer, notice, or other language. However, evaluating documents using simple text-searching approaches is inadequate because the language may vary between documents. Accordingly, conventional techniques for evaluating documents against a policy involve considerable manual review.
In contrast to conventional techniques, techniques and mechanisms described herein provide for automated processes for evaluating documents against a policy. According to various embodiments, a user may specify a policy that includes one or more terms described in natural language. The system may then identify areas of documents that implicate those terms and determine whether the documents comply with the terms. If a document is identified as non-compliant, the system may perform one or more remedial actions such as summarizing the differences and/or suggesting changes to the document to bring it into compliance.
According to various embodiments, techniques and mechanisms described herein provide for novel text generation in domain-specific contexts. A text generation interface system may take as input one or more arbitrary documents, process them via optical text recognition, segment them into portions, and process the segmented text via various tasks based on need. Different workflows are provided for different tasks, and this application describes a number of examples of such workflows. In many workflows, an input document is divided into chunks via a chunking technique. Then, chunks are inserted into prompt templates for processing by a large language model such as the GPT-3 or GPT-4 available from OpenAI. The large language model's response is then parsed and potentially used to trigger additional analysis, such as one or more database searches, one or more additional prompts sent back to the large language model, and/or a response returned to a client machine.
According to various embodiments, techniques and mechanisms described herein provide for retrieval augmented generation. A search is conducted base on a search query. Then, the search results are provided to an artificial intelligence system. The artificial intelligence system then further processes the search results to produce an answer based on those search results. In this context, a large language model may be used to determine the search query, apply one or more filters and/or tags, and/or synthesize potentially many different types of search.
According to various embodiments, techniques and mechanisms described herein provide for a sophisticated document processing pipeline. The pipeline receives one or more input documents, identifies text that should be kept together, identifies extraneous text such as headers, footers, and line numbers, and segments the text accordingly. In this way, the quality of the text provided to the rest of the system is improved.
According to various embodiments, techniques and mechanisms described herein provide for new approaches to text segmentation. Large language models often receive as input a portion of input text and generate in response a portion of output text. In many systems, the large language model imposes a limit on the input text size. Accordingly, in the event that the large language model is asked to summarize a length document, the document may need to be segmented into portions in order to achieve the desired summarization.
Conventional text segmentation techniques frequently create divisions in text that negatively affect the performance of the model, particularly in domains-specific contexts such as law. For example, consider a caption page of a legal brief, which includes text in a column on the left that encompasses the parties, text in a column on the right that includes the case number, a title that follows lower on the page, and line numbering on the left. In such a configuration, the text in the different columns should not be mixed and should be treated separately from the line numbers, while both columns should precede the document title, when converting the document to an input query for a large language model. However, conventional techniques would result in these semantically different elements of text being jumbled together, resulting in an uninformative query provided to the large language model and hence a low quality response. In contrast to these conventional techniques, techniques and mechanisms described herein provide for a pipeline that cleans such raw text so that it can be provided to a large language model.
According to various embodiments, techniques and mechanisms described herein provide for the division of text into chunks, and the incorporation of those chunks into prompts that can be provided to a large language model. For instance, a large language model may impose a limit of, for instance, 8,193 tokens on a task, including text input, text output, and task instructions. In order to process longer documents, the system may split them. However, splitting a document can easily destroy meaning depending on where and how the document is split. Techniques and mechanisms described herein provide for evenly splitting a document or documents into chunks, and incorporating those chunks into prompts, in ways that retain the semantic content associated with the raw input document or documents.
In some embodiments, techniques and mechanisms described herein may be applied to generate novel text in domain-specific contexts, such as legal analysis. Large language models, while powerful, have a number of drawbacks when used for technical, domain-specific tasks. When using conventional techniques, large language models often invent âfactsâ that are actually not true. For instance, if asked to summarize the law related to non-obviousness in the patent context, a large language model might easily invent a court case, complete with caption and ruling, that in fact did not occur. In contrast to conventional techniques, techniques and mechanisms described herein provide for the generation of novel text in domain-specific contexts while avoiding such drawbacks.
According to various embodiments, techniques and mechanisms described herein may be used to automate complex, domain-specific tasks that were previously the sole domain of well-trained humans. Moreover, such tasks may be executed in ways that are significantly faster, less expensive, and more auditable than the equivalent tasks performed by humans. For example, a large language model may be employed to produce accurate summaries of legal texts, to perform legal research tasks, to generate legal documents, to generate questions for legal depositions, and the like.
In some embodiments, techniques and mechanisms described herein may be used to divide text into portions while respecting semantic boundaries and simultaneously reducing calls to the large language model. The cost of using many large language models depends on the amount of input and/or output text. Accordingly, techniques and mechanisms described herein provide for reduced overhead associated with prompt instructions while at the same time providing for improved model context to yield an improved response.
In some embodiments, techniques and mechanisms described herein may be used to process an arbitrary number of unique documents (e.g., legal documents) that cannot be accurately parsed and processed via existing optical character recognition and text segmentation solutions.
In some embodiments, techniques and mechanisms described herein may be used to link a large language model with a legal research database, allowing the large language model to automatically determine appropriate searches to perform and then ground its responses to a source of truth (e.g., in actual law) so that it does not âhallucinateâ a response that is inaccurate.
In some embodiments, techniques and mechanisms described herein provide for specific improvements in the legal domain. For example, tasks that were previously too laborious for attorneys with smaller staffs may now be more easily accomplished. As another example, attorneys may automatically analyze large volumes of documents rather than needing to perform such tasks manually. As another example, text chunking may reduce token overhead and hence cost expended on large language model prompts. As yet another example, text chunking may reduce calls to a large language model, increasing response speed. As still another example, text chunking may increase and preserve context provided to a large language model by dividing text into chunks in semantically meaningful ways.
According to various embodiments, techniques and mechanisms described herein may provide for automated solutions for generated text in accordance with a number of specialized applications. Such applications may include, but are not limited to: simplifying language, generating correspondence, generating a timeline, reviewing documents, editing a contract clause, drafting a contract, performing legal research, preparing for a depositions, drafting legal interrogatories, drafting requests for admission, drafting requests for production, briefing a litigation case, responding to requests for admission, responding to interrogatories, responding to requests for production, analyzing cited authorities, and answering a complaint.
FIG. 1 illustrates a document policy evaluation overview method 100 , performed in accordance with one or more embodiments. In some implementations, the method 100 may be performed at a text generation interface system such as the system 200 shown in FIG. 2 . For instance, the method 100 may be performed at the text generation interface system 210 .
At 102 , relevance scores for portions of text included in a document are determined based on a comparison with a natural language criterion. A subset of the text portions are selected at 104 based on the relevance scores. In some embodiments, one or more of the relevance scores may be determined via a machine learning model, such as a bi-encoder and/or a cross-encoder. Alternatively, or additionally, one or more of the relevance scores may be determined based on communication with a remote text generation modeling system. Additional details regarding the determination of relevance scores and the selection of a subset of the text portions are discussed throughout the application and particularly with respect to FIG. 13 , FIG. 14 , FIG. 15 , and FIG. 16 .
At 106 , a criteria evaluation prompt is transmitted to a remote text generation modeling system. According to various embodiments, the criteria evaluation prompt includes an instruction to evaluate some or all of the subset of the text portions against the natural language criterion. A policy evaluation response message is sent to a client machine at 108 . The policy evaluation response message may identify an instance of noncompliance with the designated policy. Additional details regarding the evaluation of a text portion against a natural language criterion are discussed with respect to the method 1700 shown in FIG. 17 .
FIG. 2 illustrates a text generation system 200 , configured in accordance with one or more embodiments. The text generation system 200 includes client machines 202 through 204 in communication with a text generation interface system 210 , which in turn is in communication with a text generation modeling system 270 . The text generation modeling system 270 includes a communication interface 272 , a text generation API 274 , and a text generation model 276 . The text generation interface system 210 includes a communication interface 212 , a database system 214 , a testing module 220 , and an orchestrator 230 . The testing module 220 includes a query cache 222 , a test repository 224 , and a prompt testing utility 226 . The orchestrator 230 includes skills 232 through 234 , and prompt templates 236 through 238 . The orchestrator also includes a chunker 240 and a scheduler 242 . The orchestrator also includes API interfaces 250 , which include a model interface 252 , an external search interface 254 , an internal search interface 256 , and a chat interface 258 .
According to various embodiments, a client machine may be any suitable computing device or system. For instance, a client machine may be a laptop computer, desktop computer, mobile computing device, or the like. Alternatively, or additionally, a client machine may be an interface through which multiple remote devices communicate with the text generation interface system 210 .
According to various embodiments, a client machine may interact with the text generation interface system in any of various ways. For example, a client machine may access the text generation interface system via a text editor plugin, a dedicated application, a web browser, other types of interactions techniques, or combinations thereof.
According to various embodiments, the text generation modeling system 270 may be configured to receive, process, and respond to requests via the communication interface 272 , which may be configured to facilitate communications via a network such as the internet.
In some embodiments, some or all of the communication with the text generation modeling system 270 may be conducted in accordance with the text generation API 274 , which may provide remote access to the text generation model 276 . The text generation API 274 may provide functionality such as defining standardized message formatting, enforcing maximum input and/or output size for the text generation model, and/or tracking usage of the text generation model.
According to various embodiments, the text generation model 276 may be a large language model. The text generation model 276 may be trained to predict successive words in a sentence. It may be capable of performing functions such as generating correspondence, summarizing text, and/or evaluating search results. The text generation model 276 may be pre-trained using many gigabytes of input text and may include billions or trillions of parameters.
In some embodiments, large language models impose a tradeoff. A large language model increases in power with the number of parameters and the amount of training data used to train the model. However, as the model parameters and input data increase in magnitude, the model's training cost, storage requirements, and required computing resources increase as well. Accordingly, the large language model may be implemented as a general-purpose model configured to generate arbitrary text. The text generation interface system 210 may serve as an interface between the client machines and the text generation modeling system 270 to support the use of the text generation modeling system 270 for performing complex, domain-specific tasks in fields such as law. That is, the text generation interface system 210 may be configured to perform one or more methods described herein.
According to various embodiments, the orchestrator 230 facilitates the implementation of one or more skills, such as the skills 232 through 234 . A skill may act as a collection of interfaces, prompts, actions, data, and/or metadata that collectively provide a type of functionality to the client machine. For instance, a skill may involve receiving information from a client machine, transmitting one or more requests to the text generation modeling system 270 , processing one or more response received form the text generation modeling system 270 , performing one or more searches, and the like. Skills are also referred to herein as text generation flows. Additional details regarding specific skills are provided with reference to FIGS. 8 - 10 .
In some embodiments, a skill may be associated with one or more prompts. For instance, the skill 234 is associated with the prompt templates
236 and 238 . A prompt template may include information such as instructions that may be provided to the text generation modeling system 270 . A prompt template may also include one or more fillable portions that may be filled based on information determined by the orchestrator 230 . For instance, a prompt template may be filled based on information received from a client machine, information returned by a search query, or another information source. Additional details regarding prompt templates are provided with reference to FIGS. 8 - 10 .
In some implementations, the chunker 240 is configured to divide text into smaller portions. Dividing text into smaller portions may be needed at least in part to comply with one or more size limitations associated with the text. For instance, the text generation API 274 may impose a maximum size limit on prompts provided to the text generation model 276 . The chunker may be used to subdivide text included in a request from a client, retrieved from a document, returned in a search result, or received from any other source.
According to various embodiments, the API interfaces 250 include one or more APIs for interacting with internal and/or external services. The model interface 252 may expose one or more functions for communicating with the text generation modeling system 270 . For example, the model interface 252 may provide access to functions such as transmitting requests to the text generation modeling system 270 , receiving responses from the text generation modeling system 270 , and the like.
In some embodiments, the external search interface 254 may be used to search one or more external data sources such as information repositories that are generalizable to multiple parties. For instance, the external search interface 254 may expose an interface for searching legal case law and secondary sources.
In some implementations, the internal search interface 256 may facilitate the searching of private documents. For instance, a client may upload or provide access to a set of private documents, which may then be indexed by the text generation interface system 210 .
According to various embodiments, the chat interface 258 may facilitate text-based communication with the client machines. For instance, the chat interface 258 may support operations such as parsing chat messages, formulating responses to chat messages, identifying skills based on chat messages, and the like. In some configurations, the chat interface 258 may orchestrate text-based chat communication between a user at a client machine and the text generation model 276 , for instance via web sockets.
In some embodiments, the query cache 222 may store queries such as testing queries sent to the text generation modeling system 270 . Then, the query cache 222 may be instructed to return a predetermined result to a query that has already been sent to the text generation modeling system 270 rather than sending the same query again.
In some embodiments, the prompt testing utility 226 is configured to perform operations such as testing prompts created based on prompt templates against tests stored in the test repository 224 .
In some embodiments, the communication interface 212 is configured to facilitate communications with the client machines and/or the text generation modeling system 270 via a network such as the internet. The scheduler 242 may be responsible for scheduling one or more tasks performed by the text generation interface system 210 . For instance, the scheduler may schedule requests for transmission to the text generation modeling system 270 .
In some embodiments, the database system 214 is configured to store information determined based on natural language. For example, the database system 214 may be configured to store one or more database tables that include fields corresponding with information extracted from natural language documents. As another example, the database system 214 may be configured to store metadata information about documents based on information extracted from those documents. As yet another example, the database system 214 may be configured to store linkages between documents and document portions.
According to various embodiments, the database system 214 may be configured using any of a variety of suitable database technologies. For instance, the database system 214 may be configured as a relational database system, a non-relational database system, or any other type of database system capable of supporting the storage and querying of information described herein.
FIG. 3 illustrates a document parsing method 300 , performed in accordance with one or more embodiments. According to various embodiments, the method 300 may be performed on any suitable computing system. For instance, the method 300 may be performed on the text generation interface system 230 shown in FIG. 2 . The method 300 may be performed in order to convert a document into usable text while at the same time retaining metadata information about the text, such as the page, section, and/or document at which the text was located.
A request to parse a document is received at 302 . In some embodiments, the request to parse a document may be generated when a document is identified for analysis. For example, as discussed herein, a document may be uploaded or identified by a client machine as part of communication with the text generation interface system 230 . As another example, a document may be returned as part of a search result.
The document is converted to portable document format (PDF) or another suitable document format at 304 . In some embodiments, the document need only be converted to PDF if the document is not already in the PDF format. Alternatively, PDF conversion may be performed even on PDFs to ensure that PDFs are properly formatted. PDF conversion may be performed, for instance, by a suitable Python library or the like. For instance, PDF conversion may be performed with the Hyland library.
Multipage pages are split into individual pages at 306 . In some implementations, multipage pages may be split into individual pages via a machine learning model. The machine learning model may be trained to group together portions of text on a multipage page. For instance, a caption page in a legal decision may include text in a column on the left that encompasses the parties, text in a column on the right that includes the case number, a title that follows lower on the page, and line numbering on the left. In such a configuration, the machine learning model may be trained to treat separately the text in the different columns, and to separate the text from the line numbers. The document title may be identified as a first page, with the left column identified as the second page and the right column identified as the third page.
Optical character recognition is performed on individual pages or on the document as a whole at 308 . In some implementations, optical character recognition may be performed locally via a library. Alternatively, optical character recognition may be performed by an external service. For instance, documents or pages may be sent to a service such as Google Vision. Performing optical character recognition on individual pages may provide for increased throughout via parallelization.
Individual pages are combined in order at 310 . In some implementations, combining pages in order may be needed if optical character recognition were applied to individual pages rather than to the document as a whole.
Inappropriate text splits are identified and corrected at 312 . In some embodiments, inappropriate text splits include instances where a paragraph, sentence, word, or other textual unit was split across different pages. Such instances may be identified by, for example, determining whether the first textual unit in a page represents a new paragraph, sentence, word, or other unit, or if instead it represents the continuation of a textual unit from the previous page. When such a split is identified, the continuation of the textual unit may be excised from the page on which it is located and moved to the end of the previous page. Such an operation may be performed by, for instance, the Poppler library available in Python.
Segmented JSON text is determined at 314 . In some embodiments, the segmented JSON text may include the text returned by the optical character recognition performed at operation 308 . In addition, the segmented JSON text may include additional information, such as one or more identifiers for the page, section, and/or document on which the text resides. The output of the segmented JSON may be further processed, for instance via the text sharding method 500 shown in FIG. 5 and/or the text chunking method 600 shown in FIG. 6 .
FIG. 4 illustrates a text generation method 400 , performed in accordance with one or more embodiments. According to various embodiments, the method 400 may be performed on any suitable computing system. For instance, the method 400 may be performed on the text generation interface system 230 shown in FIG. 2 . The method 400 may be performed in order to identify and implement a text generation flow based on input text.
A request from a client machine to generate a novel text portion is received at 402 . In some embodiments, the request may include a query portion. The query portion may include natural language text, one or more instructions in a query language, user input in some other format, or some combination thereof. For instance, the query portion may include an instruction to âwrite an emailâ, âsummarize documentsâ, or âresearch case lawâ.
In some embodiments, the request may include an input text portion. For example, the request may link to, upload, or otherwise identify documents. As another example, the request may characterize the task to be completed. For instance, the request may discuss the content of the desired email or other correspondence. The particular types of input text included in the request may depend in significant part on the type of request. Accordingly, many variations are possible.
A text generation flow is determined at 404 . In some embodiments, the text generation flow may be explicitly indicated as part of the request received from the client machine. For instance, the client machine may select a particular text generation flow from a list. Alternatively, the text generation flow may be determined at least in part by analyzing the request received from the client machine. For example, the request may be analyzed to search for keywords or other indications that a particular text generation flow is desired. As another example, all or a portion of the request may be provided to a machine learning model to predict the requested text generation flow. In some configurations, a predicted text generation flow may be provided to the client machine for confirmation before proceeding.
Input text is determined at 406 . In some embodiments, the input text may be determined by applying one or more text processing, search, or other operations based on the request received from the client machine. For example, the input text may be determined at least in part by retrieving one or more documents identified in or included with the request received from the client machine. As another example, the input text may be determined at least in part by applying one or more natural language processing techniques such as cleaning or tokenizing raw text.
In some embodiments, determining input text may involve executing a search query. For example, a search of a database, set of documents, or other data source may be executed base at least in part on one or more search parameters determined based on a request received from a client machine. For instance, the request may identify one or more search terms and a set of documents to be searched using the one or more search terms.
In some embodiments, determining input text may involve processing responses received from a text generation modeling system. For instance, all or a portion of the results from an initial request to summarizing a set of text portions may then be used to create a new set of more compressed input text, which may then be provided to the text generation modeling system for further summarization or other processing.
One or more prompt templates are determined at 408 based on the input text and the text generation flow. As discussed with respect to FIG. 2 , different text generation flows may be associated with different prompt templates. Prompt templates may be selected from the prompt library based on the particular text generation flow. Additional details regarding the content of particular prompt templates is discussed with respect to the text generation flows illustrated in FIGS. 8 - 10 .
At 410 , one or more prompts based on the prompt templates are determined. In some embodiments, a prompt may be determined by supplementing and/or modifying a prompt template based on the input text. For instance, a portion of input text may be added to a prompt template at an appropriate location. As one example, a prompt template may include a set of instructions for causing a large language model to generate a correspondence document. The prompt template may be modified to determine a prompt by adding a portion of input text that characterizes the nature of the correspondence document to be generated. The added input text may identify information such as the correspondence recipient, source, topic, and discussion points.
The one or more prompts are transmitted to a text generation modeling system at 412 . In some embodiments, the text generation modeling system may be implemented at a remote computing system. The text generation modeling system may be configured to implement a text generation model. The text generation modeling system may expose an application procedure interface via a communication interface accessible via a network such as the internet.
One or more text response messages are received from the remote computing system at 414 . According to various embodiments, the one or more text response messages include one or more novel text portions generated by a text generation model implemented at the remote computing system. The novel text portions may be generated based at least in part on the prompt received at the text generation modeling system, including the instructions and the input text.
The one or more responses are parsed at 416 to produce a parsed response. In some embodiments, parsing the one or more responses may involve performing various types of processing operations. For example, in some systems a large language model may be configured to complete a prompt. Hence, a response message received from the large language model may include the instructions and/or the input text. Accordingly, the response message may be parsed to remove the instructions and/or the input text.
In some implementations, parsing the one or more responses may involve combining text from different responses. For instance, a document may be divided into a number of portions, each of which is summarized by the large language model. The resulting summaries may then be combined to produce an overall summary of the document.
A determination is made at 418 as to whether to provide a response to the client machine. In some embodiments, the determination made at 418 may depend on the process flow. For example, in some process flows, additional user input may be solicited by providing a response message determined based at least in part on one or more responses received from the text generation modeling system. As another example, in some process flows, a parsed response message may be used to produce an output message provided to the client machine.
If a response is to be provided to the client machine, then a client response message including a novel text passage is transmitted to the client machine at 420 . In some embodiments, the client response message may be determined based in part on the text generation flow determined at 404 and in part based on the one or more text response messages received at 414 and parsed at 416 . Additional details regarding the generation of a novel text passage are discussed with respect to the text generation flows illustrated in FIGS. 8 - 10 .
A determination is made at 422 as to whether to generate an additional prompt. According to various embodiments, the determination as to whether to generation an additio
CLAIMS
Claims ( 20 )
The invention claimed is:
1. A method comprising:
selecting via a processor a subset of a plurality of text portions included in a document based on a comparison of a respective text portion of the plurality of text portions with one or more criteria specified in natural language;
determining one or more criteria evaluation prompts by filling a first fillable portion of a criteria evaluation prompt template with some or all of the one or more criteria specified in natural language and filling a second fillable portion of the criteria evaluation prompt template with the subset of the plurality of text portions, the one or more criteria evaluation prompts and the criteria evaluation prompt template each including a natural language instruction to evaluate one or more of the subset of the plurality of text portions against some or all of the one or more criteria;
determining novel text generated by a generative language model completing the one or more criteria evaluation prompts by executing the natural language instruction;
identifying an instance of noncompliance with a designated policy by evaluating the novel text; and
transmitting to a client machine a policy evaluation response message, the policy evaluation response message identifying the instance of noncompliance with the designated policy.
2. The method recited in claim 1 , wherein the policy evaluation response message identifies a difference between the one or more criteria and a text element included in the plurality of text portions.
3. The method recited in claim 1 , wherein the policy evaluation response message identifies a proposed revision to bring the document into compliance with the designated policy.
4. The method recited in claim 3 , wherein the policy evaluation response message identifies a difference between the proposed revision and the document.
5. The method recited in claim 1 , wherein selecting via the subset of the plurality of text portions comprises determining a plurality of relevance scores, wherein a relevance score of the plurality of relevance scores indicates relevance of a respective text portion of the plurality of text portions to the one or more criteria.
6. The method recited in claim 5 , wherein the plurality of relevance scores are determined based on application of one or more machine learning models to the plurality of text portions and the one or more criteria.
7. The method recited in claim 6 , wherein the one or more machine learning models include a cross-encoder based on a transformer network.
8. The method recited in claim 6 , wherein the one or more machine learning models include a bi-encoder based on an embedding of text into a vector space.
9. The method recited in claim 5 , wherein determining the plurality of relevance scores comprises:
transmitting one or more input relevance prompts to a remote text generation system, an input relevance prompt of the one or more input relevance prompts including one or more of the plurality of text portions and a criterion portion determined based on the one or more criteria; and
receiving from the generative language model one or more completed relevance prompts each including a respective one or more of the plurality of relevance scores.
10. One or more non-transitory computer readable media having instructions stored thereon for performing a method, the method comprising:
selecting via a processor a subset of a plurality of text portions included in a document based on a comparison of a respective text portion of the plurality of text portions with one or more criteria specified in natural language;
determining one or more criteria evaluation prompts by filling a first fillable portion of a criteria evaluation prompt template with some or all of the one or more criteria specified in natural language and filling a second fillable portion of the criteria evaluation prompt template with the subset of the plurality of text portions, the one or more criteria evaluation prompts and the criteria evaluation prompt template each including a natural language instruction to evaluate one or more of the subset of the plurality of text portions against some or all of the one or more criteria;
determining novel text generated by a generative language model completing the one or more criteria evaluation prompts by executing the natural language instruction;
identifying an instance of noncompliance with a designated policy by evaluating the novel text; and
transmitting to a client machine a policy evaluation response message, the policy evaluation response message identifying the instance of noncompliance with the designated policy.
11. The one or more non-transitory computer readable media recited in claim 10 , wherein the policy evaluation response message identifies a difference between the one or more criteria and a text element included in the plurality of text portions.
12. The one or more non-transitory computer readable media recited in claim 10 , wherein the policy evaluation response message identifies a proposed revision to bring the document into compliance with the designated policy.
13. The one or more non-transitory computer readable media recited in claim 12 , wherein the policy evaluation response message identifies a difference between the proposed revision and the document.
14. The one or more non-transitory computer readable media recited in claim 10 , wherein selecting via the subset of the plurality of text portions comprises determining a plurality of relevance scores, wherein a relevance score of the plurality of relevance scores indicates relevance of a respective text portion of the plurality of text portions to the one or more criteria.
15. The one or more non-transitory computer readable media recited in claim 14 , wherein the plurality of relevance scores are determined based on application of one or more machine learning models to the plurality of text portions and the one or more criteria.
16. The one or more non-transitory computer readable media recited in claim 15 , wherein the one or more machine learning models include a cross-encoder based on a transformer network or a bi-encoder based on an embedding of text into a vector space.
17. A system including a communication interface and one or more processors, the system configured to perform a method comprising:
selecting via a processor a subset of a plurality of text portions included in a document based on a comparison of a respective text portion of the plurality of text portions with one or more criteria specified in natural language;
determining one or more criteria evaluation prompts by filling a first fillable portion of a criteria evaluation prompt template with some or all of the one or more criteria specified in natural language and filling a second fillable portion of the criteria evaluation prompt template with the subset of the plurality of text portions, the one or more criteria evaluation prompts and the criteria evaluation prompt template each including a natural language instruction to evaluate one or more of the subset of the plurality of text portions against some or all of the one or more criteria;
determining novel text generated by a generative language model completing the one or more criteria evaluation prompts by executing the natural language instruction;
identifying an instance of noncompliance with a designated policy by evaluating the novel text; and
transmitting to a client machine a policy evaluation response message, the policy evaluation response message identifying the instance of noncompliance with the designated policy.
18. The system recited in claim 17 , wherein the policy evaluation response message identifies a difference between the one or more criteria and a text element included in the plurality of text portions.
19. The system recited in claim 17 , wherein the policy evaluation response message identifies a proposed revision to bring the document into compliance with the designated policy.
20. The system recited in claim 17 , wherein selecting via the subset of the plurality of text portions comprises determining a plurality of relevance scores, wherein a relevance score of the plurality of relevance scores indicates relevance of a respective text portion of the plurality of text portions to the one or more criteria, wherein the plurality of relevance scores are determined based on application of one or more machine learning models to the plurality of text portions and the one or more criteria.
US18/640,591
2023-02-28
2024-04-19
Large language model artificial intelligence text evaluation system
Active
US12299406B2
( en )
Priority Applications (1)
Application Number
Priority Date
Filing Date
Title
US18/640,591
US12299406B2
( en )
2023-02-28
2024-04-19
Large language model artificial intelligence text evaluation system
Applications Claiming Priority (3)
Application Number
Priority Date
Filing Date
Title
US202363487330P
2023-02-28
2023-02-28
US18/329,039
US11995411B1
( en )
2023-02-28
2023-06-05
Large language model artificial intelligence text evaluation system
US18/640,591
US12299406B2
( en )
2023-02-28
2024-04-19
Large language model artificial intelligence text evaluation system
Related Parent Applications (1)
Application Number
Title
Priority Date
Filing Date
US18/329,039
Continuation
US11995411B1
( en )
2023-02-28
2023-06-05
Large language model artificial intelligence text evaluation system
Publications (2)
Publication Number
Publication Date
US20240289561A1
US20240289561A1 ( en )
2024-08-29
US12299406B2
true
US12299406B2 ( en )
2025-05-13
Family
ID=91197002
Family Applications (2)
Application Number
Title
Priority Date
Filing Date
US18/329,039
Active
US11995411B1
( en )
2023-02-28
2023-06-05
Large language model artificial intelligence text evaluation system
US18/640,591
Active
US12299406B2
( en )
2023-02-28
2024-04-19
Large language model artificial intelligence text evaluation system
Family Applications Before (1)
Application Number
Title
Priority Date
Filing Date
US18/329,039
Active
US11995411B1
( en )
2023-02-28
2023-06-05
Large language model artificial intelligence text evaluation system
Country Status (4)
Country
Link
US
( 2 )
US11995411B1
( en )
EP
( 1 )
EP4630962A1
( en )
AU
( 1 )
AU2023434189A1
( en )
WO
( 1 )
WO2024182041A1
( en )
Families Citing this family (37)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US12159119B2
( en )
2023-02-15
2024-12-03
Casetext, Inc.
Text generation interface system
US11861320B1
( en )
2023-02-27
2024-01-02
Casetext, Inc.
Text reduction and analysis interface to a text generation modeling system
US20240289545A1
( en )
*
2023-02-28
2024-08-29
Microsoft Technology Licensing, Llc
Automated plan generation and execution system for large language models
US12481832B2
( en )
*
2023-04-05
2025-11-25
Artica Inc.
Providing item discovery guidance based on automatically-discerned subjective considerations
US12475151B1
( en )
2023-05-04
2025-11-18
Vijay Madisetti
Fault tolerant multi-agent generative AI applications
US12405979B2
( en )
*
2023-05-04
2025-09-02
Vijay Madisetti
Method and system for optimizing use of retrieval augmented generation pipelines in generative artificial intelligence applications
US12493638B2
( en )
2023-05-04
2025-12-09
Vijay Madisetti
Method and system for multi-level artificial intelligence supercomputer design
US12405977B1
( en )
2023-09-20
2025-09-02
Vijay Madisetti
Method and system for optimizing use of retrieval augmented generation pipelines in generative artificial intelligence applications
US12572575B2
( en )
*
2023-05-31
2026-03-10
Highspot, Inc.
Using large language models to generate search query answers
US20240428005A1
( en )
*
2023-06-20
2024-12-26
Microsoft Technology Licensing, Llc
Generating grounded documents using large language models
US20250005060A1
( en )
*
2023-06-28
2025-01-02
Jpmorgan Chase Bank, N.A.
Systems and methods for runtime input and output content moderation for large language models
US11861321B1
( en )
*
2023-06-29
2024-01-02
Casetext, Inc.
Systems and methods for structure discovery and structure-based analysis in natural language processing models
US12541347B2
( en )
2023-07-31
2026-02-03
Discover Beagle, Inc.
Integration of machine learning models into software systems using software library
US12572351B2
( en )
*
2023-07-31
2026-03-10
Discover Beagle, Inc.
Integration of machine learning models into software systems using software library
US20250053899A1
( en )
*
2023-08-09
2025-02-13
Chase Hughes
Computer-Implemented Methods and Computer Systems for Artificial Intelligence (AI) Based Automated Provision of Management Consulting
US12242550B1
( en )
*
2023-08-28
2025-03-04
Snowflake Inc.
Browser plug-in for marketplace recommendations
US20250094233A1
( en )
*
2023-09-20
2025-03-20
Microsoft Technology Licensing, Llc
Request segmentation for reduced memory consumption by trained sequential models
US12332925B2
( en )
*
2023-10-05
2025-06-17
Nasdaq, Inc.
Systems and methods of chained conversational prompt engineering for information retrieval
US20250124067A1
( en )
*
2023-10-11
2025-04-17
Google Llc
Method for Text Ranking with Pairwise Ranking Prompting
US12505145B2
( en )
*
2023-11-15
2025-12-23
Hanzo Ltd
Method of classifying a very large corpus of documents
US12561314B2
( en )
*
2023-11-17
2026-02-24
Goldman Sachs & Co. LLC
Retrieval-augmented generation (RAG) system optimization
US12236193B1
( en )
*
2024-03-15
2025-02-25
CAST AI Group, Inc.
Automated selection of large language models in cloud computing environments
CN118657192B
( en )
*
2024-05-31
2025-12-12
浪潮çµåä¿¡æ¯äº§ä¸è¡ä»½æéå ¬å¸
A text filtering method, apparatus, electronic device, and storage medium
CN118261254B
( en )
*
2024-05-31
2024-08-20
å京深å¿ç§ææéå ¬å¸
A method and device for compressing long text
US20260004078A1
( en )
*
2024-06-26
2026-01-01
Microsoft Technology Licensing, Llc
Iterative prompt generation loop
US12436957B1
( en )
2024-07-26
2025-10-07
Bank Of America Corporation
Context-specific query response platform using large language models
US20260044536A1
( en )
*
2024-08-09
2026-02-12
Joshua James Montgomery
System and method for artificial intelligence assisted processing of legal research queries
CN119003759B
( en )
*
2024-08-22
2025-05-30
èªå¤©ç©èç½ææ¯æéå ¬å¸
Conference summary generation method based on large language model
CN118898533A
( en )
*
2024-08-23
2024-11-05
ç§å¤§è®¯é£è¡ä»½æéå ¬å¸
Scientific inquiry practice observation evaluation methods, systems, electronic equipment and storage media
CN118714416B
( en )
*
2024-08-29
2025-05-30
é¿éå¥åº·ç§æ(æå·)æéå ¬å¸
Method, device, storage medium and program product for processing speech and text
JP7644302B1
( en )
2024-09-27
2025-03-11
大éªç¦æ¯æ ªå¼ä¼ç¤¾
Evaluation System and Evaluation Program
JP7672120B1
( en )
*
2024-09-30
2025-05-07
æå¹´ å·ä¸
Idea support program and method
US20260127185A1
( en )
*
2024-11-07
2026-05-07
International Business Machines Corporation
Human guided text attention mechanism
US12277489B1
( en )
*
2024-11-13
2025-04-15
Airia LLC
Artificial intelligence agent output through caching predicted inputs
US12405985B1
( en )
*
2024-12-12
2025-09-02
Dell Products L.P.
Retrieval-augmented generation processing using dynamically selected number of document chunks
JP7817481B1
( en )
*
2025-03-28
2026-02-18
å¼è·å£«ãããã³ã æ ªå¼ä¼ç¤¾
Program, method, and information processing device
US12517941B1
( en )
*
2025-04-04
2026-01-06
Poma Ai Gmbh
Retrieval-augmented generation for large language models
Citations (57)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US20020169595A1
( en )
2001-03-30
2002-11-14
Yevgeny Agichtein
Method for retrieving answers from an information retrieval system
US20050108219A1
( en )
1999-07-07
2005-05-19
Carlos De La Huerga
Tiered and content based database searching
US20070022109A1
( en )
2005-07-25
2007-01-25
Tomasz Imielinski
Systems and methods for answering user questions
US7257766B1
( en )
2000-06-29
2007-08-14
Egocentricity Ltd.
Site finding
US7293012B1
( en )
2003-12-19
2007-11-06
Microsoft Corporation
Friendly URLs
US20070266331A1
( en )
2006-05-12
2007-11-15
Sap Ag
Editable table modification
US20090083248A1
( en )
2007-09-21
2009-03-26
Microsoft Corporation
Multi-Ranker For Search
US20100145673A1
( en )
2008-12-09
2010-06-10
Xerox Corporation
Cross language tool for question answering
US20110125734A1
( en )
2009-11-23
2011-05-26
International Business Machines Corporation
Questions and answers generation
US20120030201A1
( en )
2010-07-30
2012-02-02
International Business Machines Corporation
Querying documents using search terms
US8380710B1
( en )
2009-07-06
2013-02-19
Google Inc.
Ordering of ranked documents
US20140172907A1
( en )
2012-12-17
2014-06-19
International Business Machines Corporation
Corpus search improvements using term normalization
US8812291B2
( en )
2007-03-26
2014-08-19
Google Inc.
Large language models in machine translation
US20140358889A1
( en )
2013-06-04
2014-12-04
Google Inc.
Natural language search results for intent queries
US20180075011A1
( en )
2016-09-13
2018-03-15
International Business Machines Corporation
Hybrid Approach to Handling Hypotheticals in Texts
US20180322110A1
( en )
2017-05-02
2018-11-08
eHealth Technologies
Methods for improving natural language processing with enhanced automated screening for automated generation of a clinical summarization report and devices thereof
CN109213870A
( en )
2017-06-30
2019-01-15
å森å²ç¯ç解å³æ¹æ¡æéå ¬å¸
Document process
US20190042551A1
( en )
2017-08-01
2019-02-07
Samsung Electronics Co., Ltd.
Apparatus and method for providing summarized information using an artificial intelligence model
US20190286753A1
( en )
2018-03-03
2019-09-19
Thomson Reuters Global Resources Unlimited Company
System and methods for generating an enhanced output of relevant content to facilitate content analysis
US20190311064A1
( en )
2018-04-07
2019-10-10
Microsoft Technology Licensing, Llc
Intelligent question answering using machine reading comprehension
US20200019642A1
( en )
2018-07-12
2020-01-16
International Business Machines Corporation
Question Answering Using Trained Generative Adversarial Network Based Modeling of Text
US10565639B1
( en )
2019-05-02
2020-02-18
Capital One Services, Llc
Techniques to facilitate online commerce by leveraging user activity
US20200159783A1
( en )
2018-11-16
2020-05-21
Yandex Europe Ag
Method of and system for updating search index database
US20200243076A1
( en )
2019-01-27
2020-07-30
Min Ku Kim
Multi-Dimensional Parsing Method and System for Natural Language Processing
US20200342862A1
( en )
2018-06-27
2020-10-29
Google Llc
Rendering responses to a spoken utterance of a user utilizing a local text-response map
US20200342036A1
( en )
2019-04-29
2020-10-29
Ip.Com I, Llc
Method, System, and Data Storage Device for Automating Solution Prompts Based Upon Semantic Representation
US20200364403A1
( en )
2019-05-17
2020-11-19
Samsung Electronics Co., Ltd.
Electronic apparatus and controlling method thereof
US20210124876A1
( en )
2019-10-28
2021-04-29
Salesforce.Com, Inc.
Evaluating the Factual Consistency of Abstractive Text Summarization
US20210326428A1
( en )
2020-04-16
2021-10-21
Capital One Services, Llc
Systems and methods for authenticating users
US20210374341A1
( en )
2020-06-01
2021-12-02
Salesforce.Com, Inc.
Generative-discriminative language modeling for controllable text generation
US20210406735A1
( en )
2020-06-25
2021-12-30
Pryon Incorporated
Systems and methods for question-and-answer searching using a cache
US20220051479A1
( en )
*
2020-08-14
2022-02-17
Accenture Global Solutions Limited
Automated apparel design using machine learning
US11281976B2
( en )
2018-07-12
2022-03-22
International Business Machines Corporation
Generative adversarial network based modeling of text for natural language processing
US11321329B1
( en )
2020-06-24
2022-05-03
Amazon Technologies, Inc.
Systems, apparatuses, and methods for document querying
US20220164397A1
( en )
*
2020-11-24
2022-05-26
Thomson Reuters Enterprise Centre Gmbh
Systems and methods for analyzing media feeds
US20220180051A1
( en )
2020-12-04
2022-06-09
Patent Draftr, LLC
Text generation for patent applications using natural language modeling and programmable templating language
US20220197958A1
( en )
2020-12-22
2022-06-23
Yandex Europe Ag
Methods and servers for ranking digital documents in response to a query
US20220253447A1
( en )
2021-02-08
2022-08-11
Robert Bosch Gmbh
Interaction layer neural network for search, retrieval, and ranking
US20220261429A1
( en )
2021-02-17
2022-08-18
Gsi Technology Inc.
System and method for improved similarity search for search engines
US20220284174A1
( en )
2021-03-03
2022-09-08
Oracle International Corporation
Correcting content generated by deep learning
US20220300718A1
( en )
2021-03-22
2022-09-22
National University Of Defense Technology
Method, system, electronic device and storage medium for clarification question generation
US20220318255A1
( en )
2021-04-05
2022-10-06
Baidu Usa Llc
Cross-lingual language models and pretraining of cross-lingual language models
US20220366127A1
( en )
2020-03-23
2022-11-17
Chetan Desh
Legal Document Generation
US20220374459A1
( en )
2021-05-17
2022-11-24
Salesforce.Com, Inc.
Systems and methods for hierarchical retrieval of semantic-based passages in deep learning
US20220382975A1
( en )
*
2021-05-28
2022-12-01
Adobe Inc.
Self-supervised document representation learning
US20230034011A1
( en )
2021-07-29
2023-02-02
Hewlett Packard Enterprise Development Lp
Natural language processing workflow
US20230080674A1
( en )
2021-09-14
2023-03-16
Duolingo, Inc.
Systems and Methods for Automated Generation of Passage-Based Items for Use in Testing or Evaluation
US20230108863A1
( en )
2021-10-04
2023-04-06
International Business Machines Corporation
Deep learning document generation from conversation transcripts
US20230121711A1
( en )
2021-10-14
2023-04-20
Adobe Inc.
Content augmentation with machine generated content to meet content gaps during interaction with target entities
US20230237277A1
( en )
2022-01-25
2023-07-27
Oracle International Corporation
Aspect prompting framework for language modeling
US20230245051A1
( en )
*
2022-01-31
2023-08-03
Walmart Apollo, Llc
Methods and apparatus for automatic item mapping using machine learning processes
US20230274084A1
( en )
2022-02-28
2023-08-31
Adobe Inc.
Facilitating generation of fillable document templates
US20230351105A1
( en )
2022-04-29
2023-11-02
Leverage Technologies, LLC
Systems and methods for enhanced document generation
US11860914B1
( en )
2023-02-27
2024-01-02
Casetext, Inc.
Natural language database generation and query system
US11861321B1
( en )
2023-06-29
2024-01-02
Casetext, Inc.
Systems and methods for structure discovery and structure-based analysis in natural language processing models
US11861320B1
( en )
2023-02-27
2024-01-02
Casetext, Inc.
Text reduction and analysis interface to a text generation modeling system
US11972223B1
( en )
2023-06-30
2024-04-30
Casetext, Inc.
Query evaluation in natural language processing systems
Family Cites Families (5)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
JPWO2014064777A1
( en )
*
2012-10-24
2016-09-05
æ ªå¼ä¼ç¤¾æ¥ç«è£½ä½æ
Document evaluation support system and document evaluation support method
US11151325B2
( en )
*
2019-03-22
2021-10-19
Servicenow, Inc.
Determining semantic similarity of texts based on sub-sections thereof
CN112307712B
( en )
*
2019-07-31
2024-04-16
æ ªå¼ä¼ç¤¾çå
Text evaluation device and method, storage medium, and computer device
WO2021176648A1
( en )
*
2020-03-05
2021-09-10
å¯å£«éæ ªå¼ä¼ç¤¾
Document evaluation program, document evaluation method, and document evaluation device
US12524618B2
( en )
2021-09-20
2026-01-13
Salesforce, Inc.
Database systems and methods of representing conversations
2023
2023-06-05
US
US18/329,039
patent/US11995411B1/en
active
Active
2023-12-19
EP
EP23925582.1A
patent/EP4630962A1/en
active
Pending
2023-12-19
AU
AU2023434189A
patent/AU2023434189A1/en
active
Pending
2023-12-19
WO
PCT/US2023/084919
patent/WO2024182041A1/en
not_active
Ceased
2024
2024-04-19
US
US18/640,591
patent/US12299406B2/en
active
Active
Patent Citations (59)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US20050108219A1
( en )
1999-07-07
2005-05-19
Carlos De La Huerga
Tiered and content based database searching
US7257766B1
( en )
2000-06-29
2007-08-14
Egocentricity Ltd.
Site finding
US20020169595A1
( en )
2001-03-30
2002-11-14
Yevgeny Agichtein
Method for retrieving answers from an information retrieval system
US7293012B1
( en )
2003-12-19
2007-11-06
Microsoft Corporation
Friendly URLs
US20070022109A1
( en )
2005-07-25
2007-01-25
Tomasz Imielinski
Systems and methods for answering user questions
US20070266331A1
( en )
2006-05-12
2007-11-15
Sap Ag
Editable table modification
US8812291B2
( en )
2007-03-26
2014-08-19
Google Inc.
Large language models in machine translation
US20090083248A1
( en )
2007-09-21
2009-03-26
Microsoft Corporation
Multi-Ranker For Search
US20100145673A1
( en )
2008-12-09
2010-06-10
Xerox Corporation
Cross language tool for question answering
US8380710B1
( en )
2009-07-06
2013-02-19
Google Inc.
Ordering of ranked documents
US20110125734A1
( en )
2009-11-23
2011-05-26
International Business Machines Corporation
Questions and answers generation
US20120030201A1
( en )
2010-07-30
2012-02-02
International Business Machines Corporation</s