Conceptio › Archive › Google Patents

Google Patentsopen access

… the automated analysis or use of data, including use of a large language model — Unlikely Artificial Intelligence Limited (US12353827B2)

Unlikely Artificial Intelligence Limited · Google Patents

Google Patents · Patents · License: Open Access

pedoe unlikelyartificialintelligencelimited williamtunstall

patent, google patents, intellectual property, US12353827B2, Unlikely Artificial Intelligence Limited, William Tunstall-Pedoe, en, 2025

ABSTRACT

Abstract

There is provided a computer-implemented method for ensuring that a large language model (LLM) generates original text, including (i) providing or accessing a database of previous text that the LLM should not generate, wherein the database includes text used to train the LLM; (ii) checking potential continuations generated by the LLM against the database; (iii) when a potential continuation generated by the LLM matches text in the database, adjusting the potential continuation generated by the LLM to no longer match that text in the database, to produce an adjusted potential continuation, and (iv) storing the adjusted potential continuation.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This is a continuation of U.S. application Ser. No. 18/648,788, filed on Apr. 29, 2024, which is a continuation of U.S. application Ser. No. 18/301,615, filed on Apr. 17, 2023, which is a continuation of International Application No. PCT/GB2023/050405, filed on Feb. 22, 2023, which claims priority to GB Application No. GB2202347.7, filed on Feb. 22, 2022; GB Application No. GB2219268.6, filed on Dec. 20, 2022; GB Application No. GB2300624.0, filed on Jan. 16, 2023; and GB Application No. GB2302085.2, filed on Feb. 14, 2023, and is a continuation-in-part of U.S. application Ser. No. 18/001,368, filed on Dec. 9, 2022, which is the US national stage of International Application No. PCT/GB2021/052196, filed on Aug. 24, 2021, the entire contents of each of which being fully incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The field of the invention relates to computer implemented methods for the automated analysis or use of data, including use of a large language model (LLM), and to related computer implemented methods and systems.

2. Technical Background

Natural language (NL) is language evolved for humans such as the English language. Although significant advances have been made in computers' ability to process natural language, computers are still not able to deeply understand the meaning of natural language and use that meaning internally.

For this reason most computer applications typically use structured data to store information that they need for processingâe.g. a relational database: designing the schema, populating the database and writing code to process the fields in the database.

Use of structured data can work well if the application has limited requirements for the type of data required. However, some applications naturally require an extremely broad, heterogeneous collection of data to work well. This means that the schema required would have to be enormous, making building and coding for such an application impractical. We refer to such applications herein as HUB applications (Heterogeneous and Unreasonably Broad).

Examples of HUB applications include an application for managing a person's general health data where there are thousands of tests, thousands of medical conditions and thousands of symptoms. Another related application could be a nutrition tracking application where there are many thousands of substances and foods that can be ingested, each with different metabolic effects on the body.

Another example is an application to match the rÃ©sumÃ© of potential candidates with a job specification: in principle such an application would need structured data to represent every skill that might be of value to any role, every type of experience, every type of previous job.

Accounting is another application where vast heterogeneous data would be valuable: the perfect accounting application would represent every type of contract, every type of service.

In practice some of these applications, where they exist, work with a limited schema that doesn't cover the full range of their ideal properties. Health applications for example, typically work like this ignoring many types of data that they do not cover and instead end up being narrowâlimiting the application to only certain verticals within health.

Applications may also use natural language or augment a limited schema with natural languageâsuch as with current rÃ©sumÃ© matching applications which might represent a few key skills in a structured form but rely largely on keyword searching or statistical natural language processing (NLP) techniques on written rÃ©sumÃ©s otherwise.

In the case of accounting, transactions are represented with limited structured dataâdebits and credits on virtual ledgers with natural language names. The meaning of the natural language names and thus what these transactions represent is generally opaque to the application. Virtual ledgers often group different types of transaction together but fail to represent semantic differences which may be important.

There is no exact threshold for when an application becomes a HUB application but the difficulty of building an application with a hand created schema grows more than linearly with the number of tables as managing these tables as well as the code that maintains them becomes increasingly difficult to do.

These issues could be addressed if there existed a language or way of representing data that computers could fully process and understand but that also had an extremely broad scope.

In conventional Artificial Intelligence (AI), statistical Machine Learning (ML)âparticularly Deep Learning (DL)âhas been widely used. This has provided significant progress with many problems. Despite the progress, the results cannot be explained in a way that makes sense to human users as the solutions are the result of a calculation that may involve billions of weights. It can also be argued that the system lacks ârealâ understanding of the data or at least that the understanding is very different from the way a human user would understand it. The statistical nature of the technology also means that it is inherently unreliable with weights that are initiated randomly and then improved through a process of training where they are exposed to data. For complex problems, this improvement (learning) generally results in a model that works only a percentage of the time and is unable to reliably know when the result it produces is accurate.

Much processing of natural language is now done with DL. One example of DL applied to language is large language models (LLMs). LLMs are usually very large deep learning models potentially containing hundreds of billions of parameters and trained on tens of terabytes of text. Although they can generate explanations, those explanations are not derived from what has happened within the model. The text that LLMs produce is frequently incorrect and often describes things that are not true.

3. Discussion of Related Art

The Wikipedia page about Cyc, of 18 Jul. 2019, states that Cyc is the world's longest-lived artificial intelligence project, attempting to assemble a comprehensive ontology and knowledge base that spans the basic concepts and ârules of thumbâ about how the world works (think common sense knowledge but focusing more on things that rarely get written down or said, in contrast with facts one might find somewhere on the internet or retrieve via a search engine or Wikipedia), with the goal of enabling AI applications to perform human-like reasoning and be less âbrittleâ when confronted with novel situations that were not preconceived.

The Wikipedia page about Cyc, of 18 Jul. 2019, states that the objective of the Cyc project was to codify, in machine-usable form, the millions of pieces of knowledge that compose human common sense; this entailed, along the way, (1) developing an adequately expressive representation language, CycL, (2) developing an ontology spanning all human concepts down to some appropriate level of detail, (3) developing a knowledge base on that ontological framework, comprising all human knowledge about those concepts down to some appropriate level of detail, and (4) developing an inference engine exponentially faster than those used in then-conventional expert systems, to be able to infer the same types and depth of conclusions that humans are capable of, given their knowledge of the world.

The Wikipedia page about Cyc, of 18 Jul. 2019, states that most of Cyc's knowledge, outside math and games, is only true by default; for example, Cyc knows that as a default parents love their children, when you're made happy you smile, taking your first step is a big accomplishment, when someone you love has a big accomplishment that makes you happy, and only adults have children; when asked whether a picture captioned âSomeone watching his daughter take her first stepâ contains a smiling adult person, Cyc can logically infer that the answer is Yes, and âshow its workâ by presenting the step by step logical argument using those five pieces of knowledge from its knowledge base: these are formulated in the language CycL, which is based on predicate calculus and has a syntax similar to that of the Lisp programming language.

The Wikipedia page about Cyc, of 18 Jul. 2019, states that the Cyc project has been described as âone of the most controversial endeavors of the artificial intelligence historyâ; Catherine Havasi, CEO of Luminoso, says that Cyc is the predecessor project to IBM's Watson; machine-learning scientist Pedro Domingos refers to the project as a âcatastrophic failureâ for several reasons, including the unending amount of data required to produce any viable results and the inability for Cyc to evolve on its own; Robin Hanson, a professor of economics at George Mason University, gives a more balanced analysis: âOf course the CYC project is open to criticism on its many particular choices. People have complained about its logic-like and language-like representations, about its selection of prototypical cases to build from (e.g., encyclopedia articles), about its focus on answering over acting, about how often it rebuilds vs. maintaining legacy systems, and about being private vs. publishing everything. But any large project like this would produce such disputes, and it is not obvious any of its choices have been seriously wrong. They had to start somewhere, and in my opinion they have now collected a knowledge base with a truly spectacular size, scope, and integration. Other architectures may well work better, but if knowing lots is anywhere near as important as Lenat thinks, I'd expect serious AI attempts to import CYC's knowledge, translating it into a new representation. No other source has anywhere near CYC's size, scope, and integration.â

The True Knowledge system provided open-domain question answering using structured knowledge and inference. In the True Knowledge system, knowledge in the knowledge base was represented in a single unified format: named relations between pairs of named entities referred to as âfacts.â Facts, and the relations themselves, were first-class entities so facts about facts and facts about the properties of relations were fully supported (Tunstall-Pedoe, W. (2010). True Knowledge: Open-Domain Question Answering Using Structured Knowledge and Inference. AI Magazine, 31(3), 80-92. https://ojs.aaai.org/aimagazine/index.php/aimagazine/article/view/2298).

EP2527990A2 and EP2527990B1 disclose a method of using large language models in machine translation in which a translation model is partitioned into a plurality of language model partitions stored on a plurality of different language model servers. Segments of text are distributed to the servers for translation according to server workload. The language model information is usually collected by computing the frequency of occurrence of sequences of words in a large training corpus of documents.

SUMMARY OF THE INVENTION

According to a first aspect of the invention, there is provided a method of interacting with a LLM, including the step of a processing system using a structured, machine-readable representation of data that conforms to a machine-readable language, such as a universal language, to provide new context data for the LLM, in order to improve the output, such as continuation text output, generated by the LLM in response to a prompt.

According to a second aspect of the invention, there is provided a method of interacting with a LLM, including the step of providing continuation data generated by the LLM to a processing system that uses a structured, machine-readable representation of data that conforms to a machine-readable language, such as a universal language, in which the processing system is configured to analyse the continuation output (e.g. text output) generated by the LLM in response to a prompt to enable an improved version of that continuation output to be provided to a user. An example is shown in FIG. 10 . An advantage is that improved continuation output to that generated by the LLM may be provided.

According to a third aspect of the invention, there is provided a method of interacting with a LLM, including the step of generating continuation data by an LLM, by the LLM using the output of a processing system that uses a structured, machine-readable representation of data that conforms to a machine-readable language, such as a universal language, in which the LLM is configured to use the output of the processing system as a prompt and to generate a continuation output (e.g. text output) that is an improved version of the output from the processing system. An advantage is that an improved version of the output from the processing system may be generated by the LLM.

According to a fourth aspect of the invention, there is provided a method of interacting with a LLM, including the step of training the LLM on the output from a processing system using a structured, machine-readable representation of data that conforms to a machine-readable language, such as a universal language. An advantage is that an improved LLM may be provided.

According to a fifth aspect of the invention, there is provided a method for interacting with one or more users in natural language comprising the steps of:

providing a structured representation of data and a system operable to process language with reference to the structured representation of data to provide additional useful text relevant to the language; providing an LLM operable to provide a continuation of text; receiving a language input from one or more users; processing the language input with the system to produce additional useful text; providing the LLM with the language input plus the additional useful text; providing the one or more users with language derived from a continuation generated by the LLM. An advantage is that additional useful text to that provided by a system operable to process language with reference to the structured representation of data may be provided by a continuation generated by the LLM.

According to a sixth aspect of the invention, there is provided a method of validating natural language for factual accuracy comprising the steps of:

Extracting from the natural language a collection of one or more factual assertions asserted within the natural language; Checking the one or more factual assertions for factual accuracy; Outputting, or making available, the results of the checking.

An advantage is validating natural language for factual accuracy.

According to a seventh aspect of the invention, there is provided a method of learning knowledge in structured form comprising the steps of:

Providing an LLM trained on a large corpus of natural language documents; Providing the LLM with a prompt that will elicit desired knowledge; Eliciting a continuation of the prompt from the LLM; Translating the continuation into the structured form.

An advantage is learning knowledge in structured form.

According to an eighth aspect of the invention, there is provided a method for ensuring that an LLM generates original text comprising the steps of:

Providing a database of previous text that it should not generate; Performing a beam search; Checking potential continuations against the database; When a potential continuation matches non-original text in the database, adjusting the continua

CROSS-REFERENCE TO RELATED APPLICATIONS

BACKGROUND OF THE INVENTION

1. Field of the Invention

2. Technical Background

Accounting is another application where vast heterogeneous data would be valuable: the perfect accounting application would represent every type of contract, every type of service.

These issues could be addressed if there existed a language or way of representing data that computers could fully process and understand but that also had an extremely broad scope.

3. Discussion of Related Art

SUMMARY OF THE INVENTION

According to a fifth aspect of the invention, there is provided a method for interacting with one or more users in natural language comprising the steps of:

According to a sixth aspect of the invention, there is provided a method of validating natural language for factual accuracy comprising the steps of:

An advantage is validating natural language for factual accuracy.

According to a seventh aspect of the invention, there is provided a method of learning knowledge in structured form comprising the steps of:

An advantage is learning knowledge in structured form.

According to an eighth aspect of the invention, there is provided a method for ensuring that an LLM generates original text comprising the steps of:

An advantage is avoiding copyright infringement.

According to a ninth aspect of the invention, there is provided a method of avoiding hallucination in an LLM comprising the steps of:

Analysing different continuations for a given prompt; Comparing the continuations to identify parts of the response that do not vary and parts of the response that do vary; Identifying the parts that vary as more likely to be hallucination; Replacing or not showing the parts identified as hallucination to the user.

An advantage is avoiding hallucination in an LLM.

According to a tenth aspect of the invention, there is provided a method for adding citations to text generated by an LLM comprising the steps of:

Providing a database of text used to train the LLM which includes sources associated with each section of text stored; checking sections of the continuation generated by the LLM against the database; retrieving sources where the sections of the continuation match text contained within the database. An advantage is that citations are added to text generated by an LLM.

According to an eleventh aspect of the invention, there is provided a computer-implemented method including the steps of

(i) Receiving output from a natural language processing computer process, the output including an answer to a question; (ii) Inputting the received output to a large language model (LLM); (iii) The large language model (LLM) processing the input to the LLM, to generate output based on the input to the LLM. (iv) Storing the output based on the input to the LLM.

An advantage is that an improved answer to the question may be provided by the LLM output.

According to a twelfth aspect of the invention, there is provided a computer-implemented method of training a large language model (LLM), including the steps of

(i) receiving output from a natural language processing computer process, the output including an answer to a question; (ii) repeating step (i) at least one thousand times using a set of at least one thousand different questions, and storing the answers to the questions in a training file; (iii) using the training file to train a large language model (LLM); (iv) storing weights characterizing the trained LLM.

An advantage is that a trained LLM with training improved by the use of the training file is provided.

According to a thirteenth aspect of the invention, there is provided a computer-implemented method of generating a training file for a large language model (LLM), including the steps of

According to a fourteenth aspect of the invention, there is provided a computer-implemented method of re-training a large language model (LLM), the LLM having been previously trained using a training file, the method including the steps of

(i) receiving output from a natural language processing computer process, the output including an answer to a question; (ii) repeating step (i) at least one thousand times using a set of at least one thousand different questions, and storing the answers to the questions in a re-training file; (iii) combining the training file and the re-training file, to generate a combined training file; (iv) using the combined training file to re-train the large language model (LLM); (v) storing weights characterizing the re-trained LLM.

An advantage is that an LLM with an improved training is provided.

According to a fifteenth aspect of the invention, there is provided a computer-implemented method of updating the training of a large language model (LLM), the LLM having been previously trained using a training file, the method including the steps of

(i) receiving output from a natural language processing computer process, the output including an answer to a question; (ii) repeating step (i) at least one thousand times using a set of at least one thousand different questions, and storing the answers to the questions in a re-training file; (iii) using the re-training file to update the training of the large language model (LLM); (iv) storing weights characterizing the LLM with the updated training. An advantage is that an LLM with an improved training is provided.

According to a sixteenth aspect of the invention, there is provided a computer-implemented method of improving output using an LLM, including the steps of

(i) receiving a natural language question (e.g. any natural language to which a natural language response is appropriate. It could be for example a command or a request for data or even some kind of social interaction or discussion); (ii) using a computer system including a processor and a memory, the processor configured to use a processing language in which semantic nodes are represented in the processing language, the semantic nodes including semantic links between semantic nodes wherein the semantic links are themselves semantic nodes, in which each semantic node denotes one specific meaning, in which a combination of semantic nodes defines a semantic node, in which expressions in the processing language may be nested, in which the natural language question is represented in the processing language, in which reasoning steps are represented in the processing language to represent semantics of the reasoning steps, in which computation units are represented in the processing language, wherein the memory is configured to store the representations in the processing language; (iii) the processor answering the question using the reasoning steps, the computation units and the semantic nodes, and (iv) inputting the natural language question, and the processor's answer to the question to the LLM; (v) the large language model (LLM) processing the input to the LLM, to generate output based on the input to the LLM; (vi) storing the output based on the input to the LLM.

An advantage is that an improved answer to the question may be provided by the LLM output.

According to a seventeenth aspect of the invention, there is provided a computer-implemented method of improving output from an LLM, including the steps of

(i) receiving a first natural language question; (ii) Inputting or providing the received first natural language question to a large language model (LLM); (iii) The large language model (LLM) processing the input to the LLM, to generate output based on the input to the LLM; (iv) translating the output into a processing language in which semantic nodes are represented in the processing language, the semantic nodes including semantic links between semantic nodes wherein the semantic links are themselves semantic nodes, in which each semantic node denotes one specific meaning, in which a combination of semantic nodes defines a semantic node, in which expressions in the processing language may be nested; (v) constructing a question which asks if the output is true, in which the question is represented in the processing language; (vi) inputting the question to a computer system including a processor and a memory, the processor configured to use the processing language, in which reasoning steps are represented in the processing language to represent semantics of the reasoning steps, in which computation units are represented in the processing language, wherein the memory is configured to store the representations in the processing language; (vii) the processor answering the question using the reasoning steps, the computation units and the semantic nodes, and (viii) the processor storing an answer to the question in the memory.

An advantage is that the LLM output is checked for accuracy.

According to an eighteenth aspect of the invention, there is provided a computer-implemented method of fact checking output from a large language model (LLM), including the steps of

(i) receiving a text input; (ii) inputting or providing the received text input to a large language model (LLM); (iii) the large language model (LLM) processing the input to the LLM, to generate output based on the input to the LLM; (iv) translating the output into a processing language in which semantic nodes are represented in the processing language, the semantic nodes including semantic links between semantic nodes wherein the semantic links are themselves semantic nodes, in which each semantic node denotes one specific meaning, in which a combination of semantic nodes defines a semantic node, in which expressions in the processing language may be nested, wherein translating the output includes extracting the assertions in text generated by the LLM; (v) constructing one or more questions which ask if the extracted assertions are individually true, in which the one or more questions are represented in the processing language; (vi) inputting the one or more questions to a computer system including a processor and a memory, the processor configured to use the processing language, in which reasoning steps are represented in the processing language to represent semantics of the reasoning steps, in which computation units are represented in the processing language, wherein the memory is configured to store the representations in the processing language; (vii) the processor answering the one or more questions using the reasoning steps, the computation units and the semantic nodes, and (viii) the processor storing an individual answer to each of the one or more questions in the memory. An advantage is fact checking output from a large language model.

According to a nineteenth aspect of the invention, there is provided a computer-implemented method of improving output from a computer system including a processor and a memory, the system operable to process the natural language question using a processing language in which semantic nodes are represented in the processing language, the method including using an LLM, the method including the steps of

(i) receiving a natural language question; (ii) using a computer system including a processor and a memory, the system operable to process the natural language question using a processing language in which semantic nodes are represented in the processing language, the semantic nodes including semantic links between semantic nodes wherein the semantic links are themselves semantic nodes, in which each semantic node denotes one specific meaning, wherein the memory is configured to store the representations in the processing language; (iii) the processor answering the question, and (iv) inputting the natural language question, and the processor's answer to the question to the LLM; (v) the large language model (LLM) processing the input to the LLM, to generate output based on the input to the LLM; (vi) Storing the output based on the input to the LLM.

An advantage is that output from the computer system including a processor and a memory, the system operable to process the natural language question using a processing language in which semantic nodes are represented in the processing language, is improved using the LLM.

According to a twentieth aspect of the invention, there is provided a computer-implemented method of improving output from an LLM, including the steps of

(i) receiving a first natural language question; (ii) inputting the received first natural language question to a large language model (LLM); (iii) the large language model (LLM) processing the input to the LLM, to generate output based on the input to the LLM; (iv) translating the output into a processing language in which semantic nodes are represented in the processing language, the semantic nodes including semantic links between semantic nodes wherein the semantic links are themselves semantic nodes, in which each semantic node denotes one specific meaning; (v) constructing a question which asks if the output is true, in which the question is represented in the processing language; (vi) inputting the question to a computer system including a processor and a memory, the processor configured to use the processing language, wherein the memory is configured to store the representations in the processing language; (vii) the processor answering the question, and (viii) the processor storing an answer to the question in the memory. An advantage is that the LLM output is checked for accuracy.

According to a 21st aspect of the invention, there is provided a computer-implemented method of fact checking output from a large language model (LLM), including the steps of

(i) receiving a text input; (ii) inputting the received text input to a large language model (LLM); (iii) the large language model (LLM) processing the input to the LLM, to generate output based on the input to the LLM; (iv) translating the output into a processing language in which semantic nodes are represented in the processing language, the semantic nodes including semantic links between semantic nodes wherein the semantic links are themselves semantic nodes, in which each semantic node denotes one specific meaning, wherein translating the output includes extracting the assertions in text generated by the LLM; (v) constructing one or more questions which ask if the extracted assertions are individually true, in which the one or more questions are represented in the processing language; (vi) inputting the one or more questions to a computer system including a processor and a memory, the processor configured to use the processing language, wherein the memory is configured to store the representations in the processing language; (vii) the processor answering the one or more questions, and (viii) the processor storing an individual answer to each of the one or more questions in the memory.

An advantage is fact checking output from a large language model.

According to a 22nd aspect of the invention, there is provided a computer implemented method of augmenting search results, including the steps of:

(i) receiving a search query; (ii) an (e.g. internet) search engine processing the search query to generate (e.g. internet) search results; (iii) transforming the search results into a structured, machine-readable representation of data that conforms to a machine-readable language, such as a universal language; (iv) processing the transformed search results using a processing system which uses the structured, machine-readable representation of data that conforms to the machine-readable language, such as a universal language, to produce output; (v) supplying the output of step (iv) as input to a large language model (LLM), and the LLM generating continuation data output in response to the input. An advantage is that search results found by a search engine are augmented.

According to a 23rd aspect of the invention, there is provided a method of fact-checking the output of a large language model (LLM), including the step of providing continuation data generated by the LLM to a processing system that uses a structured, machine-readable representation of data that conforms to a machine-readable language, in which the processing system analyses the continuation output generated by the LLM in response to a prompt, to enable an improved, fact-checked version of that continuation output to be generated and provided to a user. An advantage is fact checking output from a large language model.

Further aspects of the invention are given in the dependent Claims, and in the independent Claims. Aspects of the invention may be combined with each other, or with any of the Concepts described herein, or with any of the Claims.

BRIEF DESCRIPTION OF THE FIGURES

Aspects of the invention will now be described, by way of example(s), with reference to the following Figures, in which:

FIG. 1 shows example screen output for a notification of a job match.

FIG. 2 shows example screen output for a description of a job match.

FIG. 3 shows an example conversation within an app where nutritional data is being communicated with the app.

FIG. 4 shows some example insights that can be derived from a period of time where horizontal health and nutrition data was shown.

FIG. 5 shows an example graph showing daily calories in versus calories out which is an extremely common thing for someone to track if the user is aiming to lose (or gain) weight.

FIG. 6 shows an example of a visualisation capable of being generated from an example of the present invention: it is comparing estimated caffeine in the user's body at the time they went to bed with a calculation of sleep quality.

FIG. 7 shows examples of explanations generated. (a) shows examples of simplified explanations; (b) shows an example of a detailed explanation.

FIG. 8 shows an example of a voice assistant product, referred to herein as âBrianâ and how it fits in with the wider UL platform and other applications built on the UL platform.

FIG. 9 shows an alternative to the example of FIG. 8 .

FIG. 10 shows an example of a method of interacting with a LLM, including the step of providing continuation data generated by the LLM to a processing system that uses a structured, machine-readable representation of data that conforms to a machine-readable language, such as a universal language, in which the processing system is configured to analyse the continuation output (e.g. text output) generated by the LLM in response to a prompt to enable an improved version of that continuation output to be provided to a user.

FIG. 11 shows an example of an intermediate representation semantic graph.

DETAILED DESCRIPTION

This disclosure includes a computer implemented method for the automated analysis or use of data, which comprises the steps of:

(a) storing in a memory a structured, machine-readable representation of data that conforms to a machine-readable language (âmachine representationâ); the machine representation including representations of user speech or text input to a human/machine interface; (b) automatically processing the machine representations to analyse the user speech or text input.

This disclosure includes a computer-based system configured to analyse data, in which the system is configured to:

(a) store in a memory a structured, machine-readable representation of data that conforms to a machine-readable language; the structured, machine-readable representation of data including representations of user speech or text input to a human/machine interface; (b) automatically process the structured representations to analyse the user speech or text input to a human/machine interface.

These aspects of the disclosure may be implemented in a voice assistant or chatbot; the technical advantage achieved is to enable more extensive and faster scaling of the voice assistant or chatbot's capabilities; the invention enables voice assistants and chatbot to be able to answer a broader range of questions, and to do so more accurately; this makes it easier for the voice assistant or chatbot to work with large numbers of different natural languages.

One point of interpretation: the conjunction âorâ should not be narrowly construed to indicate mutual exclusivity but to cover inclusivity too. Hence, the phrase âuser speech or text inputâ means âuser speech on its own, or user text on its own, or both user speech and also user textâ. When the conjunction âorâ is meant exclusively, the phrase âeither . . . orâ is used.

The machine-readable language is extremely expressive yet also extremely simple; simplicity requires less computer processing and thus delivers faster performance. Further details are in the appended Concepts and Claims.

According to a further aspect of the disclosure, there is provided a computer system including a processor and a memory, the processor configured to answer a question, the processor configured to use a processing language in which semantic nodes are represented in the processing language, the semantic nodes including semantic links between semantic nodes wherein the semantic links are themselves semantic nodes, in which each semantic node denotes one specific meaning, in which a combination of semantic nodes defines a semantic node, in which expressions in the processing language may be nested, in which the question is represented in the processing language, in which reasoning steps are represented in the processing language to represent semantics of the reasoning steps, in which computation units are represented in the processing language, wherein the memory is configured to store the representations in the processing language, and wherein the processor is configured to answer the question using the reasoning steps, the computation units and the semantic nodes, and to store an answer to the question in the memory.

An advantage is that because semantic links between semantic nodes are themselves semantic nodes, semantic links and semantic nodes do not need to be processed in a distinctly different way, which simplifies processing, which speeds up response times, which is a technical effect.

An advantage is that because semantic nodes are very widely used in the processing language, processing of the processing language is speeded up, which speeds up response times, which is a technical effect.

The technical effect operates at the level of the architecture of the computer system; that is to say the effect is produced irrespective of the data being processed.

The technical effect results in the computer system being made to operate in a new way, because the computer finds answers to questions faster than using prior art approaches, because semantic nodes are very widely used in the processing language, which means that processing of the processing language is speeded up.

The processing language contributes to the technical character because it produces a technical effect: a processor processing the processing language has the effect that the computer finds answers to questions faster than using prior art approaches, because semantic nodes are very widely used in the processing language, which means that processing of the processing language is speeded up.

The computer system may be configured to output the answer to the question.

The computer system may be configured to output the answer to the question to a display device.

The computer system may be one wherein expressions in the processing language may be nested with no limit inherent to the processing language.

The computer system may be one wherein the semantic nodes each includes a unique identifier.

The computer system may be one wherein the computation units are semantic nodes.

The computer system may be one wherein the question is represented in the processing language with a passage comprising a semantic node that identifies the passage as a question, a list of zero, one or more semantic nodes representing unknown entities being asked about and at least one further passage which represents the semantics of the question in the context of the zero, one or more unknown entities.

The computer system may be one wherein the processing language is universal language.

The computer system may be one wherein the processing language is not a natural language.

The computer system may be one wherein the question relates to search and analysis of documents or web pages, wherein the semantic nodes include representations of at least parts of the documents or the web pages stored in a document store.

The computer system may be one wherein the question relates to a location-based search, using mapping data represented as semantic nodes in the processing language.

The computer system may be one wherein the question relates to a search for defined advertisements or news, wherein the semantic nodes include representations of advertisements, news articles or other information items.

The computer system may be one wherein the question relates to a request for a summary of a news topic, wherein the semantic nodes include representations of news from multiple sources, e.g. to provide a summary or aggregation of the news.

The computer system may be one wherein the question relates to a request for a compatibility match between persons, wherein the semantic nodes include representations of personal information defining one or more attributes of a person, for a plurality of people.

The computer system may be one wherein the question relates to compliance with requirements preventing abusive or illegal social media postings, wherein the semantic nodes include representations of social media postings.

The computer system may be one wherein the question relates to analysing customer reviews, wherein the semantic nodes include representations of customer reviews.

The computer system may be one wherein the question relates to a user's product request, wherein the semantic nodes include representations of product descriptions and user product requests.

The computer system may be one wherein the question relates to a job search, wherein the semantic nodes include representations of job descriptions and job applicants' skills and experience, to determine which job applicants match a job description, or to determine which job descriptions match a job applicant's skills and experience.

The computer system may be one wherein the question relates to health of an individual, wherein the semantic nodes include health data relating to the individual, and health data relating to human beings.

The computer system may be one wherein the question relates to nutrition, wherein the semantic nodes include nutritional data for foods and drinks.

The computer system may be one wherein the question relates to accounting or finance, wherein the semantic nodes include representations of financial or accounting information.

The computer system may be one wherein the question is received by a voice assistant or chatbot, wherein the semantic nodes include representations of user speech input to a human/machine interface and include representations of the human/machine interface itself.

According to a further aspect of the disclosure, there is provided a computer-implemented method, the method using a computer system including a processor and a memory, the processor configured to use a processing language in which semantic nodes are represented in the processing language, the semantic nodes including semantic links between semantic nodes wherein the semantic links are themselves semantic nodes, in which each semantic node denotes one specific meaning, in which a combination of semantic nodes defines a semantic node, in which expressions in the processing language may be nested, in which the question is represented in the processing language, in which reasoning steps are represented in the processing language to represent semantics of the reasoning steps, in which computation units are represented in the processing language, wherein the memory is configured to store the representations in the processing language, the method including the steps of:

(i) the processor answering the question using the reasoning steps, the computation units and the semantic nodes, and (ii) the processor storing an answer to the question in the memory.

Advantages include the advantages of the previous aspect of the disclosure.

The method may be one wherein the question is represented in the processing language with a passage comprising a semantic node that identifies the passage as a question, a list of zero, one or more semantic nodes representing unknown entities being asked about and at least one further passage which represents the semantics of the question in the context of the zero, one or more unknown entities.

The method may be one wherein the unknowns in the question are identified and the passage making up the body of the question is selected for further analysis; processing begins on a list of passages from the body of the question and the selected unknowns; a first passage in the list of passages is selected for processing; processing a single passage comprises three methods: using statically stored processing language passages, utilising computation units and utilising processing language generated from reasoning:

in which the first method is to lookup in the passage store if there are any passages that can be directly mapped with the passage being processed; if the passage is exactly the same structure as a passage in the passage store, with all nodes matching other than the unknowns, then the values the unknowns match against are valid results; the second method is to check if any results can be found by executing computation units; it is checked if this passage matches against any passages in a computation unit description; all non-unknown nodes in the passage being processed must match the same nodes in the corresponding position in the computation description or align with a computation input unknown; the unknowns being processed must align to output unknowns in the description; the computation unit is then called to get valid output values for the processed passage's unknowns; the third method is to see if this passage can be proved by applying any reasoning steps; reasoning steps are searched for where a passage in the second half of the reasoning passage can be unified with the passage being processed; all nodes and structure must be equal between the two passages, other than unknowns in the focus passage or the reasoning passage; if a reasoning passage like this is found it means that this reasoning step could be used to prove the passage being processed; a multi-stage process is used to first find any mappings for unknowns in the processed passage when matching with the reasoning passage; secondly, mappings for unknowns used in the reasoning passage are found by mapping with the passage being processed; this mapping can then be applied to the front half of the reasoning passage to generate a list of passages that, if they can be matched with known or generated processing language and mappings found for them, will prove and find valid mappings for the focus passage; solutions for the list of passages can then be found recursively.

The method may use a computer system of any aspect of the previous aspect of the disclosure.

Aspects of the disclosure may be combined.

Examples of the present invention include systems and methods for creating and using a structured representation of data that is aimed to be as expressive as natural language in terms of breadth but which also can be processed

CLAIMS

Claims ( 30 )

The invention claimed is:

1. A method of improving operation of a generative AI large language model (LLM)-based data processing system, in which a large language model is a deep learning model capable of processing natural language, in which the method includes the steps of:

(a) providing an input to a non-LLM data processing system that uses symbolic representations to analyse the input;

(b) the non-LLM system using, accessing or searching a knowledge or data source external to the LLM-based system to construct or enable an enhanced or augmented version of that input;

(c) the non-LLM system providing the enhanced or augmented version of that input to the LLM-based system as a prompt or other context, and

(d) the LLM-based system then using that prompt or other context to generate a continuation or other output that is fact-checked, accurate and reliable.

2. The method of claim 1 in which the non-LLM system acts as an agent for the LLM-based system.

3. The method of claim 1 in which the non-LLM system provides a semantic backbone or support for the LLM-based system.

4. The method of claim 1 in which the non-LLM system is programmed to use a non-statistical symbolic representation of one or more of the following: things, relations between things, facts, relations between facts.

5. The method of claim 1 in which the non-LLM system uses a symbolic representation to provide the prompt or other context to the LLM-based system, where this prompt or other context improves one or more of the following parameters of the continuation or other output generated by the LLM-based system: factual accuracy and/or factual scope of the continuation or other output; internal, logical self-consistency of the continuation or other output; correspondence of the continuation or other output to how people understand the real world or reason in the real world; bias reduction or removal in the continuation or other output; inclusion of dynamic or real-time information in the continuation or other output.

6. The method of claim 1 in which the non-LLM system uses a symbolic representation to provide the prompt or other context to the LLM-based system, where the prompt or other context includes one or more the following: (i) dynamic or real-time information; (ii) reasoned text, such as text derived from a non-statistical reasoning process; (iii) an answer to a question, where the question is derived from a continuation output of the LLM-based system; (iv) an answer to a question that has been derived from a factual assertion, itself derived from an output of the LLM-based system.

7. The method of claim 1 in which the non-LLM system provides the prompt or other context to the LLM-based system, where the prompt or other context is labelled with a level of certainty or uncertainty, or trust or lack of trust.

8. The method of claim 1 in which the input from the LLM-based system is a partial continuation, namely a continuation or other output made before the LLM-based system has stopped generating or whilst the LLM-based system is still generating.

9. The method of claim 1 in which a classifier operates to (a) identify when an input to the LLM-based system is likely to result in a continuation or other output from the LLM-based system where accuracy is important, and (b) to then use the non-LLM system to improve factual accuracy and/or factual scope of that continuation or other output.

10. The method of claim 1 in which the method includes the step of providing the input to the non-LLM system, where the non-LLM system uses symbolic representations to analyse the input and then accesses or searches an authoritative knowledge or data source external to the LLM-based system to generate an enhanced or augmented version of that input and provide that enhanced or augmented version of that input to the LLM-based system as an augmented prompt or other context.

11. The method of claim 10 in which the authoritative knowledge or data source external to the LLM-based system is one or more of: the web, social media, and document stores;

and the non-LLM system provides the augmented prompt or other context to the LLM-based system to improve one or more of the following parameters of the continuation or other output of the LLM-based system: factual accuracy and/or factual scope of the continuation or other output; internal, logical self-consistency of the continuation or other output; correspondence of the continuation or other output to how people understand the real world or reason in the real world; bias reduction or removal in the continuation or other output; inclusion of dynamic or real-time information in the continuation or other output.

12. The method of claim 10 in which the non-LLM system analyses the input to the non-LLM system by extracting from it a collection of one or more factual assertions, and checks one or more of the factual assertions for accuracy, and then outputs, or makes available, results of the checking to enable the enhanced or augmented version of that input to be provided to the LLM-based system as the prompt or other context.

13. The method of claim 10 in which the non-LLM system includes a reasoning system able to reason with a symbolic representation of the world, and the LLM-based system provides the input to the non-LLM system which then (i) translates the input into one or more assertions in a form compatible with the reasoning system and (ii) utilises the reasoning system to identify one or more assertions which are true or false.

14. The method of claim 10 in which the non-LLM processing system includes a symbolic reasoning system able to reason with symbolic representations, and to translate assertions into questions and to answer those questions.

15. The method of claim 1 in which the LLM-based system generates an initial continuation or other output, and the non-LLM system (a) uses a symbolic process to analyse the initial continuation or other output against an authoritative data source to identify accurate citations or sources relevant to that initial continuation or other output, and (b) the non-LLM system then provides the accurate citations or sources back to the LLM-based system.

16. The method of claim 15 in which the LLM-based system then includes the accurate citations or sources into a new continuation or other output provided to an end-user.

17. The method of claim 15 in which the LLM-based system revises its initial continuation or other output using information from the accurate citations or sources so that the new continuation or other output it provides to an end-user is consistent with the accurate citations or sources.

18. The method of claim 1 which includes the step of the non-LLM system providing labels or other data to the LLM-based system that the LLM-based system uses to control the output generated by the LLM-based system, such as labels that relate to certainty, brevity, expectation that the response will be spoken, level of formality; use or non-use of profanity, context of age or other details of person being addressed, emotion to be conveyed in the response;

and the labels or other data are hidden from the continuation or other output displayed to an end-user by the LLM-based system.

19. The method of claim 1 in which the LLM-based system is configured to operate steps that are not shown to an end-user for brevity, clarity or other purposes, such as creating something shorter that can be spoken or for other reasons where an intermediate data or calculation or step is better not shown to the end-user.

20. The method of claim 1 in which the method includes the step of the non-LLM system performing a symbolic computation on the input using multi-dimensional vectors.

21. The method of claim 1 in which the method includes the step of the non-LLM system providing the prompt or other context as training data to the LLM-based system.

22. The method of claim 21 in which the training data is reasoned text derived using a non-statistical, symbolic reasoning process.

23. The method of claim 21 in which if the LLM-based system is generating an answer to a question, then the training data provided to the LLM-based system by the non-LLM system is an answer to that question derived using a non-statistical, symbolic reasoning process.

24. The method of claim 21 in which the training data provided to the LLM-based system includes dynamic or real-time information.

25. The method of claim 21 in which the training data provided to the LLM-based system is labelled with a level of certainty or uncertainty, or trust or lack of trust.

26. The method of claim 21 in which the non-LLM system provides training data to the LLM-based system to improve any of the following aspects of the output of the LLM-based system: (i) factual accuracy and/or factual scope; (ii) internal, logical self-consistency; (iii) correspondence to how people understand the real world or reason in the real world; (iv) bias.

27. The method of claim 1 in which the LLM-based system provides an input to the non-LLM system that then (a) automatically analyses the input using a symbolic process to generate factually accurate training data, and (b) provides the factually accurate training data back to the LLM-based system to train the LLM-based system.

28. The method of claim 27 in which the input sent to the non-LLM system is an answer to a question, or is processed by the non-LLM system into an answer to a question, and the non-LLM system generates, using a symbolic process, factually accurate training data that includes a factually accurate answer to that question.

29. The method of claim 1 in which the LLM-based system and the non-LLM system are combined, in which the combined LLM-based system and the non-LLM system is configured to process data that relates to documents stored in a document store;

and the method includes the step of the combined LLM-based system and non-LLM system automatically processing the data for one or more of the following: to derive facts or relationships, to reason, to learn, to translate, to answer questions, to process natural language content, to enable man-machine interaction, to represent and to police rules or tenets, to enable one or more vertical applications that require a search for, or analysis of, documents.

30. A computer implemented system including a generative AI large language model (LLM)-based data processing system, in which a large language model is a deep learning model capable of processing natural language, and a non-LLM data processing system; in which:

(a) the non-LLM system is configured to use symbolic representations to analyse an input from the LLM-based system, wherein the LLM-based system includes a deep learning model capable of processing natural language;

(b) the non-LLM system is configured to use, access or search a knowledge or data source external to the LLM-based system to construct or enable an enhanced or augmented version of that input;

(c) the non-LLM system is configured to provide the enhanced or augmented version of that input to the LLM-based system as a context or prompt; and

d) the LLM-based system is configured to use that context or prompt to generate a continuation or other output that is fact-checked, accurate and reliable.

US18/923,851

2021-08-24

2024-10-23

Computer implemented methods for the automated analysis or use of data, including use of a large language model

Active

US12353827B2

( en )

Priority Applications (1)

Application Number

Priority Date

Filing Date

Title

US18/923,851

US12353827B2

( en )

2021-08-24

2024-10-23

Computer implemented methods for the automated analysis or use of data, including use of a large language model

Applications Claiming Priority (18)

Application Number

Priority Date

Filing Date

Title

PCT/GB2021/052196

WO2022043675A2

( en )

2020-08-24

2021-08-24

A computer implemented method for the automated analysis or use of data

GB2202347

2022-02-22

GB2202347.7

2022-02-22

GBGB2202347.7A

GB202202347D0

( en )

2022-02-22

Unlikely ai 2nd gen

US202218001368A

2022-12-09

GB2219268.6

2022-12-20

GB2219268

2022-12-20

GBGB2219268.6A

GB202219268D0

( en )

2022-12-20

Unlikely ai 2nd gen ii

GB2300624.0

2023-01-16

GBGB2300624.0A

GB202300624D0

( en )

2023-01-16

Unlikely ai 2nd gen iii

GB2300624

2023-01-16

GB2302085.2

2023-02-14

GB2302085

2023-02-14

GBGB2302085.2A

GB202302085D0

( en )

2023-02-14

Unlikely AI 2nd gen IV

PCT/GB2023/050405

WO2023161630A1

( en )

2022-02-22

2023-02-22

Computer implemented methods for the automated analysis or use of data, including use of a large language model

US18/301,615

US11989507B2

( en )

2021-08-24

2023-04-17

Computer implemented methods for the automated analysis or use of data, including use of a large language model

US18/648,788

US12164868B2

( en )

2021-08-24

2024-04-29

Computer implemented methods for the automated analysis or use of data, including use of a large language model

US18/923,851

US12353827B2

( en )

2021-08-24

2024-10-23

Computer implemented methods for the automated analysis or use of data, including use of a large language model

Related Parent Applications (1)

Application Number

Title

Priority Date

Filing Date

US18/648,788

Continuation

US12164868B2

( en )

2021-08-24

2024-04-29

Computer implemented methods for the automated analysis or use of data, including use of a large language model

Publications (2)

Publication Number

Publication Date

US20250045520A1

US20250045520A1 ( en )

2025-02-06

US12353827B2

true

US12353827B2 ( en )

2025-07-08

Family

ID=87766942

Family Applications (14)

Application Number

Title

Priority Date

Filing Date

US18/301,615

Active

US11989507B2

( en )

2021-08-24

2023-04-17

Computer implemented methods for the automated analysis or use of data, including use of a large language model

US18/648,788

Active

US12164868B2

( en )

2021-08-24

2024-04-29

Computer implemented methods for the automated analysis or use of data, including use of a large language model

US18/914,717

Active

US12456008B2

( en )

2021-08-24

2024-10-14

Computer implemented methods for the automated analysis or use of data, including use of a large language model

US18/914,659

Active

US12430504B2

( en )

2021-08-24

2024-10-14

Computer implemented methods for the automated analysis or use of data, including use of a large language model

US18/914,537

Active

US12430503B2

( en )

2021-08-24

2024-10-14

Computer implemented methods for the automated analysis or use of data, including use of a large language model

US18/914,686

Active

US12430505B2

( en )

2021-08-24

2024-10-14

Computer implemented methods for the automated analysis or use of data, including use of a large language model

US18/923,851

Active

US12353827B2

( en )

2021-08-24

2024-10-23

Computer implemented methods for the automated analysis or use of data, including use of a large language model

US19/364,721

Pending

US20260080163A1

( en )

2021-08-24

2025-10-21

Computer implemented methods for the automated analysis or use of data, including use of a large language model

US19/383,139

Pending

US20260064999A1

( en )

2021-08-24

2025-11-07

Computer implemented methods for the automated analysis or use of data, including use of a large language model

US19/383,076

Pending

US20260127365A1

( en )

2021-08-24

2025-11-07

Computer implemented methods for the automated analysis or use of data, including use of a large language model

US19/383,036

Pending

US20260127364A1

( en )

2021-08-24

2025-11-07

Computer implemented methods for the automated analysis or use of data, including use of a large language model

US19/397,156

Pending

US20260080164A1

( en )

2021-08-24

2025-11-21

Computer implemented methods for the automated analysis or use of data, including use of a large language model

US19/397,080

Pending

US20260127367A1

( en )

2021-08-24

2025-11-21

Computer implemented methods for the automated analysis or use of data, including use of a large language model

US19/396,998

Pending

US20260127366A1

( en )

2021-08-24

2025-11-21

Computer implemented methods for the automated analysis or use of data, including use of a large language model

Family Applications Before (6)

Application Number

Title

Priority Date

Filing Date

US18/301,615

Active

US11989507B2

( en )

2021-08-24

2023-04-17

Computer implemented methods for the automated analysis or use of data, including use of a large language model

US18/648,788

Active

US12164868B2

( en )

2021-08-24

2024-04-29

Computer implemented methods for the automated analysis or use of data, including use of a large language model

US18/914,717

Active

US12456008B2

( en )

2021-08-24

2024-10-14

Computer implemented methods for the automated analysis or use of data, including use of a large language model

US18/914,659

Active

US12430504B2

( en )

2021-08-24

2024-10-14

Computer implemented methods for the automated analysis or use of data, including use of a large language model

US18/914,537

Active

US12430503B2

( en )

2021-08-24

2024-10-14

Computer implemented methods for the automated analysis or use of data, including use of a large language model

US18/914,686

Active

US12430505B2

( en )

2021-08-24

2024-10-14

Computer implemented methods for the automated analysis or use of data, including use of a large language model

Family Applications After (7)

Application Number

Title

Priority Date

Filing Date

US19/364,721

Pending

US20260080163A1

( en )

2021-08-24

2025-10-21

Computer implemented methods for the automated analysis or use of data, including use of a large language model

US19/383,139

Pending

US20260064999A1

( en )

2021-08-24

2025-11-07

Computer implemented methods for the automated analysis or use of data, including use of a large language model

US19/383,076

Pending

US20260127365A1

( en )

2021-08-24

2025-11-07

Computer implemented methods for the automated analysis or use of data, including use of a large language model

US19/383,036

Pending

US20260127364A1

( en )

2021-08-24

2025-11-07

Computer implemented methods for the automated analysis or use of data, including use of a large language model

US19/397,156

Pending

US20260080164A1

( en )

2021-08-24

2025-11-21

Computer implemented methods for the automated analysis or use of data, including use of a large language model

US19/397,080

Pending

US20260127367A1

( en )

2021-08-24

2025-11-21

Computer implemented methods for the automated analysis or use of data, including use of a large language model

US19/396,998

Pending

US20260127366A1

( en )

2021-08-24

2025-11-21

Computer implemented methods for the automated analysis or use of data, including use of a large language model

Country Status (1)

Country

Link

( 14 )

US11989507B2

( en )

Cited By (1)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20260127232A1

( en )

2024-11-01

2026-05-07

Google Llc

Merging generative model prompts based on context

Families Citing this family (86)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US12423507B2

( en )

2021-07-12

2025-09-23

International Business Machines Corporation

Elucidated natural language artifact recombination with contextual awareness

US20230169561A1

( en )

2021-11-30

2023-06-01

The Florida State University Research Foundation, Inc.

Systems and methods for automatically extracting information from text data

US12299404B2

( en )

2022-08-22

2025-05-13

Rohirrim, Inc.

Computer-generated content based on text classification, semantic relevance, and activation of deep learning large language models

US20240289545A1

( en )

2023-02-28

2024-08-29

Microsoft Technology Licensing, Llc

Automated plan generation and execution system for large language models

US12481823B2

( en )

2023-03-10

2025-11-25

Microsoft Technology Licensing, Llc

Predicate-guided preparation for LLM integrations in spreadsheet environments

US12405982B2

( en )

2023-04-18

2025-09-02

Twilio Inc.

Tool for providing contextual data for natural language queries

US12511282B1

( en )

2023-05-02

2025-12-30

Microstrategy Incorporated

Generating structured query language using machine learning

US12475151B1

( en )

2023-05-04

2025-11-18

Vijay Madisetti

Fault tolerant multi-agent generative AI applications

US12493638B2

( en )

2023-05-04

2025-12-09

Vijay Madisetti

Method and system for multi-level artificial intelligence supercomputer design

US12405977B1

( en )

2023-09-20

2025-09-02

Vijay Madisetti

Method and system for optimizing use of retrieval augmented generation pipelines in generative artificial intelligence applications

US12405978B2

( en )

2023-05-04

2025-09-02

Vijay Madisetti

Method and system for optimizing use of retrieval augmented generation pipelines in generative artificial intelligence applications

US12619501B2

( en )

2023-05-04

2026-05-05

Cohesity, Inc.

Data retrieval using embeddings for data in backup systems

US12468899B2

( en )

2023-05-08

2025-11-11

Adobe Inc.

Hallucination prevention for natural language insights

US20240386207A1

( en )

2023-05-18

2024-11-21

Noblis, Inc.

Systems and methods for detecting errors and hallucinations in generative model output data

US12625901B2

( en )

2023-05-23

2026-05-12

Palantir Technologies Inc.

Machine learning and language model-assisted geospatial data analysis and visualization

US12417356B2

( en )

2023-05-30

2025-09-16

Google Llc

Large-scale, privacy preserving personalized large language models (LLMs)

CN119091870A

( en )

2023-06-05

2024-12-06

åäº¬æç«¹å±ç½ç»ææ¯æéå¬å¸

Method for generating speech translation model, translation method and device

US12353457B2

( en )

2023-06-22

2025-07-08

Avalor Technologies, Ltd.

System and method for training a multi-tenant language model

US20250021468A1

( en )

2023-07-11

2025-01-16

Intuit Inc.

Controlling uncertain output by large language models

US12579609B2

( en )

2023-07-19

2026-03-17

Nvidia Corporation

High resolution input processing in a neural network

CN121941995A

( en )

2023-07-28

2026-04-28

D5Aiæéè´£ä»»å¬å¸

Interpretable adaptive artificial intelligence network

US12423364B2

( en )

2023-08-03

2025-09-23

Avid Technology, Inc.

Using artificial intelligence and machine learning to recommend supporting material for media projects

TWI902418B

( en )

2023-08-23

2025-10-21

åéåç´å¯¶è¡ä»½æéå¬å¸

Method and system for recommending question for association with content of information provider as answer to user query

US12608370B2

( en )

2023-08-28

2026-04-21

Wiz, Inc.

System and method for natural language query processing utilizing language model techniques

US12493615B2

( en )

2023-08-28

2025-12-09

Wiz, Inc.

System and method for improving efficiency in natural language query processing utilizing language model

US12001550B1

( en )

2023-08-28

2024-06-04

Wiz, Inc.

Cybersecurity incident response techniques utilizing artificial intelligence

US12572855B1

( en )

2023-08-28

2026-03-10

Dmitry Shapiro

Systems and methods for refining machine learning models based on user responses to prompts

US12424210B2

( en )

2023-08-28

2025-09-23

Amazon Technologies, Inc.

Natural language processing

US12322363B2

( en )

2023-08-31

2025-06-03

Aimi Inc.

Techniques for generating musical plan based on both explicit user parameter adjustments and automated parameter adjustments based on conversational interface

US20250077659A1

( en )

2023-08-31

2025-03-06

Dell Products L.P.

Managing challenges regarding impact of poisoned inferences on inference consumers

US20250077848A1

( en )

2023-09-05

2025-03-06

Nec Laboratories America, Inc.

Demonstration uncertainty-based artificial intelligence model for open information extraction

WO2025054413A1

( en )

2023-09-06

2025-03-13

Augmentir, Inc.

Automated generation of a scoped and verified response from a large language model to a user query

US20250086234A1

( en )

2023-09-11

2025-03-13

Orbsurgical Ltd.

Surgical System Leveraging Large Language Models

US12530112B2

( en )

2023-09-11

2026-01-20

Salesforce, Inc.

Generating virtual space headers utilizing machine-learned models

US12093965B1

( en )

2023-09-13

2024-09-17

Dmitry Shapiro

Systems and methods for generating response records for individual users based on user responses to prompts

US20250103624A1

( en )

2023-09-25

2025-03-27

International Business Machines Corporation

Combinatorial prompting for large language models

US12462096B2

( en )

2023-09-26

2025-11-04

Dropbox, Inc.

Generating field objects for auto-populating fillable documents utilizing a large language model

CN116992006B

( en )

2023-09-26

2024-01-12

æ¦æ±çæ¨¡ç§æè¡ä»½æéå¬å¸

Chain type natural language interaction method and system driven by large language model

CN117336539B

( en )

2023-09-28

2024-05-14

åäº¬é£å¹³æºè½ç§ææéå¬å¸

Video script production method and system for short video IP (Internet protocol) construction

CN117056494B

( en )

2023-09-28

2024-01-23

è¾è®¯ç§æï¼æ·±å³ï¼æéå¬å¸

Open domain question and answer method, device, electronic equipment and computer storage medium

US11922143B1

( en )

2023-10-05

2024-03-05

Dmitry Shapiro

Systems and methods for providing a user interface that facilitates application development

US12316715B2

( en )

2023-10-05

2025-05-27

The Toronto-Dominion Bank

Dynamic push notifications

US12614025B2

( en )

2023-10-05

2026-04-28

Bank Of America Corporation

Authorship source analysis for large language models (LLM) using a distributed ledger

US20250117583A1

( en )

2023-10-06

2025-04-10

The Hong Kong University Of Science And Technology

Knowledge-constrained tree search decoding with token-level hallucination detection

US20250117381A1

( en )

2023-10-10

2025-04-10

Google Llc

Utilizing large language model (llm) in responding to multifaceted queries

CN117874179B

( en )

2023-11-02

2024-06-04

çµæäºç¢³(åäº¬)ç§ææéå¬å¸

CCER intelligent question answering method and device, electronic equipment and storage medium

WO2025096269A1

( en )

2023-11-03

2025-05-08

Meta Platforms, Inc.

Methods and systems to improve post opportunities

US20250148501A1

( en )

2023-11-03

2025-05-08

Meta Platforms, Inc.

Methods and systems to improve post opportunities

EP4550168A1

( en )

2023-11-06

2025-05-07

Amadeus S.A.S.

Search request processing

US12561314B2

( en )

2023-11-17

2026-02-24

Goldman Sachs & Co. LLC

Retrieval-augmented generation (RAG) system optimization

CN117592483B

( en )

2023-11-21

2024-05-28

åè¥å·¥ä¸å¤§å¦

Implicit emotion analysis method and device based on thinking tree

US12511497B1

( en )

2023-11-30

2025-12-30

Amazon Technologies, Inc.

Embedding-based large language model tuning

KR102642981B1

( en )

2023-12-11

2024-03-05

ì£¼ìíì¬ ë¯¸ëë ì½ë

Method for providing chat-bot service and user terminal therefor

US20250225561A1

( en )

2024-01-08

2025-07-10

Motorola Mobility Llc

Compatible product recommendations

US20250225337A1

( en )

2024-01-09

2025-07-10

Google Llc

Detection of hallucinations in large language model responses

WO2025155426A1

( en )

2024-01-16

2025-07-24

Apple Inc.

Providing a suitability prompt to evaluate and improve the output of a generative model without fine tuning

US20250245665A1

( en )

2024-01-29

2025-07-31

Actimize Ltd

Fraud risk analysis system incorporating a large language model

US12452126B2

( en )

2024-02-13

2025-10-21

T-Mobile Usa, Inc.

Provisioning flow troubleshooting tool

US20250265345A1

( en )

2024-02-15

2025-08-21

Cisco Technology, Inc.

Vulnerability Defense System for Large Language Models

US12572756B2

( en )

2024-02-22

2026-03-10

Microsoft Technology Licensing, Llc

Discovery and selection of content based on language model token restrictions

US20250278433A1

( en )

2024-03-04

2025-09-04

Microsoft Technology Licensing, Llc

Personalized input suggestions for query interfaces

US12254005B1

( en )

2024-03-29

2025-03-18

nference, inc.

Systems and methods for retrieving patient information using large language models

US12602541B2

( en )

2024-04-24

2026-04-14

Ebay Inc.

Minimizing large language model hallucinations in generated summaries

US12242994B1

( en )

2024-04-30

2025-03-04

People Center, Inc.

Techniques for automatic generation of reports based on organizational data

WO2025244883A1

( en )

2024-05-20

2025-11-27

System, Inc.

Systems and methods for improving accuracy of large language models

US12541488B2

( en )

2024-05-23

2026-02-03

AtomBeam Technologies Inc.

System and method for generating thoughts with large language models using codewords

US20250363295A1

( en )

2024-05-23

2025-11-27

Ema Unlimited Inc.

Systems and methods for using large language model(s) to write, edit, and rewrite coherent content items

US12271696B1

( en )

2024-05-23

2025-04-08

AtomBeam Technologies Inc.

System and method for training and operating large language models using codewords

WO2025250446A1

( en )

2024-05-29

2025-12-04

Massachusetts Institute Of Technology

Large language model with exact arithmetic

US20250371611A1

( en )

2024-05-31

2025-12-04

Intuit Inc.

Graph based technique for automatically identifying a transactions table within a document

WO2026003510A1

( en )

2024-06-24

2026-01-02

Unlikely Artificial Intelligence Limited

Computer implemented methods for the automated analysis or use of data, and related systems

US12591622B2

( en )

2024-07-05

2026-03-31

National Potato Promotion Board

Methods and systems for monitoring and replying to social media content

<tr itemprop="forwardRef

… the automated analysis or use of data, including use of a large language model — Unlikely Artificial Intelligence Limited (US12353827B2)

Related documents