Conceptio › Archive › Google Patents

Google Patentsopen access

… the automated analysis or use of data, including use of a large language model — Unlikely Artificial Intelligence Limited (US12008333B2)

Unlikely Artificial Intelligence Limited · Google Patents

Google Patents · Patents · License: Open Access

pedoe unlikelyartificialintelligencelimited williamtunstall

patent, google patents, intellectual property, US12008333B2, Unlikely Artificial Intelligence Limited, William Tunstall-Pedoe, en, 2024

ABSTRACT

Abstract

Methods are provided, such as a method of interacting with a large language model (LLM), including the step of a processing system using a structured, machine-readable representation of data that conforms to a machine-readable language, such as a universal language, to provide new context data for the LLM, in order to improve the output, such as continuation text output, generated by the LLM in response to a prompt; and such as a method of interacting with a LLM, including the step of providing continuation data generated by the LLM to a processing system that uses a structured, machine-readable representation of data that conforms to a machine-readable language, such as a universal language, in which the processing system is configured to analyse the continuation output generated by the LLM in response to a prompt to enable an improved version of that continuation output to be provided to a user. Related computer systems are provided.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This is a continuation of U.S. application Ser. No. 18/301,639, filed on Apr. 17, 2023, which is a continuation of International Application No. PCT/GB2023/050405, filed on Feb. 22, 2023, which claims priority to GB Application No. GB2202347.7, filed on Feb. 22, 2022; GB Application No. GB2219268.6, filed on Dec. 20, 2022; GB Application No. GB2300624.0, filed on Jan. 16, 2023; and GB Application No. GB2302085.2, filed on Feb. 14, 2023, and application Ser. No. 18/301,639 is a continuation-in-part of U.S. application Ser. No. 18/001,368, filed on Dec. 9, 2022, which claims priority to International Application No. PCT/GB2021/052196, filed on Aug. 24, 2021, the entire contents of each of which being fully incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The field of the invention relates to computer implemented methods for the automated analysis or use of data, including use of a large language model (LLM), and to related computer implemented methods and systems.

2. Technical Background

Natural language (NL) is language evolved for humans such as the English language. Although significant advances have been made in computers' ability to process natural language, computers are still not able to deeply understand the meaning of natural language and use that meaning internally.

For this reason most computer applications typically use structured data to store information that they need for processingâe.g. a relational database: designing the schema, populating the database and writing code to process the fields in the database.

Use of structured data can work well if the application has limited requirements for the type of data required. However, some applications naturally require an extremely broad, heterogeneous collection of data to work well. This means that the schema required would have to be enormous, making building and coding for such an application impractical. We refer to such applications herein as HUB applications (Heterogeneous and Unreasonably Broad).

Examples of HUB applications include an application for managing a person's general health data where there are thousands of tests, thousands of medical conditions and thousands of symptoms. Another related application could be a nutrition tracking application where there are many thousands of substances and foods that can be ingested, each with different metabolic effects on the body.

Another example is an application to match the rÃ©sumÃ© of potential candidates with a job specification: in principle such an application would need structured data to represent every skill that might be of value to any role, every type of experience, every type of previous job.

Accounting is another application where vast heterogeneous data would be valuable: the perfect accounting application would represent every type of contract, every type of service.

In practice some of these applications, where they exist, work with a limited schema that doesn't cover the full range of their ideal properties. Health applications for example, typically work like this ignoring many types of data that they do not cover and instead end up being narrowâlimiting the application to only certain verticals within health.

Applications may also use natural language or augment a limited schema with natural languageâsuch as with current rÃ©sumÃ© matching applications which might represent a few key skills in a structured form but rely largely on keyword searching or statistical natural language processing (NLP) techniques on written rÃ©sumÃ©s otherwise.

In the case of accounting, transactions are represented with limited structured dataâdebits and credits on virtual ledgers with natural language names. The meaning of the natural language names and thus what these transactions represent is generally opaque to the application. Virtual ledgers often group different types of transaction together but fail to represent semantic differences which may be important.

There is no exact threshold for when an application becomes a HUB application but the difficulty of building an application with a hand created schema grows more than linearly with the number of tables as managing these tables as well as the code that maintains them becomes increasingly difficult to do.

These issues could be addressed if there existed a language or way of representing data that computers could fully process and understand but that also had an extremely broad scope.

In conventional Artificial Intelligence (AI), statistical Machine Learning (ML)âparticularly Deep Learning (DL)âhas been widely used. This has provided significant progress with many problems. Despite the progress, the results cannot be explained in a way that makes sense to human users as the solutions are the result of a calculation that may involve billions of weights. It can also be argued that the system lacks ârealâ understanding of the data or at least that the understanding is very different from the way a human user would understand it. The statistical nature of the technology also means that it is inherently unreliable with weights that are initiated randomly and then improved through a process of training where they are exposed to data. For complex problems, this improvement (learning) generally results in a model that works only a percentage of the time and is unable to reliably know when the result it produces is accurate.

Much processing of natural language is now done with DL. One example of DL applied to language is large language models (LLMs). LLMs are usually very large deep learning models potentially containing hundreds of billions of parameters and trained on tens of terabytes of text. Although they can generate explanations, those explanations are not derived from what has happened within the model. The text that LLMs produce is frequently incorrect and often describes things that are not true.

3. Discussion of Related Art

The Wikipedia page about Cyc, of 18 Jul. 2019, states that Cyc is the world's longest-lived artificial intelligence project, attempting to assemble a comprehensive ontology and knowledge base that spans the basic concepts and ârules of thumbâ about how the world works (think common sense knowledge but focusing more on things that rarely get written down or said, in contrast with facts one might find somewhere on the internet or retrieve via a search engine or Wikipedia), with the goal of enabling AI applications to perform human-like reasoning and be less âbrittleâ when confronted with novel situations that were not preconceived.

The Wikipedia page about Cyc, of 18 Jul. 2019, states that the objective of the Cyc project was to codify, in machine-usable form, the millions of pieces of knowledge that compose human common sense; this entailed, along the way, (1) developing an adequately expressive representation language, CycL, (2) developing an ontology spanning all human concepts down to some appropriate level of detail, (3) developing a knowledge base on that ontological framework, comprising all human knowledge about those concepts down to some appropriate level of detail, and (4) developing an inference engine exponentially faster than those used in then-conventional expert systems, to be able to infer the same types and depth of conclusions that humans are capable of, given their knowledge of the world.

The Wikipedia page about Cyc, of 18 Jul. 2019, states that most of Cyc's knowledge, outside math and games, is only true by default; for example, Cyc knows that as a default parents love their children, when you're made happy you smile, taking your first step is a big accomplishment, when someone you love has a big accomplishment that makes you happy, and only adults have children; when asked whether a picture captioned âSomeone watching his daughter take her first stepâ contains a smiling adult person, Cyc can logically infer that the answer is Yes, and âshow its workâ by presenting the step by step logical argument using those five pieces of knowledge from its knowledge base: these are formulated in the language CycL, which is based on predicate calculus and has a syntax similar to that of the Lisp programming language.

The Wikipedia page about Cyc, of 18 Jul. 2019, states that the Cyc project has been described as âone of the most controversial endeavors of the artificial intelligence historyâ; Catherine Havasi, CEO of Luminoso, says that Cyc is the predecessor project to IBM's Watson; machine-learning scientist Pedro Domingos refers to the project as a âcatastrophic failureâ for several reasons, including the unending amount of data required to produce any viable results and the inability for Cyc to evolve on its own; Robin Hanson, a professor of economics at George Mason University, gives a more balanced analysis: âOf course the CYC project is open to criticism on its many particular choices. People have complained about its logic-like and language-like representations, about its selection of prototypical cases to build from (e.g., encyclopedia articles), about its focus on answering over acting, about how often it rebuilds vs. maintaining legacy systems, and about being private vs. publishing everything. But any large project like this would produce such disputes, and it is not obvious any of its choices have been seriously wrong. They had to start somewhere, and in my opinion they have now collected a knowledge base with a truly spectacular size, scope, and integration. Other architectures may well work better, but if knowing lots is anywhere near as important as Lenat thinks, I'd expect serious AI attempts to import CYC's knowledge, translating it into a new representation. No other source has anywhere near CYC's size, scope, and integration.â

The True Knowledge system provided open-domain question answering using structured knowledge and inference. In the True Knowledge system, knowledge in the knowledge base was represented in a single unified format: named relations between pairs of named entities referred to as âfacts.â Facts, and the relations themselves, were first-class entities so facts about facts and facts about the properties of relations were fully supported (Tunstall-Pedoe, W. (2010). True Knowledge: Open-Domain Question Answering Using Structured Knowledge and Inference. AI Magazine, 31(3), 80-92. https://ojs.aaai.org/aimagazine/index.php/aimagazine/article/view/2298).

EP2527990A2 and EP2527990B1 disclose a method of using large language models in machine translation in which a translation model is partitioned into a plurality of language model partitions stored on a plurality of different language model servers. Segments of text are distributed to the servers for translation according to server workload. The language model information is usually collected by computing the frequency of occurrence of sequences of words in a large training corpus of documents.

SUMMARY OF THE INVENTION

According to a first aspect of the invention, there is provided a method of interacting with a LLM, including the step of a processing system using a structured, machine-readable representation of data that conforms to a machine-readable language, such as a universal language, to provide new context data for the LLM, in order to improve the output, such as continuation text output, generated by the LLM in response to a prompt.

According to a second aspect of the invention, there is provided a method of interacting with a LLM, including the step of providing continuation data generated by the LLM to a processing system that uses a structured, machine-readable representation of data that conforms to a machine-readable language, such as a universal language, in which the processing system is configured to analyse the continuation output (e.g. text output) generated by the LLM in response to a prompt to enable an improved version of that continuation output to be provided to a user. An example is shown in FIG. 10 . An advantage is that improved continuation output to that generated by the LLM may be provided.

According to a third aspect of the invention, there is provided a method of interacting with a LLM, including the step of generating continuation data by an LLM, by the LLM using the output of a processing system that uses a structured, machine-readable representation of data that conforms to a machine-readable language, such as a universal language, in which the LLM is configured to use the output of the processing system as a prompt and to generate a continuation output (e.g. text output) that is an improved version of the output from the processing system. An advantage is that an improved version of the output from the processing system may be generated by the LLM.

According to a fourth aspect of the invention, there is provided a method of interacting with a LLM, including the step of training the LLM on the output from a processing system using a structured, machine-readable representation of data that conforms to a machine-readable language, such as a universal language. An advantage is that an improved LLM may be provided.

According to a fifth aspect of the invention, there is provided a method for interacting with one or more users in natural language comprising the steps of:

providing a structured representation of data and a system operable to process language with reference to the structured representation of data to provide additional useful text relevant to the language; providing an LLM operable to provide a continuation of text; receiving a language input from one or more users; processing the language input with the system to produce additional useful text; providing the LLM with the language input plus the additional useful text; providing the one or more users with language derived from a continuation generated by the LLM. An advantage is that additional useful text to that provided by a system operable to process language with reference to the structured representation of data may be provided by a continuation generated by the LLM.

According to a sixth aspect of the invention, there is provided a method of validating natural language for factual accuracy comprising the steps of:

Extracting from the natural language a collection of one or more factual assertions asserted within the natural language; Checking the one or more factual assertions for factual accuracy; Outputting, or making available, the results of the checking.

An advantage is validating natural language for factual accuracy.

According to a seventh aspect of the invention, there is provided a method of learning knowledge in structured form comprising the steps of:

Providing an LLM trained on a large corpus of natural language documents; Providing the LLM with a prompt that will elicit desired knowledge; Eliciting a continuation of the prompt from the LLM; Translating the continuation into the structured form.

An advantage is learning knowledge in structured form.

According to an eighth aspect of the invention, there is provided a method for ensuring that an LLM generates original text comprising the steps of:

Providing a database of previous text that it should not generate; Performing a beam sear

CROSS-REFERENCE TO RELATED APPLICATIONS

BACKGROUND OF THE INVENTION

1. Field of the Invention

2. Technical Background

Accounting is another application where vast heterogeneous data would be valuable: the perfect accounting application would represent every type of contract, every type of service.

These issues could be addressed if there existed a language or way of representing data that computers could fully process and understand but that also had an extremely broad scope.

3. Discussion of Related Art

SUMMARY OF THE INVENTION

According to a fifth aspect of the invention, there is provided a method for interacting with one or more users in natural language comprising the steps of:

According to a sixth aspect of the invention, there is provided a method of validating natural language for factual accuracy comprising the steps of:

An advantage is validating natural language for factual accuracy.

According to a seventh aspect of the invention, there is provided a method of learning knowledge in structured form comprising the steps of:

An advantage is learning knowledge in structured form.

According to an eighth aspect of the invention, there is provided a method for ensuring that an LLM generates original text comprising the steps of:

Providing a database of previous text that it should not generate; Performing a beam search; Checking potential continuations against the database; When a potential continuation matches non-original text in the database, adjusting the continuation to no longer match non-original text in the database.

An advantage is avoiding copyright infringement.

According to a ninth aspect of the invention, there is provided a method of avoiding hallucination in an LLM comprising the steps of:

Analysing different continuations for a given prompt; Comparing the continuations to identify parts of the response that do not vary and parts of the response that do vary; Identifying the parts that vary as more likely to be hallucination; Replacing or not showing the parts identified as hallucination to the user.

An advantage is avoiding hallucination in an LLM.

According to a tenth aspect of the invention, there is provided a method for adding citations to text generated by an LLM comprising the steps of:

Providing a database of text used to train the LLM which includes sources associated with each section of text stored; checking sections of the continuation generated by the LLM against the database; retrieving sources where the sections of the continuation match text contained within the database. An advantage is that citations are added to text generated by an LLM.

According to an eleventh aspect of the invention, there is provided a computer-implemented method including the steps of

(i) Receiving output from a natural language processing computer process, the output including an answer to a question; (ii) Inputting the received output to a large language model (LLM); (iii) The large language model (LLM) processing the input to the LLM, to generate output based on the input to the LLM. (iv) Storing the output based on the input to the LLM.

An advantage is that an improved answer to the question may be provided by the LLM output.

According to a twelfth aspect of the invention, there is provided a computer-implemented method of training a large language model (LLM), including the steps of

(i) receiving output from a natural language processing computer process, the output including an answer to a question; (ii) repeating step (i) at least one thousand times using a set of at least one thousand different questions, and storing the answers to the questions in a training file; (iii) using the training file to train a large language model (LLM); (iv) storing weights characterizing the trained LLM.

An advantage is that a trained LLM with training improved by the use of the training file is provided.

According to a thirteenth aspect of the invention, there is provided a computer-implemented method of generating a training file for a large language model (LLM), including the steps of

According to a fourteenth aspect of the invention, there is provided a computer-implemented method of re-training a large language model (LLM), the LLM having been previously trained using a training file, the method including the steps of

(i) receiving output from a natural language processing computer process, the output including an answer to a question; (ii) repeating step (i) at least one thousand times using a set of at least one thousand different questions, and storing the answers to the questions in a re-training file; (iii) combining the training file and the re-training file, to generate a combined training file; (iv) using the combined training file to re-train the large language model (LLM); (v) storing weights characterizing the re-trained LLM.

An advantage is that an LLM with an improved training is provided.

According to a fifteenth aspect of the invention, there is provided a computer-implemented method of updating the training of a large language model (LLM), the LLM having been previously trained using a training file, the method including the steps of

(i) receiving output from a natural language processing computer process, the output including an answer to a question; (ii) repeating step (i) at least one thousand times using a set of at least one thousand different questions, and storing the answers to the questions in a re-training file; (iii) using the re-training file to update the training of the large language model (LLM); (iv) storing weights characterizing the LLM with the updated training. An advantage is that an LLM with an improved training is provided.

According to a sixteenth aspect of the invention, there is provided a computer-implemented method of improving output using an LLM, including the steps of

(i) receiving a natural language question (e.g. any natural language to which a natural language response is appropriate. It could be for example a command or a request for data or even some kind of social interaction or discussion); (ii) using a computer system including a processor and a memory, the processor configured to use a processing language in which semantic nodes are represented in the processing language, the semantic nodes including semantic links between semantic nodes wherein the semantic links are themselves semantic nodes, in which each semantic node denotes one specific meaning, in which a combination of semantic nodes defines a semantic node, in which expressions in the processing language may be nested, in which the natural language question is represented in the processing language, in which reasoning steps are represented in the processing language to represent semantics of the reasoning steps, in which computation units are represented in the processing language, wherein the memory is configured to store the representations in the processing language; (iii) the processor answering the question using the reasoning steps, the computation units and the semantic nodes, and (iv) inputting the natural language question, and the processor's answer to the question to the LLM; (v) the large language model (LLM) processing the input to the LLM, to generate output based on the input to the LLM; (vi) storing the output based on the input to the LLM.

An advantage is that an improved answer to the question may be provided by the LLM output.

According to a seventeenth aspect of the invention, there is provided a computer-implemented method of improving output from an LLM, including the steps of

(i) receiving a first natural language question; (ii) Inputting or providing the received first natural language question to a large language model (LLM); (iii) The large language model (LLM) processing the input to the LLM, to generate output based on the input to the LLM; (iv) translating the output into a processing language in which semantic nodes are represented in the processing language, the semantic nodes including semantic links between semantic nodes wherein the semantic links are themselves semantic nodes, in which each semantic node denotes one specific meaning, in which a combination of semantic nodes defines a semantic node, in which expressions in the processing language may be nested; (v) constructing a question which asks if the output is true, in which the question is represented in the processing language; (vi) inputting the question to a computer system including a processor and a memory, the processor configured to use the processing language, in which reasoning steps are represented in the processing language to represent semantics of the reasoning steps, in which computation units are represented in the processing language, wherein the memory is configured to store the representations in the processing language; (vii) the processor answering the question using the reasoning steps, the computation units and the semantic nodes, and (viii) the processor storing an answer to the question in the memory.

An advantage is that the LLM output is checked for accuracy.

According to an eighteenth aspect of the invention, there is provided a computer-implemented method of fact checking output from a large language model (LLM), including the steps of

(i) receiving a text input; (ii) inputting or providing the received text input to a large language model (LLM); (iii) the large language model (LLM) processing the input to the LLM, to generate output based on the input to the LLM; (iv) translating the output into a processing language in which semantic nodes are represented in the processing language, the semantic nodes including semantic links between semantic nodes wherein the semantic links are themselves semantic nodes, in which each semantic node denotes one specific meaning, in which a combination of semantic nodes defines a semantic node, in which expressions in the processing language may be nested, wherein translating the output includes extracting the assertions in text generated by the LLM; (v) constructing one or more questions which ask if the extracted assertions are individually true, in which the one or more questions are represented in the processing language; (vi) inputting the one or more questions to a computer system including a processor and a memory, the processor configured to use the processing language, in which reasoning steps are represented in the processing language to represent semantics of the reasoning steps, in which computation units are represented in the processing language, wherein the memory is configured to store the representations in the processing language; (vii) the processor answering the one or more questions using the reasoning steps, the computation units and the semantic nodes, and (viii) the processor storing an individual answer to each of the one or more questions in the memory. An advantage is fact checking output from a large language model.

According to a nineteenth aspect of the invention, there is provided a computer-implemented method of improving output from a computer system including a processor and a memory, the system operable to process the natural language question using a processing language in which semantic nodes are represented in the processing language, the method including using an LLM, the method including the steps of

(i) receiving a natural language question; (ii) using a computer system including a processor and a memory, the system operable to process the natural language question using a processing language in which semantic nodes are represented in the processing language, the semantic nodes including semantic links between semantic nodes wherein the semantic links are themselves semantic nodes, in which each semantic node denotes one specific meaning, wherein the memory is configured to store the representations in the processing language; (iii) the processor answering the question, and (iv) inputting the natural language question, and the processor's answer to the question to the LLM; (v) the large language model (LLM) processing the input to the LLM, to generate output based on the input to the LLM; (vi) Storing the output based on the input to the LLM.

An advantage is that output from the computer system including a processor and a memory, the system operable to process the natural language question using a processing language in which semantic nodes are represented in the processing language, is improved using the LLM.

According to a twentieth aspect of the invention, there is provided a computer-implemented method of improving output from an LLM, including the steps of

(i) receiving a first natural language question; (ii) inputting the received first natural language question to a large language model (LLM); (iii) the large language model (LLM) processing the input to the LLM, to generate output based on the input to the LLM; (iv) translating the output into a processing language in which semantic nodes are represented in the processing language, the semantic nodes including semantic links between semantic nodes wherein the semantic links are themselves semantic nodes, in which each semantic node denotes one specific meaning; (v) constructing a question which asks if the output is true, in which the question is represented in the processing language; (vi) inputting the question to a computer system including a processor and a memory, the processor configured to use the processing language, wherein the memory is configured to store the representations in the processing language; (vii) the processor answering the question, and (viii) the processor storing an answer to the question in the memory. An advantage is that the LLM output is checked for accuracy.

According to a 21st aspect of the invention, there is provided a computer-implemented method of fact checking output from a large language model (LLM), including the steps of

(i) receiving a text input; (ii) inputting the received text input to a large language model (LLM); (iii) the large language model (LLM) processing the input to the LLM, to generate output based on the input to the LLM; (iv) translating the output into a processing language in which semantic nodes are represented in the processing language, the semantic nodes including semantic links between semantic nodes wherein the semantic links are themselves semantic nodes, in which each semantic node denotes one specific meaning, wherein translating the output includes extracting the assertions in text generated by the LLM; (v) constructing one or more questions which ask if the extracted assertions are individually true, in which the one or more questions are represented in the processing language; (vi) inputting the one or more questions to a computer system including a processor and a memory, the processor configured to use the processing language, wherein the memory is configured to store the representations in the processing language; (vii) the processor answering the one or more questions, and (viii) the processor storing an individual answer to each of the one or more questions in the memory.

An advantage is fact checking output from a large language model.

According to a 22nd aspect of the invention, there is provided a computer implemented method of augmenting search results, including the steps of:

(i) receiving a search query; (ii) an (e.g. internet) search engine processing the search query to generate (e.g. internet) search results; (iii) transforming the search results into a structured, machine-readable representation of data that conforms to a machine-readable language, such as a universal language; (iv) processing the transformed search results using a processing system which uses the structured, machine-readable representation of data that conforms to the machine-readable language, such as a universal language, to produce output; (v) supplying the output of step (iv) as input to a large language model (LLM), and the LLM generating continuation data output in response to the input. An advantage is that search results found by a search engine are augmented.

According to a 23rd aspect of the invention, there is provided a method of fact-checking the output of a large language model (LLM), including the step of providing continuation data generated by the LLM to a processing system that uses a structured, machine-readable representation of data that conforms to a machine-readable language, in which the processing system analyses the continuation output generated by the LLM in response to a prompt, to enable an improved, fact-checked version of that continuation output to be generated and provided to a user. An advantage is fact checking output from a large language model.

Further aspects of the invention are given in the dependent Claims, and in the independent Claims. Aspects of the invention may be combined with each other, or with any of the Concepts described herein, or with any of the Claims.

BRIEF DESCRIPTION OF THE FIGURES

Aspects of the invention will now be described, by way of example(s), with reference to the following Figures, in which:

FIG. 1 shows example screen output for a notification of a job match.

FIG. 2 shows example screen output for a description of a job match.

FIG. 3 shows an example conversation within an app where nutritional data is being communicated with the app.

FIG. 4 shows some example insights that can be derived from a period of time where horizontal health and nutrition data was shown.

FIG. 5 shows an example graph showing daily calories in versus calories out which is an extremely common thing for someone to track if the user is aiming to lose (or gain) weight.

FIG. 6 shows an example of a visualisation capable of being generated from an example of the present invention: it is comparing estimated caffeine in the user's body at the time they went to bed with a calculation of sleep quality.

FIG. 7 shows examples of explanations generated. (a) shows examples of simplified explanations; (b) shows an example of a detailed explanation.

FIG. 8 shows an example of a voice assistant product, referred to herein as âBrianâ and how it fits in with the wider UL platform and other applications built on the UL platform.

FIG. 9 shows an alternative to the example of FIG. 8 .

FIG. 10 shows an example of a method of interacting with a LLM, including the step of providing continuation data generated by the LLM to a processing system that uses a structured, machine-readable representation of data that conforms to a machine-readable language, such as a universal language, in which the processing system is configured to analyse the continuation output (e.g. text output) generated by the LLM in response to a prompt to enable an improved version of that continuation output to be provided to a user.

FIG. 11 shows an example of an intermediate representation semantic graph.

DETAILED DESCRIPTION

This disclosure includes a computer implemented method for the automated analysis or use of data, which comprises the steps of:

(a) storing in a memory a structured, machine-readable representation of data that conforms to a machine-readable language (âmachine representationâ); the machine representation including representations of user speech or text input to a human/machine interface; (b) automatically processing the machine representations to analyse the user speech or text input.

This disclosure includes a computer-based system configured to analyse data, in which the system is configured to:

(a) store in a memory a structured, machine-readable representation of data that conforms to a machine-readable language; the structured, machine-readable representation of data including representations of user speech or text input to a human/machine interface; (b) automatically process the structured representations to analyse the user speech or text input to a human/machine interface.

These aspects of the disclosure may be implemented in a voice assistant or chatbot; the technical advantage achieved is to enable more extensive and faster scaling of the voice assistant or chatbot's capabilities; the invention enables voice assistants and chatbot to be able to answer a broader range of questions, and to do so more accurately; this makes it easier for the voice assistant or chatbot to work with large numbers of different natural languages.

One point of interpretation: the conjunction âorâ should not be narrowly construed to indicate mutual exclusivity but to cover inclusivity too. Hence, the phrase âuser speech or text inputâ means âuser speech on its own, or user text on its own, or both user speech and also user textâ. When the conjunction âorâ is meant exclusively, the phrase âeither . . . orâ is used.

The machine-readable language is extremely expressive yet also extremely simple; simplicity requires less computer processing and thus delivers faster performance. Further details are in the appended Concepts and Claims.

According to a further aspect of the disclosure, there is provided a computer system including a processor and a memory, the processor configured to answer a question, the processor configured to use a processing language in which semantic nodes are represented in the processing language, the semantic nodes including semantic links between semantic nodes wherein the semantic links are themselves semantic nodes, in which each semantic node denotes one specific meaning, in which a combination of semantic nodes defines a semantic node, in which expressions in the processing language may be nested, in which the question is represented in the processing language, in which reasoning steps are represented in the processing language to represent semantics of the reasoning steps, in which computation units are represented in the processing language, wherein the memory is configured to store the representations in the processing language, and wherein the processor is configured to answer the question using the reasoning steps, the computation units and the semantic nodes, and to store an answer to the question in the memory.

An advantage is that because semantic links between semantic nodes are themselves semantic nodes, semantic links and semantic nodes do not need to be processed in a distinctly different way, which simplifies processing, which speeds up response times, which is a technical effect.

An advantage is that because semantic nodes are very widely used in the processing language, processing of the processing language is speeded up, which speeds up response times, which is a technical effect.

The technical effect operates at the level of the architecture of the computer system; that is to say the effect is produced irrespective of the data being processed.

The technical effect results in the computer system being made to operate in a new way, because the computer finds answers to questions faster than using prior art approaches, because semantic nodes are very widely used in the processing language, which means that processing of the processing language is speeded up.

The processing language contributes to the technical character because it produces a technical effect: a processor processing the processing language has the effect that the computer finds answers to questions faster than using prior art approaches, because semantic nodes are very widely used in the processing language, which means that processing of the processing language is speeded up.

The computer system may be configured to output the answer to the question.

The computer system may be configured to output the answer to the question to a display device.

The computer system may be one wherein expressions in the processing language may be nested with no limit inherent to the processing language.

The computer system may be one wherein the semantic nodes each includes a unique identifier.

The computer system may be one wherein the computation units are semantic nodes.

The computer system may be one wherein the question is represented in the processing language with a passage comprising a semantic node that identifies the passage as a question, a list of zero, one or more semantic nodes representing unknown entities being asked about and at least one further passage which represents the semantics of the question in the context of the zero, one or more unknown entities.

The computer system may be one wherein the processing language is universal language.

The computer system may be one wherein the processing language is not a natural language.

The computer system may be one wherein the question relates to search and analysis of documents or web pages, wherein the semantic nodes include representations of at least parts of the documents or the web pages stored in a document store.

The computer system may be one wherein the question relates to a location-based search, using mapping data represented as semantic nodes in the processing language.

The computer system may be one wherein the question relates to a search for defined advertisements or news, wherein the semantic nodes include representations of advertisements, news articles or other information items.

The computer system may be one wherein the question relates to a request for a summary of a news topic, wherein the semantic nodes include representations of news from multiple sources, e.g. to provide a summary or aggregation of the news.

The computer system may be one wherein the question relates to a request for a compatibility match between persons, wherein the semantic nodes include representations of personal information defining one or more attributes of a person, for a plurality of people.

The computer system may be one wherein the question relates to compliance with requirements preventing abusive or illegal social media postings, wherein the semantic nodes include representations of social media postings.

The computer system may be one wherein the question relates to analysing customer reviews, wherein the semantic nodes include representations of customer reviews.

The computer system may be one wherein the question relates to a user's product request, wherein the semantic nodes include representations of product descriptions and user product requests.

The computer system may be one wherein the question relates to a job search, wherein the semantic nodes include representations of job descriptions and job applicants' skills and experience, to determine which job applicants match a job description, or to determine which job descriptions match a job applicant's skills and experience.

The computer system may be one wherein the question relates to health of an individual, wherein the semantic nodes include health data relating to the individual, and health data relating to human beings.

The computer system may be one wherein the question relates to nutrition, wherein the semantic nodes include nutritional data for foods and drinks.

The computer system may be one wherein the question relates to accounting or finance, wherein the semantic nodes include representations of financial or accounting information.

The computer system may be one wherein the question is received by a voice assistant or chatbot, wherein the semantic nodes include representations of user speech input to a human/machine interface and include representations of the human/machine interface itself.

According to a further aspect of the disclosure, there is provided a computer-implemented method, the method using a computer system including a processor and a memory, the processor configured to use a processing language in which semantic nodes are represented in the processing language, the semantic nodes including semantic links between semantic nodes wherein the semantic links are themselves semantic nodes, in which each semantic node denotes one specific meaning, in which a combination of semantic nodes defines a semantic node, in which expressions in the processing language may be nested, in which the question is represented in the processing language, in which reasoning steps are represented in the processing language to represent semantics of the reasoning steps, in which computation units are represented in the processing language, wherein the memory is configured to store the representations in the processing language, the method including the steps of:

(i) the processor answering the question using the reasoning steps, the computation units and the semantic nodes, and (ii) the processor storing an answer to the question in the memory.

Advantages include the advantages of the previous aspect of the disclosure.

The method may be one wherein the question is represented in the processing language with a passage comprising a semantic node that identifies the passage as a question, a list of zero, one or more semantic nodes representing unknown entities being asked about and at least one further passage which represents the semantics of the question in the context of the zero, one or more unknown entities.

The method may be one wherein the unknowns in the question are identified and the passage making up the body of the question is selected for further analysis; processing begins on a list of passages from the body of the question and the selected unknowns; a first passage in the list of passages is selected for processing; processing a single passage comprises three methods: using statically stored processing language passages, utilising computation units and utilising processing language generated from reasoning:

in which the first method is to lookup in the passage store if there are any passages that can be directly mapped with the passage being processed; if the passage is exactly the same structure as a passage in the passage store, with all nodes matching other than the unknowns, then the values the unknowns match against are valid results; the second method is to check if any results can be found by executing computation units; it is checked if this passage matches against any passages in a computation unit description; all non-unknown nodes in the passage being processed must match the same nodes in the corresponding position in the computation description or align with a computation input unknown; the unknowns being processed must align to output unknowns in the description; the computation unit is then called to get valid output values for the processed passage's unknowns; the third method is to see if this passage can be proved by applying any reasoning steps; reasoning steps are searched for where a passage in the second half of the reasoning passage can be unified with the passage being processed; all nodes and structure must be equal between the two passages, other than unknowns in the focus passage or the reasoning passage; if a reasoning passage like this is found it means that this reasoning step could be used to prove the passage being processed; a multi-stage process is used to first find any mappings for unknowns in the processed passage when matching with the reasoning passage; secondly, mappings for unknowns used in the reasoning passage are found by mapping with the passage being processed; this mapping can then be applied to the front half of the reasoning passage to generate a list of passages that, if they can be matched with known or generated processing language and mappings found for them, will prove and find valid mappings for the focus passage; solutions for the list of passages can then be found recursively.

The method may use a computer system of any aspect of the previous aspect of the disclosure.

Aspects of the disclosure may be combined.

Examples of the present invention include systems and methods for creating and using a structured representation of data that is aimed to be as expressive as natural language in terms of breadth but which also can be processed and understood by automated systems. The representation referred to herein as UL (for universal language) is a preferred example. Examples of the invention include systems and methods relating to specific HUB and other applications as well as systems

CLAIMS

Claims ( 23 )

The invention claimed is:

1. A computer-implemented method of automatically removing hallucinations from natural language text generated by a large language model (LLM), including the steps of:

(a) providing a prompt or query to the LLM;

(b) automatically generating a baseline response to the prompt or query, the baseline response including factual assertions;

(c) automatically generating one or more verification questions to test the accuracy of one or more of the factual assertions for factual accuracy or inaccuracy;

(d) systematically answering the or each verification questions in a manner that is not dependent on the baseline response;

(e) using the answers to the or each verification question to identify one or more factual inaccuracies or hallucinations present in the baseline response;

(f) automatically using the or each answer to the verification question or questions to generate a final natural language output, in which one or more factual inaccuracies or hallucinations present in the baseline response, have been removed.

2. The method of claim 1 in which there are multiple verification questions forming a sequence or chain of verification.

3. The method of claim 1 in which one or more of the verification questions do not include phrasing that matches the prompt or the baseline response.

4. The method of claim 1 in which one or more of the verification questions are simpler than the original prompt.

5. The method of claim 1 in which one or more of the verification questions are answered independently.

6. The method of claim 5 in which one or more of the verification questions do not include phrasing that matches the prompt or query, or the baseline response.

7. The method of claim 1 in which one or more of the verification questions are automatically answered as either true or false by the LLM.

8. The method of claim 1 in which one or more of the verification questions are open verification questions.

9. The method of claim 1 in which one or more of the verification questions are generated by the LLM.

10. The method of claim 9 in which the verification questions are generated by the LLM in response to a prompt.

11. The method of claim 1 in which the method includes the step of using explicit reasoning steps when cross-checking whether each of the answers to the verification questions indicate an inconsistency with the baseline response.

12. The method of claim 1 including the step of extracting from the baseline response a collection of one or more factual assertions by providing the LLM with the baseline response and a prompt asking the LLM to provide a continuation generating the factual assertions.

13. The method of claim 12 where a factual assertion derived from the baseline response is automatically translated into a verification question.

14. The method of claim 1 where, if a verification question is answered as false, then the related factual assertions derived from the draft natural language text is treated as false and is not included in the final natural language output.

15. The method of claim 1 where the sources of any previous known knowledge used to validate a factual assertion as false are returned as part of the process of checking one or more of the factual assertions for factual accuracy or inaccuracy.

16. The method of claim 1 where the checking the one or more factual assertions for factual accuracy or inaccuracy includes a further step of generating a natural language explanation of why a factual assertion is determined to be true or false.

17. The method of claim 1 in which the method includes the step of supplementing the information stored in the weights of the LLM by referencing a semantic representation of knowledge.

18. The method of claim 17 in which the semantic representation of knowledge is used to provide additional context to the LLM.

19. The method of claim 1 in which the method includes the step of supplementing the information stored in the weights of the LLM by referencing dynamic real-time information.

20. The method of claim 1 which includes the step of providing a reasoning system able to reason with a symbolic representation of the world, translating the verification questions or assertions into a representation compatible with the reasoning system and utilising the reasoning system to identify assertions which are true or false.

21. The method of claim 20 in which the reasoning system uses a universal language.

22. A computer implemented system configured to automatically remove hallucinations from natural language text generated by a LLM, the system being configured to:

(a) receive a prompt or query to the LLM;

(b) automatically generate a baseline response to the prompt or query, the baseline response including factual assertions;

(c) automatically generate one or more verification questions to test the accuracy of one or more of the factual assertions for factual accuracy or inaccuracy;

(d) systematically answer the or each verification questions in a manner that is not dependent on the baseline response;

(e) use the answers to the or each verification question to identify one or more factual inaccuracies or hallucinations present in the baseline response;

(f) automatically use the or each answer to the verification question or questions to generate a final natural language output, in which one or more factual inaccuracies or hallucinations present in the baseline response, have been removed.

23. A large language model system configured to automatically remove hallucinations from natural language text generated by the LLM system, the LLM system being configured to:

(a) receive a prompt or query;

(b) automatically generate a baseline response to the prompt or query, the baseline response including factual assertions;

(c) automatically generate one or more verification questions to test the accuracy of one or more of the factual assertions for factual accuracy or inaccuracy;

(d) systematically answer the or each verification questions in a manner that is not dependent on the baseline response;

(e) use the answers to the or each verification question to identify one or more factual inaccuracies or hallucinations present in the baseline response;

US18/515,488

2021-08-24

2023-11-21

Computer implemented methods for the automated analysis or use of data, including use of a large language model

Active

US12008333B2

( en )

Priority Applications (1)

Application Number

Priority Date

Filing Date

Title

US18/515,488

US12008333B2

( en )

2021-08-24

2023-11-21

Computer implemented methods for the automated analysis or use of data, including use of a large language model

Applications Claiming Priority (17)

Application Number

Priority Date

Filing Date

Title

PCT/GB2021/052196

WO2022043675A2

( en )

2020-08-24

2021-08-24

A computer implemented method for the automated analysis or use of data

GB2202347.7

2022-02-22

GB2202347

2022-02-22

GBGB2202347.7A

GB202202347D0

( en )

2022-02-22

Unlikely ai 2nd gen

US202218001368A

2022-12-09

GB2219268.6

2022-12-20

GBGB2219268.6A

GB202219268D0

( en )

2022-12-20

Unlikely ai 2nd gen ii

GB2219268

2022-12-20

GB2300624

2023-01-16

GBGB2300624.0A

GB202300624D0

( en )

2023-01-16

Unlikely ai 2nd gen iii

GB2300624.0

2023-01-16

GB2302085.2

2023-02-14

GBGB2302085.2A

GB202302085D0

( en )

2023-02-14

Unlikely AI 2nd gen IV

GB2302085

2023-02-14

PCT/GB2023/050405

WO2023161630A1

( en )

2022-02-22

2023-02-22

Computer implemented methods for the automated analysis or use of data, including use of a large language model

US18/301,639

US11977854B2

( en )

2021-08-24

2023-04-17

Computer implemented methods for the automated analysis or use of data, including use of a large language model

US18/515,488

US12008333B2

( en )

2021-08-24

2023-11-21

Computer implemented methods for the automated analysis or use of data, including use of a large language model

Related Parent Applications (1)

Application Number

Title

Priority Date

Filing Date

US18/301,639

Continuation

US11977854B2

( en )

2021-08-24

2023-04-17

Computer implemented methods for the automated analysis or use of data, including use of a large language model

Publications (2)

Publication Number

Publication Date

US20240095468A1

US20240095468A1 ( en )

2024-03-21

US12008333B2

true

US12008333B2 ( en )

2024-06-11

Family

ID=88195841

Family Applications (2)

Application Number

Title

Priority Date

Filing Date

US18/301,639

Active

US11977854B2

( en )

2021-08-24

2023-04-17

Computer implemented methods for the automated analysis or use of data, including use of a large language model

US18/515,488

Active

US12008333B2

( en )

2021-08-24

2023-11-21

Computer implemented methods for the automated analysis or use of data, including use of a large language model

Family Applications Before (1)

Application Number

Title

Priority Date

Filing Date

US18/301,639

Active

US11977854B2

( en )

2021-08-24

2023-04-17

Computer implemented methods for the automated analysis or use of data, including use of a large language model

Country Status (1)

Country

Link

( 2 )

US11977854B2

( en )

Families Citing this family (92)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

EP4200717A2

( en )

2020-08-24

2023-06-28

Unlikely Artificial Intelligence Limited

A computer implemented method for the automated analysis or use of data

US12423507B2

( en )

2021-07-12

2025-09-23

International Business Machines Corporation

Elucidated natural language artifact recombination with contextual awareness

US11989507B2

( en )

2021-08-24

2024-05-21

Unlikely Artificial Intelligence Limited

Computer implemented methods for the automated analysis or use of data, including use of a large language model

US12067362B2

( en )

2021-08-24

2024-08-20

Unlikely Artificial Intelligence Limited

Computer implemented methods for the automated analysis or use of data, including use of a large language model

US12073180B2

( en )

2021-08-24

2024-08-27

Unlikely Artificial Intelligence Limited

Computer implemented methods for the automated analysis or use of data, including use of a large language model

US11977854B2

( en )

2021-08-24

2024-05-07

Unlikely Artificial Intelligence Limited

Computer implemented methods for the automated analysis or use of data, including use of a large language model

US11989527B2

( en )

2021-08-24

2024-05-21

Unlikely Artificial Intelligence Limited

Computer implemented methods for the automated analysis or use of data, including use of a large language model

WO2023183914A1

( en )

2022-03-25

2023-09-28

OrangeDot, Inc.

Method and system for automatically determining responses in a messaging platform

US12292912B2

( en )

2022-03-25

2025-05-06

OrangeDot, Inc.

Method and system for intent-based action recommendations and/or fulfillment in a messaging platform

US12299404B2

( en )

2022-08-22

2025-05-13

Rohirrim, Inc.

Computer-generated content based on text classification, semantic relevance, and activation of deep learning large language models

US12518737B2

( en )

2022-12-08

2026-01-06

Google Llc

Structured description-based chatbot development techniques

WO2024145642A1

( en )

2022-12-30

2024-07-04

Theai, Inc.

Pre-fetching results from large language models

US20240256784A1

( en )

2023-01-31

2024-08-01

Microsoft Technology Licensing, Llc

Extensible chatbot framework

US20240296295A1

( en )

2023-03-03

2024-09-05

Microsoft Technology Licensing, Llc

Attribution verification for answers and summaries generated from large language models (llms)

US20240319965A1

( en )

2023-03-24

2024-09-26

Wevo, Inc.

Systems And Methods For Automating Analyses Of User Experience Tests With Generative Language Models

US20240378391A1

( en )

2023-05-12

2024-11-14

Servicenow, Inc.

Service Platform Integration with Generative Natural Language Models

US12142371B1

( en )

2023-05-15

2024-11-12

HealthGPT, Inc.

Low-latency conversational artificial intelligence (AI) architecture with a parallelized in-depth analysis feedback loop

US12597512B2

( en )

2023-05-15

2026-04-07

HealthGPT, Inc.

Real-time use of multiple parallel automatic speech recognition (ASR) modules in a conversational artificial intelligence (AI) architecture

US20240386207A1

( en )

2023-05-18

2024-11-21

Noblis, Inc.

Systems and methods for detecting errors and hallucinations in generative model output data

US12613928B2

( en )

2023-05-30

2026-04-28

Snowflake Inc.

Data listing search utilizing large language models

US12417356B2

( en )

2023-05-30

2025-09-16

Google Llc

Large-scale, privacy preserving personalized large language models (LLMs)

US20240411994A1

( en )

2023-06-07

2024-12-12

NEC Laboratories Europe GmbH

Extracting information from reports using large language models

US20240419912A1

( en )

2023-06-13

2024-12-19

Microsoft Technology Licensing, Llc

Detecting hallucination in a language model

US20250013873A1

( en )

2023-07-07

2025-01-09

Sri International

Method, apparatus and system for consistency enhanced large language models

US12511487B2

( en )

2023-07-26

2025-12-30

Micro Focus Llc

Augmented question and answer (Q and A) with large language models

CN117097790A

( en )

2023-08-08

2023-11-21

åäº¬åè·³ç½ç»ææ¯æéå¬å¸

Information pushing method and device, computer equipment and storage medium

US12424210B2

( en )

2023-08-28

2025-09-23

Amazon Technologies, Inc.

Natural language processing

US12572855B1

( en )

2023-08-28

2026-03-10

Dmitry Shapiro

Systems and methods for refining machine learning models based on user responses to prompts

US12204524B1

( en )

2023-08-29

2025-01-21

Quantiphi, Inc

Method and system for electronic processing of user queries maintaining factual consistency during processing

US12530112B2

( en )

2023-09-11

2026-01-20

Salesforce, Inc.

Generating virtual space headers utilizing machine-learned models

US20250086211A1

( en )

2023-09-12

2025-03-13

Bitvore Corp.

Grounding large language models using real-time content feeds and reference data

US12093965B1

( en )

2023-09-13

2024-09-17

Dmitry Shapiro

Systems and methods for generating response records for individual users based on user responses to prompts

US12531820B2

( en )

2023-09-29

2026-01-20

Amazon Technologies, Inc.

Knowledge graph assisted large language models

US11922143B1

( en )

2023-10-05

2024-03-05

Dmitry Shapiro

Systems and methods for providing a user interface that facilitates application development

US20250117593A1

( en )

2023-10-06

2025-04-10

Mcafee, Llc

Methods and apparatus to self-guardrail large language model responses

WO2025080599A1

( en )

2023-10-09

2025-04-17

Quabbin Patent Holdings

Computer-implemented methods and systems for dynamic prompt generation and integration with large language models for document revision

US12554924B2

( en )

2023-10-09

2026-02-17

Quabbin Patent Holdings, Inc.

Computer-implemented methods and systems for generative text painting

US12292896B1

( en )

2023-10-20

2025-05-06

Promoted.ai, Inc.

Multi-dimensional content organization and arrangement control in a user interface of a computing device

US12306842B1

( en )

2024-07-01

2025-05-20

Promoted.ai, Inc.

Within-context semantic relevance inference of machine learning model generated output

US12277162B1

( en )

2023-10-20

2025-04-15

Promoted.ai, Inc.

Using generative AI models for content searching and generation of confabulated search results

US12488050B2

( en )

2023-10-20

2025-12-02

Dropbox, Inc.

Using generative AI models for content searching and generation of confabulated search results

US20250139263A1

( en )

2023-10-25

2025-05-01

Varonis Systems Inc.

Object Security Control

WO2025096582A1

( en )

2023-10-31

2025-05-08

Aira Technologies, Inc.

Method and system for context-aware telecommunications, cellular, and radio based generative pre-trained transformer

WO2025097140A1

( en )

2023-11-03

2025-05-08

Sluice Ai, Inc.

Document generation system and method

US20250149033A1

( en )

2023-11-07

2025-05-08

Qsc, Llc

Artificial intelligence assistance for an audio, video and control system

WO2025111700A1

( en )

2023-11-28

2025-06-05

GroupBy Inc.

Fitment based product search and filtering

WO2025111916A1

( en )

2023-11-30

2025-06-05

Qualcomm Incorporated

Accelerating prompt inferencing of large language models

US12511497B1

( en )

2023-11-30

2025-12-30

Amazon Technologies, Inc.

Embedding-based large language model tuning

US12093658B1

( en )

2023-12-06

2024-09-17

Citibank, N.A.

Systems and methods for implementing an artificial intelligence-based solution for prompt orchestration by segregating processing requests into non-serialized tasks

US12572471B1

( en )

2025-02-12

2026-03-10

AtomBeam Technologies Inc.

Mobile-optimized multi-stage LLM with generalized thought caching

US12602549B1

( en )

2025-02-12

2026-04-14

AtomBeam Technologies Inc.

Persistent cognitive machine with curated long term memory

US12608631B1

( en )

2025-02-12

2026-04-21

AtomBeam Technologies Inc.

Mobile-optimized multi-stage LLM with autonomous reasoning

US12387050B1

( en )

2025-02-12

2025-08-12

AtomBeam Technologies Inc.

Multi-stage LLM with unlimited context

US12626167B2

( en )

2024-05-23

2026-05-12

AtomBeam Technologies Inc.

System and method for large language model with integrated memory during inference using manifold traversal architecture

US12579437B2

( en )

2024-05-23

2026-03-17

AtomBeam Technologies Inc.

Mobile-optimized multi-stage LLM with federated persistent cognitive architecture

US12367971B2

( en )

2023-12-18

2025-07-22

HealthGPT, Inc.

Low-latency conversational artificial intelligence (AI) architecture with a parallelized in-depth analysis feedback loop

CN118136231B

( en )

2023-12-29

2025-03-28

åäº¬å¤§å¦é¿æ²è®¡ç®ä¸æ°åç»æµç ç©¶é¢

Multi-role collaborative information processing method and device based on large language model

US20250232091A1

( en )

2024-01-16

2025-07-17

Halliburton Energy Services, Inc.

Large Language Model Driven Design Verifications And Reviews

US12001464B1

( en )

2024-01-19

2024-06-04

OneSource Solutions International, Inc

System and method for medical data governance using large language models

US20250245665A1

( en )

2024-01-29

2025-07-31

Actimize Ltd

Fraud risk analysis system incorporating a large language model

US12450168B2

( en )

2024-01-31

2025-10-21

International Business Machines Corporation

Dynamic updating of content addressable associative memories for large language models

US12452126B2

( en )

2024-02-13

2025-10-21

T-Mobile Usa, Inc.

Provisioning flow troubleshooting tool

US12499167B2

( en )

2024-03-06

2025-12-16

Ebay Inc.

Item retrieval using generative model output

US20250284719A1

( en )

2024-03-11

2025-09-11

Microsoft Technology Licensing, Llc

Machine cognition workflow engine with rewinding mechanism

WO2025221712A1

( en )

2024-04-17

2025-10-23

Elementum Ltd

Systems and methods for a data-driven workflow platform with artificial intelligence (ai) integration

KR20250156497A

( en )

2024-04-25

2025-11-03

ì¼ì±ì ìì£¼ìíì¬

Methof for removing hallucination in inference result of neural network model, and electronic apparatus for performing the same

US12619781B2

( en )

2024-04-30

2026-05-05

Microsoft Technology Licensing, Llc

Verbatim feedback processing for compliant retention

US20250349195A1

( en )

2024-05-10

2025-11-13

Adeia Guides Inc.

Customizable environmental interaction and response system for immersive device users

US20250362902A1

( en )

2024-05-22

2025-11-27

Charter Communications Operating, Llc

Methods and systems for automatically evaluating code and software documentation for compliance with relevant standards, requirements, and policies

US12572830B1

( en )

2024-05-23

2026-03-10

AtomBeam Technologies Inc.

System and method for enterprise hierarchical persistent cognitive machines with organizational hierarchy awareness and compliance integration

US12541488B2

( en )

2024-05-23

2026-02-03

AtomBeam Technologies Inc.

System and method for generating thoughts with large language models using codewords

US12271696B1

( en )

2024-05-23

2025-04-08

AtomBeam Technologies Inc.

System and method for training and operating large language models using codewords

EP4672025A1

( en )

2024-06-25

2025-12-31

Tata Consultancy Services Limited

Calculation of variability and confidence values for answers generated by large language models (LLMS).

WO2026010841A1

( en )

2024-07-03

2026-01-08

Yale University

Method, system, and product for parsing data into structured and formatted output data with iterative prompting

KR102916056B1

( en )

2024-07-15

2026-01-21

ì£¼ìíì¬ ë¸í

Technique for reducing hallucination answer of artificial intelligence based language model

US12347573B1

( en )

2024-09-03

2025-07-01

Sully.Ai

Artificial intelligence (AI) to create a patient visit note based on a conversation between a doctor and a patient

US12254966B1

( en )

2024-09-03

2025-03-18

Sully.Ai

Artificial intelligence (AI) to provide decision support insights including while a doctor is engaged in conversation with a patient

US12436966B1

( en )

2024-09-23

2025-10-07

Aravo Solutions, Inc.

Utilizing a large language model to generate a computing structure

US12423265B1

( en )

2024-09-23

2025-09-23

Aravo Solutions, Inc.

Prompting a large language model for vector embeddings and metadata to generate an indexed computing file

US20260086986A1

( en )

2024-09-23

2026-03-26

Aravo Solutions, Inc.

Utilizing a large language model to generate a computing structure

US12367213B1

( en )

2024-09-23

2025-07-22

Aravo Solutions, Inc.

Generating structured data from unstructured data using metadata from a large language model

US12399889B1

( en )

2024-09-23

2025-08-26

Aravo Solutions, Inc.

Modifying a database using a large language model

US20260094193A1

( en )

2024-09-30

2026-04-02

Google Llc

Systems and Methods for the Generation of a Comparative Data Structure using a Large Language Model

DE102024128971A1

( en )

2024-10-08

2026-04-09

Schaeffler Technologies AG & Co. KG

Human-machine interaction methods with validation

US12517936B1

( en )

2024-11-06

2026-01-06

Dropbox, Inc.

Retrieval-augmented generation and relevancy annotation to abort impact of irrelevant queries using generative artificial intelligence

US12499333B1

( en )

2025-02-06

2025-12-16

NewsGenie, Inc.

Autonomous journalism with artificial intelligence and sensory data processing

US12572748B1

( en )

2025-02-12

2026-03-10

AtomBeam Technologies Inc.

Scalable expert foundry system using hierarchical supervisory networks and geometric manifold architectures for multi-domain cognitive processing

US12373649B1

( en )

2025-02-12

2025-07-29

U.S. Bancorp, National Association

Remediating hallucinations in language models

US12585882B1

( en )

2025-02-12

2026-03-24

Atobeam Technologies Inc.

Evolutionary thought caching for multi-stage language model systems

US12368931B1

( en )

2025-04-04

2025-07-22

Lumana Inc.

Multimedia content management using reduced representations

US12619831B1

( en )

2025-04-14

2026-05-05

Charles DeLisi

Systems and methods for generating deliberative committees of age-stratified large language models

CN120722752B

( en )

2025-08-18

2025-11-11

åå·çè¾è¾¾çµåå¨æå¶é æéå¬å¸

A collaborative control optimization method and system for solar-powered acoustic and optical device groups based on the Internet of Things

Citations (241)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US4974191A

( en )

1987-07-31

1990-11-27

Syntellect Software Inc.

Adaptive natural language computer interface system

US5386556A

( en )

1989-03-06

1995-01-31

International Business Machines Corporation

Natural language analyzing apparatus and method

US6085186A

( en )

1996-09-20

2000-07-04

Netbot, Inc.

Method and system using information written in a wrapper description language to execute query on a network

US20020173971A1

( en )

2001-03-28

2002-11-21

Stirpe Paul Alan

System, method and application of ontology driven inferencing-based personalization systems

US20030130976A1

( en )

1998-05-28

2003-07-10

Lawrence Au

Semantic network methods to disambiguate natural language meaning

US20040054626A1

( en )

2002-09-17

2004-03-18

Adn Technologies

Device for processing local accounts data with different formats, equipment and a method for treating associated data

US20040078756A1

( en )

2002-10-15

2004-04-22

Napper Jonathon Leigh

Method of improving recognition accuracy in form-based data entry systems

US20040117189A1

( en )

1999-11-12

2004-06-17

Bennett Ian M.

Query engine for processing voice based queries including semantic decoding

US20040174976A1

( en )

2003-03-07

2004-09-09

Nortel Networks Limited

User controlled call routing for multiple telephony devices

US7085708B2

( en )

2000-09-23

2006-08-01

Ravenflow, Inc.

Computer system with natural language to machine language translator

US20070043708A1

( en )

2000-11-28

2007-02-22

Semscript Ltd.

Knowledge storage and retrieval system and method

US20070055656A1

( en )

2005-08-01

2007-03-08

Semscript Ltd.

Knowledge repository

US20070094224A1

( en )

1998-05-28

2007-04-26

Lawrence Au

Method and system for determining contextual meaning for network search applications

US7231633B2

( en )

2003-07-15

2007-06-12

Lsi Corporation

Debugging with set verbosity level during read and analysis of executable code and associated comments while logging according to set verbosity level

US20070136222A1

( en )

2005-12-09

2007-06-14

Microsoft Corporation

Question and answer architecture for reasoning and clarifying intentions, goals, and needs from contextual clues and content

US20070197882A1

( en )

2006-02-17

<td itemprop="publica

… the automated analysis or use of data, including use of a large language model — Unlikely Artificial Intelligence Limited (US12008333B2)

Related documents