ABSTRACT
Abstract
Methods are provided, such as a method of interacting with a large language model (LLM), including the step of a processing system using a structured, machine-readable representation of data that conforms to a machine-readable language, such as a universal language, to provide new context data for the LLM, in order to improve the output, such as continuation text output, generated by the LLM in response to a prompt; and such as a method of interacting with a LLM, including the step of providing continuation data generated by the LLM to a processing system that uses a structured, machine-readable representation of data that conforms to a machine-readable language, such as a universal language, in which the processing system is configured to analyse the continuation output generated by the LLM in response to a prompt to enable an improved version of that continuation output to be provided to a user. Related computer systems are provided.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
This is a continuation of International Application No. PCT/GB2023/050405, filed on Feb. 22, 2023, which claims priority to GB Application No. GB2202347.7, filed on Feb. 22, 2022; GB Application No. GB2219268.6, filed on Dec. 20, 2022; GB Application No. GB2300624.0, filed on Jan. 16, 2023; and GB Application No. GB2302085.2, filed on Feb. 14, 2023, and is a continuation-in-part of U.S. application Ser. No. 18/001,368, filed on Dec. 9, 2022, the entire contents of each of which being fully incorporated herein by reference.
BACKGROUND OF THE INVENTION
1. Field of the Invention
The field of the invention relates to computer implemented methods for the automated analysis or use of data, including use of a large language model (LLM), and to related computer implemented methods and systems.
2. Technical Background
Natural language (NL) is language evolved for humans such as the English language. Although significant advances have been made in computers' ability to process natural language, computers are still not able to deeply understand the meaning of natural language and use that meaning internally.
For this reason most computer applications typically use structured data to store information that they need for processingâe.g. a relational database: designing the schema, populating the database and writing code to process the fields in the database.
Use of structured data can work well if the application has limited requirements for the type of data required. However, some applications naturally require an extremely broad, heterogeneous collection of data to work well. This means that the schema required would have to be enormous, making building and coding for such an application impractical. We refer to such applications herein as HUB applications (Heterogeneous and Unreasonably Broad).
Examples of HUB applications include an application for managing a person's general health data where there are thousands of tests, thousands of medical conditions and thousands of symptoms. Another related application could be a nutrition tracking application where there are many thousands of substances and foods that can be ingested, each with different metabolic effects on the body.
Another example is an application to match the résumé of potential candidates with a job specification: in principle such an application would need structured data to represent every skill that might be of value to any role, every type of experience, every type of previous job.
Accounting is another application where vast heterogeneous data would be valuable: the perfect accounting application would represent every type of contract, every type of service.
In practice some of these applications, where they exist, work with a limited schema that doesn't cover the full range of their ideal properties. Health applications for example, typically work like this ignoring many types of data that they do not cover and instead end up being narrowâlimiting the application to only certain verticals within health.
Applications may also use natural language or augment a limited schema with natural languageâsuch as with current résumé matching applications which might represent a few key skills in a structured form but rely largely on keyword searching or statistical natural language processing (NLP) techniques on written résumé s otherwise.
In the case of accounting, transactions are represented with limited structured dataâdebits and credits on virtual ledgers with natural language names. The meaning of the natural language names and thus what these transactions represent is generally opaque to the application. Virtual ledgers often group different types of transaction together but fail to represent semantic differences which may be important.
There is no exact threshold for when an application becomes a HUB application but the difficulty of building an application with a hand created schema grows more than linearly with the number of tables as managing these tables as well as the code that maintains them becomes increasingly difficult to do.
These issues could be addressed if there existed a language or way of representing data that computers could fully process and understand but that also had an extremely broad scope.
In conventional Artificial Intelligence (AI), statistical Machine Learning (ML)âparticularly Deep Learning (DL)âhas been widely used. This has provided significant progress with many problems. Despite the progress, the results cannot be explained in a way that makes sense to human users as the solutions are the result of a calculation that may involve billions of weights. It can also be argued that the system lacks ârealâ understanding of the data or at least that the understanding is very different from the way a human user would understand it. The statistical nature of the technology also means that it is inherently unreliable with weights that are initiated randomly and then improved through a process of training where they are exposed to data. For complex problems, this improvement (learning) generally results in a model that works only a percentage of the time and is unable to reliably know when the result it produces is accurate.
Much processing of natural language is now done with DL. One example of DL applied to language is large language models (LLMs). LLMs are usually very large deep learning models potentially containing hundreds of billions of parameters and trained on tens of terabytes of text. Although they can generate explanations, those explanations are not derived from what has happened within the model. The text that LLMs produce is frequently incorrect and often describes things that are not true.
3. Discussion of Related Art
The Wikipedia page about Cyc, of 18 Jul. 2019, states that Cyc is the world's longest-lived artificial intelligence project, attempting to assemble a comprehensive ontology and knowledge base that spans the basic concepts and ârules of thumbâ about how the world works (think common sense knowledge but focusing more on things that rarely get written down or said, in contrast with facts one might find somewhere on the internet or retrieve via a search engine or Wikipedia), with the goal of enabling AI applications to perform human-like reasoning and be less âbrittleâ when confronted with novel situations that were not preconceived.
The Wikipedia page about Cyc, of 18 Jul. 2019, states that the objective of the Cyc project was to codify, in machine-usable form, the millions of pieces of knowledge that compose human common sense; this entailed, along the way, (1) developing an adequately expressive representation language, CycL, (2) developing an ontology spanning all human concepts down to some appropriate level of detail, (3) developing a knowledge base on that ontological framework, comprising all human knowledge about those concepts down to some appropriate level of detail, and (4) developing an inference engine exponentially faster than those used in then-conventional expert systems, to be able to infer the same types and depth of conclusions that humans are capable of, given their knowledge of the world.
The Wikipedia page about Cyc, of 18 Jul. 2019, states that most of Cyc's knowledge, outside math and games, is only true by default; for example, Cyc knows that as a default parents love their children, when you're made happy you smile, taking your first step is a big accomplishment, when someone you love has a big accomplishment that makes you happy, and only adults have children; when asked whether a picture captioned âSomeone watching his daughter take her first stepâ contains a smiling adult person, Cyc can logically infer that the answer is Yes, and âshow its workâ by presenting the step by step logical argument using those five pieces of knowledge from its knowledge base: these are formulated in the language CycL, which is based on predicate calculus and has a syntax similar to that of the Lisp programming language.
The Wikipedia page about Cyc, of 18 Jul. 2019, states that the Cyc project has been described as âone of the most controversial endeavors of the artificial intelligence historyâ; Catherine Havasi, CEO of Luminoso, says that Cyc is the predecessor project to IBM's Watson; machine-learning scientist Pedro Domingos refers to the project as a âcatastrophic failureâ for several reasons, including the unending amount of data required to produce any viable results and the inability for Cyc to evolve on its own; Robin Hanson, a professor of economics at George Mason University, gives a more balanced analysis: âOf course the CYC project is open to criticism on its many particular choices. People have complained about its logic-like and language-like representations, about its selection of prototypical cases to build from (e.g., encyclopedia articles), about its focus on answering over acting, about how often it rebuilds vs. maintaining legacy systems, and about being private vs. publishing everything. But any large project like this would produce such disputes, and it is not obvious any of its choices have been seriously wrong. They had to start somewhere, and in my opinion they have now collected a knowledge base with a truly spectacular size, scope, and integration. Other architectures may well work better, but if knowing lots is anywhere near as important as Lenat thinks, I'd expect serious AI attempts to import CYC's knowledge, translating it into a new representation. No other source has anywhere near CYC's size, scope, and integration.â
The True Knowledge system provided open-domain question answering using structured knowledge and inference. In the True Knowledge system, knowledge in the knowledge base was represented in a single unified format: named relations between pairs of named entities referred to as âfacts.â Facts, and the relations themselves, were first-class entities so facts about facts and facts about the properties of relations were fully supported (Tunstall-Pedoe, W. (2010). True Knowledge: Open-Domain Question Answering Using Structured Knowledge and Inference. AI Magazine, 31(3), 80-92. https://ojs.aaai.org/aimagazine/index.php/aimagazine/article/view/2298).
EP2527990A2 and EP2527990B1 disclose a method of using large language models in machine translation in which a translation model is partitioned into a plurality of language model partitions stored on a plurality of different language model servers. Segments of text are distributed to the servers for translation according to server workload. The language model information is usually collected by computing the frequency of occurrence of sequences of words in a large training corpus of documents.
SUMMARY OF THE INVENTION
According to a first aspect of the invention, there is provided a method of interacting with a LLM, including the step of a processing system using a structured, machine-readable representation of data that conforms to a machine-readable language, such as a universal language, to provide new context data for the LLM, in order to improve the output, such as continuation text output, generated by the LLM in response to a prompt.
According to a second aspect of the invention, there is provided a method of interacting with a LLM, including the step of providing continuation data generated by the LLM to a processing system that uses a structured, machine-readable representation of data that conforms to a machine-readable language, such as a universal language, in which the processing system is configured to analyse the continuation output (e.g. text output) generated by the LLM in response to a prompt to enable an improved version of that continuation output to be provided to a user. An example is shown in FIG. 10 . An advantage is that improved continuation output to that generated by the LLM may be provided.
According to a third aspect of the invention, there is provided a method of interacting with a LLM, including the step of generating continuation data by an LLM, by the LLM using the output of a processing system that uses a structured, machine-readable representation of data that conforms to a machine-readable language, such as a universal language, in which the LLM is configured to use the output of the processing system as a prompt and to generate a continuation output (e.g. text output) that is an improved version of the output from the processing system. An advantage is that an improved version of the output from the processing system may be generated by the LLM.
According to a fourth aspect of the invention, there is provided a method of interacting with a LLM, including the step of training the LLM on the output from a processing system using a structured, machine-readable representation of data that conforms to a machine-readable language, such as a universal language. An advantage is that an improved LLM may be provided.
According to a fifth aspect of the invention, there is provided a method for interacting with one or more users in natural language comprising the steps of:
providing a structured representation of data and a system operable to process language with reference to the structured representation of data to provide additional useful text relevant to the language;
providing an LLM operable to provide a continuation of text;
receiving a language input from one or more users;
processing the language input with the system to produce additional useful text;
providing the LLM with the language input plus the additional useful text;
providing the one or more users with language derived from a continuation generated by the LLM. An advantage is that additional useful text to that provided by a system operable to process language with reference to the structured representation of data may be provided by a continuation generated by the LLM.
According to a sixth aspect of the invention, there is provided a method of validating natural language for factual accuracy comprising the steps of:
Extracting from the natural language a collection of one or more factual assertions asserted within the natural language;
Checking the one or more factual assertions for factual accuracy; Outputting, or making available, the results of the checking.
An advantage is validating natural language for factual accuracy.
According to a seventh aspect of the invention, there is provided a method of learning knowledge in structured form comprising the steps of:
Providing an LLM trained on a large corpus of natural language documents;
Providing the LLM with a prompt that will elicit desired knowledge;
Eliciting a continuation of the prompt from the LLM;
Translating the continuation into the structured form.
An advantage is learning knowledge in structured form.
According to an eighth aspect of the invention, there is provided a method for ensuring that an LLM generates original text comprising the steps of:
Providing a database of previous text that it should not generate;
Performing a beam search;
<div id="p-0047" num="0046" class="descri
CROSS-REFERENCE TO RELATED APPLICATIONS
This is a continuation of International Application No. PCT/GB2023/050405, filed on Feb. 22, 2023, which claims priority to GB Application No. GB2202347.7, filed on Feb. 22, 2022; GB Application No. GB2219268.6, filed on Dec. 20, 2022; GB Application No. GB2300624.0, filed on Jan. 16, 2023; and GB Application No. GB2302085.2, filed on Feb. 14, 2023, and is a continuation-in-part of U.S. application Ser. No. 18/001,368, filed on Dec. 9, 2022, the entire contents of each of which being fully incorporated herein by reference.
BACKGROUND OF THE INVENTION
1. Field of the Invention
The field of the invention relates to computer implemented methods for the automated analysis or use of data, including use of a large language model (LLM), and to related computer implemented methods and systems.
2. Technical Background
Natural language (NL) is language evolved for humans such as the English language. Although significant advances have been made in computers' ability to process natural language, computers are still not able to deeply understand the meaning of natural language and use that meaning internally.
For this reason most computer applications typically use structured data to store information that they need for processingâe.g. a relational database: designing the schema, populating the database and writing code to process the fields in the database.
Use of structured data can work well if the application has limited requirements for the type of data required. However, some applications naturally require an extremely broad, heterogeneous collection of data to work well. This means that the schema required would have to be enormous, making building and coding for such an application impractical. We refer to such applications herein as HUB applications (Heterogeneous and Unreasonably Broad).
Examples of HUB applications include an application for managing a person's general health data where there are thousands of tests, thousands of medical conditions and thousands of symptoms. Another related application could be a nutrition tracking application where there are many thousands of substances and foods that can be ingested, each with different metabolic effects on the body.
Another example is an application to match the résumé of potential candidates with a job specification: in principle such an application would need structured data to represent every skill that might be of value to any role, every type of experience, every type of previous job.
Accounting is another application where vast heterogeneous data would be valuable: the perfect accounting application would represent every type of contract, every type of service.
In practice some of these applications, where they exist, work with a limited schema that doesn't cover the full range of their ideal properties. Health applications for example, typically work like this ignoring many types of data that they do not cover and instead end up being narrowâlimiting the application to only certain verticals within health.
Applications may also use natural language or augment a limited schema with natural languageâsuch as with current résumé matching applications which might represent a few key skills in a structured form but rely largely on keyword searching or statistical natural language processing (NLP) techniques on written résumé s otherwise.
In the case of accounting, transactions are represented with limited structured dataâdebits and credits on virtual ledgers with natural language names. The meaning of the natural language names and thus what these transactions represent is generally opaque to the application. Virtual ledgers often group different types of transaction together but fail to represent semantic differences which may be important.
There is no exact threshold for when an application becomes a HUB application but the difficulty of building an application with a hand created schema grows more than linearly with the number of tables as managing these tables as well as the code that maintains them becomes increasingly difficult to do.
These issues could be addressed if there existed a language or way of representing data that computers could fully process and understand but that also had an extremely broad scope.
In conventional Artificial Intelligence (AI), statistical Machine Learning (ML)âparticularly Deep Learning (DL)âhas been widely used. This has provided significant progress with many problems. Despite the progress, the results cannot be explained in a way that makes sense to human users as the solutions are the result of a calculation that may involve billions of weights. It can also be argued that the system lacks ârealâ understanding of the data or at least that the understanding is very different from the way a human user would understand it. The statistical nature of the technology also means that it is inherently unreliable with weights that are initiated randomly and then improved through a process of training where they are exposed to data. For complex problems, this improvement (learning) generally results in a model that works only a percentage of the time and is unable to reliably know when the result it produces is accurate.
Much processing of natural language is now done with DL. One example of DL applied to language is large language models (LLMs). LLMs are usually very large deep learning models potentially containing hundreds of billions of parameters and trained on tens of terabytes of text. Although they can generate explanations, those explanations are not derived from what has happened within the model. The text that LLMs produce is frequently incorrect and often describes things that are not true.
3. Discussion of Related Art
The Wikipedia page about Cyc, of 18 Jul. 2019, states that Cyc is the world's longest-lived artificial intelligence project, attempting to assemble a comprehensive ontology and knowledge base that spans the basic concepts and ârules of thumbâ about how the world works (think common sense knowledge but focusing more on things that rarely get written down or said, in contrast with facts one might find somewhere on the internet or retrieve via a search engine or Wikipedia), with the goal of enabling AI applications to perform human-like reasoning and be less âbrittleâ when confronted with novel situations that were not preconceived.
The Wikipedia page about Cyc, of 18 Jul. 2019, states that the objective of the Cyc project was to codify, in machine-usable form, the millions of pieces of knowledge that compose human common sense; this entailed, along the way, (1) developing an adequately expressive representation language, CycL, (2) developing an ontology spanning all human concepts down to some appropriate level of detail, (3) developing a knowledge base on that ontological framework, comprising all human knowledge about those concepts down to some appropriate level of detail, and (4) developing an inference engine exponentially faster than those used in then-conventional expert systems, to be able to infer the same types and depth of conclusions that humans are capable of, given their knowledge of the world.
The Wikipedia page about Cyc, of 18 Jul. 2019, states that most of Cyc's knowledge, outside math and games, is only true by default; for example, Cyc knows that as a default parents love their children, when you're made happy you smile, taking your first step is a big accomplishment, when someone you love has a big accomplishment that makes you happy, and only adults have children; when asked whether a picture captioned âSomeone watching his daughter take her first stepâ contains a smiling adult person, Cyc can logically infer that the answer is Yes, and âshow its workâ by presenting the step by step logical argument using those five pieces of knowledge from its knowledge base: these are formulated in the language CycL, which is based on predicate calculus and has a syntax similar to that of the Lisp programming language.
The Wikipedia page about Cyc, of 18 Jul. 2019, states that the Cyc project has been described as âone of the most controversial endeavors of the artificial intelligence historyâ; Catherine Havasi, CEO of Luminoso, says that Cyc is the predecessor project to IBM's Watson; machine-learning scientist Pedro Domingos refers to the project as a âcatastrophic failureâ for several reasons, including the unending amount of data required to produce any viable results and the inability for Cyc to evolve on its own; Robin Hanson, a professor of economics at George Mason University, gives a more balanced analysis: âOf course the CYC project is open to criticism on its many particular choices. People have complained about its logic-like and language-like representations, about its selection of prototypical cases to build from (e.g., encyclopedia articles), about its focus on answering over acting, about how often it rebuilds vs. maintaining legacy systems, and about being private vs. publishing everything. But any large project like this would produce such disputes, and it is not obvious any of its choices have been seriously wrong. They had to start somewhere, and in my opinion they have now collected a knowledge base with a truly spectacular size, scope, and integration. Other architectures may well work better, but if knowing lots is anywhere near as important as Lenat thinks, I'd expect serious AI attempts to import CYC's knowledge, translating it into a new representation. No other source has anywhere near CYC's size, scope, and integration.â
The True Knowledge system provided open-domain question answering using structured knowledge and inference. In the True Knowledge system, knowledge in the knowledge base was represented in a single unified format: named relations between pairs of named entities referred to as âfacts.â Facts, and the relations themselves, were first-class entities so facts about facts and facts about the properties of relations were fully supported (Tunstall-Pedoe, W. (2010). True Knowledge: Open-Domain Question Answering Using Structured Knowledge and Inference. AI Magazine, 31(3), 80-92. https://ojs.aaai.org/aimagazine/index.php/aimagazine/article/view/2298).
EP2527990A2 and EP2527990B1 disclose a method of using large language models in machine translation in which a translation model is partitioned into a plurality of language model partitions stored on a plurality of different language model servers. Segments of text are distributed to the servers for translation according to server workload. The language model information is usually collected by computing the frequency of occurrence of sequences of words in a large training corpus of documents.
SUMMARY OF THE INVENTION
According to a first aspect of the invention, there is provided a method of interacting with a LLM, including the step of a processing system using a structured, machine-readable representation of data that conforms to a machine-readable language, such as a universal language, to provide new context data for the LLM, in order to improve the output, such as continuation text output, generated by the LLM in response to a prompt.
According to a second aspect of the invention, there is provided a method of interacting with a LLM, including the step of providing continuation data generated by the LLM to a processing system that uses a structured, machine-readable representation of data that conforms to a machine-readable language, such as a universal language, in which the processing system is configured to analyse the continuation output (e.g. text output) generated by the LLM in response to a prompt to enable an improved version of that continuation output to be provided to a user. An example is shown in FIG. 10 . An advantage is that improved continuation output to that generated by the LLM may be provided.
According to a third aspect of the invention, there is provided a method of interacting with a LLM, including the step of generating continuation data by an LLM, by the LLM using the output of a processing system that uses a structured, machine-readable representation of data that conforms to a machine-readable language, such as a universal language, in which the LLM is configured to use the output of the processing system as a prompt and to generate a continuation output (e.g. text output) that is an improved version of the output from the processing system. An advantage is that an improved version of the output from the processing system may be generated by the LLM.
According to a fourth aspect of the invention, there is provided a method of interacting with a LLM, including the step of training the LLM on the output from a processing system using a structured, machine-readable representation of data that conforms to a machine-readable language, such as a universal language. An advantage is that an improved LLM may be provided.
According to a fifth aspect of the invention, there is provided a method for interacting with one or more users in natural language comprising the steps of:
providing a structured representation of data and a system operable to process language with reference to the structured representation of data to provide additional useful text relevant to the language;
providing an LLM operable to provide a continuation of text;
receiving a language input from one or more users;
processing the language input with the system to produce additional useful text;
providing the LLM with the language input plus the additional useful text;
providing the one or more users with language derived from a continuation generated by the LLM. An advantage is that additional useful text to that provided by a system operable to process language with reference to the structured representation of data may be provided by a continuation generated by the LLM.
According to a sixth aspect of the invention, there is provided a method of validating natural language for factual accuracy comprising the steps of:
Extracting from the natural language a collection of one or more factual assertions asserted within the natural language;
Checking the one or more factual assertions for factual accuracy; Outputting, or making available, the results of the checking.
An advantage is validating natural language for factual accuracy.
According to a seventh aspect of the invention, there is provided a method of learning knowledge in structured form comprising the steps of:
Providing an LLM trained on a large corpus of natural language documents;
Providing the LLM with a prompt that will elicit desired knowledge;
Eliciting a continuation of the prompt from the LLM;
Translating the continuation into the structured form.
An advantage is learning knowledge in structured form.
According to an eighth aspect of the invention, there is provided a method for ensuring that an LLM generates original text comprising the steps of:
Providing a database of previous text that it should not generate;
Performing a beam search;
Checking potential continuations against the database;
When a potential continuation matches non-original text in the database, adjusting the continuation to no longer match non-original text in the database.
An advantage is avoiding copyright infringement.
According to a ninth aspect of the invention, there is provided a method of avoiding hallucination in an LLM comprising the steps of:
Analysing different continuations for a given prompt;
Comparing the continuations to identify parts of the response that do not vary and parts of the response that do vary;
Identifying the parts that vary as more likely to be hallucination;
Replacing or not showing the parts identified as hallucination to the user.
An advantage is avoiding hallucination in an LLM.
According to a tenth aspect of the invention, there is provided a method for adding citations to text generated by an LLM comprising the steps of:
Providing a database of text used to train the LLM which includes sources associated with each section of text stored;
checking sections of the continuation generated by the LLM against the database;
retrieving sources where the sections of the continuation match text contained within the database. An advantage is that citations are added to text generated by an LLM.
According to an eleventh aspect of the invention, there is provided a computer-implemented method including the steps of
(i) Receiving output from a natural language processing computer process, the output including an answer to a question;
(ii) Inputting the received output to a large language model (LLM);
(iii) The large language model (LLM) processing the input to the LLM, to generate output based on the input to the LLM.
(iv) Storing the output based on the input to the LLM.
An advantage is that an improved answer to the question may be provided by the LLM output.
According to a twelfth aspect of the invention, there is provided a computer-implemented method of training a large language model (LLM), including the steps of
(i) receiving output from a natural language processing computer process, the output including an answer to a question;
(ii) repeating step (i) at least one thousand times using a set of at least one thousand different questions, and storing the answers to the questions in a training file;
(iii) using the training file to train a large language model (LLM);
(iv) storing weights characterizing the trained LLM.
An advantage is that a trained LLM with training improved by the use of the training file is provided.
According to a thirteenth aspect of the invention, there is provided a computer-implemented method of generating a training file for a large language model (LLM), including the steps of
(i) receiving output from a natural language processing computer process, the output including an answer to a question;
(ii) repeating step (i) at least one thousand times using a set of at least one thousand different questions, and storing the answers to the questions in the training file. An advantage is that an improved LLM training file is generated.
According to a fourteenth aspect of the invention, there is provided a computer-implemented method of re-training a large language model (LLM), the LLM having been previously trained using a training file, the method including the steps of
(i) receiving output from a natural language processing computer process, the output including an answer to a question;
(ii) repeating step (i) at least one thousand times using a set of at least one thousand different questions, and storing the answers to the questions in a re-training file;
(iii) combining the training file and the re-training file, to generate a combined training file;
(iv) using the combined training file to re-train the large language model (LLM);
(v) storing weights characterizing the re-trained LLM.
An advantage is that an LLM with an improved training is provided.
According to a fifteenth aspect of the invention, there is provided a computer-implemented method of updating the training of a large language model (LLM), the LLM having been previously trained using a training file, the method including the steps of
(i) receiving output from a natural language processing computer process, the output including an answer to a question;
(ii) repeating step (i) at least one thousand times using a set of at least one thousand different questions, and storing the answers to the questions in a re-training file;
(iii) using the re-training file to update the training of the large language model (LLM);
(iv) storing weights characterizing the LLM with the updated training. An advantage is that an LLM with an improved training is provided.
According to a sixteenth aspect of the invention, there is provided a computer-implemented method of improving output using an LLM, including the steps of
(i) receiving a natural language question (e.g. any natural language to which a natural language response is appropriate. It could be for example a command or a request for data or even some kind of social interaction or discussion);
(ii) using a computer system including a processor and a memory, the processor configured to use a processing language in which semantic nodes are represented in the processing language, the semantic nodes including semantic links between semantic nodes wherein the semantic links are themselves semantic nodes, in which each semantic node denotes one specific meaning, in which a combination of semantic nodes defines a semantic node, in which expressions in the processing language may be nested, in which the natural language question is represented in the processing language, in which reasoning steps are represented in the processing language to represent semantics of the reasoning steps, in which computation units are represented in the processing language, wherein the memory is configured to store the representations in the processing language;
(iii) the processor answering the question using the reasoning steps, the computation units and the semantic nodes, and
(iv) inputting the natural language question, and the processor's answer to the question to the LLM;
(v) the large language model (LLM) processing the input to the LLM, to generate output based on the input to the LLM;
(vi) storing the output based on the input to the LLM.
An advantage is that an improved answer to the question may be provided by the LLM output.
According to a seventeenth aspect of the invention, there is provided a computer-implemented method of improving output from an LLM, including the steps of
(i) receiving a first natural language question;
(ii) Inputting or providing the received first natural language question to a large language model (LLM);
(iii) The large language model (LLM) processing the input to the LLM, to generate output based on the input to the LLM;
(iv) translating the output into a processing language in which semantic nodes are represented in the processing language, the semantic nodes including semantic links between semantic nodes wherein the semantic links are themselves semantic nodes, in which each semantic node denotes one specific meaning, in which a combination of semantic nodes defines a semantic node, in which expressions in the processing language may be nested;
(v) constructing a question which asks if the output is true, in which the question is represented in the processing language;
(vi) inputting the question to a computer system including a processor and a memory, the processor configured to use the processing language, in which reasoning steps are represented in the processing language to represent semantics of the reasoning steps,
in which computation units are represented in the processing language, wherein the memory is configured to store the representations in the processing language;
(vii) the processor answering the question using the reasoning steps, the computation units and the semantic nodes, and
(viii) the processor storing an answer to the question in the memory.
An advantage is that the LLM output is checked for accuracy.
According to an eighteenth aspect of the invention, there is provided a computer-implemented method of fact checking output from a large language model (LLM), including the steps of
(i) receiving a text input;
(ii) inputting or providing the received text input to a large language model (LLM);
(iii) the large language model (LLM) processing the input to the LLM, to generate output based on the input to the LLM;
(iv) translating the output into a processing language in which semantic nodes are represented in the processing language, the semantic nodes including semantic links between semantic nodes wherein the semantic links are themselves semantic nodes, in which each semantic node denotes one specific meaning, in which a combination of semantic nodes defines a semantic node, in which expressions in the processing language may be nested, wherein translating the output includes extracting the assertions in text generated by the LLM;
(v) constructing one or more questions which ask if the extracted assertions are individually true, in which the one or more questions are represented in the processing language;
(vi) inputting the one or more questions to a computer system including a processor and a memory, the processor configured to use the processing language, in which reasoning steps are represented in the processing language to represent semantics of the reasoning steps,
in which computation units are represented in the processing language, wherein the memory is configured to store the representations in the processing language;
(vii) the processor answering the one or more questions using the reasoning steps, the computation units and the semantic nodes, and
(viii) the processor storing an individual answer to each of the one or more questions in the memory. An advantage is fact checking output from a large language model.
According to a nineteenth aspect of the invention, there is provided a computer-implemented method of improving output from a computer system including a processor and a memory, the system operable to process the natural language question using a processing language in which semantic nodes are represented in the processing language, the method including using an LLM, the method including the steps of
(i) receiving a natural language question;
(ii) using a computer system including a processor and a memory, the system operable to process the natural language question using a processing language in which semantic nodes are represented in the processing language, the semantic nodes including semantic links between semantic nodes wherein the semantic links are themselves semantic nodes, in which each semantic node denotes one specific meaning, wherein the memory is configured to store the representations in the processing language;
(iii) the processor answering the question, and
(iv) inputting the natural language question, and the processor's answer to the question to the LLM;
(v) the large language model (LLM) processing the input to the LLM, to generate output based on the input to the LLM;
(vi) Storing the output based on the input to the LLM.
An advantage is that output from the computer system including a processor and a memory, the system operable to process the natural language question using a processing language in which semantic nodes are represented in the processing language, is improved using the LLM.
According to a twentieth aspect of the invention, there is provided a computer-implemented method of improving output from an LLM, including the steps of
(i) receiving a first natural language question;
(ii) inputting the received first natural language question to a large language model (LLM);
(iii) the large language model (LLM) processing the input to the LLM, to generate output based on the input to the LLM;
(iv) translating the output into a processing language in which semantic nodes are represented in the processing language, the semantic nodes including semantic links between semantic nodes wherein the semantic links are themselves semantic nodes, in which each semantic node denotes one specific meaning;
(v) constructing a question which asks if the output is true, in which the question is represented in the processing language;
(vi) inputting the question to a computer system including a processor and a memory, the processor configured to use the processing language, wherein the memory is configured to store the representations in the processing language;
(vii) the processor answering the question, and
(viii) the processor storing an answer to the question in the memory. An advantage is that the LLM output is checked for accuracy.
According to a 21st aspect of the invention, there is provided a computer-implemented method of fact checking output from a large language model (LLM), including the steps of
(i) receiving a text input;
(ii) inputting the received text input to a large language model (LLM);
(iii) the large language model (LLM) processing the input to the LLM, to generate output based on the input to the LLM;
(iv) translating the output into a processing language in which semantic nodes are represented in the processing language, the semantic nodes including semantic links between semantic nodes wherein the semantic links are themselves semantic nodes, in which each semantic node denotes one specific meaning, wherein translating the output includes extracting the assertions in text generated by the LLM;
(v) constructing one or more questions which ask if the extracted assertions are individually true, in which the one or more questions are represented in the processing language;
(vi) inputting the one or more questions to a computer system including a processor and a memory, the processor configured to use the processing language, wherein the memory is configured to store the representations in the processing language;
(vii) the processor answering the one or more questions, and
(viii) the processor storing an individual answer to each of the one or more questions in the memory.
An advantage is fact checking output from a large language model.
According to a 22nd aspect of the invention, there is provided a computer implemented method of augmenting search results, including the steps of:
(i) receiving a search query;
(ii) an (e.g. internet) search engine processing the search query to generate (e.g. internet) search results;
(iii) transforming the search results into a structured, machine-readable representation of data that conforms to a machine-readable language, such as a universal language;
(iv) processing the transformed search results using a processing system which uses the structured, machine-readable representation of data that conforms to the machine-readable language, such as a universal language, to produce output;
(v) supplying the output of step (iv) as input to a large language model (LLM), and the LLM generating continuation data output in response to the input. An advantage is that search results found by a search engine are augmented.
According to a 23rd aspect of the invention, there is provided a method of fact-checking the output of a large language model (LLM), including the step of providing continuation data generated by the LLM to a processing system that uses a structured, machine-readable representation of data that conforms to a machine-readable language, in which the processing system analyses the continuation output generated by the LLM in response to a prompt, to enable an improved, fact-checked version of that continuation output to be generated and provided to a user. An advantage is fact checking output from a large language model.
Further aspects of the invention are given in the dependent Claims, and in the independent Claims. Aspects of the invention may be combined with each other, or with any of the Concepts described herein, or with any of the Claims.
BRIEF DESCRIPTION OF THE FIGURES
Aspects of the invention will now be described, by way of example(s), with reference to the following Figures, in which:
FIG. 1 shows example screen output for a notification of a job match.
FIG. 2 shows example screen output for a description of a job match.
FIG. 3 shows an example conversation within an app where nutritional data is being communicated with the app.
FIG. 4 shows some example insights that can be derived from a period of time where horizontal health and nutrition data was shown.
FIG. 5 shows an example graph showing daily calories in versus calories out which is an extremely common thing for someone to track if the user is aiming to lose (or gain) weight.
FIG. 6 shows an example of a visualisation capable of being generated from an example of the present invention: it is comparing estimated caffeine in the user's body at the time they went to bed with a calculation of sleep quality.
FIG. 7 shows examples of explanations generated. (a) shows examples of simplified explanations; (b) shows an example of a detailed explanation.
FIG. 8 shows an example of a voice assistant product, referred to herein as âBrianâ and how it fits in with the wider UL platform and other applications built on the UL platform.
FIG. 9 shows an alternative to the example of FIG. 8 .
FIG. 10 shows an example of a method of interacting with a LLM, including the step of providing continuation data generated by the LLM to a processing system that uses a structured, machine-readable representation of data that conforms to a machine-readable language, such as a universal language, in which the processing system is configured to analyse the continuation output (e.g. text output) generated by the LLM in response to a prompt to enable an improved version of that continuation output to be provided to a user.
FIG. 11 shows an example of an intermediate representation semantic graph.
DETAILED DESCRIPTION
This disclosure includes a computer implemented method for the automated analysis or use of data, which comprises the steps of:
(a) storing in a memory a structured, machine-readable representation of data that conforms to a machine-readable language (âmachine representationâ); the machine representation including representations of user speech or text input to a human/machine interface; (b) automatically processing the machine representations to analyse the user speech or text input.
This disclosure includes a computer-based system configured to analyse data, in which the system is configured to:
(a) store in a memory a structured, machine-readable representation of data that conforms to a machine-readable language; the structured, machine-readable representation of data including representations of user speech or text input to a human/machine interface; (b) automatically process the structured representations to analyse the user speech or text input to a human/machine interface.
These aspects of the disclosure may be implemented in a voice assistant or chatbot; the technical advantage achieved is to enable more extensive and faster scaling of the voice assistant or chatbot's capabilities; the invention enables voice assistants and chatbot to be able to answer a broader range of questions, and to do so more accurately; this makes it easier for the voice assistant or chatbot to work with large numbers of different natural languages.
One point of interpretation: the conjunction âorâ should not be narrowly construed to indicate mutual exclusivity but to cover inclusivity too. Hence, the phrase âuser speech or text inputâ means âuser speech on its own, or user text on its own, or both user speech and also user textâ. When the conjunction âorâ is meant exclusively, the phrase âeither . . . orâ is used.
The machine-readable language is extremely expressive yet also extremely simple; simplicity requires less computer processing and thus delivers faster performance. Further details are in the appended Concepts and Claims.
According to a further aspect of the disclosure, there is provided a computer system including a processor and a memory, the processor configured to answer a question, the processor configured to use a processing language in which semantic nodes are represented in the processing language, the semantic nodes including semantic links between semantic nodes wherein the semantic links are themselves semantic nodes, in which each semantic node denotes one specific meaning, in which a combination of semantic nodes defines a semantic node, in which expressions in the processing language may be nested, in which the question is represented in the processing language, in which reasoning steps are represented in the processing language to represent semantics of the reasoning steps, in which computation units are represented in the processing language, wherein the memory is configured to store the representations in the processing language, and wherein the processor is configured to answer the question using the reasoning steps, the computation units and the semantic nodes, and to store an answer to the question in the memory.
An advantage is that because semantic links between semantic nodes are themselves semantic nodes, semantic links and semantic nodes do not need to be processed in a distinctly different way, which simplifies processing, which speeds up response times, which is a technical effect.
An advantage is that because semantic nodes are very widely used in the processing language, processing of the processing language is speeded up, which speeds up response times, which is a technical effect.
The technical effect operates at the level of the architecture of the computer system; that is to say the effect is produced irrespective of the data being processed.
The technical effect results in the computer system being made to operate in a new way, because the computer finds answers to questions faster than using prior art approaches, because semantic nodes are very widely used in the processing language, which means that processing of the processing language is speeded up.
The processing language contributes to the technical character because it produces a technical effect: a processor processing the processing language has the effect that the computer finds answers to questions faster than using prior art approaches, because semantic nodes are very widely used in the processing language, which means that processing of the processing language is speeded up.
The computer system may be configured to output the answer to the question.
The computer system may be configured to output the answer to the question to a display device.
The computer system may be one wherein expressions in the processing language may be nested with no limit inherent to the processing language.
The computer system may be one wherein the semantic nodes each includes a unique identifier.
The computer system may be one wherein the computation units are semantic nodes.
The computer system may be one wherein the question is represented in the processing language with a passage comprising a semantic node that identifies the passage as a question, a list of zero, one or more semantic nodes representing unknown entities being asked about and at least one further passage which represents the semantics of the question in the context of the zero, one or more unknown entities.
The computer system may be one wherein the processing language is universal language.
The computer system may be one wherein the processing language is not a natural language.
The computer system may be one wherein the question relates to search and analysis of documents or web pages, wherein the semantic nodes include representations of at least parts of the documents or the web pages stored in a document store.
The computer system may be one wherein the question relates to a location-based search, using mapping data represented as semantic nodes in the processing language.
The computer system may be one wherein the question relates to a search for defined advertisements or news, wherein the semantic nodes include representations of advertisements, news articles or other information items.
The computer system may be one wherein the question relates to a request for a summary of a news topic, wherein the semantic nodes include representations of news from multiple sources, e.g. to provide a summary or aggregation of the news.
The computer system may be one wherein the question relates to a request for a compatibility match between persons, wherein the semantic nodes include representations of personal information defining one or more attributes of a person, for a plurality of people.
The computer system may be one wherein the question relates to compliance with requirements preventing abusive or illegal social media postings, wherein the semantic nodes include representations of social media postings.
The computer system may be one wherein the question relates to analysing customer reviews, wherein the semantic nodes include representations of customer reviews.
The computer system may be one wherein the question relates to a user's product request, wherein the semantic nodes include representations of product descriptions and user product requests.
The computer system may be one wherein the question relates to a job search, wherein the semantic nodes include representations of job descriptions and job applicants' skills and experience, to determine which job applicants match a job description, or to determine which job descriptions match a job applicant's skills and experience.
The computer system may be one wherein the question relates to health of an individual, wherein the semantic nodes include health data relating to the individual, and health data relating to human beings.
The computer system may be one wherein the question relates to nutrition, wherein the semantic nodes include nutritional data for foods and drinks.
The computer system may be one wherein the question relates to accounting or finance, wherein the semantic nodes include representations of financial or accounting information.
The computer system may be one wherein the question is received by a voice assistant or chatbot, wherein the semantic nodes include representations of user speech input to a human/machine interface and include representations of the human/machine interface itself.
According to a further aspect of the disclosure, there is provided a computer-implemented method, the method using a computer system including a processor and a memory, the processor configured to use a processing language in which semantic nodes are represented in the processing language, the semantic nodes including semantic links between semantic nodes wherein the semantic links are themselves semantic nodes, in which each semantic node denotes one specific meaning, in which a combination of semantic nodes defines a semantic node, in which expressions in the processing language may be nested, in which the question is represented in the processing language, in which reasoning steps are represented in the processing language to represent semantics of the reasoning steps, in which computation units are represented in the processing language, wherein the memory is configured to store the representations in the processing language, the method including the steps of:
(i) the processor answering the question using the reasoning steps, the computation units and the semantic nodes, and
(ii) the processor storing an answer to the question in the memory.
Advantages include the advantages of the previous aspect of the disclosure.
The method may be one wherein the question is represented in the processing language with a passage comprising a semantic node that identifies the passage as a question, a list of zero, one or more semantic nodes representing unknown entities being asked about and at least one further passage which represents the semantics of the question in the context of the zero, one or more unknown entities.
The method may be one wherein the unknowns in the question are identified and the passage making up the body of the question is selected for further analysis; processing begins on a list of passages from the body of the question and the selected unknowns; a first passage in the list of passages is selected for processing; processing a single passage comprises three methods: using statically stored processing language passages, utilising computation units and utilising processing language generated from reasoning:
in which the first method is to lookup in the passage store if there are any passages that can be directly mapped with the passage being processed; if the passage is exactly the same structure as a passage in the passage store, with all nodes matching other than the unknowns, then the values the unknowns match against are valid results;
the second method is to check if any results can be found by executing computation units; it is checked if this passage matches against any passages in a computation unit description; all non-unknown nodes in the passage being processed must match the same nodes in the corresponding position in the computation description or align with a computation input unknown; the unknowns being processed must align to output unknowns in the description; the computation unit is then called to get valid output values for the processed passage's unknowns; the third method is to see if this passage can be proved by applying any reasoning steps; reasoning steps are searched for where a passage in the second half of the reasoning passage can be unified with the passage being processed; all nodes and structure must be equal between the two passages, other than unknowns in the focus passage or the reasoning passage; if
CLAIMS
Claims ( 30 )
The invention claimed is:
1. A computer-implemented method of fact-checking the output of a large language model (LLM), including the steps of:
(a) the LLM processing first input data to the LLM to generate first output from the LLM based on the first input data to the LLM;
(b) a processing system using a structured, machine-readable representation of data that conforms to a machine-readable language, in which semantic nodes are represented in the machine-readable language, the semantic nodes including semantic links between semantic nodes wherein the semantic links are themselves semantic nodes, in which each semantic node denotes one specific meaning, in which a combination of semantic nodes defines a semantic node, in which expressions in the machine-readable language are nestable, in which the first output from the LLM is represented in the machine-readable language, in which reasoning steps are represented in the machine-readable language to represent semantics of the reasoning steps, in which computation units are represented in the machine-readable language;
(c) providing the first output generated by the LLM to the processing system that uses the structured, machine-readable representation of data that conforms to the machine-readable language; and
(d) the processing system analyzing the first output generated by the LLM to fact-check the first output using the reasoning steps, the computation units and the semantic nodes, and to generate second output which is a fact-checked version of the first output and to provide the second output to a user.
2. The method of claim 1 , in which the processing system translates one or more factual assertions in the first output into the structured, machine-readable representation of data and fact-checks the translated one or more factual assertions.
3. The method claim 2 , in which the processing system fact-checks the translated one or more factual assertions by analysing for factual inaccuracies.
4. The method claim 2 , in which the processing system fact-checks the translated one or more factual assertions by analysing for contradictions.
5. The method of claim 1 , in which the improved, fact-checked version of the first output is provided to a user by the processing system.
6. The method of claim 1 , in which the improved, fact-checked version of the first output is provided to a user by the LLM.
7. The method of claim 1 , in which the improved, fact-checked version of the first output is provided to a user by a combination of the processing system and the LLM.
8. The method of claim 1 , in which a classifier operates to identify when the first input data is likely to result in first output where accuracy is important, and/or when accuracy is important in the first output, and to then use the processing system to improve the factual accuracy and/or factual scope of that first output.
9. The method of claim 1 , in which the processing system translates at least some of the first output into the structured, machine-readable representation of data to analyse the translated first output.
10. The method of claim 9 , in which the processing system analyses the translated first output for factual inaccuracies, contradictions and scope, and, where necessary or beneficial, provides corrected output for use in generating continuation output provided by the LLM, through the processing system providing the second output as an updated prompt to the LLM.
11. The method of claim 1 , in which the processing system analyses internal, logical self-consistency of the first output and, where necessary or beneficial, provides logically self-consistent output, for use in generating continuation output that is provided by the LLM, through the processing system providing the second output as an updated prompt to the LLM.
12. The method of claim 1 , in which the processing system analyses correspondence of the first output generated by the LLM to how people understand the real world or reason in the real world and, where necessary or beneficial, provides reasoned output, corresponding to how people understand the real world or reason in the real world, for use in generating continuation output that is provided by the LLM, through the processing system providing the second output as an updated prompt to the LLM.
13. The method of claim 1 , in which the processing system analyses the first output for bias and, where necessary or beneficial, provides reduced bias or bias-free output, for use in generating continuation output that is provided by the LLM, through the processing system providing the second output as an updated prompt to the LLM.
14. The method of claim 1 , in which the processing system analyses the first output for absence of relevant dynamic or real-time information and, where necessary or beneficial, provides any missing or useful dynamic or real-time information for use in generating continuation output that is provided by the LLM, through the processing system providing the second output as an updated prompt to the LLM.
15. The method of claim 1 , in which the processing system analyses the first output for any reasoned text and, where necessary or beneficial, provides improved reasoned text for use in generating continuation output that is provided by the LLM, through the processing system providing the second output as an updated prompt to the LLM.
16. The method of claim 1 , in which the LLM is answering a question and in which the processing system analyses the answer to that question in the first output and where necessary or beneficial, provides an improved answer for use in generating continuation output that is provided by the LLM, through the processing system providing the second output as an updated prompt to the LLM.
17. The method of claim 1 , in which the first output from the LLM is a partial continuation, namely an output made before the LLM has stopped generating or whilst the LLM is still generating.
18. The method of claim 1 , when used to improve one or more of: accuracy of the first output of the LLM; factual accuracy and/or factual scope of the first output of the LLM; internal, logical self-consistency (e.g. correct time ordering of events) of the first output of the LLM; correspondence of the first output generated by the LLM to how people understand the real world or reason in the real world; bias reduction or removal in the first output of the LLM; inclusion of dynamic or real-time information.
19. The method of claim 1 , when used to improve one or more parameters of the first output of the LLM: level of formality, level of brevity (e.g. briefer when the language is to be spoken), suitability for speaking via a text to speech system, other style language, level of certainty.
20. The method of claim 1 , in which the LLM is a generative AI based system.
21. The method of claim 1 , in which the LLM is an autoregressive language model.
22. The method of claim 1 , in which the LLM is an autoregressive language model, which is a Generative Pre-trained Transformer.
23. The method of claim 1 , when used for any of the following: generation of program code, solution of any problem that can be described in natural language, generation of poetry, lyrics, creative writing, generation of other forms of writing such as essays, summaries of knowledge, summaries of longer texts, essays, scientific papers; question answering; internet search.
24. The method of claim 1 , including identifying first output content that breaches predefined policies by translating the first output into the machine-readable language before the first output is displayed to the user and then checking the machine-readable language representation against the predefined policies.
25. The method of claim 1 , including a method of validating natural language for factual accuracy comprising the steps of:
(i) Extracting from the natural language a collection of one or more factual assertions asserted within the natural language;
(ii) Checking the one or more factual assertions for factual accuracy;
(iii) Outputting, or making available, the results of the checking.
26. The method of claim 25 , where the step of checking the one or more factual assertions for factual accuracy comprises the further steps of providing a reasoning system able to reason with a symbolic representation of the world, translating the assertions into a representation compatible with the reasoning system and utilising the reasoning system to identify assertions which are true or false.
27. The method of claim 26 where the reasoning system uses a universal language.
28. The method of claim 25 , where checking the one of more assertions for factual accuracy comprises translating the assertions into questions corresponding to whether the assertions are true or false and answering those questions.
29. The method of claim 25 , where the sources of any previous known knowledge used to validate a factual assertion as false are returned as part of the process of checking the one or more factual assertions for factual accuracy.
30. The method of claim 25 , where the checking the one or more factual assertions for factual accuracy comprises generating an explanation.
US18/301,594
2021-08-24
2023-04-17
Computer implemented methods for the automated analysis or use of data, including use of a large language model
Active
US12073180B2
( en )
Priority Applications (1)
Application Number
Priority Date
Filing Date
Title
US18/301,594
US12073180B2
( en )
2021-08-24
2023-04-17
Computer implemented methods for the automated analysis or use of data, including use of a large language model
Applications Claiming Priority (16)
Application Number
Priority Date
Filing Date
Title
PCT/GB2021/052196
WO2022043675A2
( en )
2020-08-24
2021-08-24
A computer implemented method for the automated analysis or use of data
GB2202347
2022-02-22
GB2202347.7
2022-02-22
GBGB2202347.7A
GB202202347D0
( en )
2022-02-22
2022-02-22
Unlikely ai 2nd gen
US202218001368A
2022-12-09
2022-12-09
GB2219268
2022-12-20
GB2219268.6
2022-12-20
GBGB2219268.6A
GB202219268D0
( en )
2022-12-20
2022-12-20
Unlikely ai 2nd gen ii
GB2300624.0
2023-01-16
GBGB2300624.0A
GB202300624D0
( en )
2023-01-16
2023-01-16
Unlikely ai 2nd gen iii
GB2300624
2023-01-16
GB2302085.2
2023-02-14
GBGB2302085.2A
GB202302085D0
( en )
2023-02-14
2023-02-14
Unlikely AI 2nd gen IV
GB2302085
2023-02-14
PCT/GB2023/050405
WO2023161630A1
( en )
2022-02-22
2023-02-22
Computer implemented methods for the automated analysis or use of data, including use of a large language model
US18/301,594
US12073180B2
( en )
2021-08-24
2023-04-17
Computer implemented methods for the automated analysis or use of data, including use of a large language model
Related Parent Applications (3)
Application Number
Title
Priority Date
Filing Date
PCT/GB2021/052196
Continuation
WO2022043675A2
( en )
2020-08-24
2021-08-24
A computer implemented method for the automated analysis or use of data
US18/001,368
Continuation
US20230343333A1
( en )
2020-08-24
2021-08-24
A computer implemented method for the aut0omated analysis or use of data
PCT/GB2023/050405
Continuation
WO2023161630A1
( en )
2021-08-24
2023-02-22
Computer implemented methods for the automated analysis or use of data, including use of a large language model
Publications (2)
Publication Number
Publication Date
US20230259705A1
US20230259705A1 ( en )
2023-08-17
US12073180B2
true
US12073180B2 ( en )
2024-08-27
Family
ID=87560055
Family Applications (1)
Application Number
Title
Priority Date
Filing Date
US18/301,594
Active
US12073180B2
( en )
2021-08-24
2023-04-17
Computer implemented methods for the automated analysis or use of data, including use of a large language model
Country Status (1)
Country
Link
US
( 1 )
US12073180B2
( en )
Cited By (14)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US20240378397A1
( en )
*
2023-05-09
2024-11-14
Jim Liu
Dynamic content generation method
US12174906B1
( en )
2023-08-24
2024-12-24
Tiny Fish Inc.
Utilizing a query response to automate a task associated with a webpage
US20250053562A1
( en )
*
2023-08-07
2025-02-13
Adobe Inc.
Machine learning recollection as part of question answering using a corpus
US12236216B1
( en )
2023-08-24
2025-02-25
Tiny Fish Inc.
Generate a script to automate a task associated with a webpage
US20250068942A1
( en )
*
2023-08-24
2025-02-27
Tiny Fish Inc.
Utilizing large language model responses to train an inference pattern engine
US20250077791A1
( en )
*
2023-08-29
2025-03-06
Intuit Inc.
Large language model and deterministic calculator systems and methods
US20250086211A1
( en )
*
2023-09-12
2025-03-13
Bitvore Corp.
Grounding large language models using real-time content feeds and reference data
US20250147967A1
( en )
*
2023-11-06
2025-05-08
Amadeus S.A.S.
Search request processing
US12368931B1
( en )
2025-04-04
2025-07-22
Lumana Inc.
Multimedia content management using reduced representations
US20250348657A1
( en )
*
2024-05-10
2025-11-13
Walkme Ltd.
Defining widgets and executing them over an application
US20250355908A1
( en )
*
2023-05-04
2025-11-20
Vijay Madisetti
Method and System for Optimizing Use of Retrieval Augmented Generation Pipelines in Generative Artificial Intelligence Applications
US12481837B2
( en )
*
2023-03-09
2025-11-25
Microsoft Technology Licensing, Llc
Prompt configuration for LLM integrations in spreadsheet environments
US20250362942A1
( en )
*
2024-05-22
2025-11-27
Microsoft Technology Licensing, Llc
Automated content generation for instructional information via generative ai integration
US12541488B2
( en )
2024-05-23
2026-02-03
AtomBeam Technologies Inc.
System and method for generating thoughts with large language models using codewords
Families Citing this family (85)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US12001462B1
( en )
*
2023-05-04
2024-06-04
Vijay Madisetti
Method and system for multi-level artificial intelligence supercomputer design
US12046011B2
( en )
*
2021-06-22
2024-07-23
Docusign, Inc.
Machine learning-based document splitting and labeling in an electronic document system
US12248898B2
( en )
2022-01-28
2025-03-11
AstrumU, Inc.
Confirming skills and proficiency in course offerings
KR102802198B1
( en )
*
2022-03-25
2025-05-02
주ìíì¬ ëë¹ë©
Apparatus and method for monitoring eating based on artificial intelligence
US12118295B2
( en )
*
2022-10-11
2024-10-15
Adobe Inc.
Text simplification with minimal hallucination
US20240289365A1
( en )
*
2023-02-28
2024-08-29
Shopify Inc.
Systems and methods for performing vector search
US20240289338A1
( en )
*
2023-02-28
2024-08-29
Salesforce, Inc.
Reusing Custom Concepts in Visual Analytics Workflows
US20240296295A1
( en )
*
2023-03-03
2024-09-05
Microsoft Technology Licensing, Llc
Attribution verification for answers and summaries generated from large language models (llms)
US12572737B2
( en )
*
2023-04-21
2026-03-10
Microsoft Technology Licensing, Llc
Generative thought starters
US12321393B2
( en )
*
2023-05-17
2025-06-03
Microsoft Technology Licensing, Llc
LLM integrations for data visualization in spreadsheet environments
US20240406202A1
( en )
*
2023-06-02
2024-12-05
Dominic Bayless
Online dating scheduling and matching platform
US20240404669A1
( en )
*
2023-06-02
2024-12-05
MDLIVE, Inc.
Physician assistant generative model
US12511487B2
( en )
*
2023-07-26
2025-12-30
Micro Focus Llc
Augmented question and answer (Q and A) with large language models
US12541347B2
( en )
2023-07-31
2026-02-03
Discover Beagle, Inc.
Integration of machine learning models into software systems using software library
US12572351B2
( en )
*
2023-07-31
2026-03-10
Discover Beagle, Inc.
Integration of machine learning models into software systems using software library
US20250053835A1
( en )
*
2023-08-07
2025-02-13
Trunk Tools, Inc.
Methods and systems for generative question answering for construction project data
CN117097790A
( en )
*
2023-08-08
2023-11-21
å京åè·³ç½ç»ææ¯æéå ¬å¸
Information pushing method and device, computer equipment and storage medium
WO2025037311A1
( en )
2023-08-14
2025-02-20
Monday.com Ltd.
Methods for implementing artificial intelligence capabilities in software applications
WO2025042894A1
( en )
*
2023-08-21
2025-02-27
Patel Surag
System and method for automated continuous source code improvement
KR20250029483A
( en )
2023-08-23
2025-03-05
ë¤ì´ë² 주ìíì¬
Method and system for dynamically generating artificial intelligence-based answer projected with message of information provider
US12424210B2
( en )
*
2023-08-28
2025-09-23
Amazon Technologies, Inc.
Natural language processing
US12493615B2
( en )
*
2023-08-28
2025-12-09
Wiz, Inc.
System and method for improving efficiency in natural language query processing utilizing language model
US12608370B2
( en )
*
2023-08-28
2026-04-21
Wiz, Inc.
System and method for natural language query processing utilizing language model techniques
AU2024334246A1
( en )
*
2023-08-28
2026-04-09
1206881 Alberta Ltd.
System and method for generating course content
US12572855B1
( en )
2023-08-28
2026-03-10
Dmitry Shapiro
Systems and methods for refining machine learning models based on user responses to prompts
CN119597865A
( en )
*
2023-08-28
2025-03-11
æå·é¿éäºé£å¤©ä¿¡æ¯ææ¯æéå ¬å¸
Information interaction system, method and electronic device
EP4517547A1
( en )
2023-08-31
2025-03-05
Arina Man
Intelligent tabular data management system utilizing language models and method thereof
WO2025054481A1
( en )
*
2023-09-06
2025-03-13
Navan, Inc.
User interface for improved large language model integration
US12468894B2
( en )
*
2023-09-08
2025-11-11
Maplebear Inc.
Using language model to generate recipe with refined content
US12530112B2
( en )
2023-09-11
2026-01-20
Salesforce, Inc.
Generating virtual space headers utilizing machine-learned models
US12093965B1
( en )
*
2023-09-13
2024-09-17
Dmitry Shapiro
Systems and methods for generating response records for individual users based on user responses to prompts
JP2025051724A
( en )
*
2023-09-22
2025-04-04
ã½ãããã³ã¯ã°ã«ã¼ãæ ªå¼ä¼ç¤¾
system
WO2025072554A1
( en )
*
2023-09-26
2025-04-03
The Regents Of The University Of California
Systems and methods for dynamic and real-time biological experiment control
CN116992006B
( en )
*
2023-09-26
2024-01-12
æ¦æ±ç模ç§æè¡ä»½æéå ¬å¸
Chain type natural language interaction method and system driven by large language model
US12614025B2
( en )
2023-10-05
2026-04-28
Bank Of America Corporation
Authorship source analysis for large language models (LLM) using a distributed ledger
US11922143B1
( en )
2023-10-05
2024-03-05
Dmitry Shapiro
Systems and methods for providing a user interface that facilitates application development
US20250117381A1
( en )
*
2023-10-10
2025-04-10
Google Llc
Utilizing large language model (llm) in responding to multifaceted queries
US12099975B1
( en )
*
2023-10-13
2024-09-24
AstrumU, Inc.
System for analyzing learners
US12020140B1
( en )
2023-10-24
2024-06-25
Mckinsey & Company, Inc.
Systems and methods for ensuring resilience in generative artificial intelligence pipelines
US12079570B1
( en )
*
2023-10-24
2024-09-03
Mckinsey & Company, Inc.
Systems and methods for packaging reusable generative artificial intelligence pipelines
WO2025094367A1
( en )
*
2023-11-02
2025-05-08
å¯å£«éæ ªå¼ä¼ç¤¾
Hallucination determination program, method, and device
US12526208B2
( en )
2023-11-03
2026-01-13
Cisco Technology, Inc.
LLM-based agent as a back-office virtual network troubleshooting assistant
US12596707B2
( en )
2023-11-06
2026-04-07
Cisco Technology, Inc.
Structured query language generation using large language models
WO2025106544A1
( en )
*
2023-11-13
2025-05-22
Autodesk, Inc.
Generative artificial intelligence (ai) construction specification interface
US20250156652A1
( en )
*
2023-11-14
2025-05-15
Sap Se
Real-time adaptation of machine learning models using large language models
US12361212B2
( en )
*
2023-11-14
2025-07-15
Green Swan Labs LTD
System and method for generating and extracting data from machine learning model outputs
CN117271778B
( en )
*
2023-11-17
2024-02-09
å京水滴ç§æéå¢æéå ¬å¸
Insurance outbound session information output method and device based on generation type large model
US12511497B1
( en )
*
2023-11-30
2025-12-30
Amazon Technologies, Inc.
Embedding-based large language model tuning
CN117932015B
( en )
*
2023-12-20
2025-06-20
å京ç¾åº¦ç½è®¯ç§ææéå ¬å¸
Question answering model training and question answering method and device
US12619822B2
( en )
2024-01-09
2026-05-05
International Business Machines Corporation
Responsible prompt recommendation
US20250245665A1
( en )
*
2024-01-29
2025-07-31
Actimize Ltd
Fraud risk analysis system incorporating a large language model
US12452126B2
( en )
2024-02-13
2025-10-21
T-Mobile Usa, Inc.
Provisioning flow troubleshooting tool
US20250265253A1
( en )
*
2024-02-15
2025-08-21
Cisco Technology, Inc.
Automatic retrieval augmented generation with expanding context
CN117724695B
( en )
*
2024-02-18
2024-04-30
æµæ±åè±é¡ºæºè½ç§ææéå ¬å¸
Code generation optimization method, device, equipment and medium for large language model
US20250272070A1
( en )
*
2024-02-27
2025-08-28
Cisco Technology, Inc.
Automatic curation of reusable code snippets for llm agents
WO2025183576A1
( en )
*
2024-02-29
2025-09-04
Xero Limited
Automated selection of processing functions in response to user message
US12450292B2
( en )
2024-03-11
2025-10-21
Wells Fargo Bank, N.A.
Document set interrogation tool
US20250291815A1
( en )
*
2024-03-15
2025-09-18
Acurai, Inc.
System and Method for Rapid Relevant Data Retrieval from an Electronic Knowledge Base
CN117892237B
( en )
*
2024-03-15
2024-06-07
å京信æ¯å·¥ç¨å¤§å¦
Multi-modal dialogue emotion recognition method and system based on hypergraph neural network
CN117892818B
( en )
*
2024-03-18
2024-05-28
æµæ±å¤§å¦
Large language model rational content generation method based on implicit thinking chain
US20250298970A1
( en )
*
2024-03-25
2025-09-25
Amazon Technologies, Inc.
Enhanced plugin selection in conversational artificial intelligence systems through contextual large language model prompts
DE102024111382A1
( en )
*
2024-04-23
2025-10-23
Bayerische Motoren Werke Aktiengesellschaft
Assistance system and assistance procedure for a vehicle
US12271696B1
( en )
*
2024-05-23
2025-04-08
AtomBeam Technologies Inc.
System and method for training and operating large language models using codewords
GB2641725A
( en )
*
2024-06-06
2025-12-17
Mastercard International Inc
Data retrieval for AI models
WO2025259117A1
( en )
*
2024-06-11
2025-12-18
Xero Limited
Apparatus, program, and computer-implemented method for automated processing of natural language instructions
US12561322B2
( en )
2024-06-13
2026-02-24
Servicenow, Inc.
Systems and methods for semantic caching
US20250384400A1
( en )
*
2024-06-14
2025-12-18
Atlassian Pty Ltd.
Use policy compliance in generative output-based features of a collaboration platform
US20250384249A1
( en )
*
2024-06-18
2025-12-18
Trustwise Inc.
Generative artificial intelligence model safety
KR102747558B1
( en )
*
2024-07-08
2025-01-02
주ìíì¬ ìë¹ëë·
Method and apparatus for analyzing medical data using large language model
WO2026024858A1
( en )
*
2024-07-23
2026-01-29
Strong Force Tp Portfolio 2022, Llc
Configured artificial intelligence systems and methods for software-defined vehicles
CN118535684B
( en )
*
2024-07-24
2024-10-15
å±±ä¸æµªæ½®ç§å¦ç 究é¢æéå ¬å¸
A text hallucination detection method and device based on large language model
US12608341B2
( en )
*
2024-08-19
2026-04-21
Lockheed Martin Corporation
Domain-focused natural language question and answer
US20260065106A1
( en )
*
2024-09-05
2026-03-05
Nec Laboratories America, Inc.
Situational awareness uncertainty propagation for medical decision making
WO2026060133A1
( en )
*
2024-09-12
2026-03-19
The Bank Of New York Mellon
Messaging system decision reapplication engine
KR102785913B1
( en )
*
2024-09-13
2025-03-26
주ìíì¬ ì¸íë©
Ai-based job training question and answer system and its operation method
US20260080181A1
( en )
*
2024-09-18
2026-03-19
Truist Bank
Data processing systems facilitating natural language processing for conversational data
US12367213B1
( en )
2024-09-23
2025-07-22
Aravo Solutions, Inc.
Generating structured data from unstructured data using metadata from a large language model
US12436966B1
( en )
*
2024-09-23
2025-10-07
Aravo Solutions, Inc.
Utilizing a large language model to generate a computing structure
US12399889B1
( en )
2024-09-23
2025-08-26
Aravo Solutions, Inc.
Modifying a database using a large language model
US12423265B1
( en )
2024-09-23
2025-09-23
Aravo Solutions, Inc.
Prompting a large language model for vector embeddings and metadata to generate an indexed computing file
US12307799B1
( en )
2024-09-23
2025-05-20
AstrumU, Inc.
Document ingestion pipeline
US20260086986A1
( en )
*
2024-09-23
2026-03-26
Aravo Solutions, Inc.
Utilizing a large language model to generate a computing structure
CN119917757A
( en )
*
2024-11-05
2025-05-02
æå·é¿ææºåç§ææéå ¬å¸
A web page data parsing method based on large language model
US12619831B1
( en )
*
2025-04-14
2026-05-05
Charles DeLisi
Systems and methods for generating deliberative committees of age-stratified large language models
US12591766B1
( en )
*
2025-05-30
2026-03-31
OpenAi OPCo, LLC.
Application programming interface with generative response engine state management
Citations (248)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US4974191A
( en )
1987-07-31
1990-11-27
Syntellect Software Inc.
Adaptive natural language computer interface system
US5386556A
( en )
1989-03-06
1995-01-31
International Business Machines Corporation
Natural language analyzing apparatus and method
US6085186A
( en )
1996-09-20
2000-07-04
Netbot, Inc.
Method and system using information written in a wrapper description language to execute query on a network
US20020173971A1
( en )
2001-03-28
2002-11-21
Stirpe Paul Alan
System, method and application of ontology driven inferencing-based personalization systems
US20030130976A1
( en )
1998-05-28
2003-07-10
Lawrence Au
Semantic network methods to disambiguate natural language meaning
US20040054626A1
( en )
2002-09-17
2004-03-18
<span