Conceptio › Archive › Google Patents

Google Patentsopen access

… the automated analysis or use of data, including use of a large language model — Unlikely Artificial Intelligence Limited (US11989527B2)

Unlikely Artificial Intelligence Limited · Google Patents

Google Patents · Patents · License: Open Access

pedoe unlikelyartificialintelligencelimited williamtunstall

patent, google patents, intellectual property, US11989527B2, Unlikely Artificial Intelligence Limited, William Tunstall-Pedoe, en, 2024

ABSTRACT

Abstract

Methods are provided, such as a method of interacting with a large language model (LLM), including the step of a processing system using a structured, machine-readable representation of data that conforms to a machine-readable language, such as a universal language, to provide new context data for the LLM, in order to improve the output, such as continuation text output, generated by the LLM in response to a prompt; and such as a method of interacting with a LLM, including the step of providing continuation data generated by the LLM to a processing system that uses a structured, machine-readable representation of data that conforms to a machine-readable language, such as a universal language, in which the processing system is configured to analyse the continuation output generated by the LLM in response to a prompt to enable an improved version of that continuation output to be provided to a user. Related computer systems are provided.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This is a continuation of International Application No. PCT/GB2023/050405, filed on Feb. 22, 2023, which claims priority to GB Application No. GB2202347.7, filed on Feb. 22, 2022; GB Application No. GB2219268.6, filed on Dec. 20, 2022; GB Application No. GB2300624.0, filed on Jan. 16, 2023; and GB Application No. GB2302085.2, filed on Feb. 14, 2023, and is a continuation-in-part of U.S. application Ser. No. 18/001,368, filed on Dec. 9, 2022, the entire contents of each of which being fully incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The field of the invention relates to computer implemented methods for the automated analysis or use of data, including use of a large language model (LLM), and to related computer implemented methods and systems.

2. Technical Background

Natural language (NL) is language evolved for humans such as the English language. Although significant advances have been made in computers' ability to process natural language, computers are still not able to deeply understand the meaning of natural language and use that meaning internally.

For this reason most computer applications typically use structured data to store information that they need for processingâe.g. a relational database: designing the schema, populating the database and writing code to process the fields in the database.

Use of structured data can work well if the application has limited requirements for the type of data required. However, some applications naturally require an extremely broad, heterogeneous collection of data to work well. This means that the schema required would have to be enormous, making building and coding for such an application impractical. We refer to such applications herein as HUB applications (Heterogeneous and Unreasonably Broad).

Examples of HUB applications include an application for managing a person's general health data where there are thousands of tests, thousands of medical conditions and thousands of symptoms. Another related application could be a nutrition tracking application where there are many thousands of substances and foods that can be ingested, each with different metabolic effects on the body.

Another example is an application to match the resume of potential candidates with a job specification: in principle such an application would need structured data to represent every skill that might be of value to any role, every type of experience, every type of previous job.

Accounting is another application where vast heterogeneous data would be valuable: the perfect accounting application would represent every type of contract, every type of service.

In practice some of these applications, where they exist, work with a limited schema that doesn't cover the full range of their ideal properties. Health applications for example, typically work like this ignoring many types of data that they do not cover and instead end up being narrowâlimiting the application to only certain verticals within health.

Applications may also use natural language or augment a limited schema with natural languageâsuch as with current resume matching applications which might represent a few key skills in a structured form but rely largely on keyword searching or statistical natural language processing (NLP) techniques on written resumes otherwise.

In the case of accounting, transactions are represented with limited structured dataâdebits and credits on virtual ledgers with natural language names. The meaning of the natural language names and thus what these transactions represent is generally opaque to the application. Virtual ledgers often group different types of transaction together but fail to represent semantic differences which may be important.

There is no exact threshold for when an application becomes a HUB application but the difficulty of building an application with a hand created schema grows more than linearly with the number of tables as managing these tables as well as the code that maintains them becomes increasingly difficult to do.

These issues could be addressed if there existed a language or way of representing data that computers could fully process and understand but that also had an extremely broad scope.

In conventional Artificial Intelligence (AI), statistical Machine Learning (ML)âparticularly Deep Learning (DL)âhas been widely used. This has provided significant progress with many problems. Despite the progress, the results cannot be explained in a way that makes sense to human users as the solutions are the result of a calculation that may involve billions of weights. It can also be argued that the system lacks ârealâ understanding of the data or at least that the understanding is very different from the way a human user would understand it. The statistical nature of the technology also means that it is inherently unreliable with weights that are initiated randomly and then improved through a process of training where they are exposed to data. For complex problems, this improvement (learning) generally results in a model that works only a percentage of the time and is unable to reliably know when the result it produces is accurate.

Much processing of natural language is now done with DL. One example of DL applied to language is large language models (LLMs). LLMs are usually very large deep learning models potentially containing hundreds of billions of parameters and trained on tens of terabytes of text. Although they can generate explanations, those explanations are not derived from what has happened within the model. The text that LLMs produce is frequently incorrect and often describes things that are not true.

3. Discussion of Related Art

The Wikipedia page about Cyc, of 18 Jul. 2019, states that Cyc is the world's longest-lived artificial intelligence project, attempting to assemble a comprehensive ontology and knowledge base that spans the basic concepts and ârules of thumbâ about how the world works (think common sense knowledge but focusing more on things that rarely get written down or said, in contrast with facts one might find somewhere on the internet or retrieve via a search engine or Wikipedia), with the goal of enabling AI applications to perform human-like reasoning and be less âbrittleâ when confronted with novel situations that were not preconceived.

The Wikipedia page about Cyc, of 18 Jul. 2019, states that the objective of the Cyc project was to codify, in machine-usable form, the millions of pieces of knowledge that compose human common sense; this entailed, along the way, (1) developing an adequately expressive representation language, CycL, (2) developing an ontology spanning all human concepts down to some appropriate level of detail, (3) developing a knowledge base on that ontological framework, comprising all human knowledge about those concepts down to some appropriate level of detail, and (4) developing an inference engine exponentially faster than those used in then-conventional expert systems, to be able to infer the same types and depth of conclusions that humans are capable of, given their knowledge of the world.

The Wikipedia page about Cyc, of 18 Jul. 2019, states that most of Cyc's knowledge, outside math and games, is only true by default; for example, Cyc knows that as a default parents love their children, when you're made happy you smile, taking your first step is a big accomplishment, when someone you love has a big accomplishment that makes you happy, and only adults have children; when asked whether a picture captioned âSomeone watching his daughter take her first stepâ contains a smiling adult person, Cyc can logically infer that the answer is Yes, and âshow its workâ by presenting the step by step logical argument using those five pieces of knowledge from its knowledge base: these are formulated in the language CycL, which is based on predicate calculus and has a syntax similar to that of the Lisp programming language.

The Wikipedia page about Cyc, of 18 Jul. 2019, states that the Cyc project has been described as âone of the most controversial endeavors of the artificial intelligence historyâ; Catherine Havasi, CEO of Luminoso, says that Cyc is the predecessor project to IBM's Watson; machine-learning scientist Pedro Domingos refers to the project as a âcatastrophic failureâ for several reasons, including the unending amount of data required to produce any viable results and the inability for Cyc to evolve on its own; Robin Hanson, a professor of economics at George Mason University, gives a more balanced analysis: âOf course the CYC project is open to criticism on its many particular choices. People have complained about its logic-like and language-like representations, about its selection of prototypical cases to build from (e.g., encyclopedia articles), about its focus on answering over acting, about how often it rebuilds vs. maintaining legacy systems, and about being private vs. publishing everything. But any large project like this would produce such disputes, and it is not obvious any of its choices have been seriously wrong. They had to start somewhere, and in my opinion they have now collected a knowledge base with a truly spectacular size, scope, and integration. Other architectures may well work better, but if knowing lots is anywhere near as important as Lenat thinks, I'd expect serious AI attempts to import CYC's knowledge, translating it into a new representation. No other source has anywhere near CYC's size, scope, and integration.â

The True Knowledge system provided open-domain question answering using structured knowledge and inference. In the True Knowledge system, knowledge in the knowledge base was represented in a single unified format: named relations between pairs of named entities referred to as âfacts.â Facts, and the relations themselves, were first-class entities so facts about facts and facts about the properties of relations were fully supported (Tunstall-Pedoe, W. (2010). True Knowledge: Open-Domain Question Answering Using Structured Knowledge and Inference. AI Magazine, 31(3), 80-92. https://ojs.aaai.org/aimagazine/index.php/aimagazine/article/view/2298).

EP2527990A2 and EP2527990B1 disclose a method of using large language models in machine translation in which a translation model is partitioned into a plurality of language model partitions stored on a plurality of different language model servers. Segments of text are distributed to the servers for translation according to server workload. The language model information is usually collected by computing the frequency of occurrence of sequences of words in a large training corpus of documents.

SUMMARY OF THE INVENTION

According to a first aspect of the invention, there is provided a method of interacting with a LLM, including the step of a processing system using a structured, machine-readable representation of data that conforms to a machine-readable language, such as a universal language, to provide new context data for the LLM, in order to improve the output, such as continuation text output, generated by the LLM in response to a prompt.

According to a second aspect of the invention, there is provided a method of interacting with a LLM, including the step of providing continuation data generated by the LLM to a processing system that uses a structured, machine-readable representation of data that conforms to a machine-readable language, such as a universal language, in which the processing system is configured to analyse the continuation output (e.g. text output) generated by the LLM in response to a prompt to enable an improved version of that continuation output to be provided to a user. An example is shown in FIG. 10 . An advantage is that improved continuation output to that generated by the LLM may be provided.

According to a third aspect of the invention, there is provided a method of interacting with a LLM, including the step of generating continuation data by an LLM, by the LLM using the output of a processing system that uses a structured, machine-readable representation of data that conforms to a machine-readable language, such as a universal language, in which the LLM is configured to use the output of the processing system as a prompt and to generate a continuation output (e.g. text output) that is an improved version of the output from the processing system. An advantage is that an improved version of the output from the processing system may be generated by the LLM.

According to a fourth aspect of the invention, there is provided a method of interacting with a LLM, including the step of training the LLM on the output from a processing system using a structured, machine-readable representation of data that conforms to a machine-readable language, such as a universal language. An advantage is that an improved LLM may be provided.

According to a fifth aspect of the invention, there is provided a method for interacting with one or more users in natural language comprising the steps of:

providing a structured representation of data and a system operable to process language with reference to the structured representation of data to provide additional useful text relevant to the language; providing an LLM operable to provide a continuation of text; receiving a language input from one or more users; processing the language input with the system to produce additional useful text; providing the LLM with the language input plus the additional useful text; providing the one or more users with language derived from a continuation generated by the LLM. An advantage is that additional useful text to that provided by a system operable to process language with reference to the structured representation of data may be provided by a continuation generated by the LLM.

According to a sixth aspect of the invention, there is provided a method of validating natural language for factual accuracy comprising the steps of:

Extracting from the natural language a collection of one or more factual assertions asserted within the natural language; Checking the one or more factual assertions for factual accuracy; Outputting, or making available, the results of the checking. An advantage is validating natural language for factual accuracy.

According to a seventh aspect of the invention, there is provided a method of learning knowledge in structured form comprising the steps of:

Providing an LLM trained on a large corpus of natural language documents; Providing the LLM with a prompt that will elicit desired knowledge; Eliciting a continuation of the prompt from the LLM; Translating the continuation into the structured form. An advantage is learning knowledge in structured form.

According to an eighth aspect of the invention, there is provided a method for ensuring that an LLM generates original text comprising the steps of:

Providing a database of previous text that it should not generate; Performing a beam search; Checking potential continuations against the database; When a potential continuation matches non-original text in the database, adjusting the continuation to no longer match non-original text in the database. <li id="ul0008-0005" num="

CROSS-REFERENCE TO RELATED APPLICATIONS

BACKGROUND OF THE INVENTION

1. Field of the Invention

2. Technical Background

Accounting is another application where vast heterogeneous data would be valuable: the perfect accounting application would represent every type of contract, every type of service.

These issues could be addressed if there existed a language or way of representing data that computers could fully process and understand but that also had an extremely broad scope.

3. Discussion of Related Art

SUMMARY OF THE INVENTION

According to a fifth aspect of the invention, there is provided a method for interacting with one or more users in natural language comprising the steps of:

According to a sixth aspect of the invention, there is provided a method of validating natural language for factual accuracy comprising the steps of:

According to a seventh aspect of the invention, there is provided a method of learning knowledge in structured form comprising the steps of:

According to an eighth aspect of the invention, there is provided a method for ensuring that an LLM generates original text comprising the steps of:

According to a ninth aspect of the invention, there is provided a method of avoiding hallucination in an LLM comprising the steps of:

Analysing different continuations for a given prompt; Comparing the continuations to identify parts of the response that do not vary and parts of the response that do vary; Identifying the parts that vary as more likely to be hallucination; Replacing or not showing the parts identified as hallucination to the user. An advantage is avoiding hallucination in an LLM.

According to a tenth aspect of the invention, there is provided a method for adding citations to text generated by an LLM comprising the steps of:

Providing a database of text used to train the LLM which includes sources associated with each section of text stored; checking sections of the continuation generated by the LLM against the database; retrieving sources where the sections of the continuation match text contained within the database. An advantage is that citations are added to text generated by an LLM.

According to an eleventh aspect of the invention, there is provided a computer-implemented method including the steps of

(i) Receiving output from a natural language processing computer process, the output including an answer to a question; (ii) Inputting the received output to a large language model (LLM); (iii) The large language model (LLM) processing the input to the LLM, to generate output based on the input to the LLM. (iv) Storing the output based on the input to the LLM.

An advantage is that an improved answer to the question may be provided by the LLM output.

According to a twelfth aspect of the invention, there is provided a computer-implemented method of training a large language model (LLM), including the steps of

(i) receiving output from a natural language processing computer process, the output including an answer to a question; (ii) repeating step (i) at least one thousand times using a set of at least one thousand different questions, and storing the answers to the questions in a training file; (iii) using the training file to train a large language model (LLM); (iv) storing weights characterizing the trained LLM.

An advantage is that a trained LLM with training improved by the use of the training file is provided.

According to a thirteenth aspect of the invention, there is provided a computer-implemented method of generating a training file for a large language model (LLM), including the steps of

According to a fourteenth aspect of the invention, there is provided a computer-implemented method of re-training a large language model (LLM), the LLM having been previously trained using a training file, the method including the steps of

(i) receiving output from a natural language processing computer process, the output including an answer to a question; (ii) repeating step (i) at least one thousand times using a set of at least one thousand different questions, and storing the answers to the questions in a re-training file; (iii) combining the training file and the re-training file, to generate a combined training file; (iv) using the combined training file to re-train the large language model (LLM); (v) storing weights characterizing the re-trained LLM.

An advantage is that an LLM with an improved training is provided.

According to a fifteenth aspect of the invention, there is provided a computer-implemented method of updating the training of a large language model (LLM), the LLM having been previously trained using a training file, the method including the steps of

(i) receiving output from a natural language processing computer process, the output including an answer to a question; (ii) repeating step (i) at least one thousand times using a set of at least one thousand different questions, and storing the answers to the questions in a re-training file; (iii) using the re-training file to update the training of the large language model (LLM); (iv) storing weights characterizing the LLM with the updated training. An advantage is that an LLM with an improved training is provided.

According to a sixteenth aspect of the invention, there is provided a computer-implemented method of improving output using an LLM, including the steps of

(i) receiving a natural language question (e.g. any natural language to which a natural language response is appropriate. It could be for example a command or a request for data or even some kind of social interaction or discussion); (ii) using a computer system including a processor and a memory, the processor configured to use a processing language in which semantic nodes are represented in the processing language, the semantic nodes including semantic links between semantic nodes wherein the semantic links are themselves semantic nodes, in which each semantic node denotes one specific meaning, in which a combination of semantic nodes defines a semantic node, in which expressions in the processing language may be nested, in which the natural language question is represented in the processing language, in which reasoning steps are represented in the processing language to represent semantics of the reasoning steps, in which computation units are represented in the processing language, wherein the memory is configured to store the representations in the processing language; (iii) the processor answering the question using the reasoning steps, the computation units and the semantic nodes, and (iv) inputting the natural language question, and the processor's answer to the question to the LLM; (v) the large language model (LLM) processing the input to the LLM, to generate output based on the input to the LLM; (vi) storing the output based on the input to the LLM.

An advantage is that an improved answer to the question may be provided by the LLM output.

According to a seventeenth aspect of the invention, there is provided a computer-implemented method of improving output from an LLM, including the steps of

(i) receiving a first natural language question; (ii) Inputting or providing the received first natural language question to a large language model (LLM); (iii) The large language model (LLM) processing the input to the LLM, to generate output based on the input to the LLM; (iv) translating the output into a processing language in which semantic nodes are represented in the processing language, the semantic nodes including semantic links between semantic nodes wherein the semantic links are themselves semantic nodes, in which each semantic node denotes one specific meaning, in which a combination of semantic nodes defines a semantic node, in which expressions in the processing language may be nested; (v) constructing a question which asks if the output is true, in which the question is represented in the processing language; (vi) inputting the question to a computer system including a processor and a memory, the processor configured to use the processing language, in which reasoning steps are represented in the processing language to represent semantics of the reasoning steps, in which computation units are represented in the processing language, wherein the memory is configured to store the representations in the processing language; (vii) the processor answering the question using the reasoning steps, the computation units and the semantic nodes, and (viii) the processor storing an answer to the question in the memory.

An advantage is that the LLM output is checked for accuracy.

According to an eighteenth aspect of the invention, there is provided a computer-implemented method of fact checking output from a large language model (LLM), including the steps of

(i) receiving a text input; (ii) inputting or providing the received text input to a large language model (LLM); (iii) the large language model (LLM) processing the input to the LLM, to generate output based on the input to the LLM; (iv) translating the output into a processing language in which semantic nodes are represented in the processing language, the semantic nodes including semantic links between semantic nodes wherein the semantic links are themselves semantic nodes, in which each semantic node denotes one specific meaning, in which a combination of semantic nodes defines a semantic node, in which expressions in the processing language may be nested, wherein translating the output includes extracting the assertions in text generated by the LLM; (v) constructing one or more questions which ask if the extracted assertions are individually true, in which the one or more questions are represented in the processing language; (vi) inputting the one or more questions to a computer system including a processor and a memory, the processor configured to use the processing language, in which reasoning steps are represented in the processing language to represent semantics of the reasoning steps, in which computation units are represented in the processing language, wherein the memory is configured to store the representations in the processing language; (vii) the processor answering the one or more questions using the reasoning steps, the computation units and the semantic nodes, and (viii) the processor storing an individual answer to each of the one or more questions in the memory. An advantage is fact checking output from a large language model.

According to a nineteenth aspect of the invention, there is provided a computer-implemented method of improving output from a computer system including a processor and a memory, the system operable to process the natural language question using a processing language in which semantic nodes are represented in the processing language, the method including using an LLM, the method including the steps of

(i) receiving a natural language question; (ii) using a computer system including a processor and a memory, the system operable to process the natural language question using a processing language in which semantic nodes are represented in the processing language, the semantic nodes including semantic links between semantic nodes wherein the semantic links are themselves semantic nodes, in which each semantic node denotes one specific meaning, wherein the memory is configured to store the representations in the processing language; (iii) the processor answering the question, and (iv) inputting the natural language question, and the processor's answer to the question to the LLM; (v) the large language model (LLM) processing the input to the LLM, to generate output based on the input to the LLM; (vi) Storing the output based on the input to the LLM.

An advantage is that output from the computer system including a processor and a memory, the system operable to process the natural language question using a processing language in which semantic nodes are represented in the processing language, is improved using the LLM.

According to a twentieth aspect of the invention, there is provided a computer-implemented method of improving output from an LLM, including the steps of

(i) receiving a first natural language question; (ii) inputting the received first natural language question to a large language model (LLM); (iii) the large language model (LLM) processing the input to the LLM, to generate output based on the input to the LLM; (iv) translating the output into a processing language in which semantic nodes are represented in the processing language, the semantic nodes including semantic links between semantic nodes wherein the semantic links are themselves semantic nodes, in which each semantic node denotes one specific meaning; (v) constructing a question which asks if the output is true, in which the question is represented in the processing language; (vi) inputting the question to a computer system including a processor and a memory, the processor configured to use the processing language, wherein the memory is configured to store the representations in the processing language; (vii) the processor answering the question, and (viii) the processor storing an answer to the question in the memory. An advantage is that the LLM output is checked for accuracy.

According to a 21st aspect of the invention, there is provided a computer-implemented method of fact checking output from a large language model (LLM), including the steps of

(i) receiving a text input; (ii) inputting the received text input to a large language model (LLM); (iii) the large language model (LLM) processing the input to the LLM, to generate output based on the input to the LLM; (iv) translating the output into a processing language in which semantic nodes are represented in the processing language, the semantic nodes including semantic links between semantic nodes wherein the semantic links are themselves semantic nodes, in which each semantic node denotes one specific meaning, wherein translating the output includes extracting the assertions in text generated by the LLM; (v) constructing one or more questions which ask if the extracted assertions are individually true, in which the one or more questions are represented in the processing language; (vi) inputting the one or more questions to a computer system including a processor and a memory, the processor configured to use the processing language, wherein the memory is configured to store the representations in the processing language; (vii) the processor answering the one or more questions, and (viii) the processor storing an individual answer to each of the one or more questions in the memory.

An advantage is fact checking output from a large language model.

According to a 22nd aspect of the invention, there is provided a computer implemented method of augmenting search results, including the steps of:

(i) receiving a search query; (ii) an (e.g. internet) search engine processing the search query to generate (e.g. internet) search results; (iii) transforming the search results into a structured, machine-readable representation of data that conforms to a machine-readable language, such as a universal language; (iv) processing the transformed search results using a processing system which uses the structured, machine-readable representation of data that conforms to the machine-readable language, such as a universal language, to produce output; (v) supplying the output of step (iv) as input to a large language model (LLM), and the LLM generating continuation data output in response to the input. An advantage is that search results found by a search engine are augmented.

According to a 23rd aspect of the invention, there is provided a method of fact-checking the output of a large language model (LLM), including the step of providing continuation data generated by the LLM to a processing system that uses a structured, machine-readable representation of data that conforms to a machine-readable language, in which the processing system analyses the continuation output generated by the LLM in response to a prompt, to enable an improved, fact-checked version of that continuation output to be generated and provided to a user. An advantage is fact checking output from a large language model.

Further aspects of the invention are given in the dependent Claims, and in the independent Claims. Aspects of the invention may be combined with each other, or with any of the Concepts described herein, or with any of the Claims.

BRIEF DESCRIPTION OF THE FIGURES

Aspects of the invention will now be described, by way of example(s), with reference to the following Figures, in which:

FIG. 1 shows example screen output for a notification of a job match.

FIG. 2 shows example screen output for a description of a job match.

FIG. 3 shows an example conversation within an app where nutritional data is being communicated with the app.

FIG. 4 shows some example insights that can be derived from a period of time where horizontal health and nutrition data was shown.

FIG. 5 shows an example graph showing daily calories in versus calories out which is an extremely common thing for someone to track if the user is aiming to lose (or gain) weight.

FIG. 6 shows an example of a visualisation capable of being generated from an example of the present invention: it is comparing estimated caffeine in the user's body at the time they went to bed with a calculation of sleep quality.

FIG. 7 shows examples of explanations generated. (a) shows examples of simplified explanations; (b) shows an example of a detailed explanation.

FIG. 8 shows an example of a voice assistant product, referred to herein as âBrianâ and how it fits in with the wider UL platform and other applications built on the UL platform.

FIG. 9 shows an alternative to the example of FIG. 8 .

FIG. 10 shows an example of a method of interacting with a LLM, including the step of providing continuation data generated by the LLM to a processing system that uses a structured, machine-readable representation of data that conforms to a machine-readable language, such as a universal language, in which the processing system is configured to analyse the continuation output (e.g. text output) generated by the LLM in response to a prompt to enable an improved version of that continuation output to be provided to a user.

FIG. 11 shows an example of an intermediate representation semantic graph.

DETAILED DESCRIPTION

This disclosure includes a computer implemented method for the automated analysis or use of data, which comprises the steps of:

(a) storing in a memory a structured, machine-readable representation of data that conforms to a machine-readable language (âmachine representationâ); the machine representation including representations of user speech or text input to a human/machine interface; (b) automatically processing the machine representations to analyse the user speech or text input.

This disclosure includes a computer-based system configured to analyse data, in which the system is configured to:

(a) store in a memory a structured, machine-readable representation of data that conforms to a machine-readable language; the structured, machine-readable representation of data including representations of user speech or text input to a human/machine interface; (b) automatically process the structured representations to analyse the user speech or text input to a human/machine interface.

These aspects of the disclosure may be implemented in a voice assistant or chatbot; the technical advantage achieved is to enable more extensive and faster scaling of the voice assistant or chatbot's capabilities; the invention enables voice assistants and chatbot to be able to answer a broader range of questions, and to do so more accurately; this makes it easier for the voice assistant or chatbot to work with large numbers of different natural languages.

One point of interpretation: the conjunction âorâ should not be narrowly construed to indicate mutual exclusivity but to cover inclusivity too. Hence, the phrase âuser speech or text inputâ means âuser speech on its own, or user text on its own, or both user speech and also user textâ. When the conjunction âorâ is meant exclusively, the phrase âeither . . . orâ is used.

The machine-readable language is extremely expressive yet also extremely simple; simplicity requires less computer processing and thus delivers faster performance. Further details are in the appended Concepts and Claims.

According to a further aspect of the disclosure, there is provided a computer system including a processor and a memory, the processor configured to answer a question, the processor configured to use a processing language in which semantic nodes are represented in the processing language, the semantic nodes including semantic links between semantic nodes wherein the semantic links are themselves semantic nodes, in which each semantic node denotes one specific meaning, in which a combination of semantic nodes defines a semantic node, in which expressions in the processing language may be nested, in which the question is represented in the processing language, in which reasoning steps are represented in the processing language to represent semantics of the reasoning steps, in which computation units are represented in the processing language, wherein the memory is configured to store the representations in the processing language, and wherein the processor is configured to answer the question using the reasoning steps, the computation units and the semantic nodes, and to store an answer to the question in the memory.

An advantage is that because semantic links between semantic nodes are themselves semantic nodes, semantic links and semantic nodes do not need to be processed in a distinctly different way, which simplifies processing, which speeds up response times, which is a technical effect.

An advantage is that because semantic nodes are very widely used in the processing language, processing of the processing language is speeded up, which speeds up response times, which is a technical effect.

The technical effect operates at the level of the architecture of the computer system; that is to say the effect is produced irrespective of the data being processed.

The technical effect results in the computer system being made to operate in a new way, because the computer finds answers to questions faster than using prior art approaches, because semantic nodes are very widely used in the processing language, which means that processing of the processing language is speeded up.

The processing language contributes to the technical character because it produces a technical effect: a processor processing the processing language has the effect that the computer finds answers to questions faster than using prior art approaches, because semantic nodes are very widely used in the processing language, which means that processing of the processing language is speeded up.

The computer system may be configured to output the answer to the question.

The computer system may be configured to output the answer to the question to a display device.

The computer system may be one wherein expressions in the processing language may be nested with no limit inherent to the processing language.

The computer system may be one wherein the semantic nodes each includes a unique identifier.

The computer system may be one wherein the computation units are semantic nodes.

The computer system may be one wherein the question is represented in the processing language with a passage comprising a semantic node that identifies the passage as a question, a list of zero, one or more semantic nodes representing unknown entities being asked about and at least one further passage which represents the semantics of the question in the context of the zero, one or more unknown entities.

The computer system may be one wherein the processing language is universal language.

The computer system may be one wherein the processing language is not a natural language.

The computer system may be one wherein the question relates to search and analysis of documents or web pages, wherein the semantic nodes include representations of at least parts of the documents or the web pages stored in a document store.

The computer system may be one wherein the question relates to a location-based search, using mapping data represented as semantic nodes in the processing language.

The computer system may be one wherein the question relates to a search for defined advertisements or news, wherein the semantic nodes include representations of advertisements, news articles or other information items.

The computer system may be one wherein the question relates to a request for a summary of a news topic, wherein the semantic nodes include representations of news from multiple sources, e.g. to provide a summary or aggregation of the news.

The computer system may be one wherein the question relates to a request for a compatibility match between persons, wherein the semantic nodes include representations of personal information defining one or more attributes of a person, for a plurality of people.

The computer system may be one wherein the question relates to compliance with requirements preventing abusive or illegal social media postings, wherein the semantic nodes include representations of social media postings.

The computer system may be one wherein the question relates to analysing customer reviews, wherein the semantic nodes include representations of customer reviews.

The computer system may be one wherein the question relates to a user's product request, wherein the semantic nodes include representations of product descriptions and user product requests.

The computer system may be one wherein the question relates to a job search, wherein the semantic nodes include representations of job descriptions and job applicants' skills and experience, to determine which job applicants match a job description, or to determine which job descriptions match a job applicant's skills and experience.

The computer system may be one wherein the question relates to health of an individual, wherein the semantic nodes include health data relating to the individual, and health data relating to human beings.

The computer system may be one wherein the question relates to nutrition, wherein the semantic nodes include nutritional data for foods and drinks.

The computer system may be one wherein the question relates to accounting or finance, wherein the semantic nodes include representations of financial or accounting information.

The computer system may be one wherein the question is received by a voice assistant or chatbot, wherein the semantic nodes include representations of user speech input to a human/machine interface and include representations of the human/machine interface itself.

According to a further aspect of the disclosure, there is provided a computer-implemented method, the method using a computer system including a processor and a memory, the processor configured to use a processing language in which semantic nodes are represented in the processing language, the semantic nodes including semantic links between semantic nodes wherein the semantic links are themselves semantic nodes, in which each semantic node denotes one specific meaning, in which a combination of semantic nodes defines a semantic node, in which expressions in the processing language may be nested, in which the question is represented in the processing language, in which reasoning steps are represented in the processing language to represent semantics of the reasoning steps, in which computation units are represented in the processing language, wherein the memory is configured to store the representations in the processing language, the method including the steps of:

(i) the processor answering the question using the reasoning steps, the computation units and the semantic nodes, and (ii) the processor storing an answer to the question in the memory.

Advantages include the advantages of the previous aspect of the disclosure.

The method may be one wherein the question is represented in the processing language with a passage comprising a semantic node that identifies the passage as a question, a list of zero, one or more semantic nodes representing unknown entities being asked about and at least one further passage which represents the semantics of the question in the context of the zero, one or more unknown entities.

The method may be one wherein the unknowns in the question are identified and the passage making up the body of the question is selected for further analysis; processing begins on a list of passages from the body of the question and the selected unknowns; a first passage in the list of passages is selected for processing; processing a single passage comprises three methods: using statically stored processing language passages, utilising computation units and utilising processing language generated from reasoning:

in which the first method is to lookup in the passage store if there are any passages that can be directly mapped with the passage being processed; if the passage is exactly the same structure as a passage in the passage store, with all nodes matching other than the unknowns, then the values the unknowns match against are valid results; the second method is to check if any results can be found by executing computation units; it is checked if this passage matches against any passages in a computation unit description; all non-unknown nodes in the passage being processed must match the same nodes in the corresponding position in the computation description or align with a computation input unknown; the unknowns being processed must align to output unknowns in the description; the computation unit is then called to get valid output values for the processed passage's unknowns; the third method is to see if this passage can be proved by applying any reasoning steps; reasoning steps are searched for where a passage in the second half of the reasoning passage can be unified with the passage being processed; all nodes and structure must be equal between the two passages, other than unknowns in the focus passage or the reasoning passage; if a reasoning passage like this is found it means that this reasoning step could be used to prove the passage being processed; a multi-stage process is used to first find any mappings for unknowns in the processed passage when matching with the reasoning passage; secondly, mappings for unknowns used in the reasoning passage are found by mapping with the passage being processed; this mapping can then be applied to the front half of the reasoning passage to generate a list of passages that, if they can be matched with known or generated processing language and mappings found for them, will prove and find valid mappings for the focus passage; solutions for the list of passages can then be found recursively.

The method may use a computer system of any aspect of the previous aspect of the disclosure.

Aspects of the disclosure may be combined.

Examples of the present invention include systems and methods for creating and using a structured representation of data that is aimed to be as expressive as natural language in terms of breadth but which also can be processed and understood by automated systems. The representation referred to herein as UL (for universal language) is a preferred example. Examples of the invention include systems and methods relating to specific HUB and other applications as well as systems and methods which process, store and utilize UL.

A natural language question may be any natural language to which a natural language response is appropriate. It could be for example a command or a request for data or even some kind of social interaction or discussion.

Use of UL also br

CLAIMS

Claims ( 30 )

The invention claimed is:

1. A computer-implemented method of interacting with a large language model (LLM), including the steps of:

(a) the LLM processing first input data to the LLM to generate first output from the LLM based on the first input data to the LLM;

(b) a processing system using a structured, machine-readable representation of data that conforms to a machine-readable language, in which semantic nodes are represented in the machine-readable language, the semantic nodes including semantic links between semantic nodes wherein the semantic links are themselves semantic nodes, in which each semantic node denotes one specific meaning, in which a combination of semantic nodes defines a semantic node, in which expressions in the machine-readable language are nestable, in which the first output from the LLM is represented in the machine-readable language, in which reasoning steps are represented in the machine-readable language to represent semantics of the reasoning steps, in which computation units are represented in the machine-readable language;

(c) the processing system verifying the first output from the LLM using the reasoning steps, the computation units and the semantic nodes, and

(d) providing second input data to the LLM, including the verified first output from the LLM in order to generate improved first output from the LLM, wherein the improved first output from the LLM is generated by the LLM in response to the second input data.

2. The method of claim 1 , in which the second input data is a correction and/or a change of continuation text output generated by the LLM.

3. The method of claim 2 , in which the LLM provides output based on the inputted correction and/or change of continuation text output generated by the LLM.

4. The method of claim 1 , in which the second input data is provided as at least part of a prompt to the LLM.

5. The method of claim 1 , in which factual accuracy and/or factual scope of the first output is improved by the second input data.

6. The method of claim 1 , in which internal, logical self-consistency (e.g. correct time ordering of events) of the first output is improved by the second input data.

7. The method of claim 1 , in which correspondence of the first output generated by the LLM to how people understand the real world or reason in the real world is improved by the second input data.

8. The method of claim 1 , in which bias in the first output is reduced by the second input data.

9. The method of claim 1 , in which the second input data provided to the LLM includes dynamic or real-time information.

10. The method of claim 1 , in which the second input data provided to the LLM includes reasoned text, e.g. text derived from a non-statistical reasoning process.

11. The method of claim 1 , in which the LLM is answering a question and the second input data provided to the LLM is an answer to that question.

12. The method of claim 1 , in which second input data text provided to the LLM is labelled with a level of certainty or uncertainty, or trust or lack of trust.

13. The method of claim 1 , in which second input data text provided to the LLM is labelled with a level of brevity.

14. The method of claim 1 , in which second input data text provided to the LLM is labelled with a level of formality.

15. The method of any claim 1 , in which second input data text provided to the LLM is labelled with a use or non-use of profanity.

16. The method of claim 1 , in which second input data text provided to the LLM is labelled with an age or other details of a person being addressed by the LLM.

17. The method of claim 1 , in which the first output from the LLM is at least partially translated to the machine-readable language and is analysed for factual inaccuracies or other contradictions.

18. The method of claim 1 , in which a classifier operates to identify when a prompt is likely to result in a continuation output where accuracy is important, and/or when accuracy is important in the continuation output, and to then use the processing system to improve factual accuracy and/or factual scope of that continuation output.

19. The method of claim 1 , in which the first output from the LLM is a partial continuation, namely an output made before the LLM has stopped generating or whilst the LLM is still generating.

20. The method of claim 1 , when used to improve one or more parameters of the first output of the LLM: factual accuracy and/or factual scope of the first output; internal, logical self-consistency (e.g. correct time ordering of events) of the first output; bias reduction or removal in the first output; inclusion of dynamic or real-time information.

21. The method of claim 1 , when used to improve one or more parameters of the first output of the LLM: level of formality, level of brevity (e.g. briefer when the language is to be spoken), suitability for speaking via a text to speech system, other style language, level of certainty.

22. The method of claim 1 , in which the LLM is a generative AI based system.

23. The method of claim 1 , in which the LLM is an autoregressive language model, such as a Generative Pre-trained Transformer.

24. The method of claim 1 , when used for any of the following: generation of program code, solution of any problem that can be described in natural language, generation of poetry, lyrics, creative writing, generation of other forms of writing such as essays, summaries of knowledge, summaries of longer texts, essays, scientific papers; question answering; internet search.

25. The method of claim 1 , including identifying output content that breaches predefined policies by translating the first output into the machine-readable language before it is displayed to the user and then checking the machine-readable language representation against the predefined policies.

26. The method of claim 1 , including the step of training the LLM on output from a processing system using a structured, machine-readable representation of data that conforms to a machine-readable language, such as a universal language.

27. The method of claim 1 , including a computer implemented method of augmenting search results, including the steps of:

(i) receiving a search query;

(ii) an (e.g. internet) search engine processing the search query to generate (e.g. internet) search results;

(iii) transforming the search results into a structured, machine-readable representation of data that conforms to a machine-readable language, such as a universal language;

(iv) processing the transformed search results using a processing system which uses the structured, machine-readable representation of data that conforms to the machine-readable language, such as a universal language, to produce output;

(v) supplying the output of step (iv) as input to the large language model (LLM), and the LLM generating continuation data output in response to the input.

28. The method of claim 27 , including the step of outputting to a user interface the LLM generated continuation data output.

29. The method of claim 1 , wherein the improved first output from the LLM is continuation text output.

30. The method of claim 1 , wherein the machine-readable language is a universal language.

US18/301,560

2021-08-24

2023-04-17

Computer implemented methods for the automated analysis or use of data, including use of a large language model

Active

US11989527B2

( en )

Priority Applications (1)

Application Number

Priority Date

Filing Date

Title

US18/301,560

US11989527B2

( en )

2021-08-24

2023-04-17

Computer implemented methods for the automated analysis or use of data, including use of a large language model

Applications Claiming Priority (16)

Application Number

Priority Date

Filing Date

Title

PCT/GB2021/052196

WO2022043675A2

( en )

2020-08-24

2021-08-24

A computer implemented method for the automated analysis or use of data

GB2202347

2022-02-22

GB2202347.7

2022-02-22

GBGB2202347.7A

GB202202347D0

( en )

2022-02-22

Unlikely ai 2nd gen

US202218001368A

2022-12-09

GB2219268

2022-12-20

GBGB2219268.6A

GB202219268D0

( en )

2022-12-20

Unlikely ai 2nd gen ii

GB2219268.6

2022-12-20

GBGB2300624.0A

GB202300624D0

( en )

2023-01-16

Unlikely ai 2nd gen iii

GB2300624.0

2023-01-16

GB2300624

2023-01-16

GB2302085.2

2023-02-14

GBGB2302085.2A

GB202302085D0

( en )

2023-02-14

Unlikely AI 2nd gen IV

GB2302085

2023-02-14

PCT/GB2023/050405

WO2023161630A1

( en )

2022-02-22

2023-02-22

Computer implemented methods for the automated analysis or use of data, including use of a large language model

US18/301,560

US11989527B2

( en )

2021-08-24

2023-04-17

Computer implemented methods for the automated analysis or use of data, including use of a large language model

Related Parent Applications (1)

Application Number

Title

Priority Date

Filing Date

PCT/GB2023/050405

Continuation

WO2023161630A1

( en )

2021-08-24

2023-02-22

Computer implemented methods for the automated analysis or use of data, including use of a large language model

Publications (2)

Publication Number

Publication Date

US20230274094A1

US20230274094A1 ( en )

2023-08-31

US11989527B2

true

US11989527B2 ( en )

2024-05-21

Family

ID=87766943

Family Applications (1)

Application Number

Title

Priority Date

Filing Date

US18/301,560

Active

US11989527B2

( en )

2021-08-24

2023-04-17

Computer implemented methods for the automated analysis or use of data, including use of a large language model

Country Status (1)

Country

Link

( 1 )

US11989527B2

( en )

Cited By (4)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20250078823A1

( en )

2023-08-28

2025-03-06

Amazon Technologies, Inc.

Natural language processing

US20250187598A1

( en )

2023-12-12

2025-06-12

Mobileye Vision Technologies Ltd.

Systems and methods for navigating a vehicle using coast control

US12368931B1

( en )

2025-04-04

2025-07-22

Lumana Inc.

Multimedia content management using reduced representations

US12541488B2

( en )

2024-05-23

2026-02-03

AtomBeam Technologies Inc.

System and method for generating thoughts with large language models using codewords

Families Citing this family (72)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20240146734A1

( en )

2017-06-24

2024-05-02

Crowdstrike, Inc.

Large language model-based authentication

US11984113B2

( en )

2020-10-06

2024-05-14

Direct Cursus Technology L.L.C

Method and server for training a neural network to generate a textual output sequence

US12423507B2

( en )

2021-07-12

2025-09-23

International Business Machines Corporation

Elucidated natural language artifact recombination with contextual awareness

US12130852B1

( en )

2022-05-19

2024-10-29

Two Six Labs, LLC

Multi-stage narrative analysis

US12299404B2

( en )

2022-08-22

2025-05-13

Rohirrim, Inc.

Computer-generated content based on text classification, semantic relevance, and activation of deep learning large language models

US20240095077A1

( en )

2022-09-20

2024-03-21

Nvidia Corporation

Prompt generator for use with one or more machine learning processes

US20240296295A1

( en )

2023-03-03

2024-09-05

Microsoft Technology Licensing, Llc

Attribution verification for answers and summaries generated from large language models (llms)

US20240370709A1

( en )

2023-05-01

2024-11-07

C3.Ai, Inc.

Enterprise generative artificial intelligence anti-hallucination and attribution architecture

US12468899B2

( en )

2023-05-08

2025-11-11

Adobe Inc.

Hallucination prevention for natural language insights

US20250005060A1

( en )

2023-06-28

2025-01-02

Jpmorgan Chase Bank, N.A.

Systems and methods for runtime input and output content moderation for large language models

US20250023930A1

( en )

2023-07-14

2025-01-16

Google Llc

Artificial intelligence (ai)-based document retrieval during a virtual meeting

US12511487B2

( en )

2023-07-26

2025-12-30

Micro Focus Llc

Augmented question and answer (Q and A) with large language models

US12130848B1

( en )

2023-08-04

2024-10-29

Kore.Ai, Inc.

Systems and methods for conversation orchestration using trained large language models

US12174906B1

( en )

2023-08-24

2024-12-24

Tiny Fish Inc.

Utilizing a query response to automate a task associated with a webpage

US12236216B1

( en )

2023-08-24

2025-02-25

Tiny Fish Inc.

Generate a script to automate a task associated with a webpage

US12450505B2

( en )

2023-08-24

2025-10-21

Tiny Fish Inc.

Utilizing large language model responses to train an inference pattern engine

US12572855B1

( en )

2023-08-28

2026-03-10

Dmitry Shapiro

Systems and methods for refining machine learning models based on user responses to prompts

US12608370B2

( en )

2023-08-28

2026-04-21

Wiz, Inc.

System and method for natural language query processing utilizing language model techniques

US12001550B1

( en )

2023-08-28

2024-06-04

Wiz, Inc.

Cybersecurity incident response techniques utilizing artificial intelligence

US12493615B2

( en )

2023-08-28

2025-12-09

Wiz, Inc.

System and method for improving efficiency in natural language query processing utilizing language model

US20250077659A1

( en )

2023-08-31

2025-03-06

Dell Products L.P.

Managing challenges regarding impact of poisoned inferences on inference consumers

US20250086952A1

( en )

2023-09-08

2025-03-13

Kneron (Taiwan) Co., Ltd.

Method of edge-cloud fusion-aware visual prompt large language model

US12530112B2

( en )

2023-09-11

2026-01-20

Salesforce, Inc.

Generating virtual space headers utilizing machine-learned models

TWI844473B

( en )

2023-09-11

2024-06-01

å¡ç±³ç¾è¡ä»½æéå¬å¸

System and method for processing confidential data using online large language model

US12238213B1

( en )

2023-09-12

2025-02-25

Portal AI Inc.

Methods and systems for verifying a worker agent

US12093965B1

( en )

2023-09-13

2024-09-17

Dmitry Shapiro

Systems and methods for generating response records for individual users based on user responses to prompts

WO2025064722A1

( en )

2023-09-19

2025-03-27

Ge Xiaolin

Systems and methods for hybrid artificial intelligence enhancement and optimization

WO2025064743A1

( en )

2023-09-21

2025-03-27

Antonick Robin

Systems and methods for setting price points

JP2025049209A

( en )

2023-09-21

2025-04-03

ã½ãããã³ã¯ã°ã«ã¼ãæ ªå¼ä¼ç¤¾

system

WO2025071376A1

( en )

2023-09-27

2025-04-03

ìë²ìì¤ ì£¼ìíì¬

Language model-based prescription information provision method and system

US11922143B1

( en )

2023-10-05

2024-03-05

Dmitry Shapiro

Systems and methods for providing a user interface that facilitates application development

US12316715B2

( en )

2023-10-05

2025-05-27

The Toronto-Dominion Bank

Dynamic push notifications

US12614025B2

( en )

2023-10-05

2026-04-28

Bank Of America Corporation

Authorship source analysis for large language models (LLM) using a distributed ledger

US20250117381A1

( en )

2023-10-10

2025-04-10

Google Llc

Utilizing large language model (llm) in responding to multifaceted queries

CN117216226B

( en )

2023-10-23

2026-02-10

ç§å¤§è®¯é£è¡ä»½æéå¬å¸

A knowledge location method, apparatus, storage medium, and device

CN119883530A

( en )

2023-10-24

2025-04-25

åä¸ºäºè®¡ç®ææ¯æéå¬å¸

Interface calling method and related equipment

CN117112777B

( en )

2023-10-24

2024-01-26

åäº¬ç¿ä¼ä¿¡æ¯ç§ææéå¬å¸

LLM-based multitasking data processing method and storage medium

EP4569416A4

( en )

2023-10-25

2025-12-03

Sas Inst Inc

METHOD AND ARCHITECTURE FOR SECURING INTERACTIONS WITH A DATA CATALOGUE WITH THE SUPPORT OF LARGE LANGUAGE MODELS

CN118657125A

( en )

2023-11-21

2024-09-17

åäº¬åè·³ç½ç»ææ¯æéå¬å¸

Information display method, device, electronic device and storage medium

US12511497B1

( en )

2023-11-30

2025-12-30

Amazon Technologies, Inc.

Embedding-based large language model tuning

US12602572B2

( en )

2023-12-23

2026-04-14

Qomplx Llc

Collaborative generative artificial intelligence content identification and verification

US20250209053A1

( en )

2023-12-23

2025-06-26

Qomplx Llc

Collaborative generative artificial intelligence content identification and verification

CN117540811B

( en )

2024-01-09

2024-04-09

åäº¬å¤§å¦æ·±å³ç ç©¶çé¢

System and method for solving illusion problem of large language model

DE102024100434A1

( en )

2024-01-09

2025-07-10

Krones Aktiengesellschaft

Method and system for analyzing and outputting data from a machine line

EP4586134A1

( en )

2024-01-12

2025-07-16

Asapp, Inc.

Automated response system with api calls and human agent interaction via language model prompts

CN118377893B

( en )

2024-01-19

2025-01-14

ä¸æµ·ç®æ³åæ°ç ç©¶é¢

Frame and method for generating full text of news comment articles based on large language model of RAG

US20250245421A1

( en )

2024-01-25

2025-07-31

Shopify Inc.

System and Method for Modifying Textual Content

US20250245665A1

( en )

2024-01-29

2025-07-31

Actimize Ltd

Fraud risk analysis system incorporating a large language model

WO2025171412A2

( en )

2024-02-09

2025-08-14

Mayo Foundation For Medical Education And Research

Fact checking llm outputs curing hallucinations using ai overread

US12452126B2

( en )

2024-02-13

2025-10-21

T-Mobile Usa, Inc.

Provisioning flow troubleshooting tool

US20250260655A1

( en )

2024-02-14

2025-08-14

Microsoft Technology Licensing, Llc

Automated text-to-optimization routing

KR20250134256A

( en )

2024-03-04

2025-09-11

ì¼ì±ì ìì£¼ìíì¬

Electronic apparatus and control method thereof

US20250298608A1

( en )

2024-03-21

2025-09-25

Prewitt Ridge, Inc.

Systems and Methods for Committing Textual Notation File of Complex Hardware Systems Model in Post-Cloud Engineering Data Management Infrastructure

US12113934B1

( en )

2024-04-04

2024-10-08

Fmr Llc

Systems and methods for intelligent call agent evaluations

US12549500B2

( en )

2024-04-25

2026-02-10

Kyndryl, Inc.

Topical and contextual content filter for large language model chatbots

CN118487766B

( en )

2024-05-06

2025-06-24

å½ç½æ²³åççµåå¬å¸ç»æµææ¯ç ç©¶é¢

Trusted mixed marking index generation method based on digital signature

KR20250165935A

( en )

2024-05-20

2025-11-27

ë¼ì¸íë¬ì¤ ì£¼ìíì¬

Method and system for prompt tuning for evaluation of artificial intelligence model

US12524449B2

( en )

2024-05-23

2026-01-13

Shutterstock, Inc.

Chat-powered search using artificial intelligence

US12271696B1

( en )

2024-05-23

2025-04-08

AtomBeam Technologies Inc.

System and method for training and operating large language models using codewords

KR102734282B1

( en )

2024-07-25

2024-11-26

ìë í¸ì¤í ì£¼ìíì¬

The Method That Generate Questions To Evaluate A System Using A Deep Learning-Based Generative Model

US20260080181A1

( en )

2024-09-18

2026-03-19

Truist Bank

Data processing systems facilitating natural language processing for conversational data

US20260086986A1

( en )

2024-09-23

2026-03-26

Aravo Solutions, Inc.

Utilizing a large language model to generate a computing structure

US12399889B1

( en )

2024-09-23

2025-08-26

Aravo Solutions, Inc.

Modifying a database using a large language model

US12423265B1

( en )

2024-09-23

2025-09-23

Aravo Solutions, Inc.

Prompting a large language model for vector embeddings and metadata to generate an indexed computing file

US12367213B1

( en )

2024-09-23

2025-07-22

Aravo Solutions, Inc.

Generating structured data from unstructured data using metadata from a large language model

US12436966B1

( en )

2024-09-23

2025-10-07

Aravo Solutions, Inc.

Utilizing a large language model to generate a computing structure

CN121833097A

( en )

2024-10-08

2026-04-10

åäº¬åè·³ç½ç»ææ¯æéå¬å¸

Method, apparatus, device, storage medium and program product for task processing

US12273381B1

( en )

2024-11-12

2025-04-08

HiddenLayer, Inc.

Detection of machine learning model attacks obfuscated in unicode

US12278836B1

( en )

2024-11-12

2025-04-15

HiddenLayer, Inc.

Canonicalization of unicode prompt injections

KR102816250B1

( en )

2024-12-20

2025-06-04

(ì£¼)ì ìí¼

Training data augmentation and inspection system using deep learning model

US12619831B1

( en )

2025-04-14

2026-05-05

Charles DeLisi

Systems and methods for generating deliberative committees of age-stratified large language models

CN121477856B

( en )

2026-01-08

2026-04-14

å¥¥ç¹é·æºè½ç§æ(åäº¬)æéå¬å¸

Multi-dimensional intention control test system and method based on large language model

Citations (235)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US4974191A

( en )

1987-07-31

1990-11-27

Syntellect Software Inc.

Adaptive natural language computer interface system

US5386556A

( en )

1989-03-06

1995-01-31

International Business Machines Corporation

Natural language analyzing apparatus and method

US6085186A

( en )

1996-09-20

2000-07-04

Netbot, Inc.

Method and system using information written in a wrapper description language to execute query on a network

US20020173971A1

( en )

2001-03-28

2002-11-21

Stirpe Paul Alan

System, method and application of ontology driven inferencing-based personalization systems

US20030130976A1

( en )

1998-05-28

2003-07-10

Lawrence Au

Semantic network methods to disambiguate natural language meaning

US20040054626A1

( en )

2002-09-17

2004-03-18

Adn Technologies

Device for processing local accounts data with different formats, equipment and a method for treating associated data

US20040078756A1

( en )

2002-10-15

2004-04-22

Napper Jonathon Leigh

Method of improving recognition accuracy in form-based data entry systems

US20040117189A1

( en )

1999-11-12

2004-06-17

Bennett Ian M.

Query engine for processing voice based queries including semantic decoding

US20040174976A1

( en )

2003-03-07

2004-09-09

Nortel Networks Limited

User controlled call routing for multiple telephony devices

US7085708B2

( en )

2000-09-23

2006-08-01

Ravenflow, Inc.

Computer system with natural language to machine language translator

US20070043708A1

( en )

2000-11-28

2007-02-22

Semscript Ltd.

Knowledge storage and retrieval system and method

US20070055656A1

( en )

2005-08-01

2007-03-08

Semscript Ltd.

Knowledge repository

US20070094224A1

( en )

1998-05-28

2007-04-26

Lawrence Au

Method and system for determining contextual meaning for network search applications

US7231633B2

( en )

2003-07-15

2007-06-12

Lsi Corporation

Debugging with set verbosity level during read and analysis of executable code and associated comments while logging according to set verbosity level

US20070136222A1

( en )

2005-12-09

2007-06-14

Microsoft Corporation

Question and answer architecture for reasoning and clarifying intentions, goals, and needs from contextual clues and content

US20070197882A1

( en )

2006-02-17

2007-08-23

Medred, Llc

Integrated method and system for diagnosis determination

AU2003266850B2

( en )

2002-10-15

2007-11-29

Silverbrook Research Pty Ltd

Method of improving recognition accuracy in form-based data entry systems

US20080033987A1

( en )

2006-08-02

2008-02-07

Carter Joe K

Primenet data management system

US20080046250A1

( en )

2006-07-26

2008-02-21

International Business Machines Corporation

Performing a safety analysis for user-defined voice commands to ensure that the voice commands do not cause speech recognition ambiguities

US20080065974A1

( en )

2006-09-08

2008-03-13

Tom Campbell

Template-based electronic presence management

US20080097748A1

( en )

2004-11-12

2008-04-24

Haley Systems, Inc.

System for Enterprise Knowledge Management and Automation

US7418443B2

( en )

2005-06-03

2008-08-26

Fuji Xerox Co., Ltd.

Question answering system, data search method, and computer program

US20080319735A1

( en )

2007-06-22

2008-12-25

International Business Machines Corporation

Systems and methods for automatic semantic role labeling of high morphological text for natural language processing applications

US20090024590A1

( en )

2007-03-15

2009-01-22

Sturge Timothy

User contributed knowledge database

US20090106612A1

( en )

2007-10-17

2009-04-23

Yogesh Pandey

Enhancing speed of simulation of an ic design while testing scan circuitry

US20090192968A1

( en )

2007-10-04

2009-07-30

True Knowledge Ltd.

Enhanced knowledge repository

US7636697B1

( en )

2007-01-29

2009-12-22

Ailive Inc.

Method and system for rapid evaluation of logical expressions

US20100054154A1

( en )

2008-07-11

2010-03-04

Lambert Paul A

Ip assignment scheme for dynamic peer-to-peer networks

US20100121839A1

( en )

2007-03-15

2010-05-13

Scott Meyer

Query optimization

US20100174692A1

( en )

2007-03-15

2010-07-08

Scott Meyer

Graph store

US20100205167A1

( en )

2009-02-10

2010-08-12

True Knowledge Ltd.

Local business and product search system and method

US20100228724A1

( en )

2009-03-09

2010-09-09

Jonah Petri

Search capability implementation for a device

US20100235162A1

( en )

2009-03-16

2010-09-16

Xerox Corporation

Method to preserve the place of parentheses and tags in statistical machine translation systems

US20100306054A1

( en )

2009-05-28

2010-12-02

Drake Robert A

… the automated analysis or use of data, including use of a large language model — Unlikely Artificial Intelligence Limited (US11989527B2)

Related documents