Supplementation of large language model knowledge via prompt modification — Sap Se (US20250094707A1)

ABSTRACT

Abstract

The present disclosure provides techniques and solutions for automatically and dynamically supplementing user prompts to large language models with information to be used by the large language model in formulating a response. In particular, entities are identified in the original prompt. A semantic framework is searched for information about such entities, and such information is added to the original user prompt to provide a modified user prompt. In a particular example, the identified entities comprise triples, and verbalized triples are added to provide the modified user prompt. The modified prompt may be hidden from the user, so that a response of the large language model appears to be in response to the original prompt.

Description

FIELD

The present disclosure generally relates to interactions with large language models. Particular implementations relate to searching for information relevant to a prompt and modifying the prompt to include such information prior to submitting the prompt to a large language model.

BACKGROUND

Large language models are a revolutionary technology rapidly integrating into the daily lives of millions of people. These models, often referred to as “chatbots,” possess the remarkable ability to process and comprehend natural human language input. They can then generate responses in the same fluid human language, making interactions with them highly accessible. The user-friendly nature of these models, which facilitate effortless input and deliver understandable responses, combined with their remarkable accuracy, contributes to their exceptional power and ease of adoption.

Nonetheless, large language models do face certain challenges. One such challenge involves their accuracy being contingent on relevant training data. If the most pertinent data for a specific query is not publicly available, such as residing within corporate information repositories, the model may either fail to provide an answer or provide an answer that is incorrect—sometimes referred to as “hallucinating.” Further, considering the time and resources required for training, these models may not be updated frequently. Consequently, an information gap might arise between the model's last training instance and its actual usage. Accordingly, room for improvement exists.

SUMMARY

This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

The present disclosure provides techniques and solutions for automatically and dynamically supplementing user prompts to large language models with information to be used by the large language model in formulating a response. In particular, entities are identified in the original prompt. A semantic framework is searched for information about such entities, and such information is added to the original user prompt to provide a modified user prompt. In a particular example, the identified entities comprise triples, and verbalized triples are added to provide the modified user prompt. The modified prompt may be hidden from the user, so that a response of the large language model appears to be in response to the original prompt.

In one aspect, the present disclosure provides a process of modifying user input to a large language model to include supplemental facts for use by the large language model prior to submitting the user input to the large language model. User input is received from a user. The user input includes a plurality of tokens. At least a portion of the plurality of tokens are analyzed. Based on the analyzing, one or more entities of a semantic framework are determined that are represented in the at least a portion of the plurality of tokens. One or more triples of the semantic framework are determined for at least a portion of the one or more entities or for associated entities. At least a portion of the one or more triples, or a representation thereof, is added to the user input to provide modified user input. The modified user input is submitted to a large language model. The modified user input is processed using the large language model to provide a response. The response is returned in response to the receiving the user input.

The present disclosure also includes computing systems and tangible, non-transitory computer readable storage media configured to carry out, or including instructions for carrying out, an above-described method. As described herein, a variety of other features and advantages can be incorporated into the technologies as desired.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an exemplary directed graph.

FIG. 2 shows the domain and range of a property in a schema of a directed graph.

FIG. 3 shows an exemplary SPARQL query and results of the query.

FIG. 4 shows a block diagram notation for subsequent figures.

FIG. 5 shows Crow's Foot notation for annotation in subsequent figures.

FIG. 6 shows an exemplary system architecture for a method according to some embodiments.

FIG. 7 shows an algorithm for computing strings from a directed graph for input to a language model.

FIG. 8 shows a sorting algorithm for sorting statements of a directed graph.

FIG. 9 shows a directed graph that may be provided as input to the sorting algorithm.

FIG. 10 shows a relationship between a condition and one or more patterns.

FIG. 11 is a flowchart of a process of creating a domain data model, including mapping elements of the domain data model to elements of a core data model.

FIG. 12 shows the algorithm of FIG. 7 expanded to handle question patterns.

FIG. 13 shows an implementation of the method for providing data from a directed graph to a language model implemented as a remote service.

FIG. 14 shows a hybrid implementation of the method for providing data from a directed graph to a language model.

FIG. 15 shows an on-premises implementation of the method for providing data from a directed graph to a language model.

FIG. 16 illustrates results provided by a large language model when the model lacks sufficient information to properly respond to user input. FIG. 16 also illustrates how disclosed innovations can provide the large language model with additional information to be used in responding to the user input, and a response produced thereby.

FIG. 17 is a flowchart of a process of determining information relevant to user input to a large language model, supplementing the user input with such information, and linking an answer to the supplemented user input to entities in a knowledge graph.

FIG. 18 A illustrates how elements of modified user input can be mapped to: the original user input, facts retrieved from a search for information about entities identified in the original user input, and additional instructions that can be added to the original user input prior to submitting the modified user input to a large language model.

FIGS. 18 B and 18 C illustrate user interface screens that display information for a recognized entity in a response of a large language model in response to user selection of such an entity.

FIG. 19 illustrates an example knowledge graph query and example subject, object, predicate triples produced therefrom.

FIG. 20 illustrates an example function to verbalize knowledge graph triples, and example verbalized triples that can be generated using the knowledge graph triples of FIG. 19 .

FIG. 21 is an example knowledge graph.

FIGS. 22 A- 22 C provide example code for querying a knowledge graph.

FIG. 23 illustrates an example computing environment in which disclosed techniques can be implemented.

FIG. 24 is a flowchart of a process of modifying user input to a large language model to include supplemental facts for use by the large language model prior to submitting the user input to the large language model.

FIG. 25 is a diagram of an example computing system in which some described embodiments can be implemented.

FIG. 26 is an example cloud computing environment that can be used in conjunction with the technologies described herein.

DETAILED DESCRIPTION

Example 1—Overview

Large language models are a revolutionary technology rapidly integrating into the daily lives of millions of people. These models, often referred to as “chatbots,” possess the remarkable ability to process and comprehend natural human language input. They can then generate responses in the same fluid human language, making interactions with them highly accessible. The user-friendly nature of these models, which facilitate effortless input and deliver understandable responses, combined with their remarkable accuracy, contributes to their exceptional power and ease of adoption.

Nonetheless, large language models do face certain challenges. One such challenge involves their accuracy being contingent on relevant training data. If the most pertinent data for a specific query is not publicly available, such as residing within corporate information repositories, the model may either fail to provide an answer or provide an answer that is incorrect—sometimes referred to as “hallucinating.” Further, considering the time and resources required for training, these models may not be updated frequently. Consequently, an information gap might arise between the model's last training instance and its actual usage. Accordingly, room for improvement exists.

The present disclosure provides techniques that can be used to influence a result provided by a large language model. For example, the large language model can be provided with a set of “facts” that it can use in whole or part to generate a response. As used in this context, “facts” refers to information that the large language model will use in preparing a response, regardless of whether the information is actually true or not. In some cases, the large language model can be told to explicitly assume that the “facts” are true, such as to bypass functionality of the large language model that might otherwise question the veracity of the information. Similarly, if desired, the large language model can be told to provide a response only using the explicitly provided facts.

The present disclosure provides a technique where a set of facts are determined from an initial user input, and added to the user input to provided modified user input, and where the large language model will respond to the modifier user input. That is, the initial user input can be “intercepted” and modified before submission to a large language model. The user input includes a plurality of tokens that can be analyzed. As used herein, a “token” of user input refers to a discrete unit of text or character sequence within the user input, including tokens that can be analyzed by a named entity recognition service. Tokenization can be achieved by scanning a text and segmenting it at specific boundaries, such as spaces or punctuation marks, to isolate distinct units, or tokens, which can then be subjected to various linguistic analyses or computational operations.

In a specific implementation, the user input is provided to a named entity recognition service that accesses one or more knowledge graphs. Information from the knowledge graphs can be the basis of the additional facts that are provided in the modified user input. In a specific example, information from the knowledge graphs can be “verbalized” before being added to the user input. That is, large language models are designed to work with normal, human language input. Converting “raw” results from a search of a knowledge graph into a verbalized form, even a simple form, can improve the results obtained from the large language model using the modified user input.

The knowledge graph can also be used to enhance the usefulness of a response. For example, a response can be submitted to a named entity recognition service, and the response can be linked to particular information in the knowledge graph (or information outside of the knowledge graph that was identified using the knowledge graph, such as by performing a query of another data source, using results from searching the knowledge graph for entities identified by the named entity recognition service).

Example 2 provides a general discussion of knowledge graphs, as well as a specific technique that can be used to convert elements retrieved from a knowledge graph into a verbalized format. Examples 3-9 describe various techniques according to the present disclosure for modifying user input with content that may not be known to a large language model. The described innovations can be used in various other ways, for example, a set of “stipulated facts” can be provided to the large language model regardless of whether the information may be part of the training corpus of the large language model.

While disclosed techniques are generally described with reference to knowledge graphs as the source of additional information to be added to a prompt, and used by a large language model in formulating a response, these techniques can use information maintained in another format.

FIELD

The present disclosure generally relates to interactions with large language models. Particular implementations relate to searching for information relevant to a prompt and modifying the prompt to include such information prior to submitting the prompt to a large language model.

BACKGROUND

Large language models are a revolutionary technology rapidly integrating into the daily lives of millions of people. These models, often referred to as “chatbots,” possess the remarkable ability to process and comprehend natural human language input. They can then generate responses in the same fluid human language, making interactions with them highly accessible. The user-friendly nature of these models, which facilitate effortless input and deliver understandable responses, combined with their remarkable accuracy, contributes to their exceptional power and ease of adoption.

Nonetheless, large language models do face certain challenges. One such challenge involves their accuracy being contingent on relevant training data. If the most pertinent data for a specific query is not publicly available, such as residing within corporate information repositories, the model may either fail to provide an answer or provide an answer that is incorrect—sometimes referred to as “hallucinating.” Further, considering the time and resources required for training, these models may not be updated frequently. Consequently, an information gap might arise between the model's last training instance and its actual usage. Accordingly, room for improvement exists.

SUMMARY

This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

The present disclosure provides techniques and solutions for automatically and dynamically supplementing user prompts to large language models with information to be used by the large language model in formulating a response. In particular, entities are identified in the original prompt. A semantic framework is searched for information about such entities, and such information is added to the original user prompt to provide a modified user prompt. In a particular example, the identified entities comprise triples, and verbalized triples are added to provide the modified user prompt. The modified prompt may be hidden from the user, so that a response of the large language model appears to be in response to the original prompt.

In one aspect, the present disclosure provides a process of modifying user input to a large language model to include supplemental facts for use by the large language model prior to submitting the user input to the large language model. User input is received from a user. The user input includes a plurality of tokens. At least a portion of the plurality of tokens are analyzed. Based on the analyzing, one or more entities of a semantic framework are determined that are represented in the at least a portion of the plurality of tokens. One or more triples of the semantic framework are determined for at least a portion of the one or more entities or for associated entities. At least a portion of the one or more triples, or a representation thereof, is added to the user input to provide modified user input. The modified user input is submitted to a large language model. The modified user input is processed using the large language model to provide a response. The response is returned in response to the receiving the user input.

The present disclosure also includes computing systems and tangible, non-transitory computer readable storage media configured to carry out, or including instructions for carrying out, an above-described method. As described herein, a variety of other features and advantages can be incorporated into the technologies as desired.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an exemplary directed graph.

FIG. 2 shows the domain and range of a property in a schema of a directed graph.

FIG. 3 shows an exemplary SPARQL query and results of the query.

FIG. 4 shows a block diagram notation for subsequent figures.

FIG. 5 shows Crow's Foot notation for annotation in subsequent figures.

FIG. 6 shows an exemplary system architecture for a method according to some embodiments.

FIG. 7 shows an algorithm for computing strings from a directed graph for input to a language model.

FIG. 8 shows a sorting algorithm for sorting statements of a directed graph.

FIG. 9 shows a directed graph that may be provided as input to the sorting algorithm.

FIG. 10 shows a relationship between a condition and one or more patterns.

FIG. 11 is a flowchart of a process of creating a domain data model, including mapping elements of the domain data model to elements of a core data model.

FIG. 12 shows the algorithm of FIG. 7 expanded to handle question patterns.

FIG. 13 shows an implementation of the method for providing data from a directed graph to a language model implemented as a remote service.

FIG. 14 shows a hybrid implementation of the method for providing data from a directed graph to a language model.

FIG. 15 shows an on-premises implementation of the method for providing data from a directed graph to a language model.

FIG. 16 illustrates results provided by a large language model when the model lacks sufficient information to properly respond to user input. FIG. 16 also illustrates how disclosed innovations can provide the large language model with additional information to be used in responding to the user input, and a response produced thereby.

FIG. 17 is a flowchart of a process of determining information relevant to user input to a large language model, supplementing the user input with such information, and linking an answer to the supplemented user input to entities in a knowledge graph.

FIG. 18 A illustrates how elements of modified user input can be mapped to: the original user input, facts retrieved from a search for information about entities identified in the original user input, and additional instructions that can be added to the original user input prior to submitting the modified user input to a large language model.

FIGS. 18 B and 18 C illustrate user interface screens that display information for a recognized entity in a response of a large language model in response to user selection of such an entity.

FIG. 19 illustrates an example knowledge graph query and example subject, object, predicate triples produced therefrom.

FIG. 20 illustrates an example function to verbalize knowledge graph triples, and example verbalized triples that can be generated using the knowledge graph triples of FIG. 19 .

FIG. 21 is an example knowledge graph.

FIGS. 22 A- 22 C provide example code for querying a knowledge graph.

FIG. 23 illustrates an example computing environment in which disclosed techniques can be implemented.

FIG. 24 is a flowchart of a process of modifying user input to a large language model to include supplemental facts for use by the large language model prior to submitting the user input to the large language model.

FIG. 25 is a diagram of an example computing system in which some described embodiments can be implemented.

FIG. 26 is an example cloud computing environment that can be used in conjunction with the technologies described herein.

DETAILED DESCRIPTION

Example 1—Overview

Large language models are a revolutionary technology rapidly integrating into the daily lives of millions of people. These models, often referred to as “chatbots,” possess the remarkable ability to process and comprehend natural human language input. They can then generate responses in the same fluid human language, making interactions with them highly accessible. The user-friendly nature of these models, which facilitate effortless input and deliver understandable responses, combined with their remarkable accuracy, contributes to their exceptional power and ease of adoption.

Nonetheless, large language models do face certain challenges. One such challenge involves their accuracy being contingent on relevant training data. If the most pertinent data for a specific query is not publicly available, such as residing within corporate information repositories, the model may either fail to provide an answer or provide an answer that is incorrect—sometimes referred to as “hallucinating.” Further, considering the time and resources required for training, these models may not be updated frequently. Consequently, an information gap might arise between the model's last training instance and its actual usage. Accordingly, room for improvement exists.

The present disclosure provides techniques that can be used to influence a result provided by a large language model. For example, the large language model can be provided with a set of “facts” that it can use in whole or part to generate a response. As used in this context, “facts” refers to information that the large language model will use in preparing a response, regardless of whether the information is actually true or not. In some cases, the large language model can be told to explicitly assume that the “facts” are true, such as to bypass functionality of the large language model that might otherwise question the veracity of the information. Similarly, if desired, the large language model can be told to provide a response only using the explicitly provided facts.

The present disclosure provides a technique where a set of facts are determined from an initial user input, and added to the user input to provided modified user input, and where the large language model will respond to the modifier user input. That is, the initial user input can be “intercepted” and modified before submission to a large language model. The user input includes a plurality of tokens that can be analyzed. As used herein, a “token” of user input refers to a discrete unit of text or character sequence within the user input, including tokens that can be analyzed by a named entity recognition service. Tokenization can be achieved by scanning a text and segmenting it at specific boundaries, such as spaces or punctuation marks, to isolate distinct units, or tokens, which can then be subjected to various linguistic analyses or computational operations.

In a specific implementation, the user input is provided to a named entity recognition service that accesses one or more knowledge graphs. Information from the knowledge graphs can be the basis of the additional facts that are provided in the modified user input. In a specific example, information from the knowledge graphs can be “verbalized” before being added to the user input. That is, large language models are designed to work with normal, human language input. Converting “raw” results from a search of a knowledge graph into a verbalized form, even a simple form, can improve the results obtained from the large language model using the modified user input.

The knowledge graph can also be used to enhance the usefulness of a response. For example, a response can be submitted to a named entity recognition service, and the response can be linked to particular information in the knowledge graph (or information outside of the knowledge graph that was identified using the knowledge graph, such as by performing a query of another data source, using results from searching the knowledge graph for entities identified by the named entity recognition service).

Example 2 provides a general discussion of knowledge graphs, as well as a specific technique that can be used to convert elements retrieved from a knowledge graph into a verbalized format. Examples 3-9 describe various techniques according to the present disclosure for modifying user input with content that may not be known to a large language model. The described innovations can be used in various other ways, for example, a set of “stipulated facts” can be provided to the large language model regardless of whether the information may be part of the training corpus of the large language model.

While disclosed techniques are generally described with reference to knowledge graphs as the source of additional information to be added to a prompt, and used by a large language model in formulating a response, these techniques can use information maintained in another format. Generally, these techniques retrieve information from a “semantic framework,” where the semantic framework can be a knowledge graph, or can instead use technologies such as ontologies, other types of semantic webs, semantic databases, graph databases, linked data, or taxonomies or folksonomies.

Similarly, aspects of the present disclosure are described as being implemented using the Resource Description Framework. However, other types of data structures or representations can be used that convey equivalent information about semantic relationships or associations, such as expressing two related entities and a relationship between such entities.

Example 2—Example Verbalization of Knowledge Graph Triples

An enterprise may have a variety of different products, services, and teams. The enterprise may also have a comprehensive knowledge graph, storing knowledge regarding skills, processes, experiences, capabilities, and insights that are relied upon in day-to-day operations of the enterprise. Contents of the knowledge graph may also include enterprise specific acronyms, departments of the enterprise, and product specifications. The knowledge may enable the enterprise to react to business situations in a fast, professional, and flexible manner. The knowledge graph may be expensive and labor intensive to construct and maintain. The knowledge graph (i.e., semantic web and/or web of linked data) may be specified using the Resource Description Framework (RDF).

In some cases, a user would like to ask questions of or provide tasks to a language model, e.g., a large language model based on a generative pre-trained transformer, such as ChatGPT. However, the language model is typically trained in an unsupervised manner on unlabeled human readable text. Hence, the language model may be unable to directly process a knowledge graph or use a knowledge graph as input, e.g., for training.

Accordingly, it may be desirable to maximize the usability of the knowledge graph, for example, using the knowledge graph as a basis for artificial intelligence applications, more particularly, to train or otherwise improve a language model. Upon training the language model, the language model may be used to answer questions or carry out tasks based on the knowledge stored in the knowledge graph.

In addition, it may be desirable to extract human readable text from the knowledge graph, e.g., for use in explaining answers provided by software (e.g., a process advisor) relying on the knowledge graph.

According to an aspect, a computer implemented method for providing data from a directed graph to a language model is provided. The method comprises defining a plurality of conditions and a plurality of patterns, wherein each of the conditions has at least one corresponding pattern. The method further comprises receiving a subset of the directed graph, wherein the subset of the directed graph includes a plurality of statements. Each of the statements includes a subject, an object and a predicate relating the subject to the object. For each of the statements in the subset of the directed graph, performing the following: when one of the conditions matches a respective statement and the pattern corresponding to the condition can be applied to the respective statement, computing a string for the respective statement using the pattern. Providing the computed strings as input to the language model.

Providing data from the directed graph to the language model may include extracting or reading the data from the directed graph and feeding or sending the data as input to the language model. The statements may be referred to as triples or triple statements, with subject, predicate and object components. A condition may correspond to a respective pattern in the sense that when the condition is determined to be true for a respective statement, it may be determined whether the respective pattern can be applied to the respective statement.

The clause, each of the conditions has at least one corresponding pattern, may be understood to mean that each of the conditions of the plurality of conditions has at least one corresponding pattern of the plurality of patterns. Hence, each one of the conditions of the plurality of conditions may be assigned at least one pattern of the plurality of patterns. Put another way, when a condition has at least one corresponding pattern, the at least one corresponding pattern is assigned to the condition. Moreover, each one of the conditions of the plurality of conditions may be assigned multiple patterns of the plurality of patterns.

The clause, when one of the conditions matches a respective statement and the pattern corresponding to the condition can be applied to the respective statement, may comprise determining whether the at least one pattern corresponding to the condition can be applied to the respective statement. Accordingly, determining whether the at least one pattern corresponding to the condition can be applied to the respective statement may involve determining whether the pattern corresponding to the condition matches the respective statement, i.e., testing whether the respective statement has the characteristics or elements specified by the pattern.

For example, determining whether a pattern can be applied to a statement including a subject may be carried out as follows:

The plurality of conditions may include a first condition:

<?s> <?p> <?o>. BIND (sap:BusinessActivity AS <?s>) BIND (rdf:type AS <?p>)

A first pattern may correspond to the first condition:

The <s.rdfs:label> is a business activity.

Since the first pattern includes a reference to a label of a subject s, if the statement including the subject fulfills the first condition, i.e., of being a BusinessActivity, but the subject of the statement does not have a label as required by the first pattern, then the first pattern could not be applied (i.e., would not match) the statement.

Computing the string from the respective statement using the pattern may involve directly outputting text of the pattern and matching pattern operators to components of the statement.

Moreover, when computing the string, not just the respective statement matching the condition but one or more further statements of the subset of the directed graph may be accessed by the pattern corresponding to the condition. In other words, the pattern can consider statements that do not match the condition. Accordingly, the condition may trigger pattern execution for the respective statement, which in turn may trigger processing of at least a portion of the subset of the directed graph (or the entire subset of the directed graph) using the pattern.

The subject, the object and the predicate may be referred to as components of their respective statement. The subject and/or the predicate may be an RDF resource (e.g., the subject and/or predicate may have the resource property of RDF, and may be a type or a label). The object may be a literal (e.g., an RDF literal) having a defined data type, such as string, integer, Boolean or double (as defined in the extensible markup language (XML) schema definition language (XSD)). Regarding RDF, please refer to the RDF specification, “Concepts and Abstract Syntax”, https://www.w3.org/TR/rdf11-concepts/

The computed strings may be provided directly to the language model (e.g., in the case of small strings, such as less than 1 GB) or may be serialized to a text file before being provided to the language model (e.g., in the case of terabytes of data computed from a comprehensive directed graph). In summary, the method accepts a subset of a directed graph as input and generates grammatically correct sentences as the computed strings. The method iterates over the input statements (i.e. triples) in the subset of the directed graph and uses the conditions filters to determine which patterns may be applicable to each of the statements. When a condition matches a statement and the statement has the elements required by a pattern corresponding to (e.g., assigned to) the condition, the pattern is applied to the condition. The result of the iteration may be a set of strings, such that a string is computed for each statement in the input. The syntax and interpretation of patterns is discussed in more detail below.

The statements of the directed graph may be close to human language.

Hence, providing the computed strings of as input to the language model may maximize the usability of the subset of the directed graph, for example, by using the subset of the directed graph as a basis for artificial intelligence applications. Once the language model has processed the computed strings, the language model may be used to answer questions or carry out tasks based on knowledge stored in the subset of the directed graph. Accordingly, the time, labor and expense invested to construct the directed graph may be exploited in further ways (e.g., to answer questions or by using the language model.

In addition or alternatively, it may be desirable to extract human readable text from the subset of the directed graph, e.g., for use in explaining answers provided by software (e.g., a process advisor) relying on the subset of the directed graph.

In some cases, each of the conditions includes at least three condition variables (variables appearing in a condition may be referred to as condition variables). Each of the condition variables may correspond to (e.g., store) a different component of a statement. For example, a first one of the condition variables matches the subject, a second one of the condition variables matches the predicate and a third one of the condition variables matches the object. At least one of the condition variables may be bound to at least one value, e.g., to an RDF property. In other words, at least one of the condition variables may specify at least one value (e.g., RDF property) that a component of a statement must have. Each of the condition variables may specify an instance of a class (e.g., an RDF class) or a literal e.g., an RDF literal). An instance of a class may be referred to as an instance. The instance may relate to a specific concept and have a definite article while the class may relate to a generic concept and have an indefinite article.

Each condition may be applied to a statement and may evaluate to TRUE or FALSE. In other words, a condition may return a Boolean value. For example, if the condition evaluates to TRUE, the condition matches the respective statement and it is determined whether the at least one pattern corresponding to the condition can be applied to the respective statement. The following are numbered examples of conditions that may be among the plurality of conditions:

1. <?s> <?p> <?o>. 2. <?s> <?p> <?o>. BIND (rdfs:label AS <?p>) 3. <?s> <?p> <?o>. <?s> rdf:type sap:BusinessActivity. BIND (rdfs:label AS <?p>) 4. <?s> <?p> <?o>. <?s> rdf:type sap:BusinessActivity. <?o> sap:requires <?r>. BIND (rdfs:label AS <?p>)

In the first condition, “<?s>” is a variable corresponding to a subject in the directed graph, “<?p>” is a variable corresponding to a predicate in the directed graph and “<?o>” is a variable corresponding to an object in the directed graph. Hence, the first condition specifies that (i.e., in order for the first condition to evaluate to TRUE) a statement must contain a subject, a predicate and an object. The second condition requires that a statement contains a subject, a predicate and an object and that the predicate is an rdfs:label. The third condition requires that a statement contains a subject, a predicate and an object, that the subject has the property (more specifically, is of type) sap:BusinessActivity and that the predicate is an rdfs:label. The fourth condition requires that a statement contains a subject, a predicate and an object, that the subject has the property (more specifically, is of type) sap:BusinessActivity, that the object has a relation (i.e., a subject-object relation) of “sap: requires” with the object “<?r>” and that the predicate is an rdfs:label. In this connection, “<?r>” is a variable bound to a requirement.

The conditions of the plurality of conditions may function to prevent patterns from being used to compute semantically incorrect strings. In other words, the conditions may be used to ensure that the computed strings are semantically and/or grammatically correct. Accordingly, by assigning patterns to conditions, the cases in which patterns are applied can be limited, thereby ensuring or facilitating computation of semantically correct strings, i.e., sentences. Without conditions, patterns could be applied to compute exemplary fantasy strings such as, “The Harry Potter Book is a business activity.”, or “The Star Wars Movie is a business activity.” However, the exemplary fantasy strings are semantically incorrect; therefore, the exemplary fantasy strings would not be helpful as input to the language model and could prolong the training of the language model or even cause the language model to produce incorrect output.

In some cases, at least one of the conditions has a plurality of corresponding patterns. Accordingly, computing a string from the respective statement using the pattern may comprise computing a plurality of strings from the respective statement using each pattern corresponding to the condition (i.e., the condition matching the respective statement) that can be applied to the respective statement.

Alternatively, computing a string from the respective statement using the pattern may comprise determining a random order of the patterns corresponding to the condition and computing a string from the respective statement only using a first one in the random order of the patterns that can be applied to the respective statement. For example, patterns 1 to 4 may be ordered 2, 4, 3, 1 and pattern 2 can be applied to the respective statement, hence, pattern 2 is applied to the respective statement.

For example, the plurality of conditions may include a sequenceID condition:

<?I1> <SequenceID> <?L1>.

The sequenceID condition may correspond to the following pattern (A):

(A) The sequence identifier of <?I1.rdf:type.rdfs:label> <?I1.rdfs:label> is <?L1>.

Continuing the example, the subset of the directed graph may include the following statements:

1. Emissions Management isA BusinessCapability. 2. BusinessCapability rdfs:label “Business Capability”. 3. EmissionsManagement SequenceID “5”.

The sequenceID condition only matches statement (3), since statement (3) includes a “SequenceID” and statements (1) and (2) do not include a “SequenceID”.

Pattern (A) can be applied to statements (1), (2) and (3) to compute the following string:

The sequence identifier of Business Capability is 5.

All three statements are needed to compute the string above because statements (1) and (2) provide context information for statement (3).

Hence, as indicated above, when computing the string, not just the respective statement matching the condition but one or more further statements of the subset of the directed graph may be accessed by the pattern corresponding to the condition. In other words, the pattern can consider statements that do not match the condition. Accordingly, the condition triggers the pattern execution for the respective statement.

Moreover, the statements of the subset of the directed graph may be iteratively checked. Accordingly, in the example above statements (1) and (2) do not cause the pattern to be triggered but statement (3) does.

In the present example, determining whether pattern (A) can be applied to statement (3) may include determining context information items of pattern (A), namely:

the label of the type of variable <?I1| the label of variable <?I1>

Both context information items are not available in statement (3) itself, but instead elsewhere in the subset of the directed graph.

The contextual information exists in the subset of the directed graph in view of statement (3) and pattern (A) can be executed for statement (3).

Whenever the forward dot notation is used, we include additional information that is not existing in the triple itself.

In some cases, each pattern includes one or more of the following:

at least one variable, wherein the variable specifies (e.g., is bound to) a class, an instance of a class, a literal or a predicate; text, such as one or more articles (e.g., grammatical articles that are definite or indefinite); at least one property that applies to the variable.

Each pattern may further include a language filter. The literal may specify a numeric value or text, where the literal may conform to the RDF schema class of literal values.

Advantageously, the patterns may enable the combination of static text with variables specifying structures (e.g., the subject, the object and the predicate of one of the statements) of the subset of the directed graph, possibly supplemented with information resulting from materializing the subset of the directed graph.

In addition or alternatively, the patterns may include at least one specific pattern (i.e. custom pattern) and a plurality of default patterns. When a condition corresponding to the specific pattern matches a respective statement and the specific pattern can be applied to the respective statement, computing a string from the respective statement using the pattern may comprise using the specific pattern. When the condition corresponding to the specific pattern does not match the respective statement, the method may further comprise determining whether a condition corresponding to one of the default patterns matches the respective statement. When the condition corresponding to one of the default patterns matches the respective statement, computing a string from the respective statement using the one of the default patterns. Defining the plurality of conditions and the plurality of patterns may further comprise defining at least three conditions and at least three patterns, where at least one of the three patterns is a specific pattern and at least one of the three conditions corresponds to the specific pattern.

The following is an exemplary pattern that may be included in the plurality of patterns:

Text <?I1.rdf:type.rdfs:label> Text

The variable <?11> may be bound to sap:PrintReceipt. Patterns, such as the exemplary pattern above, may use forward dot notation (also referred to as dot notation) to refer to a field, component or sub-property of a property. This may provide the patterns with an advantage over conventional SPARQL, which does not support forward dot notation, since forward dot notation enables more compact expressions.

Accordingly, the exemplary pattern above could be applied to (e.g., the subset of the directed graph may include) the following three statements:

sap:PrintReceipt rdf:type sap:Task. sap:Task rdfs:label “Task” @en. sap:Task rdfs:label “Process Task” @en.

In some cases, after a pattern is applied to a respective statement, the pattern is not applied to further statements matching the pattern, i.e., the further statements in the subset of the directed graph matching the pattern may be skipped. The matching of just one statement and skipping of further statements may be an option that can be configured. For example, as discussed below, the post operator may cause a Cartesian product to be computed.

For example, after a pattern is applied to a respective statement including an rdfs:label for an object, further statements including an rdfs:label for the object may be skipped, i.e., the pattern is not applied to the further statements. Accordingly, computing strings from the statements above using the exemplary pattern would yield the following: “Text Task Text”.

At least one of the patterns may include a filter condition and/or a post operator. The filter condition may specify a language. The post operator may cause a Cartesian product to be performed. As another example, the subset of the directed graph may include the following four statements:

sap:PrintReceipt rdf:type sap:Task. sap:Task rdfs:label “Task” @en. sap:Task rdfs:label “Process Task” @en. sap:Task rdfs:label “Schritt” @de.

The following further exemplary pattern may be included in the plurality of patterns and may be applied to the four statements above:

Text <?I1.rdf:type.rdfs:label(lang=‘en’)*> Text

The further exemplary pattern above includes a filter condition to specify a language and an asterisk post operator “*” that yields a cartesian product. The cartesian product may yield all possible combinations of the preceding elements. Accordingly, the following strings would be computed by applying the further exemplary pattern including the cartesian product, since the further exemplary pattern is directed to English labels and there are two English labels among the four statements above:

Text Task Text

Text Process Task Text

Hence, the statement above including “‘Schritt’@de” would not be processed since the statement does not meet the filter condition in the further exemplary pattern (i.e., the statement is not in the English language). Without the asterisk post operator in the further exemplary pattern above, only the first string “Text Task Text” would be computed.

The exemplary pattern and further exemplary pattern above may be specific patterns, i.e., patterns applicable to one directed graph or a group of directed graphs.

The strings may be computed from the respective statements using only default patterns. However, use of the specific patterns may result in computed strings that more accurately and precisely describe the contents of the subset of directed graph.

Other post operators (i.e., operators provided at the end of a pattern, also referred to as postfix operators) in addition to the asterisk may also be used. For example, an additional post operator might limit the output of a cartesian product to a specified number of combinations, e.g., about 10 combinations.

As another example, the following requirement pattern may be applied to the four statements above:

The <?I1.rdf:type.rdfs:label(lang=‘en’)*> <?I1.rdfs:label> requires a <?I2.rdfs:label>.

In this example, <?I1> may be bound to sap:PrintReceipt and <?I2> may be bound to sap: Printer. Hence, by applying the requirement pattern to the four statements above, the following strings may be computed:

The Process Task Print Receipt requires a Printer. Task Print Receipt requires a Printer.

Without the asterisk post operator in the requirement pattern, only the first string, i.e., “The Process Task Print Receipt requires a Printer.” would be computed.

In some cases, each of the computed strings is a grammatically correct sentence, wherein the conditions and/or patterns may ensure that the computed strings are grammatically correct sentences.

In addition or alternatively, the at least one specific pattern may include a plurality of specific patterns. Each of the specific patterns may be applicable to a group of directed graphs defined according to the resource description framework or a group of knowledge graphs defined according to the resource description framework. Each of the default patterns may be applicable to any directed graph defined according to the resource description framework or any knowledge graph defined according to the resource description framework.

Accordingly, each specific pattern may be defined for a single on premises network and a corresponding directed graph, or a group of on premises networks and a corresponding group of directed graphs, whereas default patterns may be applicable to any directed graph.

The patterns may include one or more of the following five patterns:

a pattern applicable to instance-to-instance statements, including variables <I1, p, I2> a pattern applicable to instance-to-class statements, including variables <I1, p, C1> a pattern applicable to class-to-class statements, including variables <C1, p, C2> a pattern applicable to instance-to-literal statements, including variables <I1, p, L1> a pattern applicable to class-to-literal statements, including variables <C1, p, L1>

The five patterns above may be made applicable to instance-to-instance statements, instance-to-class statements, class-to-class statements, instance-to-literal statements and class-to-literal statements via corresponding conditions including the respective variables <I1, p, I2>, <I1, p, C1>, <C1, p, C2>, <I1, p, L1>, <C1, p, L1>.

The five patterns mentioned above may be default patterns, in the sense that they are applicable to any directed graph, or more specifically, any knowledge graph.

For the five patterns mentioned above, “11” and “12” are variables referring to instances (i.e., instances of classes), “C1” and “C2” are variables referring to classes, “L1” is a variable referring to a literal, and “p” is a variable referring to a predicate. A first one of the five patterns applicable to instance-to-instance statements may be implemented as follows:

The <?I1.rdf:type.rdfs:label> <?I1.rdfs:label> <?p.rdfs:label> the <?I2.rdf:type.rdfs:label> <?I2.rdfs:label>.

A second one of the five patterns applicable to instance-to-class statements may be implemented as follows:

The <?I1.rdf:type.rdfs:label> <?I1.rdfs:label> <?p.rdfs:label> a <?C1.rdfs:label>.

A third one of the five patterns applicable to class-to-class statements may be implemented as follows:

A <?C1.rdfs:label> <?p.rdfs:label> a <?C2.rdfs:label>.

A fourth one of the five patterns applicable to instance-to-literal statements may be implemented as follows:

The <?I1.rdfs:label> <?p.rdfs:label> <?L1>.

A fifth one of the five patterns applicable to class-to-literal statements may be implemented as follows:

A <?C1.rdfs:label> <?p> <?L1>.

A user or administrator may define further default patterns or change the exemplary default patterns provided above.

In addition or alternatively, the plurality of patterns may include at least one text pattern and at least one question pattern. The text pattern and/or the question pattern may be a specific pattern. The text pattern and/or the question pattern may be a default pattern. Each condition may correspond to at least one text pattern and at least one condition may correspond to at least one question pattern. For example, the conditions may be defined such that each condition must correspond to at least one text pattern and each condition may correspond to at least one question pattern. A configuration option may be set to apply question patterns in addition to or instead of text patterns. When one of the conditions matches a respective statement and the configuration option is set to apply question patterns and the question pattern corresponding to the condition can be applied to the respective statement, the method may comprise computing the string from the respective statement using the question pattern and/or computing a further string from the respective statement using the question pattern in addition to a string computed from the respective statement using the text pattern.

For example, the subset of the directed graph may include the following seven statements:

sap:PrintReceipt sap: requires sap: Printer. sap:PrintReceipt rdf:type sap:Task. sap:PrintReceipt rdfs:label “Print Receipt”. sap: Printer rdfs:label “Printer”. sap:Task rdfs:label “Task”@en. sap:Task rdfs:label “Process Task”@en. sap:Task rdfs:label “Schritt”@de.

Continuing the example, the plurality of patterns may include the following pattern (e.g., text pattern):

The <I1.rdf:type.label (lang=‘en’)*> <I1.rdfs:label> requires a <I2.rdfs:label>.

In addition, the plurality of patterns may include the following question pattern preceding the text pattern directly above:

What is required by <I1.rdf:type.label (lang=‘en’)*> <I1.rdfs:label>?

The “*” (asterisk) operator (i.e., post operator) in the question pattern causes a Cartesian product to be computed. In the text and question patterns above, <?11> is bound to (i.e., holds the value) sap:PrintReceipt, <?p> is bound to sap: requires, and <?12> is bound to sap: Printer. Accordingly, a configuration option may be set to apply both question patterns and text patterns. Hence, by applying

CLAIMS

Claims ( 20 )

What is claimed is:

1 . A computing system comprising:

at least one memory; one or more hardware processing units coupled to the at least one memory; and one or more computer readable storage media storing computer-executable instructions that, when executed, cause the computing system to perform operations comprising:

receiving, from a user, user input comprising a plurality of tokens;

analyzing at least a portion of the plurality of tokens;

based on the analyzing, determining one or more entities of a semantic framework represented in the at least a portion of the plurality of tokens;

determining one or more triples of the semantic framework for at least a portion of the one or more entities or for associated entities;

adding at least a portion of the one or more triples, or a representation thereof, to the user input to provide modified user input;

submitting the modified user input to a large language model;

processing the modified user input using the large language model to provide a response; and

returning the response in response to the receiving the user input.

2 . The computing system of claim 1 , wherein the analyzing the at least a portion of the plurality of tokens comprises providing the at least a plurality of tokens to a named entity recognition service.

3 . The computing system of claim 1 , wherein adding at least a portion of the one or more triples, or a representation thereof, to the user input to provide modified using input comprises:

submitting triples of the at least a portion of the plurality of triples to a verbalization function to provide the representation, the representation being verbalized triples.

4 . The computing system of claim 1 , wherein the semantic framework comprises a knowledge graph.

5 . The computing system of claim 1 , the operations further comprising:

identifying one or more associated entities for a subset of the one or more entities by traversing the semantic framework through one or more levels of indirection from each respective entity within the set of associated entities.

6 . The computing system of claim 5 , wherein the identifying is carried out up to a specified level of indirection.

7 . The computing system of claim 5 , wherein the identifying is carried out until a threshold number of entities has been identified.

8 . The computing system of claim 5 , wherein the triples are in the form of (subject, object, predicate), and the identifying one or more associated entities is carried out for relationships where a respective entity of the one or more entities serves as a subject and for relationships where a respective entity of the one or more entities serves as an object.

9 . The computing system of claim 1 , wherein the modified input is not provided to the user.

10 . The computing system of claim 1 , wherein the user input prior to modification is not provided to the large language model without the content of the modification.

11 . The computing system of claim 1 , the operations further comprising:

adding a length constraint to the modified user input.

12 . The computing system of claim 1 , the operations further comprising:

adding an instruction to the modified user input.

13 . The computing system of claim 1 , the operations further comprising:

adding a contextual instruction to the modified user input.

14 . The computing system of claim 1 , the operations further comprising:

adding a content constraint instruction to the modified user input.

15 . The computing system of claim 1 , wherein the one or more entities correspond to a first set of one or more entities, the operations further comprising:

identifying a second set of one or more entities in the response; linking one or more entities of the second set of one or more entities to supplemental content, wherein the displaying the response in response to the user input comprises displaying the response with one or more links to supplemental content; receiving user input selecting a linked entity; and displaying the supplemental content for the linked entity.

16 . The computing system of claim 1 , wherein the based on the analyzing, determining one or more entities of a semantic framework represented in the at least a portion of the plurality of tokens comprises analyzing multiple discrete semantic frameworks.

17 . The computing system of claim 1 , wherein the based on the analyzing, determining one or more entities of a semantic framework represented in the at least a portion of the plurality of tokens comprises sending an analysis request to be executed on a semantic framework located on a remote computing system.

18 . A method, implemented in a computing system comprising at least one hardware processor and at least one memory coupled to the at least one hardware processor, the method comprising:

receiving, from a user, user input comprising a plurality of tokens; analyzing at least a portion of the plurality of tokens; based on the analyzing, determining one or more entities of a semantic framework represented in the at least a portion of the plurality of tokens; determining one or more triples of the semantic framework for at least a portion of the one or more entities or for associated entities; adding at least a portion of the one or more triples, or a representation thereof, to the user input to provide modified user input; submitting the modified user input to a large language model; processing the modified user input using the large language model to provide a response; and returning the response in response to the receiving the user input.

19 . The method of claim 18 , wherein adding at least a portion of the one or more triples, or a representation thereof, to the user input to provide modified using input comprises:

submitting triples of the at least a portion of the plurality of triples to a verbalization function to provide the representation, the representation being verbalized triples.

20 . One or more computer-readable storage media comprising:

computer-executable instructions that, when executed by a computing system comprising at least one hardware processor and at least one memory coupled to the at least one hardware processor, cause the computing system to receive, from a user, user input comprising a plurality of tokens; computer-executable instructions that, when executed by the computing system, cause the computing system to analyze at least a portion of the plurality of tokens; computer-executable instructions that, when executed by the computing system, cause the computing system to, based on the analyzing, determine one or more entities of a semantic framework represented in the at least a portion of the plurality of tokens; computer-executable instructions that, when executed by the computing system, cause the computing system to determine one or more triples of the semantic framework for at least a portion of the one or more entities or for associated entities; computer-executable instructions that, when executed by the computing system, cause the computing system to add at least a portion of the one or more triples, or a representation thereof, to the user input to provide modified user input; computer-executable instructions that, when executed by the computing system, cause the computing system to submit the modified user input to a large language model; computer-executable instructions that, when executed by the computing system, cause the computing system to process the modified user input using the large language model to provide a response; and computer-executable instructions that, when executed by the computing system, cause the computing system to return the response in response to the receiving the user input.

US18/369,065

2023-09-15

2023-09-15

Supplementation of large language model knowledge via prompt modification

Pending

US20250094707A1

( en )

Priority Applications (3)

Application Number

Priority Date

Filing Date

Title

US18/369,065

US20250094707A1

( en )

2023-09-15

2023-09-15

Supplementation of large language model knowledge via prompt modification

EP24199105.8A

EP4538926A1

( en )

2023-09-15

2024-09-09

Supplementation of large language model knowledge via prompt modification

CN202411286185.7A

CN119669435A

( en )

2023-09-15

2024-09-13

Supplementing large language model knowledge through real-time modification

Applications Claiming Priority (1)

Application Number

Priority Date

Filing Date

Title

US18/369,065

US20250094707A1

( en )

2023-09-15

2023-09-15

Supplementation of large language model knowledge via prompt modification

Publications (1)

Publication Number

Publication Date

US20250094707A1

true

US20250094707A1 ( en )

2025-03-20

Family

ID=92712383

Family Applications (1)

Application Number

Title

Priority Date

Filing Date

US18/369,065

Pending

US20250094707A1

( en )

2023-09-15

2023-09-15

Supplementation of large language model knowledge via prompt modification

Country Status (3)

Country

Link

US

( 1 )

US20250094707A1

( en )

EP

( 1 )

EP4538926A1

( en )

CN

( 1 )

CN119669435A

( en )

Cited By (12)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20250110973A1

( en )

*

2023-09-28

2025-04-03

Royal Bank Of Canada

Neural network for event prediction

US20250139112A1

( en )

*

2023-06-23

2025-05-01

Salesforce, Inc.

Systems and Methods for Federated Query Abstraction

CN120179811A

( en )

*

2025-05-20

2025-06-20

浪潮通用软件有限公司

Method, device and medium for dynamically building rule base based on large language model

US20250225335A1

( en )

*

2024-01-08

2025-07-10

Deep Water Point & Associates

Computer Program and Method for Annotated Document Processing Based on User-Defined Parameters

US12373649B1

( en )

*

2025-02-12

2025-07-29

U.S. Bancorp, National Association

Remediating hallucinations in language models

US20250323835A1

( en )

*

2024-04-12

2025-10-16

Ciena Corporation

Automated Discovery of Network Inventory From Raw Configuration Files Using Machine Learning

US12511282B1

( en )

2023-05-02

2025-12-30

Microstrategy Incorporated

Generating structured query language using machine learning

US12585683B1

( en )

*

2024-09-23

2026-03-24

International Business Machines Corporation

Solving multilingual queries using vector database

US20260087363A1

( en )

*

2024-09-24

2026-03-26

Microsoft Technology Licensing, Llc

Large language model (llm) hallucination reduction by adverserial prompt refinement

US12591825B2

( en )

*

2024-03-11

2026-03-31

Procore Technologies, Inc.

Computing system and method for progress tracking using a large language model

US12602421B1

( en )

2025-01-23

2026-04-14

Dell Products L.P.

Classifying retrieved context data for a relativistic response

US20260211917A1

( en )

*

2025-01-23

2026-07-23

Dell Products L.P.

Context data retrieval using implicit terms

Families Citing this family (2)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

CN119918508B

( en )

*

2025-04-02

2025-07-01

季华实验室

Information extraction method, apparatus, storage medium, and program product

CN122086843A

( en )

*

2026-04-24

2026-05-26

统信软件技术有限公司

Skill hot update methods, devices, electronic devices, storage media and program products

Citations (20)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20080010259A1

( en )

*

2006-07-10

2008-01-10

Nec (China) Co., Ltd.

Natural language based location query system, keyword based location query system and a natural language and keyword based location query system

US20150081663A1

( en )

*

2013-09-18

2015-03-19

First Principles, Inc.

System and method for active search environment

US20180276525A1

( en )

*

2015-12-03

2018-09-27

Huawei Technologies Co., Ltd.

Method and neural network system for human-computer interaction, and user equipment

US20190122111A1

( en )

*

2017-10-24

2019-04-25

Nec Laboratories America, Inc.

Adaptive Convolutional Neural Knowledge Graph Learning System Leveraging Entity Descriptions

US20190213258A1

( en )

*

2018-01-10

2019-07-11

International Business Machines Corporation

Machine Learning to Integrate Knowledge and Natural Language Processing

US10606958B2

( en )

*

2018-01-10

2020-03-31

International Business Machines Corporation

Machine learning modification and natural language processing

US20200380991A1

( en )

*

2019-06-03

2020-12-03

Microsoft Technology Licensing, Llc

Clarifying questions for rewriting ambiguous user utterance

US20210216819A1

( en )

*

2020-01-15

2021-07-15

Beijing Baidu Netcom Science And Technology Co., Ltd.

Method, electronic device, and storage medium for extracting spo triples

US11151320B1

( en )

*

2020-04-21

2021-10-19

Microsoft Technology Licensing, Llc

Labeled knowledge graph based priming of a natural language model providing user access to programmatic functionality through natural language input

US20220050964A1

( en )

*

2020-08-14

2022-02-17

Salesforce.Com, Inc.

Structured graph-to-text generation with two step fine-tuning

US20220138185A1

( en )

*

2020-11-03

2022-05-05

Adobe Inc.

Scene graph modification based on natural language commands

US20220230628A1

( en )

*

2021-01-20

2022-07-21

Microsoft Technology Licensing, Llc

Generation of optimized spoken language understanding model through joint training with integrated knowledge-language module

US20230177356A1

( en )

*

2020-07-29

2023-06-08

Samsung Electronics Co., Ltd.

System and method for modifying knowledge graph for providing service

US20240095460A1

( en )

*

2022-09-19

2024-03-21

Nvidia Corporation

Dialogue systems using knowledge bases and language models for automotive systems and applications

US11997056B2

( en )

*

2022-08-29

2024-05-28

Adobe Inc.

Language model with external knowledge base

US20240296285A1

( en )

*

2023-03-03

2024-09-05

NEC Laboratories Europe GmbH

Grouping and linking facts from text to remove ambiguity using knowledge graphs

US20240428044A1

( en )

*

2023-06-23

2024-12-26

Salesforce, Inc.

Systems and methods for retrieval based question answering using neura network models

US20250005299A1

( en )

*

2023-06-30

2025-01-02

Salesforce, Inc.

Language model prompt authoring and execution in a database system

US12602226B1

( en )

*

2023-09-14

2026-04-14

Amazon Technologies, Inc.

Complex code modification via chain of thought prompting

US12651599B2

( en )

*

2023-06-23

2026-06-09

Amazon Technologies, Inc.

Generating model output using a knowledge graph

2023

2023-09-15

US

US18/369,065

patent/US20250094707A1/en

active

Pending

2024

2024-09-09

EP

EP24199105.8A

patent/EP4538926A1/en

active

Pending

2024-09-13

CN

CN202411286185.7A

patent/CN119669435A/en

active

Pending

Patent Citations (23)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20080010259A1

( en )

*

2006-07-10

2008-01-10

Nec (China) Co., Ltd.

Natural language based location query system, keyword based location query system and a natural language and keyword based location query system

US20150081663A1

( en )

*

2013-09-18

2015-03-19

First Principles, Inc.

System and method for active search environment

US20180276525A1

( en )

*

2015-12-03

2018-09-27

Huawei Technologies Co., Ltd.

Method and neural network system for human-computer interaction, and user equipment

US20190122111A1

( en )

*

2017-10-24

2019-04-25

Nec Laboratories America, Inc.

Adaptive Convolutional Neural Knowledge Graph Learning System Leveraging Entity Descriptions

US20190213258A1

( en )

*

2018-01-10

2019-07-11

International Business Machines Corporation

Machine Learning to Integrate Knowledge and Natural Language Processing

US10606958B2

( en )

*

2018-01-10

2020-03-31

International Business Machines Corporation

Machine learning modification and natural language processing

US20200380991A1

( en )

*

2019-06-03

2020-12-03

Microsoft Technology Licensing, Llc

Clarifying questions for rewriting ambiguous user utterance

US20210216819A1

( en )

*

2020-01-15

2021-07-15

Beijing Baidu Netcom Science And Technology Co., Ltd.

Method, electronic device, and storage medium for extracting spo triples

US20230141200A1

( en )

*

2020-04-21

2023-05-11

Microsoft Technology Licensing, Llc

Labeled knowledge graph based priming of a natural language model providing user access to programmatic functionality through natural language input

US11151320B1

( en )

*

2020-04-21

2021-10-19

Microsoft Technology Licensing, Llc

Labeled knowledge graph based priming of a natural language model providing user access to programmatic functionality through natural language input

US20220036001A1

( en )

*

2020-04-21

2022-02-03

Microsoft Technology Licensing, Llc

Labeled Knowledge Graph Based Priming Of A Natural Language Model Providing User Access To Programmatic Functionality Through Natural Language Input

US20230177356A1

( en )

*

2020-07-29

2023-06-08

Samsung Electronics Co., Ltd.

System and method for modifying knowledge graph for providing service

US20220050964A1

( en )

*

2020-08-14

2022-02-17

Salesforce.Com, Inc.

Structured graph-to-text generation with two step fine-tuning

US20220138185A1

( en )

*

2020-11-03

2022-05-05

Adobe Inc.

Scene graph modification based on natural language commands

US12399890B2

( en )

*

2020-11-03

2025-08-26

Adobe Inc.

Scene graph modification based on natural language commands

US20220230628A1

( en )

*

2021-01-20

2022-07-21

Microsoft Technology Licensing, Llc

Generation of optimized spoken language understanding model through joint training with integrated knowledge-language module

US11997056B2

( en )

*

2022-08-29

2024-05-28

Adobe Inc.

Language model with external knowledge base

US20240095460A1

( en )

*

2022-09-19

2024-03-21

Nvidia Corporation

Dialogue systems using knowledge bases and language models for automotive systems and applications

US20240296285A1

( en )

*

2023-03-03

2024-09-05

NEC Laboratories Europe GmbH

Grouping and linking facts from text to remove ambiguity using knowledge graphs

US20240428044A1

( en )

*

2023-06-23

2024-12-26

Salesforce, Inc.

Systems and methods for retrieval based question answering using neura network models

US12651599B2

( en )

*

2023-06-23

2026-06-09

Amazon Technologies, Inc.

Generating model output using a knowledge graph

US20250005299A1

( en )

*

2023-06-30

2025-01-02

Salesforce, Inc.

Language model prompt authoring and execution in a database system

US12602226B1

( en )

*

2023-09-14

2026-04-14

Amazon Technologies, Inc.

Complex code modification via chain of thought prompting

Cited By (14)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US12511282B1

( en )

2023-05-02

2025-12-30

Microstrategy Incorporated

Generating structured query language using machine learning

US20250139112A1

( en )

*

2023-06-23

2025-05-01

Salesforce, Inc.

Systems and Methods for Federated Query Abstraction

US12681946B2

( en )

*

2023-06-23

2026-07-14

Salesforce, Inc.

Systems and methods for federated query abstraction

US20250110973A1

( en )

*

2023-09-28

2025-04-03

Royal Bank Of Canada

Neural network for event prediction

US20250225335A1

( en )

*

2024-01-08

2025-07-10

Deep Water Point & Associates

Computer Program and Method for Annotated Document Processing Based on User-Defined Parameters

US12591825B2

( en )

*

2024-03-11

2026-03-31

Procore Technologies, Inc.

Computing system and method for progress tracking using a large language model

US20250323835A1

( en )

*

2024-04-12

2025-10-16

Ciena Corporation

Automated Discovery of Network Inventory From Raw Configuration Files Using Machine Learning

US12585683B1

( en )

*

2024-09-23

2026-03-24

International Business Machines Corporation

Solving multilingual queries using vector database

US20260087363A1

( en )

*

2024-09-24

2026-03-26

Microsoft Technology Licensing, Llc

Large language model (llm) hallucination reduction by adverserial prompt refinement

US12602421B1

( en )

2025-01-23

2026-04-14

Dell Products L.P.

Classifying retrieved context data for a relativistic response

US20260211917A1

( en )

*

2025-01-23

2026-07-23

Dell Products L.P.

Context data retrieval using implicit terms

US12462108B1

( en )

*

2025-02-12

2025-11-04

U.S. Bancorp, National Association

Remediating hallucinations in language models

US12373649B1

( en )

*

2025-02-12

2025-07-29

U.S. Bancorp, National Association

Remediating hallucinations in language models

CN120179811A

( en )

*

2025-05-20

2025-06-20

浪潮通用软件有限公司

Method, device and medium for dynamically building rule base based on large language model

Also Published As

Publication number

Publication date

CN119669435A

( en )

2025-03-21

EP4538926A1

( en )

2025-04-16

Similar Documents

Publication

Publication Date

Title

EP4538926A1

( en )

2025-04-16

Supplementation of large language model knowledge via prompt modification

US11693855B2

( en )

2023-07-04

Automatic creation of schema annotation files for converting natural language queries to structured query language

US20210294781A1

( en )

2021-09-23

Feature engineering with question generation

US9740685B2

( en )

2017-08-22

Generation of natural language processing model for an information domain

US11763095B2

( en )

2023-09-19

Creating apps from natural language descriptions

CN111475534B

( en )

2023-04-14

Data query method and related equipment

EP4700605A1

( en )

2026-02-25

Supplementation of large language model knowledge and responses with media content

US12619652B2

( en )

2026-05-05

Automated ontology creation

Levkivskyi et al.

2025

Agile methodology of information engineering for semantic annotations categorization and creation in scientific articles based on NLP and machine learning methods

US20260030274A1

( en )

2026-01-29

Systems and methods for generating structured conversational ai content from unstructured and structured data sources

EP4617900A1

( en )

2025-09-17

Customized information change notification using knowledge graphs

US20240411787A1

( en )

2024-12-12

Providing data from a directed graph to a language model

Bhatia et al.

2011

Semantic web mining: Using ontology learning and grammatical rule inference technique

US12572581B2

( en )

2026-03-10

Automated ontology creation

Pietranik et al.

2012

A method for ontology alignment based on semantics of attributes

Iqbal et al.

2013

A Negation Query Engine for Complex Query Transformations

Chornyi

2025

Development of a unified output format for text parsers in the ontology construction system from text documents

Kopp et al.

2023

Computational Technology and Software Tool for Translation of Business Rules into Database Creation Scripts.

Kuchmann-Beauger

2013

Question answering system in a business intelligence context

Kedwan

2023

NLP application: Natural language questions and SQL using computational linguistics

US20250292105A1

( en )

2025-09-18

Automated ontology maintenance

Gusmita et al.

2019

QUANT-Question Answering Benchmark Curator

Canim et al.

2017

A knowledge and reasoning toolkit for cognitive applications

Ngomo et al.

2019

QUANT-Question Answering Benchmark Curator

Laitio

2011

Semantic Web Data Quality Control

Legal Events

Date

Code

Title

Description

2023-09-18

AS

Assignment

Owner name : SAP SE, GERMANY

Free format text : ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:PORTISCH, JAN;COSTA, GUILHERME;HLADIK, MICHAEL;AND OTHERS;SIGNING DATES FROM 20230912 TO 20230915;REEL/FRAME:064936/0928

2023-10-02

STPP

Information on status: patent application and granting procedure in general

Free format text : DOCKETED NEW CASE - READY FOR EXAMINATION

2025-09-03

STPP

Information on status: patent application and granting procedure in general

Free format text : NON FINAL ACTION COUNTED, NOT YET MAILED

2025-09-04

STPP

Information on status: patent application and granting procedure in general

Free format text : NON FINAL ACTION MAILED

2025-12-05

STPP

Information on status: patent application and granting procedure in general

Free format text : RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER

2025-12-14

STPP

Information on status: patent application and granting procedure in general

Free format text : RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER

2026-03-07

STPP

Information on status: patent application and granting procedure in general

Free format text : FINAL REJECTION COUNTED, NOT YET MAILED

2026-03-11

STPP

Information on status: patent application and granting procedure in general

Free format text : FINAL REJECTION MAILED

2026-05-07

STPP

Information on status: patent application and granting procedure in general

Free format text : ADVISORY ACTION COUNTED, NOT YET MAILED

2026-05-12

STPP

Information on status: patent application and granting procedure in general

Free format text : ADVISORY ACTION MAILED

2026-06-08

STPP

Information on status: patent application and granting procedure in general

Free format text : DOCKETED NEW CASE - READY FOR EXAMINATION

2026-06-13

STPP

Information on status: patent application and granting procedure in general

Free format text : NON FINAL ACTION COUNTED, NOT YET MAILED

2026-06-17

STPP

Information on status: patent application and granting procedure in general

Free format text : NON FINAL ACTION MAILED

Related documents

• MA-SAPO: Multi-Agent Reasoning for Score-Aware Prompt Optimization#1118430
• Prompt ̸ = Knowledge: On the Epistemic Limits of In-Context Control in Large Language Models#728888
• Grammar Concept Annotation at Scale: Deployed Fine-Tuned Small Language Models Outperform Prompted Frontier Models#1247754
• Application of large language models as decision support tools in occupational health and safety management: a cohort study of industrial workers.#224848
• CPInj: Uncovering Prompt Injection Risks in Textual Collaborative Prompt Optimization#386751