ABSTRACT
Abstract
A website development system automatically generates text for a webpage. The system obtains a prompt template associated with a section of the webpage, where the prompt template includes one or more parameters. Based on the webpage, the prompt template determines a first value for a first one of the one or more parameters. A request to provide input for a second value of a second parameter is sent for display to a user. Using the prompt template, the first value, and the second value, the system generates a prompt to a large language model to generate text for the section of the webpage.
Description
CROSS-REFERENCE TO RELATED APPLICATION
This application claims the benefit of U.S. Provisional Application No. 63/493,833, filed Apr. 3, 2023, which is incorporated herein by reference in its entirety.
TECHNICAL FIELD
This disclosure related to large language models (LLMs), and in particular to dynamically generating prompts to LLMs.
BACKGROUND
Webpages often include a significant amount of text that is used to describe, for example, information about an entity affiliated with the webpage or about products or services offered by the entity. Accordingly, a user creating or modifying a webpage typically must write a significant amount of text for the page.
BRIEF DESCRIPTION OF THE DRAWINGS
Detailed descriptions of implementations of the present invention will be described and explained through the use of the accompanying drawings.
FIG. 1 is a block diagram illustrating an environment in which a website development system operates, according to some implementations.
FIG. 2 is a block diagram illustrating functional modules executed by the website development system, according to some implementations.
FIG. 3 A illustrates an example website development interface.
FIGS. 3 B- 3 C illustrate example user interfaces for receiving user inputs for prompt generation.
FIG. 4 is a flowchart illustrating a process for automating generation of prompts to a large language model, according to some implementations.
FIG. 5 A is a block diagram of a simplified convolutional neural network, which may be used in examples of the present disclosure.
FIG. 5 B is a block diagram of a simplified transformer neural network, which may be used in examples of the present disclosure.
FIG. 6 is a block diagram of an example computing system, which may be used to implement examples of the present disclosure.
FIG. 7 illustrates an example e-commerce platform, according to some implementations.
FIG. 8 illustrates another implementation of an e-commerce platform.
The technologies described herein will become more apparent to those skilled in the art from studying the Detailed Description in conjunction with the drawings. Embodiments or implementations describing aspects of the invention are illustrated by way of example, and the same references can indicate similar elements. While the drawings depict various implementations for the purpose of illustration, those skilled in the art will recognize that alternative implementations can be employed without departing from the principles of the present technologies. Accordingly, while specific implementations are shown in the drawings, the technology is amenable to various modifications
DETAILED DESCRIPTION
Websites have become a critical channel for communicating information, and many entities find their websites to be a necessary component of operating their business or organizations to share information with current or potential consumers. Although it is important for an entity's website to communicate information clearly and effectively to the website's visitors, writing text for a website can be a burdensome process, and one that is often orthogonal to an entity's core competencies. Entities are therefore increasingly turning to large language models to help them quickly and efficiently prepare text for websites. However, it can be difficult to generate prompts that will cause the text produced by a large language model to be appropriate for the context in which the text will appear. It is especially difficult to generate these prompts programmatically.
To solve this problem, a website development system according to implementations herein enables LLM-based text generation for a website by automating the generation of a prompt using data retrieved or derived from the website. According to some implementations, the website development system obtains a prompt template associated with a section of a webpage. The prompt template includes one or more parameters, and a first value for a first one of the one or more parameters is determined based on the webpage. The website development system can also send a request to a user to provide input for a second value corresponding to a second one of the parameters in the prompt template. Using the prompt template, the first parameter value, and the second parameter value, the website development system generates a prompt to a large language model to generate text for the section of the webpage.
Website Development System
FIG. 1 illustrates an environment 100 in which a website development system 120 operates, according to some implementations.
A website 105 can include one or more webpages, each of which has one or more sections that represent different units of content on the webpage. For example, webpages may include sections such as an announcement bar, a description of the organization operating the webpage, a slide show with multiple image or text blocks, and/or an image with associated text that describes the image. A user can interact with the website development system 120 to modify the website 105 , including creating new webpages within the website, adding new content to a webpage, or modifying existing content on a webpage.
The website development system 120 receives data associated with the website 105 and/or user input 110 and, using the website data or user input, generates a prompt 125 to a large language model (LLM) 130 . The LLM 130 can include any commercially available or custom models, or a set or ensemble of two or more models. Example features of LLMs are described with respect to FIGS. 5 - 6 .
The LLM 130 uses the prompt to generate text 135 . The website development system 120 can then add the generated text 135 to the website 105 . Accordingly, the website development system 120 facilitates automated text generation for a website.
FIG. 2 is a block diagram illustrating functional modules executed by the website development system 120 , according to some implementations. As shown in FIG. 2 , an example implementation of the website development system 120 includes a website development tool 205 , a section recommendation module 210 , a prompt templates store 215 , a prompt template selector 220 , a parameter analysis module 225 , and a prompt generator 230 . The website development system 120 can include additional, fewer, or different modules, and functionality described herein can be divided differently between the modules. As used herein, the term âmoduleâ refers broadly to software components, firmware components, and/or hardware components. Accordingly, the
modules
205 , 210 , 220 , 225 , and 230 could each be comprised of software, firmware, and/or hardware components implemented in, or accessible to, the website development system 120 .
The website development tool 205 generates user interfaces and receives inputs from users to create or modify websites with automatically generated text. An example interface 300 output by the website development tool 205 for creating a webpage is illustrated in FIG. 3 A . As shown in FIG. 3 A , the website development interface 300 includes a canvas 310 that displays the appearance of a webpage as it is developed using the tool 205 . A navigation pane 320 displays sections of the webpage (e.g., announcement bar, header, and image banner) and enables the user to add a new section to the webpage (e.g., by selecting the âadd sectionâ control 322 ). A section editing pane 330 enables the user to edit content of a section selected in the navigation pane 320 . For example, as shown in FIG. 3 A , the âheadingâ section is selected for editing the section editing pane 330 .
Returning to FIG. 2 , the section recommendation module 210 recommends a type of section to add to a webpage. When a user selects an option in the website development tool 205 to âAdd Sectionâ (e.g., the control 322 in FIG. 3 A ), the section recommendation module 210 determines one or more recommended section types and causes the website development tool 205 to display the recommended section types to the user.
In some implementations, the section recommendation module 210 generates recommendations for a next section using a machine learning model that has been trained based on layout of other websites. For example, the section recommendation module 210 or another system analyzes websites that are operated by entities that are similar to the entity operating the website under development or websites that relate to similar products or services. The training system extracts data from these similar websites, such as types of sections that appear on the same webpage and positions of these sections on the page. The extracted data is used to train a recommendation model that can recommend a type of section to be added to a webpage based on, for example, types and positions of other sections already on the webpage, a position of the section on the webpage, an identity of the organization operating the webpage, or style or tone of text in other sections.
Alternatively, the section recommendation module 210 can recommend a section type for a new section based on an intended location for the new section on the webpage. For example, if the user indicates the section will be placed near the top of the webpage, the section recommendation module 210 may recommend an âannouncement barâ section type. A section placed
CROSS-REFERENCE TO RELATED APPLICATION
This application claims the benefit of U.S. Provisional Application No. 63/493,833, filed Apr. 3, 2023, which is incorporated herein by reference in its entirety.
TECHNICAL FIELD
This disclosure related to large language models (LLMs), and in particular to dynamically generating prompts to LLMs.
BACKGROUND
Webpages often include a significant amount of text that is used to describe, for example, information about an entity affiliated with the webpage or about products or services offered by the entity. Accordingly, a user creating or modifying a webpage typically must write a significant amount of text for the page.
BRIEF DESCRIPTION OF THE DRAWINGS
Detailed descriptions of implementations of the present invention will be described and explained through the use of the accompanying drawings.
FIG. 1 is a block diagram illustrating an environment in which a website development system operates, according to some implementations.
FIG. 2 is a block diagram illustrating functional modules executed by the website development system, according to some implementations.
FIG. 3 A illustrates an example website development interface.
FIGS. 3 B- 3 C illustrate example user interfaces for receiving user inputs for prompt generation.
FIG. 4 is a flowchart illustrating a process for automating generation of prompts to a large language model, according to some implementations.
FIG. 5 A is a block diagram of a simplified convolutional neural network, which may be used in examples of the present disclosure.
FIG. 5 B is a block diagram of a simplified transformer neural network, which may be used in examples of the present disclosure.
FIG. 6 is a block diagram of an example computing system, which may be used to implement examples of the present disclosure.
FIG. 7 illustrates an example e-commerce platform, according to some implementations.
FIG. 8 illustrates another implementation of an e-commerce platform.
The technologies described herein will become more apparent to those skilled in the art from studying the Detailed Description in conjunction with the drawings. Embodiments or implementations describing aspects of the invention are illustrated by way of example, and the same references can indicate similar elements. While the drawings depict various implementations for the purpose of illustration, those skilled in the art will recognize that alternative implementations can be employed without departing from the principles of the present technologies. Accordingly, while specific implementations are shown in the drawings, the technology is amenable to various modifications
DETAILED DESCRIPTION
Websites have become a critical channel for communicating information, and many entities find their websites to be a necessary component of operating their business or organizations to share information with current or potential consumers. Although it is important for an entity's website to communicate information clearly and effectively to the website's visitors, writing text for a website can be a burdensome process, and one that is often orthogonal to an entity's core competencies. Entities are therefore increasingly turning to large language models to help them quickly and efficiently prepare text for websites. However, it can be difficult to generate prompts that will cause the text produced by a large language model to be appropriate for the context in which the text will appear. It is especially difficult to generate these prompts programmatically.
To solve this problem, a website development system according to implementations herein enables LLM-based text generation for a website by automating the generation of a prompt using data retrieved or derived from the website. According to some implementations, the website development system obtains a prompt template associated with a section of a webpage. The prompt template includes one or more parameters, and a first value for a first one of the one or more parameters is determined based on the webpage. The website development system can also send a request to a user to provide input for a second value corresponding to a second one of the parameters in the prompt template. Using the prompt template, the first parameter value, and the second parameter value, the website development system generates a prompt to a large language model to generate text for the section of the webpage.
Website Development System
FIG. 1 illustrates an environment 100 in which a website development system 120 operates, according to some implementations.
A website 105 can include one or more webpages, each of which has one or more sections that represent different units of content on the webpage. For example, webpages may include sections such as an announcement bar, a description of the organization operating the webpage, a slide show with multiple image or text blocks, and/or an image with associated text that describes the image. A user can interact with the website development system 120 to modify the website 105 , including creating new webpages within the website, adding new content to a webpage, or modifying existing content on a webpage.
The website development system 120 receives data associated with the website 105 and/or user input 110 and, using the website data or user input, generates a prompt 125 to a large language model (LLM) 130 . The LLM 130 can include any commercially available or custom models, or a set or ensemble of two or more models. Example features of LLMs are described with respect to FIGS. 5 - 6 .
The LLM 130 uses the prompt to generate text 135 . The website development system 120 can then add the generated text 135 to the website 105 . Accordingly, the website development system 120 facilitates automated text generation for a website.
FIG. 2 is a block diagram illustrating functional modules executed by the website development system 120 , according to some implementations. As shown in FIG. 2 , an example implementation of the website development system 120 includes a website development tool 205 , a section recommendation module 210 , a prompt templates store 215 , a prompt template selector 220 , a parameter analysis module 225 , and a prompt generator 230 . The website development system 120 can include additional, fewer, or different modules, and functionality described herein can be divided differently between the modules. As used herein, the term âmoduleâ refers broadly to software components, firmware components, and/or hardware components. Accordingly, the
modules
205 , 210 , 220 , 225 , and 230 could each be comprised of software, firmware, and/or hardware components implemented in, or accessible to, the website development system 120 .
The website development tool 205 generates user interfaces and receives inputs from users to create or modify websites with automatically generated text. An example interface 300 output by the website development tool 205 for creating a webpage is illustrated in FIG. 3 A . As shown in FIG. 3 A , the website development interface 300 includes a canvas 310 that displays the appearance of a webpage as it is developed using the tool 205 . A navigation pane 320 displays sections of the webpage (e.g., announcement bar, header, and image banner) and enables the user to add a new section to the webpage (e.g., by selecting the âadd sectionâ control 322 ). A section editing pane 330 enables the user to edit content of a section selected in the navigation pane 320 . For example, as shown in FIG. 3 A , the âheadingâ section is selected for editing the section editing pane 330 .
Returning to FIG. 2 , the section recommendation module 210 recommends a type of section to add to a webpage. When a user selects an option in the website development tool 205 to âAdd Sectionâ (e.g., the control 322 in FIG. 3 A ), the section recommendation module 210 determines one or more recommended section types and causes the website development tool 205 to display the recommended section types to the user.
In some implementations, the section recommendation module 210 generates recommendations for a next section using a machine learning model that has been trained based on layout of other websites. For example, the section recommendation module 210 or another system analyzes websites that are operated by entities that are similar to the entity operating the website under development or websites that relate to similar products or services. The training system extracts data from these similar websites, such as types of sections that appear on the same webpage and positions of these sections on the page. The extracted data is used to train a recommendation model that can recommend a type of section to be added to a webpage based on, for example, types and positions of other sections already on the webpage, a position of the section on the webpage, an identity of the organization operating the webpage, or style or tone of text in other sections.
Alternatively, the section recommendation module 210 can recommend a section type for a new section based on an intended location for the new section on the webpage. For example, if the user indicates the section will be placed near the top of the webpage, the section recommendation module 210 may recommend an âannouncement barâ section type. A section placed near the bottom of the webpage may have a recommended section type of âfrequently asked questions.â In some implementations, if a user moves a section from one position to another position on the webpage, the section recommendation module 210 recommends changing the section type for the section from a first section type to a second type. Alternatively, the section recommendation module 210 can automatically modify the section type for the section based on the move. The section recommendation module 210 may notify the user or request user approval for the modification before enacting it.
The prompt templates store 215 is a repository of prompt templates that can be used by other components of the website development system 120 to generate prompts to the LLM 130 . Each prompt template includes at least one parameter, a value for which is selected to generate a unique prompt to the LLM 130 . Example parameters include the name of the entity that operates the webpage, a brand name associated with the operating entity, a type of the entity that operates the webpage or an industry in which the entity operates, a product type of a product described or sold through the website, or tone for the generated text. A prompt template can also include other fixed content that does not vary across prompts, such as content that describes the type of text to be generated or that contextualizes the parameters in the template. An example prompt template, in which parameters are indicated by curly brackets, is as follows:
Name of my brand is {{Brand name}}. We are a {{Industry}} brand selling {{Types of Products}}. The tone of the brand is {{Brand Tone}}. Can you generate a {{Style & tone}} announcement for a {{SectionType}} section, short, under 10 words, that includes {{Keywords}}?
Each prompt template can be associated with a type of webpage section. For example, a first prompt template can be associated with (and used to generate text for) a first type of webpage section (e.g., an announcement bar) while a second, different prompt template is associated with a second type of webpage section (e.g., a company description section). The available webpage section types offered by the website development system 120 can each have one or more associated prompt templates. Alternatively, each prompt template can be associated with a location on a webpage, such that a template corresponds to a section that is placed at the associated location.
The prompt template selector 220 obtains a prompt template to use to generate text for a particular section of a particular webpage. The prompt template can be selected based on section type, such as the prompt template selector 220 retrieving the prompt template associated with the section type from the prompt templates store 215 . When multiple templates are associated with the same section type, the particular prompt template used to generate text for the section can be selected based on other attributes of the webpage or the entity that operates the webpage (e.g., using a first template for a webpage associated with a first type of person or company and a second template for a webpage associated with a second type of person or company). Alternatively, a first template can be used to initially generate text for a given section, while a second template is used if a user requests regeneration of the text.
In some implementations, the prompt template selector 220 selects a prompt template based on the location of the section on the webpage. If the user moves a section from one location on the webpage to another location, the prompt template selector 220 obtains a different prompt template. For example, the selector 220 obtains a template associated with the new location of the section, or updates the section type of the section based on its relocation and obtains a template associated with the new section type.
The parameter analysis module 225 determines a value for each of the one or more parameters in the selected prompt template. The parameter analysis module 225 can use website data or previously supplied user inputs to determine some parameter values, reducing the amount of input the user needs to provide to generate the LLM prompt. For example, at the time that a user adds a new section to a webpage, the webpage (or the website containing the webpage) may already include some information relevant to the prompt template associated with the new section. Such information can appear explicitly or implicitly in the content of the webpage or website itself or in an account of an entity affiliated with the website. When a user adds a new section to a webpage, the parameter analysis module 225 accesses this webpage data to determine if the value of a parameter can be automatically retrieved or derived.
In an example, a parameter in the selected prompt template is a brand name of an entity associated with the webpage or a product or service offered by the entity. A user may have provided the brand name for another section on the website or when the user created an account to use the website development tool 205 . Accordingly, the parameter analysis module 225 can query the website data or user account data to retrieve the brand name when required by a prompt template.
Another example parameter is a product type for a product sold or advertised on the website. Like brand name, the product type may have been provided by the user for another section of the webpage or in a user account. Alternatively, the parameter analysis module 225 categorizes a product based on a description of the product provided by the user, based on an image of the product uploaded by the user, or based on other signals supplied by the user.
Still another example parameter that may be used in prompt templates is an indication of the tone or style of the text that the LLM should generate. Tones that can be used for website text include, for example, âcatchy and creative,â âbold and energetic,â âfun and playful,â âinformative and educational,â âshort and factual,â or âluxurious and relaxing.â In some implementations, the parameter analysis module 225 determines a value for a tone parameter by sending other text on the website to the LLM 130 to characterize the tone of the other text. If multiple tones are used on the website, the parameter analysis module 225 can select the tone to use for a given section based on, for example, a type of the section (e.g., such that sections of a given section type across the website use the same or similar tones) or the particular webpage on which the section will appear (e.g., to keep tone the same or similar across a given webpage within the website). Alternatively, the parameter analysis module 225 can extrapolate between multiple tones used on the website. Additionally, rather than sending the other text to the LLM for analysis, the parameter analysis module 225 can itself process or cause another system to process text from the website using a tone embedding model that uses a trained neural network to generate an embedding representing the tone of the text.
In other implementations, the parameter analysis module 225 infers tone based on a type of entity that operates the website, a type of section to be added to the webpage, or a type of item to be described by the section. For example, when a user adds a product description section for a laptop, the parameter analysis module 225 may automatically recommend a tone of âshort and factualâ for the section. When a user adds a product description section for a spa kit, the module 225 may automatically recommend a tone of âluxurious and relaxing.â Any section generated on a website operated by a gaming company targeted towards children may be assigned a âfun and playfulâ tone.
In still other implementations, the parameter analysis module 225 selects a tone for a given webpage based on a predicted audience for the webpage. For example, the parameter analysis module 225 identifies a set of websites that are similar to the webpage under development, such as websites operated by similar entities, websites selling similar types of products, or websites offering similar types of services. From the set of similar websites, the parameter analysis module 225 retrieves audience data such as age, gender, or income of the typical visitor to the similar website. Alternatively, the entity associated with the webpage under development can provide explicit input specifying the expected target audience of the webpage. The parameter analysis module 225 can maintain a mapping between audience data and corresponding tone to use for a website, and use the mapping to select the tone to use for the webpage under development based on the predicted or targeted audience.
The parameter analysis module 225 can also obtain parameter values by sending a request for input from a user of the website development tool 205 . In some implementations, if the parameter analysis module 225 identifies multiple candidate values for a given parameter, the module 225 causes the candidate values to be output to the user for selection of one of the values (or to provide an alternative value). For example, if the parameter analysis module 225 detects multiple tones that have been used throughout the website to which the webpage belongs, the module may request user input to select a tone from among the detected tones. In some implementations, the parameter analysis module 225 assigns a confidence score to its determination of a parameter value. If the prompt template includes any parameters for which the parameter analysis module 225 has not determined a corresponding value with at least a specified confidence score, the module 225 can send a request to a user to confirm a low-confidence parameter value or to provide input to specify the parameter value. The parameter analysis module 225 can also request user input to confirm any automatically determined parameter values.
The prompt generator 230 uses the prompt template and any parameters determined by the parameter analysis module 225 or received from a user to generate a prompt to the LLM 130 .
FIGS. 3 B- 3 C illustrate example user interactions with an interface 300 generated by the website development tool 205 . In the example of FIG. 3 B , a user interacts with the interface 300 to generate text for an image banner section of a webpage. When a user interacts with a menu control 332 in the section editing pane 330 , the interface displays controls to automatically generate text or to modify text. For example, the user can select the control 333 to cause the website development system to generate a âFrequently Asked Questionsâ description or a control 335 to generate an âAbout Usâ description. Alternatively, the user can select a control 337 to improve writing of an existing description or a control 339 to fix spelling and grammar of the pre-written description.
FIG. 3 C illustrates an example interface displayed when the user selects the control 335 , requesting automatic generation of text for an âabout usâ section. When the user requests automatic text generation, the website development system obtains a prompt template for the section and evaluates whether values for the parameters in the obtained template can be determined based on the existing website data. For any parameter that the system does not determine, a request for input can be presented to the user. In the example of FIG. 3 C , the website development system asks the user for input, via a textbox 342 , of a short description of the user's company, products or services, or brand mission. The system also requests user selection of a tone from a menu 344 . When the user selects the âgenerateâ control 346 , the website development system uses the prompt template, any automatically determined parameter values, and the parameter values input by the user to generate a prompt to the LLM.
Automatic Generation of LLM Prompts
FIG. 4 is a flowchart illustrating a process 400 for automating generation of prompts to a large language model, according to some implementations. The process 400 can be performed by a computer system, such as the website development system 120 . Other implementations of the process 400 include additional, fewer, or different steps, or perform the steps in different orders. For example, although the illustrated process includes steps for adding a new section to a webpage and generating a prompt to an LLM in response to the new section being added, the steps for adding the new webpage section can be omitted while prompts to the LLM are generated in similar ways to modify text in a section that already exists.
At operation 402 , the computer system receives an input to add a section to a webpage. In response to the input, the computer system can recommend a section type for the section at 404 . The recommendation for the section type can be generated using a machine learning model that is trained based on layouts of other webpages to predict a likely next section type for the webpage based on the current contents of the webpage. Section types can additionally or alternatively be recommended based on the targeted location for the new section on the webpage. A user can accept the recommended section type to continue adding the section type to the webpage or can select a different section type. In some implementations, the section is added to the webpage with default values that can be modified as the process 400 continues.
At operation 406 , the computer system obtains a prompt template associated with the section type of a target section. The target section can be a new section to be added to the webpage, whether the section type of the new section was programmatically recommended by the computer system (e.g., at operation 402 ) or selected by the user. Alternatively, the user can select a target section that was previously added to the webpage in order to modify existing content of the target section.
At operation 408 , the computer system determines, based on the webpage, a first value for a first parameter in the prompt template. For some types of parameters, the first value can be retrieved from another section of the webpage, such as explicit information the user input when creating the other section. Similarly, a value for some types of parameters can be retrieved from metadata associated with the webpage or a user or entity account associated with the user creating the webpage or the entity operating the webpage. For other types of parameters, the computer system processes other content on the webpage to determine the parameter value. For example, the system evaluates a tone of other text on the page to select a tone to use to generate text for a new section.
At operation 410 , the computer system sends a request to a user to provide input for a second value of a second parameter in the template. User input can be requested, for example, when the system determines that a value for a given parameter is unknown or cannot be determined with sufficient confidence from the webpage data. Alternatively, the system can request user input to select from among multiple candidate values for a parameter or to confirm a value determined by the system.
Using the prompt template and the first and second parameter values, the computer system generates, at operation 412 , a prompt to a large language model to cause the LLM to generate text. The text generated by the LLM can be output to the user for approval or to enable the user to request regeneration of the text. In some implementations, if regeneration of the text is requested, the computer system requests additional input from the user to generate a new prompt. For example, the computer system sends the user information about one or more automatically determined parameter values, enabling the user to confirm or modify the values. If the user supplies a modified value to replace an automatically determined value, the system modifies the prompt to the LLM to include the modified value instead of the automatically determined value.
Once the generated text has been approved by the user, at operation 414 , the computer system adds the generated text to the section on the webpage.
The process 400 can be repeated as the user adds additional sections to the website or modifies existing sections. As the user supplies inputs for parameter values, the computer system stores the inputs for use in generating prompt templates for future sections of the webpage. Accordingly, as steps of the process 400 are repeated to generate additional text for a website, fewer user inputs may be needed over time, and some prompts may be generated automatically without any additional user input.
Neural Networks and Machine Learning
To assist in understanding the present disclosure, some concepts relevant to neural networks and machine learning (ML) are first discussed.
Generally, a neural network comprises a number of computation units (sometimes referred to as âneuronsâ). Each neuron receives an input value and applies a function to the input to generate an output value. The function typically includes a parameter (also referred to as a âweightâ) whose value is learned through the process of training. A plurality of neurons may be organized into a neural network layer (or simply âlayerâ) and there may be multiple such layers in a neural network. The output of one layer may be provided as input to a subsequent layer. Thus, input to a neural network may be processed through a succession of layers until an output of the neural network is generated by a final layer. This is a simplistic discussion of neural networks and there may be more complex neural network designs that include feedback connections, skip connections, and/or other such possible connections between neurons and/or layers, which need not be discussed in detail here.
A deep neural network (DNN) is a type of neural network having multiple layers and/or a large number of neurons. The term DNN may encompass any neural network having multiple layers, including convolutional neural networks (CNNs), recurrent neural networks (RNNs), and multilayer perceptrons (MLPs), among others.
DNNs are often used as ML-based models for modeling complex behaviors (e.g., human language, image recognition, object classification, etc.) in order to improve accuracy of outputs (e.g., more accurate predictions) such as, for example, as compared with models with fewer layers. In the present disclosure, the term âML-based modelâ or more simply âML modelâ may be understood to refer to a DNN. Training a ML model refers to a process of learning the values of the parameters (or weights) of the neurons in the layers such that the ML model is able to model the target behavior to a desired degree of accuracy. Training typically requires the use of a training dataset, which is a set of data that is relevant to the target behavior of the ML model. For example, to train a ML model that is intended to model human language (also referred to as a language model), the training dataset may be a collection of text documents, referred to as a text corpus (or simply referred to as a corpus). The corpus may represent a language domain (e.g., a single language), a subject domain (e.g., scientific papers), and/or may encompass another domain or domains, be they larger or smaller than a single language or subject domain. For example, a relatively large, multilingual and non-subject-specific corpus may be created by extracting text from online webpages and/or publicly available social media posts. In another example, to train a ML model that is intended to classify images, the training dataset may be a collection of images. Training data may be annotated with ground truth labels (e.g. each data entry in the training dataset may be paired with a label), or may be unlabeled.
Training a ML model generally involves inputting into an ML model (e.g. an untrained ML model) training data to be processed by the ML model, processing the training data using the ML model, collecting the output generated by the ML model (e.g. based on the inputted training data), and comparing the output to a desired set of target values. If the training data is labeled, the desired target values may be, e.g., the ground truth labels of the training data. If the training data is unlabeled, the desired target value may be a reconstructed (or otherwise processed) version of the corresponding ML model input (e.g., in the case of an autoencoder), or may be a measure of some target observable effect on the environment (e.g., in the case of a reinforcement learning agent). The parameters of the ML model are updated based on a difference between the generated output value and the desired target value. For example, if the value outputted by the ML model is excessively high, the parameters may be adjusted so as to lower the output value in future training iterations. An objective function is a way to quantitatively represent how close the output value is to the target value. An objective function represents a quantity (or one or more quantities) to be optimized (e.g., minimize a loss or maximize a reward) in order to bring the output value as close to the target value as possible. The goal of training the ML model typically is to minimize a loss function or maximize a reward function.
The training data may be a subset of a larger data set. For example, a data set may be split into three mutually exclusive subsets: a training set, a validation (or cross-validation) set, and a testing set. The three subsets of data may be used sequentially during ML model training. For example, the training set may be first used to train one or more ML models, each ML model, e.g., having a particular architecture, having a particular training procedure, being describable by a set of model hyperparameters, and/or otherwise being varied from the other of the one or more ML models. The validation (or cross-validation) set may then be used as input data into the trained ML models to, e.g., measure the performance of the trained ML models and/or compare performance between them. Where hyperparameters are used, a new set of hyperparameters may be determined based on the measured performance of one or more of the trained ML models, and the first step of training (i.e., with the training set) may begin again on a different ML model described by the new set of determined hyperparameters. In this way, these steps may be repeated to produce a more performant trained ML model. Once such a trained ML model is obtained (e.g., after the hyperparameters have been adjusted to achieve a desired level of performance), a third step of collecting the output generated by the trained ML model applied to the third subset (the testing set) may begin. The output generated from the testing set may be compared with the corresponding desired target values to give a final assessment of the trained ML model's accuracy. Other segmentations of the larger data set and/or schemes for using the segments for training one or more ML models are possible.
Backpropagation is an algorithm for training a ML model. Backpropagation is used to adjust (also referred to as update) the value of the parameters in the ML model, with the goal of optimizing the objective function. For example, a defined loss function is calculated by forward propagation of an input to obtain an output of the ML model and comparison of the output value with the target value. Backpropagation calculates a gradient of the loss function with respect to the parameters of the ML model, and a gradient algorithm (e.g., gradient descent) is used to update (i.e., âlearnâ) the parameters to reduce the loss function. Backpropagation is performed iteratively, so that the loss function is converged or minimized. Other techniques for learning the parameters of the ML model may be used. The process of updating (or learning) the parameters over many iterations is referred to as training. Training may be carried out iteratively until a convergence condition is met (e.g., a predefined maximum number of iterations has been performed, or the value outputted by the ML model is sufficiently converged with the desired target value), after which the ML model is considered to be sufficiently trained. The values of the learned parameters may then be fixed and the ML model may be deployed to generate output in real-world applications (also referred to as âinferenceâ).
In some examples, a trained ML model may be fine-tuned, meaning that the values of the learned parameters may be adjusted slightly in order for the ML model to better model a specific task. Fine-tuning of a ML model typically involves further training the ML model on a number of data samples (which may be smaller in number/cardinality than those used to train the model initially) that closely target the specific task. For example, a ML model for generating natural language that has been trained generically on publicly-available text corpuses may be, e.g., fine-tuned by further training using the complete works of Shakespeare as training data samples (e.g., where the intended use of the ML model is generating a scene of a play or other textual content in the style of Shakespeare).
FIG. 5 A is a simplified diagram of an example CNN 510 , which is an example of a DNN that is commonly used for image processing tasks such as image classification, image analysis, object segmentation, etc. An input to the CNN 510 may be a 2D RGB image 512 .
The CNN 510 includes a plurality of layers that process the image 512 in order to generate an output, such as a predicted classification or predicted label for the image 512 . For simplicity, only a few layers of the CNN 510 are illustrated including at least one convolutional layer 514 . The convolutional layer 514 performs convolution processing, which may involve computing a dot product between the input to the convolutional layer 514 and a convolution kernel. A convolutional kernel is typically a 2D matrix of learned parameters that is applied to the input in order to extract image features. Different convolutional kernels may be applied to extract different image information, such as shape information, color information, etc.
The output of the convolution layer 514 is a set of feature maps 516 (sometimes referred to as activation maps). Each feature map 516 generally has smaller width and height than the image 512 . The set of feature maps 516 encode image features that may be processed by subsequent layers of the <figure-callout id="510" label="CNN"
CLAIMS
Claims ( 20 )
I claim:
1. A computer-implemented method comprising:
obtaining a prompt template associated with a section of a webpage, the prompt template including one or more parameters;
determining, based on the webpage, a first value for a first one of the one or more parameters;
sending for display to a user, a request to provide input for a second value corresponding to a second one of the one or more parameters; and
generating, using the prompt template and the first value and the second value, a prompt to a large language model to generate text for the section of the webpage.
2. The computer-implemented method of claim 1 , wherein sending the request for display to the user comprises:
selecting, based on the webpage, multiple candidate values for the second value;
sending the candidate values for display to the user; and
in response to receiving a selection from among the candidate values, identifying the selected candidate value as the second value.
3. The computer-implemented method of claim 1 , wherein sending the request for display to the user comprises:
predicting the second value based on the webpage;
determining a confidence score for the prediction of the second value is less than a threshold; and
sending the request to provide input for the second value to the user in response to determining the confidence score is less than the threshold.
4. The computer-implemented method of claim 1 , further comprising:
sending the generated text for display to a user; and
in response to receiving a request from the user to regenerate the text:
sending information indicative of the first value for display to the user; and
in response to receiving, from the user, a third value to replace the determined first value, modifying the prompt to the large language model based on the third value and the prompt template.
5. The computer-implemented method of claim 1 , further comprising:
receiving an input to add the section to the webpage, wherein the prompt to the large language model is generated in response to receiving the input;
generating the section on the webpage; and
adding the generated text to the generated section of the webpage.
6. The computer-implemented method of claim 5 , further comprising:
recommending a section type for the section to be added to the webpage, wherein the recommended section type is one of multiple section types available for adding to the webpage and wherein each of the multiple types of sections is associated with a different prompt template;
wherein obtaining the prompt template associated with the section of the webpage comprises obtaining the prompt template associated with the recommended section type of the section.
7. The computer-implemented method of claim 6 , wherein recommending the section type for the section to be added to the webpage comprises:
using a trained machine learning model to generate the recommended section type based on current contents of the webpage, wherein the trained machine learning model is trained based on layouts of other webpages to predict a likely next section type for the webpage based on the current contents of the webpage.
8. The computer-implemented method of claim 6 , wherein receiving the input to add the section to the webpage comprises receiving a specified location for the section on the webpage, and wherein recommending the section type of the section to be added to the webpage comprises:
selecting the recommended section type based on the specified location for the section.
9. The computer-implemented method of claim 8 , further comprising:
receiving an input to move the section from the specified location to a second location on the webpage; and
modifying the section type of the section based on the second location;
wherein obtaining the prompt template associated with the section of the webpage comprises retrieving the prompt template that is associated with the modified section type of the section.
10. The computer-implemented method of claim 1 , wherein the first one of the one or more parameters specifies a tone for the text, and wherein determining the first value based on the webpage comprises:
sending text retrieved from other sections of the webpage to the large language model for analysis of a tone of the other sections; and
selecting a tone as the first value based on the analysis of the tone of the other sections performed by the large language model.
11. The computer-implemented method of claim 1 , wherein the first one of the one or more parameters specifies a tone for the text, and wherein determining the first value based on the webpage comprises:
identifying a product category of a product to be described in the section; and
selecting a tone as the first value based on the product category.
12. The computer-implemented method of claim 1 , wherein determining the first value comprises retrieving the first value from another section on the webpage.
13. A non-transitory computer readable storage medium storing executable instructions, execution of which by a processor causing the processor to:
obtain a prompt template associated with a section of a webpage, the prompt template including one or more parameters;
determine, based on the webpage, a first value for a first one of the one or more parameters;
send for display to a user, a request to provide input for a second value corresponding to a second one of the one or more parameters; and
generate, using the prompt template and the first value and the second value, a prompt to a large language model to generate text for the section of the webpage.
14. The non-transitory computer readable storage medium of claim 13 , wherein sending the request for display to the user comprises:
selecting, based on the webpage, multiple candidate values for the second value;
sending the candidate values for display to the user; and
in response to receiving a selection from among the candidate values, identifying the selected candidate value as the second value.
15. The non-transitory computer readable storage medium of claim 13 , wherein sending the request for display to the user comprises:
predicting the second value based on the webpage;
determining a confidence score for the prediction of the second value is less than a threshold; and
sending the request to provide input for the second value to the user in response to determining the confidence score is less than the threshold.
16. The non-transitory computer readable storage medium of claim 13 , further comprising:
receiving an input to add the section to the webpage, wherein the prompt to the large language model is generated in response to receiving the input;
generating the section on the webpage; and
adding the generated text to the generated section of the webpage.
17. The non-transitory computer readable storage medium of claim 16 , further comprising:
recommending a section type for the section to be added to the webpage, wherein the recommended section type is one of multiple section types available for adding to the webpage and wherein each of the multiple types of sections is associated with a different prompt template;
wherein obtaining the prompt template associated with the section of the webpage comprises obtaining the prompt template associated with the recommended section type of the section.
18. The non-transitory computer readable storage medium of claim 13 , wherein the first one of the one or more parameters specifies a tone for the text, and wherein determining the first value based on the webpage comprises:
sending text retrieved from other sections of the webpage to the large language model for analysis of a tone of the other sections; and
selecting a tone as the first value based on the analysis of the tone of the other sections performed by the large language model.
19. The non-transitory computer readable storage medium of claim 13 , wherein determining the first value comprises retrieving the first value from another section on the webpage.
20. A system comprising:
at least one hardware processor; and
at least one non-transitory memory storing instructions, which, when executed by the at least one hardware processor, cause the system to:
obtain a prompt template associated with a section of a webpage, the prompt template including one or more parameters;
determine, based on the webpage, a first value for a first one of the one or more parameters;
send for display to a user, a request to provide input for a second value corresponding to a second one of the one or more parameters; and
generate, using the prompt template and the first value and the second value, a prompt to a large language model to generate text for the section of the webpage.
US18/322,377
2023-04-03
2023-05-23
Systems and methods for dynamic large language model prompt generation
Active
US12182506B2
( en )
Priority Applications (1)
Application Number
Priority Date
Filing Date
Title
US18/322,377
US12182506B2
( en )
2023-04-03
2023-05-23
Systems and methods for dynamic large language model prompt generation
Applications Claiming Priority (2)
Application Number
Priority Date
Filing Date
Title
US202363493833P
2023-04-03
2023-04-03
US18/322,377
US12182506B2
( en )
2023-04-03
2023-05-23
Systems and methods for dynamic large language model prompt generation
Publications (2)
Publication Number
Publication Date
US20240330579A1
US20240330579A1 ( en )
2024-10-03
US12182506B2
true
US12182506B2 ( en )
2024-12-31
Family
ID=92897600
Family Applications (2)
Application Number
Title
Priority Date
Filing Date
US18/322,377
Active
US12182506B2
( en )
2023-04-03
2023-05-23
Systems and methods for dynamic large language model prompt generation
US18/332,549
Pending
US20240329942A1
( en )
2023-04-03
2023-06-09
Systems and methods for dynamic large language model prompt generation
Family Applications After (1)
Application Number
Title
Priority Date
Filing Date
US18/332,549
Pending
US20240329942A1
( en )
2023-04-03
2023-06-09
Systems and methods for dynamic large language model prompt generation
Country Status (1)
Country
Link
US
( 2 )
US12182506B2
( en )
Cited By (3)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US20250298969A1
( en )
*
2024-03-21
2025-09-25
ManyChat, Inc.
Utilizing a large language model to generate an automated communication flow
US12499172B1
( en )
*
2025-01-31
2025-12-16
HOSTINGER operations, UAB
Website generation using language models
US12619399B1
( en )
*
2023-06-30
2026-05-05
Amazon Technologies, Inc.
Requirements discovery for generative ai software development assistant
Families Citing this family (23)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US12346387B2
( en )
*
2022-11-28
2025-07-01
Sav.com, LLC
Systems and methods for automatically generating a website and related marketing assets using generative artificial intelligence
US12079587B1
( en )
*
2023-04-18
2024-09-03
OpenAI Opco, LLC
Multi-task automatic speech recognition system
US20240419917A1
( en )
*
2023-06-14
2024-12-19
Microsoft Technology Licensing, Llc.
Customized prompt generation service for software engineering tasks
US20240427807A1
( en )
*
2023-06-23
2024-12-26
Crowdstrike, Inc.
Funnel techniques for natural language to api calls
US20250005299A1
( en )
*
2023-06-30
2025-01-02
Salesforce, Inc.
Language model prompt authoring and execution in a database system
US20250068893A1
( en )
*
2023-08-24
2025-02-27
Adobe Inc.
Generating personalized content using generative artificial intelligence
US20250086394A1
( en )
*
2023-09-07
2025-03-13
Sap Se
Digital assistant generation via large language models
US20250094788A1
( en )
*
2023-09-18
2025-03-20
Promoted.ai, Inc.
Cross-relevant refinement of generative artificial intelligence models for creative content across multiple platforms
US12307492B2
( en )
2023-09-18
2025-05-20
Promoted.ai, Inc.
Dynamic contextual generation of creative content for product listings
US20250181321A1
( en )
*
2023-12-04
2025-06-05
The Toronto-Dominion Bank
Design-based intelligent engine
US20250198790A1
( en )
*
2023-12-15
2025-06-19
Google Llc
Generating a Customized Digital Map Via a Generative Machine-Learned Model
US20250209282A1
( en )
*
2023-12-21
2025-06-26
Fujitsu Limited
Data adjustment using large language model
US12430657B2
( en )
*
2024-01-24
2025-09-30
ComplyAuto IP LLC
Dynamic automated content compliance verification
US20250252261A1
( en )
*
2024-02-07
2025-08-07
Oracle International Corporation
Multi-task learning for natural language processing tasks using a shared pre-trained language model
US12591793B2
( en )
2024-04-11
2026-03-31
Citibank, N.A.
Hierarchical cascade architecture of semantic fingerprinting operations for agent routing
US12536406B2
( en )
*
2024-04-11
2026-01-27
Citibank, N.A.
Dynamic artificial intelligence agent orchestration using a large language model gateway router
US20250335520A1
( en )
*
2024-04-29
2025-10-30
MainFunc Inc.
Generative AI Search Engine
US12437022B1
( en )
*
2024-05-15
2025-10-07
Skyword, Inc.
Dynamic generation of work requests
US20260024112A1
( en )
*
2024-07-22
2026-01-22
Yahoo Ad Tech Llc
System and method for query autosuggestion
WO2026024617A1
( en )
*
2024-07-22
2026-01-29
Endless Technologies Ltd.
A method and a system for selecting video frames
CN120045658B
( en )
*
2025-01-02
2025-11-07
ä¸ç§»(èå·)软件ææ¯æéå ¬å¸
Dialogue generation methods, apparatuses, electronic devices, storage media, and computer program products
CN120256703B
( en )
*
2025-06-05
2026-03-27
ä¸æµ·ç¨å®ç§ææéå ¬å¸
A method and apparatus for generating page parameters for a webpage.
CN120670335B
( en )
*
2025-08-21
2025-10-28
åå·ä¹ä¸ºç§ææéå ¬å¸
File access coldness and warmness calculation method, system and medium based on large model
Citations (1)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US11860914B1
( en )
*
2023-02-27
2024-01-02
Casetext, Inc.
Natural language database generation and query system
Family Cites Families (6)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US20230297887A1
( en )
*
2022-03-15
2023-09-21
Ada Support Inc.
Systems and methods for generating automatic training suggestions
US12333242B2
( en )
*
2022-03-31
2025-06-17
Docusign, Inc.
Embedded tag interface for document generation in a document management system
US12277396B2
( en )
*
2022-09-30
2025-04-15
Tenyx, Inc.
Assessing and improving the deployment of large language models in specific domains
CN116166942B
( en )
*
2022-12-27
2026-03-20
æ·±å³å¸ä¼å¿ éç§æè¡ä»½æéå ¬å¸
Classification model training methods, devices, equipment and storage media
US12159119B2
( en )
*
2023-02-15
2024-12-03
Casetext, Inc.
Text generation interface system
US11861320B1
( en )
*
2023-02-27
2024-01-02
Casetext, Inc.
Text reduction and analysis interface to a text generation modeling system
2023
2023-05-23
US
US18/322,377
patent/US12182506B2/en
active
Active
2023-06-09
US
US18/332,549
patent/US20240329942A1/en
active
Pending
Patent Citations (1)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US11860914B1
( en )
*
2023-02-27
2024-01-02
Casetext, Inc.
Natural language database generation and query system
Cited By (3)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US12619399B1
( en )
*
2023-06-30
2026-05-05
Amazon Technologies, Inc.
Requirements discovery for generative ai software development assistant
US20250298969A1
( en )
*
2024-03-21
2025-09-25
ManyChat, Inc.
Utilizing a large language model to generate an automated communication flow
US12499172B1
( en )
*
2025-01-31
2025-12-16
HOSTINGER operations, UAB
Website generation using language models
Also Published As
Publication number
Publication date
US20240330579A1
( en )
2024-10-03
US20240329942A1
( en )
2024-10-03
Similar Documents
Publication
Publication Date
Title
US20240330579A1
( en )
2024-10-03
Systems and methods for dynamic large language model prompt generation
US20240256792A1
( en )
2024-08-01
Methods and systems for dynamic generation of personalized text using large language model
US12468878B2
( en )
2025-11-11
Methods and systems for generation of text using large language model with indications of unsubstantiated information
US12561512B2
( en )
2026-02-24
Methods and systems for prompting large language model to generate formatted output
US12608548B2
( en )
2026-04-21
Methods and systems for parsing a mix of features and instructions into a prompt
US20250356111A1
( en )
2025-11-20
Methods and systems for prompting large language model to process inputs from multiple user elements
US12411699B2
( en )
2025-09-09
Dynamic generation of user interface controls
US20240256793A1
( en )
2024-08-01
Methods and systems for generating text with tone or diction corresponding to stylistic attributes of images
US12596876B2
( en )
2026-04-07
Systems and methods for improving textual descriptions using large language models
US12579377B2
( en )
2026-03-17
Summary of reviews generated by a generative language model
US20240311192A1
( en )
2024-09-19
Methods and systems for indicating resource usage parameter for prompting a large language model (llm)
US11403076B2
( en )
2022-08-02
Functional and code views in a process workflow
CA3169486A1
( en )
2023-08-11
Systems and methods for computer generation of a modifiable product description
US20240354710A1
( en )
2024-10-24
Methods and systems for the creation of parsers using large language models
US11657107B2
( en )
2023-05-23
Systems and methods for using keywords extracted from reviews
US20250165228A1
( en )
2025-05-22
Generation of grammar-compliant programming language code using machine learning
US12430661B2
( en )
2025-09-30
Systems and methods for processing product information using a machine-learning model
US20240386477A1
( en )
2024-11-21
Methods and systems for providing context for generating an online store
US20240354176A1
( en )
2024-10-24
Notification messages generated by a generative language model
US11308542B2
( en )
2022-04-19
Systems and methods for using keywords extracted from reviews
US20250166037A1
( en )
2025-05-22
Embedding-based search of an item store
US20250165125A1
( en )
2025-05-22
Methods and devices for exploring an n-dimensional embedding space using a navigation-based recommender
US20240354830A1
( en )
2024-10-24
System and Method for Identifying Complementary Objects from Images
AU2023476117A1
( en )
2026-04-23
Methods and systems for resolving and providing context for client-permissioned content
US20240378492A1
( en )
2024-11-14
Systems and methods for training and using a machine-learning model for determining the similarity of entities
Legal Events
Date
Code
Title
Description
2023-05-23
FEPP
Fee payment procedure
Free format text : ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY
2023-05-26
AS
Assignment
Owner name : SHOPIFY INC., CANADA
Free format text : ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:SAXENA, UTKARSH;REEL/FRAME:063771/0627
Effective date : 20230524
2024-08-23
STPP
Information on status: patent application and granting procedure in general
Free format text : NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS
2024-11-27
STPP
Information on status: patent application and granting procedure in general
Free format text : PUBLICATIONS -- ISSUE FEE PAYMENT RECEIVED
2024-12-02
STPP
Information on status: patent application and granting procedure in general
Free format text : PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED
2024-12-11
STCF
Information on status: patent grant
Free format text : PATENTED CASE