Conceptio › Archive › Google Patents

Google Patentsopen access

Techniques for incremental computer-based natural language understanding — Disney Enterprises, Inc. (US11749265B2)

Disney Enterprises, Inc. · Google Patents

Google Patents · Patents · License: Open Access

Open Source ↗

disneyenterprises

patent, google patents, intellectual property, US11749265B2, Disney Enterprises, Inc., Erika Varis Doggett, en, 2023

ABSTRACT

Abstract

Various embodiments disclosed herein provide techniques for performing incremental natural language understanding on a natural language understanding (NLU) system. The NLU system acquires a first audio speech segment associated with a user utterance. The NLU system converts the first audio speech segment into a first text segment. The NLU system determines a first intent based on a text string associated with the first text segment, wherein the text string represents a portion of the user utterance. The NLU system generates a first response based on the first intent prior to when the user utterance completes.

Description

BACKGROUND

Field of the Various Embodiments

The various embodiments relate generally to computer-based language processing technology and, more specifically, to techniques for incremental computer-based natural language understanding.

Description of the Related Art

Computer-based natural language understanding (NLU) has become a widely popular approach for facilitating human-machine interfaces (HMI) between human users and computer systems. Computer-based NLU is typically included as part of a dialogue system. Such dialogue systems are often deployed in automated chat-based applications, referred to herein as âchatbots,â personal digital assistants, and artificial intelligence (AI) avatars. Typically, a computer-based dialogue system receives audio input from a user via an audio input device, such as a microphone. Notably, the dialogue system waits until an entire spoken utterance from the user has been processed by an automatic speech recognition (ASR) module and received before taking any responsive actions. Such an ASR module can employ various techniques for determining that an entire utterance has been received from a user. For example, the ASR module could wait for a detectable pause in the audio input of a certain duration before concluding that the entirety of a user utterance has been received. The ASR module system then converts the audio input representing the user utterance into text. Then, the dialogue system analyzes the text, and generates an appropriate response. The response is usually in the form of text output, an audio output, and/or a visual output.

One drawback of conventional dialogue systems is that, because a dialogue system does not begin processing until after an entire user utterance has been received, the dialogue system does not generate any responses while a person is speaking, which can feel quite unnatural to the user. For example, during a typical verbal interaction between two people, each person interjects verbal and non-verbal communications into the interaction. In that regard, while the person is speaking, the second person could interject phrases, such as âuh huh,â âI understand,â âI'm so sorry,â and âI'm very happy for you,â in response to what the first person is saying. Similarly, the second person also could interject non-verbal expressions, such as winces, smiles, frowns, grimaces, or other facial expressions, in response to what the first person is saying. As noted above, conventional automated dialogue systems do not interject verbal or non-verbal communications into interactions with users, while the users are speaking. Accordingly, interactions with dialogue systems can feel âroboticâ and âstiffâ to users, which decreases the quality of the overall user experience.

As the foregoing illustrates, what is needed in the art are more effective techniques for implementing human-machine interfaces using natural language understanding.

SUMMARY

Various embodiments disclosed herein provide a method for performing incremental natural language understanding. The method includes acquiring a first audio speech segment associated with a user utterance. The method further includes converting the first audio speech segment into a first text segment. The method further includes determining a first intent based on a text string associated with the first text segment, wherein the text string represents a portion of the user utterance. The method further includes generating a first response based on the first intent prior to when the user utterance completes. The method further includes preparing for a first response or other action based on the first intent prior to when the user utterance completes.

Other embodiments include, without limitation, one or more non-transitory computer-readable media storing instructions for performing one or more aspects of the disclosed techniques, as well as a system for performing one or more aspects of the disclosed techniques.

At least one technical advantage of the disclosed techniques relative to the prior art is that the disclosed techniques enable a dialogue system to verbally and/or visually respond to audio received from a user as the user is speaking, without having to wait for the user to finish speaking. Thus, the disclosed techniques allow the dialogue system to interact with the user in a more natural, âhuman-likeâ manner, thereby enhancing the overall user experience. These technical advantages represent one or more technological advancements over prior art approaches.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the manner in which the above recited features of the various embodiments can be understood in detail, a more particular description of the inventive concepts, briefly summarized above, may be had by reference to various embodiments, some of which are illustrated in the appended drawings. It is to be noted, however, that the appended drawings illustrate only typical embodiments of the inventive concepts and are therefore not to be considered limiting of scope in any way, and that there are other equally effective embodiments.

FIG. 1 A illustrates a system configured to implement one or more aspects of the various embodiments;

FIG. 1 B illustrates a more detailed view of one of the edge devices of FIG. 1 A , according to various embodiments;

FIG. 1 C illustrates a more detailed view of the dialogue system of FIG. 1 A , according to various embodiments;

FIG. 2 illustrates how segments of audio speech are processed by the dialogue system of FIG. 1 C , according to various embodiments;

FIG. 3 is a more detailed illustration of the natural language understanding (NLU) application of FIG. 2 , according to various embodiments;

FIG. 4 illustrates a multi-modal version of the artificial intelligence (AI) response application of FIG. 1 C , according to various embodiments; and

FIGS. 5 A- 5 B set forth a flow diagram of method steps for performing incremental computer-based natural language understanding, according to various embodiments.

DETAILED DESCRIPTION

In the following description, numerous specific details are set forth to provide a more thorough understanding of the various embodiments. However, it will be apparent to one of skill in the art that the various embodiments may be practiced without one or more of these specific details.

As further described herein, a dialogue system performs incremental automatic speech recognition and incremental natural language understanding on short segments of audio speech input in real-time. In contrast to conventional approaches, the dialogue system does not wait to receive a complete user utterance before starting to analyze and respond to audio speech input. Instead, the dialogue system generates textual, verbal, and/or visual responses as audio speech input is received. User utterances are divided into smaller portions, such as characters, words, phonological phrases, or any technically feasible portions thereof. As used herein, a phonological phrase is a portion of a user utterance. A phonological phrase is typically not a full sentence. Instead, a phonological phrase is typically a phrase within a sentence. In general, audio markers are used to determine the beginning and ending of phonological phrases. These audio markers include, without limitation, intonation cues, verbal pauses, and other base cues.

The smaller portions are converted into text segments, and the text segments are analyzed to generate the predicted intent of the speaker. The dialogue system performs additional processing to further refine the predicted intent, such as by analyzing the audio speech input for user intonation cues. The dialogue system further analyzes various other verbal and non-verbal inputs to refine the predicted intent, such as cultural factors, inflections, predicted mood or emotion. The dialogue system further analyzes information from video feeds of the speaker as the speaker is talking and as the speaker is reacting to responses generated by the dialogue system. The dialogue system generates incrementally predicted intents before the full user utterance is received. The dialogue system compares a confidence score for a predicted intent against a threshold value. If the confidence score for the predicted intent is greater than the threshold value, then the dialogue system generates an appropriate context-specific response, based on the predicted intent. In addition to generating predicted intents, the dialogue system analyzes and predicts other aspects of the utterance received from the user. More particularly, the dialogue system analyzes and predicts sentiment, stance, emotion, topic, and various other elements of meaning associated with the utterance.

A natural language generation (NLG) system associated with or included in the dialogue system generates responses, and/or prepares for responses and other actions, for controlling an interactive character, such as a chatbot, a personal digital assistant, or an AI avatar. For example, if the user reacts in a certain way as the interactive character is speaking, then the dialogue system modifies the response of the interactive character in real-time before the speaker has finished talking. The dialogue system allows for programming additional personality traits that influence the responses of the interactive character, such as excitability, curiosity, and propensity to interrupt the speaker. The NLG system associated with the dialogue system continues to generate responses and/or prepare for responses and other actions based on the first intent prior to when the user utterance completes, such as interrupting and nodding head in agreement, to animate the interactive character based on the predicted intent of the speaker. An overview of the dialogue system is now described in further detail.

System Overview

FIG. 1 A illustrates a system 190 configured to implement one or more aspects of the present invention. As shown, the system 190 includes, without limitation, edge devices 120 and a dialogue system 100 in communication with each other via a computer network 130 .

Computer network 130 may be any suitable environment to enable communications among remote or local computer systems and computing devices, including, without limitation, point-to-point communications channels, Bluetooth, WiFi, cable-based networks, fiber optic networks, 4G and 5G cellular networks, infrared communications, wireless and wired LANs (Local Area Networks), and one or more internet-based WANs (Wide Area Networks).

Each edge device 120 includes, without limitation, a computing device that may be a personal computer, personal digital assistant, mobile phone, mobile device, smart speaker, vehicle, gaming device, interactive experience environment, interactive attraction, or any other device or environment suitable for implementing one or more aspects of the present invention. Illustratively, the edge device 120 communicates over network 130 via communications link 152 .

In operation, the edge device 120 receives audio, visual and other information associated with utterances spoken by a user. The edge device 120 receives audio information via an audio input device, such as a microphone. Further, the edge device 120 receives visual information via a video input device, such as a camera. The edge device 120 may receive the audio, visual and other information in analog or digital form. If the edge device 120 receives the audio, visual and other information in analog form, the edge device may perform analog-to-digital conversion to convert the information into digital form. The edge device 120 transmits the audio, visual and other information, in analog and/or digital form, to the dialogue system 100 . In response, the dialogue system 100 transmits one or more responses to the edge device 120 . The responses may include text, audio, and/or visual information. Further, the responses may be in analog or digital form. The edge device 120 presents the response on one or more output devices. The edge device 120 presents audio information via an audio output device, such as one or more speakers. Further, the edge device 120 presents text and/or visual information via a video output device, such as a display device.

The dialogue system 100 , includes, without limitation, a computing device that may be a standalone server, a cluster or âfarmâ of servers, one or more network appliances, or any other device suitable for implementing one or more aspects of the present invention. The dialogue system 100 includes any one or more of local servers, remote servers, or cloud-based servers, in any technically feasible combination. Illustratively, the <figure-cal

BACKGROUND

Field of the Various Embodiments

The various embodiments relate generally to computer-based language processing technology and, more specifically, to techniques for incremental computer-based natural language understanding.

Description of the Related Art

As the foregoing illustrates, what is needed in the art are more effective techniques for implementing human-machine interfaces using natural language understanding.

SUMMARY

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 A illustrates a system configured to implement one or more aspects of the various embodiments;

FIG. 1 B illustrates a more detailed view of one of the edge devices of FIG. 1 A , according to various embodiments;

FIG. 1 C illustrates a more detailed view of the dialogue system of FIG. 1 A , according to various embodiments;

FIG. 2 illustrates how segments of audio speech are processed by the dialogue system of FIG. 1 C , according to various embodiments;

FIG. 3 is a more detailed illustration of the natural language understanding (NLU) application of FIG. 2 , according to various embodiments;

FIG. 4 illustrates a multi-modal version of the artificial intelligence (AI) response application of FIG. 1 C , according to various embodiments; and

FIGS. 5 A- 5 B set forth a flow diagram of method steps for performing incremental computer-based natural language understanding, according to various embodiments.

DETAILED DESCRIPTION

System Overview

In operation, the dialogue system 100 acquires input information of short duration, such as audio speech segments, visual information, and other information, from one or more edge devices 120 . The dialogue system 100 converts each audio speech segment into a text segment via an automatic speech recognition process. The resulting text segment includes one or more letters that represent a portion of one or more words. Additionally or alternatively, the resulting text segment includes one or more words that represent a portion of a user utterance. The dialogue system 100 concatenates the text segment with other previously acquired text segments belonging to the same user utterance, thereby generating a concatenated text string. The dialogue system 100 then identifies one or more predicted intent results based on the concatenated text string. If the dialogue system 100 determines that the confidence scores for the predicted intent results are less than corresponding threshold values, then the dialogue system 100 generates one or more backchannel responses that are not context-specific. If, on the other hand, the dialogue system 100 determines that one or more confidence scores for the predicted intent results are greater than corresponding threshold values, then the dialogue system 100 generates one or more context-specific AI responses.

As the dialogue system 100 acquires and processes additional audio speech segments of short duration, the dialogue system 100 generates a more complete context of the user utterance being spoken and, correspondingly, an increasingly refined model of the predicted intent of the speaker. In response, the dialogue system 100 continues to generate backchannel responses and AI responses that correspond to this increasingly complete and refined model of the predicted intent of the speaker. In addition, the dialogue system 100 modifies the backchannel responses and AI responses with personality markers that define various personality traits of the chatbot, personal digital assistant, AI avatar or other responder associated with the dialogue system 100 . The dialogue system 100 further modifies the backchannel responses and AI responses based on intonation cues detected from the audio speech input, such as whether the speaker's voice is rising or trailing. The dialogue system 100 further modifies the backchannel responses and AI responses based on analysis of one or more video feeds of the speaker who is generating the user utterance.

FIG. 1 B illustrates a more detailed view of one of the edge devices 120 of FIG. 1 A , according to various embodiments. As shown, the dialogue system 100 includes, without limitation, a processor 102 A, storage 104 A, an input/output (I/O) devices interface 106 A, a network interface 108 A, an interconnect 110 A, and a system memory 112 A.

The processor 102 A retrieves and executes programming instructions stored in the system memory 112 A. Similarly, the processor 102 A stores and retrieves application data residing in the system memory 112 A. The interconnect 110 A facilitates transmission, such as of programming instructions and application data, between the processor 102 A, input/output (I/O) devices interface 106 A, storage 104 A, network interface 108 A, and system memory 112 A. The I/O devices interface 106 A is configured to receive input data from user I/ O devices 122 A. Examples of user I/ O devices 122 A may include one of more buttons, a keyboard, and a mouse or other pointing device. The I/O devices interface 106 A may also include an audio output unit configured to generate an electrical audio output signal, and user I/ O devices 122 A may further include a speaker configured to generate an acoustic output in response to the electrical audio output signal. Another example of a user I/ O device 122 A is a display device that generally represents any technically feasible means for generating an image for display. For example, the display device could be a liquid crystal display (LCD) display, CRT display, or DLP display. The display device may be a TV that includes a broadcast or cable tuner for receiving digital or analog television signals.

Processor 102 A is included to be representative of a single central processing unit (CPU), multiple CPUs, a single CPU having multiple processing cores, one or more graphics processing units (GPUs) each having multiple processing cores, and the like. And the system memory 112 A is generally included to be representative of a random access memory. The storage 104 A may be a disk drive storage device. Although shown as a single unit, the storage 104 A may be a combination of fixed and/or removable storage devices, such as fixed disc drives, floppy disc drives, tape drives, removable memory cards, or optical storage, network attached storage (NAS), or a storage area-network (SAN). Processor 102 A communicates to other computing devices and systems via network interface 108 A, where network interface 108 A is configured to transmit and receive data via a communications network, such as communications network 130 .

The system memory 112 A includes, without limitation, an edge device application 132 A and a data store 140 A. The edge device application 132 A, when executed by the processor 102 A, performs one or more operations associated with the edge device 120 , as further described herein. Data store 140 A provides memory storage for various items associated with the edge device 120 , as further described herein. The edge device application 132 A stores data in and retrieve data from data store 140 A, as further described herein.

In operation, the edge device application 132 A receives audio, visual and other information associated utterances spoken by a user. The edge device application 132 A receives audio information via an audio input device, such as a microphone. Further, the edge device application 132 A receives visual information via a video input device, such as a camera. The edge device application 132 A may receive the audio, visual and other information in analog or digital form. If the edge device application 132 A receives the audio, visual and other information in analog form, the edge device may perform analog-to-digital conversion to convert the information into digital form. The edge device application 132 A transmits the audio, visual and other information, in analog and/or digital form, to the dialogue system 100 . In response, the dialogue system 100 transmits one or more responses to the edge device application 132 A. The responses may include text, audio, and/or visual information. Further, the responses may be in analog or digital form. The edge device application 132 A presents the responses on one or more output devices. The edge device application 132 A presents audio information via an audio output device, such as one or more speakers. Further, the edge device application 132 A presents text and/or visual information via a video output device, such as a display device.

FIG. 1 C illustrates a more detailed view of the dialogue system 100 of FIG. 1 A , according to various embodiments. As shown, the dialogue system 100 includes, without limitation, a processor 102 B, storage 104 B, an input/output (I/O) devices interface 106 B, a network interface 108 B, an interconnect 110 B, and a system memory 112 B.

The processor 102 B retrieves and executes programming instructions stored in the system memory 112 B. Similarly, the processor 102 B stores and retrieves application data residing in the system memory 112 B. The interconnect 110 B facilitates transmission, such as of programming instructions and application data, between the processor 102 B, input/output (I/O) devices interface 106 B, storage 104 B, network interface 108 B, and system memory 112 B. The I/O devices interface 106 B is configured to receive input data from user I/ O devices 122 B. Examples of user I/ O devices 122 B may include one of more buttons, a keyboard, and a mouse or other pointing device. The I/O devices interface 106 B may also include an audio output unit configured to generate an electrical audio output signal, and user I/ O devices 122 B may further include a speaker configured to generate an acoustic output in response to the electrical audio output signal. Another example of a user I/ O device 122 B is a display device that generally represents any technically feasible means for generating an image for display. For example, the display device could be a liquid crystal display (LCD) display, CRT display, or DLP display. The display device may be a TV that includes a broadcast or cable tuner for receiving digital or analog television signals.

Processor 102 B is included to be representative of a single central processing unit (CPU), multiple CPUs, a single CPU having multiple processing cores, one or more graphics processing units (GPUs) each having multiple processing cores, and the like. And the system memory 112 B is generally included to be representative of a random access memory. The storage 104 B may be a disk drive storage device. Although shown as a single unit, the storage 104 B may be a combination of fixed and/or removable storage devices, such as fixed disc drives, floppy disc drives, tape drives, removable memory cards, or optical storage, network attached storage (NAS), or a storage area-network (SAN). Processor 102 B communicates to other computing devices and systems via network interface 108 B, where network interface 108 B is configured to transmit and receive data via a communications network, such as communications network 130 .

The system memory 112 B includes, without limitation, an automatic speech recognition (ASR) application 132 B, a natural language understanding (NLU) application 134 B, an artificial intelligence (AI) response application 136 B, and a data store 140 B. The ASR application 132 B, NLU application 134 B, and AI response application 136 B, when executed by the processor 102 B, performs one or more operations associated with the dialogue system 100 , as further described herein. Data store 140 B provides memory storage for various items associated with the dialogue system 100 , as further described herein. The ASR application 132 B, NLU application 134 B, and AI response application 136 BB store data in and retrieve data from data store 140 B, as further described herein.

In operation, the ASR application 132 B automatically performs recognition of speech by receiving audio input and converting the audio input into words in the form of text. The ASR application 132 B receives audio speech segments, where each audio speech segment represents a certain duration of time. After receiving an audio speech segment, the ASR application 132 B begins to generate a text segment that includes the characters and/or words represented by the audio speech segment. The ASR application 132 B continues to receive additional audio speech segments and generates corresponding text segments. Further, the ASR application 132 B applies a language model to the text segments in order to disambiguate text segments that include homonyms, context-dependent words, and the like. Consequently, the ASR application 132 B employs the language model to generate accurate text segments. The ASR application 132 B transmits each generated text segment to the NLU application 134 B.

In operation, the NLU application 134 B extracts understanding and meaning from spoken language. More specifically, the NLU application 134 B performs intent recognition and sentiment detection on language that a person has spoken. The NLU application 134 B does not directly operate on an incoming audio stream. Instead, the ASR application 132 B converts the incoming audio stream to words and other text. Then, the NLU application 134 B converts that text to a representation of meaning in the form of predicted intents and sentiments. For example, the NLU application 134 B could receive the text string âI hate broccoliâ from the ASR application 132 B. In response, the NLU application 134 B could convert the text string into a representation of a predicted intent and sentiment in the form of â<negative sentiment>[broccoli],â where â<negative sentiment>indicates a sentiment of dislike and [broccoli] specifies the subject of the dislike sentiment.

In particular, the NLU application 134 B receives each text segment generated by the ASR application 132 B. When the NLU application 134 B receives a text segment, the NLU application 134 B concatenates the text segment with other previously acquired text segments to form a concatenated text string. The NLU application 134 B applies text prediction to the concatenated text string to predict the next words received from the ASR application 132 B. In addition, the NLU application 134 B performs intent matching to generate a predicted intent based on the concatenated text string.

As the NLU application 134 B receives and processes text segments corresponding to a particular, the NLU application 134 B continually determines position in the text string corresponding to the end of an utterance. In so doing, the NLU application 134 B does not depend on a period of silence or a pause in the audio input. Instead, the NLU application 134 B predicts the end of the current utterance within the text string. Based on the predicted end of the current utterance, the NLU application 134 B predicts that the text following the predicted end of utterance corresponds to the beginning of the subsequent utterance. If the confidence score of the predicted end of utterance is greater than a threshold level, then the NLU application 134 B cuts the text string at the point of the predicted end of utterance. The text following the predicted end of utterance is not appended to the prior text string. Instead, the text following the predicted end of utterance begins a new text string for the subsequent utterance.

In some embodiments, after predicting the end of an utterance, the NLU application 134 B continually determines whether the prediction of the end of the utterance was correct. For example, if, after the predicted end of the utterance, the user continues to speak, the NLU application 134 B could determine that the previously predicted end of utterance was incorrect. The NLU application 134 B would then perform one or more actions to correct the error. For example, the NLU application 134 B could cause the AI response application 136 B to stop an AI response in progress, such as a verbal reply. The NLU application 134 B could then continue to wait and listen for the user to complete the utterance.

In operation, the AI response application 136 B performs natural language generation (NLG) which generates or writes a sentence and/or other response for an AI system to âsay.â Typically, the main focus of NLG is to generate a text response that is transmitted to a text-to-speech (TTS) application (not shown) to perform speech synthesis by translating the text response into an auditory signal. The NLG also generates non-speech responses such as a change in facial expression or body position that is consistent with the predicted intent and sentiment received from the NLU application 134 B.

In particular, the AI response application 136 B generates one or more responses that correspond to the received predicted intent and sentiment. As time progresses, the AI response application 136 B receives additional predicted intents from the NLU application 134 B. The AI response application 136 B determines which predicted intents correspond to the same user utterance or phonological phrase. The AI response application 136 B continues to update the response and generate new responses each time a new predicted intent is received from the NLU application 134 B. The AI response is based on the new predicted intent as well as previously received predicted intents that correspond to the same user utterance or phonological phrase.

In some embodiments, the AI response application 136 B may initiate the AI response or begin preparing to respond based on partial inputs from the ASR application 132 B and the NLU application 134 B, even before the utterance is complete. Further, the AI response application 136 B may acquire additional information that may be needed to generate the AI response before the utterance is complete. As a result, the dialogue system 100 may further increase the speed and efficiency of generating AI responses. As a result, the ability of the dialogue system 100 to interact with the user in a more natural, human-like manner may be further improved.

In some embodiments, one or more of the <figure-callout id="132B" label="AS

CLAIMS

Claims ( 18 )

What is claimed is:

1. A computer-implemented method for performing incremental natural language understanding, the method comprising:

acquiring, by a dialogue system that includes at least one processor and a neural network, a first audio speech segment associated with a user utterance;

converting the first audio speech segment into a first text segment;

executing the neural network to predict a first intent based on a text string associated with the first text segment, wherein the text string represents a portion of the user utterance;

determining a confidence score associated with the first intent;

modifying at least one marker that is input into the neural network based on at least one of a user associated with the user utterance or a behavior pattern of the user;

in response to determining that the confidence score is less than a threshold value, generating, using the neural network with the at least one modified marker, a first response that is unrelated to the first intent prior to when the user utterance completes, wherein the first response indicates that audio associated with the user utterance is being processed and is based on a non-intent specific response library; and

in response to determining that the confidence score is greater than the threshold value:

predicting an end of the user utterance based on the text string; and

generating, using the neural network with the at least one modified marker, a second response based on the end of the user utterance.

2. The computer-implemented method of claim 1 , further comprising determining that a second confidence score associated with a second intent is greater than the threshold value, and generating a third response by performing one or more operations based on an intent specific response library to generate a response that is related to the second intent.

3. The computer-implemented method of claim 1 , further comprising:

acquiring a second audio speech segment associated with the user utterance;

converting the second audio speech segment into a second text segment;

concatenating the second text segment to the text string to generate a concatenated text string;

determining a second intent based on the concatenated text string that is different than the first intent; and

generating a third response based on the second intent prior to the end of the user utterance.

4. The computer-implemented method of claim 1 , further comprising:

applying text prediction to the text string to determine a second text segment that is likely to follow the first text segment; and

prior to determining the first intent, concatenating the second text segment to the text string.

5. The computer-implemented method of claim 1 , further comprising:

determining a personality attribute weighting of an artificial intelligence avatar associated with the first response; and

modifying the first response based on the personality attribute weighting.

6. The computer-implemented method of claim 5 , wherein the personality attribute weighting includes at least one of an excitability weighting, a curiosity weighting, and an interruptability weighting.

7. The computer-implemented method of claim 1 , further comprising:

determining an intonation cue associated with the first audio speech segment; and

modifying the first response based on the intonation cue.

8. The computer-implemented method of claim 7 , wherein the intonation cue includes at least one of a rising intonation, a trailing intonation, and a declarative intonation.

9. The computer-implemented method of claim 1 , further comprising:

analyzing a video feed associated with the user utterance;

determining a second intent based on the video feed; and

modifying the first response based on the second intent.

10. One or more non-transitory computer-readable media storing program instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of:

acquiring, by a dialogue system that includes at least one processor and a neural network, a first audio speech segment associated with a user utterance;

converting the first audio speech segment into a first text segment;

concatenating the first text segment to a text string that represents a portion of the user utterance;

executing the neural network to predict a first intent based on the text string;

determining a confidence score associated with the first intent;

modifying at least one marker that is input into the neural network based on at least one of a user associated with the user utterance or a behavior pattern of the user;

in response to determining that the confidence score is greater than the threshold value:

predicting an end of the user utterance based on the text string; and

generating, using the neural network with the at least one modified marker, a second response based on the end of the user utterance.

11. The one or more non-transitory computer-readable media of claim 10 , further comprising determining that a second confidence score associated with a second intent is greater than the threshold value, and generating a third response by performing one or more operations based on an intent specific response library to generate a response that is related to the second intent.

12. The one or more non-transitory computer-readable media of claim 10 , further comprising:

acquiring a second audio speech segment associated with the user utterance;

converting the second audio speech segment into a second text segment;

concatenating the second text segment to the text string to generate a concatenated text string;

determining a second intent based on the concatenated text string that is different than the first intent; and

generating a third response based on the second intent prior to the end of the user utterance.

13. The one or more non-transitory computer-readable media of claim 12 , wherein a first duration of time represented by the first audio speech segment overlaps with a second duration of time represented by the second audio speech segment.

14. The one or more non-transitory computer-readable media of claim 12 , wherein a first duration of time represented by the first audio speech segment is non-overlapping with a second duration of time represented by the second audio speech segment.

15. The one or more non-transitory computer-readable media of claim 10 , further comprising:

applying text prediction to the text string to determine a second text segment that is likely to follow the first text segment; and

prior to determining the first intent, concatenating the second text segment to the text string.

16. The one or more non-transitory computer-readable media of claim 10 , further comprising:

determining a personality attribute weighting of an artificial intelligence avatar associated with the first response; and

modifying the first response based on the personality attribute weighting.

17. The one or more non-transitory computer-readable media of claim 10 , further comprising:

determining an intonation cue associated with the first audio speech segment; and

modifying the first response based on the intonation cue.

18. A system, comprising:

one or more memories that include instructions, and

one or more processors that are coupled to the one or more memories and, when executing the instructions, perform the steps of:

acquiring, by a dialogue system that includes a neural network, an audio speech segment associated with a user utterance;

converting the audio speech segment into a text segment;

executing the neural network to predict a first intent based on a text string associated with the text segment, wherein the text string represents a portion of the user utterance;

determining a confidence score associated with the first intent;

modifying at least one marker that is input into the neural network based on at least one of a user associated with the user utterance or a behavior pattern of the user;

in response to determining that the confidence score is greater than the threshold value:

predicting an end of the user utterance based on the text string; and

generating, using the neural network with the at least one modified marker, a second response based on the end of the user utterance.

US16/593,939

2019-10-04

Techniques for incremental computer-based natural language understanding

Active

2040-09-08

US11749265B2

( en )

Priority Applications (1)

Application Number

Priority Date

Filing Date

Title

US16/593,939

US11749265B2

( en )

2019-10-04

Techniques for incremental computer-based natural language understanding

Applications Claiming Priority (1)

Application Number

Priority Date

Filing Date

Title

US16/593,939

US11749265B2

( en )

2019-10-04

Techniques for incremental computer-based natural language understanding

Publications (2)

Publication Number

Publication Date

US20210104236A1

US20210104236A1 ( en )

2021-04-08

US11749265B2

true

US11749265B2 ( en )

2023-09-05

Family

ID=75274953

Family Applications (1)

Application Number

Title

Priority Date

Filing Date

US16/593,939

Active

2040-09-08

US11749265B2

( en )

2019-10-04

Techniques for incremental computer-based natural language understanding

Country Status (1)

Country

Link

( 1 )

US11749265B2

( en )

Cited By (3)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20220165264A1

( en )

2020-11-26

2022-05-26

Hyundai Motor Company

Dialogue system, vehicle, and method of controlling dialogue system

US20220230643A1

( en )

2022-04-01

2022-07-21

Intel Corporation

Technologies for enhancing audio quality during low-quality connection conditions

US12462110B2

( en )

2023-04-18

2025-11-04

Bank Of America Corporation

Natural language generation system for automated translation of digital multimedia

Families Citing this family (54)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US8977255B2

( en )

2007-04-03

2015-03-10

Apple Inc.

Method and system for operating a multi-function portable electronic device using voice-activation

US8676904B2

( en )

2008-10-02

2014-03-18

Apple Inc.

Electronic devices with voice command and contextual data processing capabilities

EP4560630A3

( en )

2013-02-07

2025-08-06

Apple Inc.

Voice trigger for a digital assistant

US9715875B2

( en )

2014-05-30

2017-07-25

Apple Inc.

Reducing the need for manual start/end-pointing and trigger phrases

US10170123B2

( en )

2014-05-30

2019-01-01

Apple Inc.

Intelligent assistant for home automation

US9338493B2

( en )

2014-06-30

2016-05-10

Apple Inc.

Intelligent automated assistant for TV user interactions

US9886953B2

( en )

2015-03-08

2018-02-06

Apple Inc.

Virtual assistant activation

US10460227B2

( en )

2015-05-15

2019-10-29

Apple Inc.

Virtual assistant in a communication session

US10747498B2

( en )

2015-09-08

2020-08-18

Apple Inc.

Zero latency digital assistant

US10331312B2

( en )

2015-09-08

2019-06-25

Apple Inc.

Intelligent automated assistant in a media environment

US10691473B2

( en )

2015-11-06

2020-06-23

Apple Inc.

Intelligent automated assistant in a messaging environment

US10586535B2

( en )

2016-06-10

2020-03-10

Apple Inc.

Intelligent digital assistant in a multi-tasking environment

US12197817B2

( en )

2016-06-11

2025-01-14

Apple Inc.

Intelligent device arbitration and control

DK201670540A1

( en )

2016-06-11

2018-01-08

Apple Inc

Application integration with a digital assistant

US11204787B2

( en )

2017-01-09

2021-12-21

Apple Inc.

Application integration with a digital assistant

DK179496B1

( en )

2017-05-12

2019-01-15

Apple Inc.

USER-SPECIFIC Acoustic Models

DK201770428A1

( en )

2017-05-12

2019-02-18

Apple Inc.

Low-latency intelligent automated assistant

US10303715B2

( en )

2017-05-16

2019-05-28

Apple Inc.

Intelligent automated assistant for media exploration

US10818288B2

( en )

2018-03-26

2020-10-27

Apple Inc.

Natural assistant interaction

DK180639B1

( en )

2018-06-01

2021-11-04

Apple Inc

DISABILITY OF ATTENTION-ATTENTIVE VIRTUAL ASSISTANT

US11462215B2

( en )

2018-09-28

2022-10-04

Apple Inc.

Multi-modal inputs for voice commands

US11348573B2

( en )

2019-03-18

2022-05-31

Apple Inc.

Multimodality in digital assistant systems

US11307752B2

( en )

2019-05-06

2022-04-19

Apple Inc.

User configurable task triggers

DK201970509A1

( en )

2019-05-06

2021-01-15

Apple Inc

Spoken notifications

US11790903B2

( en )

2019-10-29

2023-10-17

Lg Electronics Inc.

Voice recognition method and device

CN113362828B

( en )

2020-03-04

2022-07-05

é¿æ³¢ç½æºè(åäº¬)ç§ææéå¬å¸

Method and apparatus for recognizing speech

KR102870766B1

( en )

2020-03-18

2025-10-14

ë¼ì¸íë¬ì¤ ì£¼ìíì¬

Method and apparatus for controlling avatars based on sound

US12301635B2

( en )

2020-05-11

2025-05-13

Apple Inc.

Digital assistant hardware abstraction

CN111611366B

( en )

2020-05-20

2023-08-11

åäº¬ç¾åº¦ç½è®¯ç§ææéå¬å¸

Intention recognition optimization processing method, device, equipment and storage medium

US11741965B1

( en )

2020-06-26

2023-08-29

Amazon Technologies, Inc.

Configurable natural language output

US11438683B2

( en )

2020-07-21

2022-09-06

Apple Inc.

User identification using headphones

US11984124B2

( en )

2020-11-13

2024-05-14

Apple Inc.

Speculative task flow execution

US11798549B2

( en )

2021-03-19

2023-10-24

Mitel Networks Corporation

Generating action items during a conferencing session

US11669294B2

( en )

2021-05-28

2023-06-06

Microsoft Technology Licensing, Llc

Computing device headset input

US11792364B2

( en )

2021-05-28

2023-10-17

Microsoft Technology Licensing, Llc

Headset virtual presence

CN113571064B

( en )

2021-07-07

2024-01-30

èåºå°é¹æ°è½æºæèµæéå¬å¸

Natural language understanding method and device, vehicle and medium

CN113643696B

( en )

2021-08-10

2024-06-21

é¿æ³¢ç½æºè(åäº¬)ç§ææéå¬å¸

Voice processing method, device, equipment, storage medium and program

JP7705481B2

( en )

2021-08-17

2025-07-09

ã°ã¼ã°ã« ã¨ã«ã¨ã«ã·ã¼

Enabling natural conversations using soft end pointing for automated assistants

KR102840099B1

( en )

2021-08-30

2025-08-01

íêµì ìê¸°ì ì°êµ¬ì

Method and system for automatic back-channel generation in interactive agent system

US12243517B1

( en )

2021-10-13

2025-03-04

Interactions Llc

Utterance endpointing in task-oriented conversational systems

US12567404B1

( en )

2021-10-27

2026-03-03

Meta Platforms, Inc.

Tunable confidence intervals for on-device natural language understanding models

CN116472705A

( en )

2021-11-19

2023-07-21

äº¬ä¸æ¹ç§æéå¢è¡ä»½æéå¬å¸

Method for displaying conference content, conference system and conference equipment

US12260856B2

( en )

2021-12-23

2025-03-25

Y.E. Hub Armenia LLC

Method and system for recognizing a user utterance

US20250182742A1

( en )

2022-02-24

2025-06-05

Nippon Telegraph And Telephone Corporation

Training device, estimation device, training method, estimation method, training program, and estimation program

CN114528822B

( en )

2022-02-25

2024-02-06

å¹³å®ç§æï¼æ·±å³ï¼æéå¬å¸

Conversation flow control method and device of customer service robot, server and medium

WO2023177145A1

( en )

2022-03-16

2023-09-21

ì¼ì±ì ìì£¼ìíì¬

Electronic device and method for controlling electronic device

US12400634B2

( en )

2022-04-21

2025-08-26

Google Llc

Dynamically adapting given assistant output based on a given persona assigned to an automated assistant

CN115134466B

( en )

2022-06-07

2024-08-09

é©¬ä¸æ¶è´¹éèè¡ä»½æéå¬å¸

Intention recognition method and device and electronic equipment

US11646014B1

( en )

2022-07-25

2023-05-09

Intuit Inc.

Ensemble of machine learning models for real-time predictions in expert electronic chats

US12412031B2

( en )

2022-07-29

2025-09-09

Intuit Inc.

Natural language query disambiguation

US11983808B2

( en )

2022-08-24

2024-05-14

Disney Enterprises, Inc.

Conversation-driven character animation

US11822894B1

( en )

2022-12-30

2023-11-21

Fmr Llc

Integrating common and context-specific natural language understanding processing in a virtual assistant application

US20250077341A1

( en )

2023-08-30

2025-03-06

Mark Cummings

Orchestration of or with artificial intelligence

WO2025076397A1

( en )

2023-10-05

2025-04-10

Mia Labs Inc.

Voice interaction with ai models

Citations (52)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20030236664A1

( en )

2002-06-24

2003-12-25

Intel Corporation

Multi-pass recognition of spoken dialogue

US20050261905A1

( en )

2004-05-21

2005-11-24

Samsung Electronics Co., Ltd.

Method and apparatus for generating dialog prosody structure, and speech synthesis method and system employing the same

US20080226058A1

( en )

2005-04-14

2008-09-18

Das Subrata K

System and method for expedited resolution of inquiries for an automated agent

US20090112590A1

( en )

2007-10-30

2009-04-30

At&T Corp.

System and method for improving interaction with a user through a dynamically alterable spoken dialog system

US20090275316A1

( en )

2008-05-05

2009-11-05

Nuance Communications, Inc.

Minimal Distraction Capture of Spoken Contact Information

US8041565B1

( en )

2007-05-04

2011-10-18

Foneweb, Inc.

Precision speech to text conversion

US20120034904A1

( en )

2010-08-06

2012-02-09

Google Inc.

Automatically Monitoring for Voice Input Based on Context

US20130132079A1

( en )

2011-11-17

2013-05-23

Microsoft Corporation

Interactive speech recognition

US20140025383A1

( en )

2012-07-17

2014-01-23

Lenovo (Beijing) Co., Ltd.

Voice Outputting Method, Voice Interaction Method and Electronic Device

US20140058732A1

( en )

2012-08-21

2014-02-27

Nuance Communications, Inc.

Method to provide incremental ui response based on multiple asynchronous evidence about user input

US20150053781A1

( en )

2013-08-21

2015-02-26

Honeywell International Inc.

Devices and methods for interacting with an hvac controller

US20160063118A1

( en )

2014-08-29

2016-03-03

Facebook, Inc.

Priming Search Results on Online Social Networks

US20160104486A1

( en )

2011-04-22

2016-04-14

Angel A. Penilla

Methods and Systems for Communicating Content to Connected Vehicle Users Based Detected Tone/Mood in Voice Input

US9361084B1

( en )

2013-11-14

2016-06-07

Google Inc.

Methods and systems for installing and executing applications

US9437186B1

( en )

2013-06-19

2016-09-06

Amazon Technologies, Inc.

Enhanced endpoint detection for speech recognition

US20160358605A1

( en )

2013-03-12

2016-12-08

Nuance Communications, Inc.

Methods and apparatus for detecting a voice command

US20170075879A1

( en )

2015-09-15

2017-03-16

Kabushiki Kaisha Toshiba

Detection apparatus and method

US20170359707A1

( en )

2016-06-08

2017-12-14

Google Inc.

Providing a personal assistant module with a selectively-traversable state machine

US20170366909A1

( en )

2016-06-15

2017-12-21

Echostar Technologies L.L.C.

Systems and methods for audio calibration using acoustic measurements

US20180032884A1

( en )

2016-07-27

2018-02-01

Wipro Limited

Method and system for dynamically generating adaptive response to user interactions

US20180068660A1

( en )

2016-09-06

2018-03-08

Kyoto University

Voice interaction apparatus and automatic interaction method using voice interaction apparatus

US20180232436A1

( en )

2017-02-16

2018-08-16

Google Llc

Streaming real-time dialog management

US20180357286A1

( en )

2017-06-08

2018-12-13

Microsoft Technology Licensing, Llc

Emotional intelligence for a conversational chatbot

US20190027135A1

( en )

2017-07-18

2019-01-24

Samsung Electronics Co., Ltd.

Method for operating speech recognition service and electronic device and server for supporting the same

US20190042560A1

( en )

2017-08-01

2019-02-07

Intuit Inc.

Extracting domain-specific actions and entities in natural language commands

US10224030B1

( en )

2013-03-14

2019-03-05

Amazon Technologies, Inc.

Dynamic gazetteers for personalized entity recognition

US20190095171A1

( en )

2013-06-09

2019-03-28

Apple Inc.

Device, method, and graphical user interface for enabling conversation persistence across two or more instances of a digital assistant

US20190206397A1

( en )

2017-12-29

2019-07-04

Microsoft Technology Licensing, Llc

Full duplex communication for conversation between chatbot and human

US20190244619A1

( en )

2014-01-21

2019-08-08

Samsung Electronics Co., Ltd.

Electronic device and voice recognition method thereof

US10403272B1

( en )

2013-03-07

2019-09-03

Nuance Communications, Inc.

Facilitating participation in a virtual meeting using an intelligent assistant

US20190290965A1

( en )

2018-03-21

2019-09-26

Peloton Interactive, Inc

Systems and methods for the production, management, syndication and distribution of digital assets through a network in a micro-subscription-based platform for use with an exercise apparatus

US20190318219A1

( en )

2018-04-11

2019-10-17

International Business Machines Corporation

Personalized artificial intelligence interactions and customized responses of a computer system

US10497250B1

( en )

2017-09-27

2019-12-03

State Farm Mutual Automobile Insurance Company

Real property monitoring systems and methods for detecting damage and other conditions

US10499146B2

( en )

2016-02-22

2019-12-03

Sonos, Inc.

Voice control of a media playback system

US20190378493A1

( en )

2017-01-17

2019-12-12

Samsung Electronics Co., Ltd.

Method for sensing end of speech, and electronic apparatus implementing same

US20190394547A1

( en )

2018-06-20

2019-12-26

Amazon Technologies, Inc.

Voice activated device with integrated heatsink and speaker

US20200027455A1

( en )

2017-03-10

2020-01-23

Nippon Telegraph And Telephone Corporation

Dialog system, dialog method, dialog apparatus and program

US20200074993A1

( en )

2016-12-20

2020-03-05

Samsung Electronics Co., Ltd.

Electronic device, method for determining utterance intention of user thereof, and non-transitory computer-readable recording medium

US20200098346A1

( en )

2017-11-01

2020-03-26

Bose Corporation

Adaptive null forming and echo cancellation for selective audio pick-up

US20200135212A1

( en )

2018-10-24

2020-04-30

Samsung Electronics Co., Ltd.

Speech recognition method and apparatus in environment including plurality of apparatuses

US20200168220A1

( en )

2017-05-08

2020-05-28

Signify Holding B.V.

Voice control

US20200273448A1

( en )

2019-02-27

2020-08-27

Comcast Cable Communications, Llc

Voice Command Detection And Prediction

US20200302112A1

( en )

2016-04-22

2020-09-24

Sony Mobile Communications Inc.

Speech to text enhanced media editing

US20200335128A1

( en )

2019-04-19

2020-10-22

Magic Leap, Inc.

Identifying input for speech recognition engine

US20200342858A1

( en )

2019-04-26

2020-10-29

Rovi Guides, Inc.

Systems and methods for enabling topic-based verbal interaction with a virtual assistant

US20200342862A1

( en )

2018-06-27

2020-10-29

Google Llc

Rendering responses to a spoken utterance of a user utilizing a local text-response map

US20210026896A1

( en )

2019-07-22

2021-01-28

Microsoft Technology Licensing, Llc

Asynchronous context communication for speech services

US20210049237A1

( en )

2019-08-15

2021-02-18

International Business Machines Corporation

Multiple parallel delineated topics of a conversation within the same virtual assistant

US11062696B2

( en )

2015-10-19

2021-07-13

Google Llc

Speech endpointing

US11086858B1

( en )

2018-04-20

2021-08-10

Facebook, Inc.

Context-based utterance prediction for assistant systems

US20210280180A1

( en )

2018-12-28

2021-09-09

Google Llc

Supplementing voice inputs to an automated assistant according to selected suggestions

US11211058B1

( en )

2019-09-20

2021-12-28

Amazon Technologies, Inc.

Disambiguation in automatic speech processing

2019

2019-10-04

US16/593,939

patent/US11749265B2/en

active

Active

Patent Citations (53)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20030236664A1

( en )

2002-06-24

2003-12-25

Intel Corporation

Multi-pass recognition of spoken dialogue

US20050261905A1

( en )

2004-05-21

2005-11-24

Samsung Electronics Co., Ltd.

Method and apparatus for generating dialog prosody structure, and speech synthesis method and system employing the same

US20080226058A1

( en )

2005-04-14

2008-09-18

Das Subrata K

System and method for expedited resolution of inquiries for an automated agent

US8041565B1

( en )

2007-05-04

2011-10-18

Foneweb, Inc.

Precision speech to text conversion

US20090112590A1

( en )

2007-10-30

2009-04-30

At&T Corp.

System and method for improving interaction with a user through a dynamically alterable spoken dialog system

US20090275316A1

( en )

2008-05-05

2009-11-05

Nuance Communications, Inc.

Minimal Distraction Capture of Spoken Contact Information

US20120034904A1

( en )

2010-08-06

2012-02-09

Google Inc.

Automatically Monitoring for Voice Input Based on Context

US20160104486A1

( en )

2011-04-22

2016-04-14

Angel A. Penilla

Methods and Systems for Communicating Content to Connected Vehicle Users Based Detected Tone/Mood in Voice Input

Techniques for incremental computer-based natural language understanding — Disney Enterprises, Inc. (US11749265B2)

Related documents