ABSTRACT
Abstract
Techniques described herein relate to facilitating end-to-end multilingual communications with automated assistants. In various implementations, speech recognition output may be generated based on voice input in a first language. A first language intent may be identified based on the speech recognition output and fulfilled in order to generate a first natural language output candidate in the first language. At least part of the speech recognition output may be translated to a second language to generate an at least partial translation, which may then be used to identify a second language intent that is fulfilled to generate a second natural language output candidate in the second language. Scores may be determined for the first and second natural language output candidates, and based on the scores, a natural language output may be selected for presentation.
Description
BACKGROUND
Humans may engage in human-to-computer dialogs with interactive software applications referred to herein as âautomated assistantsâ (also referred to as âchatbots,â âinteractive personal assistants,â âintelligent personal assistants,â âpersonal voice assistants,â âconversational agents,â etc.). For example, humans (which when they interact with automated assistants may be referred to as âusersâ) may provide commands, queries, and/or requests (collectively referred to herein as âqueriesâ) using free form natural language input which may be vocal utterances converted into text and then processed, and/or typed free form natural language input. Automated assistants are typically invoked using predetermined vocal utterances (e.g., âOK Assistantâ) and often perform various types of processing, such as speech-to-text processing and/or natural language processing, only on those vocal utterances that follow an invocation phrase.
Automated assistants are adept at communicating with users in some widely-used languages, such as English, but are less able to communicate in other languages. However, conventional techniques for configuring automated assistants to communicate in new languages are resource-intensive. For each new language, it may be necessary to generate new trigger grammars (i.e., words or phrases intended to invoke a particular action by the automated assistant), recognize query intents in the new language (which require different natural language processing components for different languages), and to generate natural language output that is comprehensible in the new language. Adding to the challenge, each language has myriad idiosyncrasies (e.g., morphologically rich, pro-drop, gender neutrality, etc.) and language experts who can build these systems in particular languages may be hard to find.
SUMMARY
Techniques are described herein for facilitating end-to-end processing of human-to-computer dialogs in various languages by leveraging the processing in a particular language. Put another way, techniques described herein increase language coverage for automated assistants. In various implementations, user queries provided to automated assistants may be processed using multiple pipelines. For example, in some implementations, a first pipeline may be employed to attempt to process the user's query in the original language of the query. A second pipeline may be employed to translate the user's query to a different language that the automated assistant is better-equipped to handle, and to use the translation to identify and fulfill the user's intent. Natural language output returned via the second pipeline may or may not be translated back to the user's original language. In various implementations, the natural language output generated via the multiple pipelines may be ranked, e.g., based on confidence scores, and the highest ranked natural language output may be presented to the user.
It may be desirable to preserve portions of a user's original query in the user's spoken language. For example, in some implementations, slot values may be identified in the user's original query (or more specifically, in speech recognition output generated from the user's original voice input). These slot values may not need to be translated, and in some cases translating them might lead to errors. For example, suppose a user utters the phrase, âWhat is Akash's phone number?â The name âAkashâ may translate to some word in English that has a particular meaning (e.g., sky) that is unrelated to the user's intent. However, if the word âAkashâ can be identified as a slot value and thereby preserved in the user's spoken language, the remainder of the user's query may be properly translated so that an appropriate responsive action may be taken.
Most languages have various colloquialisms. In addition, colloquialisms may be built into the natural language output generation process so that automated assistants can provide output that sounds more âhuman.â For example, suppose a user asks about the weather today in a language that is not yet well supported. The user's query may be translated to a more widely-supported language, such as English, and then the user's intent may be identified and fulfilled. However, for common queries like this it is highly possible that informal (e.g., âhumanlikeâ) natural language output may be available as alternatives to straight-forward natural language output. For example, suppose it is supposed to be hot and sunny today. Rather than simply generating natural language output such as âIt will be 90 degrees and sunny,â the automated assistant may instead select natural language output that is, for instance, generated manually, to sound more human, such as âBetter grab your sunglasses because it's gonna be a scorcher.â Translating this colloquial-sounding natural language output back to the user's native language may yield results that are nonsensical to the user. Accordingly, in some implementations, natural language output that may be difficult to translate, e.g., because it includes manually-generated colloquialisms, slang, etc., may be replaced with more straight-forward natural language output that is more likely to be properly translatable to the user's native language.
In some implementations, a method performed by one or more processors is provided that includes: receiving voice input provided by a user at an input component of a client device in a first language; generating speech recognition output from the voice input, wherein the speech recognition output is in the first language; identifying a first language intent of the user based on the speech recognition output; fulfilling the first language intent to generate first fulfillment information; based on the first fulfillment information, generating a first natural language output candidate in the first language; translating at least a portion of the speech recognition output from the first language to a second language to generate an at least partial translation of the speech recognition output; identifying a second language intent of the user based on the at least partial translation; fulfilling the second language intent to generate second fulfillment information; based on the second fulfillment information, generating a second natural language output candidate in the second language; determining scores for the first and second natural language output candidates; based on the scores, selecting, from the first and second natural language output candidates, a natural language output to be presented to the user; and causing the client device to present the selected natural language output at an output component of the client device.
These and other implementations of technology disclosed herein may optionally include one or more of the following features.
In various implementations, the method may further include generating a third natural language output candidate in the first language that is responsive to the second language intent. In some such implementations, determining the scores may further include determining scores for the first, second, and third content.
In various implementations, the method may further include translating the second natural language output candidate to the first language prior to the scoring. In various implementations, translating the second natural language output candidate may be based on a machine learning model that is trained using one or more logs of natural language outputs provided by one or more automated assistants during human-to-computer dialogs.
In various implementations, the translating to generate the at least partial translation of the speech recognition output may be based on a machine learning model that is trained using one or more logs of user queries submitted to one or more automated assistants during human-to-computer dialogs. In various implementations, the scoring may be based on respective confidence scores assigned to the first and second natural language intents. In various implementations, the scoring may be based on time required to obtain the first and second natural language output candidates.
In various implementations, the method may further include identifying a slot value in the speech recognition output. In some such implementations, the at least partial translation of the speech recognition output may exclude the slot value, and the second language intent may be further identified based on the slot value in the first language.
In another aspect, a method may include the following operations: receiving voice input provided by a user at an input component of a client device in a first language; generating speech recognition output of the voice input in the first language; translating at least a portion of the speech recognition output from the first language to a second language to generate an at least partial translation of the speech recognition output; identifying a second language intent of the user based on the at least partial translation; fulfilling the second language intent to generate fulfillment information; generating natural language output in the second language based on the second language intent; translating the natural language output to the first language to generate translated natural language output; determining whether the translated natural language output satisfies one or more criteria; based on the determining, selecting output that is based on the translated natural language output or alternative natural language output; and causing the client device to present the output at an output component of the client device; and causing the client device to present the selected natural language output at an output component of the client device.
In another aspect, a method for generating training data for training a machine translation model to translate from a first language to a second language may include: applying a textual query in the first language as input across a machine learning model to generate a first embedding of the textual query in a reduced dimensionality space; identifying one or more additional embeddings in the reduced dimensionality space based on one or more respective proximities of the one or more additional embeddings to the first embedding in the reduced dimensionality space, wherein the one or more additional embeddings were generated based on one or more respective textual queries in the second language; and generating and storing at least one training example of the training data using the textual query in the first language and one of the textual queries in the second language that was used to generate a respective one of the additional embeddings.
In various implementations, the method may further include: receiving voice input provided by a user at an input component of a client device in the first language; and performing speech recognition on the voice input to generate the textual query in the first language. In various implementations, the method may further include training the machine translation model using the training data. In various implementations, the machine translation model comprises a neural machine translation model.
In various implementations, the one or more additional embeddings comprise a plurality of embeddings, and the method further includes selecting the one of the textual queries in the second language from the plurality of embeddings. In various implementations, the selecting is based on respective proximities of the plurality of embeddings to the first embedding.
In various implementations, the method may further include: performing natural language understanding processing in the first language to identify a first language intent of the user based on the textual query in the first language; translating the textual query in the first language to generate a translation of the textual query in the second language; performing natural language understanding processing in the second language to identify a second language intent of the user based on the translation of the textual query in the second language; comparing the first and second language intents; and in response to determining, based on the comparing, that the first and second language intents match, generating and storing an additional training example of the training data using the textual query in the first language and the translation of the textual query in the second language. In various implementations, the comparing includes comparing one or more arguments associated with the first language intent with one or more arguments associated with the second language intent. In various implementations, the translating comprises using the one of the textual queries in the second language as the translation of the textual query in the second language. In various implementations, the translating is performed using the machine translation model.
In another aspect, a method for generating training data for training a machine translation model to translate from a first language to a second language may include: performing natural language understanding processing in the first language to identify a first language intent of a user based on a textual query in the first language that is obtained from input provided by the user; translating the textual query in the first language to generate a translation of the textual query in the second language; performing natural language understanding processing in the second language to identify a second language intent of the user based on the translation of the textual query in the second language; comparing the first and second language intents; and in response to determining, based on the comparing, that the first and second language intents match, generating and storing a training example of the training data using the textual query in the first language and the translation of the textual query in the second language.
In various implementations, the comparing includes comparing one or more arguments associated with the first language intent with one or more arguments associated with the second language intent. In various implementations, the translating is performed using the machine translation model.
In various implementations, the translating comprises using a previously-identified textual query in the second language as the translation of the textual query in the second language, wherein the previously-identified textual query in the second language was identified by: applying the textual query in the first language as input across a machine learning model to generate a first embedding of the textual query in a reduced dimensionality space; identifying one or more additional embeddings in the reduced dimensionality space based on one or more respective proximities of the one or more additional embeddings to the first embedding in the reduced dimensionality space, wherein the one or more additional embeddings were generated based on one or more respective textual queries in the second language; and identifying, as the previously-identified query in the second language, one of the textual queries in the second language that was used to generate a respective one of the additional embeddings.
In addition, some implementations include one or more processors of one or more computing devices, where the one or more processors are operable to execute instructions stored in associated memory, and where the instructions are configured to cause performance of any of the aforementioned methods. Some implementations also include one or more non-transitory computer readable storage media storing computer instructions executable by one or more processors to perform any of the aforementioned methods.
It should be appreciated that all combinations of the foregoing concepts and additional concepts described in greater detail herein are contemplated as being part of the subject matter disclosed herein. For example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 is a block diagram of an example environment in which implementations disclosed herein may be implemented.
FIG. 2 depicts an example process flow that demonstrates various aspects of the present disclosure, in accordance with various implementations.
FIG. 3A and FIG. 3B depict example dialogs between a user and an automated assistant, in accordance with various implementations.
<div id="p-0026" num="00
BACKGROUND
Humans may engage in human-to-computer dialogs with interactive software applications referred to herein as âautomated assistantsâ (also referred to as âchatbots,â âinteractive personal assistants,â âintelligent personal assistants,â âpersonal voice assistants,â âconversational agents,â etc.). For example, humans (which when they interact with automated assistants may be referred to as âusersâ) may provide commands, queries, and/or requests (collectively referred to herein as âqueriesâ) using free form natural language input which may be vocal utterances converted into text and then processed, and/or typed free form natural language input. Automated assistants are typically invoked using predetermined vocal utterances (e.g., âOK Assistantâ) and often perform various types of processing, such as speech-to-text processing and/or natural language processing, only on those vocal utterances that follow an invocation phrase.
Automated assistants are adept at communicating with users in some widely-used languages, such as English, but are less able to communicate in other languages. However, conventional techniques for configuring automated assistants to communicate in new languages are resource-intensive. For each new language, it may be necessary to generate new trigger grammars (i.e., words or phrases intended to invoke a particular action by the automated assistant), recognize query intents in the new language (which require different natural language processing components for different languages), and to generate natural language output that is comprehensible in the new language. Adding to the challenge, each language has myriad idiosyncrasies (e.g., morphologically rich, pro-drop, gender neutrality, etc.) and language experts who can build these systems in particular languages may be hard to find.
SUMMARY
Techniques are described herein for facilitating end-to-end processing of human-to-computer dialogs in various languages by leveraging the processing in a particular language. Put another way, techniques described herein increase language coverage for automated assistants. In various implementations, user queries provided to automated assistants may be processed using multiple pipelines. For example, in some implementations, a first pipeline may be employed to attempt to process the user's query in the original language of the query. A second pipeline may be employed to translate the user's query to a different language that the automated assistant is better-equipped to handle, and to use the translation to identify and fulfill the user's intent. Natural language output returned via the second pipeline may or may not be translated back to the user's original language. In various implementations, the natural language output generated via the multiple pipelines may be ranked, e.g., based on confidence scores, and the highest ranked natural language output may be presented to the user.
It may be desirable to preserve portions of a user's original query in the user's spoken language. For example, in some implementations, slot values may be identified in the user's original query (or more specifically, in speech recognition output generated from the user's original voice input). These slot values may not need to be translated, and in some cases translating them might lead to errors. For example, suppose a user utters the phrase, âWhat is Akash's phone number?â The name âAkashâ may translate to some word in English that has a particular meaning (e.g., sky) that is unrelated to the user's intent. However, if the word âAkashâ can be identified as a slot value and thereby preserved in the user's spoken language, the remainder of the user's query may be properly translated so that an appropriate responsive action may be taken.
Most languages have various colloquialisms. In addition, colloquialisms may be built into the natural language output generation process so that automated assistants can provide output that sounds more âhuman.â For example, suppose a user asks about the weather today in a language that is not yet well supported. The user's query may be translated to a more widely-supported language, such as English, and then the user's intent may be identified and fulfilled. However, for common queries like this it is highly possible that informal (e.g., âhumanlikeâ) natural language output may be available as alternatives to straight-forward natural language output. For example, suppose it is supposed to be hot and sunny today. Rather than simply generating natural language output such as âIt will be 90 degrees and sunny,â the automated assistant may instead select natural language output that is, for instance, generated manually, to sound more human, such as âBetter grab your sunglasses because it's gonna be a scorcher.â Translating this colloquial-sounding natural language output back to the user's native language may yield results that are nonsensical to the user. Accordingly, in some implementations, natural language output that may be difficult to translate, e.g., because it includes manually-generated colloquialisms, slang, etc., may be replaced with more straight-forward natural language output that is more likely to be properly translatable to the user's native language.
In some implementations, a method performed by one or more processors is provided that includes: receiving voice input provided by a user at an input component of a client device in a first language; generating speech recognition output from the voice input, wherein the speech recognition output is in the first language; identifying a first language intent of the user based on the speech recognition output; fulfilling the first language intent to generate first fulfillment information; based on the first fulfillment information, generating a first natural language output candidate in the first language; translating at least a portion of the speech recognition output from the first language to a second language to generate an at least partial translation of the speech recognition output; identifying a second language intent of the user based on the at least partial translation; fulfilling the second language intent to generate second fulfillment information; based on the second fulfillment information, generating a second natural language output candidate in the second language; determining scores for the first and second natural language output candidates; based on the scores, selecting, from the first and second natural language output candidates, a natural language output to be presented to the user; and causing the client device to present the selected natural language output at an output component of the client device.
These and other implementations of technology disclosed herein may optionally include one or more of the following features.
In various implementations, the method may further include generating a third natural language output candidate in the first language that is responsive to the second language intent. In some such implementations, determining the scores may further include determining scores for the first, second, and third content.
In various implementations, the method may further include translating the second natural language output candidate to the first language prior to the scoring. In various implementations, translating the second natural language output candidate may be based on a machine learning model that is trained using one or more logs of natural language outputs provided by one or more automated assistants during human-to-computer dialogs.
In various implementations, the translating to generate the at least partial translation of the speech recognition output may be based on a machine learning model that is trained using one or more logs of user queries submitted to one or more automated assistants during human-to-computer dialogs. In various implementations, the scoring may be based on respective confidence scores assigned to the first and second natural language intents. In various implementations, the scoring may be based on time required to obtain the first and second natural language output candidates.
In various implementations, the method may further include identifying a slot value in the speech recognition output. In some such implementations, the at least partial translation of the speech recognition output may exclude the slot value, and the second language intent may be further identified based on the slot value in the first language.
In another aspect, a method may include the following operations: receiving voice input provided by a user at an input component of a client device in a first language; generating speech recognition output of the voice input in the first language; translating at least a portion of the speech recognition output from the first language to a second language to generate an at least partial translation of the speech recognition output; identifying a second language intent of the user based on the at least partial translation; fulfilling the second language intent to generate fulfillment information; generating natural language output in the second language based on the second language intent; translating the natural language output to the first language to generate translated natural language output; determining whether the translated natural language output satisfies one or more criteria; based on the determining, selecting output that is based on the translated natural language output or alternative natural language output; and causing the client device to present the output at an output component of the client device; and causing the client device to present the selected natural language output at an output component of the client device.
In another aspect, a method for generating training data for training a machine translation model to translate from a first language to a second language may include: applying a textual query in the first language as input across a machine learning model to generate a first embedding of the textual query in a reduced dimensionality space; identifying one or more additional embeddings in the reduced dimensionality space based on one or more respective proximities of the one or more additional embeddings to the first embedding in the reduced dimensionality space, wherein the one or more additional embeddings were generated based on one or more respective textual queries in the second language; and generating and storing at least one training example of the training data using the textual query in the first language and one of the textual queries in the second language that was used to generate a respective one of the additional embeddings.
In various implementations, the method may further include: receiving voice input provided by a user at an input component of a client device in the first language; and performing speech recognition on the voice input to generate the textual query in the first language. In various implementations, the method may further include training the machine translation model using the training data. In various implementations, the machine translation model comprises a neural machine translation model.
In various implementations, the one or more additional embeddings comprise a plurality of embeddings, and the method further includes selecting the one of the textual queries in the second language from the plurality of embeddings. In various implementations, the selecting is based on respective proximities of the plurality of embeddings to the first embedding.
In various implementations, the method may further include: performing natural language understanding processing in the first language to identify a first language intent of the user based on the textual query in the first language; translating the textual query in the first language to generate a translation of the textual query in the second language; performing natural language understanding processing in the second language to identify a second language intent of the user based on the translation of the textual query in the second language; comparing the first and second language intents; and in response to determining, based on the comparing, that the first and second language intents match, generating and storing an additional training example of the training data using the textual query in the first language and the translation of the textual query in the second language. In various implementations, the comparing includes comparing one or more arguments associated with the first language intent with one or more arguments associated with the second language intent. In various implementations, the translating comprises using the one of the textual queries in the second language as the translation of the textual query in the second language. In various implementations, the translating is performed using the machine translation model.
In another aspect, a method for generating training data for training a machine translation model to translate from a first language to a second language may include: performing natural language understanding processing in the first language to identify a first language intent of a user based on a textual query in the first language that is obtained from input provided by the user; translating the textual query in the first language to generate a translation of the textual query in the second language; performing natural language understanding processing in the second language to identify a second language intent of the user based on the translation of the textual query in the second language; comparing the first and second language intents; and in response to determining, based on the comparing, that the first and second language intents match, generating and storing a training example of the training data using the textual query in the first language and the translation of the textual query in the second language.
In various implementations, the comparing includes comparing one or more arguments associated with the first language intent with one or more arguments associated with the second language intent. In various implementations, the translating is performed using the machine translation model.
In various implementations, the translating comprises using a previously-identified textual query in the second language as the translation of the textual query in the second language, wherein the previously-identified textual query in the second language was identified by: applying the textual query in the first language as input across a machine learning model to generate a first embedding of the textual query in a reduced dimensionality space; identifying one or more additional embeddings in the reduced dimensionality space based on one or more respective proximities of the one or more additional embeddings to the first embedding in the reduced dimensionality space, wherein the one or more additional embeddings were generated based on one or more respective textual queries in the second language; and identifying, as the previously-identified query in the second language, one of the textual queries in the second language that was used to generate a respective one of the additional embeddings.
In addition, some implementations include one or more processors of one or more computing devices, where the one or more processors are operable to execute instructions stored in associated memory, and where the instructions are configured to cause performance of any of the aforementioned methods. Some implementations also include one or more non-transitory computer readable storage media storing computer instructions executable by one or more processors to perform any of the aforementioned methods.
It should be appreciated that all combinations of the foregoing concepts and additional concepts described in greater detail herein are contemplated as being part of the subject matter disclosed herein. For example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 is a block diagram of an example environment in which implementations disclosed herein may be implemented.
FIG. 2 depicts an example process flow that demonstrates various aspects of the present disclosure, in accordance with various implementations.
FIG. 3A and FIG. 3B depict example dialogs between a user and an automated assistant, in accordance with various implementations.
FIG. 4 depicts a flowchart illustrating an example method according to implementations disclosed herein.
FIG. 5 depicts a flowchart illustrating an example method according to implementations disclosed herein.
FIG. 6 illustrates an example architecture of a computing device.
FIG. 7 , FIG. 8 , and FIG. 9 demonstrate another aspect in which training data is automatically generated for training a machine translation model to translate text from a first language to a second language, with little or no human intervention, in accordance with various implementations.
DETAILED DESCRIPTION
Now turning to FIG. 1 , an example environment in which techniques disclosed herein may be implemented is illustrated. The example environment includes a plurality of client computing devices 106 1-N . Each client device 106 may execute a respective instance of an automated assistant client 118 . One or more cloud-based automated assistant components 119 , such as a natural language understanding engine 135 , may be implemented on one or more computing systems (collectively referred to as a âcloudâ computing system) that are communicatively coupled to client devices 106 1-N via one or more local and/or wide area networks (e.g., the Internet) indicated generally at 110 .
In some implementations, plurality of client computing devices 106 1-N (also referred to herein simply as âclient devicesâ) may be associated with each other in various ways in order to facilitate performance of techniques described herein. For example, in some implementations, plurality of client computing devices 106 1-N may be associated with each other by virtue of them being members of a coordinated ecosystem of client devices 106 that are operated by one or more users (e.g., an individual, a family, employees of an organization, other predefined groups, etc.).
As noted in the background, an instance of an automated assistant client 118 , by way of its interactions with one or more cloud-based automated assistant components 119 , may form what appears to be, from the user's perspective, a logical instance of an automated assistant 120 with which the user may engage in a human-to-computer dialog. Two instances of such an automated assistant 120 are depicted in FIG. 1 . A first automated assistant 120 A encompassed by a dashed line serves a first user (not depicted) operating first client device 106 1 and includes automated assistant client 118 1 and one or more cloud-based automated assistant components 119 . A second automated assistant 120 B encompassed by a dash-dash-dot line serves a second user (not depicted) operating another client device 106 N and includes automated assistant client 118 N and one or more cloud-based automated assistant components 119 . It thus should be understood that each user that engages with an automated assistant client 118 executing on a client device 106 may, in effect, engage with his or her own logical instance of an automated assistant 120 . For the sakes of brevity and simplicity, the term âautomated assistantâ as used herein as âservingâ a particular user will refer to the combination of an automated assistant client 118 executing on a client device 106 operated by the user and one or more cloud-based automated assistant components 119 (which may be shared amongst multiple automated assistant clients 118 ). It should also be understood that in some implementations, automated assistant 120 may respond to a request from any user regardless of whether the user is actually âservedâ by that particular instance of automated assistant 120 .
The client devices 106 1-N may include, for example, one or more of: a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device of a vehicle of the user (e.g., an in-vehicle communications system, an in-vehicle entertainment system, an in-vehicle navigation system), a standalone interactive speaker, a smart appliance such as a smart television, and/or a wearable apparatus of the user that includes a computing device (e.g., a watch of the user having a computing device, glasses of the user having a computing device, a virtual or augmented reality computing device). Additional and/or alternative client computing devices may be provided.
Each of the client computing devices 106 1-N may operate a variety of different applications, such as a corresponding one of a plurality of message exchange clients 107 1-N . Message exchange clients 107 1-N may come in various forms and the forms may vary across the client computing devices 106 1-N and/or multiple forms may be operated on a single one of the client computing devices 106 1-N . In some implementations, one or more of the message exchange clients 107 1-N may come in the form of a short messaging service (âSMSâ) and/or multimedia messaging service (âMMSâ) client, an online chat client (e.g., instant messenger, Internet relay chat, or âIRC,â etc.), a messaging application associated with a social network, a personal assistant messaging service dedicated to conversations with automated assistant 120 , and so forth. In some implementations, one or more of the message exchange clients 107 1-N may be implemented via a webpage or other resources rendered by a web browser (not depicted) or other application of client computing device 106 .
As described in more detail herein, automated assistant 120 engages in human-to-computer dialog sessions with one or more users via user interface input and output devices of one or more client devices 106 1-N . In some implementations, automated assistant 120 may engage in a human-to-computer dialog session with a user in response to user interface input provided by the user via one or more user interface input devices of one of the client devices 106 1-N . In some of those implementations, the user interface input is explicitly directed to automated assistant 120 . For example, one of the message exchange clients 107 1-N may be a personal assistant messaging service dedicated to conversations with automated assistant 120 and user interface input provided via that personal assistant messaging service may be automatically provided to automated assistant 120 . Also, for example, the user interface input may be explicitly directed to automated assistant 120 in one or more of the message exchange clients 107 1-N based on particular user interface input that indicates automated assistant 120 is to be invoked. For instance, the particular user interface input may be one or more typed characters (e.g., @AutomatedAssistant), user interaction with a hardware button and/or virtual button (e.g., a tap, a long tap), an oral command (e.g., âHey Automated Assistantâ), and/or other particular user interface input.
In some implementations, automated assistant 120 may engage in a human-to-computer dialog session in response to user interface input, even when that user interface input is not explicitly directed to automated assistant 120 . For example, automated assistant 120 may examine the contents of user interface input and engage in a dialog session in response to certain terms being present in the user interface input and/or based on other cues. In many implementations, automated assistant 120 may engage interactive voice response (âIVRâ), such that the user can utter commands, searches, etc., and the automated assistant may utilize speech recognition to convert the utterances into text, and respond to the text accordingly, e.g., by providing search results, general information, and/or taking one or more responsive actions (e.g., playing media, launching a game, ordering food, etc.). In some implementations, the automated assistant 120 can additionally or alternatively respond to utterances without converting the utterances into text. For example, the automated assistant 120 can convert voice input into an embedding, into entity representation(s) (that indicate entity/entities present in the voice input), and/or other ânon-textualâ representation and operate on such non-textual representation. Accordingly, implementations described herein as operating based on text converted from voice input may additionally and/or alternatively operate on the voice input directly and/or other non-textual representations of the voice input.
Each of the client computing devices 106 1-N and computing device(s) operating cloud-based automated assistant components 119 may include one or more memories for storage of data and software applications, one or more processors for accessing data and executing applications, and other components that facilitate communication over a network. The operations performed by one or more of the client computing devices 106 1-N and/or by automated assistant 120 may be distributed across multiple computer systems. Automated assistant 120 may be implemented as, for example, computer programs running on one or more computers in one or more locations that are coupled to each other through a network.
As noted above, in various implementations, each of the client computing devices 106 1-N may operate an automated assistant client 118 . In various implementations, each automated assistant client 118 may include a corresponding speech capture/text-to-speech (âTTSâ)/speech-to-text (âSTTâ) module 114 . In other implementations, one or more aspects of speech capture/TTS/STT module 114 may be implemented separately from automated assistant client 118 . In various implementations, speech capture/TTS/STT module 114 may generate speech recognition output based on a vocal query. This speech recognition output may be analyzed pursuant to techniques described herein.
Each speech capture/TTS/STT module 114 may be configured to perform one or more functions: capture a user's speech, e.g., via a microphone; convert that captured audio to text (and/or to other representations or embeddings); and/or convert text to speech. For example, in some implementations, because a client device 106 may be relatively constrained in terms of computing resources (e.g., processor cycles, memory, battery, etc.), the speech capture/TTS/STT module 114 that is local to each client device 106 may be configured to convert a finite number of different spoken phrasesâparticularly phrases that invoke automated assistant 120 âto text (or to other forms, such as lower dimensionality embeddings). Other speech input may be sent to cloud-based automated assistant components 119 , which may include a cloud-based TTS module 116 and/or a cloud-based STT module 117 .
Cloud-based STT module 117 may be configured to leverage the virtually limitless resources of the cloud to convert audio data captured by speech capture/TTS/STT module 114 into text (which may then be provided to natural language processor 122 ). Cloud-based TTS module 116 may be configured to leverage the virtually limitless resources of the cloud to convert textual data (e.g., natural language responses formulated by automated assistant 120 ) into computer-generated speech output. In some implementations, TTS module 116 may provide the computer-generated speech output to client device 106 to be output directly, e.g., using one or more speakers. In other implementations, textual data (e.g., natural language responses) generated by automated assistant 120 may be provided to speech capture/TTS/STT module 114 , which may then convert the textual data into computer-generated speech that is output locally.
Automated assistant 120 (and in particular, cloud-based automated assistant components 119 ) may include a natural language understanding engine 135 , the aforementioned TTS module 116 , the aforementioned STT module 117 , and other components that are described in more detail below. In some implementations, one or more of the engines and/or modules of automated assistant 120 may be omitted, combined, and/or implemented in a component that is separate from automated assistant 120 . In some implementations, to protect privacy, one or more of the components of automated assistant 120 , such as natural language processor 122 , speech capture/TTS/STT module 114 , etc., may be implemented at least on part on client devices 106 (e.g., to the exclusion of the cloud).
In some implementations, automated assistant 120 generates responsive content in response to various inputs generated by a user of one of the client devices 106 1-N during a human-to-computer dialog session with automated assistant 120 . Automated assistant 120 may provide the responsive content (e.g., over one or more networks when separate from a client device of a user) for presentation to the user as part of the dialog session. For example, automated assistant 120 may generate responsive content in response to free-form natural language input provided via one of the client devices 106 1-N . As used herein, free-form input is input that is formulated by a user and that is not constrained to a group of options presented for selection by the user.
As used herein, a âdialog sessionâ may include a logically-self-contained exchange of one or more messages between a user and automated assistant 120 (and in some cases, other human participants). Automated assistant 120 may differentiate between multiple dialog sessions with a user based on various signals, such as passage of time between sessions, change of user context (e.g., location, before/during/after a scheduled meeting, etc.) between sessions, detection of one or more intervening interactions between the user and a client device other than dialog between the user and the automated assistant (e.g., the user switches applications for a while, the user walks away from then later returns to a standalone voice-activated product), locking/sleeping of the client device between sessions, change of client devices used to interface with one or more instances of automated assistant 120 , and so forth.
Natural language processor 122 of natural language understanding engine 135 processes natural language input generated by users via client devices 106 1-N and may generate annotated output (e.g., in textual form) for use by one or more other components of automated assistant 120 . For example, the natural language processor 122 may process natural language free-form input that is generated by a user via one or more user interface input devices of client device 106 1 . The generated annotated output includes one or more annotations of the natural language input and one or more (e.g., all) of the terms of the natural language input.
In some implementations, the natural language processor 122 is configured to identify and annotate various types of grammatical information in natural language input. For example, the natural language processor 122 may include a morphological engine that may separate individual words into morphemes and/or annotate the morphemes, e.g., with their classes. Natural language processor 122 may also include a part of speech tagger configured to annotate terms with their grammatical roles. For example, the part of speech tagger may tag each term with its part of speech such as ânoun,â âverb,â âadjective,â âpronoun,â etc. Also, for example, in some implementations the natural language processor 122 may additionally and/or alternatively include a dependency parser (not depicted) configured to determine syntactic relationships between terms in natural language input. For example, the dependency parser may determine which terms modify other terms, subjects and verbs of sentences, and so forth (e.g., a parse tree)âand may make annotations of such dependencies.
In some implementations, the natural language processor 122 may additionally and/or alternatively include an entity tagger (not depicted) configured to annotate entity references in one or more segments such as references to people (including, for instance, literary characters, celebrities, public figures, etc.), organizations, locations (real and imaginary), and so forth. In some implementations, data about entities may be stored in one or more databases, such as in a knowledge graph (not depicted). In some implementations, the knowledge graph may include nodes that represent known entities (and in some cases, entity attributes), as well as edges that connect the nodes and represent relationships between the entities. For example, a âbananaâ node may be connected (e.g., as a child) to a âfruitâ node,â which in turn may be connected (e.g., as a child) to âproduceâ and/or âfoodâ nodes. As another example, a restaurant called âHypothetical Caféâ may be represented by a node that also includes attributes such as its address, type of food served, hours, contact information, etc. The âHypothetical Caféâ node may in some implementations be connected by an edge (e.g., representing a child-to-parent relationship) to one or more other nodes, such as a ârestaurantâ node, a âbusinessâ node, a node representing a city and/or state in which the restaurant is located, and so forth.
The entity tagger of the natural language processor 122 may annotate references to an entity at a high level of granularity (e.g., to enable identification of all references to an entity class such as people) and/or a lower level of granularity (e.g., to enable identification of all references to a particular entity such as a particular person). The entity tagger may rely on content of the natural language input to resolve a particular entity and/or may optionally communicate with a knowledge graph or other entity database to resolve a particular entity.
In some implementations, the natural language processor 122 may additionally and/or alternatively include a coreference resolver (not depicted) configured to group, or âcluster,â references to the same entity based on one or more contextual cues. For example, the coreference resolver may be utilized to resolve the term âthereâ to âHypothetical Caféâ in the natural language input âI liked Hypothetical Café last time we ate there.â
In some implementations, one or more components of the natural language processor 122 may rely on annotations from one or more other components of the natural language processor 122 . For example, in some implementations the named entity tagger may rely on annotations from the coreference resolver and/or dependency parser in annotating all mentions to a particular entity. Also, for example, in some implementations the coreference resolver may rely on annotations from the dependency parser in clustering references to the same entity. In some implementations, in processing a particular natural language input, one or more components of the natural language processor 122 may use related prior input and/or other related data outside of the particular natural language input to determine one or more annotations.
Natural language understanding engine 135 may also include an intent matcher 136 that is configured to determine an intent of a user engaged in a human-to-computer dialog session with automated assistant 120 . While depicted separately from natural language processor 122 in FIG. 1 , in other implementations, intent matcher 136 may be an integral part of natural language processor 122 (or more generally, of a pipeline that includes natural language processor 122 ). In some implementations, natural language processor 122 and intent matcher 136 may collectively form the aforementioned ânatural language understandingâ engine 135 .
Intent matcher 136 may use various techniques to determine an intent of the user, e.g., based on output from natural language processor 122 (which may include annotations and terms of the natural language input). In some implementations, intent matcher 136 may have access to one or more databases 137 that include, for instance, a plurality of mappings between grammars and responsive actions (or more generally, intents). In many cases, these grammars may be selected and/or learned over time, and may represent the most common intents of users. For example, one grammar, âplay <artist>â, may be mapped to an intent that invokes a responsive action that causes music by the <artist> to be played on the client device 106 operated by the user. Another grammar, â[weather|forecast] today,â may be match-able to user queries such as âwhat's the weather todayâ and âwhat's the forecast for today?â As will be described in more detail below, in addition to or instead of grammars, in some implementations, intent matcher 136 may employ one or more trained machine learning models, alone or in combination with one or more grammars. These trained machine learning models may also be stored in one or more databases 137 .
As seen in the âplay <artist>â example grammar, some grammars have slots (e.g., <artist>) that can be filled with slot values (or âparametersâ). Slot values may be determined in various ways. Often users will provide the slot values proactively. For example, for a grammar âOrder me a <topping> pizza,â a user may likely speak the phrase âorder me a sausage pizza,â in which case the slot <topping> is filled automatically. Additionally or alternatively, if a user invokes a grammar that includes slots to be filled with slot values, without the user proactively providing the slot values, automated assistant 120 may solicit those slot values from the user (e.g., âwhat type of crust do you want on your pizza?â).
In some implementations, automated assistant 120 may facilitate (or âbrokerâ) transactions between users and third party applications. These third party applications may or may not operate on computing systems that are separate from those that operate, for instance, cloud-based automated assistant components 119 . Accordingly, one kind of user intent that may be identified by intent matcher 136 is to engage a third party application. For example, automated assistant 120 may provide access to an application programming interface (âAPIâ) to a pizza delivery service. A user may invoke automated assistant 120 and provide a command such as âI'd like to order a pizza.â Intent matcher 136 may map this command to a grammar (which may be added to database 137 in some cases by the third party) that triggers automated assistant 120 to engage with the third party pizza delivery service. The third party pizza delivery service may provide automated assistant 120 with a minimum list of slots that need to be filled in order to fulfill a pizza delivery order. Automated assistant 120 may generate and provide to the user (via client device 106 ) natural language output that solicits parameters for the slots.
In the human-to-computer dialog domain, the vocabulary employed by users and by automated assistant 120 may be somewhat less comprehensive than what is employed in everyday conversation, literature, writing, etc. In other words, a limited language space employed during human-to-computer dialogs is typically going to be a subset of a more comprehensive language space used for more general purpose human-to-human communication (e.g., talking, writing, etc.). For example, users tend to engage with automated assistants 120 to cause one or more tasks to be performed. Even in the context of general search queries, users tend to engage with automated assistants to perform particular types of queries, such as asking about the weather, traffic, transit times, restaurant hours, etc.
Accordingly, in some implementations, natural language understanding engine 135 , e.g., by way of natural language processor 122 or intent matcher 136 , may have access to one or more models (e.g., in database 137 ) that are trained in a particular language to understand the specific and limited vocabulary typically employed in human-to-computer dialogs with automated assistant 129 . These models may be trained in a particular language, e.g., using logs of natural language inputs translated to the particular language, to determine users' intents in the particular language. This is especially helpful for languages that are not yet fully supported (i.e., for which comprehensive translation is not yet available). For example, common queries (e.g., order food, check weather, set timer, etc.) in a widely supported language, such as English, may be obtained from log(s) and translated (e.g., automatically by translator 138 , manually by linguists/translators, etc.) into a less-widely-supported second language. These translated common queries may be used in conjunction with their matching intents as training examples to train a model for the less-widely supported second language.
For example, the English queries may be translated, e.g., using translator 138 and/or human linguists/translators, into the less-widely supported language. The translations may be used as training examples, labelled with corresponding intents (which also may be translated into the less-widely supported language), to train the model. If the model is some flavour of a neural network, the labelled training examples may be applied as input across the neural network to generate output. The output may be compared to the corresponding labels to generate an error. Based on the error, various training techniques, such as gradient descent (stochastic or batch) and back propagation may be employed to adjust weights of one or more hidden layers of the neural network.
In some implementations, previously-unseen queries in the less-widely supported language may be applied as input across the model to generate output. In some such implementations, the output may take the form of embeddings into a reduced-dimensionality space associated with the language. Intents may be matched to these previously-unseen queries based on their proximity (e.g., Euclidian distance, cosine similarity, etc.) with other embeddings in the reduced-dimensionality space. If a user who provides a previously-unseen query rejects the ultimate outcome, then the user's query may be used as a negative training example to further train the model. If a user who provides a new query provides positive feedback (or no feedback, which may indicate the user is satisfied), then the user's query may be used as a positive training example to further train the model. In this manner it is possible for automated assistant 120 to âlearnâ how to predict intents in a new language.
Fulfillment engine 124 may be configured to receive the intent output by intent matcher 136 , as well as an associated slot values (whether provided by the user proactively or solicited from the user) and fulfill the intent. In various implementations, fulfillment of the user's intent may cause various fulfillment information (which may be language-agnostic in some instances) to be generated/obtained, e.g., by fulfillment engine 124 . As will be described below, the fulfillment information may in some implementations be provided to a natural language generator (âNLGâ in some FIG. 126 , which may generate natural language output based on the fulfillment information.
Fulfillment information may take various forms because an intent can be fulfilled in a variety of ways. Suppose a user requests pure information, such as âWhere were the outdoor shots of âThe Shiningâ filmed?â The intent of the user may be determined, e.g., by intent matcher 136 , as being a search query. The intent and content of the search query may be provided to fulfillment engine 124 , which as depicted in FIG. 1 may be in communication with one or more search engines 150 configured to search corpuses of documents and/or other data sources (e.g., knowledge graphs, etc.) for responsive information. Fulfillment engine 124 may provide data indicative of the search query (e.g., the text of the query, a reduced dimensionality embedding, etc.) to search engine 150 . Search engine 150 may provide responsive information, such as GPS coordinates, or other more explicit information, such as âTimberline Lodge, Mt. Hood, Oreg.â This responsive information may form part of the fulfillment information generated by fulfillment engine 124 .
Additionally or alternatively, fulfillment engine 124 may be configured to receive, e.g., from natural language understanding engine 135 , a user's intent and any slot values provided by th
CLAIMS
Claims ( 15 )
What is claimed is:
1. A method for generating training data for training a machine translation model to translate from a first language to a second language, the method implemented by one or more processors and comprising:
performing natural language understanding processing in the first language to identify a first language intent of a user based on a textual query in the first language that is obtained from input provided by the user;
using the machine translation model, translating the textual query in the first language to generate a translation of the textual query in the second language;
performing natural language understanding processing in the second language to identify a second language intent of the user based on the translation of the textual query in the second language;
comparing the first and second language intents;
in response to determining, based on the comparing, that the first and second language intents match, generating and storing a training example of the training data using the textual query in the first language and the translation of the textual query in the second language; and
updating the machine translation model based on the training data.
2. The method of claim 1 , further comprising:
receiving voice input provided by a user at an input component of a client device in the first language; and
performing speech recognition on the voice input to generate the textual query in the first language.
3. The method of claim 1 , wherein the updating comprises training the machine translation model using the training data.
4. The method of claim 1 , wherein the machine translation model comprises a neural machine translation model.
5. The method of claim 1 , wherein the comparing includes comparing one or more arguments associated with the first language intent with one or more arguments associated with the second language intent.
6. At least one non-transitory computer-readable medium comprising instructions that, in response to execution of the instructions by one or more processors, cause the one or more processors to perform the following operations to generate training data for training a machine translation model to translate from a first language to a second language:
perform natural language understanding processing in the first language to identify a first language intent of a user based on a textual query in the first language that is obtained from input provided by the user;
using the machine translation model, translate the textual query in the first language to generate a translation of the textual query in the second language;
perform natural language understanding processing in the second language to identify a second language intent of the user based on the translation of the textual query in the second language;
compare the first and second language intents;
in response to a determination that the first and second language intents match, generate and store a training example of the training data using the textual query in the first language and the translation of the textual query in the second language; and
update the machine translation model based on the training data.
7. The at least one non-transitory computer-readable medium of claim 6 , further comprising instructions to:
receive voice input provided by a user at an input component of a client device in the first language; and
perform speech recognition on the voice input to generate the textual query in the first language.
8. The at least one non-transitory computer-readable medium of claim 6 , wherein the instructions to update include instructions to train the machine translation model using the training data.
9. The at least one non-transitory computer-readable medium of claim 6 , wherein the machine translation model comprises a neural machine translation model.
10. The at least one non-transitory computer-readable medium of claim 6 , wherein the comparison includes a comparison of one or more arguments associated with the first language intent to one or more arguments associated with the second language intent.
11. A system comprising one or more processors and memory storing instructions that, in response to execution of the instructions by one or more of the processors, cause the one or more processors to perform the following operations to generate training data for training a machine translation model to translate from a first language to a second language:
perform natural language understanding processing in the first language to identify a first language intent of a user based on a textual query in the first language that is obtained from input provided by the user;
using the machine translation model, translate the textual query in the first language to generate a translation of the textual query in the second language;
perform natural language understanding processing in the second language to identify a second language intent of the user based on the translation of the textual query in the second language;
compare the first and second language intents;
in response to a determination that the first and second language intents match, generate and store a training example of the training data using the textual query in the first language and the translation of the textual query in the second language; and
update the machine translation model based on the training data.
12. The system of claim 11 , further comprising instructions to:
receive voice input provided by a user at an input component of a client device in the first language; and
perform speech recognition on the voice input to generate the textual query in the first language.
13. The system of claim 11 , wherein the instructions to compare comprise instructions to train the machine translation model using the training data.
14. The system of claim 11 , wherein the machine translation model comprises a neural machine translation model.
15. The system of claim 11 , wherein the comparison includes a comparison of one or more arguments associated with the first language intent to one or more arguments associated with the second language intent.
US16/792,572
2018-03-07
2020-02-17
Facilitating communications with automated assistants in multiple languages
Active
2038-08-27
US11354521B2
( en )
Priority Applications (2)
Application Number
Priority Date
Filing Date
Title
US16/792,572
US11354521B2
( en )
2018-03-07
2020-02-17
Facilitating communications with automated assistants in multiple languages
US17/825,778
US11942082B2
( en )
2018-03-07
2022-05-26
Facilitating communications with automated assistants in multiple languages
Applications Claiming Priority (4)
Application Number
Priority Date
Filing Date
Title
US201862639740P
2018-03-07
2018-03-07
PCT/US2018/027774
WO2019172946A1
( en )
2018-03-07
2018-04-16
Facilitating end-to-end communications with automated assistants in multiple languages
US16/082,175
US10984784B2
( en )
2018-03-07
2018-04-16
Facilitating end-to-end communications with automated assistants in multiple languages
US16/792,572
US11354521B2
( en )
2018-03-07
2020-02-17
Facilitating communications with automated assistants in multiple languages
Related Parent Applications (2)
Application Number
Title
Priority Date
Filing Date
PCT/US2018/027774
Continuation-In-Part
WO2019172946A1
( en )
2018-03-07
2018-04-16
Facilitating end-to-end communications with automated assistants in multiple languages
US16/082,175
Continuation-In-Part
US10984784B2
( en )
2018-03-07
2018-04-16
Facilitating end-to-end communications with automated assistants in multiple languages
Related Child Applications (1)
Application Number
Title
Priority Date
Filing Date
US17/825,778
Continuation
US11942082B2
( en )
2018-03-07
2022-05-26
Facilitating communications with automated assistants in multiple languages
Publications (2)
Publication Number
Publication Date
US20200184158A1
US20200184158A1 ( en )
2020-06-11
US11354521B2
true
US11354521B2 ( en )
2022-06-07
Family
ID=70972775
Family Applications (1)
Application Number
Title
Priority Date
Filing Date
US16/792,572
Active
2038-08-27
US11354521B2
( en )
2018-03-07
2020-02-17
Facilitating communications with automated assistants in multiple languages
Country Status (1)
Country
Link
US
( 1 )
US11354521B2
( en )
Cited By (2)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US11915692B2
( en )
2018-03-07
2024-02-27
Google Llc
Facilitating end-to-end communications with automated assistants in multiple languages
US20240202468A1
( en )
*
2022-12-15
2024-06-20
Adp, Inc.
Automated translations for autonomous chat agents
Families Citing this family (28)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
GB2576161B
( en )
*
2018-08-06
2022-11-09
Brown Patrick
Wearable strap
US10902942B1
( en )
*
2018-08-29
2021-01-26
Big Health Inc.
Generation and delivery of customized content programs
CN111368565B
( en )
*
2018-09-05
2022-03-18
è ¾è®¯ç§æï¼æ·±å³ï¼æéå ¬å¸
Text translation method, apparatus, storage medium and computer equipment
US11043214B1
( en )
*
2018-11-29
2021-06-22
Amazon Technologies, Inc.
Speech recognition using dialog history
US11222176B2
( en )
*
2019-05-24
2022-01-11
International Business Machines Corporation
Method and system for language and domain acceleration with embedding evaluation
US11386276B2
( en )
*
2019-05-24
2022-07-12
International Business Machines Corporation
Method and system for language and domain acceleration with embedding alignment
US11514899B2
( en )
*
2020-01-21
2022-11-29
Motorola Solutions, Inc.
Using multiple languages during speech to text input
WO2022039745A1
( en )
*
2020-08-20
2022-02-24
Google Llc
Selecting from arrays of multilingual content
EP4200717A2
( en )
2020-08-24
2023-06-28
Unlikely Artificial Intelligence Limited
A computer implemented method for the automated analysis or use of data
US11562028B2
( en )
*
2020-08-28
2023-01-24
International Business Machines Corporation
Concept prediction to create new intents and assign examples automatically in dialog systems
US11664010B2
( en )
*
2020-11-03
2023-05-30
Florida Power & Light Company
Natural language domain corpus data set creation based on enhanced root utterances
US11574130B2
( en )
*
2020-11-24
2023-02-07
International Business Machines Corporation
Enhancing multi-lingual embeddings for cross-lingual question-answer system
US11783824B1
( en )
*
2021-01-18
2023-10-10
Amazon Technologies, Inc.
Cross-assistant command processing
US20220269734A1
( en )
*
2021-02-19
2022-08-25
Zebra Technologies Corporation
Intelligent natural language dialogue systems and methods for creating intelligent natural language dialogues for efficient retrieval of items in one or more large databases
US12524400B2
( en )
*
2021-04-20
2026-01-13
Microsoft Technology Licensing, Llc
Unified multilingual command recommendation model
CN113139816B
( en )
*
2021-04-26
2024-07-16
å京æ²ä¸å¤©éªä¿¡æ¯ææ¯æéå ¬å¸
Information processing method, device, electronic device and storage medium
US11853712B2
( en )
*
2021-06-07
2023-12-26
International Business Machines Corporation
Conversational AI with multi-lingual human chatlogs
US11977854B2
( en )
2021-08-24
2024-05-07
Unlikely Artificial Intelligence Limited
Computer implemented methods for the automated analysis or use of data, including use of a large language model
US12073180B2
( en )
2021-08-24
2024-08-27
Unlikely Artificial Intelligence Limited
Computer implemented methods for the automated analysis or use of data, including use of a large language model
US11989527B2
( en )
2021-08-24
2024-05-21
Unlikely Artificial Intelligence Limited
Computer implemented methods for the automated analysis or use of data, including use of a large language model
US12067362B2
( en )
2021-08-24
2024-08-20
Unlikely Artificial Intelligence Limited
Computer implemented methods for the automated analysis or use of data, including use of a large language model
US11989507B2
( en )
2021-08-24
2024-05-21
Unlikely Artificial Intelligence Limited
Computer implemented methods for the automated analysis or use of data, including use of a large language model
CN114661910B
( en )
*
2022-03-25
2025-05-06
å¹³å®ç§æï¼æ·±å³ï¼æéå ¬å¸
Intention recognition method, device, electronic device and storage medium
US20240169165A1
( en )
*
2022-11-17
2024-05-23
Samsung Electronics Co., Ltd.
Automatically Generating Annotated Ground-Truth Corpus for Training NLU Model
US12579370B2
( en )
*
2022-11-23
2026-03-17
Bank Of America Corporation
Multilingual chatbot
CN115840824B
( en )
*
2022-11-29
2026-02-06
æµ·å°ä¼å®¶æºè½ç§æï¼å京ï¼æéå ¬å¸
Intelligent home user spoken language understanding system oriented to environment information amplification
US20240386879A1
( en )
*
2023-05-18
2024-11-21
Amdocs Development Limited
System, method, and computer program for real-time language translation using generative artificial intelligence
US20260050753A1
( en )
*
2024-08-19
2026-02-19
Adobe Inc.
Guiding language translation with translation documents using machine learning
Citations (45)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
JPH11259089A
( en )
1998-03-16
1999-09-24
Atr Onsei Honyaku Tsushin Kenkyusho:Kk
Natural language understanding device and system therefor
CN1350685A
( en )
1999-03-09
2002-05-22
ç家è²å©æµ¦çµåæéå ¬å¸
Method with a pulrality of speech recognizers
US20070022134A1
( en )
2005-07-22
2007-01-25
Microsoft Corporation
Cross-language related keyword suggestion
US7212964B1
( en )
2000-12-08
2007-05-01
At&T Corp.
Language-understanding systems employing machine translation components
US20070198245A1
( en )
2006-02-20
2007-08-23
Satoshi Kamatani
Apparatus, method, and computer program product for supporting in communication through translation between different languages
CN101154220A
( en )
2006-09-25
2008-04-02
æ ªå¼ä¼ç¤¾ä¸è
Machine translation device and method
US20090024595A1
( en )
2007-07-20
2009-01-22
Google Inc.
Automatic expanded language search
US20090125497A1
( en )
2006-05-12
2009-05-14
Eij Group Llc
System and method for multi-lingual information retrieval
US7797688B1
( en )
2005-03-22
2010-09-14
Dubagunta Saikumar V
Integrating applications in multiple languages
US20110125483A1
( en )
2009-11-20
2011-05-26
Manuel-Devadoss Johnson Smith Johnson
Automated Speech Translation System using Human Brain Language Areas Comprehension Capabilities
US7984034B1
( en )
2007-12-21
2011-07-19
Google Inc.
Providing parallel resources in search results
US20120150532A1
( en )
*
2010-12-08
2012-06-14
At&T Intellectual Property I, L.P.
System and method for feature-rich continuous space language models
CN102799579A
( en )
2012-07-18
2012-11-28
西å®ç工大å¦
Statistical machine translation method with error self-diagnosis and self-correction functions
US20130144900A1
( en )
2011-11-30
2013-06-06
Bruce Winston MURPHY
System and method for determining user language intent
US20130151235A1
( en )
2008-03-26
2013-06-13
Google Inc.
Linguistic key normalization
US20130238312A1
( en )
2012-03-08
2013-09-12
Mobile Technologies, Llc
Device for extracting information from a dialog
US20140272821A1
( en )
2013-03-15
2014-09-18
Apple Inc.
User training by intelligent digital assistant
US20140278343A1
( en )
2013-03-15
2014-09-18
Bao Tran
Assistive agent
US20150142704A1
( en )
2013-11-20
2015-05-21
Justin London
Adaptive Virtual Intelligent Agent
US20150142447A1
( en )
2008-05-27
2015-05-21
Voicebox Technologies Corporation
System and method for an integrated, multi-modal, multi-device natural language voice services environment
US9164985B1
( en )
2011-11-29
2015-10-20
Google Inc.
Techniques for detecting poor machine translations of key terms
CN104991892A
( en )
2015-07-09
2015-10-21
ç¾åº¦å¨çº¿ç½ç»ææ¯ï¼å京ï¼æéå ¬å¸
Speech translation method and apparatus
US20150340033A1
( en )
2014-05-20
2015-11-26
Amazon Technologies, Inc.
Context interpretation in natural language processing using previous dialog acts
US20150370780A1
( en )
2014-05-30
2015-12-24
Apple Inc.
Predictive conversion of language input
US20160042748A1
( en )
*
2014-08-11
2016-02-11
Rawles Llc
Voice application architecture
US20160147744A1
( en )
2013-12-25
2016-05-26
Beijing Baidu Netcom Science And Technology Co., Ltd.
On-line voice translation method and device
US20160307567A1
( en )
*
2015-04-17
2016-10-20
Microsoft Technology Licensing, Llc
Context carryover in language understanding systems or methods
US20170060854A1
( en )
*
2015-08-25
2017-03-02
Alibaba Group Holding Limited
Statistics-based machine translation method, apparatus and electronic device
US9607102B2
( en )
2014-09-05
2017-03-28
Nuance Communications, Inc.
Task switching in dialogue processing
US20170091177A1
( en )
2015-09-30
2017-03-30
Kabushiki Kaisha Toshiba
Machine translation apparatus, machine translation method and computer program product
US9633317B2
( en )
2013-06-20
2017-04-25
Viv Labs, Inc.
Dynamically evolving cognitive architecture system based on a natural language intent interpreter
CN107170453A
( en )
2017-05-18
2017-09-15
ç¾åº¦å¨çº¿ç½ç»ææ¯ï¼å京ï¼æéå ¬å¸
Across languages phonetic transcription methods, equipment and computer-readable recording medium based on artificial intelligence
US20170270925A1
( en )
2007-12-11
2017-09-21
Voicebox Technologies Corporation
System and method for providing a natural language voice user interface
WO2017197187A1
( en )
2016-05-13
2017-11-16
Google Llc
Media transfer among media output devices
US9959272B1
( en )
*
2017-07-21
2018-05-01
Memsource a.s.
Automatic classification and translation of written segments
US10073843B1
( en )
*
2003-09-05
2018-09-11
Spoken Translation Inc.
Method and apparatus for cross-lingual communication
US10088853B2
( en )
2012-05-02
2018-10-02
Honeywell International Inc.
Devices and methods for interacting with an HVAC controller
US20190034429A1
( en )
*
2017-07-29
2019-01-31
Splunk Inc.
Translating a natural language request to a domain-specific language request using templates
US20190129947A1
( en )
*
2017-10-26
2019-05-02
Electronics And Telecommunications Research Institute
Neural machine translation method and apparatus
US20190213999A1
( en )
2018-01-08
2019-07-11
Apple Inc.
Multi-directional dialog
US10452782B1
( en )
2018-02-20
2019-10-22
Facebook, Inc.
Systems and methods for distributing intent models
US20190325873A1
( en )
2012-12-19
2019-10-24
Amazon Technologies, Inc.
Architecture for multi-domain natural language processing
US10546001B1
( en )
2015-04-15
2020-01-28
Arimo, LLC
Natural language queries based on user defined attributes
US10733222B1
( en )
2018-06-04
2020-08-04
Amazon Technologies, Inc.
Profile disambiguation
EP3723084A1
( en )
2018-03-07
2020-10-14
Google LLC
Facilitating end-to-end communications with automated assistants in multiple languages
2020
2020-02-17
US
US16/792,572
patent/US11354521B2/en
active
Active
Patent Citations (50)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
JPH11259089A
( en )
1998-03-16
1999-09-24
Atr Onsei Honyaku Tsushin Kenkyusho:Kk
Natural language understanding device and system therefor
CN1350685A
( en )
1999-03-09
2002-05-22
ç家è²å©æµ¦çµåæéå ¬å¸
Method with a pulrality of speech recognizers
US7212964B1
( en )
2000-12-08
2007-05-01
At&T Corp.
Language-understanding systems employing machine translation components
US10073843B1
( en )
*
2003-09-05
2018-09-11
Spoken Translation Inc.
Method and apparatus for cross-lingual communication
US7797688B1
( en )
2005-03-22
2010-09-14
Dubagunta Saikumar V
Integrating applications in multiple languages
US20070022134A1
( en )
2005-07-22
2007-01-25
Microsoft Corporation
Cross-language related keyword suggestion
US20070198245A1
( en )
2006-02-20
2007-08-23
Satoshi Kamatani
Apparatus, method, and computer program product for supporting in communication through translation between different languages
JP2007220045A
( en )
2006-02-20
2007-08-30
Toshiba Corp
Communication support device, communication support method, and communication support program
US20090125497A1
( en )
2006-05-12
2009-05-14
Eij Group Llc
System and method for multi-lingual information retrieval
CN101154220A
( en )
2006-09-25
2008-04-02
æ ªå¼ä¼ç¤¾ä¸è
Machine translation device and method
US20090024595A1
( en )
2007-07-20
2009-01-22
Google Inc.
Automatic expanded language search
US20170270925A1
( en )
2007-12-11
2017-09-21
Voicebox Technologies Corporation
System and method for providing a natural language voice user interface
US7984034B1
( en )
2007-12-21
2011-07-19
Google Inc.
Providing parallel resources in search results
US20130151235A1
( en )
2008-03-26
2013-06-13
Google Inc.
Linguistic key normalization
US20150142447A1
( en )
2008-05-27
2015-05-21
Voicebox Technologies Corporation
System and method for an integrated, multi-modal, multi-device natural language voice services environment
US20170294189A1
( en )
*
2008-05-27
2017-10-12
Voicebox Technologies Corporation
System and method for an integrated, multi-modal, multi-device natural language voice services environment
US20160217785A1
( en )
*
2008-05-27
2016-07-28
Voicebox Technologies Corporation
System and method for an integrated, multi-modal, multi-device natural language voice services environment
US9305548B2
( en )
*
2008-05-27
2016-04-05
Voicebox Technologies Corporation
System and method for an integrated, multi-modal, multi-device natural language voice services environment
US20110125483A1
( en )
2009-11-20
2011-05-26
Manuel-Devadoss Johnson Smith Johnson
Automated Speech Translation System using Human Brain Language Areas Comprehension Capabilities
US20120150532A1
( en )
*
2010-12-08
2012-06-14
At&T Intellectual Property I, L.P.
System and method for feature-rich continuous space language models
US9164985B1
( en )
2011-11-29
2015-10-20
Google Inc.
Techniques for detecting poor machine translations of key terms
US20130144900A1
( en )
2011-11-30
2013-06-06
Bruce Winston MURPHY
System and method for determining user language intent
WO2013134106A2
( en )
2012-03-08
2013-09-12
Mobile Technologies, Llc
Device for extracting information from a dialog
US20130238312A1
( en )
2012-03-08
2013-09-12
Mobile Technologies, Llc
Device for extracting information from a dialog
US10088853B2
( en )
2012-05-02
2018-10-02
Honeywell International Inc.
Devices and methods for interacting with an HVAC controller
CN102799579A
( en )
2012-07-18
2012-11-28
西å®ç工大å¦
Statistical machine translation method with error self-diagnosis and self-correction functions
US20190325873A1
( en )
2012-12-19
2019-10-24
Amazon Technologies, Inc.
Architecture for multi-domain natural language processing
US20140272821A1
( en )
2013-03-15
2014-09-18
Apple Inc.
User training by intelligent digital assistant
US20140278343A1
( en )
2013-03-15
2014-09-18
Bao Tran
Assistive agent
US9633317B2
( en )
2013-06-20
2017-04-25
Viv Labs, Inc.
Dynamically evolving cognitive architecture system based on a natural language intent interpreter
US20150142704A1
( en )
2013-11-20
2015-05-21
Justin London
Adaptive Virtual Intelligent Agent
US20160147744A1
( en )
2013-12-25
2016-05-26
Beijing Baidu Netcom Science And Technology Co., Ltd.
On-line voice translation method and device
US20150340033A1
( en )
2014-05-20
2015-11-26
Amazon Technologies, Inc.
Context interpretation in natural language processing using previous dialog acts
US20150370780A1
( en )
2014-05-30
2015-12-24
Apple Inc.
Predictive conversion of language input
US20160042748A1
( en )
*
2014-08-11
2016-02-11
Rawles Llc
Voice application architecture
US9607102B2
( en )
2014-09-05
2017-03-28
Nuance Communications, Inc.
Task switching in dialogue processing
US10546001B1
( en )
2015-04-15
2020-01-28
Arimo, LLC
Natural language queries based on user defined attributes
US20160307567A1
( en )
*
2015-04-17
2016-10-20
Microsoft Technology Licensing, Llc
Context carryover in language understanding systems or methods
CN104991892A
( en )
2015-07-09
2015-10-21
ç¾åº¦å¨çº¿ç½ç»ææ¯ï¼å京ï¼æéå ¬å¸
Speech translation method and apparatus
US20170060854A1
( en )
*
2015-08-25
2017-03-02
Alibaba Group Holding Limited
Statistics-based machine translation method, apparatus and electronic device
US20170091177A1
( en )
2015-09-30
2017-03-30
Kabushiki Kaisha Toshiba
Machine translation apparatus, machine translation method and computer program product
WO2017197187A1
( en )
2016-05-13
2017-11-16
Google Llc
Media transfer among media output devices
CN107170453A
( en )
2017-05-18
2017-09-15
ç¾åº¦å¨çº¿ç½ç»ææ¯ï¼å京ï¼æéå ¬å¸
Across languages phonetic transcription methods, equipment and computer-readable recording medium based on artificial intelligence
US9959272B1
( en )
*
2017-07-21
<td itemprop="publicat