Conceptio › Archive › Google Patents

Google Patentsopen access

Adapting automated assistants for use with multiple languages — Google Llc (US11113481B2)

Google Llc · Google Patents

Google Patents · Patents · License: Open Access

Open Source ↗

googlellc

patent, google patents, intellectual property, US11113481B2, Google Llc, Melvin Jose Johnson Premkumar, en, 2021

ABSTRACT

Abstract

Techniques described herein may serve to increase the language coverage of an automated assistant system, i.e. they may serve to increase the number of queries in one or more non-native languages for which the automated assistant is able to deliver reasonable responses. For example, techniques are described herein for training and utilizing a machine translation model to map a plurality of semantically-related natural language inputs in one language to one or more canonical translations in another language. In various implementations, the canonical translations may be selected and/or optimized for determining an intent of the speaker by the automated assistant, so that one or more responsive actions can be performed based on the speaker's intent. Put another way, the canonical translations may be specifically formatted for indicating the intent of the speaker to the automated assistant.

Description

BACKGROUND

Humans may engage in human-to-computer dialogs with interactive software applications referred to herein as âautomated assistantsâ (also referred to as âchatbots,â âinteractive personal assistants,â âintelligent personal assistants,â âpersonal voice assistants,â âconversational agents,â etc.). For example, humans (which when they interact with automated assistants may be referred to as âusersâ) may provide commands, queries, and/or requests (collectively referred to herein as âqueriesâ) using free form natural language input which may be vocal utterances converted into text and then processed, and/or typed free form natural language input. Automated assistants are typically invoked using predetermined vocal utterances (e.g., âOK Assistantâ) and often perform various types of processing, such as speech-to-text processing and/or natural language processing, only on those vocal utterances that follow an invocation phrase.

Automated assistants are adept at communicating with users in some widely-used languages, such as English, but are less able to communicate in other languages. However, conventional techniques for configuring automated assistants to communicate in new languages are resource-intensive. For each new language, it may be necessary to generate new trigger grammars (i.e., words or phrases intended to invoke a particular action by the automated assistant), recognize query intents in the new language (which require different natural language processing components for different languages), and to generate natural language output that is comprehensible in the new language. Adding to the challenge, each language has myriad idiosyncrasies (e.g., morphologically rich, pro-drop, gender neutrality, etc.) and language experts who can build these systems in particular languages may be hard to find.

SUMMARY

Techniques described herein relate in part to multilingual models that automated assistants to interpret and respond to queries, commands, and requests in multiple different languages, beyond a ânativeâ language of the automated assistant. Techniques described herein may also serve to increase the language coverage of an automated assistant system, i.e. they may serve to increase the number of queries in one or more non-native languages for which the automated assistant is able to deliver reasonable responses.

Some implementations are described herein for training and utilizing a machine translation model to map a plurality of semantically-related natural language inputs in one language to one or more canonical translations in another language. In various implementations, the canonical translations may be selected and/or optimized for determining an intent of the speaker by the automated assistant, so that one or more responsive actions can be performed based on the speaker's intent. Put another way, the canonical translations may be specifically formatted for indicating the intent of the speaker to the automated assistant. While example described herein will primarily relate to multiple speech inputs in one language being speech recognized and mapped to a single canonical translation in another language, this is not meant to be limiting. Techniques described herein may be applicable to unspoken natural language input, such as typed natural language input.

In various implementations, a spoken utterance may be captured, e.g., by one or more microphones, to generate audio data. The audio data may take various forms, such as an audio recording, an embedding generated based on the spoken utterance, etc. The spoken utterance may be in a first language that may not be natively supported by an automated assistant, or that at least may not be as well supported by the automated assistant as a second language.

In some implementations, speech-recognition processing may be performed on the audio data to generate speech-recognition output. Speech-recognition output may take various forms, such as text, a semantic embedding, etc. The speech-recognition output may be applied as input across a trained machine translation model to generate output. The output generated based on the machine translation model may be indicative of a canonical translation of the speech-recognition output from the first language to the second language. Based on the canonical translation, an intent conveyed by the spoken utterance may be determined and used to trigger one or more responsive actions by the automated assistant.

In various implementations, the machine translation model may be trained to map a plurality of semantically-related phrases in the first language to the same canonical translation in the second language. The canonical translation may vary syntactically from at least some of the plurality of semantically-related phrases. In other words, at least some of the plurality of semantically-related phrases in the first language will no longer be mapped to their âtrueâ or strict translations in the second language. Instead they may be mapped by the machine translation model to canonical translations in the second language. However, the canonical translations may be selected or optimized for use in the automated assistant domain. Consequently, language coverage for the first languageâi.e. the number of queries in the first language for which the automated assistant is able to deliver reasonable responsesâmay be increased. This may increase the utility of automated assistant systems and may decrease the number of instances of (and the computational resource use associated with) the automated assistant system being unable to handle a query received in a particular, non-native language.

Suppose the first language not natively supported by the automated assistant is German, and that the second language natively supported by the automated assistant is English. There may be several different German phrases that convey the intent of setting an alarm in the morning. Each of these German phrases may, when translated using a conventional machine translation model, generate a slightly different English translation than the others. Put another way, a conventional machine translation model may impose a many-to-many relationship between the various German phrases and English translations thereof. However, it may be the case that only a subset of the English translations are easily mapped by an automated assistant to the intent of setting an alarm. The English translations outside of this subsetâwhich may be âlong tailâ or less common ways of conveying the intent of setting an alarmâmay be more difficult for an automated assistant to respond to confidently and/or without seeking additional input from the user.

A machine translation model trained in accordance with the present disclosure may instead impose a many-to-one, or at least a many-to-fewer, relationship between the various German phrases and one or more canonical English translations. In some implementations, these one or more canonical English translations may be directly mapped to structured intents, in some cases with as little as a one-to-one mapping between the canonical translation and the structured intent. Put another way, a machine translation model trained accordance to the present disclosure may normalize a plurality of different translations of a plurality of semantically-related phrases into one or more canonical translations. As a consequence, any of the semantically-related phrases may be mapped to the one or more canonical translations, notwithstanding how they might be translated using a conventional machine translation model or naturally by a human translator.

In various implementations, a machine translation model configured with selected aspects of the present disclosure may be a sequence-to-sequence model such as a recurrent neural network. In some implementations, the machine translation model may be a sequence-to-sequence model such as an encoder-decoder model, sometimes referred to as an âautoencoder,â that includes an encoder portion and a decoder portion. The encoder portion may be trained to map data indicative of words or phrases in a first language to an embedding in language-neutral semantic space. The decoder portion may be trained to map embeddings in the semantic space to words or phrases in a second language.

In some implementations, the machine translation model may start as a conventional encoder-decoder model that is then optimized for use in the automated assistant domain. For example, the conventional encoder-decoder may be further trained using training data in which the target outputs (i.e., the desired word or phrase to be generated by the decoder portion) are rewritten in order to tune the machine translation model to map multiple semantically-related phrases to the same canonical translation.

As an example, suppose there are N (positive integer greater than one) different ways to convey a particular intent in German, and that these N phrases map to M (positive integer greater than one) different respective translations in English if traditional translation is applied. In some implementations, at least N training examples may be generated, at least one training example for each different German phrase. The target outputs of each of these N training examples may be rewritten from the traditional translations to one canonical translation. As noted previously, this one canonical translation may have a one-to-one relationship with a structured intent that is well-understood by the automated assistant.

The training data may then be used to further train the machine translation model. With each training example, the output generated by the model may be compared to the canonical translation. To the extent the output and the canonical translation differ (i.e., the error), techniques such as back propagation and/or gradient descent may be applied to adjust parameters of the machine translation model. Once a sufficient number of training examples are utilized in this manner, the machine translation model may be trained to map multiple semantically-related German phrases to the same canonical translation.

In some implementations in which the machine translation model is an encoder-decoder, a decoder portion may be trained to map multiple outputs of an encoder portionâe.g., one or more semantic embeddings representing a plurality of semantically-related phrases in a first languageâto the same canonical translation. In some such implementations, the decoder portion may be trained to map an entire cluster of semantic embeddings to the same canonical translation, such that any individual embedding of the cluster is mapped to the canonical translation regardless of what its âtrueâ or strict translation would be. Additionally or alternatively, in some implementations, an encoder portion of the encoder-decoder may be trained to map a plurality of semantically-related phrases in the first language to a lower number of semantic embeddings than would otherwise be mapped with a conventional machine translation model. These fewer semantic embeddings may in turn be mapped to canonical translation(s).

Techniques described herein may be applicable to machine translation models that are trained to translate from more than one language to a natively supported language. For example, multiple encoders may be trained, one for each language other than the natively-supported language. Each encoder may generate semantic embeddings from input in a respective language. In some implementations, these semantic embeddings may be language neutral. For example, the same (or substantially similar) semantic embedding may be generated by a French encoder based on a French utterance as a German encoder based on a semantically-similar German utterance. A single decoder portion may then be used to process semantic embeddings generated by any of the encoders. As described above, the single decoder may map the semantic embeddings to canonical translations.

In some examples, a final performance of the model may be improved using multi-task training. More specifically, the decoder portion may additionally receive embeddings generated by an additional encoder that is trained to map semantically-related phrases received in the second, natively-supported language to semantic embeddings. The semantic embeddings generated by the additional encoder may then be processed by the decoder portion in the manner described above.

There are various ways to generate training data used to train machine translation models in accordance with the present disclosure. In some implementations, one or more canonical phrases may be selected for a given responsive action, such as setting an alarm, setting a reminder, etc. These canonical phrases may be selected, for instance, based on being the most frequent or popular phrases used to identify structured intents that are mapped to responsive actions. For example, users may convey the intent to set an alarm at a particular <time> in several ways: âset an alarm for <time>,â âplease wake me up at <time>,â âI need an alarm set for <time>,â âat <time> please wake me up,â etc. However, one of these phrases, such as âset an alarm for <time>,â may be the most frequently used. Accordingly, that phrase may be selected as the canonical translation to which multiple different semantically-related phrases in other languages may be mapped, regardless of those phrases' âtrueâ translations.

In some implementations, a method performed by one or more processors is provided that includes: capturing a spoken utterance at a microphone to generate audio data, wherein the spoken utterance is spoken in a first language; performing speech recognition processing on the audio data to generate speech-recognition output; applying the speech-recognition output as input across a trained machine translation model to generate output, wherein the output is indicative of a canonical translation of the speech-recognition output from the first language to a second language; determining an intent conveyed by the spoken utterance based on the canonical translation; and performing one or more responsive actions based on the intent. In various implementations, the machine translation model is trained to map a plurality of semantically-related phrases in the first language to the same canonical translation, wherein the canonical translation varies syntactically from at least some of the plurality of semantically-related phrases.

These and other implementations of technology disclosed herein may optionally include one or more of the following features.

In various implementations, the machine translation model may take the form of a sequence-to-sequence model. In various implementations, the machine translation model may take the form of an encoder-decoder. In various implementations, a decoder portion of the encoder-decoder is trained to map one or more semantic embeddings representing the plurality of semantically-related phrases in the first language to the same canonical translation. In various implementations, an encoder portion of the encoder-decoder is trained to map the plurality of semantically-related phrases in the first language to a fewer number of semantic embeddings.

In various implementations, the method may further include: capturing another spoken utterance at the same microphone or a different microphone to generate additional audio data, wherein the another spoken utterance is spoken in a third language; performing speech recognition processing on the additional audio data to generate additional speech-recognition output; applying the additional speech-recognition output as input across the trained machine translation model to generate additional output, wherein the additional output is indicative of the canonical translation of the speech-recognized text; determining an intent conveyed by the another spoken utterance based on the canonical translation; and performing one or more additional responsive actions based on the intent. In some such implementations, the machine translation model may be trained to map a plurality of semantically-related phrases in the third language to the same canonical translation, wherein the canonical translation varies syntactically from at least some of the plurality of semantically-related phrases in the third language.

In various implementations, the intent may be determined based on a mapping between the canonical translation and the intent. In various implementations, determining the intent may include performing natural language processing on the canonical translation to determine the intent. In some implementations, the trained machine translation module includes a word piece vocabulary that is shared across multiple languages.

In another aspect, a method implemented using one or more processors may inc

BACKGROUND

SUMMARY

These and other implementations of technology disclosed herein may optionally include one or more of the following features.

In another aspect, a method implemented using one or more processors may include: identifying a plurality of semantically-related phrases in a first language; identifying a plurality of translations of the plurality of semantically-related phrases in a second language; generating a canonical translation in the second language, the canonical translation corresponding to the plurality of translations of the plurality of semantically-related phrases; generating training data based on the plurality of semantically-related phrases and the canonical translation; and training the machine translation model using the training data, wherein upon being trained, the machine translation model maps the plurality of semantically-related phrases in the first language to the canonical translation in the second language. In various implementations, prior to being trained, the machine learning model may have mapped the plurality of semantically-related phrases in the first language to the plurality of translations in the second language.

In addition, some implementations include one or more processors of one or more computing devices, where the one or more processors are operable to execute instructions stored in associated memory, and where the instructions are configured to cause performance of any of the aforementioned methods. Some implementations also include one or more non-transitory computer readable storage media storing computer instructions executable by one or more processors to perform any of the aforementioned methods.

It should be appreciated that all combinations of the foregoing concepts and additional concepts described in greater detail herein are contemplated as being part of the subject matter disclosed herein. For example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an example environment in which implementations disclosed herein may be implemented.

FIG. 2 depicts an example process flow that demonstrates various aspects of the present disclosure, in accordance with various implementations.

FIGS. 3, 4, 5A, and 5B depict example dialogs between a user and an automated assistant, in accordance with various implementations.

FIG. 6 depicts a flowchart illustrating an example method according to implementations disclosed herein.

FIG. 7 depicts a flowchart illustrating another example method according to implementations disclosed herein.

FIG. 8 illustrates an example architecture of a computing device.

DETAILED DESCRIPTION

Now turning to FIG. 1 , an example environment in which techniques disclosed herein may be implemented is illustrated. The example environment includes a plurality of client computing devices 106 1-N . Each client device 106 may execute a respective instance of an automated assistant client 118 . One or more cloud-based automated assistant components 119 , such as a natural language understanding engine 135 , may be implemented on one or more computing systems (collectively referred to as a âcloudâ computing system) that are communicatively coupled to client devices 106 1-N via one or more local and/or wide area networks (e.g., the Internet) indicated generally at 110 .

In some implementations, the plurality of client computing devices 106 1-N (also referred to herein simply as âclient devicesâ) may be associated with each other in various ways in order to facilitate performance of techniques described herein. For example, in some implementations, plurality of client computing devices 106 1-N may be associated with each other by virtue of them being members of a coordinated ecosystem of client devices 106 that are operated by one or more users (e.g., an individual, a family, employees of an organization, other predefined groups, etc.).

As noted in the background, an instance of an automated assistant client 118 , by way of its interactions with one or more cloud-based automated assistant components 119 , may form what appears to be, from the user's perspective, a logical instance of an automated assistant 120 with which the user may engage in a human-to-computer dialog. Two instances of such an automated assistant 120 are depicted in FIG. 1 . A first automated assistant 120 A encompassed by a dashed line serves a first user (not depicted) operating first client device 106 1 and includes automated assistant client 118 1 and one or more cloud-based automated assistant components 119 .

A second automated assistant 120 B encompassed by a dash-dash-dot line serves a second user (not depicted) operating another client device 106 N and includes automated assistant client 118 N and one or more cloud-based automated assistant components 119 . It thus should be understood that each user that engages with an automated assistant client 118 executing on a client device 106 may, in effect, engage with his or her own logical instance of an automated assistant 120 . For the sake of brevity and simplicity, the term âautomated assistantâ as used herein as âservingâ a particular user will refer to the combination of an automated assistant client 118 executing on a client device 106 operated by the user and one or more cloud-based automated assistant components 119 (which may be shared amongst multiple automated assistant clients 118 ). It should also be understood that in some implementations, automated assistant 120 may respond to a request from any user regardless of whether the user is actually âservedâ by that particular instance of automated assistant 120 .

The client devices 106 1-N may include, for example, one or more of: a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device of a vehicle of the user (e.g., an in-vehicle communications system, an in-vehicle entertainment system, an in-vehicle navigation system), a standalone interactive speaker, a smart appliance such as a smart television, and/or a wearable apparatus of the user that includes a computing device (e.g., a watch of the user having a computing device, glasses of the user having a computing device, a virtual or augmented reality computing device). Additional and/or alternative client computing devices may be provided.

Each of the client computing devices 106 1-N may operate a variety of different applications, such as a corresponding one of a plurality of message exchange clients 107 1-N . Message exchange clients 107 1-N may come in various forms and the forms may vary across the client computing devices 106 1-N and/or multiple forms may be operated on a single one of the client computing devices 106 1-N . In some implementations, one or more of the message exchange clients 107 1-N may come in the form of a short messaging service (âSMSâ) and/or multimedia messaging service (âMMSâ) client, an online chat client (e.g., instant messenger, Internet relay chat, or âIRC,â etc.), a messaging application associated with a social network, a personal assistant messaging service dedicated to conversations with automated assistant 120 , and so forth. In some implementations, one or more of the message exchange clients 107 1-N may be implemented via a webpage or other resources rendered by a web browser (not depicted) or other application of client computing device 106 .

As described in more detail herein, automated assistant 120 engages in human-to-computer dialog sessions with one or more users via user interface input and output devices of one or more client devices 106 1-N . In some implementations, automated assistant 120 may engage in a human-to-computer dialog session with a user in response to user interface input provided by the user via one or more user interface input devices of one of the client devices 106 1-N . In some of those implementations, the user interface input is explicitly directed to automated assistant 120 . For example, one of the message exchange clients 107 1-N may be a personal assistant messaging service dedicated to conversations with automated assistant 120 and user interface input provided via that personal assistant messaging service may be automatically provided to automated assistant 120 . Also, for example, the user interface input may be explicitly directed to automated assistant 120 in one or more of the message exchange clients 107 1-N based on particular user interface input that indicates automated assistant 120 is to be invoked. For instance, the particular user interface input may be one or more typed characters (e.g., @AutomatedAssistant), user interaction with a hardware button and/or virtual button (e.g., a tap, a long tap), an oral command (e.g., âHey Automated Assistantâ), and/or other particular user interface input.

In some implementations, automated assistant 120 may engage in a human-to-computer dialog session in response to user interface input, even when that user interface input is not explicitly directed to automated assistant 120 . For example, automated assistant 120 may examine the contents of user interface input and engage in a dialog session in response to certain terms being present in the user interface input and/or based on other cues. In many implementations, the user may utter commands, searches, etc., and the automated assistant may utilize speech recognition to convert the utterances into text, and respond to the text accordingly, e.g., by providing search results, general information, and/or taking one or more responsive actions (e.g., playing media, launching a game, ordering food, etc.). In some implementations, the automated assistant 120 can additionally or alternatively respond to utterances without converting the utterances into text. For example, the automated assistant 120 can convert voice input into an embedding, into entity representation(s) (that indicate entity/entities present in the voice input), and/or other ânon-textualâ representation and operate on such non-textual representation. Accordingly, implementations described herein as operating based on text converted from voice input may additionally and/or alternatively operate on the voice input directly and/or other non-textual representations of the voice input.

Each of the client computing devices 106 1-N and computing device(s) operating cloud-based automated assistant components 119 may include one or more memories for storage of data and software applications, one or more processors for accessing data and executing applications, and other components that facilitate communication over a network. The operations performed by one or more of the client computing devices 106 1-N and/or by automated assistant 120 may be distributed across multiple computer systems. Automated assistant 120 may be implemented as, for example, computer programs running on one or more computers in one or more locations that are coupled to each other through a network.

As noted above, in various implementations, each of the client computing devices 106 1-N may operate an automated assistant client 118 . In various implementations, each automated assistant client 118 may include a corresponding speech capture/text-to-speech (âTTSâ)/speech-to-text (âSTTâ) module 114 . In other implementations, one or more aspects of speech capture/TTS/STT module 114 may be implemented separately from automated assistant client 118 . In various implementations, speech capture/TTS/STT module 114 may generate speech recognition output based on a vocal query. This speech recognition output may be analyzed pursuant to techniques described herein.

Each speech capture/TTS/STT module 114 may be configured to perform one or more functions: capture a user's speech, e.g., via a microphone; convert that captured audio to text (and/or to other representations or embeddings) using various speech recognition techniques; and/or convert text to speech. For example, in some implementations, because a client device 106 may be relatively constrained in terms of computing resources (e.g., processor cycles, memory, battery, etc.), the speech capture/TTS/STT module 114 that is local to each client device 106 may be configured to convert a finite number of different spoken phrasesâparticularly phrases that invoke automated assistant 120 âto text (or to other forms, such as lower dimensionality embeddings). Other speech input may be sent to cloud-based automated assistant components 119 , which may include a cloud-based TTS module 116 and/or a cloud-based STT module 117 .

Cloud-based STT module 117 may be configured to leverage the virtually limitless resources of the cloud to convert audio data captured by speech capture/TTS/STT module 114 into text (which may then be provided to natural language processor 122 ) using various speech recognition techniques. Cloud-based TTS module 116 may be configured to leverage the virtually limitless resources of the cloud to convert textual data (e.g., natural language responses formulated by automated assistant 120 ) into computer-generated speech output. In some implementations, TTS module 116 may provide the computer-generated speech output to client device 106 to be output directly, e.g., using one or more speakers. In other implementations, textual data (e.g., natural language responses) generated by automated assistant 120 may be provided to speech capture/TTS/STT module 114 , which may then convert the textual data into computer-generated speech that is output locally.

Automated assistant 120 (and in particular, cloud-based automated assistant components 119 ) may include a natural language understanding engine 135 , the aforementioned TTS module 116 , the aforementioned STT module 117 , and other components that are described in more detail below. In some implementations, one or more of the engines and/or modules of automated assistant 120 may be omitted, combined, and/or implemented in a component that is separate from automated assistant 120 . In some implementations, to protect privacy, one or more of the components of automated assistant 120 , such as natural language processor 122 , speech capture/TTS/STT module 114 , etc., may be implemented at least on part on client devices 106 (e.g., to the exclusion of the cloud).

In some implementations, automated assistant 120 generates responsive content in response to various inputs generated by a user of one of the client devices 106 1-N during a human-to-computer dialog session with automated assistant 120 . Automated assistant 120 may provide the responsive content (e.g., over one or more networks when separate from a client device of a user) for presentation to the user as part of the dialog session. For example, automated assistant 120 may generate responsive content in response to free-form natural language input provided via one of the client devices 106 1-N . As used herein, free-form input is input that is formulated by a user and that is not constrained to a group of options presented for selection by the user.

As used herein, a âdialog sessionâ may include a logically-self-contained exchange of one or more messages between a user and automated assistant 120 (and in some cases, other human participants). Automated assistant 120 may differentiate between multiple dialog sessions with a user based on various signals, such as passage of time between sessions, change of user context (e.g., location, before/during/after a scheduled meeting, etc.) between sessions, detection of one or more intervening interactions between the user and a client device other than dialog between the user and the automated assistant (e.g., the user switches applications for a while, the user walks away from then later returns to a standalone voice-activated product), locking/sleeping of the client device between sessions, change of client devices used to interface with one or more instances of automated assistant 120 , and so forth.

Natural language processor 122 of natural language understanding engine 135 processes natural language input generated by users via client devices 106 1-N and may generate annotated output (e.g., in textual form) for use by one or more other components of automated assistant 120 . For example, the natural language processor 122 may process natural language free-form input that is generated by a user via one or more user interface input devices of client device 106 1 . The generated annotated output includes one or more annotations of the natural language input and one or more (e.g., all) of the terms of the natural language input.

In some implementations, the natural language processor 122 is configured to identify and annotate various types of grammatical information in natural language input. For example, the natural language processor 122 may include a morphological engine that may separate individual words into morphemes and/or annotate the morphemes, e.g., with their classes. Natural language processor 122 may also include a part of speech tagger configured to annotate terms with their grammatical roles. For example, the part of speech tagger may tag each term with its part of speech such as ânoun,â âverb,â âadjective,â âpronoun,â etc. Also, for example, in some implementations the natural language processor 122 may additionally and/or alternatively include a dependency parser (not depicted) configured to determine syntactic relationships between terms in natural language input. For example, the dependency parser may determine which terms modify other terms, subjects and verbs of sentences, and so forth (e.g., a parse tree)âand may make annotations of such dependencies.

In some implementations, the natural language processor 122 may additionally and/or alternatively include an entity tagger (not depicted) configured to annotate entity references in one or more segments such as references to people (including, for instance, literary characters, celebrities, public figures, etc.), organizations, locations (real and imaginary), and so forth. In some implementations, data about entities may be stored in one or more databases, such as in a knowledge graph (not depicted). In some implementations, the knowledge graph may include nodes that represent known entities (and in some cases, entity attributes), as well as edges that connect the nodes and represent relationships between the entities. For example, a âbananaâ node may be connected (e.g., as a child) to a âfruitâ node,â which in turn may be connected (e.g., as a child) to âproduceâ and/or âfoodâ nodes. As another example, a restaurant called âHypothetical CafÃ©â may be represented by a node that also includes attributes such as its address, type of food served, hours, contact information, etc. The âHypothetical CafÃ©â node may in some implementations be connected by an edge (e.g., representing a child-to-parent relationship) to one or more other nodes, such as a ârestaurantâ node, a âbusinessâ node, a node representing a city and/or state in which the restaurant is located, and so forth.

The entity tagger of the natural language processor 122 may annotate references to an entity at a high level of granularity (e.g., to enable identification of all references to an entity class such as people) and/or a lower level of granularity (e.g., to enable identification of all references to a particular entity such as a particular person). The entity tagger may rely on content of the natural language input to resolve a particular entity and/or may optionally communicate with a knowledge graph or other entity database to resolve a particular entity.

In some implementations, the natural language processor 122 may additionally and/or alternatively include a coreference resolver (not depicted) configured to group, or âcluster,â references to the same entity based on one or more contextual cues. For example, the coreference resolver may be utilized to resolve the term âthereâ to âHypothetical CafÃ©â in the natural language input âI liked Hypothetical CafÃ© last time we ate there.â

In some implementations, one or more components of the natural language processor 122 may rely on annotations from one or more other components of the natural language processor 122 . For example, in some implementations the named entity tagger may rely on annotations from the coreference resolver and/or dependency parser in annotating all mentions to a particular entity. Also, for example, in some implementations the coreference resolver may rely on annotations from the dependency parser in clustering references to the same entity. In some implementations, in processing a particular natural language input, one or more components of the natural language processor 122 may use related prior input and/or other related data outside of the particular natural language input to determine one or more annotations.

Natural language understanding engine 135 may also include an intent matcher 136 that is configured to determine an intent of a user engaged in a human-to-computer dialog session with automated assistant 120 . While depicted separately from natural language processor 122 in FIG. 1 , in other implementations, intent matcher 136 may be an integral part of natural language processor 122 (or more generally, of a pipeline that includes natural language processor 122 ). In some implementations, natural language processor 122 and intent matcher 136 may collectively form the aforementioned ânatural language understandingâ engine 135 .

Intent matcher 136 may use various techniques to determine an intent of the user, e.g., based on output from natural language processor 122 (which may include annotations and terms of the natural language input). In some implementations, intent matcher 136 may have access to one or more databases 137 that include, for instance, a plurality of mappings between grammars and responsive actions, or more generally, intents. In many cases, these grammars may be selected and/or learned over time, and may represent the most common intents of users. For example, one grammar, âplay <artist>â, may be mapped to an intent that invokes a responsive action that causes music by the <artist> to be played on the client device 106 operated by the user. Another grammar, â[weather|forecast] today,â may be match-able to user queries such as âwhat's the weather todayâ and âwhat's the forecast for today?â As will be described in more detail below, in addition to or instead of grammars, in some implementations, intent matcher 136 may employ one or more trained machine learning models, alone or in combination with one or more grammars. These trained machine learning models may also be stored in one or more databases 137 .

As seen in the âplay <artist>â example grammar, some grammars have slots (e.g., <artist>) that can be filled with slot values (or âparametersâ). Slot values may be determined in various ways. Often users will provide the slot values proactively. For example, for a grammar âOrder me a <topping> pizza,â a user may likely speak the phrase âorder me a sausage pizza,â in which case the slot <topping> is filled automatically. Additionally or alternatively, if a user invokes a grammar that includes slots to be filled with slot values, without the user proactively providing the slot values, automated assistant 120 may solicit those slot values from the user (e.g., âwhat type of crust do you want on your pizza?â).

In some implementations, automated assistant 120 may facilitate (or âbrokerâ) transactions between users and third party applications. These third party applications may or may not operate on computing systems that are separate from those that operate, for instance, cloud-based automated assistant components 119 . Accordingly, one kind of user intent that may be identified by intent matcher 136 is to engage a third party application. For example, automated assistant 120 may provide access to an application programming interface (âAPIâ) to a pizza delivery service. A user may invoke automated assistant 120 and provide a command such as âI'd like to order a pizza.â Intent matcher 136 may map this command to a grammar (which may be added to database 137 in some cases by the third party) that triggers automated assistant 120 to engage with the third party pizza delivery service. The third party pizza delivery service may provide automated assistant 120 with a minimum list of slots that need to be filled in order to fulfill a pizza delivery order. Automated assistant 120 may generate and provide to the user (via client device 106 ) natural language output that solicits parameters for the slots.

Fulfillment engine 124 may be configured to receive the intent output by intent matcher 136 , as well as an associated slot values (whether provided by the user proactively or solicited from the user) and fulfill the intent. In various implementations, fulfillment of the user's intent may cause various fulfillment information (which may be language-agnostic in some instances) to be generated/obtained, e.g., by fulfillment engine 124 . As will be described below, the fulfillment information may in some implementations be provided to a natural language generator (âNLGâ in some FIGS.) 126 , which may generate natural language output based on the fulfillment information.

Fulfillment information may take various forms because an intent can be fulfilled in a variety of ways. Suppose a user requests pure information, such as âWhere were the outdoor shots of âThe Shiningâ filmed?â The intent of the user may be determined, e.g., by intent matcher 136 , as being a search query. The intent and content of the search query may be provided to fulfillment engine 124 , which as depicted in FIG. 1 may be in communication with one or more search engines 150 configured to search corpuses of documents and/or other data sources (e.g., knowledge graphs, etc.) for responsive information. Fulfillment engine 124 may provide data indicative of the search query (e.g., the text of the query, a reduced dimensionality embedding, etc.) to search engine 150 . Search engine 150 may provide responsive information, such as GPS coordinates, or other more explicit information, such as âTimberline Lodge, Mt. Hood, Oreg.â This responsive information may form part of the fulfillment information generated by fulfillment engine 124 .

Additionally or alternatively, fulfillment engine 124 may be configured to receive, e.g., from natural language understanding engine 135 , a user's intent and any slot values provided by the user or determined using other means (e.g., GPS coordinates of the user, user preferences, etc.) and trigger a responsive action. Responsive actions may include, for instance, ordering a good/service, starting a timer, setting a reminder, initiating a phone call, playing media, sending a message, etc. In some such implementations, fulfillment information may include slot values associated with the fulfillment, confirmation responses (which may be selected from predetermined responses in some cases), etc.

As noted above, natural language generator 126 may be configured to generate and/or select natural language output (e.g., spoken words/phrases that are designed to mimic human speech) based on data obtained from various sources. In some implementations, natural language generator 126 may be configured to receive, as input, fulfillment information associated with fulfillment of an intent, and to generate natural language output based on the fulfillment information. Additionally or alternatively, natural language generator 126 may receive information from other sources, such as third party applications (e.g., required slots), which it may use to compose natural language output for the user.

If a user's intent is to search for general information, then natural language generator 126 may generate natural language output that conveys information responsive to the user's, e.g., in sentence form. In some instances, the natural language output may be extracted, e.g., by <figure-callout id="126" label="natural langu

CLAIMS

Claims ( 13 )

What is claimed is:

1. A method implemented using one or more processers, comprising:

capturing a spoken utterance at a microphone to generate audio data, wherein the spoken utterance is spoken in a first language;

performing speech recognition processing on the audio data to generate speech-recognition output;

applying the speech-recognition output as input across a trained encoder-decoder machine translation model to generate output, wherein the output comprises a canonical second language translation of the speech-recognition output from the first language;

determining an intent conveyed by the spoken utterance based on the canonical second language translation; and

performing one or more responsive actions based on the intent;

wherein the encoder-decoder machine translation model was trained previously to map a plurality of syntactically-distinct but semantically-similar phrases in the first language to the same canonical second language translation, wherein the encoder-decoder machine translation model was trained previously by:

processing the plurality of phrases in the first language based on the encoder-decoder machine translation model to generate a plurality of syntactically-distinct but semantically-similar second language translations; and

comparing the plurality of syntactically-distinct but semantically-similar second language translations to the canonical second language translation to generate corresponding errors, whereby the encoder-decoder machine translation model was trained previously based on the corresponding errors.

2. The method of claim 1 , wherein a decoder portion of the encoder-decoder machine translation model is trained to map one or more semantic embeddings representing the plurality of syntactically-different but semantically-related phrases in the first language to the same canonical translation.

3. The method of claim 1 , wherein an encoder portion of the encoder-decoder machine translation model is trained to map the syntactically-different but plurality of semantically-related phrases in the first language to a lower number of semantic embeddings.

4. The method of claim 1 , further comprising:

capturing another spoken utterance at the same microphone or a different microphone to generate additional audio data, wherein the another spoken utterance is spoken in a third language;

performing speech recognition processing on the additional audio data to generate additional speech-recognition output;

applying the additional speech-recognition output as input across the trained machine translation model to generate additional output, wherein the additional output comprises the canonical second language translation of the speech-recognized text;

determining an additional intent conveyed by the another spoken utterance based on the canonical second language translation; and

performing one or more additional responsive actions based on the additional intent;

wherein the encoder-decoder machine translation model is trained to map a plurality of semantically-related phrases in the third language to the same canonical second language translation, wherein the canonical second language translation varies syntactically from at least some of the plurality of semantically-related phrases in the third language.

5. The method of claim 1 , wherein the trained encoder-decoder machine translation model includes a word piece vocabulary that is shared across multiple languages.

6. The method of claim 1 , wherein determining the intent comprises performing natural language processing on the canonical second language translation to determine the intent.

7. A method of training encoder-decoder machine translation model to map a plurality of syntactically-distinct but semantically-similar phrases in a first language to a single canonical second language translation, comprising:

processing the plurality of syntactically-distinct but semantically-similar phrases in the first language based on the encoder-decoder machine translation model to generate a plurality of syntactically-distinct but semantically-similar second language translations;

comparing the plurality of syntactically-distinct but semantically-similar second language translations to the single canonical second language translation to generate corresponding errors; and

training the encoder-decoder machine translation model based on the corresponding errors.

8. The method of claim 7 , wherein the training comprises training a decoder portion of the encoder-decoder machine translation model to map one or more semantic embeddings representing the plurality of syntactically-distinct but semantically-related phrases in the first language to the single canonical second language translation.

9. The method of claim 7 , wherein the training comprises training an encoder portion of the encoder-decoder machine translation model to map the plurality of syntactically-distinct but semantically-related phrases in the first language to a lower number of semantic embeddings.

10. A system comprising one or more processors and memory storing instructions that, in response to execution of the instructions by the one or more processors, cause the one or more processors to:

capture a spoken utterance at a microphone to generate audio data, wherein the spoken utterance is spoken in a first language;

perform speech recognition processing on the audio data to generate speech-recognition output;

apply the speech-recognition output as input across a trained encoder-decoder machine translation model to generate output, wherein the output comprises a canonical second language translation of the speech-recognition output from the first language;

determine an intent conveyed by the spoken utterance based on the canonical second language translation; and

perform one or more responsive actions based on the intent;

11. The system of claim 10 , wherein a decoder portion of the encoder-decoder machine translation model is trained to map one or more semantic embeddings representing the plurality of syntactically-different but semantically-related phrases in the first language to the same canonical second language translation.

12. The system of claim 10 , wherein an encoder portion of the encoder-decoder machine translation model is trained to map the plurality of syntactically-different but semantically-related phrases in the first language to a lower number of semantic embeddings.

13. The system of claim 10 , wherein the intent is determined based on a mapping between the canonical second language translation and the intent.

US16/621,578

2019-05-02

Adapting automated assistants for use with multiple languages

Active

2039-06-24

US11113481B2

( en )

Applications Claiming Priority (1)

Application Number

Priority Date

Filing Date

Title

PCT/US2019/030343

WO2020222846A1

( en )

2019-05-02

Adapting automated assistants for use with multiple languages

Publications (2)

Publication Number

Publication Date

US20210064828A1

US20210064828A1 ( en )

2021-03-04

US11113481B2

true

US11113481B2 ( en )

2021-09-07

Family

ID=66530529

Family Applications (1)

Application Number

Title

Priority Date

Filing Date

US16/621,578

Active

2039-06-24

US11113481B2

( en )

2019-05-02

Adapting automated assistants for use with multiple languages

Country Status (4)

Country

Link

( 1 )

US11113481B2

( en )

( 1 )

EP3899927B1

( en )

( 1 )

CN113412515B

( en )

( 1 )

WO2020222846A1

( en )

Cited By (1)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20220327288A1

( en )

2021-04-13

2022-10-13

Microsoft Technology Licensing, Llc

Semantic Parsing of Utterance Using Contractive Paraphrasing

Families Citing this family (17)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US11475875B2

( en )

2018-10-26

2022-10-18

Sriram Chakravarthy

Method and system for implementing language neutral virtual assistant

US11567788B1

( en )

2019-10-18

2023-01-31

Meta Platforms, Inc.

Generating proactive reminders for assistant systems

US11861674B1

( en )

2019-10-18

2024-01-02

Meta Platforms Technologies, Llc

Method, one or more computer-readable non-transitory storage media, and a system for generating comprehensive information for products of interest by assistant systems

US12574627B2

( en )

2019-10-18

2026-03-10

Meta Platforms Technologies, Llc

Smart cameras enabled by assistant systems

US11580312B2

( en )

2020-03-16

2023-02-14

Servicenow, Inc.

Machine translation of chat sessions

US11385916B2

( en )

2020-03-16

2022-07-12

Servicenow, Inc.

Dynamic translation of graphical user interfaces

US11443193B2

( en )

2020-04-24

2022-09-13

Adobe Inc.

Domain adaptation for machine learning models

US11551695B1

( en )

2020-05-13

2023-01-10

Amazon Technologies, Inc.

Model training system for custom speech-to-text models

US11218594B1

( en )

2020-08-11

2022-01-04

Genesys Telecommunications Laboratories, Inc.

System and method for creating bots for automating first party touchpoints

US12524400B2

( en )

2021-04-20

2026-01-13

Microsoft Technology Licensing, Llc

Unified multilingual command recommendation model

US12380475B2

( en )

2021-09-17

2025-08-05

The Toronto-Dominion Bank

Systems and methods for automated response to online reviews

US12518744B2

( en )

2021-11-30

2026-01-06

Google Llc

Selectively masking query content to provide to a secondary digital assistant

US12481840B2

( en )

2022-04-06

2025-11-25

Avodah, Inc.

Translating documents using neural networks and hierarchical structuring of documents

US20240062014A1

( en )

2022-08-16

2024-02-22

Nvidia Corporation

Generating canonical forms for task-oriented dialogue in conversational ai systems and applications

WO2024048816A1

( en )

2022-09-01

2024-03-07

ìì§ì ì ì£¼ìíì¬

Device and method for transmitting or receiving signal in wireless communication system

KR20240032581A

( en )

2022-09-02

2024-03-12

ì¼ì±ì ìì£¼ìíì¬

Electronic device for generating a translation using a translation method selected from among a plurality of translation methods and a translation generating mwthod thereof

US20250284683A1

( en )

2024-03-07

2025-09-11

Palo Alto Networks, Inc.

Natural language query to domain-specific database query conversion with language models

Citations (8)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20010029455A1

( en )

2000-03-31

2001-10-11

Chin Jeffrey J.

Method and apparatus for providing multilingual translation over a network

US20030144832A1

( en )

2002-01-16

2003-07-31

Harris Henry M.

Machine translation system

US8583416B2

( en )

2007-12-27

2013-11-12

Fluential, Llc

Robust information extraction from utterances

US20140205974A1

( en )

2011-06-30

2014-07-24

Rosetta Stone, Ltd.

Statistical machine translation framework for modeling phonological errors in computer assisted pronunciation training system

WO2017112813A1

( en )

2015-12-22

2017-06-29

Sri International

Multi-lingual virtual personal assistant

US20190138606A1

( en )

2016-07-12

2019-05-09

Huawei Technologies Co., Ltd.

Neural network-based translation method and apparatus

US10867136B2

( en )

2016-07-07

2020-12-15

Samsung Electronics Co., Ltd.

Automatic interpretation method and apparatus

US10891435B1

( en )

2018-02-20

2021-01-12

Interactions Llc

Bootstrapping multilingual natural language understanding via machine translation

Family Cites Families (3)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US7577562B2

( en )

2004-11-04

2009-08-18

Microsoft Corporation

Extracting treelet translation pairs

CN107329960B

( en )

2017-06-29

2019-01-01

åå°æ»¨å·¥ä¸å¤§å¦

Unregistered word translating equipment and method in a kind of neural network machine translation of context-sensitive

CN107526727B

( en )

2017-07-31

2021-01-19

èå·å¤§å¦

Language generation method based on statistical machine translation

2019

2019-05-02

US16/621,578

patent/US11113481B2/en

active

Active

2019-05-02

EP19724021.1A

patent/EP3899927B1/en

active

Active

2019-05-02

CN201980091601.9A

patent/CN113412515B/en

active

Active

2019-05-02

PCT/US2019/030343

patent/WO2020222846A1/en

not_active

Ceased

Patent Citations (9)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20010029455A1

( en )

2000-03-31

2001-10-11

Chin Jeffrey J.

Method and apparatus for providing multilingual translation over a network

US20030144832A1

( en )

2002-01-16

2003-07-31

Harris Henry M.

Machine translation system

US8583416B2

( en )

2007-12-27

2013-11-12

Fluential, Llc

Robust information extraction from utterances

US20140205974A1

( en )

2011-06-30

2014-07-24

Rosetta Stone, Ltd.

Statistical machine translation framework for modeling phonological errors in computer assisted pronunciation training system

WO2017112813A1

( en )

2015-12-22

2017-06-29

Sri International

Multi-lingual virtual personal assistant

US20180314689A1

( en )

2015-12-22

2018-11-01

Sri International

Multi-lingual virtual personal assistant

US10867136B2

( en )

2016-07-07

2020-12-15

Samsung Electronics Co., Ltd.

Automatic interpretation method and apparatus

US20190138606A1

( en )

2016-07-12

2019-05-09

Huawei Technologies Co., Ltd.

Neural network-based translation method and apparatus

US10891435B1

( en )

2018-02-20

2021-01-12

Interactions Llc

Bootstrapping multilingual natural language understanding via machine translation

Non-Patent Citations (10)

* Cited by examiner, â Cited by third party

Title

Dong, Daxiang, et al. " Multi-task learning for multiple language translation. " Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (vol. 1: Long Papers). 2015. (Year: 2015).

European Patent Office; International Search Report and Written Opinion of Ser. No. PCT/US2019/030343; 18 pages; dated Dec. 3, 2019.

European Patent Office; Invitation to Pay Fees of Ser. No. PCT/US2019/030343; dated Oct. 4, 2019. 13 pages.

Koehn " Neural Machine Translation " Center for Speech and Language Processing Department of Computer Science Johns Hopkins University 2017. 133 pages.

Lee et al., " Fully Character-Level Neural Machine Translation without Explicit Segmentation " Association for Computational Linguistics, vol. 5, pp. 365-378. Published Oct. 2017. 14 pages.

Luong, M. et al., " Multi-Task Sequence to Sequence Learning "; retrieved from Internet: URL:https://arxiv.org/pdf/1511.06114.pdf; dated Mar. 1, 2016. 10 pages.

Schwenk, Holger, and Matthijs Douze. " Learning joint multilingual sentence representations with neural machine translation. " arXiv preprint arXiv:1704.04154 (2017). (Year: 2017).

Shi, Chen, et al. " Knowledge-based semantic embedding for machine translation. " Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (vol. 1: Long Papers). 2016. (Year: 2016).

Song, Linfeng, et al. " Semantic neural machine translation using AMR. " Transactions of the Association for Computational Linguistics 7 (Mar. 1, 2019): 19-31. (Year: 2019).

Wu, Yonghui, et al. " Google's neural machine translation system: Bridging the gap between human and machine translation. " arXiv preprint arXiv:1609.08144 (2016). (Year: 2016).

Cited By (2)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20220327288A1

( en )

2021-04-13

2022-10-13

Microsoft Technology Licensing, Llc

Semantic Parsing of Utterance Using Contractive Paraphrasing

US12314670B2

( en )

2021-04-13

2025-05-27

Microsoft Technology Licensing, Llc

Semantic parsing of utterance using contractive paraphrasing

Also Published As

Publication number

Publication date

WO2020222846A1

( en )

2020-11-05

US20210064828A1

( en )

2021-03-04

CN113412515B

( en )

2025-01-14

EP3899927A1

( en )

2021-10-27

EP3899927B1

( en )

2024-07-24

CN113412515A

( en )

2021-09-17

Adapting automated assistants for use with multiple languages — Google Llc (US11113481B2)

Related documents