ABSTRACT
Abstract
A speech recognition and natural language understanding system performs insertion, deletion, and replacement edits of tokens at positions with low probabilities according to both a forward and a backward statistical language model (SLM) to produce rewritten token sequences. Multiple rewrites can be produced with scores depending on the probabilities of tokens according to the SLMs. The rewritten token sequences can be parsed according to natural language grammars to produce further weighted scores. Token sequences can be rewritten iteratively using a graph-based search algorithm to find the best rewrite. Mappings of input token sequences to rewritten token sequences can be stored in a cache, and searching for a best rewrite can be bypassed by using cached rewrites when present. Analysis of various initial token sequences that produce the same new rewritten token sequence can be useful to improve natural language grammars.
Description
FIELD OF THE INVENTION
The present invention is in the field of speech recognition and natural language understanding.
BACKGROUND
Automatic speech recognition (ASR) is prone to errors.
ASR performs spectral analysis on audio signals and extracts features, from which the process hypothesizes multiple phoneme sequences, each with a score representing the likelihood that it is correct, given the acoustic analysis of the input audio. ASR proceeds to tokenize phoneme sequence hypotheses into token sequence hypotheses according to a dictionary, maintaining a score for each hypothesis. Tokens can be alphabetic words such as English words, logographic characters such as Chinese characters, or discernable elemental units of other types of writing systems. Tokenization is imprecise since, for example, English speakers pronounce the phrases âI screamâ and âice creamâ almost identically. To deal with such ambiguities, ASR systems use the statistics of known frequencies of neighboring word tokens common in the spoken language to hypothesize which of multiple token sequence hypotheses is correct. For example, the word âiceâ frequently follows the word âeatâ, as in âeat ice creamâ, but the word âIâ rarely follows the word âeatâ. Therefore, if the word sequence hypotheses âI screamâ and âice creamâ follow the word âeatâ, then the score of the word sequence hypothesis with âice creamâ increases while the score of the word sequence hypothesis with âI screamâ decreases.
For example, Mandarin Chinese speakers pronounce the phrases â
â and â
â identically. Therefore, speech recognition uses the statistics of known frequencies of neighboring tokens common in the spoken language to hypothesize which of multiple token sequence hypotheses is correct. For example, the word â
â frequently follows the word â
â, as in â
â, but the word â
â rarely follows the word â
â. Therefore, if the word sequence hypotheses â
â and â
â follow the word â
â, then the score of the word sequence hypothesis with â
â increases while the score of the word sequence hypothesis with â
â decreases.
Conventional speech recognition and natural language understanding systems are relatively inaccurate and slow. They can produce transcriptions that are grammatically incorrect. Furthermore, their grammar rules are complex to create and improve. Also, grammars usually do not capture all of the informal and approximate ways that users express themselves, and as a result have insufficient coverage.
SUMMARY OF THE INVENTION
The present invention relates to methods, systems, and software to program systems for rewriting the token sequences that result from speech recognition using a combination of forward and backward SLMs. This can involve computing scores for token sequences according to the combination of probabilities of the tokens within a sequence.
The present invention provides technical improvements to the accuracy of speech recognition, grammatical correctness of transcriptions, and enhancements to natural language grammars.
This can involve editing a token sequence by the deletion, insertion, or replacement of a specific token. A token whose forward and backward probabilities are both low is a candidate for deletion, or for replacement by a new token in the token sequence. An adjacent pair of tokens for which the earlier token has a low score in the backward direction and the later token has a low score in the forward direction is a candidate for deletion of either token, or for insertion of a new token in between.
Choosing a new token for insertion or replacement can involve checking either the forward SLM or backward SLM to create a list of the highest probability tokens or checking both SLMs to find a token with a high probability in both SLMs. In an embodiment that chooses a new token based on probabilities in both SLMs, the best choice is often one that is not the highest on either list, but rather one that is fairly high on both lists.
Any operation of deletion, insertion, or replacement creates a new token sequence, which has a new combined score. Some embodiments produce multiple possible rewritten token sequences, compute scores for each one, and choose to output the one with the best score. Some embodiments perform multiple successive rewrites of a token sequence.
Some embodiments submit token sequences as input to a natural language parser that determines parse scores for token sequences according to a set of grammar rules. In such embodiments, the ultimate choice of a best token sequence (the original or any particular rewrite) is determined by either the grammar parse score or a combination of the grammar parse score and the token sequence probability score. Some such grammar parsers simply produce an output indicating whether the token sequence is valid or invalid according to the grammar. Parsers for stochastic grammars assign probability scores to parses. Semantic parsers use semantic grammars, where the score of a parse may be affected by its semantics.
Some embodiments perform a partial syntactic analysis of token sequences, such as with part of speech tagging and the application of grammar patterns known to be legal within a spoken language.
Some embodiments use one or more of SLMs, syntax checks, and grammar rules that are general to a diverse corpus of expressions generally used in one or more entire human languages. Some use domain specific SLMs, syntax checks, and grammars, which can be more accurate at rewriting token sequences according to particular domains of knowledge.
Some embodiments use entity tagging to replace known specific tokens with special tag tokens that indicate their generic class. An example is to replace each token that is the name of a city with a tag that represents cities in general. This can improve the number of token sequences that satisfy a pattern, thereby making statistical probabilities more accurate.
Some embodiments perform rewrites on token sequences from real-time ASR within a delayed buffer of real-time speech. Some embodiments use larger delay buffers and attempt parsing of complete expression phrases.
Some embodiments store a history cache of tokens used recently, and increase the probability-based score of tokens that are present in the history cache. This improves accuracy by taking advantage of the fact that topic-specific words tend to be spoken multiple times within short periods of conversation.
Some embodiments store, in a cache, both original input token sequences and the chosen one or top few rewrites. This enables completion of future rewrite requests by a lookup in the cache, without the need for a new search of many possible rewrites. Some embodiments store rewritten token sequences and one or more input token sequences that resulted in the rewrite. This can provide useful information, especially when sorted by frequency of occurrence, for grammar developers to create and improve domain-specific grammars.
Whereas speech recognition systems that do not use bidirectional probabilistic rewriting might require relatively sophisticated and complex grammars, by using bidirectional probabilistic rewriting, speech recognition systems are able to expand the coverage of relatively simple grammars by mapping approximate expressions to grammatically correct ones. As a result, less sophisticated grammar developers are able to rely on the speech recognition system employing programmatic rules for bidirectional probabilistic rewriting as a tool supporting the creation of custom domain grammars that meet the requirements of their applications. This has the effect of both higher performance and more accurate speech recognition systems and natural language understanding systems.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 shows a flow diagram of a speech-enabled natural language understanding and processing system according to an embodiment.
FIG. 2 shows typical speech recognition errors in English.
FIG. 3 shows typical speech recognition errors in Mandarin Chinese.
FIG. 4 shows a flow diagram of rewriting according to an embodiment.
FIG. 5A shows a simple example corpus of English expressions.
FIG. 5B shows a forward SLM built according to the corpus of FIG. 5A .
FIG. 5C shows a backward SLM built according to the corpus of FIG. 5A .
FIG. 6A shows a simple example corpus of Mandarin Chinese expressions.
FIG. 6B shows a forward SLM built according to the corpus of FIG. 6A .
FIG. 6C shows a backward SLM built according to the corpus of FIG. 6A .
<figref id
FIELD OF THE INVENTION
The present invention is in the field of speech recognition and natural language understanding.
BACKGROUND
Automatic speech recognition (ASR) is prone to errors.
ASR performs spectral analysis on audio signals and extracts features, from which the process hypothesizes multiple phoneme sequences, each with a score representing the likelihood that it is correct, given the acoustic analysis of the input audio. ASR proceeds to tokenize phoneme sequence hypotheses into token sequence hypotheses according to a dictionary, maintaining a score for each hypothesis. Tokens can be alphabetic words such as English words, logographic characters such as Chinese characters, or discernable elemental units of other types of writing systems. Tokenization is imprecise since, for example, English speakers pronounce the phrases âI screamâ and âice creamâ almost identically. To deal with such ambiguities, ASR systems use the statistics of known frequencies of neighboring word tokens common in the spoken language to hypothesize which of multiple token sequence hypotheses is correct. For example, the word âiceâ frequently follows the word âeatâ, as in âeat ice creamâ, but the word âIâ rarely follows the word âeatâ. Therefore, if the word sequence hypotheses âI screamâ and âice creamâ follow the word âeatâ, then the score of the word sequence hypothesis with âice creamâ increases while the score of the word sequence hypothesis with âI screamâ decreases.
For example, Mandarin Chinese speakers pronounce the phrases â
â and â
â identically. Therefore, speech recognition uses the statistics of known frequencies of neighboring tokens common in the spoken language to hypothesize which of multiple token sequence hypotheses is correct. For example, the word â
â frequently follows the word â
â, as in â
â, but the word â
â rarely follows the word â
â. Therefore, if the word sequence hypotheses â
â and â
â follow the word â
â, then the score of the word sequence hypothesis with â
â increases while the score of the word sequence hypothesis with â
â decreases.
Conventional speech recognition and natural language understanding systems are relatively inaccurate and slow. They can produce transcriptions that are grammatically incorrect. Furthermore, their grammar rules are complex to create and improve. Also, grammars usually do not capture all of the informal and approximate ways that users express themselves, and as a result have insufficient coverage.
SUMMARY OF THE INVENTION
The present invention relates to methods, systems, and software to program systems for rewriting the token sequences that result from speech recognition using a combination of forward and backward SLMs. This can involve computing scores for token sequences according to the combination of probabilities of the tokens within a sequence.
The present invention provides technical improvements to the accuracy of speech recognition, grammatical correctness of transcriptions, and enhancements to natural language grammars.
This can involve editing a token sequence by the deletion, insertion, or replacement of a specific token. A token whose forward and backward probabilities are both low is a candidate for deletion, or for replacement by a new token in the token sequence. An adjacent pair of tokens for which the earlier token has a low score in the backward direction and the later token has a low score in the forward direction is a candidate for deletion of either token, or for insertion of a new token in between.
Choosing a new token for insertion or replacement can involve checking either the forward SLM or backward SLM to create a list of the highest probability tokens or checking both SLMs to find a token with a high probability in both SLMs. In an embodiment that chooses a new token based on probabilities in both SLMs, the best choice is often one that is not the highest on either list, but rather one that is fairly high on both lists.
Any operation of deletion, insertion, or replacement creates a new token sequence, which has a new combined score. Some embodiments produce multiple possible rewritten token sequences, compute scores for each one, and choose to output the one with the best score. Some embodiments perform multiple successive rewrites of a token sequence.
Some embodiments submit token sequences as input to a natural language parser that determines parse scores for token sequences according to a set of grammar rules. In such embodiments, the ultimate choice of a best token sequence (the original or any particular rewrite) is determined by either the grammar parse score or a combination of the grammar parse score and the token sequence probability score. Some such grammar parsers simply produce an output indicating whether the token sequence is valid or invalid according to the grammar. Parsers for stochastic grammars assign probability scores to parses. Semantic parsers use semantic grammars, where the score of a parse may be affected by its semantics.
Some embodiments perform a partial syntactic analysis of token sequences, such as with part of speech tagging and the application of grammar patterns known to be legal within a spoken language.
Some embodiments use one or more of SLMs, syntax checks, and grammar rules that are general to a diverse corpus of expressions generally used in one or more entire human languages. Some use domain specific SLMs, syntax checks, and grammars, which can be more accurate at rewriting token sequences according to particular domains of knowledge.
Some embodiments use entity tagging to replace known specific tokens with special tag tokens that indicate their generic class. An example is to replace each token that is the name of a city with a tag that represents cities in general. This can improve the number of token sequences that satisfy a pattern, thereby making statistical probabilities more accurate.
Some embodiments perform rewrites on token sequences from real-time ASR within a delayed buffer of real-time speech. Some embodiments use larger delay buffers and attempt parsing of complete expression phrases.
Some embodiments store a history cache of tokens used recently, and increase the probability-based score of tokens that are present in the history cache. This improves accuracy by taking advantage of the fact that topic-specific words tend to be spoken multiple times within short periods of conversation.
Some embodiments store, in a cache, both original input token sequences and the chosen one or top few rewrites. This enables completion of future rewrite requests by a lookup in the cache, without the need for a new search of many possible rewrites. Some embodiments store rewritten token sequences and one or more input token sequences that resulted in the rewrite. This can provide useful information, especially when sorted by frequency of occurrence, for grammar developers to create and improve domain-specific grammars.
Whereas speech recognition systems that do not use bidirectional probabilistic rewriting might require relatively sophisticated and complex grammars, by using bidirectional probabilistic rewriting, speech recognition systems are able to expand the coverage of relatively simple grammars by mapping approximate expressions to grammatically correct ones. As a result, less sophisticated grammar developers are able to rely on the speech recognition system employing programmatic rules for bidirectional probabilistic rewriting as a tool supporting the creation of custom domain grammars that meet the requirements of their applications. This has the effect of both higher performance and more accurate speech recognition systems and natural language understanding systems.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 shows a flow diagram of a speech-enabled natural language understanding and processing system according to an embodiment.
FIG. 2 shows typical speech recognition errors in English.
FIG. 3 shows typical speech recognition errors in Mandarin Chinese.
FIG. 4 shows a flow diagram of rewriting according to an embodiment.
FIG. 5A shows a simple example corpus of English expressions.
FIG. 5B shows a forward SLM built according to the corpus of FIG. 5A .
FIG. 5C shows a backward SLM built according to the corpus of FIG. 5A .
FIG. 6A shows a simple example corpus of Mandarin Chinese expressions.
FIG. 6B shows a forward SLM built according to the corpus of FIG. 6A .
FIG. 6C shows a backward SLM built according to the corpus of FIG. 6A .
FIG. 7 shows a simple example natural language domain grammar in English.
FIG. 8 shows a simple example natural language domain grammar in Mandarin Chinese.
FIG. 9 shows a forward SLM built according to a diverse corpus of expressions parsed by the domain grammar of FIG. 7 .
FIG. 10 shows a backward SLM built according to a diverse corpus of expressions parsed by the domain grammar of FIG. 7 .
FIG. 11 shows a flow diagram of rewriting after token tag replacement according to an embodiment.
FIG. 12A shows an example of a correctly recognized expression in English.
FIG. 12B shows an example of a variant of the expression of FIG. 12A with a wrong token error causing low forward and backward probabilities at the replaced token according to an embodiment.
FIG. 12C shows an example of a variant of the expression of FIG. 12A with an extra token error causing low forward and backward probabilities at the inserted token according to an embodiment.
FIG. 12D shows an example of a variant of the expression of FIG. 12A with a missing token error causing a low backward probability on a token followed by a low forward probability on the immediately following token according to an embodiment.
FIG. 12E shows an example of a variant of the expression of FIG. 12A with a repeated token error causing a low backward probability on a token followed by a low forward probability on the immediately following token according to an embodiment.
FIG. 13A shows an example of a correctly recognized expression in Mandarin Chinese.
FIG. 13B shows an example of a variant of the expression of FIG. 13A with a wrong token error causing low forward and backward probabilities at the replaced token according to an embodiment.
FIG. 13C shows an example of a variant of the expression of FIG. 13A with an extra token error causing low forward and backward probabilities at the inserted token according to an embodiment.
FIG. 13D shows an example of a variant of the expression of FIG. 13A with a missing token error causing a low backward probability on a token followed by a low forward probability on the immediately following token according to an embodiment.
FIG. 13E shows an example of a variant of the expression of FIG. 13A with a repeated token error causing a low backward probability on a token followed by a low forward probability on the immediately following token according to an embodiment.
FIG. 14 shows a flow diagram of rewriting using a set of most probable tokens in the forward and the backward SLMs according to an embodiment.
FIG. 15A shows a replacement error in an English expression according to an embodiment.
FIG. 15B shows a set of most likely intended tokens for the error of FIG. 15A using a 2-gram SLM in the forward direction according to an embodiment.
FIG. 15C shows a set of most likely intended tokens for the error of FIG. 15A using a 2-gram SLM in the backward direction with a common token near the highest probability for each direction according to an embodiment.
FIG. 16A shows a replacement error in a Mandarin Chinese expression according to an embodiment.
FIG. 16B shows a set of most likely intended tokens for the error of FIG. 16A using a 2-gram SLM in the forward direction according to an embodiment.
FIG. 16C shows a set of most likely intended tokens for the error of FIG. 16A using a 2-gram SLM in the backward direction with a common token near the highest probability for each direction according to an embodiment.
FIG. 17A shows a replacement error in an English expression according to an embodiment.
FIG. 17B shows a set of most likely intended tokens for the error of FIG. 17A using a 3-gram SLM in the forward direction according to an embodiment.
FIG. 17C shows a set of most likely intended tokens for the error of FIG. 17A using a 3-gram SLM in the backward direction with two common tokens near the highest probability for each direction but in different orders of probability according to an embodiment.
FIG. 17D shows calculations of a rewrite score for replacement of the error token with each of the common tokens with high probability for each SLM according to an embodiment.
FIG. 18A shows a replacement error in an English expression according to an embodiment.
FIG. 18B shows a set of most likely intended tokens for the error of FIG. 18A using a 3-gram SLM in the forward direction according to an embodiment.
FIG. 18C shows a set of most likely intended tokens for the error of FIG. 18A using a 3-gram SLM in the backward direction with two common tokens near the highest probability for each direction but in different orders of probability according to an embodiment.
FIG. 18D shows calculations of a rewrite score for replacement of the error token with each of the common tokens with high probability for each SLM according to an embodiment.
FIG. 19 shows a flow diagram of rewriting using a set of most probable tokens in the forward and the backward SLMs to produce a new rewritten token sequence and a corresponding score according to an embodiment.
FIG. 20 shows a flow diagram of rewriting using a set of most probable tokens in the forward and the backward SLMs to produce a new rewritten token sequence and a corresponding score guided by a diverse corpus SLM according to an embodiment.
FIG. 21 shows a flow diagram of rewriting using a set of most probable tokens in the forward and the backward SLMs to produce a plurality of new rewritten token sequence and corresponding scores and choosing the rewritten token sequence with the highest score according to an embodiment.
FIG. 22 shows a flow diagram of rewriting using syntactic analysis to guide selection of a most probable tokens from sets in the forward and the backward SLMs according to an embodiment.
FIG. 23 shows a flow diagram of rewriting followed by natural language parsing using a domain grammar according to an embodiment.
FIG. 24 shows example grammar parse scores for different token insertions in English according to an embodiment.
FIG. 25 shows example grammar parse scores for different token insertions in Mandarin Chinese according to an embodiment.
FIG. 26 shows a flow diagram of rewriting to produce a plurality of new rewritten token sequences and corresponding scores followed by natural language parsing using a domain grammar to choose a highest scoring rewritten sequence according to an embodiment.
FIG. 27 shows a flow diagram of rewriting followed by token history caching to automate rewriting according to an embodiment.
FIG. 28 shows the contents of a token history cache for translating input token sequences to rewritten token sequences according to an embodiment.
FIG. 29 shows the contents of a token history cache for translating rewritten token sequences to input token sequences according to an embodiment.
FIG. 30 shows user interaction with a server-based speech-enabled natural language understanding and processing system according to an embodiment.
FIG. 31A shows a disk non-transitory computer readable medium according to an embodiment.
FIG. 31B shows a flash memory non-transitory computer readable medium according to an embodiment.
FIG. 31C shows a system-on-chip processor according to an embodiment.
FIG. 31D shows a system-on-chip processor medium according to an embodiment.
FIG. 32 shows a block diagram of a system-on-chip according to an embodiment.
FIG. 33 shows a server system according to an embodiment.
FIG. 34 shows a block diagram of a server system according to an embodiment.
DETAILED DESCRIPTION
Introduction
A statistical language model (SLM) captures the statistics of neighboring words in a given corpus of expressions. Applying a SLM to token sequence hypotheses significantly improves the accuracy of ASR systems.
A forward SLM represents the conditional probability of the next token given one or a sequence of prior tokens. A backward SLM represents the conditional probability of an immediately prior token given one or a sequence of following tokens. Any given pair of tokens can have very different probabilities in each of the forward and backward direction. Consider the famous American baseball player, Joe DiMaggio. In a backward SLM, there is a very high probability that the word preceding DiMaggio is Joe because there are very few famous people with the family name DiMaggio. However, in a forward SLM the probability is fairly low for the word DiMaggio following the word Joe because there are many famous people named Joe.
Consider the famous leader during China's Three Kingdom's period,
. In a forward SLM, there is a very high probability that the word following
is
because there are very few famous people with the family name
. However, in a backward SLM the probability is fairly low for the word
preceding the word
because there are many famous people named
.
Expressions are sequences of tokens. In various embodiments, tokens are alphabetic words such as in the English language or logograms such as Chinese language written characters. In some embodiments, tokens can represent multiple words that represent a single concept, such as the two words, âNewâ and âYorkâ that together represent a city or â
â and â
â that together represent the city. In some embodiments, different tokens represent different meanings of words that have the same spellings such as the âtireâ on a car and to âtireâ by becoming sleepy or â
â being angry and â
â being vital and lively.
FIG. 1 shows a high-level view of a system for speech-enabled natural language processing. A speech recognition module 11 receives speech audio from a person issuing a spoken expression. The speech recognition module 11 produces a sequence of tokens. Some embodiments produce multiple alternative token sequences when it is not perfectly clear what the person said. Some such embodiments give probability weights to the alternatives.
Some embodiments receive token sequences from a person by means other than speech audio, such as typing on a keyboard, tapping on a touch screen, eye movement, gesturing with arms, hand or fingers, or neural electrical signaling.
A rewriting module 12 receives the token sequence as an input token sequence. It applies programmatic rules, as described below, to the token sequence to perform rewriting. In the case of multiple token sequences it processes each input token sequence either sequentially or in parallel. The rewrite module uses forward SLM 13 and backward SLM 14 to produce one or more new token sequences. Some embodiments produce a corresponding score for each new token sequence, the score indicative of the likelihood that the rewritten token sequence is what the person intended to express. Some embodiments feed new token sequences back into rewriting module 12 and reprocess them in an attempt to produce even better rewritten token sequences. This is useful if, for example, token sequences have multiple errors.
Some embodiments use a natural language parsing module 15 . The parsing module receives the new token sequences or, in some embodiments, multiple new token sequences, and parses the token sequence(s) according to a domain- specific grammar 16 to produce natural language interpretations and parse scores corresponding to the interpretations. Some embodiments use multiple domain- specific grammars 16 and parse new token sequences according to each grammar 16 . Some embodiments apply the rewrite score from rewriting module 12 as weights in the parsing. Some embodiments weight the parse score by the rewrite score to produce a best estimate that the interpretation is what the person intended.
Some embodiments feed the parse score back into the rewriting module 12 , which uses the parse score to select one or more rewrites that are most likely correct among numerous possible rewrites.
Ultimately, a processing module 17 receives a highest-scoring interpretation, determines its intent and processes it accordingly, and produces a most-likely useful result. The processing module 17 produces the result and provides it as the system output to a user.
By using a rewriting module 12 embodiments are much more likely to determine the correct token sequence of the person's expression and therefore produce a correct interpretation and useful result. By further using rewrite score feedback and parse score feedback, embodiments are even more likely to determine the correct token sequence of the person's expression and therefore produce a correct interpretation and useful result. As a result, such embodiments increase the linguistic coverage of grammars (i.e., the portion of a corpus that is recognized by the grammars) by transforming erroneously recognized or grammatically incorrect input token sequences into grammatically correct forms, thereby effectively allowing the grammar to support flawed variants and perform error correction.
Bidirectional probabilistic natural language rewriting and selection expand the universe of expressions that grammars can interpret, which is effectively a form of grammar rule compression. It allows small grammars to support the range of expressions otherwise requiring much larger grammars. Correspondingly, it allows the same size of grammar to interpret a far wider range of human expressions.
Systems for automatic machine translation between human languages can also gain improved translation accuracy. Some such embodiments apply rewrites on ASR results before they are input to a translation module. The rewriting reduces the universe of expressions that might need to be translated, thereby allowing simpler translation rules to achieve the necessary requirement for translation accuracy.
FIG. 30 shows the elements of FIG. 1 in a server-implemented system that receives expressions from a user through a client coupled to the server through a network, as is typical for virtual assistants implemented as mobile phone apps or in-home devices. Some embodiments are self-contained and work independently of a network, as is typical for automotive systems, high-security systems, and extremely low-power systems.
FIG. 2 shows types and examples of English word token sequences for which a person intended to make a weather-related query, but which failed to parse correctly by a weather-related grammar.
FIG. 3 shows types and examples of Mandarin Chinese character token sequences for which a person intended to make a weather-related query, but which failed to parse correctly by a weather-related grammar.
Each example sequence in FIG. 2 and FIG. 3 is caused by one or more of the following four types of errors: wrong token, extra token, missing token, and repeated token. Each type of error can be corrected by one of the following three types of edits: token insertion, token deletion, and token replacement. Some embodiments do not have a token replacement type of edit, but achieve the same effect with a combination of a token deletion and a token insertion. Each of token insertion and token replacement creates a new token that was not present in the input token sequence. A system of programmatic rules, as described below, determines the choice of type of edit and position for making an edit for token rewriting based on conditional probabilities of tokens in view of preceding and following tokens.
For token replacement, the position of the edit is at the low probability token to be replaced. For token insertion, the position of the edit is between two tokens, the earlier having a low backward probability and the later having a low forward probability. For token deletion, the position of the edit can be at a token with a low probability in both directions, or at either of the tokens for which the earlier has a low backward probability and the later has a low forward probability.
Some embodiments perform multiple token replacements in a single rewrite. Some embodiments perform a single token replacement in any particular rewrite. Some such embodiments perform multiple rewrites iteratively.
FIG. 4 shows a token sequence rewrite flow for token replacement. A forward token probability module 41 receives an input token sequence and uses a forward SLM 42 to produce a forward probability for each token in the sequence. A backward probability module 43 receives the input token sequence and uses a backward SLM 44 to produce a backward probability for each token in the sequence. An edit module 45 receives the forward and backward probability sequences, finds a token position with a low probability in both the forward and the backward direction, and replaces that token with another to produce a new rewritten token sequence. Some embodiments as in FIG. 4 perform token replacement conditionally, only if it at least one token has a sufficiently low probability.
Some embodiments operate as in FIG. 4 but instead of performing a replacement in the edit module 45 , perform a deletion of the token with the low probabilities.
Some embodiments operate as in FIG. 4 but instead of performing a replacement in the edit module 45 , identify a pair of adjacent tokens for which the earlier one has a low probability in the backward direction and the later has a low probability in the forward direction. Some such embodiments insert a new token between the tokens in the pair. Some such embodiments replace either the first or the second token of the pair.
Thresholds
Some embodiments determine a low probability by comparing probabilities to thresholds. Some embodiments simply rewrite at the position with the lowest probability, without using thresholds. In some embodiments, thresholds are different in the forward and backward direction. In some embodiments, the thresholds depend on probabilities of neighboring tokens.
A good threshold level is one at a point in the probability distribution of all tokens in a corpus that allows a large majority of tokens to pass. A higher threshold causes more aggressive rewriting. Aggressive rewriting can correct more egregious token errors, but risks rewriting input token sequences to new ones that incorrectly represent the speaker's intent.
The distribution of forward token probabilities at the beginning of a token sequence has a higher entropy than the distribution after several tokens into the sequence from the beginning. Similarly, the backward token probabilities at the end of a token sequence are much more widely distributed than after several tokens into the sequence from the end. Some embodiments use a different threshold for the first or first few tokens in a sequence than for other tokens in the sequence.
Some embodiments, rather than using thresholds, choose for replacement the token with the lowest probability in either the forward, backward, or a combination of both the forward and backward directions.
Building SLMs
Choosing the most useful rewrites depends on having accurate SLMs. An SLM is most accurate if built from a corpus of expressions of the same type as the expressions to rewrite. For example, a corpus of expressions in Twitter⢠tweets has very different SLM probabilities than a corpus of expressions from articles in the New York Times⢠newspaper. Likewise, a corpus of expressions for a virtual assistant in general has different SLM probabilities than a corpus of expressions specific to a weather domain.
To enhance accuracy, the corpus used for building an SLM should contain only correct expressionsâthat is, expressions that an appropriate natural language grammar can parse or ones that score highly when parsed by a stochastic grammar. Furthermore, building an SLM using a corpus of expressions that refer to different specific entities (such as place names) creates many low probability sequences, one for each different specific entity. By replacing specific entity tokens with generic tags, the resulting SLM will have much higher probabilities for the same expressions with tags substituted for entities. To take advantage of an SLM built after tag substitution, it is useful for the token rewriting system to perform corresponding tag substitutions before performing token sequence rewriting.
FIG. 5A and FIG. 6A show small corpuses for purposes of example. Each corpus has five expressions, each having two tokens. FIG. 5B and FIG. 6B show the forward SLMs built from the corpuses. Each row corresponds to a sequence of most recent tokens (supporting only 1-token sequences). Columns correspond to the token following the sequence. Cells indicate probabilities. For example, in the SLM of FIG. 5B , after the token âwhoâ there is ½ chance that the next token is âisâ and in the SLM of FIG. 6B , after the token â
â there is a ¼ chance that the next token is â
â. FIG. 5C and FIG. 6C show the backward SLMs built from the corpuses. Each row corresponds to a sequence of following tokens (supporting only 1-token sequences). Columns correspond to the token preceding the sequence. For example, the in the SLM of FIG. 5C , for the token âisâ there is a ¼ chance that the preceding token is âwhoâ and in the SLM of FIG. 6C , for the token â
â there is a ¼ chance that the next token is â
â. Note that forward and backward SLMs, represented as matrices, are not transposes of each other.</di
CLAIMS
Claims ( 19 )
What is claimed is:
1. A computer-implemented method of rewriting an input token sequence of a user query when providing query results to the user, the method comprising:
receiving, over a computer network from an application executing on a remote client device, a user query from a system user, the query comprising the input token sequence;
determining forward probabilities, according to a forward statistical language model, for a plurality of tokens in the input token sequence;
determining backward probabilities, according to a backward statistical language model, for a plurality of tokens in the input token sequence;
inserting a new token at a location after a first token having a low backward probability and before an adjacent second token having a low forward probability to create a new rewritten token sequence;
processing the new rewritten token sequence to produce a result; and
providing a response, indicating the result of the processing, to the system user.
2. A computer-implemented method of rewriting an input token sequence, the method comprising:
determining forward probabilities, according to a forward statistical language model, for a plurality of tokens in the input token sequence;
determining backward probabilities, according to a backward statistical language model, for a plurality of tokens in the input token sequence;
computing a probability score for each of the plurality of tokens based on a lowest one of the forward probabilities and a lowest one of the backward probabilities; and
replacing, with a new token, a token having a lowest probability score among the computed probability scores for the plurality of tokens to create a new rewritten token sequence.
3. A computer-implemented method of rewriting an input token sequence, the method comprising:
determining forward probabilities, according to a forward statistical language model (SLM), for a plurality of tokens in the input token sequence;
determining backward probabilities, according to a backward statistical language model, for a plurality of tokens in the input token sequence; and
replacing a suspicious token having a forward probability below a first threshold and a backward probability below a second threshold with a new token to create a new rewritten token sequence.
4. The method of claim 3 , further comprising:
substituting a tag for at least one token in the input token sequence prior to determining probabilities.
5. The method of claim 3 , further comprising:
choosing, as the new token, one that is both in a list of highest probability tokens according to the forward SLM and a list of highest probability tokens according to the backward SLM.
6. The method of claim 5 , further comprising:
performing a syntactic analysis of the input token sequence according to syntax rules; and
restricting the choosing to only tokens that are syntactically legal in context of neighboring tokens according to the syntax rules.
7. The method of claim 5 , further comprising:
computing, for the new rewritten token sequence, a rewrite score that depends at least on the probability of the new token in the forward SLM and the probability of the new token in the backward SLM.
8. The method of claim 7 , further comprising:
scaling the rewrite score based on the probability of the new token in a diverse corpus SLM that was built from expressions related to a wide variety of topics.
9. The method of claim 5 , further comprising:
replacing the suspicious token with an alternative new token to create an alternative rewritten token sequence;
computing, for the alternative rewritten token sequence, an alternative score as a combination of both the probability of the alternative new token in the forward SLM and the probability of the alternative new token in the backward SLM; and
choosing whichever of the new rewritten token sequence and the alternative rewritten token sequence has a higher score.
10. The method of claim 5 , further comprising:
maintaining a token buffer of hypothesized tokens from recent continuous speech,
wherein the input token sequence is a sequence of tokens in the token buffer.
11. The method of claim 3 , further comprising:
storing a history cache of tokens present in recent token sequences;
for replacing the suspicious token, choosing the new token from each of a list of forward most probable tokens and a list of backward most probable tokens; and
increasing the probability score of at least one token that is present in the history cache.
12. The method of claim 3 , further comprising:
parsing the new rewritten token sequence according to a grammar using a natural language parser to produce a parse score.
13. The method of claim 12 , further comprising:
replacing an alternative suspicious token different from the suspicious token to create an alternative rewritten token sequence;
parsing the alternative rewritten token sequence according to the grammar using the natural language parser to produce an alternative parse score; and
choosing whichever of the new rewritten token sequence and the alternative rewritten token sequence has a higher parse score.
14. The method of claim 12 , further comprising:
replacing the suspicious token with an alternative new token to create an alternative rewritten token sequence;
parsing the alternative rewritten token sequence according to the grammar using the natural language parser to produce an alternative parse score; and
choosing whichever of the new rewritten token sequence and the alternative rewritten token sequence has a higher parse score.
15. The method of claim 12 , further comprising:
using a tree-based algorithm to iteratively perform rewrites and compute scores for each rewrite to produce a set of rewrites from which to choose one with a best score.
16. The method of claim 3 , further comprising:
storing the input token sequence in a cache;
storing the new rewritten token sequence in the cache in association with the input token sequence; and
searching the cache for the input token sequence.
17. The method of claim 16 , further comprising:
analyzing the cache to identify, for the new rewritten token sequence, a most frequent input token sequence that was rewritten to the new rewritten token sequence.
18. The method of claim 17 , further comprising:
creating a grammar rule to cover the most frequent input token sequence that was rewritten to the new rewritten token sequence.
19. The method of claim 17 , further comprising:
adapting a grammar rule to cover the most frequent input token sequence that was rewritten to the new rewritten token sequence.
US15/726,394
2017-10-06
2017-10-06
Bidirectional probabilistic natural language rewriting and selection
Active
2038-06-17
US10599645B2
( en )
Priority Applications (3)
Application Number
Priority Date
Filing Date
Title
US15/726,394
US10599645B2
( en )
2017-10-06
2017-10-06
Bidirectional probabilistic natural language rewriting and selection
CN201811151807.XA
CN109635270B
( en )
2017-10-06
2018-09-29
Bidirectional probabilistic natural language rewrite and selection
JP2018189730A
JP6675463B2
( en )
2017-10-06
2018-10-05
Bidirectional stochastic rewriting and selection of natural language
Applications Claiming Priority (1)
Application Number
Priority Date
Filing Date
Title
US15/726,394
US10599645B2
( en )
2017-10-06
2017-10-06
Bidirectional probabilistic natural language rewriting and selection
Publications (2)
Publication Number
Publication Date
US20190108257A1
US20190108257A1 ( en )
2019-04-11
US10599645B2
true
US10599645B2 ( en )
2020-03-24
Family
ID=65992537
Family Applications (1)
Application Number
Title
Priority Date
Filing Date
US15/726,394
Active
2038-06-17
US10599645B2
( en )
2017-10-06
2017-10-06
Bidirectional probabilistic natural language rewriting and selection
Country Status (3)
Country
Link
US
( 1 )
US10599645B2
( en )
JP
( 1 )
JP6675463B2
( en )
CN
( 1 )
CN109635270B
( en )
Cited By (1)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US11249756B2
( en )
*
2019-10-14
2022-02-15
Samsung Electronics Co., Ltd.
Natural language processing method and apparatus
Families Citing this family (23)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
CN109325227A
( en )
*
2018-09-14
2019-02-12
å京åèè·³å¨ç½ç»ææ¯æéå ¬å¸
Method and apparatus for generating amendment sentence
US11437025B2
( en )
2018-10-04
2022-09-06
Google Llc
Cross-lingual speech recognition
CN112151024B
( en )
*
2019-06-28
2023-09-22
声é³çæå ¬å¸
Method and apparatus for generating edited transcription of speech audio
US11205052B2
( en )
*
2019-07-02
2021-12-21
Servicenow, Inc.
Deriving multiple meaning representations for an utterance in a natural language understanding (NLU) framework
US11886461B2
( en )
*
2019-07-31
2024-01-30
Salesforce, Inc.
Machine-learnt field-specific standardization
CN110660384B
( en )
*
2019-10-14
2022-03-22
å èå¤å·¥ä¸å¤§å¦
Mongolian special-shaped homophone acoustic modeling method based on end-to-end
US11276391B2
( en )
*
2020-02-06
2022-03-15
International Business Machines Corporation
Generation of matched corpus for language model training
US11373657B2
( en )
*
2020-05-01
2022-06-28
Raytheon Applied Signal Technology, Inc.
System and method for speaker identification in audio data
US11315545B2
( en )
*
2020-07-09
2022-04-26
Raytheon Applied Signal Technology, Inc.
System and method for language identification in audio data
US12020697B2
( en )
2020-07-15
2024-06-25
Raytheon Applied Signal Technology, Inc.
Systems and methods for fast filtering of audio keyword search
US12387720B2
( en )
2020-11-20
2025-08-12
SoundHound AI IP, LLC.
Neural sentence generator for virtual assistants
US11489793B2
( en )
2020-11-22
2022-11-01
International Business Machines Corporation
Response qualification monitoring in real-time chats
CN112528980B
( en )
*
2020-12-16
2022-02-15
å京åå®ä¿¡æ¯ææ¯æéå ¬å¸
OCR recognition result correction method and terminal and system thereof
US20220284193A1
( en )
*
2021-03-04
2022-09-08
Tencent America LLC
Robust dialogue utterance rewriting as sequence tagging
US11847111B2
( en )
*
2021-04-09
2023-12-19
Bitdefender IPR Management Ltd.
Anomaly detection systems and methods
US11711469B2
( en )
*
2021-05-10
2023-07-25
International Business Machines Corporation
Contextualized speech to text conversion
CN113869069B
( en )
*
2021-09-10
2024-08-06
å¦é¨å¤§å¦
Machine translation method based on decoding path dynamic selection of translation tree structure
WO2023113784A1
( en )
*
2021-12-14
2023-06-22
Google Llc
Lattice speech corrections
US12223948B2
( en )
*
2022-02-03
2025-02-11
Soundhound, Inc.
Token confidence scores for automatic speech recognition
CN115064170B
( en )
*
2022-08-17
2022-12-13
广å·å°é¹æ±½è½¦ç§ææéå ¬å¸
Voice interaction method, server and storage medium
US12394411B2
( en )
2022-10-27
2025-08-19
SoundHound AI IP, LLC.
Domain specific neural sentence generator for multi-domain virtual assistants
US20250045523A1
( en )
*
2023-08-02
2025-02-06
Mediatek Inc.
Execution Methods of a Machine Learning Model
CN120236568A
( en )
*
2023-12-28
2025-07-01
éç¨æ±½è½¦ç¯çç§æè¿ä½æéè´£ä»»å ¬å¸
Speech recognition system based on occupant intention and action
Citations (2)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US20060224384A1
( en )
*
2005-03-31
2006-10-05
International Business Machines Corporation
System and method for automatic speech recognition
US20080270110A1
( en )
*
2007-04-30
2008-10-30
Yurick Steven J
Automatic speech recognition with textual content input
Family Cites Families (7)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US6848080B1
( en )
*
1999-11-05
2005-01-25
Microsoft Corporation
Language input architecture for converting one text form to another text form with tolerance to spelling, typographical, and conversion errors
US7822597B2
( en )
*
2004-12-21
2010-10-26
Xerox Corporation
Bi-dimensional rewriting rules for natural language processing
US9552355B2
( en )
*
2010-05-20
2017-01-24
Xerox Corporation
Dynamic bi-phrases for statistical machine translation
US8762156B2
( en )
*
2011-09-28
2014-06-24
Apple Inc.
Speech recognition repair using contextual information
CN103198149B
( en )
*
2013-04-23
2017-02-08
ä¸å½ç§å¦é¢è®¡ç®ææ¯ç 究æ
Method and system for query error correction
CN104157285B
( en )
*
2013-05-14
2016-01-20
è ¾è®¯ç§æï¼æ·±å³ï¼æéå ¬å¸
Audio recognition method, device and electronic equipment
CN105912521A
( en )
*
2015-12-25
2016-08-31
ä¹è§è´æ°çµåç§æï¼å¤©æ´¥ï¼æéå ¬å¸
Method and device for parsing voice content
2017
2017-10-06
US
US15/726,394
patent/US10599645B2/en
active
Active
2018
2018-09-29
CN
CN201811151807.XA
patent/CN109635270B/en
active
Active
2018-10-05
JP
JP2018189730A
patent/JP6675463B2/en
active
Active
Patent Citations (2)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US20060224384A1
( en )
*
2005-03-31
2006-10-05
International Business Machines Corporation
System and method for automatic speech recognition
US20080270110A1
( en )
*
2007-04-30
2008-10-30
Yurick Steven J
Automatic speech recognition with textual content input
Non-Patent Citations (13)
* Cited by examiner, â Cited by third party
Title
Ahmad Emami, et al, A Neural Syntactic Language Model, Machine Learning, 60, 195-227, 2005, Springer Science + Business Media, Inc.
Ariya Rastrow, Practical and Effcient Incorporation of Syntactic Features into Statistical Language Models, PhD thesis, The Johns Hopkins University, May 2012.
John Lee, et al, Automatic Grammar Correction for Second-Language Learners, INTERSPEECH 2006-ICSLP, Sep. 17-21, 2006, Pittsburgh, Pennsylvania.
John Lee, et al, Automatic Grammar Correction for Second-Language Learners, INTERSPEECH 2006âICSLP, Sep. 17-21, 2006, Pittsburgh, Pennsylvania.
John Sie Yuen Lee, Automatic Correction of Grammatical Errors in Non-native English Text, PhD thesis, 2009 Massachusetts Institute of Technology.
Matthew E. Peters, et al, Semi-supervised sequence tagging with bidirectional language models, Apr. 29, 2017.
Richard Schwartz, et al, Efficient, High-Performance Algorithms for N-Best Search, Jun. 1990.
Stephanie Seneff, et al, Automatic Induction of N-Gram Language Models from a Natural Language Grammar, Eighth European Conference on Speech Communication and Technology. 2003.
Steve Austin, et al, Toward a Real-Time Spoken Language System Using Commercial Hardware, 1990.
Tim Paek, et al, Rapidly Deploying Grammar-Based Speech Applications with Active Learning and Back-off Grammars, Jan. 1, 2008, Microsoft Research.
Y. Albert Park, et al, Automated Whole Sentence Grammar Correction Using a Noisy Channel Model, Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, pp. 934-944, Portland, Oregon, Jun. 19-24, 2011.
Yangfeng Ji, Document Context Language Models, International Conference on Learning Representations (ICLR), Feb. 21, 2016.
Youssef Bassil, et al, Post-Editing Error Correction Algorithm for Speech Recognition using Bing Spelling Suggestion, (IJACSA) International Journal of Advanced Computer Science and Applications, vol. 3, No. 2, 2012.
Cited By (1)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US11249756B2
( en )
*
2019-10-14
2022-02-15
Samsung Electronics Co., Ltd.
Natural language processing method and apparatus
Also Published As
Publication number
Publication date
JP6675463B2
( en )
2020-04-01
CN109635270A
( en )
2019-04-16
US20190108257A1
( en )
2019-04-11
CN109635270B
( en )
2023-03-07
JP2019070799A
( en )
2019-05-09
Similar Documents
Publication
Publication Date
Title
US20190108257A1
( en )
2019-04-11
Bidirectional probabilistic natural language rewriting and selection
CN112599128B
( en )
2024-06-11
A speech recognition method, device, equipment and storage medium
CN109754809B
( en )
2021-02-09
Voice recognition method and device, electronic equipment and storage medium
US8688435B2
( en )
2014-04-01
Systems and methods for normalizing input media
US8719021B2
( en )
2014-05-06
Speech recognition dictionary compilation assisting system, speech recognition dictionary compilation assisting method and speech recognition dictionary compilation assisting program
KR101682207B1
( en )
2016-12-12
Apparatus and method for decoding using joint tokenization and translation
CN108899013B
( en )
2023-04-18
Voice search method and device and voice recognition system
PÄiÅ et al.
2022
Capitalization and punctuation restoration: a survey
CN112489655B
( en )
2024-04-19
Method, system and storage medium for correcting voice recognition text error in specific field
CN114154487B
( en )
2026-04-07
Automatic text correction methods, apparatus, electronic devices and storage media
CN111326144A
( en )
2020-06-23
Voice data processing method, device, medium and computing equipment
US12080275B2
( en )
2024-09-03
Automatic learning of entities, words, pronunciations, and parts of speech
CN106354716A
( en )
2017-01-25
Method and device for converting text
Kirchhoff et al.
2002
Novel speech recognition models for Arabic
CN113225612A
( en )
2021-08-06
Subtitle generating method and device, computer readable storage medium and electronic equipment
US20130238321A1
( en )
2013-09-12
Dialog text analysis device, method and program
CN110826301B
( en )
2023-12-26
Punctuation mark adding method, punctuation mark adding system, mobile terminal and storage medium
Palmer et al.
2000
Robust information extraction from automatically generated speech transcriptions
Ratnam et al.
2024
Phonogram-based automatic typo correction in malayalam social media comments
Fenogenova et al.
2016
A general method applicable to the search for anglicisms in russian social network texts
Gaur et al.
2022
Multilingual second-pass rescoring for automatic speech recognition systems
CN116978367A
( en )
2023-10-31
Speech recognition method, device, electronic equipment and storage medium
Béchet
2011
Named entity recognition
JP2010044171A
( en )
2010-02-25
Subtitle output device, subtitle output method and program
Donaj et al.
2014
Manual sorting of numerals in an inflective language for language modelling
Legal Events
Date
Code
Title
Description
2017-10-06
FEPP
Fee payment procedure
Free format text : ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY
2017-10-27
AS
Assignment
Owner name : SOUNDHOUND, INC., CALIFORNIA
Free format text : ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:LEFEBURE, LUKE;SINGH, PRANAV;REEL/FRAME:043969/0801
Effective date : 20171004
2017-11-26
STPP
Information on status: patent application and granting procedure in general
Free format text : DOCKETED NEW CASE - READY FOR EXAMINATION
2019-10-11
STPP
Information on status: patent application and granting procedure in general
Free format text : NON FINAL ACTION MAILED
2019-11-12
STPP
Information on status: patent application and granting procedure in general
Free format text : NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS
2020-01-17
STPP
Information on status: patent application and granting procedure in general
Free format text : NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS
2020-02-20
STPP
Information on status: patent application and granting procedure in general
Free format text : PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED
2020-03-04
STCF
Information on status: patent grant
Free format text : PATENTED CASE
2021-06-18
AS
Assignment
Owner name : SOUNDHOUND, INC., CALIFORNIA
Free format text : SECURITY INTEREST;ASSIGNOR:OCEAN II PLO LLC, AS ADMINISTRATIVE AGENT AND COLLATERAL AGENT;REEL/FRAME:056627/0772
Effective date : 20210614
2023-04-12
AS
Assignment
Owner name : OCEAN II PLO LLC, AS ADMINISTRATIVE AGENT AND COLLATERAL AGENT, CALIFORNIA
Free format text : CORRECTIVE ASSIGNMENT TO CORRECT THE COVER SHEET PREVIOUSLY RECORDED AT REEL: 056627 FRAME: 0772. ASSIGNOR(S) HEREBY CONFIRMS THE SECURITY INTEREST;ASSIGNOR:SOUNDHOUND, INC.;REEL/FRAME:063336/0146
Effective date : 20210614
2023-04-17
AS
Assignment
Owner name : ACP POST OAK CREDIT II LLC, TEXAS
Free format text : SECURITY INTEREST;ASSIGNORS:SOUNDHOUND, INC.;SOUNDHOUND AI IP, LLC;REEL/FRAME:063349/0355
Effective date : 20230414
2023-04-19
AS
Assignment
Owner name : SOUNDHOUND, INC., CALIFORNIA
Free format text : RELEASE BY SECURED PARTY;ASSIGNOR:OCEAN II PLO LLC, AS ADMINISTRATIVE AGENT AND COLLATERAL AGENT;REEL/FRAME:063380/0625
Effective date : 20230414
Owner name : SOUNDHOUND, INC., CALIFORNIA
Free format text : RELEASE OF SECURITY INTEREST;ASSIGNOR:OCEAN II PLO LLC, AS ADMINISTRATIVE AGENT AND COLLATERAL AGENT;REEL/FRAME:063380/0625
Effective date : 20230414
2023-04-21
AS
Assignment
Owner name : SOUNDHOUND, INC., CALIFORNIA
Free format text : RELEASE BY SECURED PARTY;ASSIGNOR:FIRST-CITIZENS BANK & TRUST COMPANY, AS AGENT;REEL/FRAME:063411/0396
Effective date : 20230417
Owner name : SOUNDHOUND, INC., CALIFORNIA
Free format text : RELEASE OF SECURITY INTEREST;ASSIGNOR:FIRST-CITIZENS BANK & TRUST COMPANY, AS AGENT;REEL/FRAME:063411/0396
Effective date : 20230417
2023-06-23
AS
Assignment
Owner name : SOUNDHOUND AI IP HOLDING, LLC, CALIFORNIA
Free format text : ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:SOUNDHOUND, INC.;REEL/FRAME:064083/0484
Effective date : 20230510
2023-06-27
AS
Assignment
Owner name : SOUNDHOUND AI IP, LLC, CALIFORNIA
Free format text : ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:SOUNDHOUND AI IP HOLDING, LLC;REEL/FRAME:064205/0676
Effective date : 20230510
2023-09-19
MAFP
Maintenance fee payment
Free format text : PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY
Year of fee payment : 4
2024-06-11
AS
Assignment
Owner name : SOUNDHOUND AI IP, LLC, CALIFORNIA
Free format text : RELEASE BY SECURED PARTY;ASSIGNOR:ACP POST OAK CREDIT II LLC, AS COLLATERAL AGENT;REEL/FRAME:067698/0845
Effective date : 20240610
Owner name : SOUNDHOUND, INC., CALIFORNIA
Free format text : RELEASE BY SECURED PARTY;ASSIGNOR:ACP POST OAK CREDIT II LLC, AS COLLATERAL AGENT;REEL/FRAME:067698/0845
Effective date : 20240610
Owner name : SOUNDHOUND, INC., CALIFORNIA
Free format text : RELEASE OF SECURITY INTEREST;ASSIGNOR:ACP POST OAK CREDIT II LLC, AS COLLATERAL AGENT;REEL/FRAME:067698/0845
Effective date : 20240610
Owner name : SOUNDHOUND AI IP, LLC, CALIFORNIA
Free format text : RELEASE OF SECURITY INTEREST;ASSIGNOR:ACP POST OAK CREDIT II LLC, AS COLLATERAL AGENT;REEL/FRAME:067698/0845
Effective date : 20240610
2024-08-09
AS
Assignment
Owner name : MONROE CAPITAL MANAGEMENT ADVISORS, LLC, AS COLLATERAL AGENT, ILLINOIS
Free format text : SECURITY INTEREST;ASSIGNOR:SOUNDHOUND, INC.;REEL/FRAME:068526/0413
Effective date : 20240806
2024-12-03
AS
Assignment
Owner name : SOUNDHOUND, INC., CALIFORNIA
Free format text : TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS;ASSIGNOR:MONROE CAPITAL MANAGEMENT ADVISORS, LLC, AS COLLATERAL AGENT;REEL/FRAME:069480/0312
Effective date : 20241201