ABSTRACT
Abstract
The intelligent automated assistant system engages with the user in an integrated, conversational manner using natural language dialog, and invokes external services when appropriate to obtain information or perform various actions. The system can be implemented using any of a number of different platforms, such as the web, email, smartphone, and the like, or any combination thereof. In one embodiment, the system is based on sets of interrelated domains and tasks, and employs additional functionally powered by external services with which the system can interact.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
This application is a continuation of U.S. application Ser. No. 16/879,643, entitled âTask Flow Identification Based on User Intent,â filed May 20, 2020, (now granted as U.S. Pat. No. 11,423,886), which is a continuation of U.S. application Ser. No. 15/394,162, entitled âTask Flow Identification Based on User Intent,â filed Dec. 29, 2016, (now granted as U.S. Pat. No. 10,706,841), which is a continuation of U.S. application Ser. No. 13/492,809, entitled âIntelligent Automated Assistant,â filed Jun. 9, 2012 (now granted as U.S. Pat. No. 9,548,050), which is a continuation of U.S. application Ser. No. 12/987,982, entitled âIntelligent Automated Assistant,â filed Jan. 10, 2011 (now granted as U.S. Pat. No. 9,318,108), which claims the benefit of U.S. Provisional Patent Application Ser. No. 61/295,774, entitled âIntelligent Automated Assistantâ, filed Jan. 18, 2010, each of which are hereby incorporated by reference in their entirety for all purposes.
This application is further related to U.S. patent application Ser. No. 11/518,292 for âMethod and Apparatus for Building an Intelligent Automated Assistantâ, filed Sep. 8, 2006, which is incorporated herein by reference.
This application is further related to U.S. Provisional Patent Application Ser. No. 61/186,414 for âSystem and Method for Semantic Auto-Completionâ, filed Jun. 12, 2009, which is incorporated herein by reference.
FIELD OF THE INVENTION
The present invention relates to intelligent systems, and more specifically for classes of applications for intelligent automated assistants.
BACKGROUND OF THE INVENTION
Today's electronic devices are able to access a large, growing, and diverse quantity of functions, services, and information, both via the Internet and from other sources. Functionality for such devices is increasing rapidly, as many consumer devices, smartphones, tablet computers, and the like, are able to run software applications to perform various tasks and provide different types of information. Often, each application, function, website, or feature has its own user interface and its own operational paradigms, many of which can be burdensome to learn or overwhelming for users. In addition, many users may have difficulty even discovering what functionality and/or information is available on their electronic devices or on various websites; thus, such users may become frustrated or overwhelmed, or may simply be unable to use the resources available to them in an effective manner.
In particular, novice users, or individuals who are impaired or disabled in some manner, and/or are elderly, busy, distracted, and/or operating a vehicle may have difficulty interfacing with their electronic devices effectively, and/or engaging online services effectively. Such users are particularly likely to have difficulty with the large number of diverse and inconsistent functions, applications, and websites that may be available for their use.
Accordingly, existing systems are often difficult to use and to navigate, and often present users with inconsistent and overwhelming interfaces that often prevent the users from making effective use of the technology.
SUMMARY
According to various embodiments of the present invention, an intelligent automated assistant is implemented on an electronic device, to facilitate user interaction with a device, and to help the user more effectively engage with local and/or remote services. In various embodiments, the intelligent automated assistant engages with the user in an integrated, conversational manner using natural language dialog, and invokes external services when appropriate to obtain information or perform various actions.
According to various embodiments of the present invention, the intelligent automated assistant integrates a variety of capabilities provided by different software components (e.g., for supporting natural language recognition and dialog, multimodal input, personal information management, task flow management, orchestrating distributed services, and the like). Furthermore, to offer intelligent interfaces and useful functionality to users, the intelligent automated assistant of the present invention may, in at least some embodiments, coordinate these components and services. The conversation interface, and the ability to obtain information and perform follow-on task, are implemented, in at least some embodiments, by coordinating various components such as language components, dialog components, task management components, information management components and/or a plurality of external services.
According to various embodiments of the present invention, intelligent automated assistant systems may be configured, designed, and/or operable to provide various different types of operations, functionalities, and/or features, and/or to combine a plurality of features, operations, and applications of an electronic device on which it is installed. In some embodiments, the intelligent automated assistant systems of the present invention can perform any or all of: actively eliciting input from a user, interpreting user intent, disambiguating among competing interpretations, requesting and receiving clarifying information as needed, and performing (or initiating) actions based on the discerned intent. Actions can be performed, for example, by activating and/or interfacing with any applications or services that may be available on an electronic device, as well as services that are available over an electronic network such as the Internet. In various embodiments, such activation of external services can be performed via APIs or by any other suitable mechanism. In this manner, the intelligent automated assistant systems of various embodiments of the present invention can unify, simplify, and improve the user's experience with respect to many different applications and functions of an electronic device, and with respect to services that may be available over the Internet. The user can thereby be relieved of the burden of learning what functionality may be available on the device and on web-connected services, how to interface with such services to get what he or she wants, and how to interpret the output received from such services; rather, the assistant of the present invention can act as a go-between between the user and such diverse services.
In addition, in various embodiments, the assistant of the present invention provides a conversational interface that the user may find more intuitive and less burdensome than conventional graphical user interfaces. The user can engage in a form of conversational dialog with the assistant using any of a number of available input and output mechanisms, such as for example speech, graphical user interfaces (buttons and links), text entry, and the like. The system can be implemented using any of a number of different platforms, such as device APIs, the web, email, and the like, or any combination thereof. Requests for additional input can be presented to the user in the context of such a conversation. Short and long term memory can be engaged so that user input can be interpreted in proper context given previous events and communications within a given session, as well as historical and profile information about the user.
In addition, in various embodiments, context information derived from user interaction with a feature, operation, or application on a device can be used to streamline the operation of other features, operations, or applications on the device or on other devices. For example, the intelligent automated assistant can use the context of a phone call (such as the person called) to streamline the initiation of a text message (for example to determine that the text message should be sent to the same person, without the user having to explicitly specify the recipient of the text message). The intelligent automated assistant of the present invention can thereby interpret instructions such as âsend him a text messageâ, wherein the âhimâ is interpreted according to context information derived from a current phone call, and/or from any feature, operation, or application on the device. In various embodiments, the intelligent automated assistant takes into account various types of available context data to determine which address book contact to use, which contact data to use, which telephone number to use for the contact, and the like, so that the user need not re-specify such information manually.
In various embodiments, the assistant can also take into account external events and respond accordingly, for example, to initiate action, initiate communication with the user, provide alerts, and/or modify previously initiated action in view of the external events. If input is required from the user, a conversational interface can again be used.
In one embodiment, the system is based on sets of interrelated domains and tasks, and employs additional functionally powered by external services with which the system can interact. In various embodiments, these external services include web-enabled services, as well as functionality related to the hardware device itself. For example, in an embodiment where the intelligent automated assistant is implemented on a smartphone, personal digital assistant, tablet computer, or other device, the assistant can control many operations and functions of the device, such as to dial a telephone number, send a text message, set reminders, add events to a calendar, and the like.
In various embodiments, the system of the present invention can be implemented to provide assistance in any of a number of different domains. Examples include:
Local Services (including location- and time-specific services such as restaurants, movies, automated teller machines (ATMs), events, and places to meet); Personal and Social Memory Services (including action items, notes, calendar events, shared links, and the like); E-commerce (including online purchases of items such as books, DVDs, music, and the like); Travel Services (including flights, hotels, attractions, and the like).
One skilled in the art will recognize that the above list of domains is merely exemplary. In addition, the system of the present invention can be implemented in any combination of domains.
In various embodiments, the intelligent automated assistant systems disclosed herein may be configured or designed to include functionality for automating the application of data and services available over the Internet to discover, find, choose among, purchase, reserve, or order products and services. In addition to automating the process of using these data and services, at least one intelligent automated assistant system embodiment disclosed herein may also enable the combined use of several sources of data and services at once. For ex-ample, it may combine information about products from several review sites, check prices and availability from multiple distributors, and check their locations and time constraints, and help a user find a personalized solution to their problem. Additionally, at least one intelligent automated assistant system embodiment disclosed herein may be configured or designed to include functionality for automating the use of data and services available over the Internet to discover, investigate, select among, reserve, and otherwise learn about things to do (including but not limited to movies, events, performances, exhibits, shows and at-tractions); places to go (including but not limited to travel destinations, hotels and other places to stay, landmarks and other sites of interest, etc.); places to eat or drink (such as restaurants and bars), times and places to meet others, and any other source of entertainment or social interaction which may be found on the Internet. Additionally, at least one intelligent automated assistant system embodiment disclosed herein may be configured or designed to include functionality for enabling the operation of applications and services via natural language dialog that may be otherwise provided by dedicated applications with graphical user interfaces including search (including location-based search); navigation (maps and directions); database lookup (such as finding businesses or people by name or other properties); getting weather conditions and forecasts, checking the price of market items or status of financial transactions; monitoring traffic or the status of flights; accessing and updating calendars and schedules; managing reminders, alerts, tasks and projects; communicating over email or other messaging platforms; and operating devices locally or remotely (e.g., dialing telephones, controlling light and temperature, controlling home security devices, playing music or video, etc.). Further, at least one intelligent automated assistant system embodiment disclosed herein may be configured or designed to include functionality for identifying, generating, and/or providing personalized recommendations for activities, products, services, source of entertainment, time management, or any other kind of recommendation service that benefits from an interactive dialog in natural language and automated access to data and services.
In various embodiments, the intelligent automated assistant of the present invention can control many features and operations of an electronic device. For example, the intelligent automated assistant can call services that interface with functionality and applications on a device via APIs or by other means, to perform functions and operations that might otherwise be initiated using a conventional user interface on the device. Such functions and operations may include, for example, setting an alarm, making a telephone call, sending a text message or email message, adding a calendar event, and the like. Such functions and operations may be performed as add-on functions in the context of a conversational dialog between a user and the assistant. Such functions and operations can be specified by the user in the context of such a dialog, or they may be automatically performed based on the context of the dialog. One skilled in the art will recognize that the assistant can thereby be used as a control mechanism for initiating and controlling various operations on the electronic device, which may be used as an alternative to conventional mechanisms such as buttons or graphical user interfaces.
BRIEF DESCRIPTION OF THE DRAWINGS
The accompanying drawings illustrate several embodiments of the invention and, together with the description, serve to explain the principles of the invention according to the embodiments. One skilled in the art will recognize that the particular embodiments illustrated in the drawings are merely exemplary, and are not intended to limit the scope of the present invention.
FIG. 1 is a block diagram depicting an example of one embodiment of an intelligent automated assistant system.
FIG. 2 illustrates an example of an interaction between a user and an intelligent automated assistant according to at least one embodiment.
FIG. 3 is a block diagram depicting a computing device suitable for implementing at least a portion of an intelligent automated assistant according to at least one embodiment.
FIG. 4 is a block diagram depicting an architecture for implementing at least a portion of an intelligent automated assistant on a standalone computing system, according to at least one embodiment.
FIG. 5 is a block diagram depicting an architecture for implementing at least a portion of an intelligent automated assistant on a distributed computing network, according to at least one embodiment.
FIG. 6 is a block diagram depicting a system architecture illustrating several different types of clients and modes of operation.
FIG. 7 is a block diagram depicting a client and a server, which communicate with each other to implement the present invention according to one embodiment.
FIG. 8 is a block diagram depicting a fragment of an active ontology ac-cording to one embodiment.
FIG. 9 is a block diagram depicting an example of an alternative embodiment of an intelligent au
CROSS-REFERENCE TO RELATED APPLICATIONS
This application is a continuation of U.S. application Ser. No. 16/879,643, entitled âTask Flow Identification Based on User Intent,â filed May 20, 2020, (now granted as U.S. Pat. No. 11,423,886), which is a continuation of U.S. application Ser. No. 15/394,162, entitled âTask Flow Identification Based on User Intent,â filed Dec. 29, 2016, (now granted as U.S. Pat. No. 10,706,841), which is a continuation of U.S. application Ser. No. 13/492,809, entitled âIntelligent Automated Assistant,â filed Jun. 9, 2012 (now granted as U.S. Pat. No. 9,548,050), which is a continuation of U.S. application Ser. No. 12/987,982, entitled âIntelligent Automated Assistant,â filed Jan. 10, 2011 (now granted as U.S. Pat. No. 9,318,108), which claims the benefit of U.S. Provisional Patent Application Ser. No. 61/295,774, entitled âIntelligent Automated Assistantâ, filed Jan. 18, 2010, each of which are hereby incorporated by reference in their entirety for all purposes.
This application is further related to U.S. patent application Ser. No. 11/518,292 for âMethod and Apparatus for Building an Intelligent Automated Assistantâ, filed Sep. 8, 2006, which is incorporated herein by reference.
This application is further related to U.S. Provisional Patent Application Ser. No. 61/186,414 for âSystem and Method for Semantic Auto-Completionâ, filed Jun. 12, 2009, which is incorporated herein by reference.
FIELD OF THE INVENTION
The present invention relates to intelligent systems, and more specifically for classes of applications for intelligent automated assistants.
BACKGROUND OF THE INVENTION
Today's electronic devices are able to access a large, growing, and diverse quantity of functions, services, and information, both via the Internet and from other sources. Functionality for such devices is increasing rapidly, as many consumer devices, smartphones, tablet computers, and the like, are able to run software applications to perform various tasks and provide different types of information. Often, each application, function, website, or feature has its own user interface and its own operational paradigms, many of which can be burdensome to learn or overwhelming for users. In addition, many users may have difficulty even discovering what functionality and/or information is available on their electronic devices or on various websites; thus, such users may become frustrated or overwhelmed, or may simply be unable to use the resources available to them in an effective manner.
In particular, novice users, or individuals who are impaired or disabled in some manner, and/or are elderly, busy, distracted, and/or operating a vehicle may have difficulty interfacing with their electronic devices effectively, and/or engaging online services effectively. Such users are particularly likely to have difficulty with the large number of diverse and inconsistent functions, applications, and websites that may be available for their use.
Accordingly, existing systems are often difficult to use and to navigate, and often present users with inconsistent and overwhelming interfaces that often prevent the users from making effective use of the technology.
SUMMARY
According to various embodiments of the present invention, an intelligent automated assistant is implemented on an electronic device, to facilitate user interaction with a device, and to help the user more effectively engage with local and/or remote services. In various embodiments, the intelligent automated assistant engages with the user in an integrated, conversational manner using natural language dialog, and invokes external services when appropriate to obtain information or perform various actions.
According to various embodiments of the present invention, the intelligent automated assistant integrates a variety of capabilities provided by different software components (e.g., for supporting natural language recognition and dialog, multimodal input, personal information management, task flow management, orchestrating distributed services, and the like). Furthermore, to offer intelligent interfaces and useful functionality to users, the intelligent automated assistant of the present invention may, in at least some embodiments, coordinate these components and services. The conversation interface, and the ability to obtain information and perform follow-on task, are implemented, in at least some embodiments, by coordinating various components such as language components, dialog components, task management components, information management components and/or a plurality of external services.
According to various embodiments of the present invention, intelligent automated assistant systems may be configured, designed, and/or operable to provide various different types of operations, functionalities, and/or features, and/or to combine a plurality of features, operations, and applications of an electronic device on which it is installed. In some embodiments, the intelligent automated assistant systems of the present invention can perform any or all of: actively eliciting input from a user, interpreting user intent, disambiguating among competing interpretations, requesting and receiving clarifying information as needed, and performing (or initiating) actions based on the discerned intent. Actions can be performed, for example, by activating and/or interfacing with any applications or services that may be available on an electronic device, as well as services that are available over an electronic network such as the Internet. In various embodiments, such activation of external services can be performed via APIs or by any other suitable mechanism. In this manner, the intelligent automated assistant systems of various embodiments of the present invention can unify, simplify, and improve the user's experience with respect to many different applications and functions of an electronic device, and with respect to services that may be available over the Internet. The user can thereby be relieved of the burden of learning what functionality may be available on the device and on web-connected services, how to interface with such services to get what he or she wants, and how to interpret the output received from such services; rather, the assistant of the present invention can act as a go-between between the user and such diverse services.
In addition, in various embodiments, the assistant of the present invention provides a conversational interface that the user may find more intuitive and less burdensome than conventional graphical user interfaces. The user can engage in a form of conversational dialog with the assistant using any of a number of available input and output mechanisms, such as for example speech, graphical user interfaces (buttons and links), text entry, and the like. The system can be implemented using any of a number of different platforms, such as device APIs, the web, email, and the like, or any combination thereof. Requests for additional input can be presented to the user in the context of such a conversation. Short and long term memory can be engaged so that user input can be interpreted in proper context given previous events and communications within a given session, as well as historical and profile information about the user.
In addition, in various embodiments, context information derived from user interaction with a feature, operation, or application on a device can be used to streamline the operation of other features, operations, or applications on the device or on other devices. For example, the intelligent automated assistant can use the context of a phone call (such as the person called) to streamline the initiation of a text message (for example to determine that the text message should be sent to the same person, without the user having to explicitly specify the recipient of the text message). The intelligent automated assistant of the present invention can thereby interpret instructions such as âsend him a text messageâ, wherein the âhimâ is interpreted according to context information derived from a current phone call, and/or from any feature, operation, or application on the device. In various embodiments, the intelligent automated assistant takes into account various types of available context data to determine which address book contact to use, which contact data to use, which telephone number to use for the contact, and the like, so that the user need not re-specify such information manually.
In various embodiments, the assistant can also take into account external events and respond accordingly, for example, to initiate action, initiate communication with the user, provide alerts, and/or modify previously initiated action in view of the external events. If input is required from the user, a conversational interface can again be used.
In one embodiment, the system is based on sets of interrelated domains and tasks, and employs additional functionally powered by external services with which the system can interact. In various embodiments, these external services include web-enabled services, as well as functionality related to the hardware device itself. For example, in an embodiment where the intelligent automated assistant is implemented on a smartphone, personal digital assistant, tablet computer, or other device, the assistant can control many operations and functions of the device, such as to dial a telephone number, send a text message, set reminders, add events to a calendar, and the like.
In various embodiments, the system of the present invention can be implemented to provide assistance in any of a number of different domains. Examples include:
Local Services (including location- and time-specific services such as restaurants, movies, automated teller machines (ATMs), events, and places to meet); Personal and Social Memory Services (including action items, notes, calendar events, shared links, and the like); E-commerce (including online purchases of items such as books, DVDs, music, and the like); Travel Services (including flights, hotels, attractions, and the like).
One skilled in the art will recognize that the above list of domains is merely exemplary. In addition, the system of the present invention can be implemented in any combination of domains.
In various embodiments, the intelligent automated assistant systems disclosed herein may be configured or designed to include functionality for automating the application of data and services available over the Internet to discover, find, choose among, purchase, reserve, or order products and services. In addition to automating the process of using these data and services, at least one intelligent automated assistant system embodiment disclosed herein may also enable the combined use of several sources of data and services at once. For ex-ample, it may combine information about products from several review sites, check prices and availability from multiple distributors, and check their locations and time constraints, and help a user find a personalized solution to their problem. Additionally, at least one intelligent automated assistant system embodiment disclosed herein may be configured or designed to include functionality for automating the use of data and services available over the Internet to discover, investigate, select among, reserve, and otherwise learn about things to do (including but not limited to movies, events, performances, exhibits, shows and at-tractions); places to go (including but not limited to travel destinations, hotels and other places to stay, landmarks and other sites of interest, etc.); places to eat or drink (such as restaurants and bars), times and places to meet others, and any other source of entertainment or social interaction which may be found on the Internet. Additionally, at least one intelligent automated assistant system embodiment disclosed herein may be configured or designed to include functionality for enabling the operation of applications and services via natural language dialog that may be otherwise provided by dedicated applications with graphical user interfaces including search (including location-based search); navigation (maps and directions); database lookup (such as finding businesses or people by name or other properties); getting weather conditions and forecasts, checking the price of market items or status of financial transactions; monitoring traffic or the status of flights; accessing and updating calendars and schedules; managing reminders, alerts, tasks and projects; communicating over email or other messaging platforms; and operating devices locally or remotely (e.g., dialing telephones, controlling light and temperature, controlling home security devices, playing music or video, etc.). Further, at least one intelligent automated assistant system embodiment disclosed herein may be configured or designed to include functionality for identifying, generating, and/or providing personalized recommendations for activities, products, services, source of entertainment, time management, or any other kind of recommendation service that benefits from an interactive dialog in natural language and automated access to data and services.
In various embodiments, the intelligent automated assistant of the present invention can control many features and operations of an electronic device. For example, the intelligent automated assistant can call services that interface with functionality and applications on a device via APIs or by other means, to perform functions and operations that might otherwise be initiated using a conventional user interface on the device. Such functions and operations may include, for example, setting an alarm, making a telephone call, sending a text message or email message, adding a calendar event, and the like. Such functions and operations may be performed as add-on functions in the context of a conversational dialog between a user and the assistant. Such functions and operations can be specified by the user in the context of such a dialog, or they may be automatically performed based on the context of the dialog. One skilled in the art will recognize that the assistant can thereby be used as a control mechanism for initiating and controlling various operations on the electronic device, which may be used as an alternative to conventional mechanisms such as buttons or graphical user interfaces.
BRIEF DESCRIPTION OF THE DRAWINGS
The accompanying drawings illustrate several embodiments of the invention and, together with the description, serve to explain the principles of the invention according to the embodiments. One skilled in the art will recognize that the particular embodiments illustrated in the drawings are merely exemplary, and are not intended to limit the scope of the present invention.
FIG. 1 is a block diagram depicting an example of one embodiment of an intelligent automated assistant system.
FIG. 2 illustrates an example of an interaction between a user and an intelligent automated assistant according to at least one embodiment.
FIG. 3 is a block diagram depicting a computing device suitable for implementing at least a portion of an intelligent automated assistant according to at least one embodiment.
FIG. 4 is a block diagram depicting an architecture for implementing at least a portion of an intelligent automated assistant on a standalone computing system, according to at least one embodiment.
FIG. 5 is a block diagram depicting an architecture for implementing at least a portion of an intelligent automated assistant on a distributed computing network, according to at least one embodiment.
FIG. 6 is a block diagram depicting a system architecture illustrating several different types of clients and modes of operation.
FIG. 7 is a block diagram depicting a client and a server, which communicate with each other to implement the present invention according to one embodiment.
FIG. 8 is a block diagram depicting a fragment of an active ontology ac-cording to one embodiment.
FIG. 9 is a block diagram depicting an example of an alternative embodiment of an intelligent automated assistant system.
FIG. 10 is a flow diagram depicting a method of operation for active input elicitation component(s) according to one embodiment.
FIG. 11 is a flow diagram depicting a method for active typed-input elicitation according to one embodiment.
FIGS. 12 to 21 are screen shots illustrating some portions of some of the procedures for active typed-input elicitation according to one embodiment.
FIG. 22 is a flow diagram depicting a method for active input elicitation for voice or speech input according to one embodiment.
FIG. 23 is a flow diagram depicting a method for active input elicitation for GUI-based input according to one embodiment.
FIG. 24 is a flow diagram depicting a method for active input elicitation at the level of a dialog flow according to one embodiment.
FIG. 25 is a flow diagram depicting a method for active monitoring for relevant events according to one embodiment.
FIG. 26 is a flow diagram depicting a method for multimodal active input elicitation according to one embodiment.
FIG. 27 is a set of screen shots illustrating an example of various types of functions, operations, actions, and/or other features which may be provided by domain models component(s) and services orchestration according to one embodiment.
FIG. 28 is a flow diagram depicting an example of a method for natural language processing according to one embodiment.
FIG. 29 is a screen shot illustrating natural language processing according to one embodiment.
FIGS. 30 and 31 are screen shots illustrating an example of various types of functions, operations, actions, and/or other features which may be provided by dialog flow processor component(s) according to one embodiment.
FIG. 32 is a flow diagram depicting a method of operation for dialog flow processor component(s) according to one embodiment.
FIG. 33 is a flow diagram depicting an automatic call and response procedure, according to one embodiment.
FIG. 34 is a flow diagram depicting an example of task flow for a constrained selection task according to one embodiment.
FIGS. 35 and 36 are screen shots illustrating an example of the operation of constrained selection task according to one embodiment.
FIG. 37 is a flow diagram depicting an example of a procedure for executing a service orchestration procedure according to one embodiment.
FIG. 38 is a flow diagram depicting an example of a service invocation procedure according to one embodiment.
FIG. 39 is a flow diagram depicting an example of a multiphase output procedure according to one embodiment.
FIGS. 40 and 41 are screen shots depicting examples of output processing according to one embodiment.
FIG. 42 is a flow diagram depicting an example of multimodal output processing according to one embodiment.
FIGS. 43 A and 43 B are screen shots depicting an example of the use of short term personal memory component(s) to maintain dialog context while changing location, according to one embodiment.
FIGS. 44 A through 44 C are screen shots depicting an example of the use of long term personal memory component(s), according to one embodiment.
FIG. 45 depicts an example of an abstract model for a constrained selection task.
FIG. 46 depicts an example of a dialog flow model to help guide the user through a search process.
FIG. 47 is a flow diagram depicting a method of constrained selection according to one embodiment.
DETAILED DESCRIPTION OF THE EMBODIMENTS
Various techniques will now be described in detail with reference to a few example embodiments thereof as illustrated in the accompanying drawings. In the following description, numerous specific details are set forth in order to provide a thorough understanding of one or more aspects and/or features described or reference herein. It will be apparent, however, to one skilled in the art, that one or more aspects and/or features described or reference herein may be practiced without some or all of these specific details. In other instances, well known process steps and/or structures have not been described in detail in order to not obscure some of the aspects and/or features described or reference herein.
One or more different inventions may be described in the present application. Further, for one or more of the invention(s) described herein, numerous embodiments may be described in this patent application, and are presented for illustrative purposes only. The described embodiments are not intended to be limiting in any sense. One or more of the invention(s) may be widely applicable to numerous embodiments, as is readily apparent from the disclosure. These embodiments are described in sufficient detail to enable those skilled in the art to practice one or more of the invention(s), and it is to be understood that other embodiments may be utilized and that structural, logical, software, electrical and other changes may be made without departing from the scope of the one or more of the invention(s). Accordingly, those skilled in the art will recognize that the one or more of the invention(s) may be practiced with various modifications and alterations. Particular features of one or more of the invention(s) may be described with reference to one or more particular embodiments or figures that form a part of the present disclosure, and in which are shown, by way of illustration, specific embodiments of one or more of the invention(s). It should be understood, however, that such features are not limited to usage in the one or more particular embodiments or figures with reference to which they are described. The present disclosure is neither a literal description of all embodiments of one or more of the invention(s) nor a listing of features of one or more of the invention(s) that must be present in all embodiments.
Headings of sections provided in this patent application and the title of this patent application are for convenience only, and are not to be taken as limiting the disclosure in any way.
Devices that are in communication with each other need not be in continuous communication with each other, unless expressly specified otherwise. In addition, devices that are in communication with each other may communicate directly or indirectly through one or more intermediaries.
A description of an embodiment with several components in communication with each other does not imply that all such components are required. To the contrary, a variety of optional components are described to illustrate the wide variety of possible embodiments of one or more of the invention(s).
Further, although process steps, method steps, algorithms or the like may be described in a sequential order, such processes, methods and algorithms may be configured to work in alternate orders. In other words, any sequence or order of steps that may be described in this patent application does not, in and of itself, indicate a requirement that the steps be performed in that order. The steps of described processes may be performed in any order practical. Further, some steps may be performed simultaneously despite being described or implied as occurring non-simultaneously (e.g., because one step is described after the other step). Moreover, the illustration of a process by its depiction in a drawing does not imply that the illustrated process is exclusive of other variations and modifications thereto, does not imply that the illustrated process or any of its steps are necessary to one or more of the invention(s), and does not imply that the illustrated process is preferred.
When a single device or article is described, it will be readily apparent that more than one device/article (whether or not they cooperate) may be used in place of a single device/article. Similarly, where more than one device or article is described (whether or not they cooperate), it will be readily apparent that a single device/article may be used in place of the more than one device or article.
The functionality and/or the features of a device may be alternatively embodied by one or more other devices that are not explicitly described as having such functionality/features. Thus, other embodiments of one or more of the invention(s) need not include the device itself.
Techniques and mechanisms described or reference herein will sometimes be described in singular form for clarity. However, it should be noted that particular embodiments include multiple iterations of a technique or multiple instantiations of a mechanism unless noted otherwise.
Although described within the context of intelligent automated assistant technology, it may be understood that the various aspects and techniques described herein (such as those associated with active ontologies, for example) may also be deployed and/or applied in other fields of technology involving human and/or computerized interaction with software.
Other aspects relating to intelligent automated assistant technology (e.g., which may be utilized by, provided by, and/or implemented at one or more intelligent automated assistant system embodiments described herein) are disclosed in one or more of the following references:
U.S. Provisional Patent Application Ser. No. 61/295,774 for âIntelligent Automated Assistantâ, filed Jan. 18, 2010, the disclosure of which is incorporated herein by reference; U.S. patent application Ser. No. 11/518,292 for âMethod And Apparatus for Building an Intelligent Automated Assistantâ, filed Sep. 8, 2006, the disclosure of which is incorporated herein by reference; and U.S. Provisional Patent Application Ser. No. 61/186,414 for âSystem and Method for Semantic Auto-Completionâ, filed Jun. 12, 2009, the disclosure of which is incorporated herein by reference.
Hardware Architecture
Generally, the intelligent automated assistant techniques disclosed herein may be implemented on hardware or a combination of software and hardware. For example, they may be implemented in an operating system kernel, in a separate user process, in a library package bound into network applications, on a specially constructed machine, or on a network interface card. In a specific embodiment, the techniques disclosed herein may be implemented in software such as an operating system or in an application running on an operating system.
Software/hardware hybrid implementation(s) of at least some of the intelligent automated assistant embodiment(s) disclosed herein may be implemented on a programmable machine selectively activated or reconfigured by a computer program stored in memory. Such network devices may have multiple network interfaces which may be configured or designed to utilize different types of network communication protocols. A general architecture for some of these machines may appear from the descriptions disclosed herein. According to specific embodiments, at least some of the features and/or functionalities of the various intelligent automated assistant embodiments disclosed herein may be implemented on one or more general-purpose network host machines such as an end-user computer system, computer, network server or server system, mobile computing device (e.g., personal digital assistant, mobile phone, smartphone, laptop, tablet computer, or the like), consumer electronic device, music player, or any other suitable electronic device, router, switch, or the like, or any combination thereof. In at least some embodiments, at least some of the features and/or functionalities of the various intelligent automated assistant embodiments disclosed herein may be implemented in one or more virtualized computing environments (e.g., network computing clouds, or the like).
Referring now to FIG. 3 , there is shown a block diagram depicting a computing device 60 suitable for implementing at least a portion of the intelligent automated assistant features and/or functionalities disclosed herein. Computing device 60 may be, for example, an end-user computer system, network server or server system, mobile computing device (e.g., personal digital assistant, mobile phone, smartphone, laptop, tablet computer, or the like), consumer electronic device, music player, or any other suitable electronic device, or any combination or portion thereof. Computing device 60 may be adapted to communicate with other computing devices, such as clients and/or servers, over a communications network such as the Internet, using known protocols for such communication, whether wireless or wired.
In one embodiment, computing device 60 includes central processing unit (CPU) 62 , interfaces 68 , and a bus 67 (such as a peripheral component inter-connect (PCI) bus). When acting under the control of appropriate software or firmware, CPU 62 may be responsible for implementing specific functions associated with the functions of a specifically configured computing device or machine. For example, in at least one embodiment, a user's personal digital assistant (PDA) may be configured or designed to function as an intelligent automated assistant system utilizing CPU 62 , memory
61 , 65 , and interface(s) 68 . In at least one embodiment, the CPU 62 may be caused to perform one or more of the different types of intelligent automated assistant functions and/or operations under the control of software modules/components, which for example, may include an operating system and any appropriate applications software, drivers, and the like.
CPU 62 may include one or more processor(s) 63 such as, for example, a processor from the Motorola or Intel family of microprocessors or the MIPS family of microprocessors. In some embodiments, processor(s) 63 may include specially designed hardware (e.g., application-specific integrated circuits (ASICs), electrically erasable programmable read-only memories (EEPROMs), field-programmable gate arrays (FPGAs), and the like) for controlling the operations of computing device 60 . In a specific embodiment, a memory 61 (such as non-volatile random access memory (RAM) and/or read-only memory (ROM)) also forms part of CPU 62 . However, there are many different ways in which memory may be coupled to the system. Memory block 61 may be used for a variety of purposes such as, for example, caching and/or storing data, programming instructions, and the like.
As used herein, the term âprocessorâ is not limited merely to those integrated circuits referred to in the art as a processor, but broadly refers to a microcontroller, a microcomputer, a programmable logic controller, an application-specific integrated circuit, and any other programmable circuit.
In one embodiment, interfaces 68 are provided as interface cards (sometimes referred to as âline cardsâ). Generally, they control the sending and receiving of data packets over a computing network and sometimes support other peripherals used with computing device 60 . Among the interfaces that may be provided are Ethernet interfaces, frame relay interfaces, cable interfaces, DSL interfaces, token ring interfaces, and the like. In addition, various types of interfaces may be provided such as, for example, universal serial bus (USB), Serial, Ethernet, Firewire, PCI, parallel, radio frequency (RF), Bluetoothâ¢, near-field communications (e.g., using near-field magnetics), 802.11 (WiFi), frame relay, TCP/IP, ISDN, fast Ethernet interfaces, Gigabit Ethernet interfaces, asynchronous transfer mode (ATM) interfaces, high-speed serial interface (HSSI) interfaces, Point of Sale (POS) interfaces, fiber data distributed interfaces (FDDIs), and the like. Generally, such interfaces 68 may include ports appropriate for communication with the appropriate media. In some cases, they may also include an independent processor and, in some instances, volatile and/or nonvolatile memory (e.g., RAM).
Although the system shown in FIG. 3 illustrates one specific architecture for a computing device 60 for implementing the techniques of the invention described herein, it is by no means the only device architecture on which at least a portion of the features and techniques described herein may be implemented. For example, architectures having one or any number of processors 63 can be used, and such processors 63 can be present in a single device or distributed among any number of devices. In one embodiment, a single processor 63 handles communications as well as routing computations. In various embodiments, different types of intelligent automated assistant features and/or functionalities may be implemented in an intelligent automated assistant system which includes a client device (such as a personal digital assistant or smartphone running client software) and server system(s) (such as a server system described in more detail below).
Regardless of network device configuration, the system of the present invention may employ one or more memories or memory modules (such as, for example, memory block 65 ) configured to store data, program instructions for the general-purpose network operations and/or other information relating to the functionality of the intelligent automated assistant techniques described herein. The program instructions may control the operation of an operating system and/or one or more applications, for example. The memory or memories may also be configured to store data structures, keyword taxonomy information, advertisement information, user click and impression information, and/or other specific non-program information described herein.
Because such information and program instructions may be employed to implement the systems/methods described herein, at least some network device embodiments may include nontransitory machine-readable storage media, which, for example, may be configured or designed to store program instructions, state information, and the like for performing various operations described herein. Examples of such nontransitory machine-readable storage media include, but are not limited to, magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROM disks; magneto-optical media such as floptical disks, and hardware devices that are specially configured to store and perform program instructions, such as read-only memory devices (ROM), flash memory, memristor memory, random access memory (RAM), and the like. Examples of program instructions include both machine code, such as produced by a compiler, and files containing higher level code that may be executed by the computer using an interpreter.
In one embodiment, the system of the present invention is implemented on a standalone computing system. Referring now to FIG. 4 , there is shown a block diagram depicting an architecture for implementing at least a portion of an intelligent automated assistant on a standalone computing system, according to at least one embodiment. Computing device 60 includes processor(s) 63 which run software for implementing intelligent automated assistant 1002 . Input device 1206 can be of any type suitable for receiving user input, including for example a keyboard, touch screen, microphone (for example, for voice input), mouse, touchpad, trackball, five-way switch, joystick, and/or any combination thereof. Output device 1207 can be a screen, speaker, printer, and/or any combination thereof. Memory 1210 can be random-access memory having a structure and architecture as are known in the art, for use by processor(s) 63 in the course of running software. Storage device 1208 can be any magnetic, optical, and/or electrical storage device for storage of data in digital form; examples include flash memory, magnetic hard drive, CD-ROM, and/or the like.
In another embodiment, the system of the present invention is implemented on a distributed computing network, such as one having any number of clients and/or servers. Referring now to FIG. 5 , there is shown a block diagram depicting an architecture for implementing at least a portion of an intelligent automated assistant on a distributed computing network, according to at least one embodiment.
In the arrangement shown in FIG. 5 , any number of clients 1304 are provided; each client 1304 may run software for implementing client-side portions of the present invention. In addition, any number of servers 1340 can be provided for handling requests received from clients 1304 . Clients 1304 and servers 1340 can communicate with one another via electronic network 1361 , such as the Internet. Network 1361 may be implemented using any known net-work protocols, including for example wired and/or wireless protocols.
In addition, in one embodiment, servers 1340 can call external services 1360 when needed to obtain additional information or refer to store data concerning previous interactions with particular users. Communications with external services 1360 can take place, for example, via network 1361 . In various embodiments, external services 1360 include web-enabled services and/or functionality related to or installed on the hardware device itself. For example, in an embodiment where assistant 1002 is implemented on a smartphone or other electronic device, assistant 1002 can obtain information stored in a calendar application (âappâ), contacts, and/or other sources.
In various embodiments, assistant 1002 can control many features and operations of an electronic device on which it is installed. For example, assistant 1002 can call external services 1360 that interface with functionality and applications on a device via APIs or by other means, to perform functions and operations that might otherwise be initiated using a conventional user interface on the device. Such functions and operations may include, for example, setting an alarm, making a telephone call, sending a text message or email message, adding a calendar event, and the like. Such functions and operations may be performed as add-on functions in the context of a conversational dialog between a user and assistant 1002 . Such functions and operations can be specified by the user in the context of such a dialog, or they may be automatically performed based on the context of the dialog. One skilled in the art will recognize that assistant 1002 can thereby be used as a control mechanism for initiating and controlling various operations on the electronic device, which may be used as an alternative to conventional mechanisms such as buttons or graphical user interfaces.
For example, the user may provide input to assistant 1002 such as âI need to wake tomorrow at 8 am.â Once assistant 1002 has determined the user's intent, using the techniques described herein, assistant 1002 can call external services 1340 to interface with an alarm clock function or application on the device. Assistant 1002 sets the alarm on behalf of the user. In this manner, the user can use assistant 1002 as a replacement for conventional mechanisms for setting the alarm or performing other functions on the device. If the user's requests are ambiguous or need further clarification, assistant 1002 can use the various techniques described herein, including active elicitation, paraphrasing, suggestions, and the like, to obtain the needed information so that the correct services 1340 are called and the intended action taken. In one embodiment, assistant 1002 may prompt the user for confirmation before calling a service 1340 to perform a function. In one embodiment, a user can selectively disable assistant's 1002 ability to call particular services 1340 , or can disable all such service-calling if desired.
The system of the present invention can be implemented with many different types of clients 1304 and modes of operation. Referring now to FIG. 6 , there is shown a block diagram depicting a system architecture illustrating several different types of clients 1304 and modes of operation. One skilled in the art will recognize that the various types of clients 1304 and modes of operation shown in FIG. 6 are merely exemplary, and that the system of the present invention can be implemented using clients 1304 and/or modes of operation other than those depicted. Additionally, the system can include any or all of such clients 1304 and/or modes of operation, alone or in any combination. Depicted examples include:
Computer devices with input/output devices and/or sensors 1402 . A client component may be deployed on any such computer device 1402 . At least one embodiment may be implemented using a web browser 1304 A or other software application for enabling communication with servers 1340 via network 1361 . Input and output channels may of any type, including for example visual and/or auditory channels. For example, in one embodiment, the system of the invention can be implemented using voice-based communication methods, allowing for an embodiment of the assistant for the blind whose equivalent of a web browser is driven by speech and uses speech for output. Mobile Devices with I/O and sensors 1406 , for which the client may be implemented as an application on the mobile device 1304 B. This includes, but is not limited to, mobile phones, smartphones, personal digital assistants, tablet devices, networked game consoles, and the like. Consumer Appliances with I/O and sensors 1410 , for which the client may be implemented as an embedded application on the appliance 1304 C. Automobiles and other vehicles with dashboard interfaces and sensors 1414 , for which
CLAIMS
Claims ( 39 )
What is claimed is:
1. An electronic device, comprising:
one or more processors;
a memory; and
one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for:
receiving, within a communication session between a user and a digital assistant, a user request for the digital assistant to facilitate a restaurant reservation;
in response to receiving the user request for the digital assistant to facilitate the restaurant reservation, facilitating the restaurant reservation by:
identifying one or more restaurant reservation parameters;
determining one or more values for the one or more restaurant reservation parameters;
transmitting the one or more values to a restaurant reservation service;
receiving a response from the restaurant reservation service indicating whether the restaurant reservation was successfully made; and
providing, in the communication session, an output corresponding to the received response.
2. The electronic device of claim 1 , the one or more programs including instructions for:
displaying, in the communication session, a plurality of restaurants; and
receiving, from the user, a selection of a restaurant from the plurality of restaurants.
3. The electronic device of claim 1 , wherein identifying one or more restaurant reservation parameters further comprises:
identifying a restaurant domain; and
selecting the one or more restaurant reservation parameters from the restaurant domain.
4. The electronic device of claim 1 , the one or more programs including instructions for:
identifying the restaurant reservation service based on at least one of user history or user preference.
5. The electronic device of claim 1 , the one or more programs including instructions for:
identifying the restaurant reservation service based on at least one of reliability information, user reviews, and cost information.
6. The electronic device of claim 1 , the one or more programs including instructions for:
after receiving the response from the restaurant reservation service, storing the restaurant reservation.
7. The electronic device of claim 1 , wherein the restaurant reservation service is a third party service.
8. The electronic device of claim 1 , wherein determining the one or more values for the one or more restaurant reservation parameters further comprises:
prompting the user to provide the one or more values for the one or more restaurant reservation parameters; and
receiving, from the user, an input specifying the one or more values for the one or more restaurant reservation parameters.
9. The electronic device of claim 8 , wherein prompting the user to provide the one or more values for one or more one or more values for the one or more restaurant reservation parameters further comprises:
displaying, in the communication session, a plurality of candidate values; and
receiving, from the user, the input specifying the one or more values for the one or more restaurant reservation parameters further comprises receiving, from the user, a selection of a candidate value of the plurality of candidate values.
10. The electronic device of claim 1 , the one or more programs including instructions for:
performing a semantic analysis on the user request to determine whether the user request includes a request for the digital assistant to facilitate the restaurant reservation.
11. The electronic device of claim 10 , wherein performing the semantic analysis includes performing natural language processing.
12. The electronic device of claim 10 , wherein performing the semantic analysis further comprises:
determining whether one or more words of the user request are associated with a restaurant domain; and
determining whether the one or more words of the user request are associated with a task of reserving a table identified by the restaurant domain.
13. The electronic device of claim 10 , the one or more programs including instructions for:
identifying a restaurant of a plurality of restaurants based on the semantic analysis of the user request.
14. A non-transitory computer-readable medium storing one or more programs comprising instructions, wherein the instructions, when executed by one or more processors of an electronic device, cause the electronic device to:
receive, within a communication session between a user and a digital assistant, a user request for the digital assistant to facilitate a restaurant reservation;
in response to receiving the user request for the digital assistant to facilitate the restaurant reservation, facilitate the restaurant reservation by:
identifying one or more restaurant reservation parameters;
determining one or more values for the one or more restaurant reservation parameters;
transmitting the one or more values to a restaurant reservation service;
receiving a response from the restaurant reservation service indicating whether the restaurant reservation was successfully made; and
providing, in the communication session, an output corresponding to the received response.
15. The non-transitory computer-readable medium of claim 14 , wherein the instructions cause the electronic device to:
display, in the communication session, a plurality of restaurants; and
receive, from the user, a selection of a restaurant from the plurality of restaurants.
16. The non-transitory computer-readable medium of claim 14 , wherein identifying one or more restaurant reservation parameters further comprises:
identifying a restaurant domain; and
selecting the one or more restaurant reservation parameters from the restaurant domain.
17. The non-transitory computer-readable medium of claim 14 , wherein the instructions cause the electronic device to:
identify the restaurant reservation service based on at least one of user history or user preference.
18. The non-transitory computer-readable medium of claim 14 wherein the instructions cause the electronic device to:
identify the restaurant reservation service based on at least one of reliability information, user reviews, and cost information.
19. The non-transitory computer-readable medium of claim 14 , wherein the instructions cause the electronic device to:
after receiving the response from the restaurant reservation service, store the restaurant reservation.
20. The non-transitory computer-readable medium of claim 14 , wherein the restaurant reservation service is a third party service.
21. The non-transitory computer-readable medium of claim 14 , wherein determining the one or more values for the one or more restaurant reservation parameters further comprises:
prompting the user to provide the one or more values for the one or more restaurant reservation parameters; and
receiving, from the user, an input specifying the one or more values for the one or more restaurant reservation parameters.
22. The non-transitory computer-readable medium of claim 21 , wherein prompting the user to provide the one or more values for the one or more restaurant reservation parameters further comprises:
displaying, in the communication session, a plurality of candidate values; and
receiving, from the user, the input specifying the one or more values for the one or more restaurant reservation parameters further comprises receiving, from the user, a selection of a candidate value of the plurality of candidate values.
23. The non-transitory computer-readable medium of claim 14 , wherein the instructions cause the electronic device to:
perform a semantic analysis on the user request to determine whether the user request includes a request for the digital assistant to facilitate the restaurant reservation.
24. The non-transitory computer-readable medium of claim 23 , wherein performing the semantic analysis includes performing natural language processing.
25. The non-transitory computer-readable medium of claim 23 , wherein performing the semantic analysis further comprises:
determining whether one or more words of the user request are associated with a restaurant domain; and
determining whether the one or more words of the user request are associated with a task of reserving a table identified by the restaurant domain.
26. The non-transitory computer-readable medium of claim 23 , wherein the instructions cause the electronic device to:
identify a restaurant of a plurality of restaurants based on the semantic analysis of the user request.
27. A computer-implemented method, comprising:
at an electronic device with one or more processors and memory:
receiving, within a communication session between a user and a digital assistant, a user request for the digital assistant to facilitate a restaurant reservation;
in response to receiving the user request for the digital assistant to facilitate the restaurant reservation, facilitating the restaurant reservation by:
identifying one or more restaurant reservation parameters;
determining one or more values for the one or more restaurant reservation parameters;
transmitting the one or more values to a restaurant reservation service;
receiving a response from the restaurant reservation service indicating whether the restaurant reservation was successfully made; and
providing, in the communication session, an output corresponding to the received response.
28. The method of claim 27 , comprising:
displaying, in the communication session, a plurality of restaurants; and
receiving, from the user, a selection of a restaurant from the plurality of restaurants.
29. The method of claim 27 , wherein identifying one or more restaurant reservation parameters further comprises:
identifying a restaurant domain; and
selecting the one or more restaurant reservation parameters from the restaurant domain.
30. The method of claim 27 , comprising:
identifying the restaurant reservation service based on at least one of user history or user preference.
31. The method of claim 27 , comprising:
identifying the restaurant reservation service based on at least one of reliability information, user reviews, and cost information.
32. The method of claim 27 , comprising:
after receiving the response from the restaurant reservation service, storing the restaurant reservation.
33. The method of claim 27 , wherein the restaurant reservation service is a third party service.
34. The method of claim 27 , wherein determining the one or more values for the one or more restaurant reservation parameters further comprises:
prompting the user to provide the one or more values for the one or more restaurant reservation parameters; and
receiving, from the user, an input specifying the one or more values for the one or more restaurant reservation parameters.
35. The method of claim 34 , wherein prompting the user to provide the one or more values for the one or more restaurant reservation parameters further comprises:
displaying, in the communication session, a plurality of candidate values; and
receiving, from the user, the input specifying the one or more values for the one or more restaurant reservation parameters further comprises receiving, from the user, a selection of a candidate value of the plurality of candidate values.
36. The method of claim 27 , comprising:
performing a semantic analysis on the user request to determine whether the user request includes a request for the digital assistant to facilitate the restaurant reservation.
37. The method of claim 36 , wherein performing the semantic analysis includes performing natural language processing.
38. The method of claim 36 , wherein performing the semantic analysis further comprises:
determining whether one or more words of the user request are associated with a restaurant domain; and
determining whether the one or more words of the user request are associated with a task of reserving a table identified by the restaurant domain.
39. The method of claim 36 , comprising:
identifying a restaurant of a plurality of restaurants based on the semantic analysis of the user request.
US17/732,011
2010-01-18
2022-04-28
Intelligent automated assistant
Active
2031-01-12
US12165635B2
( en )
Priority Applications (1)
Application Number
Priority Date
Filing Date
Title
US17/732,011
US12165635B2
( en )
2010-01-18
2022-04-28
Intelligent automated assistant
Applications Claiming Priority (6)
Application Number
Priority Date
Filing Date
Title
US29577410P
2010-01-18
2010-01-18
US12/987,982
US9318108B2
( en )
2010-01-18
2011-01-10
Intelligent automated assistant
US13/492,809
US9548050B2
( en )
2010-01-18
2012-06-09
Intelligent automated assistant
US15/394,162
US10706841B2
( en )
2010-01-18
2016-12-29
Task flow identification based on user intent
US16/879,643
US11423886B2
( en )
2010-01-18
2020-05-20
Task flow identification based on user intent
US17/732,011
US12165635B2
( en )
2010-01-18
2022-04-28
Intelligent automated assistant
Related Parent Applications (1)
Application Number
Title
Priority Date
Filing Date
US16/879,643
Continuation
US11423886B2
( en )
2010-01-18
2020-05-20
Task flow identification based on user intent
Publications (2)
Publication Number
Publication Date
US20220254338A1
US20220254338A1 ( en )
2022-08-11
US12165635B2
true
US12165635B2 ( en )
2024-12-10
Family
ID=44304930
Family Applications (17)
Application Number
Title
Priority Date
Filing Date
US12/987,982
Active
2032-03-26
US9318108B2
( en )
2006-09-08
2011-01-10
Intelligent automated assistant
US13/492,809
Active
US9548050B2
( en )
2010-01-18
2012-06-09
Intelligent automated assistant
US13/725,481
Active
US8903716B2
( en )
2006-09-08
2012-12-21
Personalized vocabulary for digital assistant
US13/725,656
Active
US8660849B2
( en )
2006-09-08
2012-12-21
Prioritizing selection criteria by automated assistant
US13/725,761
Active
US9117447B2
( en )
2006-09-08
2012-12-21
Using event alert text as input to an automated assistant
US13/725,713
Active
US8799000B2
( en )
2006-09-08
2012-12-21
Disambiguation based on active input elicitation by intelligent automated assistant
US13/725,616
Active
US8892446B2
( en )
2006-09-08
2012-12-21
Service orchestration for intelligent automated assistant
US13/725,550
Active
US8942986B2
( en )
2006-09-08
2012-12-21
Determining user intent based on ontologies of domains
US13/725,742
Active
US8706503B2
( en )
2006-09-08
2012-12-21
Intent deduction based on previous user interactions with voice assistant
US13/725,512
Active
US8670979B2
( en )
2006-09-08
2012-12-21
Active input elicitation by intelligent automated assistant
US13/784,694
Active
US8930191B2
( en )
2006-09-08
2013-03-04
Paraphrasing of user requests and results by automated digital assistant
US13/784,707
Active
US8731942B2
( en )
2006-09-08
2013-03-04
Maintaining context information between user interactions with a voice assistant
US15/394,162
Active
2032-02-16
US10706841B2
( en )
2010-01-18
2016-12-29
Task flow identification based on user intent
US16/879,643
Active
US11423886B2
( en )
2010-01-18
2020-05-20
Task flow identification based on user intent
US17/732,011
Active
2031-01-12
US12165635B2
( en )
2010-01-18
2022-04-28
Intelligent automated assistant
US17/882,457
Active
2031-12-01
US12431128B2
( en )
2010-01-18
2022-08-05
Task flow identification based on user intent
US19/329,341
Pending
US20260011323A1
( en )
2010-01-18
2025-09-15
Task flow identification based on user intent
Family Applications Before (14)
Application Number
Title
Priority Date
Filing Date
US12/987,982
Active
2032-03-26
US9318108B2
( en )
2006-09-08
2011-01-10
Intelligent automated assistant
US13/492,809
Active
US9548050B2
( en )
2010-01-18
2012-06-09
Intelligent automated assistant
US13/725,481
Active
US8903716B2
( en )
2006-09-08
2012-12-21
Personalized vocabulary for digital assistant
US13/725,656
Active
US8660849B2
( en )
2006-09-08
2012-12-21
Prioritizing selection criteria by automated assistant
US13/725,761
Active
US9117447B2
( en )
2006-09-08
2012-12-21
Using event alert text as input to an automated assistant
US13/725,713
Active
US8799000B2
( en )
2006-09-08
2012-12-21
Disambiguation based on active input elicitation by intelligent automated assistant
US13/725,616
Active
US8892446B2
( en )
2006-09-08
2012-12-21
Service orchestration for intelligent automated assistant
US13/725,550
Active
US8942986B2
( en )
2006-09-08
2012-12-21
Determining user intent based on ontologies of domains
US13/725,742
Active
US8706503B2
( en )
2006-09-08
2012-12-21
Intent deduction based on previous user interactions with voice assistant
US13/725,512
Active
US8670979B2
( en )
2006-09-08
2012-12-21
Active input elicitation by intelligent automated assistant
US13/784,694
Active
US8930191B2
( en )
2006-09-08
2013-03-04
Paraphrasing of user requests and results by automated digital assistant
US13/784,707
Active
US8731942B2
( en )
2006-09-08
2013-03-04
Maintaining context information between user interactions with a voice assistant
US15/394,162
Active
2032-02-16
US10706841B2
( en )
2010-01-18
2016-12-29
Task flow identification based on user intent
US16/879,643
Active
US11423886B2
( en )
2010-01-18
2020-05-20
Task flow identification based on user intent
Family Applications After (2)
Application Number
Title
Priority Date
Filing Date
US17/882,457
Active
2031-12-01
US12431128B2
( en )
2010-01-18
2022-08-05
Task flow identification based on user intent
US19/329,341
Pending
US20260011323A1
( en )
2010-01-18
2025-09-15
Task flow identification based on user intent
Country Status (12)
Country
Link
US
( 17 )
US9318108B2
( en )
EP
( 3 )
EP2526511A4
( en )
JP
( 16 )
JP2013517566A
( en )
KR
( 19 )
KR101593739B1
( en )
CN
( 2 )
CN105808200B
( en )
AU
( 1 )
AU2011205426B2
( en )
BR
( 10 )
BR122012028969B1
( en )
CA
( 14 )
CA2793248C
( en )
GB
( 1 )
GB2490444A
( en )
MX
( 6 )
MX2012008369A
( en )
RU
( 11 )
RU2544787C2
( en )
WO
( 1 )
WO2011088053A2
( en )
Cited By (2)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US20220383864A1
( en )
*
2010-01-18
2022-12-01
Apple Inc.
Task flow identification based on user intent
US20240296278A1
( en )
*
2023-03-03
2024-09-05
Microsoft Technology Licensing, Llc
Efficient multi-turn generative ai model suggested message generation
Families Citing this family (2556)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
WO2001013255A2
( en )
*
1999-08-13
2001-02-22
Pixo, Inc.
Displaying and traversing links in character array
US7516190B2
( en )
*
2000-02-04
2009-04-07
Parus Holdings, Inc.
Personal voice-based information retrieval system
US8645137B2
( en )
2000-03-16
2014-02-04
Apple Inc.
Fast, language-independent method for user authentication by voice
US7287071B2
( en )
*
2000-09-28
2007-10-23
Vignette Corporation
Transaction management system
ITFI20010199A1
( en )
2001-10-22
2003-04-22
Riccardo Vieri
SYSTEM AND METHOD TO TRANSFORM TEXTUAL COMMUNICATIONS INTO VOICE AND SEND THEM WITH AN INTERNET CONNECTION TO ANY TELEPHONE SYSTEM
US7373300B1
( en )
*
2002-12-18
2008-05-13
At&T Corp.
System and method of providing a spoken dialog interface to a website
US7669134B1
( en )
*
2003-05-02
2010-02-23
Apple Inc.
Method and apparatus for displaying information during an instant messaging session
US10032452B1
( en )
2016-12-30
2018-07-24
Google Llc
Multimodal transmission of packetized data
WO2006128183A2
( en )
2005-05-27
2006-11-30
Schwegman, Lundberg, Woessner & Kluth, P.A.
Method and apparatus for cross-referencing important ip relationships
US8677377B2
( en )
*
2005-09-08
2014-03-18
Apple Inc.
Method and apparatus for building an intelligent automated assistant
US7633076B2
( en )
2005-09-30
2009-12-15
Apple Inc.
Automated response to and sensing of user activity in portable devices
US11019161B2
( en )
2005-10-26
2021-05-25
Cortica, Ltd.
System and method for profiling users interest based on multimedia content analysis
US9330189B2
( en )
2005-10-26
2016-05-03
Cortica, Ltd.
System and method for capturing a multimedia content item by a mobile device and matching sequentially relevant content to the multimedia content item
US10621988B2
( en )
2005-10-26
2020-04-14
Cortica Ltd
System and method for speech to text translation using cores of a natural liquid architecture system
US9384196B2
( en )
2005-10-26
2016-07-05
Cortica, Ltd.
Signature generation for multimedia deep-content-classification by a large-scale matching system and method thereof
US10585934B2
( en )
2005-10-26
2020-03-10
Cortica Ltd.
Method and system for populating a concept database with respect to user identifiers
US11361014B2
( en )
2005-10-26
2022-06-14
Cortica Ltd.
System and method for completing a user profile
US9646005B2
( en )
2005-10-26
2017-05-09
Cortica, Ltd.
System and method for creating a database of multimedia content elements assigned to users
US9191626B2
( en )
2005-10-26
2015-11-17
Cortica, Ltd.
System and methods thereof for visual analysis of an image on a web-page and matching an advertisement thereto
US9286623B2
( en )
2005-10-26
2016-03-15
Cortica, Ltd.
Method for determining an area within a multimedia content element over which an advertisement can be displayed
US9953032B2
( en )
2005-10-26
2018-04-24
Cortica, Ltd.
System and method for characterization of multimedia content signals using cores of a natural liquid architecture system
US10535192B2
( en )
2005-10-26
2020-01-14
Cortica Ltd.
System and method for generating a customized augmented reality environment to a user
US10387914B2
( en )
2005-10-26
2019-08-20
Cortica, Ltd.
Method for identification of multimedia content elements and adding advertising content respective thereof
US9477658B2
( en )
2005-10-26
2016-10-25
Cortica, Ltd.
Systems and method for speech to speech translation using cores of a natural liquid architecture system
US10360253B2
( en )
2005-10-26
2019-07-23
Cortica, Ltd.
Systems and methods for generation of searchable structures respective of multimedia data content
US10191976B2
( en )
2005-10-26
2019-01-29
Cortica, Ltd.
System and method of detecting common patterns within unstructured data elements retrieved from big data sources
US9529984B2
( en )
2005-10-26
2016-12-27
Cortica, Ltd.
System and method for verification of user identification based on multimedia content elements
US11216498B2
( en )
2005-10-26
2022-01-04
Cortica, Ltd.
System and method for generating signatures to three-dimensional multimedia data elements
US11003706B2
( en )
2005-10-26
2021-05-11
Cortica Ltd
System and methods for determining access permissions on personalized clusters of multimedia content elements
US8818916B2
( en )
2005-10-26
2014-08-26
Cortica, Ltd.
System and method for linking multimedia data elements to web pages
US11620327B2
( en )
2005-10-26
2023-04-04
Cortica Ltd
System and method for determining a contextual insight and generating an interface with recommendations based thereon
US9218606B2
( en )
2005-10-26
2015-12-22
Cortica, Ltd.
System and method for brand monitoring and trend analysis based on deep-content-classification
US10848590B2
( en )
2005-10-26
2020-11-24
Cortica Ltd
System and method for determining a contextual insight and providing recommendations based thereon
US9235557B2
( en )
2005-10-26
2016-01-12
Cortica, Ltd.
System and method thereof for dynamically associating a link to an information resource with a multimedia content displayed in a web-page
US10949773B2
( en )
2005-10-26
2021-03-16
Cortica, Ltd.
System and methods thereof for recommending tags for multimedia content elements based on context
US9558449B2
( en )
2005-10-26
2017-01-31
Cortica, Ltd.
System and method for identifying a target area in a multimedia content element
US9087049B2
( en )
*
2005-10-26
2015-07-21
Cortica, Ltd.
System and method for context translation of natural language
US9031999B2
( en )
2005-10-26
2015-05-12
Cortica, Ltd.
System and methods for generation of a concept based database
US10614626B2
( en )
2005-10-26
2020-04-07
Cortica Ltd.
System and method for providing augmented reality challenges
US10691642B2
( en )
2005-10-26
2020-06-23
Cortica Ltd
System and method for enriching a concept database with homogenous concepts
US10607355B2
( en )
2005-10-26
2020-03-31
Cortica, Ltd.
Method and system for determining the dimensions of an object shown in a multimedia content item
US10380164B2
( en )
2005-10-26
2019-08-13
Cortica, Ltd.
System and method for using on-image gestures and multimedia content elements as search queries
US11604847B2
( en )
2005-10-26
2023-03-14
Cortica Ltd.
System and method for overlaying content on a multimedia content element based on user interest
US9489431B2
( en )
2005-10-26
2016-11-08
Cortica, Ltd.
System and method for distributed search-by-content
US10380623B2
( en )
2005-10-26
2019-08-13
Cortica, Ltd.
System and method for generating an advertisement effectiveness performance score
US11032017B2
( en )
2005-10-26
2021-06-08
Cortica, Ltd.
System and method for identifying the context of multimedia content elements
US10180942B2
( en )
2005-10-26
2019-01-15
Cortica Ltd.
System and method for generation of concept structures based on sub-concepts
US9767143B2
( en )
2005-10-26
2017-09-19
Cortica, Ltd.
System and method for caching of concept structures
US10776585B2
( en )
2005-10-26
2020-09-15
Cortica, Ltd.
System and method for recognizing characters in multimedia content
US10372746B2
( en )
2005-10-26
2019-08-06
Cortica, Ltd.
System and method for searching applications using multimedia content elements
US10698939B2
( en )
2005-10-26
2020-06-30
Cortica Ltd
System and method for customizing images
US10635640B2
( en )
2005-10-26
2020-04-28
Cortica, Ltd.
System and method for enriching a concept database
US9466068B2
( en )
2005-10-26
2016-10-11
Cortica, Ltd.
System and method for determining a pupillary response to a multimedia data element
US9639532B2
( en )
2005-10-26
2017-05-02
Cortica, Ltd.
Context-based analysis of multimedia content items using signatures of multimedia elements and matching concepts
US8312031B2
( en )
2005-10-26
2012-11-13
Cortica Ltd.
System and method for generation of complex signatures for multimedia data content
US10742340B2
( en )
2005-10-26
2020-08-11
Cortica Ltd.
System and method for identifying the context of multimedia content elements displayed in a web-page and providing contextual filters respective thereto
US10193990B2
( en )
2005-10-26
2019-01-29
Cortica Ltd.
System and method for creating user profiles based on multimedia content
US10380267B2
( en )
2005-10-26
2019-08-13
Cortica, Ltd.
System and method for tagging multimedia content elements
US8326775B2
( en )
2005-10-26
2012-12-04
Cortica Ltd.
Signature generation for multimedia deep-content-classification by a large-scale matching system and method thereof
US11386139B2
( en )
2005-10-26
2022-07-12
Cortica Ltd.
System and method for generating analytics for entities depicted in multimedia content
US9396435B2
( en )
2005-10-26
2016-07-19
Cortica, Ltd.
System and method for identification of deviations from periodic behavior patterns in multimedia content
US9256668B2
( en )
2005-10-26
2016-02-09
Cortica, Ltd.
System and method of detecting common patterns within unstructured data elements retrieved from big data sources
US9372940B2
( en )
2005-10-26
2016-06-21
Cortica, Ltd.
Apparatus and method for determining user attention using a deep-content-classification (DCC) system
US8266185B2
( en )
2005-10-26
2012-09-11
Cortica Ltd.
System and methods thereof for generation of searchable structures respective of multimedia data content
US11403336B2
( en )
2005-10-26
2022-08-02</