U.S. patent application number 16/116112 was filed with the patent office on 2018-12-27 for context-sensitive handling of interruptions.
The applicant listed for this patent is Apple Inc.. Invention is credited to Thomas R. GRUBER, Donald W. PITSCHEL.
Application Number | 20180373487 16/116112 |
Document ID | / |
Family ID | 50487195 |
Filed Date | 2018-12-27 |
View All Diagrams
United States Patent
Application |
20180373487 |
Kind Code |
A1 |
GRUBER; Thomas R. ; et
al. |
December 27, 2018 |
CONTEXT-SENSITIVE HANDLING OF INTERRUPTIONS
Abstract
A list of notification items is received, the list including a
plurality of notification items, wherein each respective one of the
plurality of notification items is associated with a respective
urgency value. An information item is detected. In some
implementations, the information item is a communication (e.g., an
email). In some implementations, the information item is a change
in context of a user. Upon determining that the information item is
relevant to the urgency value of the first notification item, the
urgency value of the first notification item is adjusted. Upon
determining that the adjusted urgency value satisfies the
predetermined threshold, a first audio prompt is provided to a
user.
Inventors: |
GRUBER; Thomas R.; (Santa
Cruz, CA) ; PITSCHEL; Donald W.; (San Francisco,
CA) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Apple Inc. |
Cupertino |
CA |
US |
|
|
Family ID: |
50487195 |
Appl. No.: |
16/116112 |
Filed: |
August 29, 2018 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
14213812 |
Mar 14, 2014 |
10078487 |
|
|
16116112 |
|
|
|
|
61799996 |
Mar 15, 2013 |
|
|
|
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
G06F 3/165 20130101;
G06F 3/167 20130101 |
International
Class: |
G06F 3/16 20060101
G06F003/16 |
Claims
1. A non-transitory computer readable storage medium storing one or
more programs, the one or more programs comprising instructions,
which when executed by one or more processors of an electronic
device, cause the device to: generate a speech output to be
provided to a user of the device; engage in a communication session
with a remote device; while the device is engaged in the
communication session with the remote device: determine an urgency
value of the speech output; determine whether the urgency value of
the speech output satisfies a predetermined threshold; upon
determining that the urgency value of the speech output satisfies
the predetermined threshold, provide the speech output to the user
of the device; and upon determining that the urgency value of the
speech output does not satisfy the predetermined threshold, forgo
providing the speech output to the user of the device.
2. The non-transitory computer readable storage medium of claim 1,
wherein the urgency value of the speech output is based on a
user-configurable criterion associated with the speech output.
3. The non-transitory computer readable storage medium of claim 1,
wherein the urgency value of the speech output is based on a
context of the speech output.
4. The non-transitory computer readable storage medium of claim 1,
wherein the one or more programs further comprise instructions,
which when executed by the one or more processors, cause the device
to: determine whether a mode of operation of the electronic device
satisfies a predetermined mode of operation; and in accordance with
a determination that that the mode of operation of the electronic
device satisfies the predetermined mode of operation, provide the
speech output to the user of the device.
5. The non-transitory computer readable storage medium of claim 4,
wherein the predetermined mode of operation is based on a user
setting of the device.
6. The non-transitory computer readable storage medium of claim
1,wherein the one or more programs further comprise instructions,
which when executed by the one or more processors, cause the device
to: upon determining that the urgency value of the speech output
does not satisfy the predetermined threshold, delay providing the
speech output for a predetermined time.
7. The non-transitory computer readable storage medium of claim 6,
wherein the predetermined time is based on whether the device is
receiving speech input from the user.
8. The non-transitory computer readable storage medium of claim 6,
wherein the one or more programs further comprise instructions,
which when executed by the one or more processors, cause the device
to: after delaying providing the speech output for the
predetermined time: determine whether the communication session has
ended; and in accordance with a determination that the
communication session has ended, provide the speech output to the
user of the device.
9. The non-transitory computer readable storage medium of claim 1,
wherein the one or more programs further comprise instructions,
which when executed by the one or more processors, cause the device
to: upon determining that the urgency value of the speech output
does not satisfy the predetermined threshold, provide a visual
output corresponding to the speech output on a display of the
electronic device.
10. The non-transitory computer readable storage medium of claim 1,
wherein the communication session is a telephone conversation.
11. The non-transitory computer readable storage medium of claim 1,
wherein the one or more programs further comprise instructions,
which when executed by the one or more processors, cause the device
to: receive a request from the user to perform a task; perform the
task requested by the user; and wherein the speech output to be
provided to the user of the device corresponds to the performance
of the task.
12. The non-transitory computer readable storage medium of claim 1,
wherein the one or more programs further comprise instructions,
which when executed by the one or more processors, cause the device
to: while the device is engaged in the communication session:
determine whether the device is currently receiving speech input
from the user; and in accordance with a determination that the
device is not currently receiving speech input from the user,
provide the speech output to the user of the device.
13. The non-transitory computer readable storage medium of claim
12, wherein determining whether the device is currently receiving
speech input from the user includes: determining whether a previous
speech input from the user was received within a predetermined
period of time.
14. The non-transitory computer readable storage medium of claim
12, wherein determining whether the device is currently receiving
speech input from the user includes : determining whether a
strength a characteristic of the speech input exceeds a
predetermined strength threshold.
15. The non-transitory computer readable storage medium of claim 1,
wherein the urgency value of the speech output is based on a
position of the speech output in a list of scheduled speech
outputs.
16. The non-transitory computer readable storage medium of claim 1:
wherein engaging in the communication session with the remote
device includes: audibly providing audio data received from the
remote device to the user; and wherein providing the speech output
to the user of the device includes: temporarily muting the
provision of the audio data received from the remote device; and
outputting the speech output to the user of the device while the
audio data received from the remote device is muted.
17. A method of operating a digital assistant, comprising: at a
device having one or more processors and memory: generating a
speech output to be provided to a user of the device; engaging in a
communication session with a remote device; while the device is
engaged in the communication session with the remote device:
determining an urgency value of the speech output; determining
whether the urgency value of the speech output satisfies a
predetermined threshold; upon determining that the urgency value of
the speech output satisfies the predetermined threshold, providing
the speech output to the user of the device; and upon determining
that the speech output does not satisfy the predetermined
threshold, forgo providing the speech output to the user of the
device.
18. An electronic device, comprising: one or more processors;
memory; and one or more programs, wherein the one or more programs
are stored in the memory and configured to be executed by the one
or more processors, the one or more programs including instructions
for: generating a speech output to be provided to a user of the
device; engaging in a communication session with a remote device;
while the device is engaged in the communication session with the
remote device: determining an urgency value of the speech output;
determining whether the urgency value of the speech output
satisfies a predetermined threshold; upon determining that the
urgency value of the speech output satisfies the predetermined
threshold, providing the speech output to the user of the device;
and upon determining that the speech output does not satisfy the
predetermined threshold, forgo providing the speech output to the
user of the device.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This Application is a continuation of U.S. application Ser.
No. 14/213,812, filed on Mar. 14, 2014, entitled CONTEXT-SENSITIVE
HANDLING OF INTERRUPTIONS, which claims the benefit of U.S.
Provisional Application No. 61/799,996, filed on Mar. 15, 2013,
entitled CONTEXT-SENSITIVE HANDLING OF INTERRUPTIONS, which are
hereby incorporated by reference in their entity for all
purposes.
TECHNICAL FIELD
[0002] The disclosed embodiments relate generally to digital
assistants, and more specifically, to digital assistants that
intelligently handle notifications based on the current
context.
BACKGROUND
[0003] Just like human personal assistants, digital assistants or
virtual assistants can perform requested tasks and provide
requested advice, information, or services. An assistant's ability
to fulfill a user's request is dependent on the assistant's correct
comprehension of the request or instructions. Recent advances in
natural language processing have enabled users to interact with
digital assistants using natural language, in spoken or textual
forms, rather than employing a conventional user interface (e.g.,
menus or programmed commands). Such digital assistants can
interpret the user's input to infer the user's intent, translate
the inferred intent into actionable tasks and parameters, execute
operations or deploy services to perform the tasks, and produce
outputs that are intelligible to the user. Ideally, the outputs
produced by a digital assistant should fulfill the user's intent
expressed during the natural language interaction between the user
and the digital assistant.
[0004] However, many digital assistants are merely reactive, in
that they provide outputs to the user only in response to a user's
requests for information. For example, a digital assistant will
provide driving directions when a user asks for them, will set an
alarm, search the web, or the like. But by only reacting to overt
requests for information, the helpfulness of digital assistants is
inherently limited. Human personal assistants, on the other hand,
are proactive. They can determine what information a person may
need and provide it before it is requested. And they can determine
whether and how to interrupt a user to provide such information in
the most appropriate manner.
[0005] Accordingly, there is a need for methods of operating a
digital assistant that intelligently and intuitively determine
information to provide to a user, without first receiving a
specific request for it, and intelligently determine what
information warrants interrupting a user, and how to present that
information in an appropriate manner.
SUMMARY
[0006] The embodiments described below offer an improved method for
providing audio prompts associated with notification items. In
particular, in some embodiments, a digital assistant dynamically
and intelligently assigns urgency values to the notifications in a
notification list, and provides audio prompts only if the urgency
values satisfy a predetermined threshold. Thus, whereas traditional
notification techniques either provide all notifications to a user,
or none (e.g., if the device has been placed in a silent mode), the
present methods allow a digital assistant to intelligently
determine whether to interrupt a user based on the urgency of the
message and the user's context. Moreover, in some implementations,
the digital assistant detects changing conditions and monitors new
incoming notifications to adjust urgency values of existing
notification items. Thus, the digital assistant provides real-time
triage of the user's notifications, so that truly important
notifications--including notifications that have only recently
become important because of changing conditions or newly received
information--are not missed simply because the user didn't want to
be interrupted with notifications of sports scores updates during a
business lunch.
[0007] The embodiments disclosed herein provide methods, systems,
computer readable storage medium and user interfaces for a digital
assistant to intelligently and dynamically determine whether to
provide a speech output. The method includes providing a list of
notification items, the list including a plurality of notification
items, wherein each respective one of the plurality of notification
items is associated with a respective urgency value. The method
further includes detecting an information item, and determining
whether the information item is relevant to an urgency value of a
first notification item of the plurality of notification items. The
method further includes, upon determining that the information item
is relevant to the urgency value of the first notification item,
adjusting the urgency value of the first notification item. The
method further includes determining whether the adjusted urgency
value of the first notification item satisfies a predetermined
threshold, and upon determining that the adjusted urgency value
satisfies the predetermined threshold, providing a first audio
prompt to a user.
[0008] In some embodiments, the method further includes
establishing the predetermined threshold in accordance with a
location of the device. In some embodiments, the method further
includes establishing the predetermined threshold in accordance
with a time of day. In some embodiments, the method further
includes establishing the predetermined threshold in accordance
with a calendar item associated with a current time. In some
embodiments, the method further includes establishing the
predetermined threshold in accordance with a user setting of the
device.
[0009] In some embodiments, the information item is one of the
group consisting of: an email; a voicemail; and a text message. In
some embodiments, the information item is an application
notification. In some embodiments, the information item is a change
in a context of the device.
[0010] In some embodiments, the method further includes detecting
two information items, including at least a communication and a
change in a context of the device.
[0011] In some implementations, the respective urgency values are
based on one or more of: a time associated with the respective
notification item; a location associated with the respective
notification item; and content of the respective notification
item.
[0012] In some embodiments, the method further includes determining
a topic of importance to a user of the device, and assigning at
least one of the respective urgency values to a respective
notification item based on a determination that the respective
notification item corresponds to the topic of importance. In some
implementations, the topic of importance is determined by the
device automatically without human intervention.
[0013] In some implementations, determining whether the information
item is relevant to the urgency value of the first notification
item includes determining that the information item corresponds to
a change to a location associated with the first notification item.
In some implementations, determining whether the information item
is relevant to the urgency value of the first notification item
includes determining that the information item corresponds to a
change in a time associated with the first notification item.
[0014] In some embodiments, the method further includes, upon
determining that the adjusted urgency value does not satisfy the
predetermined threshold, delaying providing the audio prompt to the
user.
[0015] In some embodiments, detecting the information item includes
receiving an incoming communication, and determining whether the
information item is relevant to the urgency value of the first
notification item is performed in response to receiving the
incoming communication.
[0016] In some embodiments, the method further includes determining
whether the user has acknowledged the first audio prompt; and upon
determining that the user has not acknowledged the first audio
prompt, providing a second audio prompt to the user, the second
audio prompt being different from the first audio prompt. In some
embodiments, the second audio prompt is louder than the first audio
prompt. In some embodiments, the second audio prompt is longer than
the first audio prompt. In some embodiments, the first audio prompt
is a first ringtone and the second audio prompt is a second
ringtone different from the first.
[0017] In some embodiments, the method further includes determining
whether the user has acknowledged the second audio prompt; and upon
determining that the user has not acknowledged the second audio
prompt, providing a third audio prompt to the user, the third audio
prompt being different from the first audio prompt and the second
audio prompt. In some embodiments, the third audio prompt is a
speech output. In some embodiments, the first audio prompt is a
ringtone, the second audio prompt is a first speech output of a
first volume, and the third audio prompt is a second speech output
of a second volume louder than the first volume. In some
embodiments, the first audio prompt is a ringtone, the second audio
prompt is a first speech output of a first length, and the third
audio prompt is a second speech output of a second length longer
than the first length.
[0018] In some embodiments, the method further includes
incorporating information from the information item into the first
notification item.
[0019] In accordance with some embodiments, an electronic device
includes a display, a touch-sensitive surface, optionally one or
more sensors to detect intensity of contacts with the
touch-sensitive surface, one or more processors, memory, and one or
more programs; the one or more programs are stored in the memory
and configured to be executed by the one or more processors and the
one or more programs include instructions for performing the
operations of any of the methods described above. In accordance
with some embodiments, a computer readable storage medium has
stored therein instructions which when executed by an electronic
device with a display, a touch-sensitive surface, and optionally
one or more sensors to detect intensity of contacts with the
touch-sensitive surface, cause the device to perform the operations
of any of the methods referred described above. In accordance with
some embodiments, an electronic device includes: a display, a
touch-sensitive surface, and optionally one or more sensors to
detect intensity of contacts with the touch-sensitive surface; and
means for performing the operations of any of the methods described
above. In accordance with some embodiments, an information
processing apparatus, for use in an electronic device with a
display and a touch-sensitive surface, optionally one or more
sensors to detect intensity of contacts with the touch-sensitive
surface, includes means for performing the operations of any of the
methods described above.
[0020] The details of one or more embodiments of the subject matter
described in this specification are set forth in the accompanying
drawings and the description below. Other features, aspects, and
advantages of the subject matter will become apparent from the
description, the drawings, and the claims.
BRIEF DESCRIPTION OF THE DRAWINGS
[0021] FIG. 1 is a block diagram illustrating an environment in
which a digital assistant operates in accordance with some
embodiments.
[0022] FIG. 2 is a block diagram illustrating a digital assistant
client system in accordance with some embodiments.
[0023] FIG. 3A is a block diagram illustrating a digital assistant
system or a server portion thereof in accordance with some
embodiments.
[0024] FIG. 3B is a block diagram illustrating functions of the
digital assistant shown in FIG. 3A in accordance with some
embodiments.
[0025] FIG. 3C is a diagram of a portion of an ontology in
accordance with some embodiments.
[0026] FIGS. 4A-4C illustrate exemplary scenarios in which a
digital assistant determines whether or not to provide a speech
output in accordance with some embodiments.
[0027] FIGS. 5A-5D are flow diagrams of an exemplary method
implemented by a digital assistant for determining whether or not
to provide a speech output to a user based on a determination of
whether or not the device is currently receiving speech input from
the user, as well as the urgency of the speech output, in
accordance with some embodiments.
[0028] FIG. 6 is a flow diagram of an exemplary method implemented
by a digital assistant for managing a notification list, in
accordance with some embodiments.
[0029] Like reference numerals refer to corresponding parts
throughout the drawings.
DESCRIPTION OF EMBODIMENTS
[0030] FIG. 1 is a block diagram of an operating environment 100 of
a digital assistant according to some embodiments. The terms
"digital assistant," "virtual assistant," "intelligent automated
assistant," or "automatic digital assistant," refer to any
information processing system that interprets natural language
input in spoken and/or textual form to infer user intent, and
performs actions based on the inferred user intent. For example, to
act on a inferred user intent, the system can perform one or more
of the following: identifying a task flow with steps and parameters
designed to accomplish the inferred user intent, inputting specific
requirements from the inferred user intent into the task flow;
executing the task flow by invoking programs, methods, services,
APIs, or the like; and generating output responses to the user in
an audible (e.g. speech) and/or visual form.
[0031] Specifically, a digital assistant is capable of accepting a
user request at least partially in the form of a natural language
command, request, statement, narrative, and/or inquiry. Typically,
the user request seeks either an informational answer or
performance of a task by the digital assistant. A satisfactory
response to the user request is either provision of the requested
informational answer, performance of the requested task, or a
combination of the two. For example, a user may ask the digital
assistant a question, such as "Where am I right now?" Based on the
user's current location, the digital assistant may answer, "You are
in Central Park near the west gate." The user may also request the
performance of a task, for example, "Please invite my friends to my
girlfriend's birthday party next week." In response, the digital
assistant may acknowledge the request by saying "Yes, right away,"
and then send a suitable calendar invite on behalf of the user to
each of the user' friends listed in the user's electronic address
book. During performance of a requested task, the digital assistant
sometimes interacts with the user in a continuous dialogue
involving multiple exchanges of information over an extended period
of time. There are numerous other ways of interacting with a
digital assistant to request information or performance of various
tasks. In addition to providing verbal responses and taking
programmed actions, the digital assistant also provides responses
in other visual or audio forms, e.g., as text, alerts, music,
videos, animations, etc.
[0032] An example of a digital assistant is described in
Applicant's U.S. Utility application Ser. No. 12/987,982 for
"Intelligent Automated Assistant," filed Jan. 10, 2011, the entire
disclosure of which is incorporated herein by reference.
[0033] As shown in FIG. 1, in some embodiments, a digital assistant
is implemented according to a client-server model. The digital
assistant includes a client-side portion 102a, 102b (hereafter "DA
client 102") executed on a user device 104a, 104b, and a
server-side portion 106 (hereafter "DA server 106") executed on a
server system 108. The DA client 102 communicates with the DA
server 106 through one or more networks 110. The DA client 102
provides client-side functionalities such as user-facing input and
output processing and communications with the DA-server 106. The DA
server 106 provides server-side functionalities for any number of
DA-clients 102 each residing on a respective user device 104.
[0034] In some embodiments, the DA server 106 includes a
client-facing I/O interface 112, one or more processing modules
114, data and models 116, and an I/O interface to external services
118. The client-facing I/O interface facilitates the client-facing
input and output processing for the digital assistant server 106.
The one or more processing modules 114 utilize the data and models
116 to determine the user's intent based on natural language input
and perform task execution based on inferred user intent. In some
embodiments, the DA-server 106 communicates with external services
120 through the network(s) 110 for task completion or information
acquisition. The I/O interface to external services 118 facilitates
such communications.
[0035] Examples of the user device 104 include, but are not limited
to, a handheld computer, a personal digital assistant (PDA), a
tablet computer, a laptop computer, a desktop computer, a cellular
telephone, a smart phone, an enhanced general packet radio service
(EGPRS) mobile phone, a media player, a navigation device, a game
console, a television, a remote control, or a combination of any
two or more of these data processing devices or other data
processing devices. More details on the user device 104 are
provided in reference to an exemplary user device 104 shown in FIG.
2.
[0036] Examples of the communication network(s) 110 include local
area networks ("LAN") and wide area networks ("WAN"), e.g., the
Internet. The communication network(s) 110 may be implemented using
any known network protocol, including various wired or wireless
protocols, such as e.g., Ethernet, Universal Serial Bus (USB),
FIREWIRE, Global System for Mobile Communications (GSM), Enhanced
Data GSM Environment (EDGE), code division multiple access (CDMA),
time division multiple access (TDMA), Bluetooth, Wi-Fi, voice over
Internet Protocol (VoIP), Wi-MAX, or any other suitable
communication protocol.
[0037] The server system 108 is implemented on one or more
standalone data processing apparatus or a distributed network of
computers. In some embodiments, the server system 108 also employs
various virtual devices and/or services of third party service
providers (e.g., third-party cloud service providers) to provide
the underlying computing resources and/or infrastructure resources
of the server system 108.
[0038] Although the digital assistant shown in FIG. 1 includes both
a client-side portion (e.g., the DA-client 102) and a server-side
portion (e.g., the DA-server 106), in some embodiments, the
functions of a digital assistant is implemented as a standalone
application installed on a user device. In addition, the divisions
of functionalities between the client and server portions of the
digital assistant can vary in different embodiments. For example,
in some embodiments, the DA client is a thin-client that provides
only user-facing input and output processing functions, and
delegates all other functionalities of the digital assistant to a
backend server.
[0039] FIG. 2 is a block diagram of a user-device 104 in accordance
with some embodiments. The user device 104 includes a memory
interface 202, one or more processors 204, and a peripherals
interface 206. The various components in the user device 104 are
coupled by one or more communication buses or signal lines. The
user device 104 includes various sensors, subsystems, and
peripheral devices that are coupled to the peripherals interface
206. The sensors, subsystems, and peripheral devices gather
information and/or facilitate various functionalities of the user
device 104.
[0040] For example, a motion sensor 210, a light sensor 212, and a
proximity sensor 214 are coupled to the peripherals interface 206
to facilitate orientation, light, and proximity sensing functions.
One or more other sensors 216, such as a positioning system (e.g.,
GPS receiver), a temperature sensor, a biometric sensor, a gyro, a
compass, an accelerometer, and the like, are also connected to the
peripherals interface 206, to facilitate related
functionalities.
[0041] In some embodiments, a camera subsystem 220 and an optical
sensor 222 are utilized to facilitate camera functions, such as
taking photographs and recording video clips. Communication
functions are facilitated through one or more wired and/or wireless
communication subsystems 224, which can include various
communication pods, radio frequency receivers and transmitters,
and/or optical (e.g., infrared) receivers and transmitters. An
audio subsystem 226 is coupled to speakers 228 and a microphone 230
to facilitate voice-enabled functions, such as voice recognition,
voice replication, digital recording, and telephony functions.
[0042] In some embodiments, an I/O subsystem 240 is also coupled to
the peripheral interface 206. The I/O subsystem 240 includes a
touch screen controller 242 and/or other input controller(s) 244.
The touch-screen controller 242 is coupled to a touch screen 246.
The touch screen 246 and the touch screen controller 242 can, for
example, detect contact and movement or break thereof using any of
a plurality of touch sensitivity technologies, such as capacitive,
resistive, infrared, surface acoustic wave technologies, proximity
sensor arrays, and the like. The other input controller(s) 244 can
be coupled to other input/control devices 248, such as one or more
buttons, rocker switches, thumb-wheel, infrared port, USB port,
and/or a pointer device such as a stylus.
[0043] In some embodiments, the memory interface 202 is coupled to
memory 250. The memory 250 can include high-speed random access
memory and/or non-volatile memory, such as one or more magnetic
disk storage devices, one or more optical storage devices, and/or
flash memory (e.g., NAND, NOR).
[0044] In some embodiments, the memory 250 stores an operating
system 252, a communication module 254, a user interface module
256, a sensor processing module 258, a phone module 260, and
applications 262. The operating system 252 includes instructions
for handling basic system services and for performing hardware
dependent tasks. The communication module 254 facilitates
communicating with one or more additional devices, one or more
computers and/or one or more servers. The user interface module 256
facilitates graphic user interface processing and output processing
using other output channels (e.g., speakers). The sensor processing
module 258 facilitates sensor-related processing and functions. The
phone module 260 facilitates phone-related processes and functions.
The application module 262 facilitates various functionalities of
user applications, such as electronic-messaging, web browsing,
media processing, Navigation, imaging and/or other processes and
functions.
[0045] As described in this specification, the memory 250 also
stores client-side digital assistant instructions (e.g., in a
digital assistant client module 264) and various user data 266
(e.g., user-specific vocabulary data, preference data, and/or other
data such as the user's electronic address book, to-do lists,
shopping lists, etc.) to provide the client-side functionalities of
the digital assistant.
[0046] In various embodiments, the digital assistant client module
264 is capable of accepting voice input (e.g., speech input), text
input, touch input, and/or gestural input through various user
interfaces (e.g., the I/O subsystem 244) of the user device 104.
The digital assistant client module 264 is also capable of
providing output in audio (e.g., speech output), visual, and/or
tactile forms. For example, output can be provided as voice, sound,
alerts, text messages, menus, graphics, videos, animations,
vibrations, and/or combinations of two or more of the above. During
operation, the digital assistant client module 264 communicates
with the digital assistant server using the communication
subsystems 224.
[0047] In some embodiments, the digital assistant client module 264
utilizes the various sensors, subsystems and peripheral devices to
gather additional information from the surrounding environment of
the user device 104 to establish a context associated with a user,
the current user interaction, and/or the current user input. In
some embodiments, the digital assistant client module 264 provides
the context information or a subset thereof with the user input to
the digital assistant server to help infer the user's intent. In
some embodiments, the digital assistant also uses the context
information to determine how to prepare and delivery outputs to the
user.
[0048] In some embodiments, the context information that
accompanies the user input includes sensor information, e.g.,
lighting, ambient noise, ambient temperature, images or videos of
the surrounding environment, etc. In some embodiments, the context
information also includes the physical state of the device, e.g.,
device orientation, device location, device temperature, power
level, speed, acceleration, motion patterns, cellular signals
strength, etc. In some embodiments, information related to the
software state of the user device 104, e.g., running processes,
installed programs, past and present network activities, background
services, error logs, resources usage, etc., of the user device 104
are provided to the digital assistant server as context information
associated with a user input.
[0049] In some embodiments, the DA client module 264 selectively
provides information (e.g., user data 266) stored on the user
device 104 in response to requests from the digital assistant
server. In some embodiments, the digital assistant client module
264 also elicits additional input from the user via a natural
language dialogue or other user interfaces upon request by the
digital assistant server 106. The digital assistant client module
264 passes the additional input to the digital assistant server 106
to help the digital assistant server 106 in intent deduction and/or
fulfillment of the user's intent expressed in the user request.
[0050] In various embodiments, the memory 250 includes additional
instructions or fewer instructions. Furthermore, various functions
of the user device 104 may be implemented in hardware and/or in
firmware, including in one or more signal processing and/or
application specific integrated circuits.
[0051] FIG. 3A is a block diagram of an example digital assistant
system 300 in accordance with some embodiments. In some
embodiments, the digital assistant system 300 is implemented on a
standalone computer system. In some embodiments, the digital
assistant system 300 is distributed across multiple computers. In
some embodiments, some of the modules and functions of the digital
assistant are divided into a server portion and a client portion,
where the client portion resides on a user device (e.g., the user
device 104) and communicates with the server portion (e.g., the
server system 108) through one or more networks, e.g., as shown in
FIG. 1. In some embodiments, the digital assistant system 300 is an
embodiment of the server system 108 (and/or the digital assistant
server 106) shown in FIG. 1. It should be noted that the digital
assistant system 300 is only one example of a digital assistant
system, and that the digital assistant system 300 may have more or
fewer components than shown, may combine two or more components, or
may have a different configuration or arrangement of the
components. The various components shown in FIG. 3A may be
implemented in hardware, software instructions for execution by one
or more processors, firmware, including one or more signal
processing and/or application specific integrated circuits, or a
combination of thereof.
[0052] The digital assistant system 300 includes memory 302, one or
more processors 304, an input/output (I/O) interface 306, and a
network communications interface 308. These components communicate
with one another over one or more communication buses or signal
lines 310.
[0053] In some embodiments, the memory 302 includes a
non-transitory computer readable medium, such as high-speed random
access memory and/or a non-volatile computer readable storage
medium (e.g., one or more magnetic disk storage devices, flash
memory devices, or other non-volatile solid-state memory
devices).
[0054] In some embodiments, the I/O interface 306 couples
input/output devices 316 of the digital assistant system 300, such
as displays, a keyboards, touch screens, and microphones, to the
user interface module 322. The I/O interface 306, in conjunction
with the user interface module 322, receive user inputs (e.g.,
voice input, keyboard inputs, touch inputs, etc.) and process them
accordingly. In some embodiments, e.g., when the digital assistant
is implemented on a standalone user device, the digital assistant
system 300 includes any of the components and I/O and communication
interfaces described with respect to the user device 104 in FIG. 2.
In some embodiments, the digital assistant system 300 represents
the server portion of a digital assistant implementation, and
interacts with the user through a client-side portion residing on a
user device (e.g., the user device 104 shown in FIG. 2).
[0055] In some embodiments, the network communications interface
308 includes wired communication port(s) 312 and/or wireless
transmission and reception circuitry 314. The wired communication
port(s) receive and send communication signals via one or more
wired interfaces, e.g., Ethernet, Universal Serial Bus (USB),
FIREWIRE, etc. The wireless circuitry 314 receives and sends RF
signals and/or optical signals from/to communications networks and
other communications devices. The wireless communications may use
any of a plurality of communications standards, protocols and
technologies, such as GSM, EDGE, CDMA, TDMA, Bluetooth, Wi-Fi,
VoIP, Wi-MAX, or any other suitable communication protocol. The
network communications interface 308 enables communication between
the digital assistant system 300 with networks, such as the
Internet, an intranet and/or a wireless network, such as a cellular
telephone network, a wireless local area network (LAN) and/or a
metropolitan area network (MAN), and other devices.
[0056] In some embodiments, memory 302, or the computer readable
storage media of memory 302, stores programs, modules,
instructions, and data structures including all or a subset of: an
operating system 318, a communications module 320, a user interface
module 322, one or more applications 324, and a digital assistant
module 326. The one or more processors 304 execute these programs,
modules, and instructions, and reads/writes from/to the data
structures.
[0057] The operating system 318 (e.g., Darwin, RTXC, LINUX, UNIX,
OS X, WINDOWS, or an embedded operating system such as VxWorks)
includes various software components and/or drivers for controlling
and managing general system tasks (e.g., memory management, storage
device control, power management, etc.) and facilitates
communications between various hardware, firmware, and software
components.
[0058] The communications module 320 facilitates communications
between the digital assistant system 300 with other devices over
the network communications interface 308. For example, the
communication module 320 may communicate with the communication
interface 254 of the device 104 shown in FIG. 2.sub.-- The
communications module 320 also includes various components for
handling data received by the wireless circuitry 314 and/or wired
communications port 312.
[0059] The user interface module 322 receives commands and/or
inputs from a user via the I/O interface 306 (e.g., from a
keyboard, touch screen, pointing device, controller, and/or
microphone), and generates user interface objects on a display. The
user interface module 322 also prepares and delivers outputs (e.g.,
speech, sound, animation, text, icons, vibrations, haptic feedback,
and light, etc.) to the user via the I/O interface 306 (e.g.,
through displays, audio channels, speakers, and touch-pads,
etc.).
[0060] The applications 324 include programs and/or modules that
are configured to be executed by the one or more processors 304.
For example, if the digital assistant system is implemented on a
standalone user device, the applications 324 may include user
applications, such as games, a calendar application, a navigation
application, or an email application. If the digital assistant
system 300 is implemented on a server farm, the applications 324
may include resource management applications, diagnostic
applications, or scheduling applications, for example.
[0061] The memory 302 also stores the digital assistant module (or
the server portion of a digital assistant) 326. In some
embodiments, the digital assistant module 326 includes the
following sub-modules, or a subset or superset thereof: an
input/output processing module 328, a speech-to-text (STT)
processing module 330, a natural language processing module 332, a
dialogue flow processing module 334, a task flow processing module
336, a service processing module 338, and an interruption handling
module 340. Each of these modules has access to one or more of the
following data and models of the digital assistant 326, or a subset
or superset thereof: ontology 360, vocabulary index 344, user data
348, task flow models 354, service models 356, and priority
parameters database 358.
[0062] In some embodiments, using the processing modules, data, and
models implemented in the digital assistant module 326, the digital
assistant performs at least some of the following: identifying a
user's intent expressed in a natural language input received from
the user; actively eliciting and obtaining information needed to
fully infer the user's intent (e.g., by disambiguating words,
names, intentions, etc.); determining the task flow for fulfilling
the inferred intent; and executing the task flow to fulfill the
inferred intent. In this specifications, more details regarding the
interruption handling module and its use of the priority parameters
arc provided later.
[0063] In some embodiments, as shown in FIG. 3B, the I/O processing
module 328 interacts with the user through the I/O devices 316 in
FIG. 3A or with a user device (e.g., a user device 104 in FIG. 1)
through the network communications interface 308 in FIG. 3A to
obtain user input (e.g., a speech input) and to provide responses
(e.g., as speech outputs) to the user input. The I/O processing
module 328 optionally obtains context information associated with
the user input from the user device, along with or shortly after
the receipt of the user input. The context information includes
user-specific data, vocabulary, and/or preferences relevant to the
user input. In some embodiments, the context information also
includes software and hardware states of the device (e.g., the user
device 104 in FIG. 1) at the time the user request is received,
and/or information related to the surrounding environment of the
user at the time that the user request was received. In some
embodiments, the I/O processing module 328 also sends follow-up
questions to, and receives answers from, the user regarding the
user request. When a user request is received by the I/O processing
module 328 and the user request contains a speech input, the I/O
processing module 328 forwards the speech input to the
speech-to-text (STT) processing module 330 for speech-to-text
conversions.
[0064] The speech-to-text processing module 330 receives speech
input (e.g., a user utterance captured in a voice recording)
through the I/O processing module 328. In some embodiments, the
speech-to-text processing module 330 uses various acoustic and
language models to recognize the speech input as a sequence of
phonemes, and ultimately, a sequence of words or tokens written in
one or more languages. The speech-to-text processing module 330 can
be implemented using any suitable speech recognition techniques,
acoustic models, and language models, such as Hidden Markov Models,
Dynamic Time Warping (DTW)-based speech recognition, and other
statistical and/or analytical techniques. In some embodiments, the
speech-to-text processing can be performed at least partially by a
third party service or on the user's device. Once the
speech-to-text processing module 330 obtains the result of the
speech-to-text processing, e.g., a sequence of words or tokens, it
passes the result to the natural language processing module 332 for
intent deduction.
[0065] The natural language processing module 332 ("natural
language processor") of the digital assistant takes the sequence of
words or tokens ("token sequence") generated by the speech-to-text
processing module 330, and attempts to associate the token sequence
with one or more "actionable intents" recognized by the digital
assistant. An "actionable intent" represents a task that can be
performed by the digital assistant, and has an associated task flow
implemented in the task flow models 354. The associated task flow
is a series of programmed actions and steps that the digital
assistant takes in order to perform the task. The scope of a
digital assistant's capabilities is dependent on the number and
variety of task flows that have been implemented and stored in the
task flow models 354, or in other words, on the number and variety
of "actionable intents" that the digital assistant recognizes. The
effectiveness of the digital assistant, however, is also dependent
on the assistant's ability to infer the correct "actionable
intent(s)" from the user request expressed in natural language.
[0066] In some embodiments, in addition to the sequence of words or
tokens obtained from the speech-to-text processing module 330, the
natural language processor 332 also receives context information
associated with the user request, e.g., from the I/O processing
module 328. The natural language processor 332 optionally uses the
context information to clarify, supplement, and/or further define
the information contained in the token sequence received from the
speech-to-text processing module 330. The context information
includes, for example, user preferences, hardware and/or software
states of the user device, sensor information collected before,
during, or shortly after the user request, prior interactions
(e.g., dialogue) between the digital assistant and the user, and
the like. As described in this specification, context information
is dynamic, and can change with time, location, content of the
dialogue, and other factors.
[0067] In some embodiments, the natural language processing is
based on e.g., ontology 360. The ontology 360 is a hierarchical
structure containing many nodes, each node representing either an
"actionable intent" or a "property" relevant to one or more of the
"actionable intents" or other "properties". As noted above, an
"actionable intent" represents a task that the digital assistant is
capable of performing, i.e., it is "actionable" or can be acted on.
A "property" represents a parameter associated with an actionable
intent or a sub-aspect of another property. A linkage between an
actionable intent node and a property node in the ontology 360
defines how a parameter represented by the property node pertains
to the task represented by the actionable intent node.
[0068] In some embodiments, the ontology 360 is made up of
actionable intent nodes and property nodes. Within the ontology
360, each actionable intent node is linked to one or more property
nodes either directly or through one or more intermediate property
nodes. Similarly, each property node is linked to one or more
actionable intent nodes either directly or through one or more
intermediate property nodes. For example, as shown in FIG. 3C, the
ontology 360 may include a "restaurant reservation" node (i.e., an
actionable intent node). Property nodes "restaurant," "date/time"
(for the reservation), and "party size" are each directly linked to
the actionable intent node (i.e., the "restaurant reservation"
node). In addition, property nodes "cuisine," "price range," "phone
number," and "location" are sub-nodes of the property node
"restaurant," and are each linked to the "restaurant reservation"
node (i.e., the actionable intent node) through the intermediate
property node "restaurant." For another example, as shown in FIG.
3C, the ontology 360 may also include a "set reminder" node (i.e.,
another actionable intent node). Property nodes "date/time" (for
the setting the reminder) and "subject" (for the reminder) are each
linked to the "set reminder" node. Since the property "date/time"
is relevant to both the task of making a restaurant reservation and
the task of setting a reminder, the property node "date/time" is
linked to both the "restaurant reservation" node and the "set
reminder" node in the ontology 360.
[0069] An actionable intent node, along with its linked concept
nodes, may be described as a "domain." In the present discussion,
each domain is associated with a respective actionable intent, and
refers to the group of nodes (and the relationships there between)
associated with the particular actionable intent. For example, the
ontology 360 shown in FIG. 3C includes an example of a restaurant
reservation domain 362 and an example of a reminder domain 364
within the ontology 360. The restaurant reservation domain includes
the actionable intent node "restaurant reservation," property nodes
"restaurant," "date/time," and "party size," and sub-property nodes
"cuisine," "price range," "phone number," and "location." The
reminder domain 364 includes the actionable intent node "set
reminder," and property nodes "subject" and "date/time." In some
embodiments, the ontology 360 is made up of many domains. Each
domain may share one or more property nodes with one or more other
domains. For example, the "date/time" property node may be
associated with many different domains (e.g., a scheduling domain,
a travel reservation domain, a movie ticket domain, etc.), in
addition to the restaurant reservation domain 362 and the reminder
domain 364.
[0070] While FIG. 3C illustrates two example domains within the
ontology 360, other domains (or actionable intents) include, for
example, "initiate a phone call," "find directions," "schedule a
meeting," "send a message," and "provide an answer to a question,"
"read a list", "providing navigation instructions," "provide
instructions for a task" and so on. A "send a message" domain is
associated with a "send a message" actionable intent node, and may
further include property nodes such as "recipient(s)", "message
type", and "message body." The property node "recipient" may be
further defined, for example, by the sub-property nodes such as
"recipient name" and "message address."
[0071] In some embodiments, the ontology 360 includes all the
domains (and hence actionable intents) that the digital assistant
is capable of understanding and acting upon. In some embodiments,
the ontology 360 may be modified, such as by adding or removing
entire domains or nodes, or by modifying relationships between the
nodes within the ontology 360.
[0072] In some embodiments, nodes associated with multiple related
actionable intents may be clustered under a "super domain" in the
ontology 360. For example, a "travel" super-domain may include a
cluster of property nodes and actionable intent nodes related to
travels. The actionable intent nodes related to travels may include
"airline reservation," "hotel reservation," "car rental," "get
directions," "find points of interest," and so on. The actionable
intent nodes under the same super domain (e.g., the "travels" super
domain) may have many property nodes in common. For example, the
actionable intent nodes for "airline reservation," "hotel
reservation," "car rental," "get directions," "find points of
interest" may share one or more of the property nodes "start
location," "destination," "departure date/time," "arrival
date/time," and "party size."
[0073] In some embodiments, each node in the ontology 360 is
associated with a set of words and/or phrases that are relevant to
the property or actionable intent represented by the node. The
respective set of words and/or phrases associated with each node is
the so-called "vocabulary" associated with the node. The respective
set of words and/or phrases associated with each node can be stored
in the vocabulary index 344 in association with the property or
actionable intent represented by the node. For example, returning
to FIG. 3B, the vocabulary associated with the node for the
property of "restaurant" may include words such as "food,"
"drinks," "cuisine," "hungry," "eat," "pizza," "fast food," "meal,"
and so on. For another example, the vocabulary associated with the
node for the actionable intent of "initiate a phone call" may
include words and phrases such as "call," "phone," "dial," "ring,"
"call this number," "make a call to," and so on. The vocabulary
index 344 optionally includes words and phrases in different
languages.
[0074] The natural language processor 332 receives the token
sequence (e.g., a text string) from the speech-to-text processing
module 330, and determines what nodes are implicated by the words
in the token sequence. In some embodiments, if a word or phrase in
the token sequence is found to be associated with one or more nodes
in the ontology 360 (via the vocabulary index 344), the word or
phrase will "trigger" or "activate" those nodes. Based on the
quantity and/or relative importance of the activated nodes, the
natural language processor 332 will select one of the actionable
intents as the task that the user intended the digital assistant to
perform. In some embodiments, the domain that has the most
"triggered" nodes is selected. In some embodiments, the domain
having the highest confidence value (e.g., based on the relative
importance of its various triggered nodes) is selected. In some
embodiments, the domain is selected based on a combination of the
number and the importance of the triggered nodes. In some
embodiments, additional factors are considered in selecting the
node as well, such as whether the digital assistant has previously
correctly interpreted a similar request from a user.
[0075] In some embodiments, the digital assistant also stores names
of specific entities in the vocabulary index 344, so that when one
of these names is detected in the user request, the natural
language processor 332 will be able to recognize that the name
refers to a specific instance of a property or sub-property in the
ontology. In some embodiments, the names of specific entities are
names of businesses, restaurants, people, movies, and the like. In
some embodiments, the digital assistant searches and identifies
specific entity names from other data sources, such as the user's
address book, a movies database, a musicians database, and/or a
restaurant database. In some embodiments, when the natural language
processor 332 identifies that a word in the token sequence is a
name of a specific entity (such as a name in the user's address
book), that word is given additional significance in selecting the
actionable intent within the ontology for the user request.
[0076] For example, when the words "Mr. Santo" are recognized from
the user request, and the last name "Santo" is found in the
vocabulary index 344 as one of the contacts in the user's contact
list, then it is likely that the user request corresponds to a
"send a message" or "initiate a phone call" domain. For another
example, when the words "ABC Cafe" are found in the user request,
and the term "ABC Cafe" is found in the vocabulary index 344 as the
name of a particular restaurant in the user's city, then it is
likely that the user request corresponds to a "restaurant
reservation" domain.
[0077] User data 348 includes user-specific information, such as
user-specific vocabulary, user preferences, user address, user's
default and secondary languages, user's contact list, and other
short-term or long-term information for each user. In some
embodiments, the natural language processor 332 uses the
user-specific information to supplement the information contained
in the user input to further define the user intent. For example,
for a user request "invite my friends to my birthday party," the
natural language processor 332 is able to access user data 348 to
determine who the "friends" are and when and where the "birthday
party" would be held, rather than requiring the user to provide
such information explicitly in his/her request.
[0078] Other details of searching an ontology based on a token
string is described in U.S. Utility application Ser. No. 12/341,743
for "Method and Apparatus for Searching Using An Active Ontology,"
filed Dec. 22, 2008, the entire disclosure of which is incorporated
herein by reference.
[0079] In some embodiments, once the natural language processor 332
identifies an actionable intent (or domain) based on the user
request, the natural language processor 332 generates a structured
query to represent the identified actionable intent. In some
embodiments, the structured query includes parameters for one or
more nodes within the domain for the actionable intent, and at
least some of the parameters are populated with the specific
information and requirements specified in the user request. For
example, the user may say "Make me a dinner reservation at a sushi
place at 7." In this case, the natural language processor 332 may
be able to correctly identify the actionable intent to be
"restaurant reservation" based on the user input. According to the
ontology, a structured query for a "restaurant reservation" domain
may include parameters such as {Cuisine), {Time}, {Date}, {Party
Size}, and the like. In some embodiments, based on the information
contained in the user's utterance, the natural language processor
332 generates a partial structured query for the restaurant
reservation domain, where the partial structured query includes the
parameters {Cuisine="Sushi"} and {Time="7 pm"}. However, in this
example, the user's utterance contains insufficient information to
complete the structured query associated with the domain.
Therefore, other necessary parameters such as {Party Size} and
{Date} are not specified in the structured query based on the
information currently available. In some embodiments, the natural
language processor 332 populates some parameters of the structured
query with received context information. For example, in some
embodiments, if the user requested a sushi restaurant "near me,"
the natural language processor 332 populates a {location} parameter
in the structured query with GPS coordinates from the user device
104.
[0080] In some embodiments, the natural language processor 332
passes the structured query (including any completed parameters) to
the task flow processing module 336 ("task flow processor"). The
task flow processor 336 is configured to receive the structured
query from the natural language processor 332, complete the
structured query, if necessary, and perform the actions required to
"complete" the user's ultimate request. In some embodiments, the
various procedures necessary to complete these tasks are provided
in task flow models 354. In some embodiments, the task flow models
include procedures for obtaining additional information from the
user, and task flows for performing actions associated with the
actionable intent.
[0081] As described above, in order to complete a structured query,
the task flow processor 336 may need to initiate additional
dialogue with the user in order to obtain additional information,
and/or disambiguate potentially ambiguous utterances. When such
interactions are necessary, the task flow processor 336 invokes the
dialogue processing module 334 ("dialogue processor 334") to engage
in a dialogue with the user. In some embodiments, the dialogue
processor 334 determines how (and/or when) to ask the user for the
additional information, and receives and processes the user
responses. The questions are provided to and answers are received
from the users through the I/O processing module 328. In some
embodiments, the dialogue processor 334 presents dialogue output to
the user via audio and/or visual output, and receives input from
the user via spoken or physical (e.g., clicking) responses.
Continuing with the example above, when the task flow processor 336
invokes the dialogue flow processor 334 to determine the "party
size" and "date" information for the structured query associated
with the domain "restaurant reservation," the dialogue flow
processor 335 generates questions such as "For how many people?"
and "On which day?" to pass to the user. Once answers are received
from the user, the dialogue flow processor 334 can then populate
the structured query with the missing information, or pass the
information to the task flow processor 336 to complete the missing
information from the structured query.
[0082] In some cases, the task flow processor 336 may receive a
structured query that has one or more ambiguous properties. For
example, a structured query for the "send a message" domain may
indicate that the intended recipient is "Bob," and the user may
have multiple contacts named "Bob." The task flow processor 336
will request that the dialogue processor 334 disambiguate this
property of the structured query. In turn, the dialogue processor
334 may ask the user "Which Bob?", and display (or read) a list of
contacts named "Bob" from which the user may choose.
[0083] Once the task flow processor 336 has completed the
structured query for an actionable intent, the task flow processor
336 proceeds to perform the ultimate task associated with the
actionable intent. Accordingly, the task flow processor 336
executes the steps and instructions in the task flow model
according to the specific parameters contained in the structured
query. For example, the task flow model for the actionable intent
of "restaurant reservation" may include steps and instructions for
contacting a restaurant and actually requesting a reservation for a
particular party size at a particular time. For example, using a
structured query such as: {restaurant reservation, restaurant =ABC
Cafe, date=Mar. 12, 2012, time=7 pm, party size=5}, the task flow
processor 336 may perform the steps of: (1) logging onto a server
of the ABC Cafe or a restaurant reservation system such as
OPENTABLE.RTM., (2) entering the date, time, and party size
information in a form on the website, (3) submitting the form, and
(4) making a calendar entry for the reservation in the user's
calendar.
[0084] In some embodiments, the task flow processor 336 employs the
assistance of a service processing module 338 ("service processor")
to complete a task requested in the user input or to provide an
informational answer requested in the user input. For example, the
service processor 338 can act on behalf of the task flow processor
336 to make a phone call, set a calendar entry, invoke a map
search, invoke or interact with other user applications installed
on the user device, and invoke or interact with third party
services (e.g. a restaurant reservation portal, a social networking
website, a banking portal, etc.). In some embodiments, the
protocols and application programming interfaces (API) required by
each service can be specified by a respective service model among
the services models 356. The service processor 338 accesses the
appropriate service model for a service and generates requests for
the service in accordance with the protocols and APIs required by
the service according to the service model.
[0085] For example, if a restaurant has enabled an online
reservation service, the restaurant can submit a service model
specifying the necessary parameters for making a reservation and
the APIs for communicating the values of the necessary parameter to
the online reservation service. When requested by the task flow
processor 336, the service processor 338 can establish a network
connection with the online reservation service using the web
address stored in the service model, and send the necessary
parameters of the reservation (e.g., time, date, party size) to the
online reservation interface in a format according to the API of
the online reservation service.
[0086] In some embodiments, the natural language processor 332,
dialogue processor 334, and task flow processor 336 are used
collectively and iteratively to infer and define the user's intent,
obtain information to further clarify and refine the user intent,
and finally generate a response (i.e., an output to the user, or
the completion of a task) to fulfill the user's intent.
[0087] In some embodiments, after all of the tasks needed to
fulfill the user's request have been performed, the digital
assistant 326 formulates a confirmation response, and sends the
response back to the user through the I/O processing module 328. If
the user request seeks an informational answer, the confirmation
response presents the requested information to the user. In some
embodiments, the digital assistant also requests the user to
indicate whether the user is satisfied with the response produced
by the digital assistant 326.
[0088] More details on the digital assistant can be found in the
U.S. Utility application Ser. No. 12/987,982, entitled "Intelligent
Automated Assistant", filed Jan. 18, 2010, U.S. Utility Application
No. 61/493,201, entitled "Generating and Processing Data Items That
Represent Tasks to Perform", filed Jun. 3, 2011, the entire
disclosures of which are incorporated herein by reference.
[0089] In most scenarios, when the digital assistant receives a
user input from a user, the digital assistant attempts to provide
an appropriate response to the user input with as little delay as
possible. For example, suppose the user requests certain
information (e.g., current traffic information) by providing a
speech input (e.g., "How does the traffic look right now?"). Right
after the digital assistant receives and processes the speech
input, the digital assistant optionally provides a speech output
(e.g., "Looking up traffic information...") acknowledging receipt
of the user request. After the digital assistant obtains the
requested information in response to the user request, the digital
assistant proceeds to provide the requested information to the user
without further delay. For example, in response to the user's
traffic information request, the digital assistant may provide a
series of one or more discrete speech outputs separated by brief
pauses (e.g., "There are 2 accidents on the road. <Pause> One
accident is on 101 north bound near Whipple Avenue. <Pause>
And a second accident is on 85 north near 280."), immediately after
the speech outputs are generated.
[0090] For the purpose of this specification, the initial
acknowledgement of the user request and the series of one or more
discrete speech outputs provided in response to the user request
are all considered sub-responses of a complete response to the user
request. In other words, the digital assistant initiates an
information provision process for the user request upon receipt of
the user request, and during the information provision process, the
digital assistant prepares and provides each sub-response of the
complete response to the user request without requiring further
prompts from the user.
[0091] Sometimes, additional information or clarification (e.g.,
route information) is required before the requested information can
be obtained. In such scenarios, the digital assistant outputs a
question (e.g., "Where are you going?") to the user asking for the
additional information or clarification. In some embodiments, the
question provided by the digital assistant is considered a complete
response to the user request because the digital assistant will not
take further actions or provide any additional response to the user
request until a new input is received from the user. In some
embodiments, once the user provides the additional information or
clarification, the digital assistant initiates a new information
provision process for a "new" user request established based on the
original user request and the additional user input.
[0092] In some embodiments, the digital assistant initiates a new
information provision process upon receipt of each new user input,
and each existing information provision process terminates either
(1) when all of the sub-responses of a complete response to the
user request have been provided to the user or (2) when the digital
assistant provides a request for additional information or
clarification to the user regarding a previous user request that
started the existing information provision process.
[0093] In general, after a user request for information or
performance of a task is received by the digital assistant, it is
desirable that the digital assistant provides a response (e.g.,
either an output containing the requested information, an
acknowledgement of a requested task, or an output to request a
clarification) as promptly as possible. Real-time responsiveness of
the digital assistant is one of the key factors in evaluating
performance of the digital assistant. In such cases, a response is
prepared as quickly as possible, and a default delivery time for
the response is a time immediately after the response is
prepared.
[0094] Sometimes, however, after an initial sub-response provided
immediately after receipt of the user input, the digital assistant
provides the remaining one or more sub-responses one at a time over
an extended period of time. In some embodiments, the information
provision process for a user request is stretched out over an
extended period of time that is longer than the sum of the time
required to provide each sub-response individually. For example, in
some embodiments, short pauses (i.e., brief periods of silence) are
inserted between an adjacent pair of sub-responses (e.g., a pair of
consecutive speech outputs) when they are delivered to the user
through an audio-output channel.
[0095] In some embodiments, a sub-response is held in abeyance
after it is prepared and is delivered only when a predetermined
condition has been met. In some embodiments, the predetermined
condition is met when a predetermined trigger time has been reached
according to a system clock and/or when a predetermined trigger
event has occurred. For example, if the user says to the digital
assistant "set me a timer for 5 minutes," the digital assistant
initiates an information provision process upon receipt of the user
request. During the information provision process, the digital
assistant provides a first sub-response (e.g., "OK, timer
started.") right away, and does not provide a second and final
sub-response (e.g., "OK, five minutes are up") until 5 minutes
later. In such cases, the default delivery time for the first
sub-response is a time immediately after the first sub-response is
prepared, and the default delivery time for the second, final
sub-response is a time immediately after the occurrence of the
trigger event (e.g., the elapse of 5 minutes from the start of the
timer). The information provision process is terminated when the
digital assistant finishes providing the final sub-response to the
user. In various embodiments, the second sub-response is prepared
any time (e.g., right after the first sub-response is prepared, or
until shortly before the default delivery time for the second
sub-response) before the default delivery time for the second
sub-response.
[0096] As will be described in more details later in this
specification, a context-sensitive interruption handler (e.g., the
interruption handling module 340 in FIG. 3A) is implemented on top
of the default rules for providing responses to the user requests
and/or for providing the alert items for reminders and
notifications. In some embodiments, the interruption handler
gathers information regarding the present context in real-time, and
determines in real-time whether the default rules for provision of
responses, reminders and/or notifications need to be altered (e.g.,
because the device is currently receiving speech input for a user,
or because additional information has been detected that alters the
urgency of a reminder, notification, or other speech output). For
example, in some contexts, it would be more suitable to delay
(e.g., staying, at least temporary) provision of a non-urgent
speech output because a user is speaking into the device, or
because the user's context suggests that interruptions should be
avoided (e.g., because the user is in a meeting or is asleep) while
other times it may be more suitable to provide an urgent speech
output immediately (e.g., "barge-in," or interrupt the user). In
addition, in some contexts, it is acceptable and in fact, more
suitable to forgo providing the speech output altogether.
[0097] The context-sensitive interruption handler also selects
audio prompts from among several possible audio prompts with which
to alert the user to some information (e.g., a notification item).
In some embodiments, the particular audio prompt or type of audio
prompt that is selected is based on the urgency of the notification
and/or the user's context. For example, in some embodiments, a
notification item with a low urgency is provided as soon as it is
received if the user's context suggests that a barge-in would not
be inconvenient. On the other hand, in some embodiments, a
notification item with a higher urgency can be delayed if the
user's context suggests that a barge-in, even for somewhat
important information, would be unwelcome.
[0098] FIGS. 4A-4C illustrate exemplary scenarios in which a
digital assistant provides a speech output, or does not provide a
speech output, in accordance with some embodiments. In FIGS. 4A-4C,
solid boxes corresponding to speech outputs (e.g., SO1 in FIG. 4A)
indicate speech outputs that are actually provided by the device.
Dashed boxes, on the other hand (e.g., SO2 in FIG. 4A), indicate
speech outputs that are not actually provided by the device at the
corresponding time and location, but otherwise would be provided by
the device if not for the detection of speech input by the user, as
explained in greater detail with reference to the individual
figures.
[0099] FIG. 4A illustrates an exemplary scenario in which a speech
output is permanently forgone by the device. At the outset, the
user is heading East (401) on East Alder Ave. At a first location
designated by 402-1, the user requests that the device provide
turn-by-turn directions to a library by stating, "Take me to the
library" as a speech input SI1. The location at which the user
finishes the speaking is designated by 402-2, which is distinct
from 402-1 by virtue of the fact that the user is speaking while
moving. Thus, the bars corresponding to respective speech inputs
and outputs (e.g., the bar between 402-1 and 402-2 corresponding to
SI1) indicate a distance or, equivalently, a length of time that
the respective input/output requires to recite (e.g., speak).
[0100] The device receives the speech input and performs the
necessary operations to, for example, determine the location of the
nearest library, as described in greater detail with reference to
method 500 and FIGS. 5A-5D. In this example, the device determines
that library 404 is the nearest library and responds promptly with
a speech output SO1 ("Ok, Continue Straight")
[0101] At a location designated by 405-1, a phone feature included
on the same device as the digital assistant receives an incoming
call, as indicated by ring-tone icon 406. The user answers the
phone by providing speech input SI2, stating, "Hey John! Haven't
heard from you in ages. How is the family?" At the completion of
speech input SI2, the user is at a location designated by 405-2.
However, in the interim between 405-1 and 405-2, the device
receives (e.g., from a server or a different module on the same
device) a speech output SO2 indicating, "Turn right on First Street
in 3 miles." In this example, speech output SO2 has a low measure
of urgency, because the device is scheduled to warn the user of the
upcoming turn one or more additional times before the user reaches
First Street (e.g., additional warning such as, "Turn Right in 1
mile," and/or, "Turn right now onto First Street"). Because the
device was receiving speech input SI2 when speech output SO2 was to
be outputted (407-1 until 407-2), the device stays output of speech
output SO2. Furthermore, because of the very low priority
associated with speech output SO2 (e.g., due to the redundancy
associated with the turn-by-turn direction in this example), the
stay actually forgoes output of speech output SO2 altogether (e.g.,
never outputs a command to turn left in three miles, relying
instead on the 1 mile and immediate warnings).
[0102] FIG. 4B illustrates an exemplary scenario in which a speech
output is immediately provided to a user, in accordance with some
embodiments. Like reference numerals shared between FIG. 4A and 4B
refer to analogous aspects of the respective scenarios. Thus, for
brevity, those analogous aspects are not repeated here.
[0103] FIG. 4B differs from FIG. 4A in that the phone receives an
incoming call, as indicated by the ring-tone icon 406, at a much
closer proximity to First Street than in FIG. 4A. While the user is
answering the phone via speech input SI2, the device receives a
speech output SO3 corresponding to a turn-by-turn direction command
indicating that the user should turn right very soon (e.g., in this
example, 100 feet). Because of the urgency of the message, the
device "barges-in" (e.g., interrupts the user while the user is
speaking) to output, "Turn right in 100 feet."
[0104] FIG. 4C illustrates an exemplary scenario in which a speech
output is temporarily stayed, and then later provided to a user, in
accordance with some embodiments. Like reference numerals shared
between FIG. 4A and 4C refer to analogous aspects of the respective
scenarios. Thus, for brevity, those analogous aspects are not
repeated here.
[0105] During a speech input SI3, the user requests that the device
inform the user of the Knicks' score whenever the game should end,
stating, "Tell me the Knicks' score when the game ends." The device
responds promptly acknowledging the request, stating, "Ok, I will
tell you the score of the Knicks' game when it ends." As explained
previously, the phone receives an incoming call, which the user
answers in speech input SI2 by stating, "Hey John! Haven't heard
from you in ages. How is the family?" During SI2, the Knicks' game
ends the device receives a speech output SO5 indicating the score,
as requested, to be provided to the user. In this example, speech
output SO5 is not considered urgent because the Knicks' score will
not change in the time that the user is speaking (e.g., during the
time that the device is receiving speech input). For this reason,
the device stays speech output SO5, as indicated by arrow 408,
until the user has finished speaking, and then outputs speech
output SO5. However, in some embodiments, the device response to
the user request in a non-audible fashion, such as by displaying
the Knicks' score on a display of the device. In some embodiments,
because a displayed response will not interrupt the user's speech,
such a response is provided without delay. In some embodiments,
such a displayed response is provided in conjunction with, or
alternatively, in lieu of, a stayed speech output (e.g., when the
displayed response is in lieu of a speech output, the speech output
is forgone altogether).
[0106] Devices on which digital assistants are provided (or through
which users interact with digital assistants) are frequently used
to provide notifications of various kinds to a user. For example, a
device (e.g., user device 104) can receive emails, telephone calls,
voicemails, text messages, and other electronic communications, and
provide notifications to the user that such communications have
been received. For example, the device may provide an audible
output (e.g., ringtone, beep, etc.) and/or a visual output to alert
the user that a communication has been received. In addition, the
device can provide notifications from many other sources, such as
application notifications (e.g., messages from applications
installed on the device, including social networking applications,
utilities, games, etc.), task list notifications (e.g., reminders
related to items that a user placed on a task or reminder list),
calendar notifications (e.g., alerts or reminders related to
calendar items and/or appointments), and the like. In some cases,
these notifications are provided to the user when they are received
and/or when an associated reminder or alert time is reached. If a
user does not wish to be bothered, they can simply turn off all
notifications, such as by putting the device in a "silent" mode.
Or, users can create rules that allow the device to provide some
types of notifications but not others. For example, a user could
manually set a rule that notifications should only be provided for
communications from a certain person or people.
[0107] In some embodiments, a digital assistant (e.g., the digital
assistant system 300) manages a list of notification items for a
user, and intelligently determines whether and how to interrupt a
user to provide notifications. Thus, the digital assistant can
prevent a user from being unnecessarily bothered by notifications
with low urgency, while also ensuring that high-urgency
notifications are provided to the user even if it is at a somewhat
inconvenient time. As a specific example, if a user is in an
important work meeting, the digital assistant determines that a
notification is of low urgency (e.g., a notification from a banking
application indicating that a session has timed out), and delays or
foregoes providing an audio prompt to the user for that
notification item. If the user is simply watching TV, though, the
digital assistant provides the audio prompt because the user's
context suggests that interruptions or barge-ins will not be a
nuisance. However, if an important communication is received (e.g.,
a text message or voicemail regarding a family emergency), the
digital assistant determines that, even though the user is in an
important meeting, the urgency of the communication warrants an
interruption.
[0108] Moreover, as discussed below with reference to FIG. 6, the
digital assistant can escalate its notifications based on the
user's context and the urgency of the particular notification. For
example, when a notification item is deemed urgent enough to
warrant an interruption, the device provides a first audio prompt
to alert the user. If the user does not acknowledge the prompt, the
device outputs a second audio prompt. If the user does not
acknowledge the second audio prompt, the device outputs a third
audio prompt. In some embodiments, the audio prompts are of
different types, and increase in distinctiveness and/or intensity
as they are provided. For example, the first audio prompt may be a
single beep or ringtone, the second may be a repetitive beep or
ringtone (or a louder beep or ringtone, or simply a different
ringtone), and the third may be a speech output (e.g., the
assistant speaking "1 am sorry to interrupt you, but this is very
important."). Additional details and embodiments related to a
digital assistant managing a list of notifications are provided
below with respect to FIG. 6.
[0109] Notification lists are not static, though, because new
notifications are constantly arriving, and new information that
affects the urgency value of already existing notification items is
frequently detected or detectable. A human assistant would take
this information into account when determining how and whether to
interrupt a user to provide a notification. Thus, in some
embodiments, the digital assistant disclosed herein adjusts the
urgency values of notifications based on changing conditions
related to the user and/or the notification. Accordingly, the
digital assistant does more than just react to a static set of
rules established by a user (e.g., a rule to only alert for mails
marked as "high importance"), and actually adjusts urgency values
based on unanticipated and/or spontaneous occurrences. As a
specific example, if a user is in a work meeting when the digital
assistant detects an email requesting that the user take some
action (e.g., "please send me the latest sales figures"), the
digital assistant determines that this message is not urgent enough
to warrant an interruption during this busy time period. However,
if the digital assistant detects a follow up email that changes a
deadline or otherwise increases the urgency of the previous message
(e.g., "I need those figures within the next 5 minutes or we will
lose the sale."), the digital assistant adjusts the urgency value
of the notification associated with the original message (and/or
combines the two messages into one notification item with a
heightened urgency value). If the new urgency value is high enough
to warrant an interruption, the digital assistant will alert the
user about the emails. Thus, the user is alerted to important
messages that otherwise would not have passed a simple rule based
"do-not-disturb" filter (and certainly would not have been provided
if the device were in a silent mode). As described in greater
detail below, many different circumstances cause the digital
assistant to adjust the urgency of a notification item, such as
received communications (e.g., follow up emails and telephone
calls), changes in traffic conditions, changes in weather
conditions, changes in the context of the device, and the like.
[0110] FIGS. 5A-5D are flow diagrams of an exemplary method 500
implemented by a digital assistant for determining whether or not
to provide a speech output to a user based on a determination or
whether or not the device is currently receiving speech input from
a user, as well as the urgency of the speech output. In some
embodiments, the determination of whether or not to provide the
speech output is performed dynamically by an interruption handler
(e.g., the interruption handler 340 in FIG. 3A) of the digital
assistant in real-time based on the present-context.
[0111] In some embodiments, prior to receiving the speech output
(cf. 506), the device receives (502) a request from the user to
perform a digital assistant task. For example, the user requests
that the digital assistant find a cheap nearby restaurant by
stating as a speech input, for example, "Find me something for
dinner, not too expensive." Alternatively, the user requests that
the digital assistant make a reservation at a particular
restaurant, for example, by stating as a speech input, "Make me a
reservation at Boulevard for four," Alternatively, the user asks
for turn-by-turn directions to a local landmark ("Directions to the
Golden Gate Bridge"), or ask for a baseball score ("How did the Sox
do?"), or a stock price ("How did Apple's stock do today?").
[0112] In some embodiments, prior to receiving a speech output (cf.
506), the device sends (504) the request to a digital assistant
server. The speech output is received from the server in response
to the request. In some embodiments, prior to sending the request
to the server, the device performs a speech-to-text operation
(e.g., with STT Processing Module 330). In some embodiments,
speech-to-text is performed at the server. In some embodiments, the
device performs the natural language processing (e.g., with Natural
Language Processing Module 322) including performing the ontology,
vocabulary analysis and context matching using user data (for
example, to disambiguate which "Sox" team the user is interested
in, based on preferences such as favorites, browser history and/or
digital assistant request history). The server then performs any
remaining operations necessary to service the request (e.g.,
identifies one or more actionable items, one or more missing
properties from the actionable properties, searches one or more
database and/or the Internet for missing information, etc.) In some
embodiments, the server prepares a response (e.g., a text string)
and returns the response to the user. In some embodiments, the
server prepares a speech response (e.g., audio data) and transmits
the speech response to the digital assistant.
[0113] In any event, the device receives (506) a speech output to
be provided to a user of the device. In some embodiments, the
speech output is received from the server in response to the
request (e.g., the speech output is an appropriate response to the
request made by the user, be it a request for a dinner reservation
or turn-by-turn directions). In some embodiments, receiving the
speech output includes (508) generating the speech output at the
device (e.g., for example, the server returns a text string in
response to the request and the device generates the speech output
from the text string using a text-to-speech engine). It should be
understood that, in some embodiments, receiving a speech output
means receiving from a server (which optionally includes additional
processing operations such as text-to-speech operations).
Alternatively, or in addition, receiving a speech output means
receiving at a first device component (e.g., a module such as
interruption handling module 340 or a processor 304 executing
instructions held in a module such as interruption handling module
340) from a second device component (e.g., a module such as natural
language processing module 332 or a processor 304 executing
instructions held in a module such as natural language processing
module 332).
[0114] The device determines (510) if the device is currently
receiving speech input from a user. For example, in some
embodiments, the device is (512) a telephone, and determining if
the device is currently receiving speech input from the user
includes determining if the user is participating in a telephone
conversation with a remote user. In such embodiments, the device
determines that it is currently receiving speech input from the
user if the user is currently speaking in the conversation. In some
embodiments, when a party on the other end of the telephone
conversation is speaking, or if there is silence while the user and
the other party go about doing other things, the device determines
that it is not currently receiving speech input (e.g., in some
embodiments, an active telephone conversation is sufficient for a
determination that the device is receiving speech input, while in
alternative embodiments, the device determines that speech input is
being received when the user is actually the one speaking in the
conversation).
[0115] In some embodiments, determining if the device is currently
receiving speech input from the user includes (514) determining if
a last speech input was received within a predetermined period of
time. For example, because there are natural pauses in the
ebb-and-flow of conversation (e.g., pauses to catch one's breath,
pauses to consider what to say next), in some embodiments, the
devices waits a predetermined amount of time before concluding that
the user is not speaking, rather than detecting speech input in an
instantaneous or nearly instantaneous fashion. In some embodiments,
the predetermined period of time is (516) a function of a measure
of a urgency of the output. For example, when the device has an
urgent message ("Turn right NOW!") in an output queue, the device
will wait a shorter amount of time before determining that the user
is not speaking, thus barging-in the moment the user pauses to
catch his or her breath or consider what to say next). In some
embodiments, determining if the device is currently receiving
speech input includes a squelch determination (e.g., based on a
particular strength or directionality threshold at a device
microphone) to disambiguate, for example, background noise and/or
speech made by the user but not intended as speech input (e.g.,
during a telephone conversation, when the user pauses the
conversation to talk to another party in-person).
[0116] Upon determining that the device is not currently receiving
speech input from the user, the device provides (518) the speech
output to the user. In some embodiments, the device provides (520)
audio data received from the remote user (cf. 512, when the user is
participating in a telephone) and the speech output to the user
contemporaneously without staying provision of the speech output
due to the received audio data. For example, in such embodiments,
when the remote user (i.e., the other party) is talking during a
telephone conversation, the device will nevertheless provide speech
output from the digital assistant. In some embodiments, providing
audio data (e.g., speech) received from the remote user and the
speech output to the user contemporaneously means muting the audio
data from the remote user temporarily while the speech output is
provided. For example, in such embodiments, when the remote user
says, "Four score and seven years ago our fathers brought forth on
this continent a new nation, conceived in liberty, and dedicated to
the proposition that all men are created equal" and the speech
output in an output queue is, "Turn left," the audio actually
provided to the user will be, "Four score and seven years ago our
fathers brought forth on this continent a . . . `Turn Left` . . . ,
conceived in liberty, and dedicated to the proposition that all men
are created equal." The user will thus be aware that the remote
user is reciting Lincoln's Gettysburg address, and will also
understand the instructions to turn left. In some embodiments, the
audio data received from the remote user and the speech output are
provided using different vocal accents and/or volumes to
disambiguate the remote user from the digital assistant.
[0117] In some circumstances, the user will have configured the
device to override provision of the speech output. For example,
when the device is in a do-not-disturb mode of operation, provision
of the speech output is forgone (522). In some embodiments, the
device is in a do-not-disturb mode of operation when the user has
configured the device to be in a do-not-disturb mode of operation.
In some embodiments, the device is in a do-not-disturb mode of
operation when the user has configured a device to operation in a
mode distinct from do-not-disturb, but nevertheless includes
do-not-disturb as a feature (e.g., the device is in an airplane
mode, or a quiet mode, or the user has configured the device to be
in a quiet mode during particular hours of the day, etc.).
[0118] In some embodiments, during provision (524) of the speech
output, the device receives (526) speech input from the user. For
example, the device is in the midst of providing a speech output
when the user interrupts by talking as part of a telephone
conversation or speaking another request for a digital assistant
operation. As an example of the latter scenario, when a user has
previously requested that the device locate a nearby Chinese
restaurant, the user may interrupt the response to indicate that he
or she also needs to send an SMS message to a coworker. In such
embodiments, the device will discontinue (528) speech output. In
such embodiments, the device will determine (530) if completion
criteria corresponding to the speech output have been met. In some
embodiments, the completion criteria are met (532) when a
predefined percentage of the speech output has already been
provided to the user. For example, in some embodiments, the
predefined percentage of speech output is (534) a percentage from
the group consisting of: 50%, 60%, 70%, and 80%. Upon determining
that the completion criteria have not been met, the device stays
(536) at least part of the speech output for later time, and upon
determining that the completion criteria have been met, the device
forgoes output of the remainder of the speech output altogether. In
some embodiments, the completion criteria are met when the device
determines that the remainder of the message is moot (e.g., after
requesting Chinese food, and during a recitation by the device of a
list of local Chinese restaurants, the user declares, "Never mind,
I want Thai food.")
[0119] Upon determining that the device is receiving speech input
from the user, the device determines (538) if provision of the
speech output is urgent. In some embodiments, the speech output is
urgent (540) when the speech output meets user-configurable
criteria for immediate provision. For example, in some embodiments,
the user-configurable criteria are met (542) when the device
receives an electronic message from a person that the user has
previously identified as a very important person (VIP).
Alternatively or in addition, in some embodiments, the
user-configurable criteria are met (544) when the device receives a
stock price update and the user has previously configured the
device to provide the stock price update immediately (for example,
the user has configured the device to alert him or her when a
particular stock price exceeds a particular value, so that the user
can consider selling the stock as fast as possible). In some
embodiments, a determination is made as to whether or not provision
of the speech input is urgent based on context. For example, when
the speech output includes directions to turn in the near future
("Turn left NOW!") the device recognizes that the message is
urgent. Upon determining that provision of the speech output is
urgent, the device provides (546) the speech output to the user
(e.g., the device "barges-in" and provides the speech output
despite receiving speech input from the user). In some embodiments,
the device provides (548) the speech output to the user without
delay (e.g., additional delay added on account of the fact that the
user is speaking, on top of any required processing time needed to
produce the output).
[0120] Upon determining that provision of the speech output is not
urgent, the device stays (550) provision of the speech output to
the user. As explained in greater detail below, in some
circumstances staying provision of the speech output means delaying
provision of the speech output until a later time, and then
providing the speech output, while in other circumstances staying
means forgoing provision of the speech output altogether and never
providing that particular speech output. In some circumstances,
whether staying means temporarily delaying provision of the speech
output or permanently forgoing provision of the speech output
depends on the particular embodiment, implementation and the
context surrounding the speech output (cf. 562). In some
embodiments, when the device is in a special mode of operation, the
device provides (552) the speech output without delay (e.g., even
if the device is currently receiving speech input from the user).
For example, in some embodiments the device includes an "Interrupt
Me" mode of operation whereby the user is to be interrupted by the
digital assistant (e.g., the digital assistant is to barge-in)
regardless of whether the device is receiving speech input. In some
embodiments, the special mode of operation is (553) one or more of
the group consisting of a hold mode of operation and a mute mode of
operation.
[0121] Flow paths 553-1, 553-2, and 553-3 represent additional
operation that are optionally performed upon determining that
provision of the speech output is not urgent, in accordance with
some embodiments of method 500. It should be understood that the
various operations described with respect to flow paths 553 are not
necessarily mutually exclusive and, in some circumstances,
combined.
[0122] For example, according to some embodiments, upon determining
that the device is no longer receiving speech input from the user,
the device provides (554) the speech output to the user. In some
embodiments, determining that the device is no longer receiving
speech input from the user includes (556) determining that a
predefined amount of time has elapsed between a time of a last
speech input and a current time. In some embodiments, the
predefined amount of time is a function of a measure of the urgency
of the speech output. In some embodiments, the predetermined amount
of time is (560) a monotonically decreasing function of the measure
of the urgency of the speech output, thereby providing speech
outputs with a greater measure of urgency in a lesser amount of
time . For example, in these embodiments, the device waits a
shorter amount of time before providing an urgent speech output
after the user has finished speaking than if the speech output was
less urgent.
[0123] In some embodiments, the device determines (562) if the
output meets message skipping criteria. In some embodiments, the
message skipping criteria are met (564) when the measure of the
urgency is lower than a predefined threshold. For example, when the
speech output is one of several warnings in a sequence of warnings,
in some circumstances it is unnecessary to provide the user with
each warning in the sequence of warnings. In some embodiments, the
message skipping criteria are met (566) when the speech output is a
navigational command in a set of turn-by-turn directions and the
device is scheduled to give a corresponding navigational command at
a later time. For example, the device is scheduled to provide
navigation commands at 2 miles, 1 miles, 1/2 a mile and moments
before a turn. In such circumstances, when the user is providing
speech input when the 1 mile command would otherwise be recited,
the device forgoes provision of the 1 mile command altogether. The
driver will correspondingly still be notified of the turn by the
1/2 mile command as well as moments before the turn.
[0124] In some embodiments, when the device includes a display,
upon determining that provision of the speech output is not urgent,
the device provides (568) a displayed output corresponding to the
speech output.
[0125] FIG. 6 is a flow diagram of an exemplary method 600
implemented by a digital assistant for managing a list of
notification items and providing audio prompts for notification
items. In some embodiments, the method is performed at one or more
devices having one or more processors and memory (e.g., the device
104 and/or components of the digital assistant system 300,
including, for example, server system 108). In some embodiments,
the determination of whether or not to provide an audio prompt for
a notification item is performed dynamically by an interruption
handler (e.g., the interruption handler 340 in FIG. 3A) based on
the present context of the device and/or the user. While the
following steps may be understood as being performed by a device
(e.g., one device), the method is not limited to this particular
embodiment. For example, in some embodiments, the steps may be
performed by different devices, including several devices working
together to perform a single step, several devices each
individually performing one or more steps, etc.
[0126] A list of notification items is provided (602). Notification
items are items that are configured to cause a notification to be
provided to a user. For example, notification items may be
associated with and/or triggered by communications (e.g., received
emails, text messages, voicemail messages, etc.), calendar alerts
(e.g., reminders associated with appointments or other entries in a
calendar application or service), reminder alerts (e.g., reminders
or task items associated with a task list or reminder list),
application alerts, and the like. Application alerts are alerts
that are issued by an application installed on the electronic
device, and may contain any information. For example, an
application alert may include a notification of an action taken by
the application (e.g., notifying the user that an online banking
session will be terminated for security purposes), or a
notification from a service associated with the application (e.g.,
notifying the user of activity in a social network to which the
application provides access). For example, notification items
correspond to items that are displayed in the "Notification Center"
in APPLE, INC.'s IOS.
[0127] The list of notification items includes a plurality of
notification items, wherein each respective one of the plurality of
notification items is associated with a respective urgency value.
Urgency values are assigned to notification items by the digital
assistant. In some embodiments, urgency values are not assigned by
a user. In some embodiments, urgency values are not determined
based on user-defined notification rules. For example, in some
embodiments, urgency values are not based on a user's request to
allow or deny notifications from certain people, domains,
applications, etc. In some embodiments, however, urgency values
take user-defined notification rules into account when assigning
urgency values, though the rules can be overridden or ignored by
the digital assistant as appropriate.
[0128] Urgency values may be based on various different factors, as
discussed below. Urgency values may be any metric, such as a
numerical range between 0 and 10, where a higher value corresponds
to a more urgent notification. Urgency values may also be "high
urgency," "medium urgency," and "low urgency." Any other
appropriate value or metric may be used as well.
[0129] In some embodiments, urgency values are based on one or more
of: a time associated with the respective notification item; a
location associated with the respective notification item; and
content of the respective notification item. In some
implementations, the urgency values are based on a combination of
these components, such as a weighted average of the urgency impact
of each component. In some implementations, the urgency values are
based on one or a subset of these components.
[0130] In some embodiments, the urgency values for notification
items associated with a certain time, such as calendar entries and
time-based reminders, account for the temporal proximity of the
notification. Thus, the time component of the urgency value is
higher if the notification relates to an event or reminder that is
close in time (e.g., relative to other events or reminders), and
lower if the notification relates to an event or reminder that is
further away in time (e.g., relative to other events or
reminders).
[0131] In some embodiments, the urgency values for notification
items associated with locations, such as calendar entries that
specify a location, account for how far away the user currently is
from that location. Thus, in some embodiments, the location
component of the urgency value is higher if the notification
relates to an appointment requiring a longer travel time (e.g.,
relative to other appointments), and lower if the notification
relates to an event or reminder that requires a shorter travel time
(e.g., relative to other appointments).
[0132] In some embodiments, urgency values are automatically
determined based on the semantic content of the notification. For
example, the digital assistant determines the meaning of each
notification (e.g., with the natural language processing module
322) and assigns an urgency value based on the determined meaning.
For example, the digital assistant can determine whether the
content of a notification item (e.g., the body of an email or text
message, or the textual content of an application notification)
relates to one of a known set of meanings. For example, in some
embodiments, the digital assistant can determine whether a
notification item likely relates to a medical emergency, a work
emergency, a family emergency, a routine application notification,
a calendar item, a reminder or task list item, and the like. In
some embodiments, known meanings and/or classes of meanings are
associated with urgency values and/or ranges of urgency values, and
the digital assistant assigns an urgency value to a notification
item in accordance with its determined meaning and/or class of
meaning.
[0133] In some embodiments, determining urgency values includes
determining a topic of importance to a user of the device, and
assigning at least one of the respective urgency values to a
respective notification item based on a determination that the
respective notification item corresponds to the topic of
importance. For example, in some embodiments, the digital assistant
determines a topic of importance to the user based on any of the
following: historical data associated with the user (e.g., by
determining that the user typically responds to communications
about a certain topic quickly), an amount of notification items in
the list of notification items that relate to that topic (e.g., by
determining that the number of notification items relating to that
topic satisfies a predetermined threshold, such as 2, 3, 5, or more
notification items), a user-specified topic (e.g., the user
requests to be alerted to any notifications relating to a
particular topic), and the like. In some embodiments, the topic of
importance is determined by the device automatically without human
intervention, such as by determining a topic of importance based on
historical data associated with the user, as described noted
above.
[0134] In some embodiments, urgency values are further based on
urgency values that were previously assigned by the digital
assistant to similar notifications, embedded flags or importance
indicators associated with a notification (e.g., an email sent with
"high importance"), keywords in the notification (e.g., "boss,"
"urgent," "emergency," "hospital," "died," "birth," "now," "where
are you," etc.), the application or type of application that issued
the notification (e.g., applications that are less likely to
provide important notifications, such as games, are typically less
important than those from applications that allow human-to-human
communications), senders and recipients of communications, user
history relating to similar notifications (e.g., whether the user
has a history of quickly looking at and/or acting on similar
notifications, or whether they are frequently ignored and/or
dismissed, or how quickly the user tends to respond to
communications from a certain person), and the like.
[0135] Returning to FIG. 6, an information item is detected (604).
In some embodiments, the information item is a communication (e.g.,
an email, a voicemail, a text message, etc.). In some embodiments,
the information item is an application notification. In some
embodiments, the information item corresponds to a change in
context of the device, such as an indication that the device is in
a vehicle. For example, the digital assistant can determine that it
is in a vehicle by detecting certain motions, speeds, and/or
locations of the device with a GPS receiver or accelerometer, or by
detecting that the device has been communicatively coupled to a
vehicle, for example, via BLUETOOTH or a docking station. Another
example of an information item corresponding to change in context
is an indication that the device has changed location (e.g., an
indication that the user has arrived at a workplace, or at home,
etc.). In some embodiments, two information items are detected,
including at least a communication (e.g., an email, voicemail, or
text message) and a change in context of the device (e.g.,
detecting that the device has changed location).
[0136] The digital assistant determines whether the information
item is relevant to an urgency value of a first notification item
of the plurality of notification items (606). In some embodiments,
where two information items are received, the digital assistant
determines whether the combination of the two information items are
relevant to a first notification item of the plurality of
notification items.
[0137] For example, in some embodiments, the digital assistant
determines whether an incoming communication (e.g., the information
item) relates to any of the notification items in the list of
notification items. For example, an incoming communication relates
to a notification in the list of notification items if they have
the same or similar subject matter, are from the same sender, have
the same or similar semantic classification (as determined by a
natural language processing module, as described above), have one
or more common words and/or keywords, etc. As a specific example, a
transcribed voicemail from a particular sender may refer to a
recent email that is included in the notification list (e.g., "I
just forwarded you an email from Josh--please call me as soon as
you get it."). The digital assistant then determines from
information associated with the transcribed voicemail that the
voicemail relates to a particular email (e.g., based on the fact
that they were both sent by the same person, they both refer to a
forwarded email from "Josh," etc.)
[0138] In some embodiments, the digital assistant determines
whether a change in context of the device relates to any of the
notification items in the list of notification items. For example,
the digital assistant determines whether a change in the location
of the device affects the travel time necessary to get to an
upcoming appointment. In some embodiments, the digital assistant
determines whether an application notification relates to any of
the notification items in the list of notification items. For
example, the digital assistant can determine that a notification
from a reminder or task list application has the same or similar
content as an existing notification relating to a calendar entry.
Specifically, the digital assistant can determine that a reminder
to "pick up a birthday present for Judy" relates to a notification
of a calendar entry of "Judy's Birthday."
[0139] In some embodiments, once the digital assistant determines
that the information item relates to a first notification item of
the plurality of notification items, the digital assistant
determines whether the information item is relevant to the urgency
of that notification item (cf. 606). For example, the digital
assistant determines whether the information item affects any of
the following: a time associated with the notification item (e.g.,
the information item changes an appointment to an earlier or later
time, the information item indicates a flight or other travel
delay), a location associated with the notification item (e.g.,
changes the location of an appointment), a travel time to an
appointment (e.g., because the user is now further away from a
location of an upcoming appointment, or because traffic conditions
have changed), an importance of the notification item (e.g.,
because multiple communications relating to a particular topic have
been detected, or because the semantic content of the information
item indicates an escalation of importance of the notification
item), and the like.
[0140] Upon determining that the information item is relevant to
the urgency value of the first notification item, the digital
assistant adjusts the urgency value of the first notification item
(608). In some embodiments, urgency values are adjusted to be more
urgent or less urgent depending on how the detected information
item affects the first notification item. In some implementations,
the digital assistant incorporates the information item in the
first notification item, such as by changing a due date,
appointment time, location, etc., of the first notification item.
For example, in some implementations, if a notification item
relates to a calendar entry associated with a particular time, and
the information item is an email indicating that the calendar entry
has been rescheduled, the digital assistant will update the
notification item to show that the time has been changed. In some
implementations, the digital assistant generates a new notification
item including information from both the first notification item
and the information item and assigns to it an urgency value based
on both of them. For example, in some implementations, the digital
assistant will create a notification item that relates to multiple
communications, such as an original email (e.g., the first
notification item) and a follow up voicemail (e.g., the information
item). In some implementations, the new notification item has a
different urgency value than its constituent notification items
and/or information items.
[0141] The digital assistant determines whether the adjusted
urgency value of the first notification item satisfies a
predetermined threshold (610). In some embodiments, the threshold
establishes the urgency level that a notification item must possess
in order to warrant an interruption of the user at that time. In
some embodiments, the threshold may be determined by the user or
the digital assistant. For example, in some embodiments, the
digital assistant continuously determines and applies a particular
urgency threshold, and the user can override and/or adjust the
automatically determined threshold at any time.
[0142] In some embodiments, there are a predetermined number of
threshold values. For example, the digital assistant and/or the
user can select from a low, medium, or high urgency threshold. In
some embodiments, a low urgency threshold indicates that any and
all notifications can be provided without restriction; a medium
urgency threshold indicates that only notifications with a medium
or high urgency value will be provided; and a high urgency
threshold indicates that only notifications with a high urgency
value will be provided. As noted above, urgency values may be
correspond to a numerical range rather than (or in addition to) a
low/medium/high classification. Thus, in some embodiments, urgency
values within a first sub range of values correspond to a low
urgency (e.g., 1-5, where urgency values range from 1-10), urgency
values within a second sub range of values correspond to a medium
urgency (e.g., 6-8, where urgency values range from 1-10), and
urgency values within a third sub range of values correspond to a
high urgency (e.g., 9-10, where urgency values range from 1-10).
Any other appropriate overall range and sub ranges may also be
used. In the present description, thresholds are referred to as
low, medium, or high, though it is understood that this does not
limit urgency values to specific "low/medium/high" scheme, and that
other thresholds and urgency values may be used instead of or in
addition to those described. In some embodiments, the user can set
a threshold value by manipulating a slider control (e.g., displayed
on a touchscreen of the device) to a desired point. For example, a
higher value (e.g., to the right) on the slider control may
correspond to a higher urgency threshold.
[0143] As noted above, in some embodiments, the digital assistant
establishes the predetermined threshold automatically without user
intervention. In some embodiments, the digital assistant
establishes the predetermined threshold in accordance with a
location of the device. In some embodiments, certain locations may
be associated with certain thresholds by default (changeable by the
user either temporarily or permanently). For example, a home may be
associated with a low threshold, a bedroom associated with a medium
threshold, a workplace with a medium threshold, a movie theater
with a high threshold, etc. In some embodiments, the threshold to
be used in various locations is first established by the user,
e.g., as part of an initialization or training of the digital
assistant. Thus, one user can specify that a low threshold should
be used when he or she is at work, and another user can specify a
high threshold while at work. In some cases, however, the digital
assistant assigns a default threshold to certain locations for all
users. For example, the digital assistant can select a high
threshold by default whenever the device is in a theater, park,
church, museum, store, etc., even without the user associating the
location with the threshold. In some embodiments, the digital
assistant notifies the user when it is applying anything other than
a low threshold without the user's having previously trained it to
do so. Accordingly, the user can easily opt out (or opt in) to the
elevated threshold. In some implementations, the user must
specifically enable a mode where the device will automatically
select any threshold higher than a low threshold. This way, the
user can be confident that the device will only raise the urgency
threshold under conditions that are specifically requested by the
user.
[0144] In some embodiments, the digital assistant establishes the
predetermined threshold in accordance with a context of the device.
For example, when the device is in a car (e.g., as detected by
motion/location/speed profiles, ambient noise profiles, or by
detecting a communication link with the vehicle), the digital
assistant establishes a low urgency threshold.
[0145] In some embodiments, the digital assistant establishes the
predetermined threshold in accordance with a time of day. For
example, daylight hours may be associated with a low urgency
threshold, and night time hours with a high urgency threshold.
[0146] In some embodiments, the digital assistant establishes the
predetermined threshold in accordance with a calendar item
associated with a current time. In particular, in some embodiments,
the digital assistant can infer where a user is and what the user
is doing based on the user's calendar entries. If the user is
scheduled to be in a meeting during a certain time, for example,
the device can infer that the user is likely to be in that meeting
during that time. (In some embodiments, the digital assistant can
confirm whether the user is attending a scheduled event by
comparing a location of the event with the user's actual location.)
Thus, if the calendar entry includes information suggesting that a
certain threshold is appropriate, the device will establish the
threshold accordingly. The digital assistant determines that a
calendar event suggests a certain threshold, for example, based on
attendees of the meeting (e.g., the number and/or names of the
attendees), the location of the meeting, the topic of the meeting,
or any text associated with the calendar entry (e.g., "lunch" may
correspond to a low threshold, while "job interview" may correspond
to a high threshold).
[0147] In some embodiments, the digital assistant establishes the
predetermined threshold in accordance with a user setting of the
device. For example, a high urgency threshold can be used if the
user has activated a "do-not-disturb" mode.
[0148] Returning to FIG. 6, upon determining that the adjusted
urgency value satisfies the predetermined threshold (cf. 610), the
digital assistant provides a first audio prompt to a user
(612).
[0149] In some embodiments, upon determining that the adjusted
urgency value does not satisfy the predetermined threshold, the
digital assistant delays providing the audio prompt to the user
(614). In some embodiments, the delayed audio prompt is provided
once the urgency threshold changes, or once the urgency value of
the notification item changes. In some embodiments, upon
determining that the adjusted urgency value does not satisfy the
predetermined threshold, the digital assistant simply does not
provide an audio prompt for that notification item (unless the
urgency value changes in response to a later detected information
item). In some embodiments, upon determining that the adjusted
urgency value does not satisfy the predetermined threshold, the
digital assistant provides a visual prompt to the user (e.g., a
banner or popup notification on a screen of the device 104). In
some embodiments, the a textual component of the notification item
remains in a notification user interface such that the user can
view, acknowledge, and/or act on the notification item at a later
time, and the notification item is not lost.
[0150] In some embodiments, the digital assistant determines
whether the user has acknowledged the first audio prompt (616); and
upon determining that the user has not acknowledged the first audio
prompt (e.g., within a certain predetermined duration), provides a
second audio prompt to the user, the second audio prompt being
different from the first audio prompt (618). In some embodiments,
the second prompt is more distinctive and/or intense than the first
audio prompt. For example, in some embodiments, the second audio
prompt is louder than the first audio prompt. In some embodiments,
the second audio prompt is longer than the first audio prompt. For
example, the second audio prompt may be a longer ringtone, or a
tone or sound that repeats more times (and/or more quickly) than
the first audio prompt. In some embodiments, the first audio prompt
is a first sound (e.g., a first ringtone) and the second audio
prompt is a second sound (e.g., a second ringtone) different from
the first. Thus, the user can differentiate the first audio prompt
from the second audio prompt. In some embodiments, the user can
select the particular sounds and/or ringtones to be associated with
first and second audio prompts. In some implementations, a
vibration of the device is considered an audio prompt.
[0151] In some embodiments, one of the first, second, or third
audio prompts corresponds to a telephone call or a voicemail. For
example, the digital assistant may actually place a telephone call
(or a virtual telephone call), causing the user's smartphone to
ring, thus alerting the user to the urgency of the notification. In
particular, notifications for incoming telephone calls may be
handled differently than the notifications, such that incoming
telephone calls bypass the managed notification list. Thus, by
placing a telephone call to the user (e.g., such that the normal
telephone ringtone and visual notification is provided to the
user), the user can be alerted to the urgency of the notification.
In some implementations, the digital assistant will actually
vocalize the notification (e.g., using a text-to-speech engine)
when the telephone call is answered by the user. In some
implementations, if the telephone call is not answered by the user,
the digital assistant leaves a verbal voicemail for the user.
[0152] In some embodiments, the digital assistant determines
whether the user has acknowledged the second audio prompt (620);
and upon determining that the user has not acknowledged the second
audio prompt (e.g., within a certain predetermined duration),
provides a third audio prompt to the user, the third audio prompt
being different from the first audio prompt and the second audio
prompt (622). In some embodiments, the third audio prompt is louder
and/or longer than both the first and the second audio prompts. In
some embodiments, the third audio prompt is a speech output.
[0153] In some embodiments, the first audio prompt is a ringtone,
the second audio prompt is a first speech output of a first volume,
and the third audio prompt is a speech output of a second volume
louder than the first volume.
[0154] In some embodiments, the first audio prompt is a ringtone,
the second audio prompt is a first speech output of a first length,
and the third audio prompt is a second speech output of a second
length longer than the first length.
[0155] In some embodiments, the process of escalating audio prompts
(e.g., corresponding to using the first, second, and third audio
prompts, above) is combined with the threshold determination to
provide a comprehensive and minimally intrusive proactive
notification scheme. In particular, in some embodiments, even the
most urgent notifications start out with a minimally intrusive
audio prompt (e.g., a single tone or beep, or even a non-audio
prompt, such as a tactile or visual output such as a vibration or a
popup notification). If the audio prompt is not acknowledged (e.g.,
because the user does not interact with the device by pressing a
button, switch, or turning on the screen to view the notification),
the second audio prompt will be provided. If the second audio
prompt is not acknowledged, the third audio prompt is provided. If
the notification is less urgent, however, it may not result in
additional audio prompts. Thus, a lower urgency message may result
in a first audio prompt being provided to the user, but will not
result in subsequent audio prompts. In some embodiments, all
notification items cause a first audio prompt to be provided, but
only notifications satisfying a predetermined threshold will
escalate to the second or the third audio prompt.
[0156] The operations described above with reference to FIGS. 5A-6
are, optionally, implemented by components depicted in FIG. 2
and/or FIG. 3. For example, receiving operation 504, providing
operation 520, receiving operation 526 are, optionally, implemented
by digital assistant 326, I/O processing module 328, interruption
handling module 340, and/or natural language processing module 332,
which are described in detail above. Similarly, it would be clear
to a person having ordinary skill in the art how other processes
can be implemented based on the components depicted in FIG. 2
and/or FIG. 3.
[0157] It should be understood that the particular order in which
the operations have been described above is merely exemplary and is
not intended to indicate that the described order is the only order
in which the operations could be performed. One of ordinary skill
in the art would recognize various ways to reorder the operations
described herein.
[0158] The foregoing description, for purpose of explanation, has
been described with reference to specific embodiments. However, the
illustrative discussions above are not intended to be exhaustive or
to limit the invention to the precise forms disclosed. Many
modifications and variations are possible in view of the above
teachings. The embodiments were chosen and described in order to
best explain the principles of the invention and its practical
applications, to thereby enable others skilled in the art to best
utilize the invention and various embodiments with various
modifications as are suited to the particular use contemplated.
* * * * *