U.S. patent application number 14/744223 was filed with the patent office on 2017-07-13 for system and method of providing generated speech via a network.
The applicant listed for this patent is Nuance Communications, Inc.. Invention is credited to Pamela Leigh Dragosh, David Bjorn Roe, Robert Douglas Sharp.
Application Number | 20170200450 14/744223 |
Document ID | / |
Family ID | 46272960 |
Filed Date | 2017-07-13 |
United States Patent
Application |
20170200450 |
Kind Code |
A1 |
Dragosh; Pamela Leigh ; et
al. |
July 13, 2017 |
SYSTEM AND METHOD OF PROVIDING GENERATED SPEECH VIA A NETWORK
Abstract
A system and method of operating an automatic speech recognition
application over an Internet Protocol network is disclosed. The ASR
application communicates over a packet network such as an Internet
Protocol network or a wireless network. A grammar for recognizing
received speech from a user over the IP network is selected from a
plurality of grammars according to a user-selected application. A
server receives information representing speech over the IP
network, performs speech recognition using the selected grammar,
and returns information based upon the recognized speech.
Sub-grammars may be included within the grammar to recognize speech
from sub-portions of a dialog with the user.
Inventors: |
Dragosh; Pamela Leigh;
(Bedminster, NJ) ; Roe; David Bjorn; (Santa
Barbara, CA) ; Sharp; Robert Douglas; (Morristown,
NJ) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Nuance Communications, Inc. |
Burlington |
MA |
US |
|
|
Family ID: |
46272960 |
Appl. No.: |
14/744223 |
Filed: |
June 19, 2015 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
13527151 |
Jun 19, 2012 |
9065914 |
|
|
14744223 |
|
|
|
|
11548882 |
Oct 12, 2006 |
8209184 |
|
|
13527151 |
|
|
|
|
11031386 |
Jan 7, 2005 |
7139715 |
|
|
11548882 |
|
|
|
|
10347977 |
Jan 21, 2003 |
6856960 |
|
|
11031386 |
|
|
|
|
10067464 |
Feb 5, 2002 |
6604077 |
|
|
10347977 |
|
|
|
|
09533820 |
Mar 24, 2000 |
6366886 |
|
|
10067464 |
|
|
|
|
08833210 |
Apr 14, 1997 |
6078886 |
|
|
09533820 |
|
|
|
|
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
H04M 7/006 20130101;
G10L 15/19 20130101; G10L 2015/223 20130101; G10L 13/00 20130101;
G10L 15/22 20130101; G10L 15/30 20130101; H04M 3/493 20130101; H04M
3/4936 20130101 |
International
Class: |
G10L 15/30 20060101
G10L015/30; G10L 15/22 20060101 G10L015/22; G10L 13/00 20060101
G10L013/00; G10L 15/19 20060101 G10L015/19; H04M 3/493 20060101
H04M003/493; H04M 7/00 20060101 H04M007/00 |
Claims
1. A method comprising: transmitting over a network an
identification of a spoken dialog application, the spoken dialog
application having an associated grammar; transmitting user speech
over the network; and receiving synthesized speech in response to
the user speech, wherein the user speech was recognized by the
spoken dialog application using the associated grammar.
2. The method of claim 1, wherein recognizing the user speech using
the associated grammar further comprises recognizing the user
speech using a sub-grammar based on a sub-component of the user
speech.
3. The method of claim 2, wherein the sub-grammar is associated
with a task.
4. The method of claim 1, wherein the network is an internet
protocol network.
5. The method of claim 1, wherein the spoken dialog application
carries on a dialog with a user communicating with a client.
6. The method of claim 1, further comprising: receiving information
based on the synthesized speech over an internet protocol network
from a client.
7. The method of claim 1, further comprising: modifying the
associated grammar based at least in part on the user speech.
8. A system comprising: a processor; and a computer-readable
storage medium storing instructions which, when executed on the
processor, cause the processor to perform a method comprising:
transmitting over a network an identification of a spoken dialog
application, the spoken dialog application having an associated
grammar; transmitting user speech over the network; and receiving
synthesized speech in response to the user speech, wherein the user
speech was recognized by the spoken dialog application using the
associated grammar.
9. The system of claim 8, wherein recognizing the user speech using
the associated grammar further comprises recognizing the user
speech using a sub-grammar based on a sub-component of the user
speech.
10. The system of claim 9, wherein the sub-grammar is associated
with a task.
11. The system of claim 8, wherein the network is an internet
protocol network.
12. The system of claim 8, wherein the spoken dialog application
carries on a dialog with a user communicating with a client.
13. The system of claim 8, the computer-readable storage medium
storing additional instructions which, when executed on the
processor cause the processor to perform a step comprising:
receiving information based on the synthesized speech over an
internet protocol network from a client.
14. The system of claim 8, the computer-readable storage medium
storing additional instructions which, when executed on the
processor, cause the processor to perform a step comprising:
modifying the associated grammar based at least in part on the user
speech.
15. A computer-readable storage medium storing instructions which,
when executed by a processor, cause the processor, to perform steps
comprising: transmitting over a network an identification of a
spoken dialog application, the spoken dialog application having an
associated grammar; transmitting user speech over the network; and
receiving synthesized speech in response to the user speech,
wherein the user speech was recognized by the spoken dialog
application using the associated grammar.
16. The computer-readable storage medium of claim 15, wherein
recognizing the user speech using the associated grammar further
comprises recognizing the user speech by using a sub-grammar based
on a sub-component of the user speech.
17. The computer-readable storage medium of claim 16, wherein the
sub-grammar is associated with a task.
18. The computer-readable storage medium of claim 15, wherein the
network is an internet protocol network.
19. The computer-readable storage medium of claim 15, wherein the
spoken dialog application carries on a dialog with a user
communicating with a client.
20. The computer-readable storage medium of claim 15 storing
additional instructions which, when executed on the processor,
cause the processor to perform a step comprising: receiving
information based on the synthesized speech over an internet
protocol network from a client.
Description
RELATED APPLICATIONS DATA
[0001] The present application is a continuation of U.S. patent
application Ser. No. 11/548,882, filed Oct. 12, 2006, which is a
continuation of U.S. patent application Ser. No. 11/031,386, filed
Jan. 7, 2005, which is a continuation of U.S. patent application
Ser. No. 10/347,977, filed Jan. 21, 2003 (now U.S. Pat. No.
6,856,960), which is a continuation of U.S. patent application Ser.
No. 10/067,464, filed Feb. 5, 2002 (now U.S. Pat. No. 6,604,077),
which is a continuation of U.S. patent application Ser. No.
09/533,820, filed Mar. 24, 2000 (now U.S. Pat. No. 6,366,886),
which is a continuation of U.S. patent application Ser. No.
08/833,210, filed Apr. 14, 1997 (now U.S. Pat. No. 6,078,886). The
contents of each of the related patent applications are
incorporated herein by reference.
TECHNICAL FIELD
[0002] This invention relates to speech recognition in general and,
more particularly, the present invention provides remotely
accessible automatic speech recognition and speech generation
services via a packet or Internet Protocol network.
BACKGROUND OF THE INVENTION
[0003] Techniques for accomplishing automatic speech recognition
(ASR) are well known. Among known ASR techniques are those that use
grammars. A grammar is a representation of the language or phrases
expected to be used or spoken in a given context. In one sense,
then, ASR grammars typically constrain the speech recognizer to a
vocabulary that is a subset of the universe of potentially-spoken
words; and grammars may include sub-grammars. An ASR grammar rule
can then be used to represent the set of "phrases" or combinations
of words from one or more grammars or sub-grammars that may be
expected in a given context. "Grammar" may also refer generally to
a statistical language model (where a model represents phrases),
such as those used in language understanding systems.
[0004] Products and services that utilize some form of automatic
speech recognition ("ASR") methodology have been recently
introduced commercially. For example, AT&T has developed a
grammar-based ASR engine called WATSON that enables development of
complex ASR services. Desirable attributes of complex ASR services
that would utilize such ASR technology include high accuracy in
recognition; robustness to enable recognition where speakers have
differing accents or dialects, and/or in the presence of background
noise; ability to handle large vocabularies; and natural language
understanding. In order to achieve these attributes for complex ASR
services, ASR techniques and engines typically require
computer-based systems having significant processing capability in
order to achieve the desired speech recognition capability.
Processing capability as used herein refers to processor speed,
memory, disk space, as well as access to application databases.
Such requirements have restricted the development of complex ASR
services that are available at one's desktop, because the
processing requirements exceed the capabilities of most desktop
systems, which are typically based on personal computer (PC)
technology.
[0005] Packet networks are general-purpose data networks that are
well-suited for sending stored data of various types, including
speech or audio. The Internet, the largest and most renowned of the
existing packet networks, connects over 4 million computers in some
140 countries. The Internet's global and exponential growth is
common knowledge today.
[0006] Typically, one accesses a packet network, such as the
Internet, through a client software program executing on a
computer, such as a PC, and so packet networks are inherently
client/server oriented. One way of accessing information over a
packet network is through use of a Web browser (such as the
Netscape Navigator, available from Netscape Communications, Inc.,
and the Internet Explorer, available from Microsoft Corp.) that
enables a client to interact with Web servers. Web servers and the
information available therein are typically identified and
addressed through a Uniform Resource Locator (URL)-compatible
address. URL addressing is widely used in Internet and intranet
applications and is well known to those skilled in the art (an
"intranet" is a packet network modeled in functionality based upon
the Internet and is used, e.g., by companies locally or
internally). Many of these networks use the well-known Internet
Protocol for communication. Further, in addition to land-line
networks, wireless networks also employ packet technologies and use
microbrowsers and the like to enable hand-held wireless devices to
communicate with wireless web servers.
[0007] What is desired is a way of enabling ASR services that may
be made available to users at a location, such as at their desktop
or from a wireless device, that is remote from the system hosting
the ASR engine and TTS engine.
SUMMARY OF THE INVENTION
[0008] A system and method of operating an automatic speech
recognition service over a packet or Internet Protocol network is
used to make a spoken dialog service accessible at a client
location remote from the location of the main ASR and speech
generation engines. The speech generated by the spoken dialog
service may be generated from recorded speech segments, a
text-to-speech (TTS) engine or other speech generation techniques.
These various speech generation engines may be used to carry on a
spoken dialog with a user as shown below. In this regard, the
spoken dialog service may be termed either an ASR server when
performing speech recognition, a speech generation server or TTS
server when generating speech for presenting to a user as part of a
conversation or spoken dialog. As used below, the TTS server may
refer to any of these inherent means of generating speech to carry
on a spoken dialog with a user. In accordance with the present
invention, using client-server communications over a packet
network, such as the Internet, the ASR server receives grammar
information from the client which may be identified by virtue of a
selected spoken dialog application such as a pizza ordering
application, receives information representing speech from the
client, performs speech recognition, and the TTS server returns
information based upon the recognized speech to the client. The ASR
server and TTS server may reside on a single computer device or
multiple devices within a network. Accordingly, it is immaterial to
the present invention regarding the physical location within a
network where the ASR and TTS processes operate.
[0009] According to an embodiment of the invention, the invention
relates to a method of operating an automatic speech recognition
service remotely accessible over a packet or IP network. The method
comprises receiving over the packet or IP network an identification
of a grammar, based on the grammar identifier, selecting a grammar
from the plurality of grammars, receiving over the packet or IP
network information representing speech and recognizing the
received speech information. The grammar may be identified
according to what ASR application or spoken dialog application the
user selects via a URL.
[0010] The present invention enables a dialogue between a human
user and a computer device having an ASR and TTS engine over a
packet network. The packet network may be associated with a
land-line network or a wireless network. For example, packet
networks may provide communication of data between a wireless
device and a wireless network node, or between two nodes in a
wireless network. Accordingly, the present invention may be
applicable to an entire network or a portion of a network employing
packet technology.
[0011] Alternative embodiments of the present invention include a
variety of ways to obtain access to the desired grammar and use of
compression or feature extraction as a processing step at the ASR
client prior to transferring speech information to the ASR
server.
BRIEF DESCRIPTION OF THE DRAWINGS
[0012] FIG. 1 is a diagram showing a client-server relationship for
a system providing remote ASR/TTS services in accordance with the
present invention.
[0013] FIG. 2 is a diagram showing a setup process for enabling
remote ASR/TTS services in accordance with the present
invention.
[0014] FIG. 3 is a diagram showing an alternative setup process for
enabling remote ASR/TTS services in accordance with the present
invention.
[0015] FIG. 4 is a diagram showing a process for rule selection in
accordance with the present invention.
[0016] FIG. 5 is a diagram showing a process for enabling remote
automatic speech recognition and text to speech synthesis in
accordance with the present invention.
[0017] FIG. 6 is a diagram showing an alternative process for
enabling remote automatic speech recognition and text to speech
synthesis in accordance with the present invention.
[0018] FIG. 7 is a diagram showing another alternative process for
enabling remote automatic speech recognition and text to speech
synthesis in accordance with the present invention.
DETAILED DESCRIPTION
[0019] The present invention is directed to a client-server based
system for providing remotely-available ASR and TTS services. In
accordance with the present invention, ASR and TTS services may be
provided to a user--e.g., at the user's desktop or from a user's
wireless device--over a packet network, such as the Internet,
Internet Protocol network, or wireless network, without the need
for the user to obtain computer hardware having the extensive
processing capability required for executing full ASR and TTS
techniques.
[0020] A basic client-server architecture used in accordance with
the present invention is shown in FIG. 1. ASR/TTS server 100 is an
ASR software engine and TTS engine which execute on a system,
denoted as server node 110, and can be linked across packet network
120 (such as the Internet or wireless network) to other computers.
Server node 110 may typically be a computer having processing
capability sufficient for running complex ASR-based and TTS-based
applications, such as the AT&T WATSON system. Packet network
120 may, illustratively, be the Internet, wireless network or an
intranet. The packet network 120 may also comprise a portion of a
network wherein packet-based technology is utilized.
[0021] ASR client 130 is a relatively small program (when compared
to ASR server 100) that executes on client PC 140. Client PC 140 is
a computer, such as a personal computer (PC) or wireless device,
having sufficient processing capability for running client
applications, such as a Web browser. Client PC includes hardware,
such as a microphone, and software for the input and capture of
audio sounds, such as speech. Methods for connecting microphones to
a PC and capturing audio sounds, such as speech, at the PC are well
known. Examples of speech handling capabilities for PCS include the
Speech Application Programmer Interface (SAPI) from Microsoft and
the AT&T Advanced Speech Application Programmer Interface
(ASAPI). Details of the Microsoft SAPI are found in, e.g., a
publication entitled "Speech API Developers Guide, Windows 95
Edition," Vers. 1.0, Microsoft Corporation (1995), and details of
the AT&T ASAPI are provided in a publication entitled "Advanced
Speech API Developers Guide," Vers. 1.0, AT&T Corporation
(1996); each of these publications is incorporated herein by
reference. An alternative embodiment of the present invention may
utilize an interface between ASR client 130 and one or more voice
channels, such that speech input may be provided by audio sources
other than a microphone.
[0022] Client PC 140 also has the capability of communicating with
other computers over a packet network (such as the Internet or a
wireless network). Methods for establishing a communications link
to other computers over a packet network are well known and
include, e.g., use of a modem to dial into an Internet service
provider over a telephone line or using a wireless communication
means to access a wireless network.
[0023] ASR/TTS server 100, through server node 110, and ASR client
130, through client PC 140, may communicate with one another over
packet network 120 using known methods suitable for communicating
information (including the transmission of data) over a packet
network using, e.g., a standard communications protocol such as the
Transmission Control Protocol/Internet Protocol (TCP/IP) socket. A
TCP/IP socket is analogous to a "pipe" through which information
may be transmitted over a packet network from one point to
another.
[0024] Establishment of a TCP/IP socket between ASR/TTS server 100
and ASR client 130 will enable the transfer of data between ASR
server 100 and ASR client 130 over packet network 120 necessary to
enable remote ASR and TTS services in accordance with the present
invention. ASR client 130 also interfaces with audio/speech input
and output capabilities and text/graphics display capabilities of
client PC 140. Methods and interfaces for handling input and output
of audio and speech are well known, and text and graphics display
handling methods and interfaces are also well known.
[0025] ASR client 130 may be set up to run in client PC 140 in
several ways. For example, ASR client 130 may be loaded onto client
PC 140 from a permanent data storage medium, such as a magnetic
disk or CD-ROM. In the alternative, ASR client 130 may be
downloaded from an information or data source locatable over a
packet network, such as the Internet. Downloading of ASR client 130
may, e.g., be accomplished once to reside permanently in client PC
140; alternatively, ASR client 130 may be downloaded for single or
limited use purposes. ASR client 130 may be implemented, e.g., as a
small plug-in software module for another program, such as a Web
browser, that executes on client PC 140. One way of accomplishing
this is to make ASR client 130 an Active-X software component
according to the Microsoft Active-X standard. In this way, ASR
client 130 may, e.g., be loaded into client PC 140 in conjunction
with a Web browsing session as follows: a user browsing the World
Wide Web using client PC 140 enters a Web site having ASR
capability; the Web site asks the user permission to download an
ASR client module into client PC 140 in accordance with signed
Active-X control; upon the user's authorization, ASR client 130 is
downloaded into client PC 140.
[0026] Similarly, ASR/TTS server 100 may be set up to run in server
node 110 in several ways, for example, ASR/TTS server 100 may be
loaded onto server node 100 from a permanent data storage medium,
such as a magnetic disk or CD-ROM, or, in the alternative, ASR/TTS
server 100 may be downloaded from an information or data source
locatable over a packet network.
[0027] Further details of providing remote ASR and TTS services in
accordance with the present invention will now be described with
reference to FIGS. 2-7. It is presumed for the discussion to follow
with respect to each of these figures that the client-server
relationship is as shown in FIG. 1. A setup phase is used to
prepare ASR/TTS server 100 and ASR client 130 for performing an
automatic speech recognition task and text to speech task as part
of an ASR/TTS application. For convenience, items shown in FIG. 1
and appearing in other figures will be identified by the same
reference numbers as in FIG. 1.
[0028] Referring now to FIG. 2, a setup phase in a process of
providing remote ASR/TTS services will now be described. At step
201, ASR client 130 receives a request from the application to load
a client grammar. The client grammar is illustratively a data file
containing information representing the language (e.g., words and
phrases) expected to be spoken in the context of the particular ASR
application. The data file may be in a known format, such as the
Standard Grammar Format (SGF) which is part of the Microsoft
SAPI.
[0029] For purposes of illustration, an ASR/TTS application for
taking a pizza order will be used in describing the present
invention. An ASR/TTS service application, such as an application
for pizza-ordering, would typically include a program that
interfaces with and uses ASR client 130 as a resource used for
accomplishing the tasks of the ASR/TTS application. Such an ASR/TTS
application could reside and execute, in whole or in part, in
client PC 140.
[0030] Considering the pizza ordering example, the client grammar
PIZZA would include information representing words that one may use
in ordering pizza, such as "pizza," "pepperoni," etc. In fact,
sub-grammars may be used to build an appropriate grammar. For the
pizza ordering example, sub-grammars for the PIZZA grammar could
include SIZE and TOPPING. These are tasks that the user needs to
complete to successfully complete the dialog with the system and
order the pizza. Each sub-grammar is associated with a task. The
sub-grammar SIZE could consist of words used to describe the size
of the pizza desired, such as "small," "medium" and "large." The
sub-grammar TOPPING may consist of words used to describe the
various toppings one may order with a pizza, e.g., "sausage,"
"pepperoni," "mushroom" and the like.
[0031] ASR client 130 may be given the desired grammar from the
application or, alternatively, ASR client 130 may choose the
grammar from a predetermined group of grammars based upon
information provided by the application. Either way, if the group
of grammars is located remotely from the ASR/TTS server 100, ASR
client 130 then at step 202 sends the desired grammar file to
ASR/TTS server 100 over a TCP/IP socket. A new TCP/IP socket may
have to be set up as part of establishing a new communications
session between client PC 140 and server node 100, or the TCP/IP
socket may already exist as the result of an established
communications session between client PC 140 and server node 110
that has not been terminated. In the pizza ordering illustration,
ASR client 130 would cause transmission of a file containing the
PIZZA grammar to ASR/TTS server 100 over a TCP/IP socket. As is
inherent in the above description, when a person uses the client
device 130 to select communication with the pizza ASR/TTS service
application, that connection may comprise the sending of the
grammar file which explicitly or inherently identifies the PIZZA
grammar to be used for ASR.
[0032] At step 203, ASR/TTS server 100 receives the client grammar
sent from ASR client 130, received locally on the ASR/TTS server
100 or from some other remote location and, at step 204, ASR/TTS
server 100 loads the transmitted client grammar. As used herein,
"loading" of the client grammar means to have the grammar
accessible for use by ASR/TTS server 100, e.g. by storing the
grammar in RAM of server node 110. At step 205, ASR/TTS server 100
returns a grammar "handle" to ASR client 130. A grammar "handle" is
a marker, such as, e.g., a pointer to memory containing the loaded
grammar, that enables ASR client 130 to easily refer to the grammar
during the rest of the communications session or application
execution. ASR client 130 receives the grammar handle from ASR/TTS
server 100 at step 206 and returns the handle to the application at
step 207. For the pizza ordering example, ASR/TTS server 100
receives and loads the transmitted PIZZA grammar file and transmits
back to ASR client 130 a handle pointing to the loaded PIZZA
grammar. ASR client 130, in turn, receives the PIZZA handle from
ASR/TTS server 100 and returns the PIZZA handle to the pizza
ordering application. In this way, the application can simply refer
to the PIZZA handle when carrying out or initiating an ASR or TTS
task as part of the pizza ordering application.
[0033] An alternative setup approach will now be described with
reference to FIG. 3. It is assumed for the remainder of the
description herein that transmission or communication of
information or data between ASR/TTS server 100 and ASR client 130
take place over an established TCP/IP socket. At step 301, ASR
client 130 receives a request from the application to load a client
grammar. Rather than send the client grammar as a data file to
ASR/TTS server 100 at step 302, however, ASR client 130 instead
sends to ASR/TTS server 100 an identifier representing a "canned"
grammar; a "canned" grammar would, e.g., be a common grammar, such
as TIME-OF-DAY or DATE, which ASR/TTS server 100 would already have
stored. In other words, the ASR/TTS client 130 may transmit over
the packet network data such as a grammar identifier directing the
ASR/TTS server 100 to select a particular grammar from a plurality
of grammars stored at the ASR/TTS server 100. Alternatively, ASR
client 130 could send to ASR/TTS server 100 an IP address, such as
a URL-compatible address, where ASR/TTS server 100 could find the
desired grammar file.
[0034] The remote location (i.e., a server located at a remote
location from the ASR/TTS server) in this example would store a
plurality of grammar files available to the ASR/TTS server 100.
ASR/TTS server 100 at step 303 receives the grammar identifier or
URL grammar address from ASR client 130, locates and loads the
requested client grammar at step 304, and at step 305 returns a
grammar handle to ASR client 130. Similar to the steps described
above with respect to FIG. 2, ASR client 130 receives the grammar
handle from ASR/TTS server 100 at step 306 and returns the handle
to the application at step 307. For the pizza ordering example, the
steps described above in connection with FIG. 2 would be the same,
except that ASR client 130 would send to ASR/TTS server 100 a
grammar identifier for the PIZZA grammar (if it were a "canned"
grammar) or a URL address for the location of a file containing the
PIZZA grammar; ASR/TTS server 100 would, in turn, retrieve a file
for the PIZZA grammar based upon the grammar identifier or URL
address (as sent by the ASR client) and then load the requested
PIZZA grammar. The URL of the selected application may also
identify the appropriate grammar, such as www.pizza.com may be used
to identify the PIZZA grammar.
[0035] After the grammar has been loaded and a grammar handle
returned to ASR client 130, an ASR/TTS service application needs to
select a grammar rule to be activated. FIG. 4 shows a process for
grammar rule selection in accordance with the present invention.
ASR client 130 receives from the application a request to activate
a grammar rule at step 401. At step 402, ASR client 130 sends a
rule activate request to ASR/TTS server 100; as shown in FIG. 4,
ASR client 130 may also at step 402 send to ASR/TTS server 100 the
previously-returned grammar handle (which may enable ASR/TTS server
to activate the appropriate grammar rule for the particular grammar
as identified by the grammar handle). ASR/TTS server 100 at step
403 receives the rule activate request and grammar handle (if
sent). At step 404, ASR/TTS server 100 activates the requested rule
and, at step 405, returns to ASR client 130 notification that the
requested rule has been activated. ASR client 130 receives at step
406 the notification of rule activation and notifies the
application at step 407 that the rule has been activated. Once the
application receives notice of rule activation, it may then
initiate recognition of speech.
[0036] For purposes of illustrating the process shown in FIG. 4,
again consider the pizza ordering example. A rule that may be used
for recognizing a pizza order may set the desired phrase for an
order to include the sub-grammars SIZE and TOPPINGS along with the
word "pizza," and might be denoted in the following manner:
{ORDER=SIZE "pizza" "with" TOPPINGS}. With reference again to FIG.
4, ASR client 130 would receive from the application a request to
activate a pizza ordering rule and send the ORDER rule set out
above to ASR/TTS server 100 along with the PIZZA grammar handle.
ASR/TTS server 100 receives the rule activate request along with
the PIZZA grammar handle and activates the ORDER rule, such that
the recognizer would be constrained to recognizing words from the
SIZE sub-grammar, the word "pizza," the word "with" and words from
the sub-grammar TOPPINGS. After activating the ORDER rule, ASR/TTS
server 100 sends notification of the rule activation to ASR client
130 that, in turn notifies the application.
[0037] Once a grammar rule has been activated, the processing of
speech for purposes of recognizing words in the grammar according
to the rule can take place. Referring to FIG. 5, at step 501, ASR
client 130 receives a request from the application to initiate a
speech recognition task. At step 502, ASR client 130 requests
streaming audio from the audio input of client PC 140. Streaming
audio refers to audio being processed "on the fly" as more audio
comes in; the system does not wait for all of the audio input
(i.e., the entire speech) before sending the audio along for
digital processing; streaming audio may also refer to partial
transmission of part of the audio signal as additional audio is
input. Illustratively, a request for streaming audio may be
accomplished by making an appropriate software call to the
operating system running on client PC 140 such that streaming audio
from the microphone input is digitized by the sound processor of
client PC 140. Streaming audio digitized from the microphone input
is then passed along to ASR client 130. ASR client 130 then
initiates transmission of streaming digitized audio to ASR/TTS
server 100 at step 503; like the audio input from the microphone,
the digitized audio is sent to ASR/TTS server 100 "on the fly" even
while speech input continues.
[0038] At step 504, ASR/TTS server 100 performs speech recognition
on the streaming digitized audio as the audio is received from ASR
client 130. Speech recognition is performed using known recognition
algorithms, such as those employed by the AT&T WATSON speech
recognition engine, and is performed within the constraints of the
selected grammar as defined by the activated rule. At step 505,
ASR/TTS server 100 returns streaming text (i.e., partially
recognized speech) as the input speech is recognized. Thus, as
ASR/TTS server 100 reaches its initial results, it returns those
results to ASR client 130 even as ASR server 100 continues to
process additional streaming audio being sent by ASR client 130.
This process of returning recognized text "on the fly" permits ASR
client 130 (or the application interfacing with ASR client 130) to
provide feedback to the speaker. As ASR server 100 continues to
process additional streaming input audio, it may correct the
results of the earlier speech recognition, such that the returned
text may actually update (or correct) parts of the text already
returned to ASR client 130 as part of the speech recognition task.
Once all of the streaming audio has been received from ASR client
130, ASR server completes its speech recognition processing and
returns a final version of recognized text (including corrections)
at step 506.
[0039] At step 507, ASR client 130 receives the recognized text
from ASR server 100 and returns the text to the application at step
508. Again, this may be done "on the fly" as the recognized text
comes in, and ASR client 130 passes along to the application any
corrections to recognized text received from ASR/TTS server
100.
[0040] Referring to the pizza ordering example, once the ORDER rule
has been activated and the application notified, ASR client 130
will receive a request to initiate speech recognition and will
initiate streaming audio from the microphone input. The TTS engine
at the ASR/TTS server 100 will provide the generated responses to
the ASR client 130 to carry on a dialogue with the user. The user
may be prompted to speak the pizza order, and once speaking begins,
ASR client 130 sends digitized streaming audio to ASR/TTS server
100. Thus, as the speaker states, e.g., that she wants to order a
"large pizza with sausage and pepperoni," ASR client 130 will have
sent digitized streaming audio for the first word of the order
along to ASR/TTS server 100 even as the second word is being
spoken. ASR/TTS server 100 will, as the order is being spoken,
return the first word as text "large" as the rest of the order is
being spoken. Ultimately, once the speaker stops speaking, the
final recognized text for the order, "large pizza with sausage,
pepperoni" can be returned to ASR client 130 and, hence, to the
application.
[0041] An alternative embodiment for carrying out the speech
recognition process in accordance with the present invention is
shown in FIG. 6. Similar to the speech recognition process shown in
FIG. 5, at step 601 ASR client 130 receives a request from the
application to initiate a speech recognition task and, at step 602,
ASR client 130 requests streaming audio from the audio input of
client PC 140. Streaming audio digitized from the microphone input
is then passed along to ASR client 130. At step 603, ASR client 130
compresses the digitized audio "on the fly" and then initiates
transmission of streaming compressed digitized audio to ASR/TTS
server 100, while speech input continues.
[0042] At step 604, ASR/TTS server 100 decompresses the compressed
audio received from ASR client 130 before performing speech
recognition on the streaming digitized audio. As described above
with reference to FIG. 5, speech recognition is performed within
the constraints of the selected grammar as defined by the activated
rule. At step 605, ASR/TTS server 100 returns streaming text (i.e.,
partially recognized speech) as the input speech is recognized.
Thus, ASR server 100 returns initial results to ASR client 130 even
as ASR/TTS server 100 continues to process additional compressed
streaming audio being sent by ASR client 130, and may update or
correct parts of the text already returned to ASR client 130 as
part of the speech recognition task. Once all of the streaming
audio has been received from ASR client 130, ASR/TTS server 100
completes its speech recognition processing, generates a response
from the TTS engine, and returns a final version of recognized text
(including corrections) at step 606. ASR client 130 receives the
recognized text and generated speech from ASR/TTS server 100 at
step 607 as it comes in and returns the text and synthetic speech
to the application at step 608.
[0043] Another alternative embodiment for carrying out the speech
recognition process in accordance with the present invention is
shown in FIG. 7. Similar to the speech recognition process shown in
FIGS. 5 and 6, at step 701 ASR client 130 receives a request from
the application to initiate a speech recognition task and, at step
702, ASR client 130 requests streaming audio from the audio input
of client PC 140. Streaming audio digitized from the microphone
input is then passed along to ASR client 130. At step 703, ASR
client 130 processes the digitized audio "on the fly" to extract
features useful for speech recognition processing and then
initiates transmission of extracted features to ASR/TTS server 100,
while speech input continues. Extraction of relevant features from
speech involves grammar-independent processing that is typically
part of algorithms employed for speech recognition, and may be done
using methods known to those skilled in the art, such as those
based upon linear predictive coding (LPC) or Mel filter bank
processing. Feature extraction provides information obtained from
characteristics of voice signals while eliminating unnecessary
information, such as volume.
[0044] Upon receiving extracted features from ASR client 130,
ASR/TTS server 100 at step 704 performs speech recognition on the
incoming features that are arriving "on the fly" (i.e., in manner
analogous to streaming audio). Speech recognition is performed
within the constraints of the selected grammar as defined by the
activated rule. As is the case with the embodiments discussed above
with reference to FIGS. 5 and 6, at step 705 ASR/TTS server 100
returns streaming text (i.e., partially recognized speech) to ASR
client 130 as the input features are recognized. ASR/TTS server 100
continues to process additional extracted features being sent by
ASR client 130, and may update or correct parts of the text already
returned to ASR client 130. ASR/TTS server 100 completes its speech
recognition processing upon receipt of all of the extracted
features from ASR client 130, and returns a final version of
recognized text (including corrections) and synthesized text at
step 706. ASR client 130 receives the recognized text from ASR/TTS
server 100 at step 707 as it comes in and returns the text to the
application at step 708.
[0045] The alternative embodiments described above with respect to
FIGS. 6 and 7 each provide for additional processing at the client
end. For the embodiment in FIG. 6, this entails compression of the
streaming audio (with audio decompression at the server end); for
the embodiment in FIG. 7, this included part of the speech
recognition processing in the form of feature extraction. Using
such additional processing at the client end significantly reduces
the amount of data transmitted from ASR client 130 to ASR/TTS
server 100. Thus, less data is required to represent the speech
signals being transmitted. Where feature extraction is accomplished
at the client end, such benefits are potentially sharply increased,
because extracted features (as opposed to digitized voice signals)
require less data and no features need be sent during periods of
silence. The reduction of data produces a dual desired benefit: (1)
it permits a reduction in bandwidth required to achieve a certain
level of performance, and (2) it reduces the transmission time in
sending speech data from ASR client to ASR/TTS server through the
TCP/IP socket.
[0046] While typically a grammar rule will be activated prior to
the initiation of transmission of speech information from ASR
client 130 to ASR/TTS server 100, rule activation could take place
after some or all of the speech information to be recognized has
been sent from ASR client 130 to ASR/TTS server 100. In such a
circumstance, ASR server 100 would not begin speech recognition
efforts until a grammar rule has been activated. The ASR/TTS server
100 may temporarily store speech sent by ASR client 130 prior to
activation of a grammar rule for processing by the recognizer.
Alternatively, such speech could be ignored.
[0047] Further, multiple speech recognition tasks may be executed
using the techniques of the present invention. For example, an ASR
application could request ASR client 130 to instruct ASR/TTS server
100 to load a canned grammar for a telephone number (i.e., "PHONE
NUMBER") and then request activation of a rule covering spoken
numbers. After a phone number is spoken and recognized in
accordance with the present invention (e.g., in response to a
prompt to speak the phone number, ASR client 130 sends digitized
spoken numbers to ASR/TTS server 100 for recognition), the ASR
application could then request ASR client 130 to set up and
initiate recognition of pizza ordering speech (e.g., load PIZZA
grammar, activate ORDER rule, and initiate speech recognition) in
accordance with the examples described above with reference to
FIGS. 2-5.
[0048] In addition to the simple pizza ordering example used above
for illustration, a wide array of potential ASR services may be
provided over a packet network in accordance with the present
invention. One example of an ASR application enabled by the present
invention is a form-filling service for completing a form in
response to spoken responses to information requested for each of a
number of blanks in the form. In accordance with the present
invention, a form-filling service may be implemented wherein ASR
client 130 sends grammars representing the possible choices for
each of the blanks to ASR/TTS server 100. For each blank, ASR
client 130 requests activation of the appropriate grammar rule and
sends a corresponding spoken answer made in response to a request
for information needed to complete the blank. ASR/TTS server 100
applies an appropriate speech recognition algorithm in accordance
with the selected grammar and rule, and returns text to be inserted
in the form.
[0049] Other ASR services may involve an exchange of information
(e.g., a dialogue) between server and client. For example, an ASR
and TTS service application for handling flight reservations may,
in accordance with the present invention as described herein,
utilize a dialogue between ASR/TTS server 100 and ASR client 130 to
accomplish the ASR and TTS tasks. A dialogue may proceed as
follows: [0050] Speaker (through ASR client 130 to ASR/TTS server
100): [0051] "I want a flight to Los Angeles." [0052] ASR/TTS
server's synthetic response to ASR client (in the form of text or,
alternatively, speech returned by ASR/TTS server 100 to ASR client
130): [0053] "From what city will you be leaving?" [0054] Speaker
(through ASR client to ASR/TTS server): [0055] "Washington, D.C."
[0056] ASR/TTS server's synthetic response to ASR client: [0057]
"What day do you want to leave" [0058] Speaker (ASR client to
ASR/TTS server): [0059] "Tuesday." [0060] ASR/TTS server's response
to ASR client: [0061] "What time do you want to leave" [0062]
Speaker (ASR client to ASR/TTS server): [0063] "4 O'clock in the
afternoon." [0064] ASR/TTS server's response to ASR client: [0065]
"I can book you on XYZ Airline flight 4567 from Washington, D.C. to
Los Angeles on Tuesday at 4 O'clock PM. Do you want to reserve a
seat on this flight?"
[0066] In this case, the information received from ASR/TTS server
100 is not literally the text from the recognized speech, but is
information based upon the recognized speech (which would depend
upon the application). Each leg of the dialogue may be accomplished
in accordance with the ASR client-server method described above. As
may be observed from this example, such an ASR/TTS service
application requires of the ASR client and ASR/TTS server not only
the ability to handle natural language, but also access to a large
database that is constantly changing. To accomplish this, it may be
desirable to have the ASR/TTS service application actually
installed and executing in server node 110, rather than in client
PC 140. Client PC 140 would, in that case, merely have to run a
relatively small "agent" program that, at the control of the
application program running at server node 110, initiates ASR
client 130 and shepherds the speech input through ASR client 130
along to and from ASR/TTS server 100. An example of such an "agent"
program may be, e.g., one that places a "talking head" on the
screen of client PC 140 to assist the interaction between an
individual using the ASR/TTS service application at client PC 140
and, through ASR client 130 and ASR/TTS server 100, send the
person's speech information along to ASR/TTS server 100 for
recognition and return synthesized speech to the individual.
[0067] In summary, the present invention provides a way of
providing ASR services that may be made available to users over a
packet network, such as the Internet, at a location remote from a
system hosting an ASR engine using a client-server
architecture.
[0068] What has been described is merely illustrative of the
application of the principles of the present invention. Those of
skill in the art may implement other arrangements and methods
without departing from the spirit and scope of the present
invention. Accordingly, the scope of the invention should be
defined by the appended claims and not by any specific examples
provided herein.
* * * * *
References