U.S. patent application number 10/685478 was filed with the patent office on 2004-04-29 for systems and methods for providing interactive speaker identification training.
Invention is credited to Kubala, Francis G., Liu, Daben.
Application Number | 20040083104 10/685478 |
Document ID | / |
Family ID | 32110223 |
Filed Date | 2004-04-29 |
United States Patent
Application |
20040083104 |
Kind Code |
A1 |
Liu, Daben ; et al. |
April 29, 2004 |
Systems and methods for providing interactive speaker
identification training
Abstract
A system (100) provides speaker identification training. The
system (100) generates speaker models and receives audio segments.
The system (100) identifies speakers corresponding to the audio
segments based on the speaker models. At least one of the audio
segments has an unidentified or misidentified speaker (i.e., an
audio segment whose speaker cannot be accurately identified). The
system (100) presents, to a user, audio segments that include an
audio segment whose speaker is unidentified or misidentified and
receives, from the user, the name of the unidentified or
misidentified speaker. The system (100) may use this information to
subsequently identify the unidentified or misidentified speaker by
name for future audio segments.
Inventors: |
Liu, Daben; (Belmont,
MA) ; Kubala, Francis G.; (Boston, MA) |
Correspondence
Address: |
Leonard C. Suchyta
c/o Christian Andersen
Verizon Corporate Services Group Inc.
600 Hidden Ridge, HQE03H01
Irving
TX
75038
US
|
Family ID: |
32110223 |
Appl. No.: |
10/685478 |
Filed: |
October 16, 2003 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
60419214 |
Oct 17, 2002 |
|
|
|
Current U.S.
Class: |
704/245 |
Current CPC
Class: |
G10L 15/32 20130101;
G10L 15/28 20130101 |
Class at
Publication: |
704/256 |
International
Class: |
G10L 015/14 |
Claims
What is claimed is:
1. A speaker identification system, comprising: an indexer
configured to: generate a plurality of speaker models, receive a
plurality of audio segments, and identify speakers corresponding to
the audio segments based on the speaker models, the indexer being
unable to correctly identify at least one of the speakers, as an
unidentified speaker, corresponding to the audio segments; and a
server configured to: receive, from a user, the name of the
unidentified speaker, and provide the name of the unidentified
speaker to the indexer for identification of the unidentified
speaker in subsequent audio segments.
2. The system of claim 1, wherein the indexer is further configured
to: generate a new speaker model for the unidentified speaker based
on an audio segment corresponding to the unidentified speaker.
3. The system of claim 1, wherein the indexer is further configured
to: generate labels for the audio segments, the labels being based
on names of speakers that can be identified and gender of speakers
that cannot be identified.
4. The system of claim 3, wherein when receiving the name of an
unidentified speaker, the server is configured to: present a
document to the user, the document including a transcription of a
plurality of the audio segments and the labels for the plurality of
the audio segments, and receive, from the user, the name of one of
the speakers that cannot be identified.
5. The system of claim 4, wherein when presenting a document, the
server is further configured to: provide audio data corresponding
to at least one of the plurality of the audio segments to the
user.
6. The system of claim 1, wherein the server is further configured
to: locate one or more additional audio segments from the
unidentified speaker, and present the one or more additional audio
segments to the user for confirmation that the one or more
additional audio segments were produced by the unidentified
speaker.
7. The system of claim 6, wherein when presenting the one or more
additional audio segments, the server is configured to continue to
present audio segments to the user for confirmation until at least
four minutes of audio data is obtained.
8. The system of claim 6, wherein the unidentified speaker
corresponds to at least one of the audio segments; and wherein when
locating one or more additional audio segments, the server is
configured to: find one or more additional audio segments similar
to the at least one of the audio segments.
9. The system of claim 1, wherein the indexer is configured to: fit
audio data from the unidentified speaker to a speaker independent
Gaussian mixture model using an expectation and maximization
process, and generate a new speaker model for the unidentified
speaker using a maximum a posteriori adaptation process.
10. The system of claim 1, wherein the unidentified speaker is a
misidentified speaker of one of the audio segments; and wherein
when receiving the name, the server is configured to: receive, from
the user, a correct name of a speaker of the one of the audio
segments.
11. The system of claim 10, wherein the indexer is further
configured to: identify one of the speaker models, as an identified
speaker model, that corresponds to the one of the audio segments,
and update a label associated with the identified speaker model to
include the correct name of the speaker of the one of the audio
segments.
12. The system of claim 10, wherein the server is further
configured to: locate one or more additional audio segments similar
to the one of the audio segments, and present the one or more
additional audio segments to the user for confirmation that the one
or more additional audio segments were produced by the speaker of
the one of the audio segments.
13. A speaker identification system, comprising: means for
generating a plurality of speaker models; means for receiving a
plurality of audio segments; means for identifying speakers
corresponding to the audio segments based on the speaker models, at
least one of the audio segments being associated with an
unidentified or misidentified speaker; means for labeling the audio
segments with names of the speakers that can be identified; means
for presenting a plurality of the audio segments, including the at
least one of the audio segments, with the labels to a user; means
for receiving, from the user, the name of the unidentified or
misidentified speaker; and means for identifying the unidentified
or misidentified speaker by name in future audio segments.
14. A method for providing speaker identification training,
comprising: generating a plurality of speaker models; receiving a
plurality of audio segments; identifying speakers corresponding to
the audio segments based on the speaker models, at least one of the
audio segments being associated with an unidentified or
misidentified speaker; presenting a plurality of the audio
segments, including the at least one of the audio segments, to a
user; receiving, from the user, the name of the unidentified or
misidentified speaker; and identifying the unidentified or
misidentified speaker by name for future audio segments.
15. The method of claim 14, wherein the unidentified or
misidentified speaker is an unidentified speaker; and wherein the
method further comprises: generating a new speaker model for the
unidentified speaker based on the at least one of the audio
segments.
16. The method of claim 14, further comprising: generating labels
for the audio segments, the labels being based on names of speakers
that can be identified and gender of speakers that cannot be
identified.
17. The method of claim 16, wherein the presenting a plurality of
the audio segments includes: providing a document to the user, the
document including a transcription of the plurality of the audio
segments and the labels for the plurality of the audio
segments.
18. The method of claim 17, wherein the providing a document
includes: providing audio data corresponding to one or more of the
plurality of the audio segments to the user.
19. The method of claim 14, further comprising: locating one or
more additional audio segments from the unidentified or
misidentified speaker, and presenting the one or more additional
audio segments to the user for confirmation that the one or more
additional audio segments were produced by the unidentified or
misidentified speaker.
20. The method of claim 19, wherein the presenting the one or more
additional audio segments includes: presenting audio segments to
the user for confirmation until at least four minutes of audio data
is obtained.
21. The method of claim 19, wherein the locating one or more
additional audio segments includes: finding one or more additional
audio segments similar to the at least one of the audio
segments.
22. The method of claim 14, wherein the unidentified or
misidentified speaker is an unidentified speaker; and wherein the
method further comprises: fitting audio data from the unidentified
speaker to a speaker independent Gaussian mixture model using an
expectation and maximization process; and generating a new speaker
model for the unidentified speaker using a maximum a posteriori
adaptation process.
23. The method of claim 14, wherein the unidentified or
misidentified speaker is a misidentified speaker; and wherein the
receiving the name includes: receiving, from the user, a correct
name of a speaker of the at least one of the audio segments.
24. The method of claim 23, further comprising: identifying one of
the speaker models, as an identified speaker model, that
corresponds to the at least one of the audio segments; and updating
a label associated with the identified speaker model to include the
correct name of the speaker of the at least one of the audio
segments.
25. A computer-readable medium that stores instructions executable
by one or more processors for speaker identification training by a
speaker identification system, comprising: instructions for
generating a plurality of speaker models based on training data;
instructions for presenting, to a user, audio segments for which no
speakers can be identified from the speaker models; instructions
for obtaining, from the user, a name of a speaker for at least one
of the audio segments; instructions for generating a new speaker
model for the speaker based on the at least one of the audio
segments; and instructions for associating the name of the speaker
with the new speaker model.
26. A speaker identification system, comprising: an indexer
configured to: receive a plurality of speech segments, each of the
speech segments being associated with a corresponding speaker,
create a plurality of documents by transcribing the speech
segments, identify names of the speakers corresponding to the
speech segments, the indexer being unable to correctly identify
names of at least one of the speakers corresponding to the speech
segments, the speakers for which the indexer can correctly identify
names being identified speakers and the speakers for which the
indexer cannot correctly identify names being unidentified
speakers; a database configured to store the documents; and a
server configured to: retrieve one or more of the documents from
the database, present the one or more of the documents to a user,
receive, from the user, a name for one of the unidentified
speakers, and provide the name for the one of the unidentified
speakers to the indexer for subsequent identification of speech
segments from the one of the unidentified speakers.
Description
RELATED APPLICATION
[0001] This application claims priority under 35 U.S.C. .sctn. 119
based on U.S. Provisional Application No. 60/419,214, filed Oct.
17, 2002, the disclosure of which is incorporated herein by
reference.
[0002] This application is related to U.S. patent application, Ser.
No. 10/______ (Docket No. 02-4042), entitled "Continuous Learning
for Speech Recognition Systems," filed concurrently herewith, and
U.S. patent application, Ser. No. 10/610,533 (Docket No. 02-4046),
entitled "Systems and Methods for Improving Recognition Results via
User-Augmentation of a Database," filed Jul. 2, 2003, the
disclosures of which are incorporated herein by reference.
BACKGROUND OF THE INVENTION
[0003] 1. Field of the Invention
[0004] The present invention relates generally to multimedia
environments and, more particularly, to systems and methods for
providing interactive speech identification training in a
multimedia environment.
[0005] 2. Description of Related Art
[0006] Conventional speaker identification systems require a huge
amount of training data to identify speakers based on audio data
received from the speakers. These systems typically train a set of
models based on the training data. When audio data is received by
one of these systems, the system segments the audio data into
speaker turns. The system then analyzes each of the speaker turns
to determine whether it matches one of the models. If a speaker
turn matches one of the models, the system labels the speaker turn
with the name of the speaker.
[0007] For proper speaker identification, the system requires a
huge amount of training data. For each speaker, for example, the
system requires several minutes of speech to generate a Gaussian
mixture model that can later be used to locate segments of speech
from the same speaker. In some situations, it is difficult to
obtain a sufficient amount of training data to generate an accurate
Gaussian mixture model. For example, it is difficult to obtain
sufficient training data from speakers who do not routinely speak
or who speak in only small bursts (i.e., one to two sentences at a
time).
[0008] As a result, there is a need for mechanisms to improve
speaker identification results even for speakers where there is
insufficient training data.
SUMMARY OF THE INVENTION
[0009] Systems and methods consistent with the present invention
collect data for training a speaker identification system in an
interactive manner. Users may be prompted to identify speaker turns
for unidentified or incorrectly identified speakers. The data from
the users may be used to generate new speaker models and/or update
links to existing speaker models. The speaker models may then be
used to correctly label speaker turns.
[0010] In one aspect consistent with the principles of the
invention, a system provides speaker identification training. The
system generates speaker models and receives audio segments. The
system identifies speakers corresponding to the audio segments
based on the speaker models. At least one of the audio segments has
an unidentified or misidentified speaker (i.e., an audio segment
whose speaker cannot be accurately identified). The system
presents, to a user, audio segments that include an audio segment
whose speaker is unidentified or misidentified and receives, from
the user, the name of the unidentified or misidentified speaker.
The system may use this information to subsequently identify the
unidentified or misidentified speaker by name for future audio
segments.
[0011] In another aspect consistent with the principles of the
invention, a speaker identification system is provided. The system
includes an indexer and a server. The indexer is configured to
generate speaker models, receive audio segments, and identify
speakers corresponding to the audio segments based on the speaker
models. The indexer is unable to correctly identify at least one of
the speakers corresponding to the audio segments. The server is
configured to receive, from a user, the name of an unidentified
speaker of the speakers corresponding to the audio segments, and
provide the name of the unidentified speaker to the indexer for
identification of the unidentified speaker in subsequent audio
segments.
[0012] In a further aspect consistent with the principles of the
invention, a computer-readable medium stores instructions
executable by one or more processors for speaker identification
training by a speaker identification system. The computer-readable
medium includes instructions for generating speaker models based on
training data; instructions for presenting, to a user, audio
segments for which no speakers can be identified from the speaker
models; instructions for obtaining, from the user, a name of a
speaker for at least one of the audio segments; instructions for
generating a new speaker model for the speaker based on the at
least one audio segment; and instructions for associating the name
of the speaker with the new speaker model.
[0013] In another aspect consistent with the principles of the
invention, a speaker identification system includes an indexer, a
database, and a server. The indexer receives speech segments, where
each of the speech segments has a corresponding speaker. The
indexer also creates documents by transcribing the speech segments
and identifies the names of the speakers corresponding to the
speech segments. The indexer is unable to correctly identify the
names of at least one of the speakers corresponding to the speech
segments. As a result, some of the speakers are identified speakers
and others of the speakers are unidentified speakers. The database
stores the documents. The server retrieves one or more of the
documents from the database and presents the one or more of the
documents to a user. The server receives, from the user, the name
of one of the unidentified speakers and provides the name of the
unidentified speaker to the indexer for subsequent identification
of speech segments from the unidentified speaker.
BRIEF DESCRIPTION OF THE DRAWINGS
[0014] The accompanying drawings, which are incorporated in and
constitute a part of this specification, illustrate the invention
and, together with the description, explain the invention. In the
drawings,
[0015] FIG. 1 is a diagram of a system in which systems and methods
consistent with the present invention may be implemented;
[0016] FIG. 2 is an exemplary diagram of the indexer of FIG. 1
according to an implementation consistent with the principles of
the invention;
[0017] FIG. 3 is an exemplary diagram of a portion of the
recognition system of FIG. 2 according to an implementation
consistent with the present invention;
[0018] FIG. 4 is an exemplary diagram of the memory system of FIG.
1 according to an implementation consistent with the principles of
the invention;
[0019] FIGS. 5A-5C are flowcharts of exemplary processing for
speaker identification training according to an implementation
consistent with the principles of the invention; and
[0020] FIG. 6 is a diagram of an exemplary graphical user interface
by which a document may be presented to a user according to an
implementation consistent with the principles of the invention.
DETAILED DESCRIPTION
[0021] The following detailed description of the invention refers
to the accompanying drawings. The same reference numbers in
different drawings may identify the same or similar elements. Also,
the following detailed description does not limit the invention.
Instead, the scope of the invention is defined by the appended
claims and equivalents.
[0022] Systems and methods consistent with the present invention
permit users to aid in the training of a speaker identification
system by, for example, correctly identifying unidentified and
misidentified speakers. The systems and methods may use the
user-identification of unidentified speakers to train new speaker
models for improved recognition results. The systems and methods
may use the user-identification of misidentified speakers to update
links to existing speaker models.
EXEMPLARY SYSTEM
[0023] FIG. 1 is a diagram of an exemplary system 100 in which
systems and methods consistent with the present invention may be
implemented. System 100 may include multimedia (MM) sources 110,
indexer 120, memory system 130, and server 140 connected to clients
150 via network 160. Network 160 may include any type of network,
such as a local area network (LAN), a wide area network (WAN)
(e.g., the Internet), a public telephone network (e.g., the Public
Switched Telephone Network (PSTN)), a virtual private network
(VPN), or a combination of networks. The various connections shown
in FIG. 1 may be made via wired, wireless, and/or optical
connections.
[0024] Multimedia sources 110 may include one or more audio sources
and/or one or more video sources. An audio source may include any
source of audio data, such as radio, telephone, and conversations
in any language. A video source may include any source of video
data with integrated audio data in any language, such as
television, satellite, and a camcorder. The audio and/or video data
may be provided to indexer 120 as a stream or file.
[0025] Indexer 120 may include mechanisms for processing audio
and/or video data. Indexer 120 may include mechanisms that receive
data from multimedia sources 110, process the data, perform feature
extraction, and output analyzed, marked-up, and enhanced language
metadata. In one implementation consistent with the principles of
the invention, indexer 120 includes mechanisms, such as the ones
described in John Makhoul et al., "Speech and Language Technologies
for Audio Indexing and Retrieval," Proceedings of the IEEE, Vol.
88, No. 8, August 2000, pp. 1338-1353, which is incorporated herein
by reference.
[0026] FIG. 2 is an exemplary diagram of indexer 120 according to
an implementation consistent with the principles of the invention.
Indexer 120 may include training system 210, statistical model 220,
and recognition system 230. Training system 210 may include logic
that estimates parameters of statistical model 220 from a corpus of
training data. The training data may initially include
human-produced data. For example, the training data might include
one hundred hours of audio data that has been meticulously and
accurately transcribed by a human. Training system 210 may use the
training data to generate parameters for statistical model 220 that
recognition system 230 may later use to recognize future data that
it receives (i.e., new audio that it has not heard before).
[0027] The training data might also include audio data for which
the speaker has been identified. Training system 210 may use the
training data to generate parameters for statistical speaker models
that recognition system 230 may later use to recognize speakers
from future data that it receives. To build a speaker model for a
new speaker, training system 210 may use a speaker independent
Gaussian mixture model with approximately 2,048 Gaussians. This
Gaussian mixture model may be trained based on a lot of diversified
speakers (e.g., speakers of different ages, different genders,
etc.). Training system 210 may use a conventional expectation and
maximization process to fit audio from a speaker to the model.
Training system 210 may then use a conventional maximum a
posteriori adaptation process to generate a final model for the
speaker.
[0028] Statistical model 220 may include acoustic models, language
models, and speaker models. The acoustic models may describe the
time-varying evolution of feature vectors for each sound or
phoneme. The acoustic models may employ continuous hidden Markov
models (HMMs) to model each of the phonemes in the various phonetic
contexts.
[0029] The language models may include n-gram language models,
where the probability of each word is a function of the previous
word (for a bi-gram language model) and the previous two words (for
a tri-gram language model). Typically, the higher the order of the
language model, the higher the recognition accuracy at the cost of
slower recognition speeds.
[0030] The speaker models may include a pool of models that are
used to identify speakers from their speech. Speech from a
particular speaker may be compared to the speaker models to
determine the likelihood that the speech was produced from each of
the models. The model with the highest likelihood may be determined
to correspond to that speech. The particular speaker corresponding
to the model may be determined based, for example, on a mapping
function that links speaker names to speaker models.
[0031] Recognition system 230 may use statistical model 220 to
process input audio data. FIG. 3 is an exemplary diagram of a
portion of recognition system 230 according to an implementation
consistent with the principles of the invention. Recognition system
230 may include speaker segmentation logic 310, speech recognition
logic 320, speaker clustering logic 330, and speaker identification
logic 340. In one implementation consistent with the principles of
the invention, the functions performed by speaker segmentation
logic 310, speech recognition logic 320, speaker clustering logic
330, and speaker identification logic 340 are similar to the
functions described in John Makhoul et al., "Speech and Language
Technologies for Audio Indexing and Retrieval," Proceedings of the
IEEE, Vol. 88, No. 8, August 2000, pp. 1338-1353, which was
previously incorporated herein by reference.
[0032] Generally, speaker segmentation logic 310 may distinguish
speech from silence, noise, and other audio signals in input audio
data. For example, speaker segmentation logic 310 may analyze each
thirty second window of the input data to determine whether it
contains speech. Speaker segmentation logic 310 may also identify
boundaries between speakers in the input stream. Speaker
segmentation logic 310 may group speech segments from the same
speaker and send the segments to speech recognition logic 320.
[0033] Speech recognition logic 320 may perform continuous speech
recognition to recognize the words spoken in the segments that it
receives from speaker segmentation logic 310. Speech recognition
logic 320 may generate a transcription of the speech using
statistical model 220. Speaker clustering logic 330 may identify
all of the segments from the same speaker in a single document
(i.e., a body of media that is contiguous in time (from beginning
to end or from time A to time B)) and group them into speaker
clusters (or speaker turns). Speaker clustering logic 330 may then
assign each of the speaker turns a unique label.
[0034] Speaker identification logic 340 may identify the speaker in
each speaker turn by name or, when the name cannot be determined,
by gender. Speaker identification logic 340 may use the speaker
models within statistical model 220 to identify speakers from their
speech. For example, speaker identification logic 340 may compare
speech from a particular speaker within a speaker turn to the
speaker models to determine the likelihood that the speech was
produced from each of the models. Speaker identification logic 340
may identify the model with the highest likelihood (above some
threshold) as the model corresponding to that speech. Speaker
identification logic 340 may include a mapping function that links
speaker names to the speaker models so that speaker identification
logic 340 can identify the speaker by name once a model has been
identified.
[0035] Returning to FIG. 1, memory system 130 may store documents
from indexer 120 and, possibly, documents from clients 150. FIG. 4
is an exemplary diagram of memory system 130 according to an
implementation consistent with the principles of the invention.
Memory system 130 may include loader 410, trainer 420, one or more
databases 430, and interface 440. Loader 410 may include logic that
receives documents from indexer 120 and stores them in database
430. Trainer 420 may include logic that sends speaker
identification information, such as audio data with identification
of the corresponding speaker, in the form of training data to
indexer 120.
[0036] Database 430 may include a conventional database, such as a
relational database, that stores documents from indexer 120.
Database 430 may also store documents received from clients 150 via
server 140. Interface 440 may include logic that interacts with
server 140 to store documents in database 130, query or search
database 130, and retrieve documents from database 130.
[0037] Returning to FIG. 1, server 140 may include a computer
(e.g., a processor and memory) or another device that is capable of
interacting with memory system 130 and clients 150 via network 170.
Server 140 may receive queries from clients 150 and use the queries
to retrieve relevant documents from memory system 130.
[0038] Clients 150 may include personal computers, laptops,
personal digital assistants, or other types of devices that are
capable of interacting with server 140 to retrieve documents from
memory system 130. Clients 150 may present information to users via
a graphical user interface, such as a web browser window.
EXEMPLARY PROCESSING
[0039] Systems and methods consistent with the present invention
permit users to assist in speaker identification training to
improve recognition results of system 100. For example, the user
may supply the name of an unidentified or misidentified speaker
that may be used to retrain indexer 120.
[0040] FIGS. 5A-5C are flowcharts of exemplary processing for
speaker identification training according to an implementation
consistent with the principles of the invention. Processing may
begin with a user desiring to retrieve one or more documents from
memory system 130. The user may use a conventional web browser of
client 150 to access server 140 in a conventional manner. To obtain
documents of interest, the user may generate a search query and
send the query to server 140 via client 150. Server 140 may use the
query to search memory system 130 and retrieve relevant
documents.
[0041] Server 140 may present the relevant documents to the user.
For example, the user may be presented with a list of relevant
documents. The documents may include any combination of audio
documents and video documents. The user may select one or more
documents on the list to view. When this happens, the user may be
presented with a transcription corresponding to the audio data from
the document. The user may also be presented with the audio and/or
video data corresponding to the transcription.
[0042] FIG. 6 is a diagram of an exemplary graphical user interface
(GUI) 600 by which a document may be presented to a user according
to an implementation consistent with the principles of the
invention. In one implementation, GUI 600 is part of an interface
of a standard Internet browser, such as Internet Explorer or
Netscape Navigator, or any browser that follows World Wide Web
Consortium (W3C) specifications for HTML.
[0043] GUI 600 may include a speaker section 610 and a
transcription section 620. Speaker section 610 may identify
boundaries between speakers, the gender of a speaker, and the name
of a speaker (when known). In this way, speaker segments are
clustered together over the entire document to group together
segments from the same speaker under the same label. In the example
of FIG. 6, one speaker, Elizabeth Vargas, has been identified by
name. Other speakers have been identified by gender and number. For
example, if "male 1" speaks again within the same document, his
speech will also be labeled "male 1." Transcription section 620 may
include a transcription of the document. In the example of FIG. 6,
the document corresponds to video data from a television broadcast
of ABC's World News Tonight.
[0044] GUI 600 may also include playback button 630. The user may
select playback button 630 to hear the audio (or possibly see the
video) corresponding to the transcription in transcription section
620. To facilitate this, the user may select a portion of the
transcription in transcription section 620 by, for example, using a
mouse to highlight the portion. The user may then select playback
button 630 to hear the audio (or see the video) corresponding to
the selected portion of the transcription.
[0045] GUI 600 may further include a correction button 640. The
user may select correction button 640 when the user desires to
supply the name of an unidentified speaker or correct the name of a
misidentified speaker within speaker section 610. Sometimes, a
speaker may be identified only by gender in speaker section 610
because indexer 120 (FIG. 1) can identify the speaker's gender but
cannot determine the speaker's name. This may occur when indexer
120 has not yet generated a speaker model corresponding to the
speaker due, for example, to a lack of audio data for the speaker.
Sometimes, a speaker may be misidentified (i.e., labeled with the
incorrect speaker name) in speaker section 610. This may occur when
indexer 120 incorrectly matches the audio data from the speaker to
a speaker model of another speaker. If the user desires, the user
may provide the name of an unidentified speaker or correct the name
of a misidentified speaker by selecting correction button 640 and
providing the correct information.
[0046] GUI 600 may receive the information provided by the user and
modify the document onscreen. This way, the user may determine
whether the information was correctly provided. GUI 600 may also
send the speaker identification information to server 140.
[0047] Returning to FIG. 5A, the user may express a desire to
provide the name of an unidentified speaker or correct the name of
a misidentified speaker. In doing so, the user may identify a
segment, such as a speaker turn, within transcription section 620
(FIG. 6) (act 505). The user may identify the speaker turn by
selecting the corresponding segment using, for example, a mouse or
by selecting one of the labels within speaker section 610. The user
may then select correction button 640 to initiate the
correction.
[0048] Client 150 may determine whether the user has selected to
identify the name of an unidentified speaker or correct the name of
a misidentified speaker (act 510). Client 150 may make this
determination based on the current speaker label provided in
speaker section 610 for the segment identified by the user. For
example, if the current speaker label includes a gender
identification, then client 150 may determine that the user has
selected to identify the name of an unidentified speaker. If, on
the other hand, the current speaker label includes something other
than a gender identification, then client 150 may determine that
the user has selected to correct the name of a misidentified
speaker.
[0049] If the user desires to provide the name of an unidentified
speaker, the user may input the name in any conventional manner
(act 515) (FIG. 5B). Client 150 may provide the name and, possibly,
other identifying information, as speaker identification
information, to server 140. The other identifying information may
include, for example, an indication of the audio segment to which
the name corresponds, the audio (or video) data corresponding to
the segment, and/or other information that may be useful for
training indexer 120 to later identify audio data from this same
speaker.
[0050] Server 140 may determine whether additional audio data is
necessary or desired (act 520). For example, server 140 may
determine whether the amount of audio data associated with the
audio segment labeled by the user is less than approximately four
minutes of audio. As explained above, the minimum amount of audio
data needed to build a good Gaussian mixture model (with over 2,000
Gaussians) is approximately four minutes of audio.
[0051] If the audio data associated with the audio segment is less
than four minutes in length, then server 140 may locate additional
audio segments that may be from the same speaker (act 525). Server
140 may identify these additional audio segments based on the
speaker clustering performed by indexer 120. Using the speaker
clustering, more, similar audio segments may be identified in the
same document and in other documents within memory system 130.
[0052] For each audio segment located, sever 140 may confirm with
the user that the audio segment is from the same speaker identified
by the user (act 530). Server 140 may continue to obtain additional
audio segments and confirm them with the user until it obtains
around four minutes or more of audio data or the user expresses a
desire to cease the confirmations.
[0053] Once server 140 collects a sufficient amount of audio data
or the user stops the confirmations, server 140 may provide the
audio data, along with the name of the speaker, to indexer 120 as
training data. Indexer 120 may use the audio data to build a
speaker model for the new speaker (act 535). As described above,
indexer 120 may use a speaker independent Gaussian mixture model
with approximately 2,048 Gaussians. This Gaussian mixture model may
be trained based on a lot of diversified speakers (e.g., speakers
of different ages, different genders, etc.). Indexer 120 may use a
conventional expectation and maximization process to fit the audio
data from the speaker to the model. Indexer 120 may then use a
conventional maximum a posteriori adaptation process to generate a
final model for the speaker.
[0054] Indexer 120 may add the speaker model to the pool of speaker
models within statistical model 220 (FIG. 2) (act 540). Thereafter,
indexer 120 may correctly identify new audio data that it receives
corresponding to the new speaker.
[0055] If, instead of providing the name of an unidentified
speaker, the user desires to correct the name of a misidentified
speaker, the user may optionally input the correct name in any
conventional manner (act 545) (FIG. 5C). Client 150 may provide the
correct name and, possibly, other identifying information, as
speaker identification information, to server 140. The other
identifying information may include, for example, an indication of
the segment (e.g., speaker turn) to which the name corresponds, the
audio (or video) data corresponding to the segment, and/or other
information that may be useful for indexer 120 to identify the
speaker model corresponding to the speaker.
[0056] Server 140 may determine whether confirmation of additional
audio data is necessary or desired (act 550). For example, a
certain amount of audio data may be needed to accurately identify
the incorrectly labeled speaker model. In this case, server 140 may
identify other audio segments that are close to the current segment
(i.e., the speaker turn that the user identified as mislabeled)
(act 555). Server 140 may use information relating to the speaker
clustering performed by indexer 120 to identify these other
potentially mislabeled audio segments. For each segment located,
sever 140 may confirm with the user that the speaker turn is also
labeled incorrectly (act 560). Server 140 may continue to obtain
additional audio segments and confirm them with the user until it
until some threshold is met or the user expresses a desire to cease
the confirmations.
[0057] Server 140 may then provide the audio data, along with the
correct name of the speaker, to indexer 120. Indexer 120 may use
the audio data to identify the speaker model from which the audio
data was produced. Indexer 120 may then update the name associated
with the identified speaker model (act 565). For example, a mapping
may exist between the names of speakers and the corresponding
speaker models in statistical model 220. Indexer 120 may update the
mapping to identify the correct name of the speaker for that
model.
[0058] In an alternate implementation, the user knows only that the
speaker is misidentified, but does not know the true identity of
the speaker. For example, the user may indicate simply that the
name of the speaker associated with the speaker turn is incorrect.
In this case, indexer 120 may simply remove the mapping of the
incorrect speaker name to the corresponding speaker model.
[0059] When indexer 120 thereafter comes across audio data
corresponding to a speaker that has been identified by the user
(either as a result of an unidentified or misidentified speaker
turn), indexer 120 compares the audio data to the models in the
speaker model pool. Indexer 120 may determine the likelihood that
the audio data was created by each of the speaker models. Indexer
120 may identify the speaker model with the highest likelihood
(above some threshold) as the speaker model corresponding to the
audio data. Indexer 120 may then use its mapping of speaker models
to speaker names to identify the speaker of that audio data by
name. Indexer 120 may also update the documents in memory system
130 with the speaker name(s) identified by the user.
CONCLUSION
[0060] Systems and methods consistent with the present invention
provide an interactive speaker identification training system. Via
these systems and methods, users are permitted to assist in the
speaker identification training by, for example, correctly
identifying unidentified and misidentified speakers. The systems
and methods may use the user-identification of unidentified
speakers to train new speaker models for improved recognition
results. The systems and methods may use the user-identification of
misidentified speakers to update links to existing speaker
models.
[0061] The foregoing description of preferred embodiments of the
present invention provides illustration and description, but is not
intended to be exhaustive or to limit the invention to the precise
form disclosed. Modifications and variations are possible in light
of the above teachings or may be acquired from practice of the
invention.
[0062] For example, an exemplary graphical user interface has been
described with regard to FIG. 6 as containing certain features
consistent with the principles of the invention. It is to be
understood that a graphical user interface, consistent with the
present invention, may include any or all of these features or
different features to facilitate user assistance in speaker
identification training.
[0063] Further, a graphical user interface has been described as
performing certain functions. It is to be understood that some, if
not all, of these functions may be performed by client 150 or
server 140.
[0064] While series of acts have been described with regard to
FIGS. 5A-5C, the order of the acts may differ in other
implementations consistent with the principles of the
invention.
[0065] Further, certain portions of the invention have been
described as "logic" that performs one or more functions. This
logic may include hardware, such as an application specific
integrated circuit or a field programmable gate array, software, or
a combination of hardware and software.
[0066] No element, act, or instruction used in the description of
the present application should be construed as critical or
essential to the invention unless explicitly described as such.
Also, as used herein, the article "a" is intended to include one or
more items. Where only one item is intended, the term "one" or
similar language is used. The scope of the invention is defined by
the claims and their equivalents.
* * * * *