U.S. patent application number 12/116472 was filed with the patent office on 2008-10-23 for synchronous multi-media recording and playback with end user control of time, data, and event visualization for playback control over a network.
This patent application is currently assigned to ATELLIS, INC.. Invention is credited to Lucas K. Huang, Chafic A. Kazoun.
Application Number | 20080261192 12/116472 |
Document ID | / |
Family ID | 39527754 |
Filed Date | 2008-10-23 |
United States Patent
Application |
20080261192 |
Kind Code |
A1 |
Huang; Lucas K. ; et
al. |
October 23, 2008 |
SYNCHRONOUS MULTI-MEDIA RECORDING AND PLAYBACK WITH END USER
CONTROL OF TIME, DATA, AND EVENT VISUALIZATION FOR PLAYBACK CONTROL
OVER A NETWORK
Abstract
A system and system are provided for providing synchronous
multimedia recording and playback. Aspects the exemplary embodiment
include in response to a training session is being conducted,
synchronously recording in real-time simulator data from a
simulator captured by a simulator capture tool, and video of the
training session captured by a plurality of A/V sources; encoding
each of the videos captured by the plurality of A/V sources as
respective digital media files formatted as streaming media; and
transmitting both the simulator data and the video media files from
a server to a client over a network, such that when the client
receives the simulator data and the stream, the respective videos
are synchronously played back with the simulator data on the
client
Inventors: |
Huang; Lucas K.; (Columbia,
MD) ; Kazoun; Chafic A.; (Chevy Chase, MD) |
Correspondence
Address: |
STRATEGIC PATENT GROUP, P.C.
P.O. BOX 1329
MOUNTAIN VIEW
CA
94042
US
|
Assignee: |
ATELLIS, INC.
Silver Spring
MD
|
Family ID: |
39527754 |
Appl. No.: |
12/116472 |
Filed: |
May 7, 2008 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
11611792 |
Dec 15, 2006 |
|
|
|
12116472 |
|
|
|
|
Current U.S.
Class: |
434/336 |
Current CPC
Class: |
G09B 9/00 20130101 |
Class at
Publication: |
434/336 |
International
Class: |
G09B 9/00 20060101
G09B009/00 |
Claims
1. A method for providing synchronous multimedia recording and
playback, comprising: in response to a training session is being
conducted, synchronously recording in real-time simulator data from
a simulator captured by a simulator capture tool, and video of the
training session captured by a plurality of A/V sources; encoding
each of the videos captured by the plurality of A/V sources as
respective digital media files formatted as streaming media; and
transmitting both the simulator data and the video media files from
a server to a client over a network, such that when the client
receives the simulator data and the stream, the respective videos
are synchronously played back with the simulator data on the
client.
2. The method of claim 1 wherein the streaming further comprises
dynamically generating a single media file that references as
sources the respective streaming media files, and streaming the
single media file.
3. The method of claim 1 further comprising playing back the
simulator data as a trend of the telemetry values along a
timeline.
4. The method of claim 3 further comprising identifying transition
points of the telemetry values during playback.
5. The method of claim 1 further comprising playing back the videos
on the client in a streaming media player, and playing back the
simulator data in a flash player.
6. The method of claim 1 further comprising allowing an end-user of
the client to advance or return to any point in time in the
synchronous play back of the videos.
7. The method of claim 1 further comprising streaming the
synchronous playback in real-time during the training session.
8. The method of claim 1 further comprising enabling an end-user to
enter annotation and assessment data while the videos and
simulation data are synchronously played back.
9. The method of claim 8 further comprising during the synchronous
playback, displaying a checklist of training tasks to be completed
by a trainee during the trading session, and allowing an end-user
to indicate through the interface which tasks were completed during
the synchronous playback for real-time annotation and assessment of
the trainee.
10. A system for providing synchronous multimedia recording and
playback, comprising: a plurality of A/V sources for capturing
video of a training session, the training session including a use
of a simulator; a simulation capture tool for capturing real-time
simulator data from the simulator, wherein the simulator data and
the video of the training session are recorded synchronously; one
or more encoders for encoding each of the videos captured by the
plurality of A/V sources as respective digital media files
formatted as streaming media; and means for transmitting both the
simulator data and the video media files from a server to a client
over a network, such that when the client receives the simulator
data and the stream, the respective videos are synchronously played
back with the simulator data on the client.
11. The system of claim 10 wherein the means for transmitting
include a skills assessment tool and a video on demand server.
12. The system of claim 11 wherein the means skills assessment tool
includes: an annotation and assessment tool for enabling an
end-user to enter annotation and assessment data of a trainee's
performance while the videos and simulation data are synchronously
played back; and a debrief tool for allowing the end-user to select
which recorded training session to view for enabling automated
debrief sessions.
13. The system of claim 10 wherein a single media file is
dynamically generated that references as sources the respective
streaming media files, and streaming the single media file.
14. The system of claim 10 wherein the simulator data is played
back as a trend of the telemetry values along a timeline.
15. The system of claim 14 wherein transition points of the
telemetry values are identified during playback.
16. The system of claim 10 wherein the videos are played back on
the client in a streaming media player, and playing back the
simulator data in a flash player.
17. The system of claim 10 wherein an end-user of the client is
allowed to advance or return to any point in time in the
synchronous play back of the videos.
18. The system of claim 10 wherein the synchronous playback is
streamed in real-time during the training session.
19. The system of claim 10 wherein during the synchronous playback,
a checklist of training tasks to be completed is displayed by a
trainee during the trading session, and an end-user is allowed to
indicate through the interface which tasks were completed during
the synchronous playback for real-time annotation and assessment of
the trainee.
20. An executable software product stored on a computer-readable
medium containing program instructions for providing synchronous
multimedia recording and playback, wherein in response to a
training session is being conducted, video of the training session
is captured by a plurality of A/V sources, the program instructions
for: synchronously recording with the video, real-time simulator
data from a simulator captured by a simulator capture tool;
encoding each of the videos captured by the plurality of A/V
sources as respective digital media files formatted as streaming
media; and transmitting both the simulator data and the video media
files from a server to a client over a network, such that when the
client receives the simulator data and the stream, the respective
videos are synchronously played back with the simulator data on the
client.
Description
CROSS-REFERENCE TO RELATED APPLICATION
[0001] This application is a continuation of co-pending patent
application Ser. No. 11/611,792, filed Dec. 15, 2006, assigned to
the assignee of the present application, and incorporated herein by
reference.
BACKGROUND OF THE INVENTION
[0002] The use of simulation training is growing rapidly. A
simulation training session is one in which training of personnel
is performed through the use of a simulator device that outputs
real-time data in response to interactions of the trainees. In the
medical industry, for example, medical training centers conduct
simulation training that generally involve students performing
simulated medical procedures and/or examinations on a mannequin
simulator, which exhibits symptoms of various ailments of a patient
during simulated examination sessions. Other types of medical
simulators include EKG machines, blood pressure monitors, and
virtual reality endoscopic, laparoscopic, and endovascular
simulators. During each simulated examination session, which
usually takes place in an assigned examination room, the student
interacts with the patient during an appointed time period to make
a diagnosis of the patient's ailment and to prescribe a proposed
treatment plan or perform a procedure. Each examination room is
equipped with monitoring equipment, including audio, visual and
time recoding devices, so that the student's simulated encounter
with the patient can be monitored in real time by an evaluator,
such as a faculty member or upper class person. Typically,
simulation training sessions are also recorded on video for
subsequent analysis and teaching purposes. A similar configuration
is used in other industries for other types of training
sessions.
[0003] The monitoring equipment in the examination rooms may
include multiple audio/video (A/V) sources, e.g. video cameras, to
provide various camera angles of the training session. A typical
recording training session may have three video feeds, for
instance, taken from different camera angles, and one of the video
feeds might show a machine that displays data from a simulator,
such as EKG, heart rate, or blood pressure data. The data from each
of the A/V sources is sent to a respective recording/playback
device, e.g., a digital video (DV) recorder, for recording onto
some type of hard recording medium, such as DVDs or DV tapes. This
results in the output of each of the video cameras, for example, to
be stored on separate medium during the training session.
Optionally, the system may have the recording/playback devices
synced together by a time sync generator. Each of the recording
mediums produced by the multiple recording/playback devices,
whether DVDs or DV tapes, are typically stored as a tape archive in
a multimedia library.
[0004] In addition, the video recorded by each of the A/V sources
may also be input to a video/audio mixer or processor of some type.
Typically, the mixer merges the video feeds from the A/V sources
and the output is recorded onto a recording medium as a merged
video with multiple windows, one for each video feed. Another
method is to overlay the simulator data as a composite image with
the video feeds, like a picture-in-picture. One problem with this
method, however, is that typically the overlay picture obscures
part of the underlying image. The recording media, e.g., DVD or DV
tape, may then also be archived in the multimedia library. After
all the video is stored and edited, each of the videos needs to be
manually associated with each of the trainees for later
retrieval.
[0005] While viewing the training session during the recording, a
trainer manually identifies performance events/issues, and manually
notes the time during the video in which the event occurred. Once
the simulation exercise is completed, the recording is stopped.
Thereafter, the trainer conducts a debriefing session with the
trainee(s) to evaluate the trainee's performance. Debriefing
sessions can be performed right after the training session using
the unmixed recordings, during which, the trainer plays back
portions of recordings and analyzes the performance of the trainee
using their notes as a guide. Since unmixed recordings are used,
this process often involves lots of rewinding and fast-forwarding
to get to points of interest.
[0006] Sometimes the trainer may desire to perform what is known as
a highlighted debriefing session. In a highlighted debriefing
session, the trainer plays back just a portion of the training
session(s) for a detailed analysis with the trainee. To enable the
highlighted debrief session, a post-video editing process is
required to extract clips of specific examples from the various
media stored in the multimedia library. For example, assume that a
trainer wants to make a video of where trainee makes the most
mistakes. This would require the finding and retrieving of the
videos that contain the key clips from the multimedia library.
During the video editing process, the clips are extracted from the
video, and then either merged to create a video overlay, or the
clips are mixed and alternated. After the video editing process is
completed, the modified video is stored in the multimedia library,
and then retrieved by the trainer in order to conduct the
highlighted debrief session.
[0007] Although recording simulation training sessions has definite
advantages in terms of being a useful teaching tool, the
conventional system described above has several problems. One
problem is that the system does not allow quantifiable individual
feedback. Instead, the training session is often subjectively
evaluated or graded based on what the trainer or reviewer thought
they saw or didn't see during recorded exercise. A related problem
is that to comment on a specific event that occurred during the
training session, the trainer or reviewer must either remember
where in the recorded event occurred, or note a time index in their
notes. In addition, if the trainer wants to highlight a specific
area of interest in the recording, the trainer may have to have the
recording played and the segment of interest recorded separately
during the editing process. For training centers that have a large
number of training sessions and a large number of recordings per
training session, such constant editing can be a significant burden
in terms of manpower and cost.
[0008] Another problem is that the training sessions are stored on
media such as DVD or DV tape that must be manually indexed and
stored for archival. Since this is a manual process, problems may
arise when attempting to find a particular recording for a
particular training session or trainee. In addition, the recordings
archived in the multimedia library may not be readily accessible to
reviewers, particularly if the reviewers are not in the same
location as the multimedia library. For these reasons, access to
the recordings may not be possible or highly cumbersome as a number
of training sessions recorded increases, which in turn, may limit
the number of users who can view the recordings for evaluation or
grading.
[0009] Accordingly, a need exists for an improved method and system
for providing synchronous multimedia recording and playback,
particularly in the area of simulation training as one example.
BRIEF SUMMARY OF THE INVENTION
[0010] A method and system is provided for providing synchronous
multimedia recording and playback. Aspects the exemplary embodiment
include in response to a training session is being conducted,
synchronously recording in real-time simulator data from a
simulator captured by a simulator capture tool, and video of the
training session captured by a plurality of A/V sources; encoding
each of the videos captured by the plurality of A/V sources as
respective digital media files formatted as streaming media; and
transmitting both the simulator data and the video media files from
a server to a client over a network, such that when the client
receives the simulator data and the stream, the respective videos
are synchronously played back with the simulator data on the
client.
[0011] According to the method and system disclosed herein, all
available training session video and simulator sources may be
encoded into a streamable format as independent video files and
streamed onto the network. This facilitates remote viewing of
activity and real-time performance assessments of the training
session. The assessment information is instantly tabulated and
available to help drive a more objective feedback/debrief session
using actual simulation trend data and multiple assessments, all
synchronously tied to multiple video feeds.
BRIEF DESCRIPTION OF SEVERAL VIEWS OF THE DRAWINGS
[0012] FIG. 1 is a block diagram of a simulation training system in
accordance with an exemplary embodiment.
[0013] FIG. 2 is a flow diagram illustrating a process implemented
by the simulation training system for providing synchronous
multimedia recording and playback in accordance with the exemplary
embodiment.
[0014] FIG. 3A is a flow diagram illustrating a process performed
by the skills assessment tool synchronous display the simulation
data and streaming video media files on the client.
[0015] FIG. 3B is a diagram graphically illustrating a composite
media file according to an exemplary embodiment.
[0016] FIGS. 4A through 4D are example screenshots of the interface
of the skills assessment tool when the playing the stream.
[0017] FIG. 5 is a flow diagram illustrating a workflow process
implemented by the simulated training system and its interaction
with the end-user for recording, annotating, and debriefing a
recorded training session.
[0018] FIG. 6A is a diagram of an example assessment display
screen.
[0019] FIG. 6B is a diagram showing another example of an
assessment display screen.
[0020] FIG. 7 is a block diagram illustrating a process performed
by the debrief tool and its interaction with the end-user for
conducting an individual debriefing session between a trainer and a
trainer.
DETAILED DESCRIPTION OF THE INVENTION
[0021] The present invention relates to synchronous multimedia
recording and playback. The following description is presented to
enable one of ordinary skill in the art to make and use the
invention and is provided in the context of a patent application
and its requirements. Various modifications to the preferred
embodiments and the generic principles and features described
herein will be readily apparent to those skilled in the art. The
end-user is also allowed to select what simulator variables are
displayed. Thus, the present invention is not intended to be
limited to the embodiments shown, but is to be accorded the widest
scope consistent with the principles and features described
herein.
[0022] The embodiments disclosed herein are mainly described in
terms of particular device and system provided in particular
implementations. However, one of ordinary skill in the art will
readily recognize that this method and system will operate
effectively in other implementations. For example, the devices
usable with the present invention can take a number of different
forms. The present invention will also be described in the context
of particular methods having certain steps. However, the method and
system operate effectively for other methods having different
and/or additional steps not inconsistent with the present
invention.
[0023] The exemplary embodiment provides a web-based simulation
training system for providing synchronous multimedia recording and
playback of recorded training sessions. Training sessions are
recorded, and the recordings include not only synchronized video
from multiple cameras, but also simulation data recorded from a
simulator. Streaming technology is then utilized to allow end-users
to access the recordings over the Internet via a browser and view
and configure the recorded training sessions in real-time. The
end-user is allowed to select which video streams are played back
and to jump to any point along the recording time line, at which
point all the videos and simulation data automatically play back at
that point in time. Thus, the system provides synchronous
multimedia recording and playback with user playback control of
time, data, and event visualization over a network.
[0024] FIG. 1 is a block diagram of a simulation training system in
accordance with an exemplary embodiment. The exemplary embodiment
provides a web-based simulation training system 10 for providing
synchronous multimedia recording and playback of recorded training
sessions. The simulation training system 10 includes a training
center 12 that includes equipment for communicating with a
simulation training website 14 over a network 16, such as Internet.
The training center 12 conducts and records simulation training
sessions in one or more training rooms equipped with multiple
audio/video (A/V) sources 18, multiple encoder/recorders 20, a time
sync generator 22, and a simulator data source 24.
[0025] The training sessions are recorded using the A/V sources 18
and the data is sent to respective encoders/recorders 20. The A/V
sources 18 in an exemplary environment will be described as video
cameras, but A/V sources 18 include any type of capture device,
such as an auxiliary microphone or a still camera, and the like.
The training sessions involve one or more trainees (not shown) who
perform simulated procedures, or otherwise interact with, at least
one simulator data source 24 that outputs real-time data in
response. The type of training conducted by the training center 12
will be described in terms of medical training that would be
suitable for doctors, nurses, and emergency response personnel, but
the exemplary embodiments are applicable any type of training that
involves the use of a any type of simulator. Example types of
simulator data sources 24 in the medical industry, for instance,
include full-body mannequin simulators, virtual reality simulators,
EKG machines, and blood pressure monitors.
[0026] The online simulation training website 14 includes a
software suite referred to as a skills assessment tool 26, a web
server 28a and a video-on-demand server 28b (collectively referred
to as server 28), a session data archive 30, a simulation data
archive 32, and a multimedia archive 34. The skills assessment tool
26 hosted on the simulation training website 14 includes a debrief
tool 38 and annotation and assessment tool 40. The server 28
hosting the simulation training website 14 may be implemented as
one server or any number of servers.
[0027] In another embodiment, the encoders/recorders 20 and the
simulation capture tool 36 may be located remote from the training
center, e.g., at the physical location of the simulation training
website 14. In another embodiment, all the components shown in the
simulation training website 14, including the encoders/recorders 20
and the simulation capture tool 36 may be implemented as a single
physical device. Further, the simulation training website 14 may be
implemented as a custom application that is installed at the
training center 12, and accessed directly by clients 42 over a
network.
[0028] FIG. 2 is a flow diagram illustrating a process implemented
by the simulation training system 10 for providing synchronous
multimedia recording and playback in accordance with the exemplary
embodiment. Referring to both FIGS. 1 and 2, the process begins in
response to a training session being conducted by synchronously
recording in real-time both simulator data from a simulator data
source 24 captured by a simulator capture tool, and video of the
training session captured by a plurality of the A/V sources 18
(block 200).
[0029] According to the exemplary embodiment, the simulator data is
captured by simulation capture tool 36. The time sync generator 22
is coupled to the encoders/recorders 20 and to the simulation
capture tool 36 to control the synchronization of the recordings.
In one embodiment, the simulation capture tool 36 is provided by
the simulation training website 14, but is installed and executed
on a computer (not shown) at the training center 12 that
communicates with the simulator data source 24. In another
embodiment, the simulation capture tool 36 may be located remote
from the simulator data source 24, such as at the simulation
training website 14.
[0030] During the recording, the each of the videos captured by A/V
sources 18 are encoded as respective digital media files in
streaming media format (block 202). As used herein, streaming media
is media that is consumed (heard and/or viewed) while the media is
being delivered. The videos captured by the A/V sources 18 may be
encoded by the encoders/decoders 20. In one embodiment, the digital
media files are encoded as MPEG-4 files, but other formats may also
be used.
[0031] In the exemplary embodiment, the simulator data may be
captured 36 as telemetry values captured in its raw and/or
compressed format. The telemetry values can then be visualized
using a thin client, such as Flash Player.TM., as a function of
time. In another embodiment, the simulator data can be captured
using one of the A/V sources 18 by recording a video of the output
of the simulator itself, e.g., by capturing a video of an EKG
display. The simulation data may be encoded by the simulation
capture tool 36.
[0032] During recording of the training session, the simulation
data and the digital media files of the video feeds are transmitted
to the simulation training website 14. The simulation data is sent
to the simulation training website 14 by simulation capture tool
36, where it stored in the simulation data archive 32 and indexed
by an ID of the training session. The video media files are sent to
the simulation training website 14 by the encoders/decoders and are
stored in the multimedia archive 34.
[0033] After all the captured data is transmitted to the simulation
training website 14 and stored, the server 28 transmits both the
simulator data and a stream of the digital media files to the
client 42 over the network 16, such that when the client 42
receives the simulator data and the stream, the respective videos
are synchronously played back with the simulator data on the client
42 (block 204).
[0034] In one embodiment, the skills assessment tool 26 causes the
server 28 to transmit the simulator data and the stream of the
video media files in response to receiving a request to view a
recorded training session. Referring to FIG. 1, according to the
exemplary embodiment, an end-user of a client 42 may access the
assessment tool 40 using a browser 44 and submit a request to view
the training session. The end-user request can be made prior to, or
during, a live training session for real-time viewing of the
recording, or after the training session is complete. If the
request is for a real-time viewing of the recording, then the
end-user may represent a trainer/faculty member, a trainee/student
observer, or other type of evaluator/reviewer. If the request is
for a prerecording training session, then the end-user may also
represent the trainers and trainees that took part in the training
session. In another embodiment, the skills assessment tool 26
causes the server 28 to transmit the simulator data and the stream
of the video media files to the client 42 automatically based on
some preconfigured settings.
[0035] FIG. 3A is a flow diagram illustrating a process performed
by the skills assessment tool 26 for synchronous display of the
simulation data and the stream of video media files on the client
42. The process begins in response to the skills assessment tool 26
receiving a request to view a training session (block 300). The
skills assessment tool 26 then dynamically generates a single
composite media file referencing as sources the video media files
stored for a training session (block 302). In one embodiment, the
composite media file is encoded as a streaming MPEG-4 file.
[0036] FIG. 3B is a diagram graphically illustrating a composite
media file according to an exemplary embodiment. In this example,
the skills assessment tool 26 created a composite media file 320 in
MPEG-4 movie format that includes three links referencing three
video media files 322.
[0037] Referring again to FIG. 3A, after the composite media file
320 is generated, one of the referenced video media files 322 is
designated as a primary audio source (block 304) to control audio
playback during synchronous video playback. In one embodiment, the
first video media file 322 referenced in the composite media file
300 may be designated the primary audio source by default. The
audio portion of the primary audio source may be used to
synchronize the videos at the time of playback. The audio portions
of the other video media files 322 may be turned-off.
[0038] The skills assessment tool 26 has a user interface that is
displayed in the browser 44 of the client 42. To fulfill the
end-user's request to view the training session, the skills
assessment tool 26 sends the composite media file 320 to the
video-on-demand server 28b. The video-on-demand server 28b streams
the composite media file 320 to browser of the designated client 42
by using the links to pull the source media files 322 and 324 from
the archives 32 and 34 and sending them through the stream (block
306). Thus, since the media files are added to the composite media
file 322 prior to playback and viewing, the exemplary embodiment
provides a system for premixing the video, rather than a mixing of
the video during playback. At the same time the skills assessment
tool 26 retrieves the related simulation data from the simulation
data archive 32 and sends the simulation data to the browser 44
(308), in one embodiment, via the web server 28a.
[0039] When the simulation data and the streamed composite media
file 322 is received on the client 42, a media player 46 compatible
with the format of the stream is automatically invoked. Example
types of media players 46 for playing streaming media include Apple
QuickTime.TM. and Flash Player.TM., for instance. The media player
46 then visualizes the simulation data and plays it in
synchronization with the videos based on the time of the
recordings. In one embodiment, the media player 46 opens within the
browser 44. In another embodiment, the media player 46 opens
outside the browser 44. In another embodiment, the videos may be
streamed individually from the server 28b, and then synchronized on
the client 42 by the media play 46.
[0040] Because the media player 46 receives a single streamed media
file referencing each of the videos, the simulator data and each of
the videos (with audio) can be synchronously played in the single
interface of the assessment tool 40 as displayed by the media
player 46. More specifically, the display screen of the assessment
tool 40 is divided into separate windows corresponding to each of
the source files included in the composite media file 300 and the
simulation data.
[0041] FIGS. 4A through 4B are example screenshots of the interface
of the skills assessment tool 26 when the playing the stream. In
the example shown in FIG. 4A, a skills assessment tool display
screen 400 is shown that is divided into respective panes or
windows 402 for playing each of the source videos files, and a
window 404 for visualizing the simulator data. The simulator data
may be displayed as a trend of telemetry values output from the
simulator data source 24 as a function of time. Also shown is a
timeline 406 of the recorded training session. In one embodiment,
in the case where a prerecorded training session is played, the
timeline 406 may also display points of interest along the timeline
that were flagged by reviewers/evaluators during an annotation
session, as described further below.
[0042] In this particular example, the simulator data shows
physiological data may have been captured by a video screen
capture, or by a visualization created from raw data captured from
the simulator data source 24. The videos in each of the four
windows 402 and 404, and the timeline 406 are played back
synchronously, but each video is an independent and fully editable
video file. By interacting with the timeline 406, the end-user is
allowed to advance or return to any point in time in the
synchronous playback of the videos.
[0043] FIG. 4B is a skills assessment tool display screen 420
synchronously playing three video windows 422, and one simulator
data window 424 displaying the simulation data in a manner that
identifies transition points of the telemetry values. An example of
a transition point is summation data showing that a mannequin
patient simulator went into cardiac arrest. Transition points are
often an indicator of where during the training the trainee needs
to perform a particular procedure and within a specified time
frame.
[0044] FIG. 4C is a skills assessment tool display screen 450
synchronously playing one video windows 452 with another video
window 454 playing video of the display screen of a medical device
data source.
[0045] FIG. 4D is a skills assessment tool display screen 460
synchronously playing three video windows 464 and a simulation data
window 464. The simulation data window shows a graph of telemetry
information as well as a list 466 of variables that may be
displayed in the graph. In this embodiment, the end-user can select
from the list which variables to display to selectively control
visualization of the telemetry information. The end-user may also
select not only which variables to visualize, but also to specify
an actual value for a selected variable to see where the value
appears along the timeline.
[0046] The above examples show that the exemplary embodiments
enable synchronous multimedia recording and playback, where with
viewer is provided with remote playback control of the recording
over a network of time, data, and event visualization.
[0047] FIG. 5 is a flow diagram illustrating a workflow process
implemented by the simulated training system 100 and its
interaction with the end-user for recording, annotating, and
debriefing a recorded training session. The process begins when a
training session is initiated and the simulator data source 24
started (block 502). In response, the simulation capture tool 36
detects the starting of the simulator data source 24 and
automatically starts training session recording by invoking the A/V
sources 18 and encoders/recorders 20 (block 504).
[0048] As stated above, an end-user can access the skills
assessment tool 26 and submit a request to view the training
session live, and in response, the recorded training session is
streamed to the client with the simulation data. According to a
further aspect of the exemplary embodiment, the annotation and
assessment tool 40 also enables the end-user to enter annotation
and assessment data of the trainee's performance while the videos
and simulation data are synchronously played back (block 506). This
process is referred to as an annotation session.
[0049] In operation, once the end-user logs in to the web server
28a from a browser 44 and accesses the assessment tool 40 (FIG. 1),
the web server 28a displays a list of live or prerecorded training
sessions to access, or the end-user can search for a training
session by entering metadata such as by case identification, or
trainee identification.
[0050] In response to the user selecting a recorded training
session, an assessment screen is displayed for displaying the
training session. In the case of a live training session, the
training session is played once the training session begins. In the
case of a pre-recorded training session, the pre-recorded trade
session is played automatically. Based on the session ID of the
selected session, the session data from the session data archive 30
and the simulation data from the simulation data archive is
retrieved and transmitted over the network by the servers 28a and
28b, as described above.
[0051] FIG. 6A is a diagram of an example assessment display screen
600. In this embodiment, the annotation and assessment tool 40
displays video windows 602 and a window 604 for displaying a
checklist 606 listing predefined training tasks to be completed by
a trainee during the training session, and allows the end-user to
indicate which of the listed tasks were completed during
synchronous playback, thereby providing real-time annotation and
assessment of the trainee. In this example, the end-user may
provide answers to the checklist by simply clicking "Yes" or "No".
In one embodiment, some of the tasks on the checklist 606 may be
associated with a timestamp that specifies at what time during the
training session particular item should have been performed. As a
reviewer checks off tasks on the checklist, the annotation and
assessment tool 40 compares the specified time with the time the
reviewer checks off the task and indicates whether the task was
performed within the specified time. Point values may be assigned
based on proper or improper execution of the predefined tasks.
Providing a predefined list of training tasks for evaluators to
complete provides a more objective approach in providing skills
assessment.
[0052] FIG. 6B is a diagram showing another example of an
assessment display screen 620. In this example, the screen 620
includes three video windows 622 and a simulation window 624
showing a trend of simulation data. In addition, a dialog box 626
is displayed that allows the reviewer to identify a predefined
event, and to enter a comment about the event to define points of
interest that occurred during the training session. In addition, a
rating can be associated with the event, and the event can be
associated with the trainee(s).
[0053] Assessment sessions may be performed both in real time and
subsequent to the training session. Because the recorded training
session is provided over the Internet, annotation sessions may be
performed by multiple reviewers at the same or different times. In
addition, assessments can easily be done remotely. The annotation
and assessment tool 40 stores the annotation and assessment data
entered by the reviewers in the session data archive 30 in
association with the training session and by the trainee(s). The
annotation and assessment tool 40 may also automatically tally the
annotations and assessments entered by the reviewers to create a
composite assessment/score.
[0054] In one embodiment, trainees participating in a training
session and the checklist of tasks to be completed during the
training session are established prior to the training session. For
example, prior to the training session, the trainer or the trainee
can log into the assessment tool 40 to perform a pre-assessment in
which the trainee enters information identifying the subject matter
and participants of training session (integrated trainee
identification). In addition, the trainer can login to the system
and create a checklist of tasks to be completed by the trainees
during the training session. The checklists can establish the order
which the tasks must be performed as well as established when
during the training session the tasks must be performed. Data
defining the training session (e.g., date/time, case number, room
number, and so on), checklist data, trainee data, and assessment
data are all stored in the session data archive. Data in the
session data archive 30 is preferably indexed by the training
session, but can also be indexed by a group of trainees or by
individual trainees. Because the recorded training session is
stored in association with the group of trainees and the individual
trainees in the session data archive 30, the assessment data can be
segmented out by individual trainees and viewed to see that
trainees performance. This allows all training sessions and
assessment entered for particular trainee, i.e., the trainee's
entire training portfolio, to be easily retrieved and viewed with
few mouse clicks.
[0055] Referring again to FIG. 5, when the training session is
completed (block 508), the simulation capture tool 36 detects the
deactivation of the simulated data source, and turns-off the A/V
sources 18 and encoders/recorders 20 to end the recording (block
510). This also ends any live assessments sessions.
[0056] As described above, training centers 12 typically hold
debriefing sessions where the recorded training session is reviewed
by one of more evaluators with the trainees as a group or
individually. According to a further aspect of the exemplary
embodiment, the skills assessment tool 26 further includes a
debrief tool 38 for enabling automated debrief sessions.
[0057] The debrief tool 38 is invoked in response to the end-user,
typically the trainer/evaluator, choosing to start a debrief
session from the browser 44 (block 512). The debrief tool 38 allows
the end-user to select which recorded training session to view, and
which videos in the training session to view, and the selected
videos are synchronously played back, as described above and as
shown in FIGS. 4A-4C, at which point the end-user begins to review
the training scenario with the trainee(s) (block 514). A debriefing
session is similar to an annotation session in that all the video
feeds and simulation data may be displayed on one screen in the
browser 44, but the debriefing session does not include a checklist
or other area for the viewers to enter annotations. The end-user
may choose to display only certain components from the recorded
training session and how the simulator data should be visualized.
The end-user can jump to segments of interest (block 516), choose
to review faculty/peer assessments (block 518), or perform the
review based on just the simulator trend data (520). For example,
the trainer may only choose to display video streams 1 and 2 as
well as the simulation data, and at the same time view the
assessment data entered by Dr. Johnson, for instance.
[0058] FIG. 7 is a block diagram illustrating a process performed
by the debrief tool 38 and its interaction with the end-user for
conducting an individual debriefing session between a trainer and a
trainee. The process begins by the debrief tool 38 receiving the
trainer's selection of a trainee's portfolio (block 700). In
response, the debrief tool 38 retrieves all the annotations related
to the trainee and combines them in real time into absolute and
comparative reports from which the trainer may select to view
(block 702). From the displayed reports, the trainer can identify
problem areas (block 704), and then select from the interface of
the debrief tool 38 a particular recorded training session to
review with the trainee (block 706. Once the recorded training
session is played back (block 714), and the trainer is allowed to
jump to any particular video clip. In addition, the trainer can
show a timeline of events (block 708), show a list of related
annotations and jump to a particular annotation point (block 710),
and show a list of assessments related to the trainee (712),
including any combination of faculty assessments, self-assessment,
and peer assessments. The trainer then discusses the data with the
trainee (block 716).
[0059] The combination of the annotation and assessment tool 40 and
debrief tool 38 enables a discussion and review of a trainee's
performance to be based on absolute and comparative metrics in
combination with multiple evaluator assessments, all linked to
video, thereby providing more objective feedback to the trainee and
an overall improved training process.
[0060] A method and system for providing synchronous multimedia
recording and playback has been disclosed. The present invention
has been described in accordance with the embodiments shown, and
one of ordinary skill in the art will readily recognize that there
could be variations to the embodiments, and any variations would be
within the spirit and scope of the present invention. In addition,
the embodiments can be implemented using hardware, software, a
computer readable medium containing program instructions, or a
combination thereof. In addition, although the debrief tool 38, and
annotation and assessment tool 40 are shown as separate components,
the functionality of each may be combined into a lesser or greater
number of components.
[0061] Software written according to the present invention is to be
either stored in some form of computer-readable medium such as
memory or CD-ROM, or is to be transmitted over a network, and is to
be executed by a processor. Consequently, a computer-readable
medium is intended to include a computer readable signal, which may
be, for example, transmitted over a network. Accordingly, many
modifications may be made by one of ordinary skill in the art
without departing from the spirit and scope of the appended
claims.
* * * * *