U.S. patent application number 13/763432 was filed with the patent office on 2014-02-20 for techniques for facilitating annotations.
This patent application is currently assigned to Ricoh Company, Ltd.. The applicant listed for this patent is Ricoh Company, Ltd.. Invention is credited to John Barrus, Hidenobu Kishi, Bradley Rhodes, Stephen Savitzky.
Application Number | 20140053052 13/763432 |
Document ID | / |
Family ID | 42738561 |
Filed Date | 2014-02-20 |
United States Patent
Application |
20140053052 |
Kind Code |
A1 |
Savitzky; Stephen ; et
al. |
February 20, 2014 |
TECHNIQUES FOR FACILITATING ANNOTATIONS
Abstract
Techniques for facilitating annotation of information.
Techniques are provided that enable one or more users to annotate
video information being displayed by an output device. Different
annotation modes may be provided. For example, in one mode, a
device is provided that can capture a video frame from video
information being output by an output device. One or more users may
then annotate the captured video frame. The annotations may be
communicated to the output device such that the output displays the
captured video frame along with annotations made to the video
frame. Other annotation modes are also provided.
Inventors: |
Savitzky; Stephen; (San
Jose, CA) ; Rhodes; Bradley; (Alameda, CA) ;
Barrus; John; (Menlo Park, CA) ; Kishi; Hidenobu;
(Cupertino, CA) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Ricoh Company, Ltd.; |
|
|
US |
|
|
Assignee: |
Ricoh Company, Ltd.
Tokyo
JP
|
Family ID: |
42738561 |
Appl. No.: |
13/763432 |
Filed: |
February 8, 2013 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
12408583 |
Mar 20, 2009 |
8380866 |
|
|
13763432 |
|
|
|
|
Current U.S.
Class: |
715/230 |
Current CPC
Class: |
H04N 21/00 20130101;
G06F 40/169 20200101; H04N 21/47205 20130101; H04N 21/8126
20130101; H04N 21/8153 20130101; G06F 15/16 20130101 |
Class at
Publication: |
715/230 |
International
Class: |
G06F 17/24 20060101
G06F017/24 |
Claims
1. A method of facilitating annotations, the method comprising:
capturing, at a first device, a video frame from a video stream
received by the first device from a second device, wherein the
capturing is performed in response to a request received by the
first device from a client device in a set of one or more client
devices, the captured video frame corresponding to a video frame
being displayed by an output device; and enabling the captured
video frame to be annotated using a client device from the set of
client devices.
Description
CROSS REFERENCES TO RELATED APPLICATIONS
[0001] This application is a continuation of U.S. patent
application Ser. No. 12/408,583 filed on Mar. 20, 2009, the content
of which is incorporated by reference herein in its entirety for
all purposes.
BACKGROUND OF THE INVENTION
[0002] Embodiments of the present invention relate to processing of
multimedia information, and more particularly to techniques for
enabling one or more users to annotate information, such as video
information.
[0003] Multimedia presentations are commonly used to convey
information during presentations and meetings. For example,
PowerPoint slide presentations are commonly used in academic,
commercial, and private environments to disseminate information. In
a typical presentation setup, a computer such as a laptop executing
the presentation is connected to an output device such as a monitor
or projector. Video information (and possibly other types of
information) generated from executing the presentation application
is then piped to the output device to be displayed by the output
device.
[0004] Even with the proliferation of multimedia presentations, the
ability of an attendee at the meeting to interact with the
presentation is still very limited. For example, there are no
simple solutions that enable attendees of the meeting to annotate
or provide feedback on a slide being presented at the meeting.
BRIEF SUMMARY OF THE INVENTION
[0005] Embodiments of the present invention provide techniques for
facilitating annotation of information. Techniques are provided
that enable one or more users to annotate video information being
displayed by an output device. Different annotation modes may be
provided. For example, in one mode, a device is provided that can
capture a video frame from video information being output by an
output device. One or more users may then annotate the captured
video frame. The annotations may be communicated to the output
device such that the output displays the captured video frame along
with annotations made to the video frame. Other annotation modes
are also provided.
[0006] According to an embodiment of the present invention,
techniques are provided for facilitating annotations. An
annotations system is configured to capture a video frame from a
video stream received by the annotations system from a video source
device, wherein the capturing is performed in response to a request
received by the annotations system from a client device in a set of
one or more client devices, the captured video frame corresponding
to a video frame being displayed by an output device. The
annotations system enables the captured video frame to be annotated
using a client device from the set of client devices.
[0007] In one embodiment, annotations system receives a set of one
or more annotations from one or more client devices in the set of
client devices, the set of one or more annotations comprising
annotations for the captured video frame made using one or more
client devices from the set of client devices. The annotations
system may communicate the set of annotations and the captured
video frame from the first device to multiple client devices in the
set of client devices thereby enabling the multiple client devices
receiving the captured video frame and the set of annotations to
display the captured video frame with the set of annotations
overlayed on the captured video frame.
[0008] In one embodiment, the annotations system is configured to
generate an overlayed image comprising the captured video frame
with the set of annotations overlayed on the captured video frame,
and communicate the overlayed image to the multiple client devices
in the set of client devices.
[0009] In one embodiment, the annotations system is configured to
communicate the set of annotations to the output device thereby
causing the output device to display the captured video frame with
the set of annotations overlayed on the captured video frame.
[0010] In one embodiment, the annotations system is configured to
capture one or more video frames from the video stream received
from the video source device, the one or more video frames
including the video frame captured in response to the request
received from the client device. The annotations system may then
generate a document comprising the one or more video frames and the
first set of annotations.
[0011] In one embodiment, wherein the set of annotations received
by the annotations system may comprise annotations received from
multiple client devices from the set of client devices. The
annotations system may communicate the set of annotations to the
output device thereby causing the output device to display the
captured video frame with the set of annotations overlayed on the
captured video frame.
[0012] In one embodiment, the annotations system may receive the
request from a first client device from the set of client devices
and, in response to the request, communicate the captured video
frame to only the first client device. The annotations system may
receive a set of one or more annotations from the first client
device, the set of annotations representing annotations made to the
captured video frame using the first client device. The annotations
system may store the set of one or more annotations.
[0013] According to one embodiment, annotations system may provide
a server that provides an interface that enables a video frame
captured by the annotations system to be annotated by a client
device from the set of client devices. The annotations system may
also enable an application to be downloaded from the annotations
system to a client device from the set of client devices, wherein
the application enables the client device receiving the application
to annotate the captured video frame.
[0014] The foregoing, together with other features and embodiments
will become more apparent when referring to the following
specification, claims, and accompanying drawings.
BRIEF DESCRIPTION OF THE DRAWINGS
[0015] FIG. 1 is a simplified block diagram of a system
incorporating an embodiment of the present invention;
[0016] FIG. 2 is a simplified flowchart depicting a method of
enabling annotations according to an embodiment of the present
invention;
[0017] FIG. 3 is a simplified flowchart depicting a method of
enabling and processing annotations in shared mode according to an
embodiment of the present invention;
[0018] FIG. 4 is a simplified flowchart depicting a method of
enabling and processing annotations in private mode according to an
embodiment of the present invention; and
[0019] FIG. 5 depicts a simplified block diagram of an annotations
system according to an embodiment of the present invention.
DETAILED DESCRIPTION OF THE INVENTION
[0020] In the following description, for the purposes of
explanation, specific details are set forth in order to provide a
thorough understanding of embodiments of the invention. However, it
will be apparent that the invention may be practiced without these
specific details.
[0021] FIG. 1 is a simplified block diagram of a system 100
incorporating an embodiment of the present invention. As shown in
FIG. 1, system 100 comprises an annotations system or device (AS)
102, a video source device 104, an output device 106, and one or
more client devices 108. Video source device 104 may be any device
that is capable of providing a stream of video information. For
example, video source device 104 may be a computer that provides a
stream of video information. The video stream may be generated as a
result of an application executing on video source device 104. For
example, video source device 104 may execute an application such as
Microsoft's PowerPoint that is capable of outputting presentations
of slides.
[0022] The video stream generated as a result of the PowerPoint
application may be output by video source device 104 and
communicated to AS 102. Other examples of applications that may
result in generation of video information include KeyNote,
QuickTime, media players of different types, a Flash application, a
web browser, other presentation software, or other application that
causes information to be output and the output information may be
displayed using an output device.
[0023] Output device 106 may be any device capable of outputting
information. In one embodiment, the information that is output by
output device 106 is received from AS 102. For example, AS 102 may
communicate video information to output device 106 that may then be
output by output device 106. In one embodiment, AS 102 receives a
video stream from video source device 104 and communicates the
received video stream to output device 106 for display by output
device 106. The information that is output by output device 106 may
be video information, audio information, images, etc. Examples of
output device 106 include a projector, a monitor, a television, and
the like.
[0024] AS 102 is configured to facilitate annotations to
information output by output device 106. In one embodiment, AS 102
provides various modes in which annotations may be made. In one
mode, users of client devices 108 may collaboratively annotate
information output by video source device 104 and that is output by
output device 106. As depicted in FIG. 1, AS 102 is communicatively
coupled to both video source device 104 and output device 106. In
one embodiment, as depicted in FIG. 1, AS 102 is configured to
receive video information from video source device 104 and
communicate the received video information to output device 106
such that video information provided by video source device 104 can
be output by output device 106.
[0025] There are various ways in which AS 102 may be
communicatively coupled with video source device 104 and output
device 106 including ways that use wired or wireless technologies,
or combinations thereof. The type of communication link between AS
102 and video source device 104 may be the same as or different
from the type of communication link between AS 102 and output
device 106. Various different protocols may be used to facilitate
the communications between video source device 104 and AS 102 and
between AS 102 and output device 106.
[0026] In one embodiment, AS 102 is configured to receive requests
or commands from client devices 108 and to perform one or more
operations in response to the requests. These requests may include
for example requests to enter modes that enable users of client
devices 108 to annotate information provided by video source device
104 and that is output by output device 106. The requests for
performing annotations may be received from one or more client
devices 108. For example, video source device 104 may execute a
PowerPoint application displaying a slide presentation. The video
information resulting from the PowerPoint presentation may be
output by output device 106. In such a scenario, AS 102 enables
users of client devices 108 to annotate the displayed slides.
Further details related to how the annotations are enabled are
provided below. It should be apparent that embodiments of the
present invention are not restricted to PowerPoint presentations.
In general, AS 102 enables users of client devices 108 to annotate
information that is received by AS 102 from video source device 104
and information that is output by output device 106.
[0027] In addition to enabling annotations, AS 102 may also be
configured to perform other operations. These operations may be
performed in response to requests received from one or more client
devices 108. For example, in one embodiment, AS 102 may be
configured to receive video information from video source device
104, format the video information to an appropriate format that can
be handled by output device 106, and then send the formatted video
information to output device 106. In this manner, AS 102 enables
information from video source device 104 to be output by output
device 106 even if the format of information output by video source
device 104 is not compatible with the format of information that
output device 106 can output. A user may also use a client device
108 to request AS 102 to capture one or more video frames from the
stream of video information received by AS 102 from video source
device 104. A user may also use a client device 108 to request AS
102 to create electronic documents based upon the received video
information from video source device 104. In one embodiment, AS 102
may create an electronic document comprising video frames captured
by AS 102. If a video frame included in the electronic document has
been annotated, then the annotations may also be included in the
electronic document. In one embodiment, the annotations may be
overlayed on the video frame in the electronic document. In this
manner, the electronic document that is created by AS 102 shows not
just the captured video frames but also any annotations made to the
video frames. A user of a client device 108 may also request AS 102
to perform other operations.
[0028] In one embodiment, in order to user the services provided by
AS 102, a client device 108 has to register with AS 102. In such an
embodiment, AS 102 provides a mechanism by which a client device
108 may register and un-register with AS 102. One or more client
devices 108 may be registered with AS 102 at any time. Only the
registered client devices are then permitted to request operations
using AS 102, including the ability to annotate video information.
For example, an AS 102 may be located in a conference room. During
a meeting held in the conference room, attendees of the meeting may
register their client devices with the AS located in the conference
room. This enables users of the registered client devices to
request operations to be performed by the AS located in the
conference room, including requests to perform annotations for a
slide presentation during the meeting. At the end of the meeting,
the users may de-register their client devices. In one embodiment,
for enhanced security, AS 102 may automatically de-register a
previously registered client device if the client device has not
communicated with the AS for a period of time, or if the distance
between the client device and AS 102 exceeds a certain
user-configurable threshold, or using some other technique. For
example, the AS 102 located in a conference room may be configured
such that only those client devices that are within the conference
room can register with the AS.
[0029] Client devices 108 are devices or systems that can be used
to send commands or requests to AS 102 requesting one or more
operations to be performed or enabled by AS 102. For example, a
user may use a client device 108 to annotate video information
provided by video source device 104 and output by output device
106. A client device 108 may comprise an input subsystem (e.g., a
stylus, a mouse, a keyboard, etc.) that enables a user to input
information and make operation requests. For example, a user may
use a stylus of a client device 108 to send a request to enable
annotation of the video information and also to make the
annotations. Other operations may also be initiated using the
stylus and the client device. A client device 108 may also comprise
an output subsystem (e.g., display, monitor) that enables
information to be output to the user of the client device.
[0030] Various different types of devices may be used as client
devices 108. These devices include but are not restricted to
various types of computers, laptops, PDAs, tablet computers, mobile
devices, E-Ink technology devices that use microspheres having at
least two reflectance states to display information (e.g., some
types of tablet computers, document readers such as the Kindle
device provided by Amazon), remote control devices, etc. In
general, any device that can be configured to communicate with AS
102 and request the AS to perform operations may be used as a
client device 108. In one embodiment, a tablet computer that uses
E-Ink to display information may be used as a client device 108.
Such a tablet device may use E-Ink to display information and
provide the user the ability to annotate the displayed information
using pen/paper like interaction using an input device such as a
digital pen or stylus.
[0031] In one embodiment, AS 102 may also provide an interface,
such as a web interface, that can be accessed by client devices 108
to perform various operations. For example, client devices 108 may
use this web interface to make annotations.
[0032] AS 102 is configured to provide different modes in which
video information received from video source device 104 and output
by output device 106 can be annotated. In one embodiment, a "shared
mode" and a "private mode" are provided. In the shared mode,
multiple users are able to collaboratively annotate the video
information and annotations made by one user are shared with other
users. A user using a client device may send a request to AS 102
for annotations to be performed in a shared mode. In one
embodiment, upon receiving such a request, AS 102 is configured to
capture the video frame or image presently being output or
displayed by output device 106. AS 102 then continues to send video
information comprising the video frame to output device 106 thereby
freezing the output of output device 106 to display the captured
video frame. In some embodiments, the video information received by
AS 102 from video source device 104 may be stopped as part of the
freezing process. AS 102 may also send the captured video frame to
the registered client devices for display by the client devices.
One or more users of the client devices may then annotate the
captured video frame using their client devices. For example, a
user using a tablet computer may use the tablet's stylus to
annotate the captured video frame displayed on the tablet's
display. The annotations made by a client device are communicated
from the client device to AS 102. Since multiple users may make
annotations in the shared mode, AS 102 may receive annotations from
multiple client devices 108. AS 102 is configured to communicate
the received annotations to the client devices such that the
annotations are displayed on the captured video frame displayed by
the client devices. AS 102 may also communicate the received
annotations to output device 106 such that the information output
by output device 106 displays the captured video frame with the
annotations overlayed on it. In one embodiment, AS 102 generates an
overlayed image by overlaying the annotations received from client
devices 108 on the captured video frame. The overlayed image thus
created is then communicated from AS 102 to output device 106 or to
client devices 108 for display. In this manner, in the shared mode,
multiple users may annotate the video information and the annotated
information is shared among the multiple users. The shared mode
thus provides a mode in which collaborative annotations may be
performed by users of client devices 108. Normal display of the
video information by output device 106 (i.e., unfreezing of the
video information) may be resumed after the shared mode has been
exited.
[0033] In the private mode, annotations made by a user of a client
device are not shared with other users. In this mode, annotations
made by a user are kept private for that user. A user may use a
client device to send a request to AS 102 requesting enabling
annotations to be performed in a private mode. In one embodiment,
upon receiving such a request, AS 102 is configured to capture the
video frame or image presently being displayed by output device
106. However, unlike the shared mode, the video information being
displayed to the users is not frozen. The captured video frame may
then be sent to the client device requesting the private mode for
display by the client device. The user of the requesting client
device may then provide annotations to the captured video frame
using the client device. For example, a user using a tablet
computer may use the tablet's stylus to annotate the captured video
frame displayed on the tablet's display. The annotations are
displayed on the user's client device overlayed on the captured
image. The annotations may also be communicated from the client
device to AS 102. AS 102 may then store the annotations. In this
manner, the annotations are only displayed to the requesting user
and not shared with the other users. AS 102 can participate in
multiple private mode sessions with individual client devices at
any time.
[0034] There are various situations in which a user may use a
private mode for annotations. For example, the user may want to
annotate the information for the user's personal notes. Since AS
102 receives the user's annotations, at the end of a presentation,
AS 102 may compile an electronic document for the user comprising
video frames captured from the video information along with
annotations provided by the user.
[0035] In one embodiment, AS 102 also provides the capability to
switch between annotation modes, for example, from the private mode
to the shared mode. For example, a user in private mode may wish to
share the user's annotations with the rest of the users. While in
private mode, the user may send a request (using the user's client
device) to AS 102 to switch the mode from private mode to shared
mode. Upon receiving such a request, AS 102 is configured to send
information to output device 106 such that the video frame captured
and possibly annotated by the user in private mode is output by
output device 106. In this manner, the output of output device 106
is frozen to display the captured video frame. The captured frame
may also be sent by AS 102 to the registered client devices for
display by the client devices and enabling one or more users of the
client devices to annotate the captured video frame. Annotations
made by the user in private mode may also be communicated to the
output device 106 and/or to the client devices 108 such that the
output device and the client devices can display the captured video
frame along with the annotations to the captured video frame made
by the user in private mode.
[0036] It should be noted that, in embodiments of the present
invention, the video source device 104 and/or output device 106 do
not have to be specially configured or modified in any way to
enable the annotation feature. There is no special hardware or
software needed on video source device 104 or output device 106 to
support the annotation feature. The software and hardware for
supporting annotations is fully provided by AS 102. In this manner,
AS 102 provides a convenient and pluggable solution to annotate
displayed information, without requiring any specialized hardware
or software on video source device 104 or output device 106 or even
on client devices 108. This not only enhances the usability of AS
102 but also enables it to be used in various environments. AS 102
can be easily used in any setting or environment where information
is being displayed to one or more users and users desire to
annotate the displayed information. Examples include during reviews
of documents (e.g., during code reviews), for teaching in a
classroom-type of setting, for website walk-throughs, or other
settings that involve users making annotations or providing
feedback on information that is displayed.
[0037] In the embodiment depicted in FIG. 1, AS 102 is shown as
being separate from video source device 104 and output device 106.
In an alternative embodiment, AS 102 may be integrated into output
device 106.
[0038] FIG. 2 is a simplified flowchart 200 depicting a method of
enabling annotations according to an embodiment of the present
invention. The method depicted in FIG. 2 may be implemented in
software (e.g., program, code, instructions executed by a processor
of AS 102), or hardware of AS 102, or combinations thereof. The
software may be stored on a computer-readable storage medium. The
method may be performed in a system such as system 100 depicted in
FIG. 1.
[0039] As depicted in FIG. 2, the method is initiated when AS 102
receives a request to enable annotations (step 202). The request
may be received from one or more client devices 108. In one
embodiment, only those client devices that are registered with AS
102 are able to send the request received in 202. In one
embodiment, the request received in 202 may also indicate a
particular mode (e.g., shared mode, private mode, etc.) in which
the annotations are to be performed. In other embodiments, a
default mode (typically the shared mode) may be provided.
[0040] In response to the request received in 202, AS 102 is
configured to capture a video frame that is presently being
displayed by an output device (step 204). AS 102 is then configured
to enable the captured video frame to be annotated (step 206). The
annotations may be enabled in a particular mode such as a private
mode or shared mode. As part of 206, the video frame captured in
204 is made available for annotations by one or more client devices
108.
[0041] FIG. 3 is a simplified flowchart 300 depicting a method of
enabling and processing annotations in shared mode according to an
embodiment of the present invention. The method depicted in FIG. 3
may be implemented in software (e.g., program, code, instructions
executed by a processor of AS 102), or hardware of AS 102, or
combinations thereof. The software may be stored on a
computer-readable storage medium. The method may be performed in a
system such as system 100 depicted in FIG. 1.
[0042] As depicted in FIG. 3, the method is initiated when AS 102
receives a request to enable annotations in shared mode (step 302).
The request may be received from one or more client devices 108. In
one embodiment, only those client devices that are registered with
AS 102 are permitted to send the request.
[0043] In response to the request, AS 102 is configured to capture
a video frame that is presently being output by an output device
(step 304). The captured video frame may also be stored by AS 102
(step 306). As part of 306, AS 102 may also store metadata
information related to the captured video frame such as the date
and time of the capture, information identifying a client device
that triggered the capture, and the like.
[0044] AS 102 may then send the video frame captured in 304 to one
or more client devices that are registered with AS 102 (step 308).
The communication may be performed using wireless technologies,
wired technologies, or combinations thereof. The video frame may be
communicated in a format that the individual client device is able
to handle. For example, the display resolutions on the client
devices may be different. For a particular client device, AS 102 is
configured to format the captured video frame to a format that the
client device can handle and then communicate the formatted
captured video frame to the client device. This allows client
devices with different capabilities to use the services provided by
AS 102 and to collaboratively perform annotations. Because the
captured video frames and annotations are stored on AS 102 and
passed to the client devices, there is no need for the client
devices to have the same screen size, color depth, etc. This allows
color images to be annotated using a client device with a
monochrome display. AS 102 may also automatically scale the
annotations so that, for example, a printable PDF document may be
annotated on a client device with a smaller screen.
[0045] AS 102 may then cause the output device to display the
captured video frame (step 310). In one embodiment, this is done by
continually sending video information comprising the captured frame
information to the output device causing the output device to
display the captured frame. Accordingly, instead of feeding output
device 106 with information received by AS 102 from video source
device 104, AS 102 keeps sending information comprising the
captured video frame. The video information received by AS 102 from
the video source device 104 may also be temporarily paused.
[0046] A client device receiving the captured video frame from AS
102 may display the captured video frame on the client device's
display and enable the user of the client device to annotate the
displayed captured video frame. One or more users of the client
devices may then make annotations to the captured video frame. The
annotations may be made at different times, even in parallel by
multiple users. AS 102 may then receive one or more annotations
made to the captured video frame by users of one or more client
devices (step 312). The annotations may be received by AS 102 in
different ways. For example, the annotations may be received in the
form of strokes (connected sequences of line segments), text, an
overlayed image comprising the captured video frame with overlayed
annotations, and the like.
[0047] AS 102 may optionally store the received annotations (step
314). In one embodiment, the annotations information may also
include metadata information such as timestamps (e.g., time of
annotation), information identifying a client device or a user
making the annotations, etc. This metadata information may be used
when displaying the annotations. For example, an icon indicative of
a user may be displayed next to the annotations made by the user.
In one embodiment, the annotations may be stored separately from
the captured image. In other embodiments, the annotations and the
captured image may be stored together.
[0048] AS 102 is configured to send the set of annotations received
in 312 to one or more client devices (step 316). In one embodiment,
the set of annotations received in 312 may be sent to all the
registered devices. In another embodiment, the set of annotations
may be sent to a subset of the registered devices. For example, if
the annotations are received in 312 from a first client device, the
annotations may be sent in 316 to other client devices other than
the first client device. However, the first client device may
receive annotations made using other client devices. In this
manner, annotations made using one client device are made available
to and shared with the other registered client devices. A client
device receiving the annotations may display the captured video
frame with the annotations overlayed on the frame. In this manner,
a client device is able to display the captured video frame along
with all the annotations made to the captured video frame by one or
more users.
[0049] In one embodiment, the set of annotations received by AS 102
may also optionally be sent to output device 106 (step 318). This
enables the output device to display the captured video frame along
with the annotations made by the various users. For example, if
output device 106 is a projector, AS 102 may send the annotations
received from one or more client devices to the projector so that
the image displayed by the projector is that of the captured video
frame along with the annotations overlayed on the captured video
frame. In this manner, the annotations made by one or more users
are displayed and shared with all the users.
[0050] There are different ways in which AS 102 may send the
captured video frame and the associated annotations to the client
devices or the output device 106. In one embodiment, AS 102 is
configured to generate an overlayed video frame comprising the
video frame captured in 304 overlayed with the annotations received
in 312. The overlayed video frame thus generated is then
communicated to the client devices and the output device. The
communications from AS 102 to output device 106 or to the client
devices 108 may be performed using wireless technologies, wired
technologies, or combinations thereof
[0051] The annotations received from the different client devices
in 312 may be of different types, shapes, and forms. In one
embodiment, when AS 102 sends the annotations to the client devices
or to the output device, the type, shape, and form of the
annotations are preserved. For example, if the annotations are
received in different colors, then those colors are preserved when
the annotations are communicated by AS 102 to the different client
devices or to the output device. In this manner, annotations in
multiple colors may be displayed by the output device or the client
devices.
[0052] In one embodiment, AS 102 may be configured to automatically
assign different colors to annotations received from different
client devices. This makes it easy for the users of the client
devices to distinguish annotations made by different users when
displayed by the client devices or by the output device.
[0053] Steps 312, 314, 316, and 318 may be repeated as additional
annotations are received from the client devices and are then
distributed to the client devices and to the output device. In this
manner, in the shared mode, AS 102 enables collaborative annotating
of the captured video frames among users of the client devices. The
shared mode may continue until a signal is received to exit the
shared mode (step 320). The request or command to exit the shared
mode may be received from one or more of the client devices. Upon
receiving a signal to exit the shared mode, AS 102 is configured to
exit the shared mode and unfreeze the output of output device 106
(step 322). AS 102 may unfreeze the output device by resuming
sending the output device video information received by AS 102 from
video source device 104. In this manner, communication of
information received by AS 102 from video source device 104 to
output device 106 is resumed for output by output device 106.
[0054] FIG. 4 is a simplified flowchart 400 depicting a method of
enabling and processing annotations in private mode according to an
embodiment of the present invention. The method depicted in FIG. 4
may be implemented in software (e.g., program, code, instructions
executed by a processor of AS 102), or hardware of AS 102, or
combinations thereof. The software may be stored on a
computer-readable storage medium. The method may be performed in a
system such as system 100 depicted in FIG. 1.
[0055] As depicted in FIG. 4, the method is initiated when AS 102
receives a request from a client device to enable annotations to be
performed in private mode (step 402). In response to the request,
AS 102 is configured to capture a video frame that is presently
being output by an output device (step 404). AS 102 may store the
captured image (step 406). As part of 406, AS 102 may also store
metadata information related to the captured video frame such as
the date and time of the capture, information identifying the
client device that triggered the capture, and the like.
[0056] AS 102 may then send the video frame captured in 404 to the
client device requesting the private mode in 402 (step 408). The
client device receiving the captured video frame may then display
the captured video frame on the client device's display to enable
the user of the client device to annotate the displayed captured
video frame. It should be noted that in the private mode, the
captured video frame is only sent to the client device requesting
the private mode and not to other client devices as in the shared
mode. Further, in private mode, the output of the output device is
not frozen to display the captured video frame. AS 102 continues to
pipe video information received from video source device 104 to
output device 106 for display by the output device.
[0057] The user of the client device receiving the captured video
frame in 408 may then make one or more annotations to the captured
video frame. AS 102 may receive the one or more annotations made to
the captured video from the client device (step 410). The
annotations may be received by AS 102 in different ways such as in
the form of strokes (connected sequences of line segments), text,
an overlayed image comprising the captured video frame with
overlayed annotations, and the like. AS 102 may store the received
annotations (step 412). Steps 410 and 412 may be repeated as
additional annotations are received from the client device.
[0058] The private mode may continue until a signal is received
from the client device to exit the private mode (step 414). The
private mode may then be exited (step 416). AS 102 is capable of
supporting multiple private mode sessions with different client
devices in parallel.
[0059] In one embodiment, a user may also be able to switch from
one annotation mode to another. For example, a user can switch from
private mode to shared mode. The user may send such a request to AS
102 using the user's client device. When such a request is
received, AS 102 may resume operations according to the shared mode
depicted in FIG. 3 and described above. For example, AS 102 may
communicate the video frame captured in the private mode to the
other client devices and cause the output of the output device to
display the captured video frame. AS 102 may also communicate the
annotations made by the user in private mode to the other client
devices. AS 102 may then receive annotations from one or more of
the client devices. The annotations may then be distributed to all
the client devices, not just to the client device initially
requesting the private mode. The annotations may also be sent to
the output device for display by the output device.
[0060] AS 102 may also be configured to perform various other
operations in response to requests received from one or more client
devices 108. In one embodiment, AS 102 may be configured to compile
an electronic document using video frames captured from the stream
of video information received by AS 102 from video source device
104 during a presentation. AS 102 may compile such a document in
response to a request received from a client device. For example,
AS 102 may aggregate one or more video frames captured by AS 102
into an electronic document. Any annotations associated with the
video frames included in the electronic document may also be
included in the electronic document and overlayed on the
corresponding frames. The electronic document may be generated for
a session (e.g., a presentation) and may comprise video frames
captured for the session and any associated annotations. The
resultant electronic document comprises the captured video frames
with overlayed annotations. In this manner, a record is preserved
of the annotation activity during a session. Further, the ability
of AS 102 to store electronic documents of video frames rather than
just video information allows AS 102 to be used for document review
and markup sessions of the electronic documents. The electronic
documents may also be printed by sending a request to AS 102 to
generate a physical paper representation of the document.
[0061] There are different ways in which video frames may be
captured by AS 102. According to one way, as described above, video
frames are captured upon receiving annotation requests. According
to another way, in addition to video frames captured as a result of
annotation requests, AS 102 may be configured to, during a
presentation, capture video frames from the video information
received from video source device 104 at regular intervals and
store ones that are sufficiently different from the previously
captured and stored video frame. In such an embodiment, AS 102 may
be configured to generate an electronic document of the captured
video frames along with annotations, if any, made to any of the
captured video frames. For video frames that have been annotated,
the generated electronic document comprises an overlayed image of
the captured video frame and its annotations. AS 102 may compile
such a document in response to a request received from a client
device. The electronic document generated by AS 102 represents a
summary of the presentation along with any annotations made by
users during the presentation.
[0062] AS 102 may also enable operations to be performed on the
electronic documents generated by AS 102. For example, services may
be provided for loading previously stored electronic documents,
browsing or navigating through the electronic documents,
communicating (e.g., email, fax, etc.) the electronic documents,
etc.
[0063] FIG. 5 depicts a simplified block diagram of an AS 102
according to an embodiment of the present invention. The embodiment
depicted in FIG. 5 is merely illustrative and is not intended to
limit the scope of the present invention. Various changes and
modifications to what is displayed in FIG. 5 are possible. As
depicted in FIG. 5, AS 102 comprises several subsystems and
components including a processor 502, a memory 504, a web server
506, a communication subsystem 508, an input/output subsystem 510,
a video input component 512, video output component 514, and a
video buffer 516. AS 102 may work on different operating systems
including Linux, Windows and its variants, and the like. In one
embodiment, AS 102 is a Linux-based computer running the X-windows
system.
[0064] Processor 502 is configured to perform a majority of the
processing performed by AS 102. For example, processor 502 may be
configured to perform the operations depicted in flowcharts in
FIGS. 2, 3, and 4 and described above. In one embodiment, processor
502 may be configured to perform the operations by executing one or
more code modules or software instructions, which may be stored on
a computer-readable storage medium such as memory 504.
[0065] Memory 504 stores the basic programming and data constructs
that provide the functionality of the present invention. Memory
subsystem 504 may include a number of memories including a main
random access memory (RAM) for storage of instructions and data
during program execution and a read only memory (ROM) in which
fixed instructions may be stored. Memory subsystem 504 may also
comprise persistent (non-volatile) storage for program and data
files, and may include a hard disk drive, a floppy disk drive along
with associated removable media, a CD or DVD drive, an optical
drive, removable media cartridges, and the like.
[0066] As indicated above, memory 504 may store the programming and
data constructs used for providing the functionality of the present
invention. As depicted in FIG. 5, memory 504 may store video frames
518 captured by AS 102 along with information related to the
captured video frames, annotations 520 received from one or more
client devices, electronic documents 522 created by AS 102
comprising one or more of the captured video frames and the
annotations, and software or code instructions or modules 524 that
when executed by processor 502 provide the functionality of the
present invention. For an electronic document, information 522 may
include metadata information identifying when the electronic
document was created, who had requested creation of the electronic
document, the presentation to which the electronic document
corresponds, and other information. Annotations 520 may include
metadata information associated with the annotations such as
timestamp information on when the annotations were made,
information identifying a user who made the annotations,
information identifying the client device used to make the
annotations, and other information related to the annotations.
[0067] As indicated in FIG. 5, the annotations 520 may be archived
separately from the captured video frames 518. This enables AS 102
to easily create electronic documents comprising captured video
frames with or without annotations. Annotations may also be stored
on a per client device basis such that AS 102 can easily determine
annotations received from a particular client device. In one
embodiment, AS 102 may store a hash-chained log of captured video
frames and annotations. Such a log allows strong assertions to be
made about the sequence and timing of frames and assertions (see
Bruce Schneier, John Kelsey, "Cryptographic Support For Secure Logs
On Untrusted Machines," Proceedings of 7th USENIX Security
Symposium, San Antonio, Tex., Jan. 26-29, 1998, p. 4-4, the entire
contents of which are incorporated herein by reference for all
purposes). Memory 504 represents a computer-readable storage medium
for storing the software/code instructions. Other information may
also be stored in memory 504.
[0068] Communication subsystem 508 is configured to facilitate
communication of data to and from AS 102. For example,
communication subsystem 508 facilitates communications between AS
102 and client devices 108. Client devices may send
requests/commands, annotations, and other information to AS 102. AS
102 may communicate captured video frames, annotations, overlayed
images, electronic documents, and other information to the client
devices. Different communication protocols may be used to
facilitate the communications including wired and wireless
protocols. In one embodiment, between a client device and AS 102,
the same protocol may be used for the communications. In
alternative embodiments, different protocols may be used.
[0069] Input/Output devices allow a user to interact with AS 102.
Input devices may include a stylus, a digital pen, a mouse, a
keyboard, voice activated input, and the like. Output devices may
include devices that enable information to be output from AS 102
such as a display or monitor.
[0070] As described above, AS 102 receives a video stream from
video source device 104 and communicates a video stream to output
device 106. Video input component 512 enables receipt of video
information from multiple sources including from video source
device 104. In one embodiment video input component 512 is a video
capture card. The video capture card may be connected to the video
output (e.g., VGA port, DVI port, HDMI port, etc.) of a video
source device executing a presentation such as a PowerPoint
presentation. Video input component 512 may be capable of receiving
the information via a wire or wirelessly.
[0071] The video information received via video input component 512
may be buffered in video buffer 516 prior to the video information
being communicated to video output component 514. Video output
component 514 enables video information from video buffer 516 (and
which may represent information received from video source device
104) to be communicated from AS 102 to an external device such as
to output device 106. In one embodiment video output component 514
is a display controller card. The display controller card may be
connected to the video input (e.g., VGA port, DVI port, HDMI port,
etc.) of output device 106. Video output component 512 is capable
of transmitting the information via a wire or wirelessly.
[0072] In one embodiment, AS 102 may provide an interface than may
be utilized by clients to enable annotations. For example, AS 102
may comprise a web server 506 that provides a web interface that
client devices may use to perform annotations. In one embodiment,
web server 506 may host a page that is accessible via a URL. A
client device may then connect to the host page using the URL. For
example, if the client device is a tablet or a laptop computer, a
user of the laptop by providing the URL for the hosted page to a
browser on the client device enables the client device to connect
to the hosted page. The hosted page may provide the client device
an interface for annotating the video frames. For example, the
hosted page may provide a simple web form to add text annotations.
In another embodiment, the hosted page may be a full Flash,
JavaScript or JavaApplet based interface that updates the
annotations on the display in real-time and offers a rich
annotation-authoring environment. For example, upon connecting to
the hosted page, the Flash, Javascript, or JavaApplet application
may be downloaded to the client device connected to the hosted
page. The downloaded application may then be executed on the client
device thereby enabling the client device to perform annotations.
Web server 506 may provide various applications that facilitate
annotations. In this manner, a client device does not need to have
preconfigured annotation capabilities--AS 102 enables a client
device to perform annotations through the use of web server 506.
Web server 506 may also enable a user of a client device to connect
to the Internet using AS 102.
[0073] AS 102 can be of various types. Due to the ever changing
nature of computers, the description of AS 102 depicted in FIG. 5
is intended only as a specific example for purposes of illustrating
an embodiment of the present invention. It should be apparent that
various other configurations of AS 102 are possible having more or
fewer subsystems or components than the system depicted in FIG.
5.
[0074] While embodiments of the present invention have been
described using a particular combination of hardware and software,
it should be recognized that other combinations of hardware and
software are also within the scope of the present invention.
Embodiments of the present invention may be implemented only in
hardware, or only in software, or using combinations thereof
[0075] The specification and drawings are, accordingly, to be
regarded in an illustrative rather than a restrictive sense. It
will, however, be evident that additions, subtractions, deletions,
and other modifications and changes may be made thereunto without
departing from the broader spirit and scope as set forth in the
claims.
* * * * *