U.S. patent application number 10/719641 was filed with the patent office on 2004-11-11 for cross-platform framework-independent synchronization abstraction layer.
Invention is credited to Mills, Michael, O'Brien, Chris, Wallace, Bruce A., Wason, Andrew.
Application Number | 20040225994 10/719641 |
Document ID | / |
Family ID | 23323778 |
Filed Date | 2004-11-11 |
United States Patent
Application |
20040225994 |
Kind Code |
A1 |
Wason, Andrew ; et
al. |
November 11, 2004 |
Cross-platform framework-independent synchronization abstraction
layer
Abstract
This invention is a synchronization abstraction layer (SAL)
providing a uniform interface to frameworks operating on sequenced
flow data. It allows content developers to design and build
interactive content that will operate interchangeably in different
multimedia frameworks (e.g., Apple Computer, Inc.'s QuickTime.TM.,
Microsoft Corporation's NetShow.TM., RealNetworks, Inc.'s
RealPlayer.TM., Sun Microsystems, Inc.'s Java.TM. Media Framework)
and on different hardware platforms (e.g., desktop PC,
Macintosh.TM., Television set-top boxes such as those from General
Instrument Corporation and Scientific Atlanta Inc., Inc., Internet
appliances such as AOL.TM.-TV, and other appliances, e.g., a
kitchen Internet radio). The uniform interface is independent of
the particular framework and the platform on which the SAL is
implemented, so that a single instance of content, whether created
in Java.TM., JavaScript, VBscript, HTML, XML, or some other
language, can run appropriately on different hardware, e.g., on a
Television set-top and on a desktop PC. In one realization, the
synchronization abstraction layer provides a Java.TM. VIRTUAL
MACHINE (JVM) interface for running Java.TM. plug-ins for streaming
media applications such as Real Networks, Inc.'s RealPlayer.TM.,
Microsoft Corporation's Windows Media Technologies (NetShow.TM.),
Apple Computer, Inc.'s QuickTime.TM., Sun Microsystems, Inc.'s
Java.TM. Media Framework. The JVM interface allows third-party
developers to design platform- and framework-independent plug-ins
for streaming media applications. This invention allows content
providers to use plug-ins or compatible software objects (such as
Java.TM. applets) to build, for example, interactive streaming
media content that is fully interactive but independent of the
particular underlying hardware and software technologies, such as
RealNetworks.TM. G2, Microsoft Corporation's NetShow.TM., a desktop
PC, or a television.
Inventors: |
Wason, Andrew; (Atlantic
Highland, NJ) ; Mills, Michael; (Holmdel, NJ)
; O'Brien, Chris; (Brooklyn, NY) ; Wallace, Bruce
A.; (Short Hills, NJ) |
Correspondence
Address: |
John V. Biernacki
Jones Dayu
North Point
901 Lakeside Avenue
Cleveland
OH
44114
US
|
Family ID: |
23323778 |
Appl. No.: |
10/719641 |
Filed: |
November 21, 2003 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
10719641 |
Nov 21, 2003 |
|
|
|
09338188 |
Jun 22, 1999 |
|
|
|
6701383 |
|
|
|
|
Current U.S.
Class: |
717/100 ;
375/E7.025; 375/E7.267 |
Current CPC
Class: |
H04N 21/2381 20130101;
H04N 21/4431 20130101; H04N 21/4437 20130101; H04N 21/4381
20130101; H04N 21/64322 20130101; H04N 21/6125 20130101; H04N
21/8193 20130101; H04N 7/52 20130101; G06Q 10/10 20130101; H04N
21/43072 20200801 |
Class at
Publication: |
717/100 |
International
Class: |
G06F 009/44 |
Claims
1-27. cancel.
28. An extension layer for extending the functionality of a
framework which is separate from an operating system, comprising: a
first access system coupled to a first framework which interacts
with the first framework; a second access system coupled to a
second framework which is not identical to the first framework and
which interacts with the second framework; and a coupling system
for connecting to an extension module for an extensible framework
providing access to content or functions of the extension module;
wherein the extension layer provides a uniform, cross-platform,
framework-independent interface between the first and second
frameworks and the extension module.
29. An extension layer for extending the functionality of a
framework which is separate from an operating system, comprising: a
first access system coupled to a first framework which interacts
with the first framework; a second access system coupled to a
second framework which is not identical to the first framework and
which interacts with the second framework; a coupling system for
connecting to an extension module for an extensible framework
providing access to content or functions of the extension module;
and a synchronization system for synchronizing data from the
extension module; wherein the extension layer provides a uniform,
cross-platform, framework-independent interface between the first
and second frameworks and the extension module.
30. An extension layer for extending the functionality of a
framework which is separate from an operating system, comprising: a
first access system coupled to a first framework which interacts
with the first framework; a second access system coupled to a
second framework which is not identical to the first framework and
which interacts with the second framework; a coupling system for
connecting to an extension module for an extensible framework
providing access to content or functions of the extension module; a
synchronization system for synchronizing data from the extension
module; and a communication system allowing for the transfer of
data or instructions between the extension module and an extensible
framework, wherein the extension layer provides a uniform,
cross-platform, framework-independent interface between the first
and second frameworks and the extension module.
31. An extension layer for accessing and using an ordered file,
comprising: a first extensible ordered-data flow-based framework; a
second extensible ordered-data flow-based framework which is not
identical to the first extensible ordered-data flow-based
framework; an extension module for an extensible ordered-data
flow-based framework containing ordered data; an abstraction layer
having a uniform cross-platform interface between the extension
module and an extensible ordered-data flow-based framework and
being separate from an operating system; and a synchronization
module for handling data flowing between the extension module and
the extension layer; wherein the extension layer provides a
uniform, cross-platform, framework-independent interface between
the first and second frameworks and the extension module.
Description
APPENDIX AND COPYRIGHT NOTICE
[0001] This document includes a partial source code listing of a
preferred embodiment of the applicant's invention. The code, listed
in Appendix A and in the drawings, forms part of the disclosure of
the specification. It is copyrighted. The copyright owner has no
objection to facsimile reproduction by anyone of the patent
document, including the copyrighted Appendix A and the drawings as
they appear in the Patent and Trademark Office file or records, but
otherwise reserves all rights.
TECHNICAL FIELD
[0002] This invention relates to the field of software plug-ins for
multimedia file players and for other applications supporting
ordered data flow files. More precisely, this invention defines a
new field of software that allows plug-ins and content to be
insulated from differences in underlying platforms and
frameworks.
BACKGROUND OF THE INVENTION
[0003] Multimedia player frameworks have become widespread. Later
versions of the most popular Internet browsers--Microsoft's
Internet Explorer and Netscape's Communicator--include at least one
player in the basic package. These are frameworks such as
RealNetworks, Inc.'s RealPlayer.TM. G2 family (collectively, the
"RealPlayer.TM."); Microsoft Corporation's Windows Media
Technologies (NetShow.TM.); Macromedia, Inc.'s Director.TM.; Apple
Computer, Inc.'s QuickTime.TM.; and Sun Microsystems, Inc.'s
Java.TM. Media Framework (JMF). Most of these frameworks are
extensible by means of plug-ins discussed below; some, notably JMF,
are extensible by means of applications built on top of the
frameworks. By being extensible we mean that a particular framework
supports a set of interfaces exposed for interaction with
additional software modules or components.
[0004] A framework is a set of interfaces, e.g., API's and
procedures, and a set of capabilities exposed for the purposes of
extensibility. It need not operate on multimedia-type files, i.e.,
files ordered sequentially and supporting the concept of a
time-line; frameworks generally need not operate on time- or
otherwise-ordered data files. In the rest of this document,
however, we will discuss predominantly frameworks operating on
ordered data flow files. We will refer to such frameworks
interchangeably as "frameworks," "extensible frameworks,"
"extensible data flow-based frameworks," or by similar expressions;
we do not imply any difference in the particular designation used,
unless otherwise indicated.
[0005] Frameworks are extended by means of extensions, for example
plug-ins. Plug-ins, also referred to as extension modules, are
modules or components dynamically loaded at run-time. Extensible
architecture and plug-ins have been used in commercial products,
the RealPlayer.TM. and Windows Media Technologies being two
instances. They also have been described in various publications.
See U.S. Pat. Nos. 5,838,906 and 5,870,559. (The specifications of
these patents are incorporated by reference as if fully set forth
herein.) A number of companies market plug-ins specifically for
extending multimedia players.
OBJECTS OF THE INVENTION
[0006] One difficulty faced by plug-in developers is that a plug-in
must be ported for each platform, i.e., for each hardware-operating
system combination. Another difficulty lies in adapting a plug-in
to various frameworks. Third difficulty, closely related to the
first two, is that platform porting and framework adaptation
require developers to have working knowledge of the various
platforms and frameworks. One object of this invention is to
provide a single interface that allows plug-in developers to build
a single, platform independent version of a plug-in. Another object
of this invention is to provide a uniform interface to the various
frameworks so that a single plug-in can extend functionality of
multiple frameworks. The third object of this invention is to
facilitate development of content that can operate similarly, i.e.,
consistently, with different multimedia players (e.g.,
RealNetworks, Inc.'s RealPlayer.TM. and Microsoft Corporation's
Windows Media Technologies (NetShow.TM.)), and platforms (TV, PC,
set-top boxes). The fourth object of this invention invention is to
accelerate the development of content and multimedia plug-ins by
promoting reuse of existing software objects (e.g., objects written
in Java.TM. and JavaScript).
SUMMARY OF THE INVENTION
[0007] This invention is an abstraction layer providing a-uniform
interface between a framework and one or more plug-ins. In the
preferred embodiment, the invention is a Synchronization
Abstraction Layer (SAL) abstracting time-based frameworks into a
common synchronization interface. The SAL synchronizes itself and
other plug-ins to a time-line of the underlying framework--and it
does that independently of the underlying framework. In other
words, the plug-ins interact with the underlying framework through
the SAL, rather than directly. Typically, the SAL is implemented on
top of the synchronization of the Application Programming
Interfaces (API's) provided by the underlying frameworks. It has at
least one point of coupling with the underlying framework: a method
for providing the SAL with the current time.
[0008] The SAL includes a uniform cross-platform interface for the
plug-ins. Preferably, the cross-platform interface has a functional
core independent of the specific framework underlying the SAL.
[0009] Some of the features of this invention are listed below:
[0010] 1. Secure e-commerce, full interactive experiences including
text and voice chat, games, graphics, animations, and advertising
can be integrated and synchronized with streaming multimedia such
as video and audio;
[0011] 2. It can be used to build a content framework that
insulates content and plug-in developers from details and
differences in hardware platforms, so that the same content or
plug-in can run on desktop platforms (PC, Macintosh, Solaris,
Linux), Televisions (GI, SA), or other kinds of devices (AOL-TV,
screen phones);
[0012] 3. It can be used to build a content framework that
insulates content and plug-in developers from specifics of, or
differences in, software platforms, so that the same content or
plug-in can run on Microsoft NetShow.TM., RealNetworks
RealPlayer.TM., Apple Quicktime.TM., Sun Java.TM. Media Framework
or any other media system;
[0013] 4. It can run without a network or on different types of
networks, such as wireless, Intranets, or the Internet;
[0014] 5. It can be used to synchronize arbitrary data types,
including text, chat, graphics, video, audio, and active content
like Java.TM., JavaScript, or Flash;
[0015] 6. The framework-independent layer can be implemented using
different languages, including Java.TM., HTML, XML, JavaScript,
VBScript, or other languages;
[0016] 7. When used for video synchronization, it can be used to
synchronize arbitrary datatypes with different spatial objects in
video, and with different temporal objects in video.
BRIEF DESCRIPTION OF THE DRAWINGS
[0017] FIG. 1 is a diagram of basic processes in a traditional
extensible time-based framework of prior art.
[0018] FIG. 2 is a diagram of basic processes in an extensible
time-based framework incorporating the Synchronization Abstraction
Layer in accordance with this invention.
[0019] FIG. 3 is a high level diagram of a preferred embodiment
implementation specific to a table of contents plug-in.
[0020] FIG. 4 lists a sample toc.rj, an XML descriptor file for a
table of contents plug-in of the previous Figure.
[0021] FIG. 5 lists a sample toc.xml, an XML descriptor file with
detailed hierarchical description of the table of contents plug-in
of FIG. 3.
[0022] FIG. 6 lists a sample toc.smi, an XML file defining a set of
synchronized plug-ins.
DETAILED DESCRIPTION OF THE INVENTION
[0023] FIG. 1 depicts conventional operation of an extensible
multimedia player framework connected to a media server. Server 110
includes Media Object 111, Clock Object 112,. and Custom Objects
113. Framework 120 and Extension Module (plug-in) 130 coexist on
the client side. As is usual for a media player, Framework 120
contains Video Media Object 121, Clock Object 122; and Audio Media
Object 123. Extension Module 130 has Custom Objects 131 and 132.
Server 110 connects to Framework 120 and Extension Module 130
through Network 150. Line 160 indicates the extensibility interface
between Framework 120 and Extension Module 130. Note that this high
level diagram of prior art omits many other components and
processes, including an operating system and rendering means.
[0024] FIG. 2 demonstrates an arrangement functionally similar to
the arrangement of FIG. 1, but using the present invention. Server
210, including Media Object 211, Clock Object 212, and Custom
Objects 213, connects to Framework 220, Extension Module (plug-in)
230, and Synchronization Abstraction Layer (SAL) 270 through
Network 250. Line 260 symbolizes the interface extending Framework
220 through SAL 270. Block 271 denotes Uniform Interface between
SAL 270 and various extension modules, including Extension Module
230.
[0025] As before, FIG. 2 omits many other components and processes,
and is rather generalized. But a number of structural changes can
be envisioned even in this, high level representation. For
instance, media files can be stored locally, obviating the need for
transmission over the network. Further, custom objects need not
reside on the same server as the media files; they may be located
on a separate server or, in some cases, locally. Several extension
modules can be simultaneously connected to the same client
framework, either through the SAL or directly. (The SAL itself is,
of course, an extension module.)
[0026] The discussion above focused on media players rendering
multimedia files supporting the concept of time-line, such as video
and audio clips, because these applications are commercially
important. The invention, however, is not limited to rendering
multimedia or time-line ordered files. Time-line sequencing is only
one example of data flow order within files, including media files.
Many other types of ordering are possible. As one example, consider
a video taken from a window of a car going down a highway. The
images can be time- or distance-sequenced.
[0027] Files of-interest in the present context, which we designate
as ordered data flow files, may also be ordered in multiple
dimensions. Imagine a video game called "Tank Commander," with the
player controlling a virtual tank moving cross country. Because the
tank can move in two dimensions, the top view of the tank and its
immediate surroundings depends on two coordinates. The video data
must therefore be ordered in two dimensions. Similarly, video of a
one-directional view from an object moving in three dimensions
requires three dimensional ordering. Clearly, the concept of
ordered files goes beyond the four dimensions of space-time.
[0028] We also need not limit this concept to files ordered
according to variables perceived as continuous, e.g., space and
time. The concept is equally applicable to ordering according to
discrete variables, such as event occurrences or quantized-space
positions. As an example, consider a server storing a library of
books and sending the content to the client for sequential viewing
page by page. The concept of ordered data flow files is therefore a
general one.
[0029] Finally, in its general form the invention need not
synchronize data flow, but may merely provide a uniform
cross-platform interface for plug-ins.
[0030] Although the invention can be implemented in various
languages, it is preferable to use a platform-independent source
code. Therefore, consistent with the spirit of the invention, its
preferred embodiment is implemented in Java.TM.--an
architecture-neutral, object-oriented, multithreaded language
intended for use in distributed environments. (Partial source code
of the preferred embodiment is listed in Appendix A and constitutes
a part of the disclosure of this specification.) The Java.TM. code
is executed by a Java.TM. Virtual Machine (JVM), which is a
byte-code interpreter. The JVM is stored in one location and
instantiated separately for each extension module, eliminating the
need to store duplicate copies of the JVM for other modules that
are also implemented in Java.TM..
[0031] Alternatively, the invention can instantiate a JVM from a
copy in the framework, in the operating system, or even in another
application.
[0032] The preferred embodiment creates a-uniform interface to the
RealPlayer.TM. of Real Networks. It can be easily adapted to extend
other multimedia players in addition to the RealPlayer.TM.,
especially if a common set of functional features exists in the
various players. This should be the case for most multimedia
players. Because the Java.TM. code of a specific extension can be
mostly independent of a particular framework being extended, large
sections of the SAL's code can be shared among different framework
adaptations of the same plug-in.
[0033] FIG. 3 illustrates a high level diagram of a more specific
example of the preferred embodiment. Here, the SAL is used to
plug-in a table of contents ("TOC") extension module into the
RealPlayer.TM.. The TOC plug-in displays a hierarchical view of the
table of contents of a video presentation described in the table of
contents. In the hierarchy, entries in the table are highlighted
during the video presentation of their associated video portions. A
user can randomly access various-portions of the video by clicking
on the corresponding entries in the TOC.
[0034] In this configuration RealPlayer.TM. 301 has three extension
modules: SAL (RealJava) 310, RealVideo 302, and RealText 303.
(Hereinafter we denominate by "rj" or "RealJava" all Java.TM.
objects compatible with the G2 architecture.) For clarity, SAL 310
is expanded within the dotted oval into its conceptual
components--File Format Filter 311 and Renderer 312. This
conceptual separation results from the G2 architecture constraints
of the RealPlayer.TM.; in a SAL adaptation compatible with another
media player, such conceptual separation may be inappropriate.
[0035] When a user invokes the RealPlayer.TM. application and the
TOC extension module, RealPlayer.TM. 301 accesses Web Server 320
over Network 330 to retrieve "toc.smi," a synchronized multimedia
integration language ("SMIL") descriptor file defining a set of
synchronized plug-ins to run simultaneously in the RealPlayer.TM.
environment. This file contains a reference to "toc.rj," an
extensible markup language ("xml") descriptor file for the table of
contents extension module compatible with the RealPlayer.TM.. Based
on the MIME type of the toc.rj file, RealPlayer.TM. 301
instantiates SAL 310 and routes the toc.rj file to it. SAL 310
includes File Format Filter 311 and Renderer 312, each implemented
using Java.TM. code linked with native platform C++ code.
[0036] File Format Filter 311 parses the toc.rj file and hands it
over to Renderer 312 through a standard protocol defined by the
RealPlayer.TM.. The parsed data contains following elements: (1)
name of the Java.TM. class to instantiate; (2) location from which
the Java.TM. class can be retrieved; and (3) any other parameters
specific to the extension module. A sample toc.rj file appears in
FIG. 4. The first line (starting with classid) specifies the
Java.TM. class names; the next two lines (codebar and archive)
specify a URL location and names of Java.TM. class files; the
following line assigns the width and the height of the window for
the table of contents; sync is the period in milliseconds between
synchronizing events, i.e., between timing calls from the
RealPlayer.TM. to the SAL; duration refers to total length of the
time-line of the presented event; and, lastly, param is the
parameter that points to the URL with the toc.xml file, the XML
file descriptor of the specific table of contents.
[0037] Renderer 312 instantiates JVM (Java.TM. Virtual Machine) 313
needed to run the Java.TM. code, and also retrieves ".jar" files
(packaged Java.TM. class files) over Network 330. Note that the URL
for the ".jar" files is available from the parsed toc.rj file
discussed above. Renderer also instantiates an instance of RealTOC
object 314 and hands to it the URL for toc.xml, the XML file
describing the specific table of contents object associated with
the multimedia presentation. This file includes the nodes and
sub-nodes within hierarchical structure of the specific table of
contents; it also includes the "begin" and "end" times
corresponding to each node and sub-node. RealTOC retrieves the
toc.xml file, parses it, and builds the specific table of contents
tree structure. A sample toc.xml file appears in FIG. 5.
[0038] At the beginning of the multimedia presentation, RealVideo
object 302 creates a video window for rendering video, while
RealTOC creates a TOC window for rendering the table of contents.
Both windows are rendered within the larger RealPlayer.TM. window.
As the time-line increases during the presentation, RealPlayer.TM.
301 periodically calls SAL 310 with the current time; SAL 310
passes the current time to JVM 313, which in turn notifies RealTOC
314; RealTOC 314 highlights the appropriate node on the table of
contents tree rendered in the TOC window. A similar process can be
constructed with the SAL keeping current time and calling the
RealPlayer.TM. with time updates.
[0039] When the user clicks on a particular heading within the
table, RealTOC 314 sends the "begin" time associated with the
heading to the "seek" function of RealPlayer.TM. 301 through JVM
313 and SAL 310; RealPlayer.TM. 301 notifies all synchronized media
servers (not shown) of the new "begin" time, waits for the all the
media servers to synchronize to the new time, then updates its
internal current time and sends the new current time to its
extension modules, i.e., RealVideo 302 and SAL 310 (and RealText
103, if applicable); SAL 310 updates the current time of RealTOC
314 through JVM module 313. Importantly, both RealTOC 314 and
RealVideo 302 are both synchronized to the current time of
RealPlayer.TM. 301. Thus, after RealTOC 314 invokes the seek
function requesting a new time, its current time does not skip to
the new time until RealPlayer.TM. 301 notifies RealTOC 314 of the
change in current time. If the seek function in RealPlayer.TM. 301
is disabled, as it should be for live video, the time line
continues uninterrupted.
[0040] Seek function is only one example of controlling the flow of
data. Fast forward and rewind functions also control the flow of
data by changing the rate of the flow. Moreover, as discussed
previously, the data can be arranged in multiple dimensions; then,
its flow direction is a vector whose direction may be controlled
from the extension module in a way similar to the control of data
flow from the table of contents described above.
[0041] Although a specific embodiment of this invention has been
shown and described, it will be understood that various
modifications may be made without departing from the spirit of this
invention.
* * * * *