U.S. patent application number 12/445055 was filed with the patent office on 2011-06-30 for method and apparatus for managing multimedia files.
This patent application is currently assigned to Tagmotion Pty Limited. Invention is credited to John Vernon Polglase, Andrew Simms.
Application Number | 20110161174 12/445055 |
Document ID | / |
Family ID | 39282366 |
Filed Date | 2011-06-30 |
United States Patent
Application |
20110161174 |
Kind Code |
A1 |
Simms; Andrew ; et
al. |
June 30, 2011 |
METHOD AND APPARATUS FOR MANAGING MULTIMEDIA FILES
Abstract
A system for locating, segmenting, annotating and retrieving
multimedia files, provides a database of metadata (68) relating to
multimedia files (70), a database manager (64), and a database
client (66) for accessing the data contained within the database.
The database client (66), together with the metadata database (68)
and database manager (64) provide a variety of different
functionalities, namely a deep linking functionality, a
segmentation functionality, a metadata annotation functionality, a
retrieval functionality, and an access functionality. The user,
through database client (68) annotates the multimedia file or
segment of multimedia file with metadata which is saved in the
database (68). When the user desires to locate a multimedia file,
the metadata is searched or browsed to locate the database entry
associated with the multimedia file in question. The system further
provides the ability for the user to load, open and start the
playing of a multimedia file from a point intermediate to its start
and end point, and without the user having to experience the
portion between the start point and the intermediate point, break
down larger files into shorter segments or individual frames (or
still images) or regions within frames (or still images) without
creating additional copies of the selected segments of the
multimedia files of interest, thereby avoiding copyright issues
from arising.
Inventors: |
Simms; Andrew; (New South
Wales, AU) ; Polglase; John Vernon; (New South Wales,
AU) |
Assignee: |
Tagmotion Pty Limited
Paddington NSW
AU
|
Family ID: |
39282366 |
Appl. No.: |
12/445055 |
Filed: |
October 11, 2007 |
PCT Filed: |
October 11, 2007 |
PCT NO: |
PCT/AU2007/001559 |
371 Date: |
January 14, 2011 |
Current U.S.
Class: |
705/14.58 ;
705/14.49; 705/14.69; 707/737; 707/E17.046 |
Current CPC
Class: |
G06F 16/435 20190101;
G06F 16/48 20190101; G06Q 30/0261 20130101; G06Q 30/0251 20130101;
G06Q 30/0263 20130101; G06F 16/41 20190101; G06F 16/4387 20190101;
G06Q 30/0273 20130101 |
Class at
Publication: |
705/14.58 ;
705/14.69; 705/14.49; 707/737; 707/E17.046 |
International
Class: |
G06Q 30/00 20060101
G06Q030/00; G06F 17/30 20060101 G06F017/30 |
Foreign Application Data
Date |
Code |
Application Number |
Oct 11, 2006 |
AU |
2006905774 |
Claims
1. An indexing and retrieval system for multimedia comprising: a
multimedia apparatus for receiving and playback of at least one
multimedia file; a data container for storing the at least one
multimedia file; a database of metadata that describes at least the
location of the at least one multimedia file; a database manager
accessible through the multimedia apparatus that further comprises:
an annotation module for defining and recording multimedia clip
metadata in the metadata database that further describes the at
least one multimedia file wherein the further metadata includes at
least: classification term hierarchies that comprise branches of a
knowledge tree and; segment information including start and end
timepoints which define a clip of the at least one multimedia file;
a search and retrieval module for identifying multimedia clips and
at least one multimedia file of interest to a user of the system
through the provision of a search interface to the metadata
database whereby the module provides the ability to search the
metadata database by reference to keywords and hierarchies of
classification terms constituting branches of the knowledge tree
and wherein the user of the system is only presented with the
segment of the at least one multimedia file defined by the segment
information without having the user first experience the playback
of the beginning of the multimedia file.
2. The system of claim 1 where there are a plurality of multimedia
apparatuses connected to the Internet, there are one or more data
containers accessed by the plurality of multimedia apparatuses over
the Internet and the database manager is accessed by the users of
the plurality of multimedia apparatuses through the Internet.
3. The system of claim 2 wherein the database manager, through its
search and retrieval module, presents the clip of the at least one
multimedia file defined by the segment information by means of
directing the multimedia apparatus to the data container that
stores the at least one multimedia file and specifically, to the
point in the at least one multimedia file defined by the clips
information's start timepoint and not to a separate, additional
multimedia file which comprises only the clip of interest.
4. The system of claim 3 wherein users of the plurality of
multimedia apparatuses connected to the metadata database via the
Internet are provided with the ability to create custom
classification terms and information branches of the knowledge tree
for their use in further defining the at least one multimedia file
and clips.
5. The system of claim 4 which further comprises an advertising hub
database which features a web based interface, said advertising hub
being connected via a network to the metadata database, wherein the
advertising hub database queries the metadata database for
hierarchies of classification terms and keywords contained therein,
and wherein the advertising hub database is accessed by advertisers
who select metadata taken from the list of hierarchies of
classification terms and keywords and purchase the right to display
advertisements in relation to the selected metadata.
6. The system of claim 5 wherein the advertisements are displayed
alongside multimedia files when the relevant metadata is being used
to find and display the at least one multimedia file and clips.
7. The system of claim 5 wherein the advertisements themselves
comprise the at least one multimedia file and wherein the database
of metadata is used to deliver contextually specific advertisements
to a potential purchaser through identification of relevant
information that is specific to the purchaser and which corresponds
to classification terms that form part of a hierarchy of
classification terms.
8. The system of claim 7 wherein the information that is specific
to the purchaser are the particular nodes of specific information
branches that are traversed by a purchaser looking for particular
multimedia content.
9. The system of claim 7 wherein the information that is specific
to the purchaser is locational information.
10. The system of claim 9 where the locational information is
comprised of the GPS co-ordinates of the purchaser.
Description
FIELD OF THE INVENTION
[0001] The present invention relates to a method and apparatus for
annotating and indexing electronic files with metadata, and
navigating said metadata for retrieval and use by the user. More
particularly the electronic files described in this embodiment of
the invention are multimedia files, such as digital audio and video
files, and the metadata applied to the files consists of both
keywords and hierarchies of classification terms. Further, the
present invention provides a method and apparatus for navigating
large volumes of structured information including hierarchies of
classification terms. The present invention also provides for a
collaborative online environment where communities and groups can
gather and exchange ideas through mash ups and the collaborative
use of personal dictionaries.
BACKGROUND TO THE INVENTION
[0002] The desire for personal control, possession and ownership
drove the evolution of computing from the handful of mainframe
computers in large corporate, government and university settings
decades ago to the situation today where personal computers now sit
in homes and businesses everywhere.
[0003] But the widespread availability and ownership of personal
computers in developed countries did nothing to quell consumers'
thirst for personal `computing` devices per se. Laptops, mobile
phones, camera phones, games consoles, portable games consoles,
portable music players and PDA's have proliferated in society
alongside personal computers.
[0004] And in more recent times, as these kinds of devices have
continued to proliferate, we have also seen major developments with
software and services on the web. In particular, we have seen the
evolution from peer-to-peer file sharing networks to social
networks like Linkedln and MySpace, to the explosion in video
sharing websites like YouTube and MSN Video. These have all played
their part in the rise of what's become known as social
networking.
[0005] One of the major uses of the internet today is to
disseminate, discuss, and otherwise consume multimedia content
including videos, photos, and audio tracks. This is being done
today, not only on personal computers, but also handheld mobile
computing devices such as PDA's, digital mobile telephones,
navigation devices, personal media players such as Apple's iPod,
and indeed, devices which incorporate all of the features of these
handheld devices such as the iPhone. Further, online multimedia
content is being delivered directly to set top boxes and games
consoles connected to the internet such as Tivo devices and
Microsoft Xbox's.
[0006] One problem associated with accessing multimedia material on
the internet is that more often than not, the segment of interest
is often found within a larger multimedia file. For example, if a
particular scene is found one hour into a movie, a link to that
movie would generally take a user to the start of the movie and not
to the section of interest.
[0007] Another problem associated with accessing multimedia
material on the internet is that metadata associated with the
multimedia file, which could be useful to determine relevancy, is
often ambiguous and of little real help in determining what the
multimedia file contains.
[0008] Another problem with prior art annotation and retrieval
systems is that they provide little scope for using natural
language and/or classification terms unique to the user. Ultimately
there is little scope for personalisation of the process.
[0009] Further problems associated with prior art systems is that
they are rigid with respect to users. Some systems provide for
single user systems, whilst others are designed with groups in mind
and are less amenable for use by a single user or different types
of users.
[0010] Further problems associated with prior art systems is that
they lack the ability to treat segments of multimedia files as
entries in their own right in a metadata database without first
physically creating segments for re-entry into the database. The
creation of further copies of portions of multimedia files is
therefore subject to copyright laws and the problems that are
associated with them.
[0011] Further, some prior art systems lack the ability to
associate database entries with logically segmented multimedia
files.
[0012] Further problems with automated tools for extracting
metadata from content (such as voice to text, voice recognition and
pattern recognition tools) are that they are limited in that they
cannot capture the range and depth of human interpretation. For
example, does the footage of an allied soldier in the desert depict
a soldier, a Sergeant, a son, an infidel, a hero, an American, a
voter or a military uniform?
[0013] Further, multimedia annotation based on keywords is devoid
of context. This ability to apply and preserve context will become
more and more important as the volume of multimedia on the web
increases. Context is important because keyword searches and
searches involving combinations of keywords return massive numbers
of hits, making it extremely difficult for users to identify the
relevant file/s, and even more difficult to identify the relevant
segment or point within a file. Further, context is important to
online advertisers who need to target the particular type content
in which they want their advertisements to appear in connection
with.
OBJECT OF THE INVENTION
[0014] It is an object of the invention to overcome at least some
of the stated deficiencies of the prior art.
SUMMARY OF THE INVENTION
[0015] According to one aspect of the invention there is provided,
a system for locating, segmenting, annotating and retrieving
multimedia files, said system providing: [0016] database of
metadata relating to multimedia files; [0017] database manager
[0018] a database client for accessing the data contained within
the database, said client providing: [0019] deep linking
functionality where the user can load, open and start the playing
of a multimedia file from a point intermediate to its start and end
point, and without the user having to experience the portion
between the start point and the intermediate point; [0020]
segmentation functionality whereby large multimedia files can be
broken down into shorter segments or individual frames (or still
images) or regions within frames (or still images); [0021] metadata
annotation functionality wherein the user can annotate whole
multimedia files, individual segments, frames and regions within
frames with alphanumeric information; [0022] retrieval
functionality for browsing and searching the metadata; and [0023]
access functionality for displaying and viewing the multimedia
file; [0024] wherein the user annotates the multimedia file or
segment of multimedia file with metadata which is saved in the
database, and wherein the user desired to locate a multimedia file,
said metadata is searched or browsed to locate the database entry
associated with multimedia file in question.
[0025] In a preferred embodiment of the invention, the annotation
of the multimedia file or portion thereof is achieved by recording
in the metadata saved in the database, a hierarchy of
classification terms that form an information branch of the
database's knowledge tree.
[0026] In a preferred embodiment of the invention the multimedia
files are stored locally or are available to the database manager
over an IP connection.
[0027] In a further preferred embodiment of the invention the IP
connection is a HTTP connection over TCP/IP.
[0028] In a preferred embodiment of the invention the database
manager provides for 2 or more types of user accounts.
[0029] In a further preferred embodiment of the invention the
database manager provides for administrator, librarian, annotator
and searcher user accounts.
[0030] In a further preferred embodiment of the invention there may
be multiples of each user type.
[0031] In a further embodiment of the invention the user is able to
annotate the multimedia file or segmented portion thereof within
still video by selecting the lowermost classification term of the
desired information branch.
[0032] In a further embodiment of the invention the user is able to
select the lowermost classification term of the information branch
by pointing, clicking and dragging the lowermost classification
term to the relevant storyboard or catalog entry.
[0033] In a further embodiment of the invention the user is able to
create overlapping segments of the one multimedia file by logically
segmenting the original file in a way such that the end point of
the first segment proceeds beyond the start point of the second
segment.
[0034] In a preferred embodiment of the invention the database
client is loaded onto a computing apparatus, said apparatus
comprising at least one data processor, input and output modules,
memory coupled to the at least one data processor and having stored
therein programming instructions for performing data
processing.
[0035] In a preferred embodiment of the invention, the computing
apparatus further comprises persistent storage.
[0036] In a preferred embodiment of the invention, the computing
apparatus further comprises a network adaptor which permits the
integration of the computing apparatus on an IP network.
[0037] In a preferred embodiment of the invention, the IP network
is the Internet, provided over a TCP/IP connection.
[0038] In a preferred embodiment of the invention the computing
device is a personal computer or laptop.
[0039] In an another embodiment of the invention, the computing
apparatus can be taken from the following list of devices: network
workstations, set top boxes, digital mobile telephones, network IP
routers, media players, games machines, NAS devices, video
conferencing equipment and satellite navigation displays.
[0040] According to a further aspect of the invention there is
provided a method for locating, segmenting and annotating
multimedia files, said method comprising the following steps:
[0041] identifying the name and location of a multimedia file;
[0042] creating a database entry for associated metadata; [0043]
loading the multimedia file into memory; [0044] assigning a short
title to the database entry [0045] assigning keywords and
information branches to the database entry [0046] segmenting the
multimedia file with into shorter video and audio portions, still
frames, and regions within still frames thereby-creating the at
least a second database entry; [0047] assigning a short title to
the at least second database entry; [0048] assigning keywords and
information branches to the at least second database entry [0049]
wherein each database entry's metadata is browsed and/or searched
during the retrieval process by the user, said metadata ultimately
providing the location and starting position of the multimedia file
or portion thereof associated with the database entry.
[0050] The method further provides for retrieving the annotated
multimedia files or portion of multimedia files by means of the
user querying the database by browsing the database's knowledge
tree to find information branches that are associated with the
desired multimedia files or portions thereof, or by conducting
keyword searches.
[0051] A preferred embodiment of this invention provides the
additional step of providing the user with multiple views of the
database's contents including at least a knowledge tree view.
[0052] In a further preferred embodiment of this method there is
provided a further step of providing a manipulated view of the
database's structure, the depiction comprising a contorted view of
at least two information branches intersecting at nodes matching
the search term, and wherein the user can jump up or drop down a
level on the intersected tree to see the classification used in
proximity to the intersected node in one view.
[0053] A further preferred embodiment of the invention involves the
user selecting individual information branches from the intersected
view for further analysis.
[0054] In a preferred embodiment of the invention, there is an
active node of the knowledge tree which is associated with a
further view depicting the contents of the knowledge tree at that
point, as determined by the information branches contained within
the database entries which correspond to certain parts of the
database's knowledge tree.
[0055] In a preferred embodiment of the invention the user browses
the knowledge tree by moving up, down and laterally to get to the
next active node.
[0056] In a preferred embodiment of the invention the content view
of each node on the knowledge tree is updated and kept live with
each movement on the knowledge tree by the user.
[0057] In a preferred embodiment of the invention, the invention
further comprises the step of highlighting on the knowledge tree
those nodes of the tree that contained database entries which
matched the search term used to query the database.
[0058] In a preferred embodiment of the invention the user can
conduct a search using a keyword or alphanumeric string over the
contents displayed as a result of selecting a certain node.
[0059] The method further provides that when the user locates a
multimedia file through reference to metadata located in a database
entry, the user is able to start playing the segmented portions of
multimedia files, particularly audio and video multimedia files,
from their start positions recorded in their database entries and
not the true start position of the multimedia file.
[0060] According to a further aspect of the invention there is
provided a system for a large group of users to locate, segment,
annotate and retrieve multimedia files wherein access privileges
can be modified for each user or type of user such that only
certain multimedia files are available to that user or type of
user, or that certain functions, such as adding new users are only
granted to certain users or types of users.
[0061] According to a further aspect of the invention there is
provided a method for providing targeted advertising for display to
users whilst searches are being conducted, said method comprising
the steps of: [0062] identifying information branches and specific
nodes on a knowledge tree; [0063] auctioning or selling the right
to display advertisements when the user browses over a node of the
knowledge tree, selects a specific information branch, or when the
user is viewing a multimedia file that has associated with its
database entry at least one information branch corresponding to the
one purchased by the advertiser; [0064] displaying the
advertisements of the highest bidder when the user browses over the
auctioned node of the knowledge tree, or selects the auctioned off
information branch, or when the user is viewing a multimedia file
that has associated with its database entry at least one
information branch that was purchased during the auction.
[0065] According to another aspect of the invention there is a
method of providing targeted advertising, said method comprising
the steps of: [0066] identifying information branches and nodes on
a knowledge tree; [0067] creating a database entry for an
advertiser which contains at least one information branch which
describes the advertising content; [0068] such that when a user is
browsing the contents of a particular node or information branch
the advertisement appears to the user as content.
[0069] According to another aspect of the invention there is
provided a method for providing targeted advertising, said method
comprising the following steps: [0070] aggregating at least the
metadata contained within disparate multimedia metadata databases
in a central database accessible over the internet; [0071]
providing advertisers with the metadata used in each of the
disparate multimedia metadata databases; [0072] the advertisers
bidding on the keyword, alphanumeric string, classification term or
information branch that they desire, said bid including the details
of which database users they wish to advertise to; [0073] the
database manager accepting the bidder or bidders that maximise the
overall revenue raising capacity of the network; [0074] the
database manager communicating with the individual multimedia
metadata databases that the advertiser has selected for advertising
to its users, said communication including directions to associate
the advertiser's advertisement with the keyword, alphanumeric
string, classification term or information branch that the
advertiser purchased such that users of the database will be
presented the advertisement if the relevant keyword, alphanumeric
string, classification term or information branch that the
advertiser purchased is found in a search result or as a result of
browsing.
[0075] In a preferred embodiment of this invention the disparate
multimedia metadata databases provide the centralised database a
detailed account of past searches conducted over the database and
the results of those searches, these are provided to the
advertisers in addition to the metadata so as to assist them in
their decision making.
[0076] In a preferred embodiment of the invention the centralised
database aggregates classification terms and information branches
used in each database's knowledge tree which are then passed on to
advertisers for their review and eventual bidding.
[0077] In a further embodiment of the invention there is provided a
method for locating files that are available on the internet,
wherein said method includes the following steps: [0078] locating
the file of interest on the internet; [0079] adding a database
entry to the metadata database in respect of the file; [0080]
annotating the database entry with metadata including at least
information branches which describes the file; [0081] incorporating
the metadata into the database including the incorporation of
information branches into a knowledge tree; [0082] searching for
the file by browsing or searching the metadata database knowledge
tree; [0083] obtaining the file from the internet once the relevant
database entry is uncovered and the URL obtained.
BRIEF DESCRIPTION OF THE DRAWINGS
[0084] In order that the invention may be more readily understood
and put into practical effect, reference will now be made to the
accompanying drawings which each depict preferred embodiments of
the present invention:
[0085] FIG. 1--depicts the devices the invention may be implemented
upon
[0086] FIG. 2--depicts the basic constituents of one aspect of the
invention
[0087] FIG. 3--depicts the logical architecture of one aspect of
the invention
[0088] FIG. 4--depicts one aspect of the invention on one computer
(locally)
[0089] FIG. 5--depicts one aspect of the invention substantially on
a LAN
[0090] FIG. 6--depicts one aspect of the invention substantially on
a WAN
[0091] FIG. 7--depicts one aspect of the invention in which the
content is hosted centrally
[0092] FIG. 8--depicts a further aspect of the invention in which
the content is hosted in a distributed manner
[0093] FIG. 9--depicts the different multimedia files managed by
one aspect of invention
[0094] FIG. 10--depicts the file formats supported by one aspect of
the invention
[0095] FIG. 11--depicts a knowledge tree according to one aspect of
the invention
[0096] FIG. 12--depicts the segmentation and annotation functions
of one aspect of the invention
[0097] FIG. 13--depicts catalog entry fields
[0098] FIG. 14--depicts overlapping segments of a multimedia file
according to one aspect of the invention
[0099] FIG. 15--depicts the hierarchy of user roles according to
one aspect of the invention
[0100] FIG. 16--depicts annotation and segmentation workflows in
one aspect of the invention
[0101] FIG. 17--depicts the retrieval process according to one
aspect of the invention
[0102] FIG. 18--depicts the retrieval process in more detail
[0103] FIG. 19--depicts a knowledge tree of classification terms
according to one aspect of the invention
[0104] FIG. 20--depicts individual information branches of the
knowledge tree
[0105] FIG. 21--depicts classification terms as slices of a
knowledge tree according to one aspect of the invention
[0106] FIG. 22--depicts a screenshot after conducting a keyword
search according to one aspect of the invention
[0107] FIG. 23--depicts a screenshot after user rises the slider up
on the intersected view according to one aspect of the
invention
[0108] FIG. 24--depicts devices accessing the metadata database
according to one aspect of the invention
[0109] FIG. 25--depicts a network of metadata databases according
to one aspect of the invention
[0110] FIG. 26--depicts a hub and advertiser server implementation
according to one aspect of the invention
[0111] FIG. 27--depicts various views of knowledge trees according
to one aspect of the invention
[0112] FIG. 28--depicts a screen shot post keyword search showing
advertisements according to one aspect of the invention
[0113] FIG. 29--depicts a screen shot after user selects one
information branch according to one aspect of the invention
[0114] FIG. 30--depicts a screen shot after user selects one of the
videos in the search results according to one aspect of the
invention;
[0115] FIG. 31--depicts a screen shot after the user selects one of
the information branches according to one aspect of the
invention.
DETAILED DESCRIPTION OF THE INVENTION
[0116] The growth in the market for internet connected multimedia
devices, including computers and laptops, combined with the rise of
social networking has its roots in distinctly personal motivations.
It is the desire to publish--whether to express oneself
artistically or intellectually--the desire to `be heard` or to
initiate online `conversations` and receive feedback, to share
one's expertise and build a community of `subscribers` to your
content, or just to `show off`.
[0117] What has become known as `citizen journalism` is but one
aspect of this phenomenon. The core motivation here is the same as
for `publishing` one's views via talkback radio, or in a journal,
book or blog. But as more and more people turn to multimedia as
their chosen form of expression and the internet as their chosen
method of publishing, and with more and more people consuming this
online content, the demand for simple, intuitive methods for
navigating this content is increasing rapidly.
[0118] The key to having a flexible navigation system such as this
is the existence of a personal metadata database as its foundation
unit. This database contains the `tags` that a person has created
to allow him to easily navigate (search, browse and retrieve)
content of interest. As more and more multimedia content is
published, it becomes more and more important to be able to find
exactly what one is interested in, from the ocean of choices
available. This is the value of a contained, personal `view`. By
combining these personal databases to form group views and/or
collective views, an individual can opt to use expanded, richer but
also more complex views to navigate content, whenever he chooses.
The creation, use and re-use of personal databases such as these is
at the heart of this invention.
[0119] A personal metadata database, as described, can represent
more value to its owner than its value purely as a superior
navigation tool for multimedia.
[0120] Since the tags represent the meaning that humans ascribe to
particular multimedia, they are of potential value to advertisers
who want to reach people who use search terms the advertiser
`owns`, as they browse, search and view the content that's been
described using tags or combinations of tags in a personal metadata
database.
[0121] Generally the present invention is concerned with providing
systems and methods for one or more users to locate, annotate
and/or retrieve multimedia files. Whilst the present invention will
be described with respect to multimedia files and computers, the
person skilled in the art will recognise the fact that the
invention may be implemented in many different devices and
situations.
[0122] Turning to FIG. 1 the range of potential devices in which
the invention may be implemented is depicted. Such devices include,
laptop computer 10, set top box 12, internet connected navigational
units 14, games consoles 16, network attached storage devices 18,
and storage devices generally, portable music and/or games device
20, video conferencing device 22, personal digital assistant 24,
and digital mobile telephone 26.
[0123] Turning to FIG. 2 the essence of the invention is disclosed
which is database 30. This database contains only metadata about
the multimedia files of interest. The multimedia files of interest
are represented by online container 32, which is connected to
database 30, through the internet 34. Ultimately the user sitting
at computer 36 queries the database 30 via a client side
application running on the computer (not shown). The database 30
instructs the computer 26 to obtain the multimedia file of interest
from online container 32.
[0124] FIG. 3 depicts the arrangement of logical components which
in the case of the present example depicted in FIG. 3 is
implemented on two computers, a server 40, and a client 42, both
containing multimedia content, in addition to an online storage
repository 54 containing further multimedia content. On the server
side 40, there are three main components, metadata database 44,
database manager 46 and a local repository of multimedia files 48.
The database manager 46 controls the access to and modifications to
the database 44 which contains the information about the multimedia
files located in local repository 48. However for these
interactions to occur, a user needs access the database from the
client computer 42 connected to server via a network connection 50
including those that run on the TCP/IP network protocol.
Specifically, the user needs to operate the database client 52 to
query database 44, through the actions of the database manager 46.
The database 44 may contain metadata in respect of multimedia files
located in repositories 54 and 56 if the contents of these
repositories have been added to the database 44, via the database
manager 46, and repository 48.
[0125] FIGS. 4, 5, and 6 all depict alternative implementations of
the invention. In particular the depiction in FIG. 4 is that of a
computer 60, which has software that embodies the invention loaded
onto it represented by the software objects 62, 64, and 66. The
depiction in FIG. 4 is a that of a local implementation. All
software modules and data reside on the one computer. The computer
60 does not communicate with any other computer on any network.
Notwithstanding the fact that the implementation is local, the
logical architecture remains the same as in the case of FIG. 3. In
particular the computer 60 still needs to have loaded onto it
client side applications being the media player 62 and database
client 66. Further the implementation in FIG. 5 requires a database
manager 64. In either of the implementations depicted in FIGS. 4, 5
and 6, the media files of interest located in the repositories are
accessed by the media player through reference to the media files
URL which may includes both HTTP: and FILE: protocols.
[0126] FIG. 5 depicts an alternative arrangement, one which is
predominantly LAN based. In this configuration the user operates
computer 72 which contains the database client application, to
query metadata database 74, which sits on the same local IP network
as repository of multimedia files 76. The figure also depicts
internet gateway 78 and a repository of multimedia files 80,
located on the WAN. This configuration might be similar to what one
would expect a small to medium media company to need to implement
to manage their content stored on their corporate LAN with the
occasional need to access materials on the greater WAN or
internet.
[0127] FIG. 6 depicts a further arrangement of components including
computer 82. However in the present example, the metadata database
84 is located on the other side of internet gateway 86 which places
it on the internet or WAN.
[0128] FIGS. 7 and 8 depicts two further alternative embodiments of
the invention. The figure depicted in FIG. 8 is representative of
the examples provided in FIGS. 6 and 7 in that the metadata
database is kept separate from the repositories of multimedia
files. As can be seen when regard is had to FIG. 8 the only data
being saved to the metadata database is metadata. No copies of the
multimedia files are maintained within the database for later
retrieval. Further, once the database server instructs the client
application resident on computer 82 where to find the multimedia
file in question the client application goes and obtains it
directly from the relevant online repository and not the server
84.
[0129] The depiction in 7 however is the opposite. In this
scenario, when a multimedia file is requested during step 88 it is
returned not only to the client application during step 90, but
also the metadata database during step 92 such that when the file
is requested during step 94, the file can be delivered centrally
from the server during step 96 (or retrieved locally if it is still
in its cache).
[0130] FIG. 9 depicts the types of multimedia files that one
embodiment of the invention can manage. Presently the invention has
been designed to operate with audio, still images, and video
(either with or without sound). The person skilled in the art
however will realise that the present invention has application
outside of these three distinct areas and that these other areas
fall within the scope of this invention. For example, the invention
could equally be applied to collections of text or indeed,
executables or other program files located on the internet. The
file formats that are currently accepted by the present invention
include those listed in FIG. 10.
[0131] FIG. 11 depicts the way in which multimedia file metadata is
arranged, being a hierarchical structure, according to one aspect
of the invention. On the diagram, the knowledge tree 100 is
depicted. It is comprised of nodes 110 which are classification
terms taken from either the global or personal dictionary of the
user. These terms are organised in a hierarchy such that the
hierarchy of terms conveys more semantic meaning than any
individual classification term or collection of terms where the
relationship between the terms is not as apparent as it is in a
hierarchy. It is this semantic meaning and attribution of the
meaning to individual media files 105 and segments thereof. The
individual media files do not form part of the knowledge tree 100.
For example node 110 of the knowledge tree 100 contains a
collection of multimedia files including HeatA.mov 130. Whilst not
shown on the diagram, each node of any level can "contain"
multimedia files, not just the lowermost one. In reality however,
the node only contains a reference to the database entries that are
associated with the multimedia file or segment thereof.
[0132] The user of the present invention is able to associate with
the database entry one or more information branches 120.
Information branches are segments of knowledge tree 100. As
indicated in the diagram, information branch 120 begins with the
root node, "Olympic Sports" and proceeds down until "100M Mens".
During annotation, the user selects the lowermost classification
term, in the present case, 100M Mens 110, and drops it onto the
database entry. The database manager then associates that
multimedia file, in the present case, file 130, with the
information branch 120 such that during browsing of the database
when the user checks the "contents" of node 110, database entry
referring to file 130 is found.
[0133] One of the more powerful features of the invention is the
ability for users to provide annotation on many different levels
(of segmentation). FIG. 12 depicts the various levels available to
an annotator. The present invention provides for the annotation at
the repository level 140 wherein all multimedia files contained
within a certain repository will be branded with the annotations
provided at this level. The invention further provides for the
annotation at the "whole of work" level 142. The user may also take
individual segments 144 or even pull individual frames 146 from
video files, or alternatively the user may have still images needed
annotating which can be achieved at this frame level 146. Users of
the invention may also decide to annotate a region within the frame
148 by means of drawing a polygon around the region.
[0134] FIG. 12 also depicts the metadata types applicable at each
level of annotation. At the first level, that of the repository
140, the only metadata applicable is that of location,
specifically, URL's. The second level is the whole of work level
142. This is the level with the greatest amount of annotation
options available to the annotating user. In particular the
annotating user can associate (1) classification terms/information
branches, (2) keywords and (3) catalog entry metadata. At the
segment level 144, the annotating user can add start and stop time
information, duration information, classification terms/information
branches and finally, keyword information. The still frame level
146 and region within still frames 148 both have the capacity to
associate classification terms/information branches, keyword
information and information relating to the point in segment/time
the frame/region relates to.
[0135] FIG. 13 depicts the catalog entry fields available to the
user in one embodiment of the invention. These fields are only
applicable to the whole of work level 142 as depicted in FIG.
12.
[0136] FIG. 14 depicts the segmentation of movie or video files. In
this figure, video file 150 is segmented into segment A 152,
segment B 154 and segment E 156. Because the multimedia files
according to this embodiment of the invention are not required to
be duplicated or actually segmented but rather are virtually
segmented, the segments 152 through 156 can be overlapping in their
content. Indeed, the concept of virtual segmentation also provides
for a further additional benefit, deep linking, and the avoidance
of copyright infringement. The present invention provides that when
a user segments a video file into shorter segments which later are
sought to be retrieved, the user is initially presented with the
shorter segment and not any of the preceding material that would
have been viewed had the whole multimedia file been played to find
the desired content. The invention achieves this by automatically
populating the time in and time out fields of the segment. When
retrieved the database manager provides the user with a link that
takes the user to that point in time of the original video file
without requiring a second smaller copy of the segment of the file
to be resident in a repository.
[0137] A further feature of a preferred embodiment of the invention
is the feature of multiple levels of access to the database. FIG.
15 depicts the 4 roles currently provided for by the invention.
Further the table provided in FIG. 15 depicts the differences in
roles carried out by each type of user. In addition to these 4
levels of users, there are various levels provided for granting
differing levels of security or access to the administrator,
librarian, annotator and searcher roles. The security levels are
such that they can be applied so that users can be restricted to
certain repositories, or restricted in other ways, including
restrictions on functionality. For example, certain users may be
prevented from viewing certain material based on the user's
location in the world, place in an organisation or age. This may be
due to the operation of copyright or censorship laws by certain
states or jurisdictions. Further, in a situation where there are
multiple metadata databases in operation by various content
aggregators and owners across a network, or the internet, a user
may be permitted to log into certain databases, but not others.
[0138] FIG. 16 depicts the process carried out, according to one
embodiment of the invention, to virtually segment and annotate
multimedia files located on the network. The first step is to
determine whether the repository (the volume mounted on the
network) is known to the metadata database manager. If the volume
is registered then the user provides the multimedia file's path,
filename and short title to the metadata database manager. This
information is sufficient for a database entry to be created and
associated with the multimedia file in question. The database entry
can be annotated with any of the fields as permitted and depicted
in FIG. 12, including keywords, alphanumeric strings, and
information branches pruned from the databases' knowledge trees. A
user can takes matters further by virtually segmenting the larger
multimedia file into clips of shorter duration or even frames or
regions within a frame. In each of these cases the resultant
segments, frames, or regions within frames are associated with
distinct database entries (by the metadata database manager) to
that of the database entry that relates to the whole multimedia
file. Once separated from the original database entry, the
subsequent database entries can also be annotated with any of the
fields as permitted and depicted in FIG. 12, including keywords,
alphanumeric strings, and information branches pruned from the
databases' knowledge trees.
[0139] Thus, the user can use any or all of the information entered
by the annotating user to locate the desired multimedia file of
interest. This allows the user to conduct keyword searches over (1)
catalog entry fields which apply only to the whole of work level,
(2) classification terms/ information branches, and (3)
keywords.
[0140] FIG. 17 depicts the broad strategy used by a user to obtain
the multimedia file that they desire. According to the present
embodiment of the invention, there is provided an iterative process
whereby the user searches by utilising keywords or browses the
knowledge trees in succession until the entry corresponding to the
multimedia file in question is found within the retrieved results.
FIG. 18 depicts a similar process in algorithmic form. In the
process so described therein, the user first selects a node of the
knowledge tree or a keyword to conduct a first level search over
the entire database or databases, then determines whether the
content of interest is contained within the node or the results
returned by the search, if not, the user either browses the tree to
arrive at a different node, or conducts a further keyword search to
obtain a different set of search results, the process being
repeated until the multimedia file desired is located.
[0141] FIG. 19 depicts a hypothetical knowledge tree 160, of a
metadata database. The knowledge tree is comprised of a
hierarchical structure with "Hollywood" at its root. When the
knowledge trees are examined for the occurrence of the term
"Simpsons" the information branches 170 are obtained from the
knowledge tree 160. As can be seen in FIG. 20 (which is a depiction
of the individual information branches of the knowledge tree) the
Simpsons feature in many of the information branches of the
knowledge tree, at various levels.
[0142] When the information branches below are arranged such that
they intersect at the common classification term "Simpsons" and
visualised in 3 dimensions the structure depicted in panel 210 of
FIG. 22 is arrived at. FIG. 21 depicts 2 dimensional cross
sectional views of this three dimensional form. Slice 200 is the
slice made at the intersection, and thus only depicts one
information branch existing at that level. However if the user
takes a slice one level above the intersection 180 the slice 210 is
arrived at. Slice 210 depicts all of the information branches as
they branch out from "Simpsons". Slices 220 to 240 represent the
information branches at those levels above the intersection 180,
and slice 250, the one level below the intersection of the
classification term "Simpsons".
[0143] FIG. 22 depicts a sample screen shot according to one aspect
of the invention. The screen is divided into a number of different
areas or panels. Panel 200 comprises the knowledge tree view, which
is in the present case, a highly complex and populated knowledge
tree, panel 210, the intersected branch view, panel 220 the keyword
search panel, panel 230 the search results list view, panel 240 the
views selector and panel 250, the navigation panel. The intersected
tree view is comprised of three elements, the 3D structure 212,
vertical slider 216 and information branches 214 that exist at the
level set by the slider 216. The search result list view 230
provides a list of database entries 232. The embodiment of the
invention depicted in FIG. 16 is highly customisable. The user can
select whether to conduct a keyword search over the current node,
or whether to conduct the search over a larger portion of the tree
or even the whole tree by selecting the appropriate radio dialog
box in the keyword search panel 220. In addition the user can
restrict searches to classification terms/information branches,
catalog entries or keyword fields.
[0144] In FIG. 23, the user has taken the results provided by a
search for "Simpsons" as depicted in FIG. 16A and further queried
the metadata database. In the present case the user has taken the
intersected tree view and slid the slider 216 up one level which
revealed classification terms 214. The user has then selected
"animated" 218. By selecting "animated" 218, the user causes the
branch of the knowledge tree to be displayed in the knowledge tree
view 200. Further, all multimedia files associated with the node
"Animated" are displayed in the search results view.
[0145] FIG. 28 depicts various representations of knowledge trees
structures possible in accordance with the teaching of the present
invention. As hierarchical information can get difficult to
represent in a traditional tree view as depicted in FIG. 19,
different ways of displaying the data have been employed such that
no matter how complex a data set, there is firstly, a way of
visualising it, and secondly, a way to zoom in and traverse it.
[0146] FIG. 24 depicts the various different devices that are
encompassed by particular embodiments of the invention. In all of
these cases there is a metadata database and manager 260 and IP
network 270. The most usual device for the invention to be
implemented upon is what is known as a personal computer, which are
often sold in desktop and notebook configurations. The personal
computer can be a client in one of a number of ways. Computer 280
has installed upon it a web enabled executable program which acts
as database client and is capable of exchanging information with
the metadata database and metadata database manager 260. This
program would need to be installed by any user wishing to access
network multimedia files described in the metadata database. The
program can be developed into two or more versions. One version of
the software is accessible by all levels of users including
librarians and administrators and facilitates the management of
users and the maintenance of the database, this is called the thick
client. In one embodiment of the invention the thin client on the
other hand is designed only for the searcher user type and thus no
annotation or database management functions are included. The only
function that this version facilitates is retrieval of already
annotated multimedia files.
[0147] A further embodiment of the invention involves the database
being accessed through a common web browser such as Firefox, Safari
or Internet Explorer as depicted by computer 290 in FIG. 17. As web
browsers are bundled with computers pre-loaded, any user should be
able to access the metadata database and manager 260 through the
common HTML mediated interface. Both fully functional and search
only web interfaces can be developed to access metadata database
and manager 260, including those that incorporate AJAX
technology.
[0148] A similar approach can be taken with mobile telephones 300,
and personal media players (not shown). Most modern mobile
telephones and PDA's have the ability to load executable program
files upon them in the form of Java, Windows Mobile CE, Symbian,
and more recently Mac OS X executables. In such scenarios the
mobile telephone or PDA can be equated with computer 280 with
software loaded onto it. Equally however, most modern mobile
telephones, PDA's, and personal media players 310 such as the iPod
and PSP also feature HTML or web browsers. Thus they may be able to
access the database and manager 260 through a browser type
interface as in the case of computer 290.
[0149] Many household appliances that feature the ability to
connect to the internet/IP network and display multimedia files
would also be candidates for embedding within their firmware a
compatible metadata database client. FIG. 17 depicts an IP enabled
portable media player device 310, set top box device 320, and games
machine 330 as being suitable candidates for inclusion of the
database client in their firmware, or indeed, as in the case of
mobile telephones and some personal media players 310, through
incorporated web browsers.
[0150] Indeed the range of devices is really only limited by their
ability to access the internet/network and display multimedia
files. Other devices that may be suitable would include video
conferencing equipment, routers and network attached storage
devices, in car navigation display units, internet kiosks and in
car/plane entertainment systems. Further, the search and retrieval
function depicted in FIGS. 22 and 23 need not be implemented in any
client device. Indeed, in one embodiment of the invention, there is
provided an open API for interaction with the metadata database
such that users may develop their own mash up applications that
utilise the power of the metadata database. For example, with
respect to the in-car navigation system, it will generally be aware
of its location via GPS. Further, the structure of a metadata
database, and importantly, the knowledge tree contained therein
might be such that it has branches that correspond to different GPS
locations. As the user passes through areas in which there are
associated database entries (given the GPS co-ordinates) certain
videos and/or audio can be displayed on its display and through its
speaker. Such technology would be ideal for the provision of
location specific advertising. A further embodiment not involving
search and retrieval functions may involve a set top box that has
loaded within it firmware that recognises the type of program being
watched (previously annotated) and which searches for similar
content over the internet for the viewer whenever a button is
pushed on the remote, or which searches for advertisements that
have been programmed to appear in conjunction with certain
programs, eg. BMW advertisements during "Top Gear" a motoring
television program.
[0151] FIG. 25 depicts a many metadata database implementation of
the invention. FIG. 25 depicts two standalone computers 350, 370
containing metadata databases 380, 390, connected through the
internet 360, which is in turn in connected to two other tagmotion
databases 400, 410. In one embodiment of the invention there is
provision for the databases to exchange information by means of
exporting database entries and dictionaries of classification
terms. This can be done manually or automatically between users in
a group where permission exists between them for this to happen. In
this way, they create a group taxonomy where each benefits from the
efforts of the other, enriching the metadata including the `global
(or collective) dictionary terms`, thus making the search process
more and more intuitive and above all, precise. Indeed, depending
on whether the user had the necessary privileges, the user can log
into any one of the databases for search and retrieval. If found,
the user might decide to export the entry into their own database,
or the dictionary terms into their own dictionary. Then, the user
may choose to import metadata such as catalog entries, dictionary
(classification) terms and storyboards (containing metadata for
segments of video files, for example) into their own database.
These imports can be from other personal databases or from the
collective one. In this way, one's own personal database or `view`
can be enriched as desired, by exploiting the annotations of
others.
[0152] In one embodiment of the invention it would be possible to
provide a system whereby all users are granted librarian access
which means that anyone could add whole of work database entries
and additions to the global dictionary from which classification
terms appearing on the knowledge trees are selected. In such an
embodiment the system would provide for an online "wiki" in which
the experience of the group using the wiki is affected by each of
the users participation. In such a scenario, potentially involving
millions of users as in the case of youtube.com or myspace.com
websites, content would be rapidly categorized according to
potentially millions of different points of view, thereby creating
the rich, metadata-dense, database which in turn results in the
formation of a detailed knowledge tree in which it is likely you
would find multimedia files of interest.
[0153] FIG. 26 depicts a similar array of metadata databases as
depicted in FIG. 19. In particular there are databases 410, 420 and
430 which are all connected the internet and through it to
advertising hub 430. Advertising hub 430 is in turn comprised of
database 440 and database manager 450. In this embodiment of the
invention the metadata from participating metadata databases is
uploaded into the database 440 where their database entries are
analysed and information branches extracted therefrom. Advertisers
460 are then able to log into the hub 430 through the manager 450
and select individual keywords, information branches and/or nodes
on knowledge trees for purchase or bidding in relation to which
their advertisements would be played.
[0154] FIGS. 27, 29, 30 and 31 depict the interaction a user has
with an advertiser who has purchased either keywords or information
branches used in the metadata databases' entries. FIG. 27 depicts
the same scenario as depicted in FIG. 22, that of a user who has
just conducted a keyword search using the word "Simpsons". In the
present case the user has not yet selected an individual
information branch and thus the basis for serving the
advertisements would be the word "Simspons" or the existence of one
or more particular information branches. As semantic meaning is
caught up in an information branch they can be used to provide more
targeted and contextually relevant advertisements.
[0155] FIG. 29 depicts the next step in which the user uses slider
216 to go up one level/slice and selects the "animated" information
branch. As the user has selected an individual information branch
this can be used as the basis to serve up further contextually
related advertisement 480.
[0156] FIG. 29 also highlights another potential advertising
possibility in that positions on the metadata database's knowledge
trees can be sold for advertisers to upload content in respect of
their goods and services. In FIG. 29 advertisement 490 has been
inserted at Hollywood:TV:Comedy:Animated, the active node. Such
positions may be used to feature content including promotional or
traditional advertising video, or it may simply act as a link to a
website or other position on the knowledge tree where the
advertiser may have more of a presence. Indeed, such an approach is
likely to facilitate the development of metadata databases in which
advertising and promotional content is the sole content of the
database. In such case the knowledge tree and associated search
methods could result in a video yellow pages in which advertisers
place their advertisements at particular places on the tree as
opposed to being arranged alphabetically by reference to the goods
and services they supply.
[0157] FIG. 30 depicts a screenshot of an embodiment of the
invention where the user has selected one of the Simpsons videos
from FIG. 29 and requested it to play. In this embodiment the
search list pane is replaced with a multimedia player and the
reproduction of the metadata recorded in respect of the multimedia
file. When the single database entry is selected, the intersected
tree view changes to reflect the 3D structure of only those
information branches associated with that particular database
entry. Further, advertisements 500 are served up with the
multimedia file as determined by the remaining information branches
and keywords contained within the various metadata database fields
available for that type of entry. As there are fewer branches at
this point, the advertisements will be more targeted than those
when there were many more information branches associated with a
given view. However, notwithstanding that there are fewer
intersecting branches, there are still more than one, and therefore
no one information branch of the knowledge trees is shown in the
knowledge tree pane 200, but rather the whole tree is shown with
highlighted "hits" either being information branches of the
knowledge trees or nodes in which there are database entries that
have the searched for keyword in their fields. FIG. 31 differs from
FIG. 23 as the user selected one particular information branch 510,
which then has the effect of highlighting that information branch
of the knowledge tree in tree view pane 200 as depicted. Also as
the intersected tree view has been used to select one particular
information branch, the same branch of the knowledge tree is
highlighted in the pane above. Further, as only one branch has been
selected, the semantic meaning of the branch should provide for the
serving of even more context specific advertisements.
[0158] In an alternative embodiment of the invention, rather than
the user navigating through a 3D representation of the knowledge
tree with the user pausing to see what is contained at each node of
the tree, the user in this alternative embodiment actually
navigates a 3D space in which the content rather than metadata
descriptors is actually traversed. In some embodiments you may have
intersected information branches being represented by the content
associated with the branches. In other embodiments it may be the
structure of the whole tree which is represented by content (that
is playing or has representative video frames depicted) by which a
user navigates themselves. Different audio or video content will
come into view as the user moves through the landscape. That is, it
will have the effect of flying through the content in a 3
dimensional space. As this happens, the content being displayed
changes according to underlying information branches (that are
hidden from view), and what they represent. This effect may also be
enhanced by spatial audio controls whereby the sound pertaining to
a particular file becomes louder as the user `flies` closer to it,
and softer as he starts to `fly` away and/or the speed of video
playback becomes slower as the user `flies` closer to it, and
faster as he starts to `fly away`. This embodiment of the invention
could be provided in the form of an executable program, an AJAX or
similar website, or indeed, inside existing three dimensional
environments such as Second Life developed by Linden Labs.
[0159] The person skilled in the art will appreciate that whilst
this invention has been described in respect of audio, images and
movie files located on the internet, the invention is equally
applicable to other items of interest located on the internet
including documents including HTML documents, information contained
in documents, executables, indeed anything of interest that can be
located by a URL and filename.
* * * * *