U.S. patent application number 14/447276 was filed with the patent office on 2014-11-20 for method, system and controller for searching a database containing data items.
The applicant listed for this patent is Someones Group Intellectual Property Holdings PTY LTD ACN 131 335 325. Invention is credited to CHRISTOPHER GLENDON BATES, KEITH DAVID DEVERELL, BRETT JAMES GRONOW, JONATHAN DAVID PAK, DAVID PETER WOLF.
Application Number | 20140344303 14/447276 |
Document ID | / |
Family ID | 43731864 |
Filed Date | 2014-11-20 |
United States Patent
Application |
20140344303 |
Kind Code |
A1 |
GRONOW; BRETT JAMES ; et
al. |
November 20, 2014 |
METHOD, SYSTEM AND CONTROLLER FOR SEARCHING A DATABASE CONTAINING
DATA ITEMS
Abstract
A method, system and controller is provided for searching a
database containing data items with a user via a user inter the
method comprising the steps of specifying an initial search subset
of some or all of the data items in the database (1201);
identifying representatives of each of a number of data categories
in the search subset (1203); displaying the representatives on the
user interface (1204); selecting one or more of the representatives
(1205); specifying a refined search subset of data items in the
search subset within the data categories corresponding to the
selected representatives (1207); and repeating the steps of
identifying and displaying representatives for the refined search
subset.
Inventors: |
GRONOW; BRETT JAMES;
(Fitzroy, AU) ; DEVERELL; KEITH DAVID;
(Collingwood, AU) ; PAK; JONATHAN DAVID; (Fitzroy
North, AU) ; BATES; CHRISTOPHER GLENDON; (Box Hill
North, AU) ; WOLF; DAVID PETER; (Northcote VIC,
AU) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Someones Group Intellectual Property Holdings PTY LTD ACN 131 335
325 |
Fitzroy |
|
AU |
|
|
Family ID: |
43731864 |
Appl. No.: |
14/447276 |
Filed: |
July 30, 2014 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
13394918 |
Mar 8, 2012 |
8832134 |
|
|
PCT/AU2010/001176 |
Sep 10, 2010 |
|
|
|
14447276 |
|
|
|
|
Current U.S.
Class: |
707/766 |
Current CPC
Class: |
G06F 16/54 20190101;
G06F 16/3328 20190101 |
Class at
Publication: |
707/766 |
International
Class: |
G06F 17/30 20060101
G06F017/30 |
Foreign Application Data
Date |
Code |
Application Number |
Sep 11, 2009 |
AU |
2009904400 |
Claims
1. A method of searching a database containing data items with a
user via a user interface, the method comprising the steps of
specifying an initial search subset of some or all of the data
items in the database; identifying representatives of each of a
number of data categories in the search subset; displaying the
representatives on the user interface; selecting one or more of the
representatives; specifying a refined search subset of data items
in the search subset within the data categories corresponding to
the representatives selected; and repeating the steps of
identifying and displaying representatives for the refined search
subset.
Description
[0001] This application is a continuation of U.S. patent
application Ser. No. 13/394,918, filed Mar. 8, 2012, which is a 371
of International Appln. No. PCT/AU2010/001176, filed Sep. 10, 2010,
which claims priority to Australian Appln. No. 2009904400, filed
Sep. 11, 2009, the contents of all incorporated herein by
reference.
FIELD
[0002] The present invention relates to a computerized method,
system and controller for searching a database of data items,
particularly but not limited to media items such as images, music,
movies, games or books.
BACKGROUND TO THE INVENTION
[0003] Searching through databases of media items such as stock
photographs is a task often undertaken both in the media industry
and by consumers generally. Websites such as gettyimages.com
images.google.com and others typically provide searching methods
that involve descriptor based indexing of image content. The
content descriptors typically relate to aspects such as concept,
emotion, location or to the characteristics of people in the image.
The user specifies values of one or more content descriptors
typically using checkboxes in drop-down lists and the image search
engine then searches the image database index for images matching
the user-specified values. Matching images are then typically
displayed, accompanied with technical information such as
resolution or origin, in a scrollable window. Usually a first page
shows the closest matches, which may involve some assessment of
additional native properties such as colour balance, and multiple
further pages contain less close matches. The user inspects the
many images and may refine the search by specifying further content
descriptor values, or by choosing one reference image as an
example.
[0004] Although an initial search using descriptors produces many
potentially relevant images, refinement of the search to select out
images of more interest is often frustrating. Further, the need to
scroll through and individually examine a mixture of a large number
of images and text is tiring and tedious.
[0005] There is therefore a need to provide improved database
searching techniques that enable more efficient and user-friendly
identification of useful data items.
[0006] The inventors of the current invention have found a much
improved method of assisting the user to locate a data item of
interest, by making advances in the use of selection and display of
representative data items.
SUMMARY OF THE INVENTION
[0007] According to a first aspect of the invention there is
provided a method of searching a database containing data items
with a user via a user interface, comprising the steps of
[0008] specifying an initial search subset of some or all of the
data items in the database;
[0009] identifying representatives of each of a number of data
categories in the search subset;
[0010] displaying the representatives on the user interface;
[0011] selecting one or more of the representatives;
[0012] specifying a refined search subset of data items in the
search subset within the data categories corresponding to the
selected representatives; and
[0013] repeating the steps of identifying and displaying
representatives for the refined search subset.
[0014] In one embodiment, the step of specifying an initial search
subset comprises receiving a restriction criterion from the user
and defining the initial subset as data items satisfying the
restriction criterion. The restriction criterion may be a value of
a search term.
[0015] In one embodiment, the data items are associated with
descriptor values in a multidimensional descriptor space and the
data categories in the search subset are sectors in the descriptor
space. The multidimensional descriptor space may be defined having
regard to obtaining categories adapted to the specified search
subset, such as from a descriptor table of stored values in a
descriptor space of larger dimensionality by an algorithm that
identifies a set of appropriate descriptors with reduced
dimensionality for the search subset. The reduced dimensionality is
chosen having regard to the number of categories.
[0016] In an embodiment, an origin of the descriptor space is near
a centre of the search subset so that the descriptor coordinates of
the search subset have positive and negative values; and
[0017] at least one of the categories is a sector in the
multidimensional descriptor space that is greater in a positive or
negative direction in one dimension compared to the other
dimensions.
[0018] In one embodiment, the number of categories is chosen so
that all the representatives can be displayed to the user at once.
The step of displaying the representatives of the refined search
set may comprise displaying the representatives of the refined
search subset next to the representatives of the initial search
subset.
[0019] In one embodiment, the process of specifying, identifying,
displaying and selecting is adapted to repeat until all the
categories have no more than one member.
[0020] In one embodiment, the data items comprise digital
media.
[0021] According to a second aspect of the invention there is
provided a database searching system for interactively searching a
database containing data items with a user, comprising
[0022] a user interface with a display;
[0023] an initial search subset specifier for specifying an initial
search subset of some or all of the data items in the database;
[0024] a representative identifier for identifying representatives
of each of a number of data categories in the search subset;
[0025] a representative display controller for displaying the
representatives on the display;
[0026] a representative selector for selecting one or more of the
representatives;
[0027] a refined search subset specifier for specifying a refined
search subset of data items in the search subset within the data
categories corresponding to the selected representatives;
[0028] a repeater for repeating the steps of identifying and
displaying representatives for the refined search subset.
[0029] According to a third aspect of the invention there is
provided a database searching controller for interactively
searching a database containing data items with a user via a user
interface having a display, comprising
[0030] an initial search subset specifier for specifying an initial
search subset of some or all of the data items in the database;
[0031] a representative identifier for identifying representatives
of each of a number of data categories in the search subset;
[0032] a representative display controller for displaying the
representatives on the display;
[0033] a representative selector for selecting one or more of the
representatives;
[0034] a refined search subset specifier for specifying a refined
search subset of data items in the search subset within the data
categories corresponding to the selected representatives;
[0035] a repeater for repeating the steps of identifying and
displaying representatives for the refined search subset.
[0036] According to a fifth aspect of the invention there is
provided a computer readable medium comprising the program code of
the fourth aspect of the invention.
[0037] According to a sixth aspect of the invention there is
provided a data signal comprising the computer program code of the
fourth aspect of the invention.
BRIEF DESCRIPTION OF DRAWINGS
[0038] FIG. 1 is a block diagram of an example database searching
system of the current invention;
[0039] FIG. 2 shows a flow diagram for the method of database
searching of an embodiment of the current invention.
[0040] FIGS. 3-6 show displays in a search session in a database of
images according to an embodiment of the current invention.
DETAILED DESCRIPTION OF EMBODIMENTS
[0041] Referring to the drawings, there is shown various aspects of
a database searching system, controller and method according to an
embodiment of the invention.
[0042] The system may take a number of different forms. In a first
form, a standalone system is provided wherein all or most
components required are present in a local user operable computer
and interface.
[0043] In a second form, a distributed architecture is provided
wherein some of the components required for implementing the system
are present in local user operable computer and interface and some
of the components are located remotely relative to the user. For
example, a "thick client" architecture may be used wherein part of
the search and/or display control is executed on a local user
operable computer and interface and part is executed remotely, such
as by a server; or a "thin client" architecture may be used wherein
most of the search and display control is executed remotely such as
by a server and a local machine is used only for the user
interface.
[0044] A typical web-based implementation involves a conventional
personal computer with display, keyboard and mouse or touch
sensitive screen, implementing the database searching controller as
a JavaScript or XML program which may be downloaded from a website.
The database will typically be stored remotely from the user and be
interrogated by the database searching controller.
[0045] Referring now to FIG. 1, the core components at the broadest
level are the database searching system 10 comprising a user
interface 20 and an database searching controller 30, communicating
with a data memory 40 which may be located both locally and
remotely. In one embodiment data memory 40 includes a data items
database 420, descriptor table 422 and restriction tag table
421.
[0046] Components of the user interface 20 include a display 200
such as an LCD monitor and a position selector 210 such as a mouse
for the user to indicate a position on the display 200 with at
least one selector button such as a left or right mouse button to
enable the user to select the position and perform other input
functions.
[0047] The database searching controller 30 is in data
communication with the user interface 20 and data memory 40 and
typically contains or is implemented on a processor that processes
the search in accordance with user input and outputs on display
200. Herein the term "processor" is used to refer generically to
any device that can perform this function and may include: a
microprocessor, microcontroller, programmable logic device or other
computational device, a general purpose computer (e.g. a PC) or a
server.
[0048] For media applications, data items database 420 is a large
database of media data items, accompanied as is standard in the art
by a restriction tag table 421 typically containing textual or
binary content-based information in relation to each media item,
but may also contain some native media information such as media
type (e.g. colour/B&W). Data memory 40 also contains descriptor
table 422 of derived descriptors for each media item as described
herein which typically originate from native audiovisual
descriptors but may also originate from content-based information.
Tables 421 and 422 are computed during a database build and may be
remotely located from media item database 420, particularly in the
case where the invention is being exercised by a party other that
the owner of the data items database 420. Tables 421 and 422 may be
ordered or indexed in whole or part such as with a binary tree to
speed identification of items in database 420 satisfying values or
value ranges in the descriptors or restriction tags, as is known in
the art.
[0049] Database searching controller 30 has an initial search
subset specifier 310 for specifying an initial search subset
typically by receiving a restriction criterion from the user via
the user interface. In simple implementations, initial search
subset specifier 310 may be trivially adapted only to specify the
entire data items in database 420 or a fixed subset thereof.
Database searching controller 30 also has a representative
identifier 320 for identifying a set of representative data items
in the search subset 420, each being representative of one of a
number of categories in the search subset, a representative display
controller 321 for displaying the representatives on display 200,
and representative selector 330 for selecting one or more of the
representatives, typically selected by the user via position
selector 210. Also provided in database searching controller 30 is
a refined search subset specifier 331 for specifying a refined
search subset of data items in the search subset within the data
categories corresponding to the selected representatives, and a
repeater 332 for transferring the refined search subset to
representative identifier 320 and for displaying the newly
identified representatives via representative display controller
321.
[0050] Referring now to FIGS. 2 to 6, one embodiment of the method
aspect of the invention is depicted in a flow diagram in FIG. 2 and
an example display layout is shown in steps of the method of the
invention in FIGS. 3 to 6. In step 1201, the user specifies an
initial search subset by typing a restriction criterion such as the
search term "fruit" in restriction criterion window 62 and clicking
on the "search" button 64b. Initial search subset specifier 310
accesses restriction tag table 421 to identify the data items
satisfying the restriction criterion, in this case images
containing fruit. This is the initial search subset. In step 1202,
20 data categories are found that are appropriate to the search
subset, using single value decomposition to a ten-dimensional
descriptor space from the 35 dimensional space represented in the
descriptor table 422. Details of these processes are provided in
the examples below. In step 1203 and 1204, representatives of each
data category are identified and displayed, also described in more
detail below. FIG. 3 shows the 20 representatives displayed is a 5
by 4 grid 64 on the left side of display 60. At the top left of
each image is a number which informs the user how many of the data
items in the search subset are in each corresponding data category.
In step 1205, the user selects one or more of the representatives
as worthy of further search, by clicking with the position selector
210 and confirming in step 1206 that the selected items are to be
the selected representatives by clicking on the "search button"
64b. In the session shown in FIG. 3, the top right hand
representative 64a is selected. In step 1207, a refined search
subset is defined as those members of the current search subset
that are contained within the data categories corresponding to the
selected representatives. The system then repeats the step of
finding appropriate categories by SVD from the descriptor table
values for the refined search subset to a new 10-dimensional
descriptor space appropriate to the refined search subset.
Representatives for each of the 20 categories are then identified
again and displayed on the right side 65 of the display. The top
left hand representative is the same as the previously selected
representative but this may not necessarily be the case. At the
bottom of the screen is a "breadcrumb" trail of the search
refinement. Clicking on the last box in the trail will revert the
search subset back to the previous search subset. Further refined
subsets are selected from representative 65a in FIG. 5 (the
pomegranate) to produce four representatives of a two dimensional
descriptor space) and as shown in FIG. 6 the pomegranate 66a, being
a representative of a category having 3 members is selected again
and the final refined search subset 67, having only 3 members, is
displayed.
[0051] As the hierarchical search proceeds, the breadcrumb trail 63
extends and the series at representatives of the search subsets are
added to a band which disappears to the left. The user can jump
back to any search subset in the sequence by clicking on the
relevant box in the breadcrumb trail, and can add further
restriction to the search subsets by editing the restriction
criterion window 62.
Example of Image-Based Descriptor Table
[0052] It is contemplated that in typical embodiments of the
invention, the multidimensional descriptor space appropriate to the
data categories is defined from a descriptor table 422 of stored
values in a descriptor space of larger dimensionality.
[0053] For application to images, there are many available image
descriptors that can be used in the invention to generate the
descriptor table 422. Examples include colour histogram, SIFT edge
detection, SURF edge detection, Canny edge detection, Tamura
texture features, and Gabor texture features.
[0054] The inventors' currently implemented embodiment for images
such as photographs or movie stills uses, for the descriptor table,
a set of 35 descriptors derived during a database build using
single value decomposition (SVD) of the 336 descriptors in the
combination of the uncompressed CEDD [Ref 1] and FTCH [Ref 2]
descriptor sets.
Example of Finding Data Categories for the Search Subset
[0055] In the embodiment illustrated in FIGS. 3 to 6,
representatives of 20 data categories in the initial search subset
of all data items having a tag of "fruit" are displayed in a 5 by 4
grid 64 on a left section of display 60. In a preliminary step, a
new 35 dimensional descriptor space appropriate to the search
subset is computed using single value decomposition (SVD) from the
descriptor table restricted to the search subset. The 35 dimensions
are reduced to 10 by selecting the 10 first dimensions, which are
the largest contributors to the variance of the 35 descriptors for
the search subset, and are therefore a good start to identity
categories adapted to the search subset. The 20 categories are then
defined as sectors in the reduced 10 dimensional descriptor space.
The output of the SVD algorithm is scaled so that an origin of the
descriptor space near the centre of the search subset so that the
descriptor coordinates of the search subset have positive and
negative values. A simple way of achieving this is to define each
coordinate dimension relative to a roughly calculated centre by
subtracting the average of the maximum and minimum values of that
dimension in the search subset. This allows definition of the
sectors as regions where one coordinate is greater in a positive or
negative direction than the other coordinates yielding two sectors
per dimension.
[0056] In terms of mathematical symbols, the positive sector of
points x.ident.(x.sub.1,x.sub.2..x.sub.n) for the kth dimension of
an n-dimensional descriptor space (e.g. of the new reduced 10
dimensional space) is defined by
S.sup.+.sub.k={all points x such that x.sub.k.gtoreq.x.sub.i and
|x.sub.k|.gtoreq.|x.sub.i| for all i.noteq.k}
and the negative sector is defined by
S.sup.-.sub.k={all points x such that x.sub.k.ltoreq.x.sub.i and
|x.sub.k|.gtoreq.|x.sub.i| for all i.noteq.k}
[0057] This way of defining sectors runs efficiently on a standard
database via the use of a B+tree.
Example of Identifying Representatives of Data Categories
[0058] For the categories found in the above example, a
representative data item can be identified for each category by a
variety of methods. An example of a rapidly executing method on a
standard database is to identity the maximum and minimum values. In
other words, the representative for the positive sector for the kth
dimension is defined by the data item in sector S with the largest
(most positive) x.sub.k, and the representative for the negative
sector for the kth dimension is defined by the data item in sector
S.sup.-.sub.k with the lowest (most negative) x.sub.k.
Example of Displaying the Representatives
[0059] For the image example detailed in this specification,
display of the representatives can be conveniently ordered by
placing the representatives for S.sup.+.sub.k and S.sup.-.sub.k at
opposing positions on the display grid for example, S.sup.+.sub.1
and S.sup.-.sub.2 are placed in the top left (row 1, column 1) and
bottom right (5, 4) positions, and S.sub.+2 and S.sup.-.sub.2 are
placed in adjacent opposing positions (2,1) and (4,4) and so
on.
[0060] As seen in FIGS. 5 and 6, as the search set is refined to
include progressively fewer data items, the number of
representatives reduces, due to some sectors being empty. Display
of the reduced number of images may be accomplished by increasing
the image size of each representative as shown in FIG. 5 and
providing irregular divisions as shown is FIG. 6 for prime numbers
of representatives that cannot be neatly placed in a grid.
Example of Audio Descriptor Space
[0061] The figures show an example of an image database, but the
invention is equally applicable to audio data such as music.
[0062] Examples of audio descriptors that can be used in the
invention include Mel-Frequency Cepstrum, Zero-Crossings, Spectral
Analysis, Spectral Centroid, Spectral Rolloff, Linear Predictive
Coding Coefficients, and Beat Histograms.
[0063] The inventors' currently implemented embodiment for music
and audio data items uses, for the multidimensional descriptor
space, a combination of a large variety of the known audio
descriptors, such as are implemented on jAudio descriptor
computation software [Ref 3], found on the internet at
http://jmir.sourceforge.net/jAudio.html. A large number of
available descriptors (in this case 330) are first tested on the
data set to remove any (typically about 30) that tend to cause
errors in the SVD algorithm for certain data types. The robust
remaining set (about 300) is then subjected to the SVD algorithm as
in the image example to provide a set of 35 descriptors to be
stored in descriptor table 422. Computation of 10-dimensional and
lower descriptor spaces in the course of the search proceed as
described above.
Example of Audio Display
[0064] In the case of photographs, the nature of the item is
apparent to the user in the display of the item. Clearly in the
case of audio, adaptations are desirable in representative display
controller 321 in the display of the audio items to convey some
information on the nature of the audio, usually in addition to
visual aspects such as the supplied music track artwork or album
cover. One approach is to condense the descriptor coordinates
further from 35 to 3 using SVD, and to represent the 3 condensed
coordinates as RGB values in a colour space. The user will came to
understand which colours correspond to which type of sounds.
Tag-Based Descriptor Applications
[0065] Although in the above examples the database is built from
descriptors that are calculated from the native audiovisual data,
such as colour balance or edge information for images and time
series descriptors in the case of music, it is possible to include
search terms in the original descriptor set, which can also be
condensed together with the native descriptors in the SVD
algorithm.
[0066] Depending on the database and application there may also be
situations where the set of descriptors in the descriptor table 422
is calculated entirely from content-based descriptors. For example,
when searching for a movie or more especially a book it may be
found that information such as actors, director, author, plot,
violence level etc is more useful in providing recommendations to
the user than the native descriptors. There may also be a wide
range of binary descriptors such as, in the case of video games,
whether the game has action or strategy elements, puzzles, elves or
guns. These content tags can all be treated the same way as the
native disrupters and reduced to a multidimensional set of a
manageable number of continuous coordinates using an algorithm such
as SVD.
[0067] It will be appreciated that while single value decomposition
from a large number of descriptors has been described as one method
of calculating a manageable descriptor space for the descriptor
table 422 and the multidimensional descriptor space of reduced
dimensionality for the initial search subset and refined search
subsets, the invention encompasses any way of categorizing and
identifying representatives however produced. In particular,
another clustering algorithm such as NIPALS could be used instead
of SVD. Furthermore, the descriptor space is not restricted to
Cartesian spaces, and may for example be a tree of nodes, the
coordinates being any way of representing position in the tree, or
any other structure. Further, although an automatic classification
adapting to the contents of the search subsets is exemplified, the
invention in its broadest form encompasses rigid non-adaptive
categorizations also.
[0068] In the claims and the description, the term "initial search
subset" extends to any search subset which precedes and is refined
to the "refined search subset", and is not required to be an
absolutely first search subset specified by the user in a searching
session.
[0069] Persons skilled in the art will also appreciate that many
other variations may be made to the invention without departing
from the scope of the invention.
[0070] In the claims which follow and in the preceding description
of the invention, except where the context requires otherwise due
to express language or necessary implication, the word "comprise"
or variations such as "comprises" or "comprising" is used in an
inclusive sense, i.e. to specify the presence of the stated
features but not to preclude the presence or addition of further
features in various embodiments of the invention.
[0071] It is to be understood that, if any prior art publication is
referred to herein, such reference does not constitute an admission
that the publication forms a part of the common general knowledge
in the art, in Australia or any other country.
REFERENCES
[0072] 1. S. A. Chatzichristofis and Y. S. Boutalis, "CEDD: COLOR
AND EDGE DIRECTIVITY DESCRIPTOR--A COMPACT DESCRIPTOR FOR IMAGE
INDEXING AND RETRIEVAL.", <<6th International Conference in
advanced research on Computer Vision Systems ICVS
2008>>Proceedings: Lecture Notes in Computer Science (LNCS)
pp. 312-322, May 12 to May 15, 2008, Santorini, Greece [0073] 2. S.
A. Chatzichristofis and Y. S. Boutalis, "FCTH: FUZZY COLOR AND
TEXTURE HISTOGRAM- A LOW LEVEL FEATURE FOR ACCURATE IMAGE
RETRIEVAL", <<9th International Workshop on Image Analysis
for Multimedia Interactive Services", Proceedings: IEEE Computer
Society pp. 191-196, May 7 to May 9, 2008, Klagenfurt, Austria
[0074] 3. McEnnis, D., C. McKay, I. Fujinaga, and P. Depalle. 2005.
jAudio: A feature extraction library Proceedings of the
International Conference on Music Information Retrieval. 600-3.
* * * * *
References