U.S. patent application number 11/179511 was filed with the patent office on 2005-11-24 for method of searching multimedia data.
This patent application is currently assigned to LG ELECTRONICS INC.. Invention is credited to Kim, Hyeon Jun, Lee, Jin Soo.
Application Number | 20050262067 11/179511 |
Document ID | / |
Family ID | 26634660 |
Filed Date | 2005-11-24 |
United States Patent
Application |
20050262067 |
Kind Code |
A1 |
Lee, Jin Soo ; et
al. |
November 24, 2005 |
Method of searching multimedia data
Abstract
A method of searching multimedia data is disclosed in which a
search for an image can re-performed by automatically updating
weights of features and/or weights of feature elements in the
respective feature in an image.
Inventors: |
Lee, Jin Soo; (Seoul,
KR) ; Kim, Hyeon Jun; (Kyonggi-do, KR) |
Correspondence
Address: |
FLESHNER & KIM, LLP
P.O. Box 221200
Chantilly
VA
20153-1200
US
|
Assignee: |
LG ELECTRONICS INC.
|
Family ID: |
26634660 |
Appl. No.: |
11/179511 |
Filed: |
July 13, 2005 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
11179511 |
Jul 13, 2005 |
|
|
|
09495250 |
Jan 31, 2000 |
|
|
|
Current U.S.
Class: |
1/1 ;
707/999.003; 707/E17.021; 707/E17.025 |
Current CPC
Class: |
G06F 16/5838 20190101;
Y10S 707/99945 20130101; G06F 16/5862 20190101; G06K 9/6212
20130101; Y10S 707/99943 20130101; Y10S 707/99944 20130101; G06F
16/40 20190101; G06K 9/4652 20130101; Y10S 707/99942 20130101 |
Class at
Publication: |
707/003 |
International
Class: |
G06F 017/30 |
Foreign Application Data
Date |
Code |
Application Number |
Feb 1, 1999 |
KR |
3182/1999 |
Feb 1, 1999 |
KR |
3183/1999 |
Claims
1-20. (canceled)
21. A method of constructing a multimedia data comprising:
incorporating a feature information including feature and feature
elements of an image; and incorporating a weight information
including weight information of said features and weight
information of said feature elements.
22. A method of claim 21, wherein the feature and the feature
elements are represented by an image characteristic structure
comprising: a global information which represents a feature of a
whole image; and a spatial information which represents a feature
of an image region, wherein the image characteristic structure
further comprises a weight information which represents the
importance of the global information and the spatial
information.
23. A method of searching a desired image from reference images
using visual feature information comprising: generating visual
feature information with respect to an interested region in each
reference image, the visual feature information utilized in an
image search system, including: providing a global feature
information; incorporating a local visual feature information,
wherein the local visual feature information includes a set of
visual feature information assigned to each sub region in each
image: incorporating a weight information of the global visual
feature information; and incorporating an importance information
assigned to each sub region of the interested region in each image;
measuring the similarities of reference images based on the global
and the local visual feature information of reference images; and
selecting an image based on the measured similarities.
24. The method of claim 23, wherein the global and local visual
feature information are plurally provided.
25. The method of claim 23, wherein the visual feature includes
color, texture, and shape.
26. The method of claim 25, wherein the global and the local visual
feature information include color histogram, region-representing
color, and texture histogram.
27. The method of claim 23, wherein the importance information
represents whether the visual feature information assigned to
corresponding sub region of the interested region of the interested
region in the image is utilized in image search,
28. The method of claim 27, wherein search performance is
determined based on weight information and the importance
information.
Description
BACKGROUND OF THE INVENTION
[0001] 1. Field of the Invention
[0002] The present invention relates to a method of searching
multimedia data, and more particularly to a method of searching
multimedia data more accurately by utilizing user feedback.
[0003] 2. Background of the Related Art
[0004] Recently, technology for digital image signal processing has
been greatly developed and has been applied in various fields. For
example, the digital image signal processing technology may be used
in a search system for automatically editing only the face of a
specific character in a moving picture file of a movie or drama, in
a security system for permitting access to only persons registered
in the system, or in a search system for searching a particular
data from an image or video detected by a detection system. The
performance of such systems basically depend on the accuracy and
speed of detecting or searching the desired object. Accordingly,
various image searching methods have been proposed in the related
art.
[0005] An image search system which detects a degree of similarity
with an image to be searched utilizing features such as color,
texture or shape is disclosed in U.S. Pat. No. 5,579,471 entitled
"An image query system and method," Depending upon the image to be
searched, the importance or a feature may vary and within one
particular feature such as the color, the importance of a feature
element such as the red or green color may also vary. However, the
above searching system does not take into consideration the
different importance of features or feature elements for each image
to be searched.
[0006] In another searching method entitled "Virage image search
engine" (www.virage.com), a user directly inputs the level of
importance for features such as the color, texture and shape by
assigning weight values. Although an image may be searched
according to an importance or a feature using this method, it may
be difficult for a user to determine the weights of features.
[0007] Therefore, Yong Rui in "Relevance feedback techniques in
interactive" SPIE Vol. 3312, discloses a method in which images
similar to a reference image are found and the importance of
features or weight for features are automatically obtained by
calculating the similarities among the found images. However, the
weight importance information is not maintained after a search for
a specific image is finished and must be calculated for each image
search, even for a same image.
[0008] In the image search and browsing system or the video (moving
image) search and browsing system of the related art, information
which describes a particular feature of an image or video data is
utilized to perform a more effective search or browsing of the
multimedia data or example, in the image query system, an image may
be divided into a plurality of regions and a representative color
of each region may be utilized as a feature information of the
image, or a whole color histogram of the image may be utilized as a
feature information. Thereafter, two images are compared to
calculate a similarity based upon the feature information and a
determination is made whether the two images are similar.
[0009] Therefore, the image search methods in the related art may
utilize weights of features such as color, texture, or shape.
However, weights of feature elements are not taken into
consideration. Accordingly, the image search methods in related art
has the limitations in intellectually training and developing
weights for searching, resulting in relatively longer searching
period to obtain search results and a deterioration of the
reliability of the search results.
SUMMARY OF THE INVENTION
[0010] Accordingly, an object of the present invention is to solve
at least the problems and disadvantages of the related art.
[0011] An object of the present invention is to provide a method of
searching multimedia data by automatically updating weights of
features included in a specified image and/or weights of feature
elements, and by applying the updated weights to search for the
specified object.
[0012] Another object of the present invention is to provide a
method or searching multimedia data which constructs image
characteristics corresponding to the types of features included in
a specified image by analyzing and classifying the judgement
standards applied when the user searches the image, and adjusts the
feature information set by taking into consideration weights of the
features and weights of feature elements during a following image
search.
[0013] Still another object of the present invention is to provide
a feature structure to be included in a multimedia data to
effectively search an image.
[0014] Additional advantages, objects, and features of the
invention will be set forth in part in the description which
follows and in part will become apparent to those having ordinary
skill in the art upon examination of the following or may be
learned from practice of L the invention. The objects and is
advantages of the invention may be realized and attained as
particularly pointed out in the appended claims.
[0015] To achieve the objects and in accordance with the purposes
of the invention, as embodied and broadly described herein, a
method of searching multimedia data in a multimedia data search
system comprises searching for a reference multimedia data selected
by a user; receiving user feedback of relevance information for the
searched multimedia data; determining importance of respective
feature elements of features included in the multimedia data
according to the relevance information; re-performing the search
for the reference multimedia data by updating the importance of
said respective elements if the user requests an additional search;
and updating previous importance to new importance obtained and
maintaining the updated importance degrees.
[0016] In another embodiment of the present invention, a method of
searching multimedia data in a multimedia data search system
comprises receiving an inquiry into previously searched multimedia
data; analyzing a judgement standard for the multimedia data
searched during the inquiry; constructing image characteristics
using at least one feature included in the multimedia data using an
analysis result of the judgement standard; and adjusting importance
of the image characteristics and re-performing a search of the
multimedia data if a user requests an additional search.
[0017] The present invention also provides a feature structure of
multimedia data comprising a first information representing a
feature of the multimedia data; a second information representing a
regional feature of the multimedia data; and a third information
representing importance of the first and second information.
BRIEF DESCRIPTION OF THE DRAWINGS
[0018] The invention will be described in detail with reference to
the following drawings in which like reference numerals refer to
like elements wherein:
[0019] FIG. 1 shows features in an image by a histogram;
[0020] FIG. 2 shows an image represented by a local grids;
[0021] FIG. 3 is a flowchart of a multimedia data search process
according to a first embodiment of the present invention;
[0022] FIGS. 4 and 5 are flowcharts of a multimedia data search
process of FIG. 2, where an initial search does not result in a
desired image;
[0023] FIG. 6 is a basic structure of image characteristics for an
image search according to a second embodiment of the present
invention;
[0024] FIG. 7 is image characteristics of FIG. 6 wherein the
feature information is constructed using a color and texture;
and
[0025] FIGS. 8 to 11 are different embodiments of the image
characteristics,
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0026] Reference will now be made in detail to the preferred
embodiments of the present invention, examples of which are
illustrated in the accompanying drawings.
[0027] In a general image search system, features of an image such
as color, texture, and shape may be represented by a histogram as
shown in FIG. 1. Particularly, FIG. 1 is a color histogram of an
image by which the colors in the image is grouped into 24 color
elements. By adjusting or altering the weights of each color
element, the degree of importance of each color element or the
extent that a color element affects a search may be
represented.
[0028] FIG. 2 is an image represented by local grids. Particularly,
an image is divided into n*m grid regions and each grid region may
be represented either by a regional color histogram or by a color
representing the and region. Here, the degree of importance of each
grid region or the extent that a grid region affects a search may
be represented by assigning weights to each of n*m grid regions.
Also, a predetermined threshold value determined by the system may
be utilized as a cap such that certain grid regions has no
influence in the search. Namely, if a grid region has a weight
which exceeds the predetermined threshold value, the grid region is
utilized in the search. Otherwise, the and region is processed as a
"Don't care" region which does not affect the image search.
[0029] The image search process will next be explained with
reference to FIGS. 3 to 5.
[0030] FIG. 3 is a flowchart of a multimedia data search process
when a plurality of reference images are selected and weights of
the respective types or elements of features included in a
specified image are assigned or updated according to the first
embodiment of the present invention. Referring to FIG. 3, a user
selects a plurality of reference images (step S301) representing
the specified image to be searched. Thereafter, the system judges
and determines the importance of the feature elements included in
the reference images.
[0031] For example, the system measures the similarities of the
features included in the reference images (step S302), and
determines the weights of each feature according to the measured
similarities of the features (step S303). Also, the system measures
the similarities of the feature elements in each feature included
in the reference images (step S304), and determines the weights of
feature elements in each respective feature according to the
measured similarities of the feature elements (step S305).
[0032] Accordingly, if a user requests an additional search, the
system re-performs the image search by using the importance, i.e.
the weights of the features and the weights of the feature elements
in the respective features (step S306). At this time, the system
may use either the weights of the features or the weights of the
feature elements, or both.
[0033] Particularly, the weights of the features and feature
elements of respective features are determined as follows. When the
user selects a plurality of the reference images, the search system
determines weights of the features and the weights of the feature
elements in the respective features by measuring the similarities
of features of the selected reference image list and the
similarities or the feature elements in the respective features.
The weights of the features are calculated by Equations 1a, 1b, and
1c below. 1 Weight_k = Cont ( k ) Cont ( ALL ) [ 1 a ] Cont ( k ) =
i = 1 n - 1 j = i + 1 n Sim ( i , j , k ) [ 1 b ] Cont ( ALL ) = k
= 1 m Cont ( k ) [ 1 c ]
[0034] In the above equations, n denotes the reference number, m
denotes the number of features used for measuring the similarity,
Weight_k denotes the weight of the k-th feature, Simi(i,j,k)
denotes the similarity between the I-th reference image and the
j-th reference image when the k-th feature is used, and Cont(k)
denotes how much the k-th feature contributes to raise the
similarity. Generally, the weights of the respective features
Weight_k increases as the similarities rises since the similarities
among the reference images are calculated based upon the respective
features and the feature having the highest similarity acts as the
most important factor.
[0035] The weights of the feature elements are determined by
Equations 2a and 2b using the similarities of the feature elements
in the respective features or the reference images.
Weight of element i: w.sub.i=af.sub.I(i) [2a]
[0036] [2b] Similarlity of element i in reference images 2 f l ( i
) = pm i qv l
[0037] In the above equations, the values of a, p, and q denote
constants, m.sub.i denotes an average of an element i in the
reference image list, and v, denotes the distribution of an element
i in the reference image list. According to Equations 2a and 2b,
the weight w.sub.i of a feature element is inversely proportional
to the distribution of the corresponding feature element and is
proportional to an average value of the corresponding feature
element component. Therefore, a feature element having a large
average value acts as an important factor even if the distribution
of the corresponding feature element is wide.
[0038] FIG. 4 is a flowchart showing a multimedia data search
process for when a plurality of reference images are and are not
selected. Generally, if the number of reference images is not
plural, the image search is performed using features with
equivalent weights. However, if the search does not result in a
desired image, other reference images similar to the specified
image to be searched are selected and added to a reference image
list. Then, the weights of the respective features and/or feature
elements are updated using the reference image list.
[0039] Referring to FIG. 4, when a user selects a plurality of
reference images (steps S401 and S402) the process (steps
S404-S408) is the same as described with reference to steps
S302-S306 of FIG. 3. Thus, an explanation will be omitted.
Thereafter, if the user is satisfied with the result of the image
search (step S408), the search operation terminates. However, if
the user is not satisfied with the search result, the user selects
other reference image(s) similar to the specified image to be
searched among the resultant images of the search (step S411). On
the other hand, when a user selects only one reference image, the
image search is performed using features with weights as assigned
(step S403). Generally, if the reference image is selected for the
first time, the search is performed using features with equal
weights assigned. If the user is not satisfied with the search
result, the user selects other reference image(s) similar to the
specified image to be searched among the resultant images of the
search (step S411).
[0040] Accordingly, the system adds the selected other reference
image(s) to an initial reference image list managed by the system
(step S412). Here, the initial reference image list includes the
reference image(s) selected in step S401. Thereafter, the system
measures the similarities of the respective features and/or feature
elements in the selected reference images (step S413), and
determines the weights of the respective features and/or feature
elements in the selected reference image by Equations 1a 1c, 2a and
2b (step S414). Thus, the system updates the weights of the
respective features and/or feature elements, and re-performs the
image search utilizing the updated weights (step S415).
[0041] FIG. 5 is a flowchart showing another multimedia data search
process for when a plurality of reference images are and are not
selected. Generally, if the number of reference images is not
plural, the image search is performed using features with
equivalent weights. However, if the search does not result in a
desired image, other reference images, both similar and dissimilar
to the specified image to be searched are selected and respectively
added to a reference image list or a dissimilar image list. Then,
the weights of the respective features and/or feature elements are
updated using the reference image list and the dissimilar image
list.
[0042] Referring to FIG. 5, when a user selects a plurality or
reference images (steps S501 and S502), the process (steps
S504-S50B) is same as described with reference to steps S302-S306
of FIG. 3. Thus, an explanation will be omitted. Thereafter, if the
user is satisfied with the result of the image search (step S509),
the search operation terminates. However, if the user is not
satisfied with the search result, the user selects other reference
image(s) similar and dissimilar to the specified image to be
searched among the resultant images of the search. On the other
hand, when a user selects only one reference image, the image
search is performed using features with weights as assigned (step
S503). Here, if the reference image is selected for the first time,
the search is performed using features with equal weights assigned.
If the user is not satisfied with the search result, the user then
selects other reference image(s) similar and dissimilar to the
specified image to be searched among the resultant images of the
search.
[0043] Accordingly, the system adds the selected similar image(s)
to an initial reference image list managed by the system (step
S511) and adds the dissimilar image(s) to an initial dissimilar
image list (step S312). Here, the initial reference image list
includes the reference image(s) selected in step S501. Thereafter,
the system measures the similarities of the respective features
and/or feature elements in the images included in the reference
image list (step S513), and measures the similarities of the
respective features and/or feature elements in the images included
in the dissimilar image list (step S514).
[0044] Using Equations 1a.about.1c, 2a and 2b, the system
determines the weights of the respective features and/or feature
elements using the images included in the reference image list and
the dissimilar image list (step S515). Thus, the system updates the
weights of the respective features and/or feature elements, and
re-performs the image search utilizing the updated weights (step
S516). Particularly, the weights of the features by the similarly
measurement of the features and/or feature elements in the images
included in the reference/dissimilar image lists are calculated by
Equations 3a.about.3d. 3 Weight_k = ( a .times. Weight l_k )
.times. ( b Weight R_k ) [ 3 a ] Weight R_k = ( Weight l_k ) = Cont
( k ) Cont ( ALL ) [ 3 b ] Cont ( k ) = l = 1 n - 1 j = i + 1 n Sim
( i , j , k ) [ 3 c ] Cont ( ALL ) = k = 1 m Cont ( k ) [ 3 d ]
[0045] In the above equations, n denotes the reference number in
the reference image list or the dissimilar image list, m denotes
the number of features used or the similarity measurement, Weight_k
denotes the final weight of the k-th feature, Simi(i,j,k) denotes
the similarity between the I-th reference image and the j-th
reference image when the k-th feature is used, Cont(k) denotes how
much the k-th feature contribute to raise the similarity,
Weight.sub.I.sub..sub.--.sub.k denotes the weight of the k-th
feature in the reference image list, and
Weight.sub.R.sub..sub.--.sub.k denotes the weight of the k-th
feature in the dissimilar image list. Generally, the similarities
of the images included in the reference image list and the
dissimilar image list are calculated respectively. As a result, the
weights of the respective features Weight_k increase as the
similarities of the images included in the reference image list
rises, while the weights decrease as the similarities in the images
included in the dissimilar image list rises.
[0046] Also, after measuring the similarities of the feature
elements in the respective features of the images included in the
reference/dissimilar image list, the weights of the feature
elements of the respective features are determined by Equations
4a.about.4b.
Weight of an element i: w.sub.i=af.sub.I(i)+bf.sub.R(i) [4a]
[0047] [4b] Similarity of an element i in the reference images: 4 f
l ( i ) = pm i qv i
[0048] In the above equations, f.sub.R(I)=pm.sub.i.times.v.sub.i,
denotes the dissimilarity of an element i in the images included in
the dissimilar image list, the values of a, b, p, and q denote
constants, m.sub.i denotes an average of the element i in the
images included in the corresponding (reference and dissimilar)
image lists, and v.sub.i denotes the distribution of the element i
in the images included in the corresponding (reference and
dissimilar) image lists. Generally, the similarities of the images
included in the reference image list and the dissimilar image list
are calculated respectively. As a result, the weights of the
respective feature elements Weight_k increase as the similarities
of the images included in the reference image list rises, while the
weights decrease as the similarities in the images included in the
dissimilar image list rises.
[0049] If the weights of the features and the weights of the
feature elements are determined as above, the similarities will be
calculated using Equation 5 during the image search. 5 A - k = 0 n
W k i = 0 km w k_i Diff ( F k_i , p , q ) [ 5 ]
[0050] Here, A is s constant, Diff(F.sub.k.sub..sub.--.sub.lp,q)
denotes the difference between the I-th elements of the k-th
feature of the image p and image q, w.sub.k.sub..sub.--.sub.i
denotes the weight of the 1-th feature element of the k-th feature,
w.sub.k denotes the weight of the k-th feature, n denotes the
number of features, and km denotes the number of feature elements
of the k-th feature. Thus, the difference is obtained by
multiplying the feature difference value of the respective image,
the feature element weight of the respective feature, and the
weight of the respective feature. Also, the similarity is obtained
by subtracting the difference from the constant.
[0051] As described above in reference to FIGS. 3.about.5, the
system automatically determines and updates both the feature
element weights of respective features and the weights of the
features of the image to be searched when the user searches an
image, Therefore, a rapid and effective search can be
performed.
[0052] Nevertheless, if the user wishes to perform a further search
of the specified image after viewing a previously searched result,
the user may raise and enter various kinds of queries to the search
system. Table 1 shows examples of queries by users, and Table 2
shows the feature information required according to the type of
query when colors and textures are used as the basic features of an
image.
1 TABLE 1 Query Type 1 What color does image have as a whole? 2
Does any portion of the image have a certain color feature? 3 About
what degree does the portion of the image have a certain color
feature? 4 What texture does the image as a whole? 5 Does any
portion of the image have a certain texture feature? 6 About what
degree does the portion of the image have a certain texture
feature? 7 Does the image have a certain color and texture feature
or have any portion having such features? 8 Does any portion of the
image have a certain color and texture feature? 9 About what degree
does the portion of the image have a certain color and texture
feature? 10 What color does the image have at a specified position?
11 What texture does the image have at a specified position? 12
What color and texture does the image have at a specified
position?
[0053]
2 TABLE 2 Query Type Main Feature Type 1 What color does image have
as a Global color whole? information 2 Does any portion have a
certain Local color color feature? information 3 What degree does
the portion Local color have a certain color feature information 4
What texture does the image as a Global texture whole? formation 5
Does any portion have a certain Local texture texture feature?
information 6 what degree does the portion Local texture have a
certain texture feature? information 7 Does the image have a
certain Global color texture color and texture feature or
information have any portion having such features? 8 Does any
portion have a certain Local color texture color and texture
feature? information 9 What degree does the portion local color
texture have a certain color and texture information feature? 10
What color does the image have Local color position at a specified
position? information 11 What texture does the image have local
texture at a specified position? position information 12 What color
and texture does the local color texture image have at a specified
position information position?
[0054] In Table 2, 12 query types are presented and to satisfy the
characteristics with respect to such queries, the search system
should have at least the following 8 image feature information.
[0055] The first image feature information is a global color
information which represents the color feature of the whole image.
A color histogram may be an example of the global color
information. The second image feature information is a global
texture information which represents the texture feature of the
whole image. A texture histogram may be an example of the global
texture information. The feature information of the color and the
texture of the whole image may be represented by a combination of
the global color information and the global texture
information.
[0056] The third image feature information is a local color
information which represents the color feature of a region, i.e.
grid region, in the image. A representative color for each local
grid may be an example of the local color information.
Alternatively, the weights of color elements obtained from the
global color information may be utilized as the local color in
formation.
[0057] The fourth image feature element is a local texture
information representing the texture feature of a grid region in
the image. A representative texture for each grid may be an example
of the local texture information. Alternatively, the weights of
texture elements obtained from the global texture information may
be utilized as the local texture information.
[0058] The fifth image feature element is a local color and texture
information which represents the color and texture features or a
grid region in the image. A representative color and texture for
each grid may be an example of the local color and texture
information. Alternatively, the weights of color and texture
elements respectively obtained from the global color information
and the global texture information may be utilized as the local
color and texture information.
[0059] The sixth image feature element is a local color position
information which represents a color feature in a region at a
particular position of the image. A color local grid feature may be
an example of the local color position information. The seventh
image feature element is a local texture position information which
represents a texture feature in a region at a particular position
of the image. A texture local grid feature may be an example of the
local texture position information. Also, the specified color and
texture feature in a region at the particular position of the image
can be represented as a combination of the sixth and seventh
information.
[0060] The eighth image feature information is a local color and
texture information which represents a specified color and texture
feature in a region at a particular position of the image. A color
and texture local grid feature may be an example of the local color
and texture position information.
[0061] The system can perform an effective search by constructing a
set of feature information, i.e. image characteristics, as
described above using the analyzed results based upon the contents
of the queries and add element weights to the constructed features.
Thus, if a user requests a search, the system adjusts the
importance of the image characteristics, i.e. the weights of the
features and/or feature element, and performs the image search.
[0062] The search method using a reference multimedia data
determines a multimedia data having the highest similarity to the
reference multimedia data by adjusting the weights of the features
and/or feature elements of the respective features included in the
multimedia data. Here, the weight adjustment of the feature and/or
feature elements of the respective features can be performed using
one of a direct adjusting method by the user, an automatic
adjusting method by the system, or an adjusting method using the
relevance information (i.e., positive and negative information) fed
back to the system by the user. The meanings of the features as
described above will now be explained in detail.
[0063] First, a color histogram represents the color distribution
in an image. Similarly, the texture histogram represents the
texture distribution in an image.
[0064] The color image grid represents the color information of a
grid region generated by dividing an image into n*m grid
regions.
[0065] The texture image grid represents the texture information of
a grid region generated by dividing an image into n*m grid regions.
The color-texture joint local grid represents the color texture
information of a grid region generated by dividing an image into
n*m grid regions.
[0066] FIG. 6 shows the structure of texture description which can
be constructed in consideration of the query types and relevance
feedbacks of the user 601. The structure comprises comprises a
global information 602 which represents a feature of a whole image,
a spatial information 603 which represents a feature of an image
region, and weight information 604 which represents the importance
of the constructed features 602 and 603. The global information
includes a global feature descriptor 605 of the whole image, and an
element weight descriptor 606 of the feature elements of the global
feature descriptor of the whole image. The spatial information 603
includes a spatial feature descriptor 607 of an image region, and a
position weight descriptor 608 of the image region.
[0067] The global information 602 of the whole image and the
spatial information 603 of the image region can be constructed by a
selective combination of features included in the image such as the
color, texture, and shape. Here, the possible combinations of the
basic features can be obtained using Equation 6 below, where n
denotes the number of the basic features. 6 k = 1 n nCk [ 6 ]
[0068] Thus, the number of feature types obtained by Equation 6
applies to local positions and for global information, since there
n number of basic features, the total number of feature types can
be obtained by Equation 7. 7 k = 1 n nCk + 2 n . [ 7 ]
[0069] The present invention will be explained utilizing two basic
features of color and texture. In such case, the total number of
feature types required by the system would be 3+2*2=7. However, if
the feature of shape is added, the total number of required feature
type would be 7:2*3=13.
[0070] FIG. 7 shows image characteristics constructed using the
features of color and texture. Referring to FIG. 7, the relevance
feedback image(s) 701 used for adjusting the weights of the image
is features according to the user feedback includes global color
information 702a of the whole image, global texture information
702b of the whole image, spatial information 703a of image regions,
spatial color information 703b of image regions, and weight
descriptor 704 of the global informations 702a and 702b, and of the
spatial informations 703a and 703b.
[0071] In FIG. 7, four feature informations 702a, 702b, 703a and
703b are used, requiring four weights. Particularly, the global
color information 702a includes a global color histogram 705
representing the color feature information of the whole image, and
an element weight descriptor 706 of the respective bins of the
global color histogram. The global texture information 702b
includes a global texture histogram 707 representing the texture
information of the whole image, and an element weight descriptor
708 of the respective bins of the global texture histogram.
[0072] Also, the spatial color information 703a includes a color
image grid 709, and a position weight descriptor 710 of the color
image grid. The spatial texture information 703b includes a texture
image grid 711, and a position weight descriptor 712 or the texture
image grid.
[0073] The color histogram 705 is used as a feature information of
the whole image and the weight of each color element in the color
histogram 705 are represented by the element weight descriptor 706.
Also, the global texture histogram 707 is used as another feature
information of the whole image and the weight of each texture
element in the global texture histogram 707 are represented by the
element weight descriptor 708.
[0074] Moreover, the color image grid 709 is used as a feature
information of the image regions and the weight or each grid
position in the color image grid 709 is represented by the position
weight descriptor 710. Similarly, the texture image grid 711 is
used as another feature information of the image regions, and the
weight of each grid position in the texture image grid 711 is
represented by the position weight descriptor 712.
[0075] As shown in FIG. 7, an image characteristic structure having
four feature information was explained in order to satisfy the
twelve query types in Table 2. However, all nine feature types for
the twelve query types is not necessary. For example, if a
color-texture joint local grid is used as a feature, the local
color, local texture, local color and texture, local position
color, local position texture and local position color and texture
can be obtained from the color-texture joint local grid.
[0076] Furthermore, in the image characteristic structure of FIG.
7, the feature weights are represented the same level as the
feature information, and the feature element weights are
represented in a level below the respective feature information.
However, image characteristics may be constructed alternatively is
with the feature weights in a level below the respective feature
information as shown in FIG. 8. For example, assuming that a global
color information 801 is the feature information, the global color
information 801 includes 2 global color feature 802, and weights
803. Here the weights 803 is composed of feature weights 804
corresponding to the global color feature and feature element
weights 805.
[0077] FIG. 9 shows another embodiment of the image characteristics
used for adjusting the weights of the image features according to
the user feedback. In this image characteristic, all information
related to weight characteristics are grouped into a set and
represented separately.
[0078] Referring to FIG. 9, the image feature structure 902, i.e.
the reference feedback, for adjusting the weights of the image
features when searching the image 901 includes image
characteristics 903 and the weight characteristics 904. The image
characteristics 903 include global information 905, local
information 906, and local position information 907. The weight
characteristics 904 include feature weights 908 and feature element
weights 909. Moreover, the global information 905 includes n
feature units 910, the local information 906 includes a number of
feature units 911 equivalent to a sum of the number or features and
possible combinations of the features, and the local position
information 907 also includes n feature units.
[0079] FIG. 10 shows another example of the image data structure of
FIG. 9 when the image information includes two basic features or
color and texture. Particularly, the image characteristics 1001
includes global information 1002, local information 1003 and local
positional information 1004. The global information 1002 includes a
global color feature unit 1005 and a global texture feature unit
1007. The local information 1003 includes a local color feature
unit 1009, a local texture feature unit 1010 and a local color and
texture feature unit 1011. The local position information 1004
includes a local position color feature unit 1013 and a local
position texture feature unit 1014.
[0080] Moreover, the global color feature unit 1005 is represented
by a global color histogram 1006, the global texture feature unit
1007 is represented by a global texture histogram 1008, and the
local color feature unit 1009 and the local position color feature
unit 1013 are represented by a color image grid 1012. Also, the
local texture feature unit 1010 and the local position texture
feature unit 1014 are represented by a texture image grid 1015.
Finally, the local color and texture feature unit 1011 is
represented by both the color image grid 1012 and the texture image
grid 1015.
[0081] Therefore, the query types in Table 2 can be satisfied by
constructing image characteristics of the seven features as
described above, and the weights are updated by adjusting the
weights in the feature weights and the feature element weights as
shown in FIG. 11. Referring to FIG. 11, the image feature structure
1102, i.e. the reference feedback, for adjusting the weights of the
image features when searching the image 1101 includes image
characteristics 1103 and weight characteristics 1104.
[0082] Particularly, the image characteristics 1103 includes global
information 1105, local information 1106 and local positional
information 1107. The weight characteristics 1104 includes feature
weights 1108 and feature element weights 1109. Here, the global
information 1105 includes a global color feature unit 1110 and a
global texture feature unit 1111. The local information 1106
includes a local color feature unit 1112, a local texture feature
unit 1113 and a local color and texture feature unit 1114. The
local position information 1109 includes a local position color
feature unit 1115 and a local position texture feature unit
1116.
[0083] As described above, according to the present invention, the
system analyzes all possible queries of the user, and provides
minimum image characteristics which satisfy all judgement standards
during the image search. Accordingly, a rapid and effective image
search can be performed by adjusting the weights of the features
and feature elements to reflect the user feedbacks.
[0084] The foregoing embodiments are merely exemplary and are not
to be construed as limiting the present invention. The present
teachings can be readily applied to other types of apparatuses. The
description of the present invention is intended to be
illustrative, and not to limit the scope of the claims. Many
alternatives, modifications, and variations will be apparent to
those skilled in the art.
* * * * *