U.S. patent application number 12/815777 was filed with the patent office on 2010-10-07 for method for dominant color setting of video region and data structure and method of confidence measure extraction.
This patent application is currently assigned to LG Electronics Inc.. Invention is credited to Hyeon Jun KIM.
Application Number | 20100254600 12/815777 |
Document ID | / |
Family ID | 34554905 |
Filed Date | 2010-10-07 |
United States Patent
Application |
20100254600 |
Kind Code |
A1 |
KIM; Hyeon Jun |
October 7, 2010 |
METHOD FOR DOMINANT COLOR SETTING OF VIDEO REGION AND DATA
STRUCTURE AND METHOD OF CONFIDENCE MEASURE EXTRACTION
Abstract
A method for a dominant color setting of a video region and a
data structure and a method of a confidence measure extraction are
disclosed. The video region dominant color setting method is
characterized in that a region dominant color descriptor is
expressed by the number of dominant colors with respect to a
certain region, a dominant color expressed, a frequency that the
dominant color appears, and an accuracy of a color value
representing the region in a region dominant color based on various
region dominant color extraction methods, for thereby expressing a
region dominant color using a plurality of colors with respect to a
region dominant color value and a confidence value of a region
dominant color information based on various region dominant color
feature extracting methods.
Inventors: |
KIM; Hyeon Jun; (Sungnam,
KR) |
Correspondence
Address: |
KED & ASSOCIATES, LLP
P.O. Box 221200
Chantilly
VA
20153-1200
US
|
Assignee: |
LG Electronics Inc.
|
Family ID: |
34554905 |
Appl. No.: |
12/815777 |
Filed: |
June 15, 2010 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
11133212 |
May 20, 2005 |
7760935 |
|
|
12815777 |
|
|
|
|
09609392 |
Jul 3, 2000 |
7417640 |
|
|
11133212 |
|
|
|
|
09239527 |
Jan 29, 1999 |
6445818 |
|
|
09609392 |
|
|
|
|
Current U.S.
Class: |
382/165 |
Current CPC
Class: |
G06F 16/785 20190101;
G06K 9/4652 20130101; Y10S 707/99945 20130101 |
Class at
Publication: |
382/165 |
International
Class: |
G06K 9/00 20060101
G06K009/00 |
Foreign Application Data
Date |
Code |
Application Number |
Jul 3, 1999 |
KR |
26784/1999 |
Claims
1. A system for searching multimedia data in a multimedia device
using a region dominant color descriptor, the system being
configured to: extract a region dominant color descriptor from
query multimedia data, the region dominant color descriptor
including a coherent degree of pixels corresponding to a dominant
color; compare the extracted region dominant color descriptor with
at least one stored region dominant color descriptor; search
multimedia data based on the coherent degree; obtain a searched
multimedia data for sorting the searched multimedia data; and sort
the searched multimedia data based on an input indicating a sorting
of the searched multimedia data.
2. The system of claim 1, wherein the region dominant color
descriptor further includes a region dominant color, a number
quantifying region dominant colors, and a frequency of the region
dominant color.
3. The system of claim 2, wherein the frequency of the region
dominant color is determined based on pixels corresponding to a
dominant color.
4. The system of claim 1, wherein the system is further configured
to: store the region dominant color descriptor.
5. The system of claim 1, wherein the region dominant color
descriptor further includes an accuracy of the region dominant
color indicating a degree of confidence of the region dominant
color descriptor.
6. The system of claim 1, wherein the system is further configured
to extract the region dominant color descriptor based on at least
one of an average-color method or a histogram method.
7. A system for searching multimedia data in a multimedia device
using a region dominant color descriptor, the system comprising: a
processor that extracts a region dominant color descriptor from
query multimedia data, the region dominant color descriptor
including a coherent degree of pixels corresponding to a dominant
color, compares the extracted region dominant color descriptor with
at least one stored region dominant color descriptor, searches
multimedia data based on the coherent degree, obtains a searched
multimedia data for sorting the searched multimedia data, and sorts
the searched multimedia data based on an input indicating a sorting
of the searched multimedia data.
8. The system of claim 7, wherein the region dominant color
descriptor further includes a region dominant color, a number
quantifying region dominant colors, and a frequency of the region
dominant color.
9. The system of claim 8, wherein the frequency of the region
dominant color is determined based on pixels corresponding to a
dominant color.
10. The system of claim 7, wherein the processor is further
configured to: store the region dominant color descriptor.
11. The system of claim 7, wherein the region dominant color
descriptor further includes an accuracy of the region dominant
color indicating a degree of confidence of the region dominant
color descriptor.
12. The system of claim 7, wherein the processor is further
configured to extract the region dominant color descriptor based on
at least one of an average-color method or a histogram method.
13. A method for searching multimedia data in a multimedia device
using a region dominant color descriptor, the method comprising:
extracting a region dominant color descriptor from query multimedia
data, the region dominant color descriptor including a coherent
degree of pixels corresponding to a dominant color; comparing the
extracted region dominant color descriptor with at least one stored
region dominant color descriptor; searching multimedia data based
on the coherent degree; obtaining a searched multimedia data for
sorting the searched multimedia data; and sorting the searched
multimedia data based on an input indicating a sorting of the
searched multimedia data.
14. The method of claim 13, wherein the region dominant color
descriptor further includes a region dominant color, a number
quantifying region dominant colors, and a frequency of the region
dominant color.
15. The method of claim 14, wherein the frequency of the region
dominant color is determined based on pixels corresponding to a
dominant color.
16. The method of claim 13, further comprising: storing the region
dominant color descriptor.
17. The method of claim 13, wherein the region dominant color
descriptor further includes an accuracy of the region dominant
color indicating a degree of confidence of the region dominant
color descriptor.
18. The method of claim 13, further comprising: extracting the
region dominant color descriptor based on at least one of an
average-color method or a histogram method.
Description
BACKGROUND OF THE INVENTION
[0001] 1. Field of the Invention
[0002] The present invention relates to a dominant color feature
description used in a content-based multimedia data retrieval
system, and in particular to a method for setting-up a video region
dominant color a data structure therefor, and a method for
extracting a confidence measure, which are capable of expressing an
object and a color of a ROI (Region Of Interest) in a video during
a multimedia indexing operation.
[0003] 2. Description of the Background Art
[0004] In a multi-media search system, there are various methods
for expressing a color feature of an object and a ROI of a video in
a multi-media search system. The above-described methods are
applied differently in accordance with each system.
[0005] There are various methods for expressing a dominant color,
such as a method for using an average color value of a region, a
method for expressing the most frequently appearing color, a method
for expressing n-number of the most frequently appearing colors, a
method for using a color appearing in a region predetermined by
threshold of P % or above, and a method using a color
histogram.
[0006] Each of the above-described conventional methods has its own
advantages and disadvantages. For example, the method of using the
histogram has an advantage to describe color information in detail.
However, it also has some problems in that the histogram has
relatively large size of data and some colors represented by
corresponding histogram bins can be considered as they have
unnecessary region dominant color values with respect to those
colors occupying relatively small regions.
[0007] In the case that a region dominant color value is expressed
by one average value, there are advantages in that it is a
compressed data descriptor and used for pre-filtering in a
content-based searching. However, in the case that the region
colors are formed in various colors, it is impossible to express
the dominant color feature accurately.
[0008] Recently, a data structure for extracting the region
dominant color is being standardized. However, if a unique method
for the extraction of the region dominant color is not standardized
and only data structure is standardized, it is impossible to
maintain a compatibility of the data built in each system where a
plurality of systems are used.
[0009] In addition, even when extracting the dominant color values
by the same method, it is hard to achieve a reliable performance in
every case.
[0010] For example, beside the problems presented when the average
color is used as a dominant color, when the histogram is used to
express the dominant color feature, the performance depends on the
number of histogram bins, namely, the number of color levels.
[0011] If there are too large number of bins, the region color is
unnecessarily expressed by too many colors for thereby decreasing a
search performance, and when the region color is expressed by too
few colors with a small number of bins, the region formed of
various colors is not properly expressed, so that the search
performance is degraded.
SUMMARY OF THE INVENTION
[0012] Accordingly, it is an object of the present invention to
provide a method for setting-up a dominant color of a video region
which is capable of expressing a region dominant color using a
plurality of colors with respect to a region dominant color value
and a confidence value of a region dominant color information based
on various region dominant color feature extraction methods.
[0013] It is another object of a present invention to provide a
data structure for the dominant color setting of a video
region.
[0014] It is still another object of the present invention is to
provide a method for extracting a confidence measure wherein the
dominant color setting of a video region according to the present
invention.
[0015] To achieve the above objects, a video region dominant color
descriptor is provided to characterize the number of dominant
colors, dominant colors, the frequency per dominant color
respectively with respect to a certain region, and the confidence
measure of the dominant color values and the frequencies extracted
based on various region dominant color extraction methods.
[0016] Additional advantages, objects and features of the present
invention will become more apparent from the description which
follows.
BRIEF DESCRIPTION OF THE DRAWINGS
[0017] The present invention will become better understood from the
and the accompanying drawings which are given by way of
illustration only, and thus are not limitative of the present
invention, and wherein:
[0018] FIG. 1 is a flow chart illustrating a region dominant color
setting method according to the present invention;
[0019] FIG. 2 is a flow chart illustrating a descriptor search
method using a region dominant color settlement according to the
present invention; and
[0020] FIG. 3 is a block diagram illustrating an interoperability
maintaining method between different systems using region dominant
color extraction description data according to the present
invention.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0021] According to one feature of the present invention, the
expression method of the region dominant color extraction method is
formulated based on an extraction method type, a pre-processing
description, a frequency condition description, color space
description, a color sub-space description, a quantization
description, a color clustering description, etc. for thereby
maintaining an interoperability between different systems.
[0022] According to another feature of the present invention, the
similarity between a dominant color and the similar color with the
dominant color, a coherency of the color with respect to a color
given, a difference between the dominant color value and the
accurate value of the color when the color is considered as a
certain color, a size of the region which covers the dominant color
in an image region, and the positions of each color pixel in the
region are adopted in order to calculate the confidence measure, so
that it is possible to compare the region dominant colors values
based on different feature extractions.
[0023] In addition, by expressing a confidence value for the entire
region dominant colors and/or each color, it is possible to obtain
a descriptor that describes the more accurate region dominant
colors.
[0024] As shown in FIG. 1, the video region dominant color setting
method according to the present invention includes a step for
extracting a region R from a visual data (video and/or images), a
step for setting a dominant color descriptor (DCD) with respect to
the region provided, and a step for storing a region descriptor
with respect to the region dominant color descriptor and the region
information.
[0025] The DCD is described by the number N of the dominant colors
of the color descriptor with respect to the region given, a certain
dominant color Ci described by a color information (e.g. r, g, b
components, etc.) and a frequency Pi which describes the degree
that the dominant color appears, and a CM (Confidence Measure) of
the color descriptor value.
[0026] As shown in FIG. 2, the descriptor search method using a
region dominant color includes a step for selecting a region by a
user and extracting a region descriptor corresponding thereto, and
a step for extracting a dominant color descriptor value with
respect to a corresponding region and comparing the extracted
dominant color descriptor value with each of all stored region
dominant color descriptors.
[0027] As shown in FIG. 3, in the interoperability maintaining
method between different systems using formalized data for a region
dominant color extraction method, a region dominant color
descriptor (DCD) with respect to a region descriptor R of each
system A and B is obtained, and the region dominant color
extraction method is formalized, and the region dominant color
descriptor and the description of region dominant color extraction
method are converted into a sharing data format, and then a
comparison search is performed with respect thereto.
[0028] The formalized data structure for description of the region
dominant color extraction method includes an extraction method type
for extracting a region dominant color, a pre-processing
description for describing a filtering method of a certain region
when obtaining the region dominant color value, a frequency
condition description type for describing a condition of a
frequency of a dominant color which is obtained by a histogram, a
color space description type for describing a descriptor with
respect to a color space used for describing the region dominant
color, a color sub-space description for defining whether the
region dominant color is expressed in a sub-space of the defined
reference color space, a quantization description for describing a
quantization method of the color space, and a color clustering
description for describing when the region color is expressed based
on the color clustering method.
[0029] In addition, the extraction method description includes a
method using an average color value of a certain region, a method
for expressing one most frequently appearing color, a method for
expressing N number of most frequently appearing colors, a method
for using colors which appears more than P % of threshold value in
the predetermined region or a method for using a color
histogram.
[0030] In the pre-processing description, it defines a filter type
established when a region dominant color value is obtained, a
filter size adapted in the image region, and a filter sliding
method of a filter window.
[0031] The frequency condition description defines a frequency
threshold for defining in a threshold value of a frequency above
which the colors are set to the region dominant color, a sorting
order description for designating the number of n top frequency
colors of a region dominant color, and a frequency sum of top n
frequency thresholds of the frequently appearing threshold value of
the higher n frequency.
[0032] The color space description defines reference color space
which is a reference of a dominant color and a transformation
description from a reference color space to define the
transformation from a well known color space to the adopted color
space, wherein the transformation description defines the number of
color channels of the reference color space (?) and a
transformation type and method.
[0033] In the color sub-space description, it defines the number of
color channels and the color channels used, and a range of the
channel, and a vector sub-space type with a method for the type,
when the region dominant color only considers a sub-space of a
color space.
[0034] In the quantization method description, in order to describe
the quantization method of the color space, the quantization
description defines the number of quantized channels and the
quantized color channels, the quantization method and the number of
the quantization levels for each channel, and a method used for a
quantization transformation.
[0035] In addition, the color clustering description defines
whether the clustering is used or not, and whether or not the
number of clustering is varied in accordance with the region, the
number of the clusters and the color channels used in the
clustering and the method to describe each cluster.
[0036] Therefore, it is possible to perform a search among data
constructed from different DOD extraction methods in different
systems using the DCD extraction method description, and a search
by unifying two DCD extraction methods into one method.
[0037] In addition, by adopting the DCD extraction method
description, the confidence measure can be obtained for expressing
the degree of accuracy of the region dominant color for thereby
enhancing a search performance and implementing a compatibility
among the region dominant colors which are extracted by different
extraction methods.
[0038] The confidence measure is determined by all or part of
factors such as a NADCA (Not Apparently Distinguishable Color
Allowance) which is a maximum variance that any two colors are
recognizable as the same color, and a coherency value for measuring
whether or not the pixels of the colors are gathered with respect
to the color given, and a CME (Color Mapping Error) which is
related to an error between all color values mapping to the
dominant color and the dominant color value i.e. CME is the
property of the color variance of the colors clustering a dominant
color, and the size of the region covered by the dominant color in
the image region, and the position of the color pixels in the
region.
[0039] The confidence measure extraction method includes a step for
initializing the confidence measure and the count sum of the
pixels, a step for obtaining a coherency value and the counting
value of the corresponding color pixels with respect to all
dominant colors Ci and adding a confidence of the initial value to
the value obtained by multiplying the coherence value and the
counting value of the color pixels for thereby obtaining a
confidence with respect to all colors, and a step for obtaining a
confidence with respect to the image region by dividing the
obtained confidence value into the region size.
[0040] In addition, a confidence is obtained with respect to each
color using a confidence extraction method.
[0041] The video region dominant color setting method will be
explained with reference to the accompanying drawings.
[0042] The DCD (Dominant Color Descriptor) capable of expressing
the colors of an object appearing in a visual data (video and/or
images) or a region of interest (ROI) during a multimedia indexing
operation is set.
[0043] The region dominant color descriptor is a color descriptor
with respect to a certain region and is determined based on the
entire images or a part of the image of the region, a video
segment, a region having an irregular shape based on the time
variance with respect to an object like a video segment, and a
region for expressing a regular position in accordance with the
time of the video segment.
[0044] The DCD is expressed based on the number N of the dominant
colors with respect to a region provided, an I-th dominant color
Ci, a frequency Pi of the dominant color Ci, and a CM (Confidence
Measure) expressed by an accurate color value which represents the
region.
[0045] Namely, DCD:[N, {Ci,Pi)|0<i.ltoreq.N}, CM]
[0046] where N represents the number of the dominant colors in the
DCD, Ci represents an i-th expressed dominant color
(0<1.ltoreq.N) in the DCD, Pi represents a frequency
(0<i.ltoreq.N) that the dominant color Ci appears in the region,
and CM represents a confidence, namely, the accuracy of the color
value and/or percentage value which represents the region.
[0047] Here, the dominant color Ci is defined by a plurality of
parameters. Namely, it is formed of a color space description, a
quantization description, a color clustering description, and a
channel description such as the number of color channels.)
[0048] Therefore, it is possible to express the region dominant
color based on an expression method of the DCD with respect to the
region dominant color value in accordance with various region
dominant color feature extraction methods, namely, a plurality of
colors, and the confidence CM of the color.
[0049] For example, when the DCD1 is expressed by DCD1=[N=1,
{C0=(r,g,b), P0=UNDEFINED)}, CM=k] based on the average color
method, the number N of the dominant colors is 1, and the expressed
dominant color (C0) becomes an average color (r,g,b) of the region,
and the frequency P0 is expressed as UNDEFINED, and the confidence
CM is a confidence value k in which the average value represents
the region.
[0050] In addition, in the histogram, in the case that the DCD5 is
expressed as DCD5=[N=64, {(C0=(r1,g1,b1), P0=10%), (C1=(r2,g2,b2),
P1=5%, . . . , (C63=(r63,g63,b63), P63=1%)}, CM=0.99], the number N
of the dominant colors is the number of the histogram bin.
Therefore, when expressing the histogram using 64 bins, N equals 64
and C0.about.C63 are expressed by the color values of a
corresponding bin.
[0051] If the number of quantization levels is too large or too
small when forming the histogram, the confidence has a small value.
Accordingly, it is possible to check whether a proper number of
quantization levels are obtained based on the confidence CM
value.
[0052] FIG. 2 illustrates a description search method which is
implemented using the region dominant color. In this method, if
user select a region, a region descriptor corresponding thereto is
extracted, and the dominant color description with respect to the
above-described corresponding region is extracted. All stored
region dominant color descriptors and the extracted dominant color
descriptor are compared.
[0053] Therefore, since all region dominant color descriptors and
the extracted dominant color descriptor are compared, it is
possible to perform a descriptor search using the dominant color
descriptor with respect to all region descriptors.
[0054] In addition, FIG. 3 illustrates a method for maintaining an
interoperability between different systems using a region dominant
color extraction method description.
[0055] In this method, a region DCD with respect to the given
region R of each system A and B is extracted, and feature
extraction method of the region dominant colors is described.
[0056] By transforming the above-described DCD into a sharing data
format, a comparison search can be performed between different
systems. In addition, by transforming each of the formalized data
of the region DCD into a sharing data format and performing a
comparison search between the different systems, an
interoperability between the different systems can be
maintained.
[0057] The description with respect to the extracting method of the
region dominant color uses the following items (item 1 through item
7) to describe different extracting method of each region dominant
color. Each item is divided into small items.
[0058] In the extraction method type of the item 1, it defines a
method using an average color of the region, a method using one
color which is most frequently appeared, a method for expressing an
n number of most frequently appearing colors, a method using a
color which appears more than P % of threshold value in the
predetermined region or an extraction method using a histogram.
[0059] In the pre-processing description of the item 2, it defines
a format of pre-processing for smoothing and burring a region when
obtaining a dominant value of the region. Such a pre-processing
description includes a filter type (for example, an average filter,
etc.), a filter size (for example, n,m/whole/, etc), and a filter
sliding method (for example, 1,1/2,3/non-over lap, etc.) for
representing how to slide the filter window when adapting the
filter.
[0060] The frequency condition description of item 3 is directed to
how to use the frequency in which the dominant colors appear by
obtaining the histogram.
[0061] In detail, it defines the threshold value of the frequency
in which the frequency below the threshold value is not considered,
the sorting order threshold value in which the dominant colors are
set with respect to a few number among the frequencies which appear
n most frequently, and the sum of the frequencies as threshold
value, which appear n most frequently.
[0062] The color space description of item 4 is directed to a
descriptor with respect to the color space itself used for
indicating the region dominant color.
[0063] In detail, the reference color space (for example, RGB, HSV,
etc.) is defined, and a transformation relationship between the
reference color space and a certain well known color space is
described.
[0064] Namely, in the transformation description from the reference
color space, the number of color channels of the adopted color
space and the type of transformation (linear type/non-linear type)
from the reference color space to the adopted color space are
defined, and the transformation is defined.
[0065] In the case that the transformation method to the color
space is a linear type, a transformation matrix is defined,
otherwise (in the case of the non-linear type), the C-code type is
used for a definition method (for example, a definition based on an
equation and a certain condition).
[0066] The color sub-space description of item 5 is directed to
recognize whether the region dominant color is expressed in a
certain sub-space of the color space defined by the color
space.
[0067] In detail, in the case that the sub-space is considered, the
number of the color channels and a corresponding color channel are
defined, and it is defined whether the type of the vector sub-space
is adopted or not (vector space type/non-vector space type), and
the range of each channel is provided.
[0068] Here, since the channel range is expressed by a variable,
and the channel range is changed dependent of the condition of the
item.
[0069] If the vector sub-space type item is a non-vector space
type, the method is not defined, and otherwise the method is
clearly defined. At this time, the re-definition is clearly
performed whenever the condition is changed.
[0070] The quantization description of item 6 is directed to a
quantization method of the color space.
[0071] In detail, the number of the quantized channels, the
quantized color channels, and the quantization type (linear
type/non-linear type/vector quantization type) are defined. In
addition, the number of quantization levels of each channel and the
thusly defined quantization type are defined in detail.
[0072] If the quantization type is a linear type, it is described,
and if the quantization type is a non-linear type, one vector is
described for one color channel. In addition, in the case of the
vector quantization type, it is defined by an equation and a
condition method. In addition, it is possible to clearly express
using a look-up table.
[0073] The color clustering description of item 7 is directed to
expressing whether the color is clustered to be color
quantization.
[0074] If the level type is not fixed, it is expressed that the
number of clustering is varied in accordance with the region, and
the number of the clusters and the clustered color channels are
expressed for thereby defining each cluster.
[0075] When defining each cluster color, it is expressed based on a
parameter of an ellipsoid and a centeroid of the ellipsoid.
[0076] For an example of the extraction method description for the
extraction method using an average color among various extraction
methods of each region dominant color, it will be explained as
follows.
[0077] 1. Extraction method type=average color
[0078] 2. Preprocessing description: [0079] 2-1. Filter
type=Average filter [0080] 2-2. Filter size=whole [0081] 2-3.
Filter sliding method=non-overlap
[0082] 3. Frequency condition description: [0083] 3-1. Frequency
threshold=0% or n/a [0084] 3-2. Sorting order threshold=n/a [0085]
3-3. Frequency sum of top n frequencies threshold=100% or n/a
[0086] 4. Color space description: [0087] 4-1. Reference color
space=RGB [0088] 4-2. Transformation from reference color space
description: [0089] 4-2-1. Number of color channels=n/a [0090]
4-2-2. Uniform type transformation=n/a [0091] 4-2-3. Method
definition=n/a
[0092] 5. Color sub-space description: [0093] 5-1. Sub-space
used=FALSE [0094] 5-2. Number of using color channels=n/a [0095]
5-3. Using color channels=n/a [0096] 5-4. Channel ranges=n/a [0097]
5-5. Vector sub-space type=n/a [0098] 5-6. Method
definition=n/a
[0099] 6. Quantization description: [0100] 6-1. Number of quantized
channels=3 [0101] 6-2. Quantized color channels={channel 1, channel
2, channel 3} [0102] 6-3. Type=uniform type [0103] 6-4. Number of
quantization levels per channel=(4,4,4) [0104] 6-5. Quantization
definition=n/a
[0105] 7. Color clustering description: [0106] 7-1. Clustering
used=FALSE [0107] 7-2. Fixed level type=n/a [0108] 7-3. Number of
clusters=n/a [0109] 7-4. Clustered color channels=n/a [0110] 7-5.
Cluster definition=n/a
[0111] Namely, in the description of the extraction method using an
average value, the type of the extraction method of item 1 is
directed to extracting an average color.
[0112] Item 2 is directed to a pre-processing description. In the
filter type 2-1 in the detailed item, an image region is
average-filtered by an average filter, and what the filter size 2-2
is "whole" represents that the entire values are averaged not
average-filtering the image region using a certain filter size. In
addition, what the filter sliding method 2-3 is "non-overlap"
represents that the earlier filter window is not overlapped with
the later filter window when adapting the filter window.
[0113] In item 3, when obtaining the histogram, and the frequency
is used, since the threshold value 3-1 is 0% or n/a, it means that
the above-described value is not considered. In addition, since the
sorting order threshold value 3-2 is n/a, it means that the
above-described value is not considered. The threshold value 3-3
represents that it is not considered since the frequency sum of top
n frequencies threshold is 100% or n/a.
[0114] Item 4 is a descriptor with respect to the color space
itself, and the reference color space 4-1 and the transformation
description 4-2 from the reference color space are directed to
expressing a transformation relationship between adopted color
space and the reference color space.
[0115] Namely, the reference color space is a RGB space, and since
the transformation description 4-2 is n/a, it means that there is
no color space which is newly adopted, and the color space which
expresses the region dominant color value is RGB.
[0116] In the case that the RGB and other color space are used, and
a transformation between the color space and the RGB is described,
if the linear type is TRUE, one transformation matrix is defined
and expressed, and in the case that the linear transformation is
not defined, the equation and/or conditional sentence is used for
thereby defining the item.
[0117] The description of the color sub-space of item 5 is directed
to checking whether the region dominant color is expressed in a
sub-space of a certain color space defined by item 4. Since the
used sub-space is set as FALSE, the region dominant color value
does not consider a certain sub-space.
[0118] The quantization description of item 6 is directed to a
quantization method of the color space. The number 6-1 of the
quantized channels is 3, and the three quantized color channels 6-2
are channel 1, channel 2 and channel 3. In addition, since the
number of the quantization types 6-3 is 4,4,4, this means that the
channel of each R,G,B is quantized by 4,4,4, respectively so that
the member of quantization levels is "64".
[0119] In the case that the quantization type of 6-3 is a
non-uniform, one vector must be described per one color channel to
define quantization point per channel, and in the case of the
vector quantization type, it is expressed by some arithmetic
expression.
[0120] In item 7, the color is not clustered in this example,
therefore this item is not used (Clustering used=FALSE).
[0121] For another example of the extraction method description,
the description with histogram extraction method is explained.
[0122] In the following extraction method, up to 10 colors are
defined as the region dominant colors which are the most frequently
appearing top ten colors, and a histogram with respect to the
region is obtained and colors corresponding to the histogram bins
are defined as the dominant colors with the condition that the
frequency below 1.5% is excluded.
[0123] The items are set as follows to express this extraction
method.
[0124] <Extraction Method Description Using Histogram>
[0125] 1. Extraction method type=At most top ten frequently
appearing colors:
[0126] 2. Preprocessing description: [0127] 2-1. Filter
type=Average filter [0128] 2-2. Filter size=5,5 (means 5 by 5
filter) [0129] 2-3. Filter sliding method=1,1
[0130] 3. Frequency condition description: [0131] 3-1. Frequency
threshold=1.5% [0132] 3-2. Sorting order threshold=10 [0133] 3-3.
Frequency sum of top n frequencies threshold=n/a (or 100%)
[0134] 4. Color space description: [0135] 4-1. Reference color
space=RGB [0136] 4-2. Transformation from reference color space
description: [0137] 4-2-1. Number of color channels=3 [0138] 4-2-2.
Uniform type transformation=FALSE [0139] 4-2-3. Method definition=
[0140] input ranges: r=(0,255), g(0,255), b=(0,255); [0141] output
ranges: C1=(0,255), C2=(0,255), C3=(0,360); [0142] C1=max(r,g,b)
[0143] if max(r,g,b)=0, C2=0;
[0144] else,
C 2 = max ( r , g , b ) - min ( r , g , b ) max ( r , g , b )
##EQU00001##
[0145] if max(r,g,b)=0, C3=UNDEFINED
[0146] else if r=max(r,g,b) & (g-b>0)
C 3 = ( g - b ) .times. 60 max ( r , g , b ) - min ( r , g , b )
##EQU00002##
[0147] else if r=max(r,g,b) & (g-b<0)
C 3 = 360 + ( g - b ) .times. 60 max ( r , g , b ) - min ( r , g ,
b ) ##EQU00003##
[0148] else if r=max,
C 3 = 120 + ( g - b ) .times. 60 max ( r , g , b ) - min ( r , g ,
b ) ##EQU00004##
[0149] else
C 3 = 240 + ( g - b ) .times. 60 max ( r , g , b ) - min ( r , g ,
b ) ##EQU00005##
[0150] 5. Color sub-space description: [0151] 5-1. Sub-space
used=TRUE [0152] 5-2. Number of using color channels=1 [0153] 5-3.
Using color channels=C1 [0154] 5-4. Channel ranges=0,360 [0155]
5-5. Vector sub-space type=FALSE [0156] 5-6. Method
definition=n/a
[0157] 6. Quantization description:
[0158] 6-1. Number of quantized channels=1 [0159] 6-2. Quantized
color channels=C1 [0160] 6-3. Type=uniform type [0161] 6-4. Number
of quantization levels per channel=24 [0162] 6-5. Quantization
definition=n/a
[0163] 7. Color clustering description: [0164] 7-1. Clustering
used=FALSE [0165] 7-2. Fixed level type=n/a [0166] 7-3. Number of
clusters=n/a [0167] 7-4. Clustered color channels=n/a [0168] 7-5.
Cluster definition=n/a
[0169] In detail, item 1 describes "at most top 10 frequently
appearing colors" as the extraction method type.
[0170] In the preprocessing description of item 2, 2-1 represents
that the average filter of the region is adopted, and 2-2
represents that the size of the filter having 5 by 5 is used, and
what the filter sliding method of 2-3 is 1,1 represents that the
center of the filter is moved by 1, 1 in the filter window in
vertical and horizontal directions.
[0171] Item 3 is for the frequency condition description. Since the
frequency threshold value of 3-1 is 1.5%, except for the frequency
that is below 1.5%, the sorting order threshold value of 3-2 is 10.
Therefore, the maximum 10 colors are designated as the dominant
colors according to the frequency of the colors in the histogram,
and n/a of 3-3 represents that this item is not considered.
[0172] Item 4 is a color space description. The reference color
space is RGB, and the number of the color channel 4-2-1 of the
color space for the transformation description 4-2 of the reference
color space is 3, and the uniform type transformation is set to
FALSE. therefore, the transformation between the color space
adopted and RGB is a non-uniform transformation. In 4-2-3, the
condition with respect to the non-uniform transformation method is
described.
[0173] In addition, in the condition 4-2-3 of the non-uniform
transformation method, the input ranges and output ranges of each
channel are defined, where the output ranges based on the input
conditions are defined.
[0174] Item 5 is the description of the color sub-space and is
directed to check whether the region dominant color is expressed in
a sub-space of the color space defined in item 4.
[0175] Since the used sub-space is set to TRUE, it is known that
the region dominant color value considers a certain sub-space, and
in 5-2, 5-3, and 5-4, it is known that one color channel C1 is
considered as a channel range value of 0.about.360.
[0176] The quantization description of item 6 is directed to a
quantization method of the color space, and number (6-1) of
quantized channels is 1, and the quantized channel 6-2 is C1, and
the quantization type 6-3 is defined as a uniform quantization
type, and it is not needed to have a method definition 6-5.
[0177] In addition, the number 6-4 of the quantization levels of
each channel represents that the channel C1 is quantized to
24-levels.
[0178] In addition, item 7 is directed to checking whether the
color is clustered or not. The use of the clustering is set to
FALSE which means that the clustering is not used.
[0179] The above-described data structure are defined in the header
part of the memory, and whenever each item is changed, the item is
re-defined.
[0180] Therefore, it is possible to clearly describe the meaning of
the dominant color description among the different feature
extraction methods based on the above-described feature extraction
method, thereby the interoperability is satisfied in comparison
search among data generated by different systems.
[0181] Namely, it is possible to conduct a comparison search by
checking an extraction method with respect to the region dominant
color descriptor using an extraction method description and by a
step (sharing data format transformation) for integrating two
region dominant color descriptors to be compatible. In addition, it
is possible to maintain an interoperability between other feature
extraction methods using a sharing data with respect to the region
dominant color extraction method.
[0182] The confidence measure CM of the region dominant color is a
descriptor which represents an accuracy of the expressed region
dominant color and represents whether a corresponding region is
expressed by one color and so on. The confidence CM is set by
numeral values which represent the degree of confidence when the
color property of the region is expressed by dominant colors.
[0183] The above-described confidence measure can be expressed by
the normalized values of 0.about.1, and the confidence measure may
be expressed by a vector value.
[0184] For example, CM=[C,ACME,P,AISI].
[0185] Here, C represents a normalized coherency (image spatial
varience), and ACME represents an average of color mapping error
value, P represents a valued obtained by summing the frequencies of
all region dominant color values, and AISI represents an average of
image space importance.
[0186] Therefore, when the confidence measure CM is expressed by a
few colors, it is more useful. Namely, it is difficult to express
the region by a few colors especially when the region consists of
various colors. At this time, the value of the confidence is very
important.
[0187] In addition, when the value of the confidence CM is low
means that the region is formed of complicated various colors.
Therefore, it is useful for a searching operation. In the case that
more than one extraction method are provided for each region, or
another feature descriptor is provided, various methods taking
advantage of the confidence measure can be used.
[0188] For example, in the case that the value of the confidence
measure of the region dominant color extracted by the average value
extraction method is low, it is possible to use other descriptors
such as a region dominant color descriptor based on the histogram
extraction method, etc.
[0189] In addition, when the region dominant colors are expressed
by a plurality of dominant color values based on a certain method
such as an extraction method of an n-number of most frequently
appearing colors, it is possible to check whether a proper number
of regions is expressed or not using the confidence measure
value.
[0190] The elements which are selectively adopted for extracting
the above-described confidence measure will be explained.
[0191] First, when one color is expressed by a certain value, the
color is varied in accordance with an increase/decrease of the
color value. At this time, the maximum variation value (NADCA: Not
Apparently Distinguish color Allowance) which may be recognized as
the similar color can exist.
[0192] Namely, it is not judged by whether people can distinguish
the slight color difference by the maximum variation. Instead, it
is judged by whether colors within the maximum variation are
recognized as the similar color by human, especially in a
content-based image search.
[0193] A blue sky image is expressed by hundreds of colors, so that
the image is naturally seen by the human eye. In the content-based
image search, it is possible to express one color, namely, a
certain blue color, so that too many color separations are not
needed during the content-based image search.
[0194] In particular, when obtaining the region dominant color
value based on an average value, it is possible to obtain the
confidence measure value based on a frequency of the region that
the average value covers the image region by defining the NADCA
value.
[0195] In addition, a coherency value (COH) is adopted to measure
whether the pixels of the color are gathered or scattered with
respect to a color given. The coherency value has a value of 0 to
1. As the coherency value is increased, the confidence value is
increased.
[0196] When a certain color Pj is considered (mapped) as a dominant
color Ci in the image region, where respective Pj and Ci is
expressed by one point in the color space, there is an error (CME:
Color Mapping Error) between the accurate value and the dominant
color value of the colors. As the difference is decreased, the
confidence is increased, and the difference is increased, the
confidence is decreased. This can be measured by color varience in
the color space.
[0197] Namely, CME is as follows:
CME = Q FO ##EQU00006##
[0198] In addition, the size Pi of the region that the dominant
color covers in the image region is reflected to the confidence. As
the size of the region that the dominant color covers is increased,
the confidence of the dominant color is increased.
[0199] The confidence is reflected based on an ISI (Image Space
Importance) in a region R of each color pixel. For example, if the
color pixels are positioned at the center portion of the image, the
colors may be considered as a more important color, and if the
colors are positioned at an edge portion of the region, the colors
may be considered as a less important color. Therefore, the
reliability is increased when the colors of the image region which
are expressed based on the representative color value are
positioned at the center portion.
[0200] Namely, when the extracted confidence is high represents
that the dominant colors are distanced from each other within the
region, and in the case that the quantization step is near an
actual NADCA value, the region colors cover the entire regions.
[0201] In addition, when the confidence is low represents that
dominant colors are mixed, or the quantization steps are actually
far from the NADCA value. At this time, the region colors do not
fully cover the region.
[0202] The algorithm for extracting the reliability is performed by
the following steps:
[0203] a) A step for setting the confidence to an initial value
(=0) is performed;
[0204] b) A step for setting the sum (SUM_COUNT_PELS) of count
pixels is set to an initial value (=0) is performed;
[0205] c) A value (COUNT PELS_Ci) obtained by counting the color
pixels corresponding to each region dominant color with respect to
all region dominant colors and a coherency COH_Ci corresponding to
each region dominant color are obtained, and the coherency value
COH_Ci and the count value COUNT_PELS_Ci of the color pixels are
multiplied, and the confidence is added to the thusly multiplied
value for thereby obtaining a confidence CM with respect to the
region dominant color;
[0206] d) The confidence value is divided into region sizes SIZE_R
for thereby obtaining a confidence with respect to the image
region; and
[0207] e) The thusly obtained confidence is outputted.
[0208] Here, the region size SIZE_R is a size in the region and is
computed by the counting of the pixels in the region R.
[0209] At this time, there are two methods for computing the
coherency COH_Ci with respect to one dominant color Ci value.
[0210] A first method includes:
[0211] a step (1) for inputting a size of a coherency checking mask
having a certain width and height, a step (2) for setting a count
(COUNT_PELS_Ci) of the color pixels and a coherent total
(TOTAL_NUM_COHERENT) to an initial value (=0), and a step (3) which
includes a step (3-1) for obtaining a count value
(COUNT_PELS_Ci_PELS_Ci+1) of the color pixels by increasing the
color pixels with respect to all pixels PELj in the region R which
satisfies that the color of the pixel PELj is mapped to the
dominant color, a step (3-2) for obtaining the number of coherent
(0.about.WIDTWHEIGHT)-1 by counting the number (except for the
central pixels) of the masked pixels in the case that the color
pixels masked by the central arrangement of the coherence checking
mask CCM are mapped to the dominant colors, and a step (3-3) for
obtaining the total number of the coherency (TOTAL_NUM_COHERENT) by
summing the number of the coherency and the total number of the
coherency, a step (4) for obtaining a coherency value (COH_Ci) with
respect to one dominant color value by dividing the total number of
the obtained coherent values by a value obtained by multiplying the
total pixels (WIDTH*HEIGHT-1) to the total number of coherences
except for the count value of the pixel, and a step (5) for
outputting a coherency value with respect to one dominant color
value and the count values of the colors and the center pixels of
the pixel colors.
[0212] The second method uses a threshold value and includes a step
(1) for inputting a size of a coherency checking mask (CCM) having
a certain width and height, a step (2) for setting a certain number
of threshold values (for example, WIDTH*HEIGHT-1), a step (3) for
setting the count values of the color pixels, the total number of
the coherency and the count value of the non-boundary pixels to an
initial value (=0), respectively, a step (4) which includes a step
(4-1) with respect to all pixels in the region which satisfies that
the pixel color is mapped to the dominant color for obtaining the
count values of the color pixels by up-counting the color pixels
one by one, a step (4-2) for obtaining the coherent number
(0.about.WIDTH.times.HEIGHT)-1 by counting the number (except for
the central pixels) of the masked pixels in the case that the color
pixels masked by the central arrangement of the coherency checking
mask CCM are mapped to the dominant color, and a step (4-3) for
obtaining a count value of the non-boundary pixels (NONBOUND_PELS)
by increasing the non-boundary pixels one by one in the case that
the coherent number is the same as or is larger than the boundary
threshold value, a step (5) for obtaining a coherency value with
respect to one dominant color by dividing the count value of the
thusly obtained non-boundary pixels by the count value of the color
pixels, and a step (6) for outputting the count values of the
coherency value and the color pixels with respect to one dominant
color.
[0213] In the above-described methods, as a condition for
determining the color which is mapped with the dominant color, when
a difference between the dominant colors which are not clearly
separated from other colors and the pixel colors is smaller than
NADCA, namely, DISTANCE (Ci, COLOR_OF_PELj)<NADCA, the
above-described condition may be changed to the above-described
satisfying condition (step (1) of method 1, and step (4) of method
2).
[0214] In addition, as a condition for using the same color as the
dominant color, when a difference between the dominant color and
the masked pixel color is smaller than NADCA, namely; DISTANCE (Ci,
COLOR_OF_MASKED_PEXELk)<NADCA, the condition may be changed to
the above-described condition (step (3-2) of the method 1, and step
(4-2) of the method 2).
[0215] As the confidence is obtained with respect to the region
dominant color by the above-described method, it is possible to
obtain an interoperability during a search with respect to the
region dominant color for a different feature extraction method
using the confidence measure.
[0216] Namely, a certain region dominant color is obtained based on
the region average value, and a certain region dominant color is
obtained based on a histogram. In this case, since there is a
certain confidence value, the confidence value may be usefully used
for a comparison of the region dominant color values based on the
different feature extractions.
[0217] In addition, the DCD extraction method shaping data is
implemented as follows:
[0218] DCD1=[N=1,{(C0=gray, P0=n/a)}, CM=0.01],
[0219] DCD2=[N=1,{(C0=gray, P0=n/a)}, CM=0.99],
[0220] DCD3[N=2,{(C0=red, P0=50%)}, (C1=cyan, P1=50%)},
CM=0.99],
[0221] DCD4=[N=2,{(C0=red, P0=50%)}, (C1=cyan, P1=50%)},
CM=0.01],
[0222] DCD5=[N=n,{(C0=red, P0=10%)}, (C1=yellow, P1=5%), . . . ,
(Cn-1=gray, Pn-1=1%, CM=0.99]; The average color obtained based on
DCD5 is assumed as "gray".
[0223] DCD6=[N=n,{(C0=red, P0=10%)}, (C1=yellow, P1=5%), . . . ,
(Cn-1=gray, Pn-1=50%, CM=0.99]; The average color obtained based on
DCD6 is assumed as "gray".
[0224] The dominant colors of DCD2&DCD4&DCD6 are similar
based on the region dominant color descriptor, and the dominant
colors of DCD1&DCD3&DCD5 are similar.
[0225] At this time, in the region dominant color descriptor, C0 is
obtained based on an average value in DCD1, and (C0,P0),(C1,P1) of
DCD3 are histogram when the histogram is recognized, so that it is
possible to obtain an average color C* and compare with the
obtained average color C* and C0 based on (C0,P0),(C1,P1).
[0226] In addition, it is possible to obtain a confidence CMi based
on each color Ci except for the confidence value with respect to
the total region dominant color descriptor DCD.
[0227] Namely, DCD=[N,{Ci, Pi, CMi|0<i.ltoreq.N)}, CM].
[0228] The confidence with respect to each color is determined
based on various elements as arranged in the confidence for the
region, namely, normalized coherency (spatial variance), color
mapping error CME (color variance), the size of region that the
dominant color covers, and the position of each color pixels in the
region for thereby obtaining a confidence value with respect to the
determined color. Therefore, CMi can be represented by a vector
such that CMi=[SpatialVariance, ColorVariance, SizeOfCovers,
Position].
[0229] The SpatialVariance which is inverse proportional to the
coherency can be defined similarly as in p24-26.
[0230] The ColorVariance value with respect to a certain color
based on the color variance which is inverse proportional to the
color mapping error CME may be obtained based on the following
equation.
ColorVariance.sub.--Ci=SUM[Distance(CENTROID.sub.--Ci,MAPPING_COLOR_POIN-
T.sub.--Pj_TO.sub.--Ci)/MAX_DISTANCE.sub.--Pj_TO.sub.--CI.times.NUM_MAPPIN-
G_COLOR_POINT.sub.--Pj_TO.sub.--Ci] for all j
CM=Sum(CM.sub.--Ci) for all i/MAX_i+1
[0231] Namely, the ColorVariance with respect to a certain dominant
color is a difference with respect to all colors which are
recognized as a dominant color, and DISTANCE(CENTROID_Ci,
MAPPING_COLOR_POINT_Pj_TO_Ci) is a difference with respect to the
color Pj when the color is assumed as the dominant color Ci.
[0232] MAX_DISTANCE_Pj_TO_Ci is a maximum distance difference
between two colors (Pj, Ci).
[0233] NUM_MAPPING_COLOR_POINT_Pj_TO_Ci is the total number that
the color Pj is mapped to Ci as the maximum value of j.
[0234] The values obtained by the above-described methods are
normalized, and the confidence with respect to a certain dominant
color has a value between 0 and 1.
[0235] The confidences with respect to all dominant color values
are summed (SUM(CM_Ci) for all i (where i represents an integer,
o<j<M) and is divided by the maximum value (MAX_i+1) for
thereby obtaining an average value of CM_Ci, namely, the confidence
CM with respect to the image region.
[0236] As described above, the region dominant color descriptor and
the confidence of the same are expressed based on a similarity of
the color with respect to the image region, an error of the same,
the size that the dominant color covers the region, and the
position of the region, so that it is possible to enhance a search
performance and to provide an interoperability between the region
dominant colors based on different extraction methods.
[0237] In addition, a standardized method is implemented by the
extraction method description of the region dominant color
descriptor using a color space descriptor, a quantization
descriptor, a color cluster descriptor, and the number of color
channels, so that it is possible to conduct a comparison search
between the extracted region dominant colors extracted by various
methods.
[0238] The present teaching can be readily applied to other types
of apparatus. The description of the present invention is intended
to be illustrative, and not to limit the scope of the claims. Many
alternatives, modifications, and variations will be apparent to
those skilled in the art.
* * * * *