U.S. patent application number 13/336964 was filed with the patent office on 2012-06-28 for apparatus for color interpolation using adjustable threshold.
This patent application is currently assigned to MTEKVISION CO., LTD.. Invention is credited to Ho-Young LEE, Jun-Oh PARK.
Application Number | 20120163734 13/336964 |
Document ID | / |
Family ID | 38218252 |
Filed Date | 2012-06-28 |
United States Patent
Application |
20120163734 |
Kind Code |
A1 |
LEE; Ho-Young ; et
al. |
June 28, 2012 |
APPARATUS FOR COLOR INTERPOLATION USING ADJUSTABLE THRESHOLD
Abstract
An apparatus for color interpolation using an adjustable
threshold is disclosed. The color interpolation apparatus
calculates the difference between the maximum value and the minimum
value of the elements of the image data and determines the color
interpolation method of the image data depending on the difference
to perform the corresponding color interpolation. With the present
invention, the improved image quality can be provided because the
color interpolation can be performed as a user desires.
Inventors: |
LEE; Ho-Young; (Seoul,
KR) ; PARK; Jun-Oh; (Seoul, KR) |
Assignee: |
MTEKVISION CO., LTD.
Seoul
KR
|
Family ID: |
38218252 |
Appl. No.: |
13/336964 |
Filed: |
December 23, 2011 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
12159365 |
Jun 26, 2008 |
|
|
|
PCT/KR2006/005884 |
Dec 29, 2006 |
|
|
|
13336964 |
|
|
|
|
Current U.S.
Class: |
382/300 |
Current CPC
Class: |
H04N 9/64 20130101 |
Class at
Publication: |
382/300 |
International
Class: |
G06K 9/32 20060101
G06K009/32 |
Foreign Application Data
Date |
Code |
Application Number |
Dec 29, 2005 |
KR |
10-2005-0132905 |
Dec 29, 2005 |
KR |
10-2005-0133516 |
Dec 29, 2005 |
KR |
10-2005-0133604 |
Dec 29, 2005 |
KR |
10-2005-0133629 |
Claims
1. A threshold determining apparatus comprising: a first
determining unit, receiving the image of the flat surface
photographed with the maximum exposure and the minimum exposure,
respectively, and randomly acquiring samples from the mask, having
a constant size, of the received image and calculating a first
difference and a second difference, to calculate a first average of
the first difference and a second average of the second difference,
the first difference being the difference between the maximum value
and the minimum value of noise in the maximum exposure, and the
second difference being the difference between the maximum value
and the minimum value of noise in the minimum exposure; a second
determining unit, determining the maximum value and the minimum
value of the threshold by using the average of the difference
between the first average and the second average, calculated by the
first determining unit; and a third determining unit, determining
the threshold by using the maximum value and the minimum value of
the threshold, determined by the second determining unit, and the
maximum exposure and the minimum exposure.
2. The threshold determining apparatus of claim 1, wherein the
second determining unit determines the maximum value of threshold
by the following formula: th.sub.max=diff.sub.max exp (where
th.sub.max refers to the maximum value of threshold, and
diff.sub.max exp refers to the average value of the difference
between the maximum value and the minimum value of noise in the
maximum exposure).
3. The threshold determining apparatus of claim 1, wherein the
second determining unit determines the minimum value of threshold
by the following formula: th.sub.min=diff.sub.min exp (where
th.sub.min refers to the minimum value of threshold, and
diff.sub.min exp refers to the average value of the difference
between the maximum value and the minimum value of noise in the
minimum exposure).
4. The threshold determining apparatus of claim 1, wherein the
third determining unit determines the threshold by the following
formula: th = th max - th min Max Exposure - Min Exposure Exposure
+ Max Exposure th min - Min Exposure th max Max Exposure - Min
Exposure ##EQU00037## (where Max Exposure refers to the maximum
exposure value, and Min Exposure refers to the minimum exposure
value).
5. A threshold determining method comprising: receiving the image
of the flat surface photographed with the maximum exposure and the
minimum exposure, respectively, and randomly acquiring samples from
the mask, having a uniform size, of the received image and
calculating a first difference and a second difference, to
calculate a first average of the first difference and a second
average of the second difference, the first difference being the
difference between the maximum value and the minimum value of noise
in the maximum exposure, and the second difference being the
difference between the maximum value and the minimum value of noise
in the minimum exposure; determining the maximum value and the
minimum value of the threshold by using the average of the
difference between the first average and the second average,
calculated by the operation of calculating the first average and
the second average; and determining the threshold by using the
maximum value and the minimum value of the threshold, determined by
the operation of determining the maximum value and the minimum
value of the threshold, and the maximum exposure and the minimum
exposure.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application is a Divisional of co-pending application
Ser. No. 12/159,365 filed on Jun. 26, 2008, which is the national
phase of PCT International Application No. PCT/KR2006/005884, filed
Dec. 29, 2006, and which claims priority to Korean Application No.
10-2005-0132905 filed on Dec. 29, 2005, Korean Application No.
10-2005-0133516 filed on Dec. 29, 2005, Korean Application No.
10-2005-0133604 filed on Dec. 29, 2005 and Korean Application No.
10-2005-0133629 filed on Dec. 29, 2005. The entire contents of all
of the above applications are hereby incorporated by reference.
BACKGROUND
[0002] 1. Technical Field
[0003] The present invention is related to an apparatus for color
interpolation using an adjustable threshold, more specifically to
an apparatus, which is employed in an image processing system, for
color interpolation using an adjustable threshold.
[0004] 2. Description of the Related Art
[0005] Today's development of multimedia apparatuses makes it
possible to perform complicated image processing. This complicated
image processing necessarily needs color interpolation.
[0006] Typically, the color interpolation creates new color
information from pre-existing color information when the standard
method of image signal is converted.
[0007] Since the color interpolation generates a new component,
which is not included in the present pixel by using other
components of the periphery of the pixel, a noise having a zipper
shape may occur at an area (e.g. edge or boundary) where there are
a lot of high frequencies.
[0008] To remove this noise, many prior arts have been disclosed.
These prior arts, however, apply the same process to all images
after the system is completed to be structured. Accordingly, the
process is not varied depending on each image. That is, it causes
waste of the system efficiency because at least the same noise
removing process must be also applied to the image having less
noise.
[0009] The conventional image processing system, the color
interpolation also applies the same process to all images after the
system is completed to be structured, which the process is not
varied depending on each image.
SUMMARY
[0010] The present invention, which is designed to solve the above
problems, provides an apparatus for color interpolation using an
adjustable threshold in order to apply a different color
interpolation method depending on each image by adjusting the
threshold in accordance with a particular value in the mask of the
image.
[0011] The present invention also provides an apparatus for color
interpolation using an adjustable threshold in order to reduce a
line memory by supplying an edge enhancement coefficient for edge
enhancement in color interpolation.
[0012] The present invention also provides an apparatus and a
method of determining a threshold used for color interpolation in
order to apply a different color interpolation method depending on
the threshold, determined in accordance with the analysis of the
properties of the image inputted according to the exposure.
[0013] In addition, the present invention provides an apparatus and
a method of determining a threshold used for removing a noise in
order to apply a different noise removing method depending on the
threshold, determined in accordance with the analysis of the
properties of the image inputted according to the exposure.
[0014] Other objects of the present invention will become more
easily understandable through the embodiments described below.
[0015] To achieve the above objects, an aspect of the present
invention features an apparatus for color interpolation using an
adjustable threshold.
[0016] According to an embodiment of the present invention, the
color interpolation apparatus can include a color interpolation
unit, performing the interpolation of an inputted m.times.m Bayer
pattern image and generating and outputting first output data and
second output data, and detecting and outputting first component
data and second component data and special data of the m.times.m
Bayer pattern image, m being a predetermined natural number; and a
determining unit, outputting an edge enhancement coefficient, the
edge enhancement coefficient being determined by using one of the
first output data and the second output data, an edge parameter,
the first edge component data and the second edge component data,
and one of the first output data and the second output data being
determined by using the size relationship of the special data,
received from the color interpolation unit, and predetermined
plural thresholds, respectively. Here, the color interpolation can
have a first interpolation unit, performing the bilinear
interpolation of the m.times.m Bayer pattern image and outputting
n.times.n data, n being a predetermined natural number and smaller
than m; a second interpolation unit, performing the interpolation
of the output data of the first interpolation unit in accordance
with the center pixel value of the data, and outputting the first
output data of 1 pixel; a first calculating unit, performing the
edge detection of the output data of the second interpolation unit,
and outputting the first edge component data of 1 pixel; a second
calculating unit, performing the edge detection of the output data
of the first interpolation unit, and outputting n.times.n data; a
first filter unit, performing the Gaussian filtering of the output
data of the second calculating unit, and outputting the second edge
component data of 1 pixel; a difference determining unit,
determining the difference between the maximum value and the
minimum value of the n.times.n output data of the second
calculating unit, and outputting the determined difference as the
special value; and a second filer unit, performing the Gaussian
filtering of the output data of the first interpolation unit, and
outputting the second output data of 1 pixel. Also, the color
interpolation apparatus can further include a converting unit
converting red/green/blue (RGB) data to luminance (Y)/chrominance
(C) data.
[0017] The color interpolation apparatus can further include an
edge enhancement unit performing an edge enhancement processing by
use of the edge enhancement coefficient, considering the Y data
converted by the converting unit and an edge flag received from the
determining unit, whereas the edge flag is outputted from the
determining unit to determine whether the edge enhancement
processing is performed by using the edge enhancement
coefficient.
[0018] The determining unit can output the second output data if
the special value is smaller than a predetermined first threshold
determined according to the flat level of image. Also, the
determining unit can output the first output data and the edge
flag, the edge flag referring to no determination of the edge
enhancement processing, if the special value is substantially the
same as or larger than the first threshold and smaller than a
predetermined second threshold determined according to the edge
level of image. Moreover, the determining unit can output the first
output data the edge flag, the edge flag referring to the
determination of the edge enhancement processing, if the difference
is substantially the same as or larger than the second
threshold.
[0019] The edge parameter can have at least one of edge coring,
edge gain, edge positive limit and edge negative limit, and the
edge enhancement coefficient can be determined by the first edge
component data and the second edge component data and the edge
parameter.
[0020] According to another embodiment of the present invention,
the color interpolation apparatus can include a color interpolation
unit, performing the interpolation of an inputted Bayer pattern
image and generating first output data and second output data,
detecting first edge component data and second edge component data
of image, and detecting the difference between the maximum value
and the minimum value of the elements of the first and second
component data; a first determining unit, determining the output
according to the difference from the color interpolation unit, and
determining an edge enhancement coefficient by using the first and
second edge component data and an edge parameter; a second
determining, determining the maximum value and the minimum value of
the elements of the output data of the first determining unit; and
a removing unit, determining the noise removing method of the image
data according to the difference, received from the color
interpolation unit, and the maximum value and the minimum value,
received from the second determining unit, and performing the
corresponding noise removing.
[0021] The color interpolation unit has a first interpolation unit,
performing the bilinear interpolation of the inputted 7.times.7
Bayer pattern image, and outputting 3.times.5 data; a second
interpolation unit, performing the interpolation of the output data
of the first interpolation unit according to the center pixel value
of the data, and outputting 3.times.3 first output data; a first
calculating unit, performing the edge detection of the output data
of the second interpolation unit, and outputting the 3.times.3
first edge component data; a second calculating unit, performing
the edge detection of the output data of the first interpolation
unit, and outputting 5.times.5 data; a first filter unit,
performing the Gaussian filtering of the output data of the second
calculating unit, and outputting the 3.times.3 second edge
component data; a difference determining unit, determining the
difference between the maximum value and the minimum value of the
5.times.5 output data of the second calculating unit; and a second
filter unit, performing the Gaussian filtering of the output data
of the first calculating unit, and outputting 3.times.3 second
output data.
[0022] The first determining unit outputs the second output data if
the difference is smaller than a predetermined first threshold
determined according to the noise level of image; outputs the first
output data and the edge flag of zero if the difference is
substantially the same as or larger than the first threshold and
smaller than a predetermined determined according to the edge level
of image; and outputs the first output data and the edge flag of 1
if the difference is substantially the same as or larger than the
second threshold. At this time, the color interpolation apparatus
further has a sequent-processing unit performing the edge
enhancement of the inputted 3.times.3 RGB data, respectively, by
using the edge enhancement coefficient if the edge flag is 1. The
subsequent-processing unit has a sign generating unit, determining
and generating the sign of the edge enhancement coefficient; a
subsequent-processing constant generating unit, determining a
subsequent-processing constant according to the area of the
pertinent value adding each of the inputted RGB data to the edge
enhancement; a minimum generating unit, generating the minimum
value of the RGB component of the subsequent-processing constants
generated by the subsequent-processing constant generating unit; a
generating unit, generating a new edge enhancement coefficient by
using the data received from the sign generating unit and the
minimum value generating unit; and an output unit, outputting the
data, having the enhanced edge, of the inputted RGB data by using
the new edge enhancement coefficient.
[0023] The removing unit has a third determining unit, determining
the noise removing method of the image data according to the
difference received from the color interpolation unit; a first
filter unit, performing the filtering of the image data from the
second determining unit; and a second filter unit, performing the
zipper noise removing filtering of the image data received from the
second determining unit.
[0024] The third determining unit determines to input the image
data into the first filter unit and to perform the filtering if the
difference is smaller than a predetermined third threshold
determined according to the intensity level of illumination of
image. The first filter unit can be a Gaussian filter.
[0025] Also, the third determining unit determines the value of the
center element of the image data, received from the second
determining unit, as the output if the difference is substantially
the same as or larger than the third threshold and smaller than a
predetermined fourth threshold determined according to the edge
level of image, and the second determining unit determines to input
the image data, received from the second determining unit, into the
second filter unit and to perform the filtering if the difference
is substantially the same as or larger than the fourth
threshold.
[0026] The second filter unit has a first filter, determining the
horizontal direction edge of the image data; a second filter,
determining the vertical direction edge of the image data; a first
detecting unit, detecting the absolute value abs_S1 of the sum of
the elements of the output data of the first filter; a second
detecting unit, detecting the absolute value abs_S2 of the sum of
the elements of the output data of the second filter; a third
calculating unit, calculating the sum abs_S of the output data of
the first and second detecting units; and a processing unit,
determining the pertinent output data according to the output data
of the third calculating unit.
[0027] The processing unit determines output data as the value of
the center element of the image data if the abs_S is smaller than a
predetermined fifth threshold; determines the output data as the
horizontal direction average of the center row, the center of which
is added to the weight if the abs_S is larger than the fifth
threshold and the abs_S1 is larger than a value adding the abs_S2
to a predetermined sixth threshold; determines the output data as
the vertical direction average of the center column, the center of
which is added to the weight if the abs_S is larger than the fifth
threshold and the abs_S2 is larger than a value adding the abs_S1
to a predetermined sixth threshold; and determines the output data
as the horizontal/vertical direction average of the center
row/column, the centers of which are added to the weight if the
abs_S is larger than the fifth threshold and the abs_S1 is smaller
than the value adding the abs_S2 to the sixth threshold or the
abs_S2 is smaller than the value adding the abs_S1 to the sixth
threshold.
[0028] According to another embodiment of the present invention,
the threshold determining apparatus can include a first determining
unit, determining a standard deviation of noise for the image of
the flat surface photographed with the maximum exposure and the
minimum exposure, respectively; a second determining unit,
determining the maximum value and the minimum value of the
threshold by using the standard deviations of noise in the maximum
exposure and the minimum exposure, respectively, determined by the
first determining unit; and a third determining unit, determining
the threshold by using the maximum value and the minimum value of
the threshold, determined by the second determining unit, and the
maximum exposure and the minimum exposure.
[0029] At this time, the second determining unit determines the
maximum value of threshold by use of the following formula:
th.sub.max=.sigma..sub.max exp (where th.sub.max refers to the
maximum value of threshold, and .sigma..sub.max exp refers to the
standard deviation of noise in the maximum exposure), and the
second determining unit determines the minimum value of threshold
by use of the following formula: th.sub.min=.sigma..sub.min exp
(where th.sub.min refers to the minimum value of threshold, and
.sigma..sub.min exp refers to the standard deviation of noise in
the minimum exposure).
[0030] The third determining unit determines the threshold by use
of the following formula:
th = th max - th min Max Exposure - Min Exposure Exposure + Max
Exposure th min - Min Exposure th max Max Exposure - Min Exposure
##EQU00001##
[0031] (where Max Exposure refers to the maximum exposure value,
and Min Exposure refers to the minimum exposure value).
[0032] According to another embodiment of the present invention,
the threshold determining apparatus can include a first determining
unit, receiving the image of the flat surface photographed with the
maximum exposure and the minimum exposure, respectively, and
randomly acquiring samples from the mask, having a constant size,
of the received image and calculating a first difference and a
second difference, to calculate a first average of the first
difference and a second average of the second difference, the first
difference being the difference between the maximum value and the
minimum value of noise in the maximum exposure, and the second
difference being the difference between the maximum value and the
minimum value of noise in the minimum exposure; a second
determining unit, determining the maximum value and the minimum
value of the threshold by using the average of the difference
between the first average and the second average, calculated by the
first determining unit; and a third determining unit, determining
the threshold by using the maximum value and the minimum value of
the threshold, determined by the second determining unit, and the
maximum exposure and the minimum exposure.
[0033] The second determining unit determines the maximum value of
threshold by the following formula: th.sub.max=cliff (where
th.sub.max refers to the maximum value of threshold, and
diff.sub.max exp refers to the average value of the difference
between the maximum value and the minimum value of noise in the
maximum exposure), and the second determining unit determines the
minimum value of threshold by the following formula:
th.sub.min=diff.sub.min exp (where th.sub.min refers to the minimum
value of threshold, and diff.sub.min exp refers to the average
value of the difference between the maximum value and the minimum
value of noise in the minimum exposure).
[0034] The third determining unit determines the threshold by the
following formula:
th = th max - th min Max Exposure - Min Exposure Exposure + Max
Exposure th min - Min Exposure th max Max Exposure - Min Exposure
##EQU00002##
[0035] (where Max Exposure refers to the maximum exposure value,
and Min Exposure refers to the minimum exposure value).
[0036] To achieve the above objects, another aspect of the present
invention features the threshold determining method.
[0037] According to an embodiment of the present invention, the
threshold determining method can include determining a standard
deviation of noise for the image of the flat surface photographed
with the maximum exposure and the minimum exposure, respectively;
determining the maximum value and the minimum value of the
threshold by using the standard deviations of noise in the maximum
exposure and the minimum exposure, respectively, determined by the
operation of determining the standard deviation of noise; and
determining the threshold by using the maximum value and the
minimum value of the threshold, determined by the operation of
determining the maximum value and the minimum value of the
threshold, and the maximum exposure and the minimum exposure.
[0038] According to another embodiment of the present invention,
the threshold determining method can include receiving the image of
the flat surface photographed with the maximum exposure and the
minimum exposure, respectively, and randomly acquiring samples from
the mask, having a uniform size, of the received image and
calculating a first difference and a second difference, to
calculate a first average of the first difference and a second
average of the second difference, the first difference being the
difference between the maximum value and the minimum value of noise
in the maximum exposure, and the second difference being the
difference between the maximum value and the minimum value of noise
in the minimum exposure; determining the maximum value and the
minimum value of the threshold by using the average of the
difference between the first average and the second average,
calculated by the operation of calculating the first average and
the second average; and determining the threshold by using the
maximum value and the minimum value of the threshold, determined by
the operation of determining the maximum value and the minimum
value of the threshold, and the maximum exposure and the minimum
exposure.
BRIEF DESCRIPTION OF THE DRAWINGS
[0039] FIG. 1 is a simplified block diagram illustrating the
operation of a color interpolation apparatus in accordance with a
first embodiment of the present invention;
[0040] FIG. 2 is simplified block diagram illustrating the
interpolation performed by the color interpolation unit of FIG.
1;
[0041] FIG. 3 is a detailed block diagram illustrating an
embodiment of the color interpolation unit of FIG. 1;
[0042] FIG. 4 is a simplified block diagram of a Bayer pattern
image for illustrating the color interpolation performed by the
interpolating unit of FIG. 3;
[0043] FIG. 5 is another simplified block diagram of a Bayer
pattern image for illustrating the color interpolation performed by
the interpolating unit of FIG. 3;
[0044] FIG. 6 is an example illustrating a first Gaussian filter
and a second Gaussian filter of FIG. 3;
[0045] FIG. 7 through FIG. 9 are examples illustrating an edge
parameter inputted into the output/edge property determining unit
of FIG. 1;
[0046] FIG. 10 is a simplified block diagram generally illustrating
a color interpolation apparatus using an adjustable threshold in
accordance with a first embodiment of the present invention;
[0047] FIG. 11 is a block diagram illustrating a color
interpolation apparatus using an adjustable threshold in accordance
with a second embodiment of the present invention;
[0048] FIG. 12 is a detailed block diagram illustrating an
embodiment of a subsequent-processing unit of FIG. 11;
[0049] FIG. 13 is a conceptual diagram illustrating the operation
of the maximum value/minimum value determining unit of FIG. 11;
[0050] FIG. 14 is a detailed block diagram illustrating an
embodiment of the noise removing unit of FIG. 11;
[0051] FIG. 15 is a detailed block diagram illustrating the zipper
noise removing unit of FIG. 14;
[0052] FIG. 16 and FIG. 17 are examples illustrating each of the
first filter and the second filter of FIG. 15;
[0053] FIG. 18 is an example illustrating Data R_n inputted into
the first filter of FIG. 16 and the second filter of FIG. 17;
[0054] FIG. 19 is a simplified block diagram generally illustrating
a color interpolation apparatus using an adjustable threshold in
accordance with a second embodiment of the present invention;
[0055] FIG. 20 is a simplified block diagram illustrating the color
interpolation performed according to the threshold determined in
accordance with a third embodiment of the present invention;
[0056] FIG. 21 illustrates a property curve of camera exposure
depending on the surroundings;
[0057] FIGS. 22 and 23 illustrate a property curve of the noise
level of camera in accordance with the exposure;
[0058] FIG. 24 is a block diagram illustrating a threshold
determining apparatus in accordance with a third embodiment of the
present invention;
[0059] FIG. 25 illustrates a graph of the maximum value and minimum
value of the exposure in accordance with the intensity of
illumination;
[0060] FIG. 26 illustrates a graph of the maximum value and minimum
value of the threshold in accordance with the exposure;
[0061] FIG. 27 is a simplified block diagram illustrating the noise
removing performed according to the threshold determined in
accordance with a fourth embodiment of the present invention;
[0062] FIG. 28 is a block diagram illustrating a threshold
determining apparatus in accordance with a fourth embodiment of the
present invention;
[0063] FIG. 29 a block diagram illustrating a noise removing
apparatus applied with the threshold determined in accordance with
a fourth embodiment of the present invention;
[0064] FIG. 30 is a conceptual diagram illustrating the operation
of the difference calculating unit of FIG. 29;
[0065] FIG. 31 is a detailed block diagram illustrating an
embodiment of the difference calculating unit of FIG. 29; and
[0066] FIG. 32 is a detailed block diagram illustrating an
embodiment of the zipper noise removing unit of FIG. 29.
DETAILED DESCRIPTION
[0067] The above objects, features and advantages will become more
apparent through the below description with reference to the
accompanying drawings.
[0068] Since there can be a variety of permutations and embodiments
of the present invention, certain embodiments will be illustrated
and described with reference to the accompanying drawings. This,
however, is by no means to restrict the present invention to
certain embodiments, and shall be construed as including all
permutations, equivalents and substitutes covered by the spirit and
scope of the present invention. Throughout the drawings, similar
elements are given similar reference numerals. Throughout the
description of the present invention, when describing a certain
technology is determined to evade the point of the present
invention, the pertinent detailed description will be omitted.
[0069] Terms such as "first" and "second" can be used in describing
various elements, but the above elements shall not be restricted to
the above terms. The above terms are used only to distinguish one
element from the other. For instance, the first element can be
named the second element, and vice versa, without departing the
scope of claims of the present invention. The term "and/or" shall
include the combination of a plurality of listed items or any of
the plurality of listed items.
[0070] When one element is described as being "connected" or
"accessed" to another element, it shall be construed as being
connected or accessed to the other element directly but also as
possibly having another element in between. On the other hand, if
one element is described as being "directly connected" or "directly
accessed" to another element, it shall be construed that there is
no other element in between.
[0071] The terms used in the description are intended to describe
certain embodiments only, and shall by no means restrict the
present invention. Unless clearly used otherwise, expressions in
the singular number include a plural meaning. In the present
description, an expression such as "comprising" or "consisting of"
is intended to designate a characteristic, a number, a step, an
operation, an element, a part or combinations thereof, and shall
not be construed to preclude any presence or possibility of one or
more other characteristics, numbers, steps, operations, elements,
parts or combinations thereof.
[0072] Unless otherwise defined, all terms, including technical
terms and scientific terms, used herein have the same meaning as
how they are generally understood by those of ordinary skill in the
art to which the invention pertains. Any term that is defined in a
general dictionary shall be construed to have the same meaning in
the context of the relevant art, and, unless otherwise defined
explicitly, shall not be interpreted to have an idealistic or
excessively formalistic meaning.
[0073] Hereinafter, preferred embodiments will be described in
detail with reference to the accompanying drawings. Identical or
corresponding elements will be given the same reference numerals,
regardless of the figure number, and any redundant description of
the identical or corresponding elements will not be repeated.
[0074] FIG. 1 is a simplified block diagram illustrating the color
interpolation apparatus in accordance with a first embodiment of
the present invention.
[0075] As illustrated of FIG. 1, the color interpolation apparatus
of the present invention can include a color interpolation unit
110, an output/edge property determining unit 120, a RGB converting
unit 130 and an edge enhancement unit 140.
[0076] The color interpolation unit 110 performs the color
interpolation of an inputted Bayer pattern image. This will be
described with reference to FIG. 3.
[0077] FIG. 2 is a simplified block diagram illustrating the
interpolation performed by the color interpolation unit of FIG.
1.
[0078] As illustrated of FIG. 2, once a 5.times.5 Bayer pattern
image 200 is inputted, the color interpolation unit 110 of the
present invention performs the bilinear interpolation of the
5.times.5 Bayer pattern image 200 to obtain a 3.times.3 output
image 201. The 3.times.3 output image 201 is referred to as R_all,
G_all and B_all.
[0079] If the interpolation of the present invention (this will be
described later) for the 3.times.3 output image 201 is first
performed, 1-pixel data 202 will be taken. This 1-pixel data 202
becomes 1-pixel output data E_R, E_G and E_B 203. If the edge
calculation of the 1-pixel data 202 is carried out, it will become
1-pixel output data E_Y 204.
[0080] If the edge calculation (this will be described later) of
the 3.times.3 output image 201 is performed, a 3.times.3 image
Y_all 205 will be taken. If the Gaussian filtering of the 3.times.3
image 205 is performed, a last 1-pixel output data G_Y 206 will be
taken. On the other hand, if the special value (the difference
between the maximum value and the minimum value) of the 3.times.3
image Y_all 205 is calculated, 1-pixel data diff_Y 207 will be
taken.
[0081] Also, if the Gaussian filtering of the 3.times.3 image 201
is carried out, 1-pixel output data G_R, G_G and G_B, which becomes
one output.
[0082] Accordingly, the color interpolation unit 110 can be
inputted with the 5.times.5 Bayer pattern image 200 and output 5
1-pixel data 203, 204, 206, 207 and 210. Hereinafter, the detailed
structure of the color interpolation unit 110 will be described
with reference to FIG. 3.
[0083] FIG. 3 is a detailed block diagram illustrating an
embodiment of the color interpolation unit of FIG. 1.
[0084] As illustrated of FIG. 3, the color interpolation unit 110
of the present invention can have a bilinear interpolation unit
310, an interpolation unit 320, a first edge calculation unit 330,
a second edge calculation unit 340, a first Gaussian filter 350, a
difference determining unit 360 and a second Gaussian filter
370.
[0085] The bilinear interpolation unit 310 performs the bilinear
interpolation of an inputted 5.times.5 Bayer pattern image and
outputs 3.times.3 data. Since this bilinear interpolation is the
same as the already-known technology, the pertinent detailed
description will be omitted.
[0086] The interpolation unit 320 performs the color interpolation
of the inputted 3.times.3 data. This operation will be described
with reference to the corresponding drawing.
[0087] FIG. 4 is a simplified block diagram of the Bayer pattern
image for illustrating the color interpolation performed by the
interpolating unit of FIG. 3 in case that the R3 of R component is
a center pixel in an RG line. Referring to FIG. 4, the below
description relates to the case that the R component is the center
pixel. However, considering that the R component and B component
exchange their locations with each other, the same description can
be applied to the case that the center pixel is the B component in
a GB line.
[0088] For the interpolation performed by the interpolation unit
320, Rn, Rw, Rs and Re and Kr1, Kr2, Kr3 and Kr4 of FIG. 4 are
defined as the following formula 1 and formula 2, respectively.
Rn = R 1 + R 3 2 Rw = R 2 + R 3 2 Rs = R 3 + R 5 2 Re = R 3 + R 4 2
[ Formula 1 ] Kr 1 = G 4 - Rn Kr 2 = G 6 - Rw Kr 3 = G 9 - Rs Kr 4
= G 7 - Re [ Formula 2 ] ##EQU00003##
[0089] Then, the output E_G of the G component of the interpolation
unit 320 is calculated by the following formula 3.
E_G = R 3 + Kr 1 + Kr 2 + Kr 3 + Kr 4 4 [ Formula 3 ]
##EQU00004##
[0090] Since the interpolation method of the interpolation unit 320
is the same as the well-known prior art (e.g. Soo-Chang Pei,
Fellow, IEEE, Io-Kuong Tam, "Effective Color Interpolation in CCD
Color Filter Arrays Using Signal Correlation," IEEE transaction on
circuits and systems for video technology, Vol. 13(6), June 2003),
the pertinent detailed description will be omitted.
[0091] Gwn, Gws, Ges and Gen and Kb1, Kb2, Kb3 and KB4 are defined
as the following formula 4 and formula 5, respectively, to
calculate the output E_B of the interpolated B component and the
output E_R of the interpolated R component.
Gwn = G 1 + G 3 + G 6 + G 4 4 Gws = G 6 + G 8 + G 11 + G 9 4 Ges =
G 7 + G 9 + G 12 + G 10 4 Gen = G 2 + G 4 + G 7 + G 5 4 [ Formula 4
] Kb 1 = Gwn - B 1 Kb 1 = Gwn - B 1 Kb 2 = Gen - B 2 Kb 3 = Gws - B
3 Kb 4 = Ges - B 4 [ Formula 5 ] ##EQU00005##
[0092] By the formula 4 and formula 5, the output E_B of the
interpolated B component and the output E_R of the interpolated R
component are determined by the formula 6 and the formula 7,
respectively.
E_B = E_G - Kb 1 + Kb 2 + Kb 3 + Kb 4 4 [ Formula 6 ] E_R = R 3 [
Formula 7 ] ##EQU00006##
[0093] On the other hand, FIG. 5 is another simplified block
diagram of the Bayer pattern image for illustrating the color
interpolation performed by the interpolating unit of FIG. 3 in case
that the G5 of the G component is the center pixel in a GB line.
Referring to FIG. 5, the below description relates to the case that
the G component is the center pixel in the GB line. However,
considering that the R component and a B component exchange their
locations with each other in the Bayer pattern image, the same
description can be applied to the case that the center pixel is the
G component in the RB line.
[0094] Gn, Gw, Gs and Ge and Kr1, Kr2, Kb1 and Kb2 are defined as
the following formula 8 and formula 9, respectively, to perform the
color interpolation of the Bayer pattern image of FIG. 5.
Gn = G 1 + G 2 + G 5 + G 3 4 Gw = G 2 + G 4 + G 7 + G 5 4 Gs = G 5
+ G 7 + G 9 + G 8 4 Ge = G 3 + G 5 + G 8 + G 6 4 [ Formula 8 ] Kr 1
= Gn - R 1 Kr 2 = Gs - R 2 Kb 1 = Gw - B 1 Kb 2 = Ge - B 2 [
Formula 9 ] ##EQU00007##
[0095] By the formula 8 and formula 9, the outputs E_G, E_R and E_B
are determined by the formula 10 through the formula 12,
respectively in the case of FIG. 5.
E_G = G 5 [ Formula 10 ] E_R = G 5 - Kr 1 + Kr 2 2 [ Formula 11 ]
E_B = G 5 - Kb 1 + Kb 2 2 [ Formula 12 ] ##EQU00008##
[0096] In case the center pixel is the G component, the R component
or the B component, each of the corresponding outputs is determined
as described above.
[0097] The first edge calculating unit 330 performs the edge
calculation of the outputs E_G, E_R and E_B of the interpolation
320. This edge calculation is represented as the following formula
13.
E_Y = E_R + 2 .times. E_G + E_B 4 [ Formula 13 ] ##EQU00009##
[0098] where E_Y is defined as the output data of the first edge
calculating unit 330.
[0099] The second edge calculating unit 340, which receives
3.times.3 data from the bilinear interpolation unit 310, performs
the edge calculation to carry out the edge detection of the
3.times.3 data. This edge calculation is the same as the following
formula 14.
Y_all = R_all + 2 .times. G_all + B_all 4 [ Formula 14 ]
##EQU00010##
[0100] The 3.times.3 data Y_all is taken by calculating each pixel
of the inputted 3.times.3 data by use of the formula 14. If the
3.times.3 data Y_all is passed through the first Gaussian filter
350, it becomes 1-pixel data G_Y. FIG. 6 is an example illustrating
the first Gaussian filter and the second Gaussian filter of FIG.
3
[0101] The difference determining unit 360 determines the
difference between the maximum value and the minimum value of
elements in the 3.times.3 output data of the second edge
calculating unit 340. This is represented by the following formula
15.
diff.sub.--Y=max(Y_all)-min(Y_all) [Formula 15]
[0102] Also, the second Gaussian filter 370 can take the output
data G_R, G_G and G_B by Gaussian filtering the 3.times.3 output
data of the bilinear interpolation unit 310.
[0103] As such, the output data of the color interpolation unit 110
of FIG. 1 is 1-pixel data E_R, E_G and E_B, 1-pixel data E_Y,
1-pixel data G_Y, 1-pixel data diff_Y and 1-pixel data G_R, G_G and
G_B.
[0104] On the other hand, the output/edge property determining unit
120 receives the output data of the color interpolation unit 110
and an edge parameter, and determines not only RGB output data but
also an edge enhancement coefficient and an edge flag.
[0105] The output/edge property determining unit 120 preset a first
threshold thr1 and a second threshold thr2, which are determined
and inputted in the previous system. These are varied depending on
the system. The first threshold is determined depending on whether
an image is flat. The second threshold is determined depending on
the edge level.
[0106] In case that the difference diff_Y received from the color
interpolation unit 110 is smaller than the first threshold, it is
determined that the corresponding image is flat (e.g. a wall part)
and the corresponding output is the output data of the second
Gaussian filter 370. That is, this is represented by the following
formula 16.
R_out=G.sub.--R
G_out=G.sub.--G
B_out=G.sub.--B [Formula 16]
[0107] If the difference diff_Y is the same as or larger than the
first threshold and smaller than the second threshold, it is
determined that the corresponding image has no edge and is not
flat, that is, the image has properties such as an outline (e.g. at
this time, referring to as a typical image). Also, the output/edge
property determining unit 120 determines the corresponding output
is the output data of the interpolation unit 320 and the
corresponding edge flag is zero. In this case, the output is
represented by the following formula 17.
R_out=E.sub.--R
G_out=E.sub.--G
B_out=E.sub.--B [Formula 17]
[0108] If the difference diff_Y is the same as or larger than the
second threshold, it is determined that the corresponding image has
an edge. Also, the output/edge property determining unit 120
determines the corresponding output is the output data of the
interpolation unit 320 and the corresponding edge flag is 1.
[0109] The output/edge property determining unit 120 further
generates an edge enhancement coefficient by using the inputted
edge parameter. This edge parameter includes edge coring, edge gain
and edge limit. This will be described with reference to FIG. 7
through FIG. 9.
[0110] FIG. 7 through FIG. 9 are examples illustrating the edge
parameter inputted into the output/edge property determining unit
of FIG. 1.
[0111] The edge coring not only removes a noise-like edge component
but also prevents small edge components from being displayed like
the noise on a screen. As illustrated in FIG. 7, a predetermined
value is subtracted from the inputted edge value, and the edge
components under the value, inclusive, become zero.
[0112] As illustrated in FIG. 8, the edge gain is performed for the
edge value passing through the edge coring. The difference between
a large edge value and a small edge value is generated by changing
the result value gradient of an outputted edge component to an
inputted edge component. Accordingly, more clear and improved image
quality is provided by making a clear edge component clearer and
maintaining a small edge component unchangeable.
[0113] In case that the absolute value of the edge component,
extracted from the edge gain operation, is very large, the edge
limit restricts the maximum edge value to prevent the edge
component having large absolute value from being specifically
displayed on a screen. The edge limit has positive limit and
negative limit.
[0114] The edge enhancement coefficient is determined by the edge
parameter and the output data E_Y and G_Y, detected from the color
interpolation unit 110. This is represented by the following
formula 18.
Srgb=edge_enhancement(edge_component,edge_parameter) [Formula
18]
[0115] where Srgb is the edge enhancement coefficient and
"edge_component" is the difference between E_Y and G_Y. The edge
enhancement coefficient is a function of the edge component and the
edge parameter, referring to "edge_component," determined depending
on the system. The edge enhancement coefficient value is not
limited.
[0116] Accordingly, the output/edge property determining unit 120
outputs the data R_out, the G_out, B_out and the Srgb and the edge
flag.
[0117] The RGB converting unit 130 receives the data R_out, G_out
and B_out and converts the data R_out and B_out to luminance and
chrominance YCbCr data. This is represented by the following
formula 19.
( Y Cb Cr ) = ( 0.299 0.587 0.114 - 0.169 - 0.331 0.5 0.5 - 0.419 -
0.081 ) .times. ( R G B ) - ( 0 128 128 ) [ Formula 19 ]
##EQU00011##
[0118] However, since the formula of converting RGB data to YCbCr
data is well known, the formula is not limited to the above formula
19. The converting ratio can be varied depending on the kind of the
image processing system.
[0119] The edge enhancement unit 140 enhances the edge by using the
luminance (Y) data, received from the above RGB converting unit 130
and the edge enhancement coefficient Srgb and the edge flag,
received from the output/edge property determining unit 120.
[0120] If the edge flag is 1, the output data of the edge
enhancement unit 140 is calculated by the following formula 20.
Y_out=Y+Srgb [Formula 20]
[0121] However, if the edge flag is zero, since it is determined
that the corresponding image has no edge in the above, the output
data of the edge enhancement unit 140 bypasses the inputted Y
data.
[0122] FIG. 10 is a simplified block diagram generally illustrating
the color interpolation apparatus using an adjustable threshold in
accordance with a first embodiment of the present invention.
[0123] As illustrated in FIG. 10, the color interpolation apparatus
of the present invention calculates the difference diff_Y. Then, if
the difference diff_Y is smaller than the first threshold, the
color interpolation apparatus performs the bilinear interpolation
and Gaussian filtering. If the difference diff_Y is smaller than
the second threshold and the same as or larger than the first
threshold, the color interpolation apparatus performs the bilinear
interpolation and the interpolation unit 320 of FIG. 3 performs the
interpolation (referred to as "effective interpolation" in FIG.
10). Also, if the difference diff_Y is the same as or larger than
the second threshold, the effective interpolation and the edge
enhancement are performed.
[0124] FIG. 11 is a block diagram illustrating the color
interpolation apparatus using an adjustable threshold in accordance
with a second embodiment of the present invention.
[0125] As illustrated in FIG. 11, the color interpolation apparatus
of the present invention includes a color interpolation unit 1110,
an output/edge property determining unit 1120, a
subsequent-processing unit 1130, a maximum value/minimum value
determining unit 1140 and a noise removing unit 1150.
[0126] The color interpolation unit 1110 performs the color
interpolation of an inputted Bayer pattern image. This will be
described with reference to the respective drawing.
[0127] Hereinafter, the method of the interpolation performed by
the color interpolation unit 1110, illustrated in FIG. 11, will be
described with reference to FIG. 2. The sizes of the Bayer pattern
image and the output data, which are different from FIG. 2, are
understandable through the below description. Further, for the
convenience of description, although the same formula as mentioned
in the first description of the first embodiment is used, a new
reference number will be added to the same formula.
[0128] Once a 7.times.7 Bayer pattern image 200 is inputted, the
color interpolation unit 1110 of the present invention performs the
bilinear interpolation of the 7.times.7 Bayer pattern image 200 to
take a 5.times.5 output image 201. The interpolation of this
5.times.5 output image 201 is performed to take 3.times.3 data 202.
This 3.times.3 data 202 becomes 3.times.3 output data E_R, E_G and
E_B 203. The edge calculation of the 3.times.3 data 202 is
performed to take 3.times.3 output data E_Y 204.
[0129] Again, the edge calculation of the 5.times.5 output data 201
can be performed to take 5.times.5 data Y_all 205. The Gaussian
filtering of the 5.times.5 data Y_all 205 can make 3.times.3 output
data G_Y 206. Also, if the difference between the maximum value and
the minimum value of elements of the data Y_all 205 is calculated,
1-pixel data diff_Y 207 can be taken.
[0130] Finally, the Gaussian filtering of the 5.times.5 output data
201 can be carried out to obtain 3.times.3 data G_R, G_G and G_B
210.
[0131] The color interpolation unit 1110 receives the 7.times.7
Bayer pattern image 200 and outputs 5 3.times.3 data 203, 204, 206,
207 and 210.
[0132] The structure of the color interpolation unit 1110 will be
now described. The detailed block diagram of the color
interpolation unit 1110 of FIG. 11 can be the same as described
with reference to FIG. 3. The color interpolation 1110 can have a
bilinear interpolation unit 310, an interpolation unit 320, a first
edge calculation unit 330, a second edge calculation unit 340, a
first Gaussian filter 350, a difference determining unit 360 and a
second Gaussian filter 370. The function of each element, however,
can be varied due to the size difference of the inputted Bayer
pattern image.
[0133] Referring to FIG. 3, the bilinear interpolation unit 310
performs the bilinear interpolation of the inputted 7.times.7 Bayer
pattern image and outputs 5.times.5 data.
[0134] The interpolation unit 320 performs the color interpolation
of the inputted 5.times.5 data. This operation will be described
with reference to the pertinent drawing.
[0135] As described above, FIG. 4 is the simplified block diagram
of the Bayer pattern image for illustrating the color interpolation
performed by the interpolating unit 320 in case that the R3 of R
component is the center pixel in an RG line. Referring to FIG. 4,
the below description relates to the case that the R component is
the center pixel. However, the same description can be applied to
the case that the center pixel is the B component in a GB line.
[0136] For the interpolation performed by the interpolation unit
320, Rn, Rw, Rs and Re and Kr1, Kr2, Kr3 and Kr4 of FIG. 4 are
defined as the following formula 21 and formula 22,
respectively.
Rn = R 1 + R 3 2 Rw = R 2 + R 3 2 Rs = R 3 + R 5 2 Re = R 3 + R 4 2
[ Formula 21 ] Kr 1 = G 4 - Rn Kr 2 = G 6 - Rw Kr 3 = G 9 - Rs Kr 4
= G 7 - Re [ Formula 22 ] ##EQU00012##
[0137] Then, the output data of E_G the G component of the
interpolation unit 320 is calculated by the following formula
23.
E_G = R 3 + Kr 1 + K r 2 + Kr 3 + Kr 4 4 [ Formula 23 ]
##EQU00013##
[0138] Since the interpolation method of the interpolation unit 320
is the same as the well-known prior art (e.g. Soo-Chang Pei,
Fellow, IEEE, Io-Kuong Tam, "Effective Color Interpolation in CCD
Color Filter Arrays Using Signal Correlation", IEEE transaction on
circuits and systems for video technology, Vol. 13(6), June 2003),
the pertinent detailed description will be omitted.
[0139] Gwn, Gws, Ges and Gen and Kb1, Kb2, Kb3 and KB4 are defined
as the following formula 24 and formula 25, respectively, to
calculate the output data E_B of the interpolated B component and
the output data E_R of the interpolated R component.
Gwn = G 1 + G 3 + G 6 + G 4 4 Gws = G 6 + G 8 + G 11 + G 9 4 Ges =
G 7 + G 9 + G 12 + G 10 4 Gen = G 2 + G 4 + G 7 + G 5 4 [ Formula
24 ] Kb 1 = G wn - B 1 Kb 2 = G en - B 2 Kb 3 = G ws - B 3 Kb 4 =
Ges - B 4 [ Formula 25 ] ##EQU00014##
[0140] By the formula 24 and formula 25, the output E_B of the
interpolated B component and the output ER of the interpolated R
component are determined by the formula 26 and the formula 27,
respectively.
E_B = E_G - Kb 1 + Kb 2 + Kb 3 + Kb 4 4 [ Formula 26 ] E_R = R 3 [
Formula 27 ] ##EQU00015##
[0141] This interpolation is applied to a 5.times.5 matrix to
output 3.times.3 output data E_R, E_G and E_B.
[0142] Also, as described below, FIG. 5 is another simplified block
diagram of the Bayer pattern image for illustrating the color
interpolation performed by the interpolating unit 320 in case that
the G5 of G component is the center pixel in a GB line. Referring
to FIG. 5, the below description relates to the case that the G
component is the center pixel in the GB line. However, the same
description can be applied to the case that the center pixel is the
G component in an RB line.
[0143] Gn, Gw, Gs and Ge and Kr1, Kr2, Kb1 and Kb2 are defined as
the following formula 28 and formula 29, respectively, to perform
the color interpolation of the Bayer pattern image of FIG. 5.
Gn = G 1 + G 2 + G 5 + G 3 4 Gw = G 2 + G 4 + G 7 + G 5 4 Gs = G 5
+ G 7 + G 9 + G 8 4 Ge = G 3 + G 5 + G 8 + G 6 4 [ Formula 28 ] Kr
1 = G n - R 1 Kr 2 = G s - R 2 Kb 1 = G w - B 1 Kb 2 = Ge - B 2 [
Formula 29 ] ##EQU00016##
[0144] By the formula 28 and formula 29, the output data E_G, E_R
and E_B are determined by the formula 30 through the formula 32,
respectively in the case of FIG. 5.
E_G = G 5 [ Formula 30 ] E_R = G 5 - Kr 1 + Kr 2 2 [ Formula 31 ]
E_B = G 5 - Kb 1 + Kb 2 2 [ Formula 32 ] ##EQU00017##
[0145] In case the center pixel is the G component, the R component
or the B component, each of the corresponding outputs is determined
as described above. This interpolation is applied to a 5.times.5
matrix to output 3.times.3 output data E_R, E_G and E_B.
[0146] The first edge calculating unit 330 of FIG. 30 performs the
edge calculation of the output data E_G, E_R and E_B of the
interpolation 320. This edge calculation is represented by the
following formula 33.
E_Y = E_R + 2 .times. E_G + E_B 4 [ Formula 33 ] ##EQU00018##
[0147] where E_y is defined as the output data of the first edge
calculating unit 330. The above calculation of each element forming
the 3.times.3 data can be performed to take the 3.times.3 data
E_Y.
[0148] The edge calculating unit, which receives 5.times.5 data
from the bilinear interpolation unit 310, performs the edge
calculation to carry out the edge detection of the 5.times.5 data.
This edge calculation is the same as the following formula 34.
Y_all = R_all + 2 .times. G_all + B_all 4 [ Formula 34 ]
##EQU00019##
[0149] The 5.times.5 data Y_all is taken by calculating each
element of the inputted 5.times.5 data by use of the formula 34. If
the 5.times.5 data Y_all is passed through the first Gaussian
filter 350, it becomes 3.times.3 data G_Y.
[0150] The difference determining unit 360 determines the
difference between the maximum value and the minimum value of the
elements of the 5.times.5 output data of the second edge
calculating unit 340. This is represented by the following formula
35.
diff.sub.--Y=max(Y_all)-min(Y_all) [Formula 35]
[0151] Also, in the color interpolation unit 1110 of the present
invention, the second Gaussian filter 370 can take the 3.times.3
output data G_R, G_G and G_B by Gaussian filtering of the 5.times.5
output data of the bilinear interpolation unit 310.
[0152] As such, the output data of the color interpolation unit
1110 is 3.times.3 data E_R, E_G and E_B, 3.times.3 data E_Y and
G_Y, 1-pixel data diff_Y and 3.times.3 data G_R, G_G and G_B.
[0153] On the other hand, the output/edge property determining unit
1120 receives the output data of the color interpolation unit 1110
and an edge parameter and determines not only the RGB output data
but also an edge enhancement coefficient and an edge flag.
[0154] The output/edge property determining unit 1120 preset a
first threshold thr1 and a second threshold thr2, which are
determined and inputted in a previous system. These are varied
depending on the system. The first threshold is determined
depending on whether an image is flat. The second threshold is
determined depending on the edge level.
[0155] In case that the difference diff_Y received from the color
interpolation unit 1110 is smaller than the first threshold, it is
determined that the corresponding image is flat (e.g. a wall part)
and the corresponding output is the output data of the second
Gaussian filter 370. That is, this is represented by the following
formula 36.
R.sub.--m=G.sub.--R
G.sub.--m=G.sub.--G
B.sub.--m=G.sub.--B [Formula 36]
[0156] If the difference diff_Y is the same as or larger than the
first threshold and smaller than the second threshold, it is
determined that the corresponding image has no edge and is not
flat, that is, the image has properties such as an outline (e.g. at
this time, referring to as a typical image). Also, the output/edge
property determining unit 1120 determines the corresponding output
is the output data of the interpolation unit 320 and the
corresponding edge flag is zero. In this case, the output is
represented by the following formula 37.
R.sub.--m=E.sub.--R
G.sub.--m.sub.--E.sub.--G
B.sub.--m=E.sub.--B [Formula 37]
[0157] If the difference diff_Y is the same as or larger than the
second threshold, it is determined that the corresponding image has
an edge. Also, the output/edge property determining unit 1120
determines the corresponding output is the output data of the
interpolation unit 320 and the corresponding edge flag is 1.
[0158] The output/edge property determining unit 1120 further
generates an edge enhancement coefficient by using the inputted
edge parameter. This edge parameter includes edge coring, edge gain
and edge limit. The edge parameter will be described with reference
to FIG. 7 through FIG. 9 in the description of the first
embodiment. That is, referring to FIG. 7 through FIG. 9, the edge
parameter inputted into the output/edge property determining unit
1120 will be described.
[0159] The edge coring not only removes a noise-like edge component
also prevents small edge components from being displayed like noise
on a screen. As illustrated in FIG. 7, a predetermined value is
subtracted from the inputted edge value, and the edge components
under the value, inclusive, become zero.
[0160] As illustrated in FIG. 8, the edge gain is performed for the
edge value passing through the edge coring. The difference between
a large edge value and a small edge value is generated by changing
the result value gradient of an outputted edge component to an
inputted edge component. Accordingly, more clear and improved image
quality is provided by making a clear edge component clearer and
maintaining a small edge component unchangeable.
[0161] In case that the absolute value of the edge component,
extracted from the edge gain operation, is very large, the edge
limit restricts the maximum edge value to prevent the edge
component having large absolute value from being specifically
displayed on a screen. The edge limit has positive limit and
negative limit.
[0162] The edge enhancement coefficient is determined by the edge
parameter and the output data E_Y and G_Y, detected from the color
interpolation unit 110. This is represented by the following
formula 38.
Srgb=edge_enhancement(edge_component,edge_parameter) [Formula
38]
[0163] where Srgb is the edge enhancement coefficient and
"edge_component" is the difference between E-Y and G_Y. The edge
enhancement coefficient is a function of the edge component and the
edge parameter, referring to "edge_component," determined depending
on the system. The edge enhancement coefficient value is not
limited. At this time, the edge enhancement coefficient also has
the 3.times.3 size.
[0164] Accordingly, the output/edge property determining unit 1120
outputs the data R_m, the G_m, B_m and the Srgb and the edge
flag.
[0165] The subsequent-processing part 1130 performs the edge
enhancement of the inputted RGB data to prevent false color from
being generated, in case that the edge flag is 1. In other words,
in case that the flag outputted from the output/edge property
determining unit 1120 is zero, the output data R_n, G_n and B_n of
the subsequent-processing part 1130 is the same as the data R_m,
G_m and B_m, that is, is bypassed.
[0166] FIG. 12 is a detailed block diagram illustrating an
embodiment of the subsequent-processing unit of FIG. 11.
[0167] The subsequent-processing unit 1130 can include a sign
generating unit 1131, a Sr generating unit 1132, a Sg generating
unit 1133, a Sb generating unit 1134, a minimum value generating
unit 1135, a new edge enhancement coefficient generating unit 1136
and an output unit 1137.
[0168] The sign generating unit 1131 determines and generates the
sign of the inputted edge enhancement coefficient Srgb. This is
represented by the following formula 39.
PN=sign(Srgb(i)) [Formula 39]
[0169] where PN is the sign of the edge enhancement coefficient,
and i represents that the sign of each element of the 3.times.3
data is generated. The below description is the same as this.
[0170] When the value of the inputted data R_m added to the edge
enhancement coefficient Srgb is referred to as R1, (i.e.
R1=R_m(i)+Srgb(i), if the R1 is larger than a predetermined
constant (e.g. 255), the Sr generating unit 1132 determines Sr by
using the following formula 40; if the R1 is smaller than zero, the
Sr generating unit 1132 determines Sr by using the following
formula 41; and otherwise, the Sr generating unit 1132 determines
Sr by using the following formula 42. At this time, the Sr refers
to the subsequent-processing constant of the R component. If Sg and
Sb similarly refer to the subsequent-processing constants of the G
and B components, respectively, the same process can be applied to
the Sg and Sb.
Sr=255-R.sub.--m(i) [Formula 40]
Sr=R.sub.--m(i) [Formula 41]
Sr=abs(Srgb(i)) [Formula 42]
[0171] At this time, "abs" represents the absolute value, and the
constant "255" is varied depending on each system. The constant is
not limited to 255.
[0172] Similarly, when the value of the data G_m and B_m,
respectively, added to the edge enhancement coefficient Srgb is
referred to as G1 and B1, (i.e. G1=G_m(i)+Srgb(i) and
B1=B_m(i)+Srgb(i)), if each of the G1 and B1 is larger than a
predetermined constant (e.g. 255), the Sg generating unit 1133 and
the Sb generating unit 134 determine Sg and Sb, respectively, by
using the following formulas 43 and 46; if the R1 is smaller than
zero, the Sg generating unit 1133 and the Sb generating unit 134
determine Sg and Sb, respectively, by using the following formulas
44 and 47; and otherwise, the Sg generating unit 1133 and the Sb
generating unit 134 determine Sg and Sb, respectively, by using the
following formulas 45 and 48.
Sg=255-G.sub.--m(i) [Formula 43]
Sg=G.sub.--m(i) [Formula 44]
Sg=abs(Srgb(i)) [Formula 45]
Sb=255-B.sub.--m(i) [Formula 46]
Sb=B.sub.--m(i) [Formula 47]
Sb=abs(Srgb(i)) [Formula 48]
[0173] Then, the minimum value generating unit 1135 generates the
minimum value of the above subsequent-processing coefficients Sr,
Sg and Sb, and the new enhancement coefficient generating unit 1136
generates a new edge enhancement coefficient by using this minimum
value and the sign PN, received from the sign generating unit 1131.
This is represented by the following formula 49.
New.sub.--Srgb=PN.times.min([Sr,Sg,Sb]) [Formula 49]
[0174] where New_Srgb is the new edge enhancement coefficient. By
using the new edge enhancement coefficient, the output unit 1137
outputs output data having enhanced edge. The output data R_n, G_n
and B_n of the RGB components outputted from the output unit 1137
are calculated by the following formulas 50 through 52,
respectively.
R.sub.--n(i)=R.sub.--m(i)+New.sub.--Srgb [Formula 50]
G.sub.--n(i)=G.sub.--m(i)+New.sub.--Srgb [Formula 51]
B.sub.--n(i)=B.sub.--m(i)+New.sub.--Srgb [Formula 52]
[0175] Then, the maximum value/minimum value determining unit 1140
determines the maximum value and minimum value of each element of
the inputted data R_n, G_n, and B_n. This is illustrated by FIG.
13. FIG. 13 is a conceptual diagram illustrating the operation of
the maximum value/minimum value determining unit 1140 of FIG.
11.
[0176] The 3.times.3 data R_n, G_n and B_n inputted from the
subsequent-processing unit 1130 refers to data RGB_n. Then, the
maximum value/minimum value determining unit 1140 outputs the
corresponding data RGB_n 1340 and the minimum value min_RGB 1320
and the maximum value max_RGB 1330 of the corresponding element. At
this time, the min_RGB refers to min_R, min_G and min_B, and the
max_RGB refers to max_R, max_G and max_B.
[0177] Then, the noise removing unit 1150 determines a noise
removing method in accordance with the threshold and output the
noise-removed data according to the determined noise removing
method. This will be described with reference to the pertinent
drawing.
[0178] FIG. 14 is a detailed block diagram illustrating an
embodiment of the noise removing unit 1150 of FIG. 11.
[0179] As illustrated in FIG. 14, the noise removing unit 1150 of
FIG. 11 can have a noise removing method determining unit 1151, a
zipper noise removing unit 1152 and a Gaussian filtering unit
1153.
[0180] The noise removing method determining unit 1151 preset a
third threshold thr 3 and a fourth threshold thr 4, which are first
determined and inputted from a previous system. These depend on the
system. The third threshold is determined depending on whether the
corresponding image is flat, and the fourth threshold is determined
depending on the edge level.
[0181] In case that the difference diff_Y received from the color
interpolation unit 1110 is smaller than the third threshold, the
noise removing method determining unit 1151 determines that the
corresponding image is flat (e.g. a wall part) and performs a
typical filtering. In other words, the noise removing method
determining unit 1151 determines to apply the Gaussian filter unit
1153 to remove the noise. The Gaussian filter unit 1153 performs
the Gaussian filtering of this 3.times.3 RGB data.
[0182] An example of the Gaussian filter unit of FIG. 14 can be
used by the same description in the first embodiment with reference
to FIG. 6. However, it shall be evident that the present invention
is not limited to the Gaussian filtering.
[0183] If the difference diff_Y is the same as or larger than the
third threshold and smaller than the fourth threshold, the noise
removing method determining unit 1151 determines that the
corresponding image has no edge and is not flat, that is, the image
has properties such as an outline (e.g. at this time, referring to
as a typical image). Also, the noise removing method determining
unit 1151 determines the output data R_out, G_out and B_out is the
center element value of the data. That is, the noise removing
method determining unit 1151 determines that the image of this area
is not necessary to remove the noise.
[0184] If the difference diff_Y is the same as or larger than the
fourth threshold, the noise removing method determining unit 1151
determines that the corresponding image has an edge and determines
to perform the zipper noise removing. Accordingly, the zipper noise
removing unit 1152 performs the zipper noise removing. This will be
described with reference to the pertinent drawing.
[0185] FIG. 15 is a detailed block diagram illustrating the zipper
noise removing unit 1152 of FIG. 14.
[0186] As illustrated in FIG. 15, the noise removing unit 1152 can
include a first filter 1510, a second filter 1520, a first absolute
value detecting unit 1530, and a second absolute value detecting
unit 1540, a sum unit 1550 and a processing unit 1560.
[0187] The first and second filter 1510 and 1520, which can be a
3.times.3 filter, determines how many horizontal and vertical
direction edge components of inputted RGB components are. For this,
the inputted RGB components are also 3.times.3 data. FIG. 16 and
FIG. 17 are examples illustrating the first filter and the second
filter, respectively, of FIG. 15. FIG. 18 is an example
illustrating Data R_n inputted into the first filter of FIG. 16 and
the second filter of FIG. 17. However, although the illustration of
FIG. 18 assumes that the data R_n is inputted into the first filter
1510 and the second filter 1520, the same illustration can be
applied to the data G_n and B_n.
[0188] The first filter of FIG. 16 and the second filter of FIG. 17
are to determine the horizontal and vertical direction edge
components of inputted data, respectively. Once the data R_n is
inputted as illustrated in FIG. 18, elements are multiplied to each
component at the same location. The symbol "x" of FIG. 18 refers to
the multiplication of elements at the same position.
[0189] The first absolute value detecting unit 1530 sums the
elements from the output data of the first filter 1510 and detects
the absolute value of the sum of the elements. The output data of
the first absolute value detecting unit 1530 is performed by the
following formula 53.
abs.sub.--S1=abs(R7+2.times.R8+R9-2.times.R2-R3) [Formula 53]
[0190] where "abs" refers to the absolute value. The second
absolute value detecting unit 1540 sums the elements from the
output data of the second filter 1520 and detects the absolute
value of the sum of the elements. The output data of the second
absolute value detecting unit 1540 is performed by the following
formula 54.
abs.sub.--S2=abs(R3+2.times.R6+R9-R1-2.times.R4-R7) [Formula
54]
[0191] The sum unit 1550 calculates the sum abs_S of the output
data of the first and second absolute detecting units 1530 and
1540. This is represented by the following formula 55.
abs.sub.--S=abs.sub.--S1+abs.sub.--S2 [Formula 55]
[0192] The processing unit 1560 outputs the data Rout filtered
according to the output abs_S of the sum unit 1550. If the sum
abs_S is larger than a predetermined fifth threshold, the
processing unit 1560 recognizes there is an edge and determines
whether the edge is the horizontal or vertical direction edge.
[0193] In other words, if abs_S1 is larger than a value adding
abs_S2 to a predetermined sixth threshold, the processing unit 1560
determines the output Rout as the horizontal direction average of
the center row, the center of which is added to the weight. In this
case, the output Rout is calculated by the following formula
56.
Rout = R 4 + 2 .times. R 5 + R 6 4 [ Formula 56 ] ##EQU00020##
[0194] If the abs_S2 is larger than another value adding the abs_S1
to the sixth threshold, the processing unit 1560 determines the
output Rout as the vertical direction average of the center column,
the center of which is added to the weight. In this case, the
output Rout is calculated by the following formula 57.
Rout = R 2 + 2 .times. R 5 + R 8 4 [ Formula 57 ] ##EQU00021##
[0195] Unless the abs_S1 is larger than the value adding the abs_S2
to the sixth threshold and the sum abs_S2 is larger than the value
adding the abs_S1 to the sixth threshold when the sum abs_S is
larger than the fifth threshold, the processing unit 1560
determines the output Rout as the horizontal/vertical direction
averages of the center row/column, the centers of which are added
to the weight. In this case, the output Rout is calculated by the
following formula 58.
Rout = 4 .times. R 5 + R 2 + R 4 + R 6 + R 8 4 [ Formula 58 ]
##EQU00022##
[0196] If the sum abs_S is smaller than the first threshold, the
processing unit 1560 can recognize the corresponding image has no
edge and determine the output Rout as R5, which is the value of the
center element of the image.
[0197] Here, the firth threshold is variable. The sixth threshold
can be determined as 50, but not limited thereto.
[0198] The above processes can be repeated for the data G_n and B_n
to determine Gout and Bout.
[0199] FIG. 19 is a simplified block diagram generally illustrating
the color interpolation apparatus using an adjustable threshold in
accordance with a second embodiment of the present invention.
[0200] As illustrated in FIG. 19, the color interpolation apparatus
of the present invention calculates the difference diff_Y. If the
difference diff_Y is smaller than the first threshold, the color
interpolation apparatus performs the bilinear interpolation and the
Gaussian filtering. If the difference diff_Y is smaller than the
second threshold and the same as or larger than the first
threshold, the color interpolation apparatus performs the bilinear
interpolation, and the interpolation unit 320 of FIG. 3 performs
the interpolation (referred to as "effective interpolation" in FIG.
19). Also, if the difference diff_Y is the same as or larger than
the second threshold, the color interpolation apparatus performs
the effective interpolation and the edge enhancement.
[0201] On the other hand, the present invention calculates the
difference diff_Y. If the difference diff_Y is smaller than the
third threshold, the present invention performs the typical
filtering such as the Gaussian filtering. If the difference diff_Y
is smaller than the fourth threshold and the same as or larger than
the third threshold, the present invention bypasses its
performance. Also, if the difference diff_Y is the same as or
larger than the fourth threshold, the present invention performs
the zipper noise removing.
[0202] FIG. 20 is a simplified block diagram illustrating the color
interpolation performed according to the threshold determined in
accordance with a three embodiment of the present invention.
[0203] As illustrated in FIG. 20, in the color interpolation where
the threshold is determined in accordance with the present
invention, the different color interpolation methods according to
the threshold are performed. This can be applied differently in
accordance with the difference (the pixel value difference) between
the maximum value and the minimum value of pixel, which is the
property of image.
[0204] In FIG. 20, the second threshold is determined by the edge
property of image. The method of determining whether the pixel
value difference has the edge property has already been well-known.
Accordingly, the method of setting the first threshold will be
disclosed in the present invention.
[0205] In FIG. 20, the object of the third area is to remove a
noise. Accordingly, the first threshold can be set depending on the
noise. At this time, the luminance of the image against the
illumination intensity of the external surroundings is typically
determined by the camera exposure. This will be described with
reference to the pertinent drawings.
[0206] FIG. 21 illustrates a property curve of camera exposure
depending on the surroundings.
[0207] As illustrated in FIG. 21, the more changed from low to high
the illumination intensity is, the smaller little by little the
exposure level becomes. The high illumination of 2,000 lux or more
maintains a constant amount of the exposure. The noise level of the
image obtained from the camera is varied depending on the exposure
level of this camera. FIGS. 22 and 23 illustrate a property curve
of the noise level of camera in accordance with the exposure.
[0208] As illustrated in FIG. 22 and FIG. 23, it typically shows a
tendency that the lower the intensity illumination is, the higher
the noise level appears and the noise level is constantly
maintained in a particular level of illumination intensity or more.
Accordingly, the first threshold can be determined depending on the
intensity level of illumination. At this time, since the
illumination intensity corresponding to the external surroundings
determines the exposure level of camera, the first threshold can be
determined depending on the exposure level of camera. Hereinafter,
"first threshold" will be referred to "threshold."
[0209] FIG. 24 is a block diagram illustrating the threshold
determining apparatus in accordance with the present invention.
[0210] As illustrated in FIG. 24, the threshold determining
apparatus of the present invention can include a standard deviation
determining unit 2410, a maximum/minimum threshold determining unit
2420 and a threshold determining unit 2430.
[0211] The standard deviation determining unit 2410 receives the
image of the flat surface photographed with the maximum exposure
and the minimum exposure, respectively, and calculates each of the
standard deviations of noise. .sigma..sub.max exp refers to the
standard deviation of noise in the maximum exposure.
.sigma..sub.min exp refers to the standard deviation of noise in
the minimum exposure.
[0212] The maximum/minimum threshold determining unit 2420
determines the maximum value and the minimum value of the threshold
by using the standard deviations of noise in the maximum exposure
and the minimum exposure, respectively, received from the
aforementioned standard deviation determining unit 2410.
[0213] If the maximum value of the threshold refers to "th.sub.max"
and the minimum value of the threshold refers to "th.sub.min," the
th.sub.max and th.sub.min can be determined depending on the
maximum noise and minimum noise, respectively. It is assumed that
the maximum noise is generated in the maximum exposure and that the
minimum noise is generated in the minimum exposure. Since the
maximum/minimum noises can be calculated by the standard deviations
of the maximum exposure and the minimum exposure, respectively, the
th.sub.max and th.sub.min are represented by the following formulas
59 and 60, respectively.
th.sub.max=.sigma..sub.max exp [Formula 59]
th.sub.min=.sigma..sub.min exp [Formula 60]
[0214] If the noise level is high, the threshold must be increased
to remove the noise in the third area of FIG. 20.
[0215] FIG. 25 illustrates a graph of the maximum value and minimum
value of the exposure in accordance with the intensity of
illumination, and FIG. 26 illustrates a graph of the maximum value
and minimum value of the threshold in accordance with the
exposure.
[0216] As described above, a lot of noises mainly appear in the low
intensity of illumination, and the low intensity of illumination
has the high exposure value as illustrated in FIG. 25. Accordingly,
a threshold th is set at a maximum assuming that there are most
noises in the maximum exposure value Max Exposure. Also, the
th.sub.min, which is the minimum of the threshold th, is set in the
minimum exposure value Min Exposure, where the exposure is
constantly maintained at a particular intensity level of
illumination. In the internal area, the threshold having linear
properties can be formed. Accordingly, in accordance with the graph
property of FIG. 26, the threshold determining unit 2430 can
calculate the threshold th depending on the exposure by the
following formula 61.
th = th max - th min Max Exposure - Min Exposure Exposure + Max
Exposure th min - Min Exposure th max Max Exposure - Min Exposure [
Formula 61 ] ##EQU00023##
[0217] The color interpolation can be performed by using the
threshold determined by the above formula 61.
[0218] Hereinafter, the threshold calculated by the formula 61
refers to "first threshold." The method of performing the color
interpolation by use of a second threshold, determined depending on
the aforementioned edge level, will be described with reference to
FIG. 1 through FIG. 9, which was referred while the first
embodiment was described. Further, for the convenience of
description, although the same formula, as described in the first
embodiment, is used in the description, a new reference number will
be added to the same formula.
[0219] Referring to FIG. 1 where illustrates the block diagram of
the color interpolation apparatus applied with the determined
threshold, the color interpolation apparatus of the present
invention can include the color interpolation unit 110, the
output/edge property determining unit 120, the RGB converting unit
130 and the edge enhancement unit 140.
[0220] The color interpolation unit 110 performs the color
interpolation of an inputted Bayer pattern image. FIG. 2 is the
simplified block diagram illustrating the interpolation performed
by the color interpolation unit of FIG. 1.
[0221] As illustrated of FIG. 2, once a 5.times.5 Bayer pattern
image 700 is inputted, the color interpolation unit 110 of the
present invention performs the bilinear interpolation of the
5.times.5 Bayer pattern image 200 to take a 3.times.3 output image
201. This 3.times.3 output image 201 is referred to as R_all, G_all
and B_all.
[0222] If the interpolation of the present invention for the
3.times.3 output image 201 is first performed, 1-pixel data 202
will be taken. This 1-pixel data 202 becomes 1-pixel output data
E_R, E_G and E_B 203. If the edge calculation of the 1-pixel data
202 is carried out, it will become 1-pixel output data E_Y 204.
[0223] If the edge calculation of the 3.times.3 output image 201 is
performed again, a 3.times.3 image Y_all 205 will be taken. If the
Gaussian filtering of the 3.times.3 image Y_all 205 is performed,
last 1-pixel output data G_Y 206 will be taken. On the other hand,
if the special value (the difference between the maximum value and
the minimum value) of the 3.times.3 image Y_all 205 is calculated,
1-pixel data diff_Y 207 will be taken.
[0224] Also, if the Gaussian filtering of the 3.times.3 image 201
is carried out, 1-pixel output data G_R, GG and G_B will be
obtained, which is one output.
[0225] Accordingly, the color interpolation unit 110 can be
inputted with the 5.times.5 Bayer pattern image 200 and output five
1-pixel data 203, 204, 206, 207 and 210.
[0226] FIG. 3 is a detailed block diagram illustrating an
embodiment of the color interpolation unit of FIG. 1. As
illustrated of FIG. 3, the color interpolation unit 110 of the
present invention can have the bilinear interpolation unit 310, the
interpolation unit 320, the first edge calculation unit 330, the
second edge calculation unit 340, the first Gaussian filter 350,
the difference determining unit 360 and the second Gaussian filter
370.
[0227] The bilinear interpolation unit 310 performs the bilinear
interpolation of an inputted 5.times.5 Bayer pattern image and
outputs 3.times.3 data.
[0228] The interpolation unit 320 performs the color interpolation
of the inputted 3.times.3 data. FIG. 4 illustrates the Bayer
pattern image for the color interpolation performed by the
interpolating unit 320. Referring to FIG. 4, the below description
relates to the case that the R3 of the R component is the center
pixel in an RG line. However, the same description can be applied
to the case that the center pixel is the B component in a GB
line.
[0229] For the interpolation performed by the interpolation unit
320, Rn, Rw, Rs and Re and Kr1, Kr2, Kr3 and Kr4 of FIG. 4 are
defined as the following formula 62 and formula 63,
respectively.
Rn = R 1 + R 3 2 Rw = R 2 + R 3 2 Rs = R 3 + R 5 2 Re = R 3 + R 4 2
[ Formula 62 ] Kr 1 = G 4 - Rn Kr 2 = G 6 - Rw Kr 3 = G 9 - Rs Kr 4
= G 7 - Re [ Formula 63 ] ##EQU00024##
[0230] Then, the output data E_G of the G component of the
interpolation unit 320 is the following formula 64.
E_G = R 3 + Kr 1 + Kr 2 + K r 3 + Kr 4 4 [ Formula 64 ]
##EQU00025##
[0231] Since the interpolation method of the interpolation unit 320
is the same as the well-known prior art, the pertinent detailed
description will be omitted.
[0232] Gwn, Gws, Ges and Gen and Kb1, Kb2, Kb3 and KB4 are defined
as the following formula 65 and formula 66, respectively, to
calculate the output data E_B of the interpolated B component and
the output data E_R of the interpolated R component.
Gwn = G 1 + G 3 + G 6 + G 4 4 Gws = G 6 + G 8 + G 11 + G 9 4 Ges =
G 7 + G 9 + G 12 + G 10 4 Gen = G 2 + G 4 + G 7 + G 5 4 [ Formula
65 ] Kb 1 = Gwn - B 1 Kb 2 = Gen - B 2 Kb 3 = Gws - B 3 Kb 4 = Ges
- B 4 [ Formula 66 ] ##EQU00026##
[0233] By the formula 65 and formula 66, the output data EB of the
interpolated B component and the output data E_R of the
interpolated R component are determined by the formula 67 and the
formula 68, respectively.
E_B = E_G - Kb 1 + Kb 2 + Kb 3 + Kb 4 4 [ Formula 67 ] E_R = R 3 [
Formula 68 ] ##EQU00027##
[0234] On the other hand, FIG. 5 is another simplified block
diagram of the Bayer pattern image illustrating the color
interpolation performed by the interpolating unit 320 of FIG. 3 in
case that the G5 of G component is the center pixel in a GB line.
The below description relates to the case that the G component is
the center pixel in the GB line. However, the same description can
be applied to the case that the center pixel is the G component in
an RB line.
[0235] Gn, Gw, Gs and Ge and Kr1, Kr2, Kb1 and Kb2 are defined as
the following formula 69 and formula 70, respectively, to perform
the color interpolation of the Bayer pattern image of FIG. 5.
Gn = G 1 + G 2 + G 5 + G 3 4 Gw = G 2 + G 4 + G 7 + G 5 4 Gs = G 5
+ G 7 + G 9 + G 8 4 Ge = G 3 + G 5 + G 8 + G 6 4 [ Formula 69 ] Kr
1 = Gn - R 1 Kr 2 = Gs - R 2 Kb 1 = Gw - B 1 Kb 2 = Ge - B 2 [
Formula 70 ] ##EQU00028##
[0236] By the formula 69 and formula 70, the output data E_G, E_R
and EB are determined by the formula 71 through the formula 73,
respectively in the case of FIG. 5.
E_G = G 5 [ Formula 71 ] E_R = G 5 - Kr 1 + Kr 2 2 [ Formula 72 ]
E_B = G 5 - Kb 1 + Kb 2 2 [ Formula 73 ] ##EQU00029##
[0237] In case the center pixel is the G component, the R component
or the B component, each of the corresponding output data is
determined as described above.
[0238] The first edge calculating unit 330 performs the edge
calculation of the output data E_G, E_R and E_B of the
interpolation unit 320. This edge calculation is represented by the
following formula 74.
E_Y = E_R + 2 .times. E_G + E_B 4 [ Formula 74 ] ##EQU00030##
[0239] where E_Y is defined as the output data of the first edge
calculating unit 330.
[0240] The second edge calculating unit 340, which receives
3.times.3 data from the bilinear interpolation unit 310, performs
the edge calculation to carry out the edge detection of the
3.times.3 data. This edge calculation is the same as the following
formula 75.
Y_all = R_all + 2 .times. G_all + B_all 4 [ Formula 75 ]
##EQU00031##
[0241] The 3.times.3 data Y_all is taken by calculating each pixel
of the inputted 3.times.3 data by use of the formula 75. If the
3.times.3 data Y_all is passed through the first Gaussian filter
350, it becomes 1-pixel data G_Y. FIG. 6 is an example illustrating
the first Gaussian filter and the second Gaussian filter of FIG.
3.
[0242] The difference determining unit 360 determines the
difference between the maximum value and the minimum value of the
elements of the 3.times.3 output data of the second edge
calculating unit 340. This is represented by the following formula
76.
diff.sub.--Y=max(Y_all)-min(Y_all) [Formula 76]
[0243] Also, the second Gaussian filter 370 can take the output
data G_R, G_G and G_B by Gaussian filtering the 3.times.3 output
data of the bilinear interpolation unit 310.
[0244] As such, the output data of the color interpolation unit 110
of FIG. 1 is 1-pixel data E_R, E_G and E_B, 1-pixel data E_Y,
1-pixel data G_Y, 1-pixel data diff_Y and 1-pixel data G_R, G_G and
G_B.
[0245] On the other hand, the output/edge property determining unit
120 receives the output data of the color interpolation unit 110
and an edge parameter, and determines not only the RGB output but
also an edge enhancement coefficient and an edge flag.
[0246] In accordance with the present invention, the output/edge
property determining unit 120 presets a first threshold, determined
by the formula 61, and a second threshold, determined depending on
the edge property. These are varied depending on the system.
[0247] In case that the difference diff_Y received from the color
interpolation unit 110 is smaller than the first threshold, it is
determined that the corresponding image is flat (e.g. a wall part)
and the corresponding output is the output data of the second
Gaussian filter 370. That is, this is represented by the following
formula 77.
R_out=G.sub.--R
G_out=G.sub.--G
B_out=G.sub.--B [Formula 77]
[0248] If the difference diff_Y is the same as or larger than the
first threshold and smaller than the second threshold, it is
determined that the corresponding image has no edge and is not
flat, that is, the image has properties such as an outline (e.g. at
this time, referring to as a typical image). Also, the output/edge
property determining unit 120 determines the corresponding output
is the output data of the interpolation unit 320 and the
corresponding edge flag is zero. In this case, the output is
represented by the following formula 78.
R_out=E.sub.--R
G_out=E.sub.--G
B_out=E.sub.--B [Formula 78]
[0249] If the difference diff_Y is the same as or larger than the
second threshold, it is determined that the corresponding image has
an edge. Also, the output/edge property determining unit 120
determines the corresponding output is the output data of the
interpolation unit 320 and the corresponding edge flag is 1.
[0250] The output/edge property determining unit 120 further
generates an edge enhancement coefficient by using the inputted
edge parameter. This edge parameter includes edge coring, edge gain
and edge limit. The edge parameter will be described with reference
to FIG. 7 through FIG. 9.
[0251] FIG. 7 through FIG. 9 are examples illustrating the edge
parameter inputted into the output/edge property determining unit
120.
[0252] The edge coring not only removes a noise-like edge component
also prevents small edge components from being displayed like noise
on a screen. As illustrated in FIG. 7, a predetermined value is
subtracted from the inputted edge value, and the edge components
under the value, inclusive, become zero.
[0253] As illustrated in FIG. 8, the edge gain is performed the
edge value passing through the edge coring. The difference between
a large edge value and a small edge value is generated by changing
the result value gradient of an outputted edge component to an
inputted edge component. Accordingly, more clear and improved image
quality is provided by making a clear edge component clearer and
maintaining a small edge component unchangeable.
[0254] In case that the absolute value of the edge component,
extracted from the edge gain operation, is very large, the edge
limit restricts the maximum edge value to prevent the edge
component having large absolute value from being specifically
displayed on a screen. The edge limit has positive limit and
negative limit.
[0255] The edge enhancement coefficient is determined by the edge
parameter and the outputs E_Y and G_Y, detected from the color
interpolation unit 110. This is represented by the following
formula 79.
Srgb=edge_enhancement(edge_component,edge_parameter) [Formula
79]
[0256] where Srgb refers to the edge enhancement coefficient and
"edge_component" refers to the difference between E_Y and G_Y. The
edge enhancement coefficient is a function of the edge component
and the edge parameter, referring to "edge_component," determined
depending on the system. The edge enhancement coefficient value is
not limited.
[0257] Accordingly, the output/edge property determining unit 120
outputs the data R_out, the G_out, B_out and the Srgb and the edge
flag.
[0258] The RGB converting unit 130 receives the data R_out, G_out
and B_out and converts the data R_out and B_out to luminance and
chrominance YCbCr data. This is represented by the following
formula 80.
( Y Cb Cr ) = ( 0.299 0.587 0.114 - 0.169 - 0.331 0.5 0.5 - 0.419 -
0.081 ) .times. ( R G B ) - ( 0 128 128 ) [ Formula 80 ]
##EQU00032##
[0259] However, since the formula of converting RGB data to YCbCr
data is well known, the formula is limited to the above formula 19.
The converting ratio can be varied depending on the kind of the
image processing system.
[0260] The edge enhancement unit 140 enhances the edge by using the
luminance (Y) data, received from the above RGB converting unit
130, the edge enhancement coefficient Srgb, received from the
output/edge property determining unit 120, and the edge flag.
[0261] If the edge flag is 1, the output data of the edge
enhancement 140 is calculated by the following formula 20.
Y_out=Y+Srgb [Formula 81]
[0262] However, if the edge flag is zero, since it is determined
that the corresponding image has no edge in the above, the output
data of the edge enhancement 140 bypasses the Y data.
[0263] FIG. 27 is a simplified block diagram illustrating the noise
removing performed according to the threshold determined in
accordance with a fourth embodiment of the present.
[0264] As illustrated in FIG. 27, in the noise removing determined
with the threshold in accordance with the present invention, the
noise removing method is differently performed according to the
threshold. This can be differently applied in accordance with the
difference (the pixel value difference), which is the property of
image, between the maximum value and the minimum value of
pixel.
[0265] In FIG. 27, the zipper noise removing of the image
corresponding to a first area is performed. Accordingly, the second
threshold must be adjusted to be applied to the area determined as
the edge. However, due to the property of sensor, in the case of
the somewhat high noise level, an area (i.e. a second area)
bypassed with the image can be controlled to perform the same
operation as the first area by adjusting the second threshold to
the first threshold. The first threshold, however, needs to be
adjusted depending on the exposure level of camera.
[0266] In FIG. 27, the second threshold is determined by the edge
property of image. The method of determining whether the pixel
value difference has the edge property has already been well-known.
Accordingly, the method of setting the first threshold will be
disclosed in the present invention.
[0267] In FIG. 27, the object of the third area is to remove not an
edge noise but a typical noise of image. Accordingly, the first
threshold can be set depending on the noise. At this time, the
luminance of the image against the illumination intensity of the
external surroundings is typically determined by the camera
exposure. This will be described with reference to FIG. 21 through
FIG. 23, which were referred to in the description of the third
embodiment.
[0268] As illustrated in FIG. 21, the more changed from low to high
the illumination intensity is, the smaller little by little the
exposure state becomes. The high illumination of 2,000 lux or more
maintains a constant amount of the exposure. The noise level of the
image obtained from the camera is varied depending on the exposure
level of this camera. FIGS. 22 and 23 illustrate the property curve
of a noise level of camera in accordance with the exposure. As
illustrated in FIG. 22 and FIG. 23, it typically shows a tendency
that the lower the illumination is, the higher the noise level
appears and the noise level is constantly maintained in a
particular level of illumination intensity or more. Accordingly,
the first threshold can be determined depending on the intensity
level of illumination. At this time, since the illumination
intensity corresponding to the external surroundings determines the
exposure level of camera, the first threshold can be set depending
on the exposure level of camera. Hereinafter, "first threshold"
will be referred to "threshold."
[0269] FIG. 28 is a block diagram illustrating the threshold
determining apparatus in accordance with a fourth embodiment of the
present.
[0270] As illustrated in FIG. 28, the threshold determining
apparatus of the present invention can include an average value
determining unit 2810, a maximum/minimum threshold determining unit
2820 and a threshold determining unit 2830.
[0271] The average value determining unit 2810 receives the image
of the flat surface photographed with the maximum exposure and the
minimum exposure, respectively, and randomly acquires samples from
the mask, having a uniform size, of the image and calculates the
differences between the maximum value and the minimum value of each
exposure. Then, the average value determining unit 2810 calculates
each average of these calculated values. At this time, diff.sub.max
exp refers to the average of the difference between the maximum
value and the minimum value of noise in the maximum exposure.
diff.sub.min exp refers to the average of the difference between
the maximum value and the minimum value of noise in the minimum
exposure.
[0272] The maximum/minimum threshold determining unit 2820
determines the maximum value and the minimum value of the threshold
by using the standard deviations of noise in the maximum exposure
and the minimum exposure, respectively, received from the
aforementioned average value determining unit 2810.
[0273] If the maximum value of the threshold refers to
"th.sub.max," and the minimum value of the threshold refers to
"th.sub.min," the th.sub.max and th.sub.min can be determined
depending on a maximum noise level and a minimum noise level,
respectively. It is assumed that the maximum noise is generated in
the maximum exposure and that the minimum noise is generated in the
minimum exposure. Since the maximum/minimum noises can be
calculated by the standard deviation of noise and the average of
the difference between the maximum value and the minimum value of
noise in each of the maximum exposure and the minimum exposure, the
th.sub.max and th.sub.min are represented by the following formulas
82 and 83, respectively.
th.sub.max=diff.sub.max exp [Formula 82]
th.sub.min=diff.sub.min exp [Formula 83]
[0274] If the noise level is high, the threshold must be increased
to remove the noise in the third area of FIG. 3.
[0275] As described above with reference to FIG. 25 and FIG. 26, a
lot of noises mainly appear in the low intensity of illumination,
and the low intensity of illumination has the high exposure value.
Accordingly, a threshold th is set at a maximum assuming that there
are most noises in the maximum exposure value Max Exposure. Also,
the th.sub.min, which is the minimum of the threshold th, is set in
the minimum exposure value Min Exposure, where the exposure is
constantly maintained at a particular intensity level of
illumination. In the internal area, the threshold having linear
properties can be formed. Accordingly, in accordance with the graph
property of FIG. 26, the threshold determining unit 2830 can
calculate the threshold th depending on the exposure by the
following formula 84.
th = th max - th min Max Exposure - Min Exposure Exposure + Max
Exposure th min - Min Exposure th max Max Exposure - Min Exposure [
Formula 84 ] ##EQU00033##
[0276] The color interpolation can be performed by using the
threshold determined by the above formula 84.
[0277] Hereinafter, the threshold calculated by the formula 84
refers to "first threshold." The method of performing the color
interpolation with a second threshold determined depending on the
edge level will be described.
[0278] FIG. 29 a block diagram illustrating a noise removing
apparatus applied with the threshold determined in accordance with
the present invention.
[0279] As illustrated in FIG. 29, the noise removing apparatus of
the present invention can have a difference calculating unit 2910,
a noise removing method determining unit 2920, a zipper noise
removing unit 2930 and a Gaussian filter unit 2940.
[0280] Hereinafter, although the below description takes the
luminance (Y) data, inputted into the noise removing apparatus, as
an example, the red/green/blue (RGB) data can be also used. This is
because the RGB data and the Y data can be simply converted to each
other by a formula.
[0281] The difference calculating unit 2910 of FIG. 29 calculates
the maximum value and the minimum value of the elements of inputted
3.times.3 Y data. The operation of the difference calculating unit
2910 will be described with reference to FIG. 30.
[0282] FIG. 30 is a conceptual diagram illustrating the operation
of the difference calculating unit 2910 of FIG. 29.
[0283] As illustrated in FIG. 30, the difference calculating unit
2910 of FIG. 29 determines the minimum value of inputted 3.times.3
Y data 3010 as the minimum value min_Y 3020 of 1 pixel and the
maximum value of the inputted 3.times.3 Y data 3010 as the maximum
value max_Y 3030 of 1 pixel. The difference diff_Y 3040 between the
max_Y 3030 and the min_Y 3020 is determined by using the max_Y 3030
and the min_Y 3020. The 3.times.3 Y data 3050 is bypassed to remove
a noise.
[0284] FIG. 31 is a detailed block diagram illustrating an
embodiment of the difference calculating unit 2910 of FIG. 29.
[0285] As illustrated in FIG. 31, the difference calculating unit
2910 can have a maximum value determining unit 2911, a minimum
value determining unit 2912 and a difference determining unit
2913.
[0286] The maximum value determining unit 2911 determines the
maximum value max_Y of the elements of 3.times.3 Y data. The
minimum value determining unit 2912 determines the minimum value
min_Y of the elements of 3.times.3 Y data. The difference
determining unit 2913 determines the difference diff_Y between the
max_Y and the min_Y.
[0287] The noise removing method determining unit 2920 preset the
first threshold determined in accordance with the present invention
and the second threshold determined in accordance with the edge
level of image. In case that the difference diff_Y is smaller than
the first threshold, the noise removing method determining unit
2920 recognizes the inputted image has no edge and the low
intensity of illumination and performs the typical filtering. In
other words, the noise removing method determining unit 2920
determines to apply the Gaussian filter unit 2940 to remove the
noise. The Gaussian filter unit 2940 performs the Gaussian
filtering of this 3.times.3 Y data. The example of the Gaussian
filter unit 2940 of FIG. 31 is illustrated in FIG. 6. Of course,
the present invention is limited to the gausian filtering.
[0288] In case that the difference diff_Y is the same as or larger
than the first threshold and smaller than the second threshold, the
noise removing method determining unit 2920 determines that the
corresponding image has no edge and is not flat, that is, the image
has properties such as an outline (e.g. at this time, referring to
as a typical image). Also, the noise removing method determining
unit 2920 determines the output Y_out as Y5. That is, the noise
removing method determining unit 2920 determines that the image of
this area is not necessary to remove the noise.
[0289] If the difference diff_Y is the same as or larger than the
second threshold, the noise removing method determining unit 2920
determines that the corresponding image has an edge and determines
to perform the zipper noise removing. Accordingly, the zipper noise
removing unit 2930 performs the zipper noise removing. This will be
described with reference to the pertinent drawing.
[0290] FIG. 32 is a detailed block diagram illustrating an
embodiment of the zipper noise removing unit 2930 of FIG. 29.
[0291] As illustrated in FIG. 32, the noise removing unit 2930 of
the present invention can include a first filter 2931, a second
filter 2932, a first absolute value detecting unit 2933, and a
second absolute value detecting unit 2934, a sum unit 2935 and a
processing unit 2936.
[0292] The first and second filter 2931 and 2932, which can be a
3.times.3 filter, checks how many horizontal and vertical direction
edge components of inputted Y components, respectively are. For
this, the inputted Y component data is also 3.times.3 data. In
accordance with the embodiment of the present invention, the data
such as the 3.times.3 data 3050 of FIG. 30 is inputted, but the
present invention is not limited thereto. It shall be evident that
an N.times.N filter can be used depending on the data inputted into
the difference calculating unit 2910.
[0293] The example of the first filter and the second filter of
FIG. 32 has been described with FIG. 16 and FIG. 17. Also, the
example illustrating the data inputted into the first filter and
the second filter has already been described with FIG. 18.
[0294] The first filter and the second filter, as described above,
are to check the horizontal and vertical direction edge components
of respective inputted Y component data. Once the Y component data
is inputted as illustrated in FIG. 18, elements are multiplied to
each component at the same location. The symbol "x" of FIG. 18
refers to the multiplication of elements at the same position.
[0295] The first absolute value detecting unit 2933 of FIG. 32 sums
the pertinent elements from the output data of the first filter
2931 and detects the absolute value of the sum of the elements. The
output data of the first absolute value detecting unit 2933 is
performed by the following formula 85.
abs.sub.--S1=abs(Y7+2.times.Y8+Y9-Y1-2.times.Y2-Y3) [Formula
85]
[0296] where "abs" refers to the absolute value. The second
absolute value detecting unit 2934 of FIG. 32 sums the respective
elements from the output data of the second filter 2932 and detects
the absolute value of the sum of the elements. The output data of
the second absolute value detecting unit 2934 is performed by the
following formula 86.
abs.sub.--S2=abs(Y3+2.times.Y6+Y9-2.times.Y4-Y7) [Formula 86]
[0297] The sum unit 2935 calculates the sum abs_S of the outputs of
the first and second absolute detecting units 2933 and 2934. This
is represented by the following formula 87.
abs.sub.--S=abs.sub.--S1+abs.sub.--S2 [Formula 87]
[0298] The processing unit 2936 outputs the data Y_out filtered
according to the output abs_S of the sum unit 2935. If the abs_S is
larger than the third threshold, the processing unit 2936
recognizes there is an edge and determines whether the edge is the
horizontal or vertical direction edge.
[0299] In other words, if abs_S1 is larger than a value adding
abs_S2 to the fourth threshold, the processing unit 2936 determines
the output data Y_out as the horizontal direction average of the
center row, the center of which is added to the weight. In this
case, the output data Y_out is calculated by the following formula
88.
Y_out = Y 4 + 2 .times. Y 5 + Y 6 4 [ Formula 88 ] ##EQU00034##
[0300] If the abs_S2 is larger than another value adding the abs_S1
to the fourth threshold, the processing unit 2936 determines the
output Rout as the vertical direction average of the center column,
the center of which is added to the weight. In this case, the
output data Y_out is calculated by the following formula 89.
Y_out = Y 2 + 2 .times. Y 5 + Y 8 4 [ Formula 89 ] ##EQU00035##
[0301] Unless the abs_S1 is larger than the value adding the abs_S2
to the fourth threshold and the abs_S2 is larger than the value
adding the abs_S1 to the fourth threshold, when the abs_S is larger
than the third threshold, the processing unit 2936 determines the
output data Y_out as the horizontal/vertical direction average of
the center row/column, the centers of which are added to the
weight. In this case, the output data Y_out is calculated by the
following formula 90.
Y_out = 4 .times. Y 5 + Y 2 + Y 4 + Y 6 + Y 8 8 [ Formula 90 ]
##EQU00036##
[0302] If the abs_S is smaller than the third threshold, the
processing unit 2936 can recognize the corresponding image has no
edge and determine the output data Y_out as Y5, which is the value
of the center element of the image.
[0303] Here, the third threshold is variable. The fourth threshold
can be determined as 50, but not limited thereto.
[0304] As such, the processing unit 2936 can control the output
data Y_out of the zipper noise removing unit 2930.
[0305] As described above, even though the conventional noise
removing method has applied one noise removing filer to whole
image, the present invention can prevent the system loss by
determining the noise removing method depending on the property of
image.
[0306] As described above, the present invention can provide
improved image quality by adjusting a threshold in accordance with
the special value (the difference between the maximum value and the
minimum value) of the pixels of the mask in an image and applying a
different color interpolation method, unlike the conventional noise
removing method that has applied one noise removing filer to whole
image.
[0307] The present invention can also select a suitable color
interpolation by adjusting a threshold in accordance with the
special value (the difference between the maximum value and the
minimum value) of the pixels of the mask in an image. Moreover, the
present invention can efficiently remove the noise as a user
desires by applying the suitable color interpolation to R, G, and B
components. The conventional noise removing method, however, has
applied one noise removing filer to whole image
[0308] The present invention can also make one line memory,
requested to remove the noise of a chrominance component, not
necessary by directly applying the noise removing to RGB data.
[0309] The present invention can also provide improved image
quality by setting a threshold in accordance with the noise caused
by the exposure level of image and applying a different color
interpolation method, unlike the conventional noise removing method
that has applied one noise removing filer to whole image.
[0310] In addition, the present invention can apply a different
noise removing method depending on the threshold, determined in
accordance with the analysis of the properties of the image
inputted according to the exposure.
[0311] The drawings and detailed description are only examples of
the present invention, serve only for describing the present
invention and by no means limit or restrict the spirit and scope of
the present invention. Thus, any person of ordinary skill in the
art shall understand that a large number of permutations and other
equivalent embodiments are possible. The true scope of the present
invention must be defined only by the spirit of the appended
claims.
* * * * *