U.S. patent application number 12/615882 was filed with the patent office on 2010-05-20 for method of detecting and rectifying red-eyes automatically when taking photos.
Invention is credited to Fuhuei Lin, Yi WANG.
Application Number | 20100124374 12/615882 |
Document ID | / |
Family ID | 40031422 |
Filed Date | 2010-05-20 |
United States Patent
Application |
20100124374 |
Kind Code |
A1 |
WANG; Yi ; et al. |
May 20, 2010 |
METHOD OF DETECTING AND RECTIFYING RED-EYES AUTOMATICALLY WHEN
TAKING PHOTOS
Abstract
The present invention discloses a method for automatically
detecting and correcting red eye in a photo taking process. The
method utilizes a preview function which all of the current digital
cameras have. When flash light is needed during the photo taking
process, an image which is closest to the shutter pressing time is
read from a preview image sequence and this image is stored. Then,
red eye detection is performed according to various related
information (e.g., chroma information) of this image and the image
actually taken. A correction process is conducted on the detected
red eye area. The corrected image is stored as a resulting image in
a storage unit in the camera. Therefore, the red eye is removed
rapidly and accurately during a single normal exposure process and
the image without red eye is thus obtained.
Inventors: |
WANG; Yi; (Shanghai, CN)
; Lin; Fuhuei; (Cupertino, CA) |
Correspondence
Address: |
DORSEY & WHITNEY LLP;INTELLECTUAL PROPERTY DEPARTMENT
SUITE 1500, 50 SOUTH SIXTH STREET
MINNEAPOLIS
MN
55402-1498
US
|
Family ID: |
40031422 |
Appl. No.: |
12/615882 |
Filed: |
November 10, 2009 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
PCT/CN2008/071009 |
May 19, 2008 |
|
|
|
12615882 |
|
|
|
|
Current U.S.
Class: |
382/167 |
Current CPC
Class: |
H04N 5/23222 20130101;
G06T 2207/30216 20130101; H04N 5/232 20130101; G06T 7/90 20170101;
H04N 5/262 20130101 |
Class at
Publication: |
382/167 |
International
Class: |
G06K 9/62 20060101
G06K009/62 |
Foreign Application Data
Date |
Code |
Application Number |
May 17, 2007 |
CN |
200710040778.5 |
Claims
1. A method for automatically detecting and correcting red eye in a
photo taking process, characterized in that, information of an
image in preview mode and information of an image actually taken
are used to perform automatic red eye detection and correction,
comprising: (1) acquiring a shutter pressing time; (2) acquiring an
image from a preview image sequence, which is closest to the
shutter pressing time; regarding the image as a correction
reference image; (3) acquiring an image actually taken with a flash
lamp and regarding the image as an image to be corrected; (4)
performing red eye detection according to the information of the
correction reference image and the image to be corrected;
determining a red eye area; (5) performing red eye correction on
the detected red eye area utilizing the information of the
correction reference image and the image to be corrected.
2. The method of claim 1, characterize in that, the shutter
pressing time in step (1) is any time during a period from pressing
the shutter to staring exposure.
3. The method of claim 1, characterize in that, the shutter
pressing time in step (1) is the time when pressing 1/2, 1/3 or 1/4
of the shutter.
4. The method of claim 1, characterize in that, the red eye
detection in step (4) comprising: (1) determining whether the
correction reference image and the image to be corrected need to be
matched; matching the correction reference image and the image to
be corrected if it is determined that the correction reference
image and the image to be corrected need to be matched; (2)
calculating values effectively indicating red portion information
in the correction reference image and the image to be corrected and
obtaining red proportion images of the correction reference image
and the image to be corrected, respectively; (3) processing a
difference between the red proportion image in the correction
reference image and the red proportion image in the image to be
corrected; obtaining a difference image; (4) determining a
candidate red eye area using the red proportion image of the image
to be corrected and the difference image; (5) further selecting a
red eye area and determining a resulting red eye area.
5. The method of claim 1, characterized in that, in step (4), the
red eye detection comprising converting the image to other color
space YUV or CIELAB for processing and performing red eye detection
processing, wherein the information of the correction reference
image and the image to be corrected comprises original information
of an image and information of an image generated after
processing.
6. The method of claim 1, characterize in that, the red eye
detection in step (5) comprising: (1) calculating values of color
proportion for an pixel in the red eye area in the correction
reference image; (2) calculating a luminance for an pixel in the
red eye area in the image to be corrected; (3) calculating a
correction value for an pixel in the red eye area in the image to
be corrected; (4) replacing a pixel in the red eye area of the
image to be corrected with its corresponding correction value.
Description
FIELD OF THE INVENTION
[0001] The present invention relates to a method for digital image
processing, and more specifically, to a method for automatically
detecting and removing red eye during photo taking.
BACKGROUND
[0002] Generally, in the case of weak light, people's eye may look
red when using flash lamp to take photos. This is called "red eye
effect". Red eye effect occurs due to the reflection of blood
vessels on the retina when the incoming light emitted by the flash
lamp is very strong. However, since the digital camera is getting
smaller and smaller, the distance between the flash lamp and the
lens becomes shorter and shorter. Such problem is increasingly
significant.
[0003] Nowadays, many cameras have been equipped with auto anti-red
eye function. Such function can be categorized into two types of
techniques. One is to prevent red eye using a pre-flash. The other
is to perform image processing to remove red eye.
[0004] CN patent application No. CN 1100814A (published on Mar. 29,
1995) discloses an apparatus which uses pre-flash to make the pupil
contracted and reduces the reflected red light. However, such
apparatus can only mitigate the red eye effect to some extent, and
it cannot remove the red eye thoroughly. Moreover, since pre-flash
can make people look unnatural, and two flashes may consume more
energy, the actual application may not achieve a desirable
effect.
[0005] CN patent application No. CN 1750017A (published on Mar. 22,
2006) discloses a method for removing red eye by post-processing.
The method performs facial recognition on the image where red eye
occurs or uses other target detection method for locating the red
eye area. Then, the information such as the color in the red eye
area is adjusted so as to remove the red eye. However, in such
method, the detection speed is slow and is subject to noise and is
not precise and can rarely achieve the practical application in
handheld or mobile devices.
[0006] JP patent application No. 2000-299812A (published on Oct.
24, 2000) discloses a photo taking device. By taking a photo with
the flash light and taking a photo without the flash light and
extracting the data in the red eye area based on the relation
between the two photos, the red eye is removed. However, such
method requires taking two photos at the same place at the same
time, which extends the photo taking time and decreases the
efficiency due to repetitive shooting.
SUMMARY
[0007] The present invention is aimed at providing an easy and fast
method for automatically detecting and correcting red eye during a
photo taking process. Consequently, the pictures with no red eye
can be taken without altering the existing digital camera device
and decreasing the efficiency.
[0008] To this end, the method for automatically detecting and
correcting red eye during a photo taking process uses information
of an image in preview mode and information of an image actually
taken to perform automatic red eye detection and correction. The
method includes:
[0009] (1) acquiring a shutter pressing time;
[0010] (2) acquiring an image from a preview image sequence, which
is closest to the shutter pressing time; regarding the image as a
correction reference image;
[0011] (3) acquiring an image actually taken with a flash lamp and
regarding the image as an image to be corrected;
[0012] (4) performing red eye detection according to the
information of the correction reference image and the image to be
corrected; determining a red eye area; and
[0013] (5) performing red eye correction on the detected red eye
area according to the information of the correction reference image
and the image to be corrected.
[0014] According to the method for automatically detecting and
correcting red eye during a photo taking process, the shutter
pressing time is any time during a period from pressing the shutter
to starting exposure. The corrected image P.sub.corrected is the
resulting image with red eye being eliminated.
[0015] Since the present invention utilizes the image in preview
mode which is closest to the shutter pressing time, there is no
need to take two photos for a same scene, thus repetitive shooting
is avoided. Furthermore, since the present invention does not
requires any additional devices as long as the camera has a preview
function, the present invention can be practiced easily. In
addition, the present invention is able to remove red eye in the
red eye area according to the information of the preview image and
the information of the image actually taken. Thus, there is no need
to perform a series of complex area detection algorithm such as
facial recognition and the efficiency is therefore enhanced.
BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Detailed description will be made to the present invention
in conjunction with embodiments and accompanying drawings.
[0017] FIG. 1 is a flowchart according to the present
invention;
[0018] FIG. 2 is an embodiment of the present invention;
[0019] FIG. 3 is an embodiment of detecting red eye according to
the present invention; and
[0020] FIG. 4 is an embodiment of correcting red eye according to
the present invention.
PREFERRED EMBODIMENT OF THE PRESENT INVENTION
[0021] According to the present invention, the red eye detection
and correction is based on information of an preview image and
information of an image actually taken. The specific implementation
flowchart is illustrated in FIG. 1 and detailed as follows.
[0022] First, a shutter pressing time T is acquired, i.e., any
moment during the period from pressing the shutter to starting
exposure.
[0023] Then, according to the acquired shutter pressing time T, the
image most closely to the shutter pressing time T is acquired from
a preview image sequence S. This image serves as a correction
reference image P1 and is stored in a temporary storing unit.
[0024] Then, a photo taking unit is used to acquire the image taken
with the flash light. This image serves as a image to be corrected
P2 and is stored in a temporary storing unit.
[0025] Then, the related information of the correction reference
image P1 and the image to be corrected P2 stored in the temporary
unit are then used for detecting the red eye area R.sub.redeye.
[0026] Then, the detected red eye area R.sub.redeye is corrected
according to the related information of the correction reference
image P1 and the image to be corrected P2 and the corrected image
P.sub.corrected is stored as the resulting image.
[0027] Detailed description will be made to the present invention
with reference to the embodiment illustrated in FIG. 2.
[0028] First, a preview image sequence S is stored in a buffer
having a size which can accommodate at least one frame. When the
buffer overflows, the principle of first-in-first-out is followed.
That is, the first stored image is moved out to release space for
storing a new image. Then, the shutter pressing time T is acquired.
The shutter pressing time T is any moment during the period from
pressing the shutter to staring exposure or the time when pressing
1/2, 1/3 or 1/4 of the shutter (in the embodiment illustrated in
FIG. 2, time T is the time when pressing 1/2 of the shutter). It is
detected whether 1/2 of the shutter is pressed. If not, the process
continues to a preview step. If 1/2 of the shutter has pressed
already, preview step is stopped and the shutter pressing time T is
recorded at the same time. Then, the image which is closest to the
shutter pressing time T is searched among the preview image
sequence S and is stored as the correction reference image P1
(usually the last viewed image). The process then waits for the
shutter to be pressed thoroughly. If the shutter button is released
at that time, the photo taking mode is quitted and the process
reverts back to preview mode. If the shutter is pressed thoroughly,
a photo is taken with flash light and this photo is stored as the
image to be corrected P2. Lastly, the correction reference image P1
and the image to be corrected P2 are read and the red eye area
R.sub.redeye is detected according to the related information of
the two images. If no red eye is detected, P2 is stored as a
resulting image; otherwise, the red eye is removed from the
detected image and the corrected image P.sub.corrected is stored as
a resulting image.
[0029] FIG. 3 is an embodiment of detecting red eye according to
the present invention.
[0030] First, the correction reference mage P1 and the image to be
corrected P2 are input and these two images are checked to see if
matching is required. If matching is required, images are matched
and a new matched correction reference image P1 and a new matched
image to be corrected P2 are obtained. If no matching is required,
the method proceeds directly to a next step.
[0031] Then, the red proportion among each pixels in the correction
reference image P1 and the image to be corrected P2 are calculated,
respectively, and red proportion images P1.sub.red and P2.sub.red
are obtained.
[0032] Then, the difference between the obtained red proportion
images P1.sub.red and P2.sub.red is calculated and a difference
image P.sub.diff is thus obtained.
[0033] Then, the difference image P.sub.diff and the red proportion
images P2.sub.red are scanned. For any u(i,j).epsilon.P.sub.diff,
and any v(i,j).epsilon.P2.sub.red, if i=a, j=b and the following
conditions are met: {circle around (1)} u(a,b)>a predetermined
difference threshold M, where M can be 0.07 in one embodiment;
{circle around (2)} v(a,b)>a given red proportion threshold N,
where N can be 0.4 in one embodiment, then the pixel (a,b) is
marked as a red eye pixel. After scan is done, the red eye pixels
are combined into a different closed-ended area R(i), i.e.,
candidate red eye area.
[0034] Lastly, determination is made on each candidate red eye area
R(i). The determination is made based on the following conditions:
{circle around (1)} the area R.sub.s of the candidate red eye area
R(a)>a given area S_min, wherein S_min is 6 in one embodiment;
{circle around (2)} the perimeter of R(a)/the area of
R(a).epsilon.(r_min,r_max), wherein r.sub.min is 0.7 and r_max is
1.3 since R(a) approximates a circle. Therefore, the determination
formula is P.sup.2/(R.sub.s.times.4.pi.).epsilon.(0.7,1.3). When
i=a and the above conditions are met, area R(a) is recorded as the
red eye area R.sub.redeye(i); otherwise, it is excluded. The red
eye detection process ends until all the R(i) are checked.
[0035] Furthermore, the red eye detection according to the present
invention may also include converting the image to other color
space YUV (a method of indicating color information, where for a
color TV in PAL mode, Y indicates luminance, U and V indicates two
color differences) or CIELAB (International Commission on
Illumination converts the wavelength of the light into a set of
data depicting luminance and chroma based on human eye
characteristics) before performing red eye detection.
[0036] FIG. 4 is an embodiment of correcting red eye according to
the present invention.
[0037] First, the proportions of three colors, i.e., r_red(x,y),
r_green(x,y), r_blue(x,y), for RGB(x,y) in the correction reference
image P1 are calculated, respectively, where
x,y.epsilon.R.sub.redeye(i).
[0038] Then, luminance information I(x,y) of the pixel RGB(x,y) in
the image to be corrected P2 is calculated, where
x,y.epsilon.R.sub.redeye(i)
[0039] The value of tricolor in red eye area R.sub.redeye is then
calculated according to a corresponding correction formula, which
is expressed as follows: R.sub.c(x,y)=I(x,y)*r_red(x,y),
G.sub.c(x,y)=I(x,y)*r_green(x,y),
B.sub.c(x,y)=I(x,y)*r_blue(x,y).
[0040] Then, the pixel value in the red eye area R.sub.redeye in
the image to be corrected P2 is replaced with correction value
R.sub.c{grave over ( )}G.sub.c{grave over ( )}B.sub.c.The red eye
removal process ends.
* * * * *