U.S. patent application number 12/947731 was filed with the patent office on 2011-05-26 for method and apparatus for detection and correction of flash-induced eye defects within digital images using preview or other reference images.
Invention is credited to Petronel Bigioi, Peter Corcoran, Alexandru Drimbarean, Florin Nanu, Yury Prilutsky, Eran Steinberg, Adrian Zamfir.
Application Number | 20110122297 12/947731 |
Document ID | / |
Family ID | 46330309 |
Filed Date | 2011-05-26 |
United States Patent
Application |
20110122297 |
Kind Code |
A1 |
Steinberg; Eran ; et
al. |
May 26, 2011 |
METHOD AND APPARATUS FOR DETECTION AND CORRECTION OF FLASH-INDUCED
EYE DEFECTS WITHIN DIGITAL IMAGES USING PREVIEW OR OTHER REFERENCE
IMAGES
Abstract
A method for red-eye detection in an acquired digital image
acquiring one or more preview or other reference images without a
flash. Any red regions that exist within the one or more reference
images are determined. A main image is acquired with a flash of
approximately a same scene as the one or more reference images. The
main image is analyzed to determine any candidate red eye defect
regions that exist within the main image. Any red regions
determined to exist within the one or more reference images are
compared with any candidate red eye defect regions determined to
exist within the main image. Any candidate red eye defect regions
within the main image corresponding to red regions determined also
to exist within the one or more reference images are removed as
candidate red eye defect regions.
Inventors: |
Steinberg; Eran; (San
Francisco, CA) ; Corcoran; Peter; (Claregalway,
IE) ; Prilutsky; Yury; (San Mateo, CA) ;
Bigioi; Petronel; (Galway, IE) ; Zamfir; Adrian;
(Bucuresti, RO) ; Nanu; Florin; (Bucuresti,
RO) ; Drimbarean; Alexandru; (Galway, IE) |
Family ID: |
46330309 |
Appl. No.: |
12/947731 |
Filed: |
November 16, 2010 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
12142134 |
Jun 19, 2008 |
|
|
|
12947731 |
|
|
|
|
11233513 |
Sep 21, 2005 |
7587085 |
|
|
12142134 |
|
|
|
|
11182718 |
Jul 15, 2005 |
|
|
|
11233513 |
|
|
|
|
11123971 |
May 6, 2005 |
7436998 |
|
|
11182718 |
|
|
|
|
10976336 |
Oct 28, 2004 |
7536036 |
|
|
11123971 |
|
|
|
|
60945558 |
Jun 21, 2007 |
|
|
|
Current U.S.
Class: |
348/241 ;
348/E5.078; 382/167 |
Current CPC
Class: |
G06T 5/008 20130101;
G06K 9/00 20130101; G06T 2207/20008 20130101; G06K 9/0061 20130101;
G06T 2207/30216 20130101; G06T 7/97 20170101; G06T 5/50 20130101;
G06T 7/90 20170101; G06T 2207/30201 20130101 |
Class at
Publication: |
348/241 ;
382/167; 348/E05.078 |
International
Class: |
H04N 5/217 20110101
H04N005/217 |
Claims
1. An portable image acquisition and processing device which
automatically detects and corrects flash-induced eye defects within
digital images, comprising a processor, image sensor and lens to
acquire digital images, and a memory having code embedded therein
for programming the processor to perform a flash-induced eye defect
detection and correction method that comprises: using the
processor, image sensor and lens to acquire and process a main
digital image; applying an analysis filter to the main digital
image; correcting one or more image defects other than
flash-induced eye defects based on said applying said analysis
filter; obtaining a subsampled version of the main digital image;
detecting a flash-induced eye defect within said subsampled version
of said digital image; and correcting a corresponding flash-induced
eye defect within said digital image.
2. The device of claim 1, wherein said correcting said one or more
image defects other than flash-induced eye defects is performed
prior to performing said detecting said flash-induced eye
defect.
3. The device of claim 1, wherein the subsampled version of the
main digital image comprises a different resolution than the main
digital image.
4. The device of claim 1, wherein the subsampled version of the
main digital image comprises a same resolution as the main digital
image.
5. The device of claim 1, wherein the obtaining said subsampled
version of the main digital image comprises subsampling luminance
and chrominance data.
6. The device of claim 1, wherein the method further comprises:
interpreting results of the applying said analysis filter;
determining an optimal correction strategy for application to the
main digital image; and determining an optimal combination of
corrective image processing and filter adaption.
7. The device of claim 1, wherein the analysis filter comprises an
analysis prefilter applied before said detecting a flash-induced
eye defect.
8. One or more processor-readable media having code embedded
therein for programming a processor to perform a flash-induced eye
defect detection and correction method that comprises: using the
processor, image sensor and lens of a portable digital image
acquisition device to acquire and process a main digital image;
applying an analysis filter to the main digital image; correcting
one or more image defects other than flash-induced eye defects
based on said applying said analysis filter; obtaining a subsampled
version of the main digital image; detecting a flash-induced eye
defect within said subsampled version of said digital image; and
correcting a corresponding flash-induced eye defect within said
digital image.
9. The one or more processor-readable media of claim 8, wherein
said correcting said one or more image defects other than
flash-induced eye defects is performed prior to performing said
detecting said flash-induced eye defect.
10. The one or more processor-readable media of claim 8, wherein
the subsampled version of the main digital image comprises a
different resolution than the main digital image.
11. The one or more processor-readable media of claim 8, wherein
the subsampled version of the main digital image comprises a same
resolution as the main digital image.
12. The one or more processor-readable media of claim 8, wherein
the obtaining said subsampled version of the main digital image
comprises subsampling luminance and chrominance data.
13. The one or more processor-readable media of claim 8, wherein
the method further comprises: interpreting results of the applying
said analysis filter; determining an optimal correction strategy
for application to the main digital image; and determining an
optimal combination of corrective image processing and filter
adaption.
14. The one or more processor-readable media of claim 8, wherein
the analysis filter comprises an analysis prefilter applied before
said detecting a flash-induced eye defect.
15. A method of detecting and correcting a flash-induced eye defect
within a digital image, comprising: using a processor, image sensor
and lens of a portable digital image acquisition device to acquire
and process a main digital image; applying an analysis prefilter to
the main digital image; correcting one or more image defects other
than flash-induced eye defects based on said applying said analysis
prefilter; obtaining a subsampled version of the main digital
image; detecting a flash-induced eye defect within said subsampled
version of said digital image; and correcting a corresponding
flash-induced eye defect within said digital image.
16. The method of claim 15, wherein said correcting said one or
more image defects other than flash-induced eye defects is
performed prior to performing said detecting said flash-induced eye
defect.
17. The method of claim 15, wherein the subsampled version of the
main digital image comprises a different resolution than the main
digital image.
18. The method of claim 15, wherein the subsampled version of the
main digital image comprises a same resolution as the main digital
image.
19. The method of claim 15, wherein the obtaining said subsampled
version of the main digital image comprises subsampling luminance
and chrominance data.
20. The method of claim 15, wherein the method further comprises:
interpreting results of the applying said analysis prefilter;
determining an optimal correction strategy for application to the
main digital image; and determining an optimal combination of
corrective image processing and filter adaption.
21. The method of claim 15, wherein the analysis filter comprises
an analysis prefilter applied before said detecting a flash-induced
eye defect.
Description
PRIORITY and RELATED APPLICATIONS
[0001] This application is a Continuation application that claims
priority to U.S. patent application Ser. No. 12/142,134, filed Jun.
19, 2008, which claims priority to U.S. provisional patent
application 60/945,558, filed Jun. 21, 2007, and which is a
continuation-in-part (CIP) of U.S. patent application Ser. No.
11/233,513, filed Sep. 21, 2005, now U.S. Pat. No. 7,587,085, which
is a continuation-in-part (CIP) which claims the benefit of
priority to U.S. patent application Ser. No. 11/182,718, filed Jul.
15, 2005, now abandoned, which is a CIP of U.S. application Ser.
No. 11/123,971, filed May 6, 2005, now U.S. Pat. No. 7,436,998, and
which is a CIP of U.S. application Ser. No. 10/976,336, filed Oct.
28, 2004, now U.S. Pat. No. 7,536,036.
[0002] This application is related to U.S. patent application Ser.
No. 11/573,713, filed Feb. 14, 2007, which claims priority to U.S.
provisional patent application No. 60/773,714, filed Feb. 14, 2006,
and to PCT application no. PCT/EP2006/008229, filed Aug. 15,
2006.
[0003] This application also is related to 11/024,046, filed Dec.
27, 2004, which is a CIP of U.S. patent application Ser. No.
10/608,772, filed Jun. 26, 2003.
[0004] This application also is related to PCT/US2006/021393, filed
Jun. 2, 2006, which is a CIP of Ser. No. 10/608,784, filed Jun. 26,
2003.
[0005] This application also is related to U.S. application Ser.
No. 10/985,657, filed Nov. 10, 2004.
[0006] This application also is related to U.S. application Ser.
No. 11/462,035, filed Aug. 2, 2006, which is a CIP of U.S.
application Ser. No. 11/282,954, filed Nov. 18, 2005.
[0007] This application also is related to U.S. patent application
Ser. No. 11/460,218, filed Jul. 26, 2006, which claims priority to
U.S. provisional patent application Ser. No. 60/776,338, filed Feb.
24, 2006.
[0008] This application also is related to U.S. patent application
Ser. No. 12/063,089, filed Feb. 6, 2008, which is a CIP of U.S.
Ser. No. 11/766,674, filed Jun. 21, 2007, which is a CIP of U.S.
Ser. No. 11/753,397, which is a CIP of U.S. Ser. No. 11/765,212,
filed Aug. 11, 2006, now U.S. Pat. No. 7,315,631.
[0009] This application also is related to U.S. patent application
Ser. No. 11/674,650, filed Feb. 13, 2007, which claims priority to
U.S. provisional patent application Ser. No. 60/773,714, filed Feb.
14, 2006.
[0010] This application is related to U.S. Ser. No. 11/836,744,
filed Aug. 9, 2007, which claims priority to U.S. provisional
patent application Ser. No. 60/821,956, filed Aug. 9, 2006.
[0011] This application is also related to U.S. Ser. Nos.
12/140,048, 12/140,125, 12/140,532, 12/140,827, 12/140,950, and
12/141,042.
[0012] Each of these priority and related applications are hereby
incorporated by reference.
FIELD OF THE INVENTION
[0013] The present invention relates to digital image processing,
and more particularly to a method and apparatus for red-eye
detection in an acquired digital image.
BACKGROUND TO THE INVENTION
[0014] Red-eye is a phenomenon in flash photography where a flash
is reflected within a subjects eye and appears in a photograph as a
red dot where the black pupil of the subject's eye would normally
appear. The unnatural glowing red of an eye is due to internal
reflections from the vascular membrane behind the retina, which is
rich in blood vessels. This objectionable phenomenon is well
understood to be caused in part by a small angle between the flash
of the camera and the lens of the camera. This angle has decreased
with the miniaturization of cameras with integral flash
capabilities. Additional contributors include the relative
closeness of the subject to the camera and ambient light
levels.
[0015] Digital cameras are becoming more popular and smaller in
size. U.S. Pat. No. 6,407,777 to DeLuca describes a method and
apparatus where a red eye filter is digitally implemented in the
capture device. The success or failure of such filter relies on the
quality of the detection and correction process.
[0016] Most algorithms that involve image analysis and
classification, are statistical in nature. There is therefore a
need to develop tools which will improve the probability of
successful detection, while reducing the probability of false
detection, while maintaining optimal execution, especially in
limited computational devices such as in digital cameras. In many
cases knowledge of the image characteristics such as image quality
may affect the design parameters and decisions the detection and
correction software needs to implement. For example an image with
suboptimal exposure may deteriorate the overall detection of
red-eye defects.
[0017] Thus, what is needed is a method of improving the efficiency
and/or success rate of algorithms for detecting and reducing
red-eye phenomenon.
SUMMARY OF THE INVENTION
[0018] A method for red-eye detection in an acquired digital image
is provided including acquiring one or more preview or other
reference images without a flash, and determining any red regions
that exist within said one or more reference images. A main image
is acquired with a flash of approximately a same scene as the one
or more reference images. The main image is analyzed to determine
any candidate red eye defect regions that exist within the main
image. Any red regions determined to exist within the one or more
reference images are compared with any candidate red eye defect
regions determined to exist within the main image. Candidate red
eye defect regions within the main image are removed as candidates
if any corresponding red regions are determined also to exist
within the one or more reference images.
[0019] The analyzing may include applying a chain of one or more
red-eye filters to said main image. The red-eye filter chain may
include: (i) a pixel locator and segmentor; (ii) a shape analyser;
(iii) a falsing analyser; and (iv) a pixel modifier. The pixel
locator and segmentor may include a pixel transformer.
[0020] A third acquired image may be corrected based on any red eye
defect regions determined in the main image. The third acquired
image may include a sub-sampled copy of the main image.
[0021] The analyzing may include recognizing one or more faces or
types of faces, or combinations thereof, in the main image.
[0022] The method may include correcting the main image based on
analysis of at least one preview image and determining a degree of
blur; a degree of dust contamination; a color imbalance; a white
imbalance; a gamma error; a texture characteristic error; or noise
characteristics, or combinations thereof. The method may also
include recognizing a same face or type of face, or both, in a
second image as in the main image, and applying one or more same
corrective processes to the second image as the main image.
[0023] The method may include determining correcting to apply to
the main image based on analysis of at least one preview image. The
correcting may include (i) contrast normalization and image
sharpening; (ii) image color adjustment and tone scaling; (iii)
exposure adjustment and digital fill flash; (iv) brightness
adjustment with color space matching; (v) image auto-gamma
determination with image enhancement.; (v) image enhancement; or
(vi) face based image enhancement, or combinations thereof.
[0024] The method may include, responsive to determining to apply
correcting to the main image, also determining that such correcting
cannot be beneficially applied to the main image, and disabling the
correcting and providing an indication of such to a user.
[0025] The determining to apply correcting may include interacting
with a user to determine the corrections to be made and/or
performing a color space transformation.
[0026] The method may include determining a sequence in which to
apply more than one correcting action to the main image.
[0027] The method may be performed in whole or in part within a
portable digital camera.
[0028] A digital image acquisition device is also provided,
including one or more optics and a sensor for acquiring digital
images including main images and relatively low resolution preview
images, a processor, and a processor readable medium having stored
thereon digital code for programming the processor to perform any
of the methods of red eye detection or other image correction
described herein.
BRIEF DESCRIPTION OF THE DRAWINGS
[0029] FIG. 1(a) shows a prior art in-camera redeye detection
system;
[0030] FIG. 1(b) shows an improved redeye detection system
according to an embodiment of the present invention;
[0031] FIG. 2(a) is a flowchart illustrating the operation of the
system of FIG. 1(b);
[0032] FIG. 2(b) is a flowchart illustrating an alternative mode of
operation of the system of FIG. 1(b);
[0033] FIG. 2(c) illustrates another alternative mode of operation
of the system of FIG. 1(b);
[0034] FIG. 2(d) is a flowchart illustrating a further alternative
mode of operation of the system of FIG. 1(b);
[0035] FIG. 2(e) is a flowchart illustrating a still further
alternative mode of operation of the system of FIG. 1(b);
[0036] FIG. 3 shows the redeye filter chain of FIG. 1(b) in more
detail;
[0037] FIG. 4(a) illustrates the operation of portions of FIGS.
2(a), 2(b), 2(d) & 2(e) in more detail;
[0038] FIG. 4(b) illustrates an alternative implementation of FIG.
4(a);
[0039] FIG. 4(c) is a flowchart illustrating the operation of a
portion of the system of FIG. 1(b); and
[0040] FIGS. 5(a) and 5(b) illustrate the operation of a red-eye
filter chain according to an embodiment of the present
invention.
DETAILED DESCRIPTION OF THE INVENTION
[0041] Several embodiments are described herein that use
information obtained from reference images for processing a main
image. That is, the data that are used to process the main image
come at least not solely from the image itself, but instead or also
from one or more separate "reference" images.
Reference Images
[0042] Reference images provide supplemental meta data, and in
particular supplemental visual data to an acquired image, or main
image. The reference image can be a single instance, or in general,
a collection of one or more images varying from each other. The
so-defined reference image(s) provides additional information that
may not be available as part of the main image.
[0043] Example of a spatial collection may be multiple sensors all
located in different positions relative to each other. Example of
temporal distribution can be a video stream.
[0044] The reference image differs from the main captured image,
and the multiple reference images differ from each other in various
potential manners which can be based on one or combination of
permutations in time (temporal), position (spatial), optical
characteristics, resolution, and spectral response, among other
parameters.
[0045] One example is temporal disparity. In this case, the
reference image is captured before and/or after the main captured
image, and preferably just before and/or just after the main image.
Examples may include preview video, a pre-exposed image, and a
post-exposed image. In certain embodiments, such reference image
uses the same optical system as the acquired image, while in other
embodiments, wholly different optical systems or optical systems
that use one or more different optical components such as a lens,
an optical detector and/or a program component.
[0046] Alternatively, a reference image may differ in the location
of secondary sensor or sensors, thus providing spatial disparity.
The images may be taken simultaneously or proximate to or in
temporal overlap with a main image. In this case, the reference
image may be captured using a separate sensor located away from the
main image sensor. The system may use a separate optical system, or
via some splitting of a single optical system into a plurality of
sensors or a plurality of sub-pixels of a same sensor. As digital
optical systems become smaller dual or multi sensor capture devices
will become more ubiquitous. Some added registration and/or
calibration may be typically involved when two optical systems are
used.
[0047] Alternatively, one or more reference images may also be
captured using different spectral responses and/or exposure
settings. One example includes an infra red sensor to supplement a
normal sensor or a sensor that is calibrated to enhance specific
ranges of the spectral response such as skin tone, highlights or
shadows.
[0048] Alternatively, one or more reference images may also be
captured using different capture parameters such as exposure time,
dynamic range, contrast, sharpness, color balance, white balance or
combinations thereof based on any image parameters the camera can
manipulate.
[0049] Alternatively, one or more reference images may also be
captured using a secondary optical system with a differing focal
length, depth of field, depth of focus, exit pupil, entry pupil,
aperture, or lens coating, or combinations thereof based on any
optical parameters of a designed lens.
[0050] Alternatively, one or more reference images may also capture
a portion of the final image in conjunction with other
differentials. Such example may include capturing a reference image
that includes only the center of the final image, or capturing only
the region of faces from the final image. This allows saving
capture time and space while keeping as reference important
information that may be useful at a later stage.
[0051] Reference images may also be captured using varying
attributes as defined herein of nominally the same scene recorded
onto different parts of a same physical sensor. As an example, one
optical subsystem focuses the scene image onto a small area of the
sensor, while a second optical subsystem focuses the scene image,
e.g., the main image, onto a much larger area of the sensor. This
has the advantage that it involves only one sensor and one
post-processing section, although the two independently acquired
scene images will be processed separately, i.e., by accessing the
different parts of the sensor array. This approach has another
advantage, which is that a preview optical system may be configured
so it can change its focal point slightly, and during a capture
process, a sequence of preview images may be captured by moving an
optical focus to different parts of the sensor. Thus, multiple
preview images may be captured while a single main image is
captured. An advantageous application of this embodiment would be
motion analysis.
[0052] Getting data from a reference image in a preview or postview
process is akin to obtaining meta data rather than the
image-processing that is performed using the meta data. That is,
the data used for processing a main image, e.g., to enhance its
quality, is gathered from one or more preview or postview images,
while the primary source of image data is contained within the main
image itself. This preview or postview information can be useful as
clues for capturing and/or processing the main image, whether it is
desired to perform red-eye detection and correction, face tracking,
motion blur processing, dust artefact correction, illumination or
resolution enhancement, image quality determination,
foreground/background segmentation, and/or another image
enhancement processing technique. The reference image or images may
be saved as part of the image header for post processing in the
capture device, or alternatively after the data is transferred on
to an external computation device. In some cases, the reference
image may only be used if the post processing software determines
that there is missing data, damaged data or need to replace
portions of the data.
[0053] In order to maintain storage and computation efficiency, the
reference image may also be saved as a differential of the final
image. Example may include a differential compression or removal of
all portions that are identical or that can be extracted from the
final image.
Correcting Eye Defects
[0054] In one example involving red-eye correction, a face
detection process may first find faces, find eyes in a face, and
check if the pupils are red, and if red pupils are found, then the
red color pupils are corrected, e.g., by changing their color to
black. Another red-eye process may involve first finding red in a
digital image, checking whether the red pixels are contained in a
face, and checking whether the red pixels are in the pupil of an
eye. Depending on the quality of face detection available, one or
the other of these may be preferred. Either of these may be
performed using one or more preview or postview images, or
otherwise using a reference image, rather than or in combination
with, checking the main image itself. A red-eye filter may be based
on use of acquired preview, postview or other reference image or
images, and can determine whether a region may have been red prior
to applying a flash.
[0055] Another known problem involves involuntary blinking. In this
case, the post processing may determine that the subject's eyes
were closed or semi closed. If there exists a reference image that
was captured time-wise either a fraction of a second before or
after such blinking, the region of the eyes from the reference
image can replace the blinking eye portion of the final image.
[0056] In some cases as defined above, the camera may store as the
reference image only high resolution data of the Region of Interest
(ROI) that includes the eye locations to offer such retouching.
Face Tools
[0057] Multiple reference images may be used, for example, in a
face detection process, e.g., a selected group of preview images
may be used. By having multiple images to choose from, the process
is more likely to have a more optimal reference image to operate
with. In addition, a face tracking process generally utilizes two
or more images anyway, beginning with the detection of a face in at
least one of the images. This provides an enhanced sense of
confidence that the process provides accurate face detection and
location results.
[0058] Moreover, a perfect image of a face may be captured in a
reference image, while a main image may include an occluded profile
or some other less than optimal feature. By using the reference
image, the person whose profile is occluded may be identified and
even have her head rotated and unblocked using reference image data
before or after taking the picture. This can involve upsampling and
aligning a portion of the reference image, or just using
information as to color, shape, luminance, etc., determined from
the reference image. A correct exposure on a region of interest or
ROI may be extrapolated using the reference image. The reference
image may include a lower resolution or even subsampled resolution
version of the main image or another image of substantially a same
scene as the main image.
[0059] Meta data that is extracted from one or more reference
images may be advantageously used in processes involving face
detection, face tracking, red-eye, dust or other unwanted image
artefact detection and/or correction, or other image quality
assessment and/or enhancement process. In this way, meta data,
e.g., coordinates and/or other characteristics of detected faces,
may be derived from one or more reference images and used for main
image quality enhancement without actually looking for faces in the
main image.
[0060] A reference image may also be used to include multiple
emotions of a single subject into a single object. Such emotions
may be used to create more comprehensive data of the person, such
as smile, frown, wink, and/or blink. Alternatively, such data may
also be used to post process editing where the various emotions can
be cut-and-pasted to replace between the captured and the reference
image. An example may include switching between a smile to a
sincere look based on the same image.
[0061] Finally, the reference image may be used for creating a
three-dimensional representation of the image which can allow
rotating subjects or the creation of three dimensional
representations of the scene such as holographic imaging or
lenticular imaging.
Motion Correction
[0062] A reference image may include an image that differs from a
main image in that it may have been captured at a different time
before or after the main image. The reference image may have
spatial differences such as movements of a subject or other object
in a scene, and/or there may be a global movement of the camera
itself. The reference image may, preferably in many cases, have
lower resolution than the main image, thus saving valuable
processing time, bytes, bitrate and/or memory, and there may be
applications wherein a higher resolution reference image may be
useful, and reference images may have a same resolution as the main
image. The reference image may differ from the main image in a
planar sense, e.g., the reference image can be infrared or Gray
Scale, or include a two bit per color scheme, while the main image
may be a full color image. Other parameters may differ such as
illumination, while generally the reference image, to be useful,
would typically have some common overlap with the main image, e.g.,
the reference image may be of at least a similar scene as the main
image, and/or may be captured at least somewhat closely in time
with the main image.
[0063] Some cameras (e.g., the Kodak V570, see
http://www.dcviews.com/_kodak/v570.htm) have a pair of CCDs, which
may have been designed to solve the problem of having a single zoom
lens. A reference image can be captured at one CCD while the main
image is being simultaneously captured with the second CCD, or two
portions of a same CCD may be used for this purpose. In this case,
the reference image is neither a preview nor a postview image, yet
the reference image is a different image than the main image, and
has some temporal or spatial overlap, connection or proximity with
the main image. A same or different optical system may be used,
e.g., lens, aperture, shutter, etc., while again this would
typically involve some additional calibration. Such dual mode
system may include a IR sensor, enhanced dynamic range, and/or
special filters that may assist in various algorithms or
processes.
[0064] In the context of blurring processes, i.e., either removing
camera motion blur or adding blur to background sections of images,
a blurred image may be used in combination with a non-blurred image
to produce a final image having a non-blurred foreground and a
blurred background. Both images may be deemed reference images
which are each partly used to form a main final image, or one may
be deemed a reference image having a portion combined into a main
image. If two sensors are used, one could save a blurred image at
the same time that the other takes a sharp image, while if only a
single sensor is used, then the same sensor could take a blurred
image followed by taking a sharp image, or vice-versa. A map of
systematic dust artefact regions may be acquired using one or more
reference images.
[0065] Reference images may also be used to disqualify or
supplement images which have with unsatisfactory features such as
faces with blinks, occlusions, or frowns.
Foreground/Background Processing
[0066] A method is provided for distinguishing between foreground
and background regions of a digital image of a scene. The method
includes capturing first and second images of nominally the same
scene and storing the captured images in DCT-coded format. These
images may include a main image and a reference image, and/or
simply first and second images either of which images may comprise
the main image. The first image may be taken with the foreground
more in focus than the background, while the second image may be
taken with the background more in focus than the foreground.
Regions of the first image may be assigned as foreground or
background according to whether the sum of selected high order DCT
coefficients decreases or increases for equivalent regions of the
second image. In accordance with the assigning, one or more
processed images based on the first image or the second image, or
both, are rendered at a digital rendering device, display or
printer, or combinations thereof.
[0067] This method lends itself to efficient in-camera
implementation due to the relatively less-complex nature of
calculations utilized to perform the task.
[0068] In the present context, respective regions of two images of
nominally the same scene are said to be equivalent if, in the case
where the two images have the same resolution, the two regions
correspond to substantially the same part of the scene. If, in the
case where one image has a greater resolution than the other image,
the part of the scene corresponding to the region of the higher
resolution image is substantially wholly contained within the part
of the scene corresponding to the region of the lower resolution
image. Preferably, the two images are brought to the same
resolution by sub-sampling the higher resolution image or
upsampling the lower resolution image, or a combination thereof.
The two images are preferably also aligned, sized or other process
to bring them to overlapping as to whatsoever relevant parameters
for matching.
[0069] Even after subsampling, upsampling and/or alignment, the two
images may not be identical to each other due to slight camera
movement or movement of subjects and/or objects within the scene.
An additional stage of registering the two images may be
utilized.
[0070] Where the first and second images are captured by a digital
camera, the first image may be a relatively high resolution image,
and the second image may be a relatively low resolution pre- or
post-view version of the first image.
[0071] While the image is captured by a digital camera, the
processing may be done in the camera as post processing, or
externally in a separate device such as a personal computer or a
server computer. In such case, both images can be stored. In the
former embodiment, two DCT-coded images can be stored in volatile
memory in the camera for as long as they are being used for
foreground/background segmentation and final image production. In
the latter embodiment, both images may be preferably stored in
non-volatile memory. In the case of lower resolution pre-or-post
view images, the lower resolution image may be stored as part of
the file header of the higher resolution image.
[0072] In some cases only selected regions of the image are stored
as two separated regions. Such cases include foreground regions
that may surround faces in the picture. In one embodiment, if it is
known that the images contain a face, as determined, for example,
by a face detection algorithm, processing can be performed just on
the region including and surrounding the face to increase the
accuracy of delimiting the face from the background.
[0073] Inherent frequency information as to DCT blocks is used to
provide and take the sum of high order DCT coefficients for a DCT
block as an indicator of whether a block is in focus or not. Blocks
whose high order frequency coefficients drop when the main subject
moves out of focus are taken to be foreground with the remaining
blocks representing background or border areas. Since the image
acquisition and storage process in a digital camera typically codes
captured images in DCT format as an intermediate step of the
process, the method can be implemented in such cameras without
substantial additional processing.
[0074] This technique is useful in cases where differentiation
created by camera flash, as described in U.S. application Ser. No.
11/217,788, published as 2006/0039690, incorporated by reference
(see also U.S. Ser. No. 11/421,027) may not be sufficient. The two
techniques may also be advantageously combined to supplement one
another.
[0075] Methods are provided that lend themselves to efficient
in-camera implementation due to the computationally less rigorous
nature of calculations used in performing the task in accordance
with embodiments described herein.
[0076] A method is also provided for determining an orientation of
an image relative to a digital image acquisition device based on a
foreground/background analysis of two or more images of a
scene.
Referring to the Figures
[0077] FIG. 1 illustrates a prior art in-camera redeye system.
Within the camera 100 a main image is acquired 105 from a sensor
subsystem. This image is further processed 110 based on image
acquisition parameters such as ambient lighting, length of
exposure, usage of pre-flash and flash, lens focal length &
aperture settings, etc. This image processing is pre-calibrated
during the design of the camera and, due to the non-linear
relationships between the various acquisition parameters, it
typically involves a significant amount of empirical testing using
as broad a range of image capture conditions as is practical. Thus,
even though modern digital cameras have much improved auto-focus
and auto-exposure algorithms it is still possible to capture images
of non-optimal quality either through incorrect camera settings or
through encountering conditions which are not fully accounted for
by the empirical calibrations process for that camera.
[0078] After this image processing is completed the main acquired
and processed image is normally committed to non-volatile storage
in camera memory, or in an onboard storage card 170. However if the
image was captured using a flash then the possibility of redeye
defects implies that the image should first be passed through an
in-camera redeye filter 90. A more detailed description of such a
filter can be found in U.S. Pat. No. 6,407,777 to DeLuca herein
incorporated by reference. Briefly it comprises of (i) a pixel
locator filter 92 which detects candidate eye-defect pixels based
on a color analysis and then groups said pixels into redeye
candidate regions; (ii) a shape analyzer filter 94 which determines
if a eye candidate region is acceptable in terms of geometry, size
and compactness and further analyzes neighbouring features such as
eyebrows and iris regions; and (iii) a falsing filter 98 which
eliminates candidate regions based on a wide range of criteria. Any
candidate regions which survive the falsing filter are then
modified by a pixel modifier 96 and the corrected image 170-2 may
then be stored in the main image store 170.
[0079] This prior art system typically will also feature a
sub-sampler which can generate lower resolution versions 170-3 of
the main acquired and processed image 170-1. This sub-sampling unit
may be implemented in either software or may be hardware based and
is, primarily, incorporated in modern digital cameras to facilitate
the generation of thumbnail images for the main camera display.
[0080] FIG. 1(b) illustrates a preferred embodiment of red-eye
detection system according to the present invention. The system
improves on the prior art by providing an additional image analysis
prefilter 130 and an image compensation prefilter 135 to the prior
art imaging chain to reduce the overall incidence of errors in the
redeye detection process 90 for non-optimally acquired images.
[0081] The image analysis prefilter 130 combines one or more
techniques for determining image quality. Such techniques are well
known to one familiar in the art of image processing and in
particular image editing and enhancements. Thus, the prefilter
provides an in-camera analysis of a number of characteristics of an
acquired, processed image with a view to determining if these
characteristics lie within acceptable limits. It will be clear to
those skilled in the art that the exact combination of analysis
techniques will be dependent on the characteristics of the
non-optimally acquired images generated by a particular digital
camera. In addition, the determination of what image quality
matters need to be addressed is primarily dependent on the effect
of such characteristics on the red eye filter 90. Thus, as
illustrative examples: (i) a low-end digital camera may omit
complex noise filtering circuitry on its sensor as it is targeted
at cost-sensitive markets and may employ low quality optics for
similar reasons. Thus it may be susceptible to a greater degree of
image noise and exhibit a poor dynamic range for white and color
balance; (ii) a high-end professional camera will have a much
greater dynamic range for color and white balance but may require
more sophisticated image analysis to compensate for motion blur,
sensor dust and other image distortions that are of concern to
professional photographers.
[0082] Accordingly we shall provide some examples of image analysis
techniques for exemplary purposes only and it will be understood
these are not intended to limit the techniques which may be
utilized in implementing the present invention.
[0083] One subsystem of the image analysis prefilter is a blur
analyzer 130-1, which performs an image analysis to determine
blurred regions within a digital image--this operate on either the
full size main image 170-1 or one or more sub-sampled copies of the
image 170-3. One technique for in-camera blur detection is outlined
in US patent application 2004/0120598 to Feng which describes a
computationally efficient means to determine blur by analysis of
DCT coefficients in a JPEG image. In common with the other
sub-systems of the prefilter 130, the analyser provides a measure
of the blur in the supplied image(s) to be used later in the
prefilter 135. This measure could be as simple as an index between
0 and 1 indicating the degree of blur. However, it could also
indicate which regions of the image are blurred and the extent to
which these are blurred.
[0084] A further subsystem of the image analysis prefilter is a
dust analyzer 130-2. The problems caused by dust on imaging devices
are well known in the prior art. In the context of the present
invention it is important to track the location and severity of
dust particles as these may interfere with the correct detection of
eye-defects when the two forms of defect overlap. Of particular
relevance are techniques where the detection of defects in a
digital image is based solely on analysis of the digital image and
that do not directly relate to the image acquisition process. For
example U.S. Pat. No. 6,233,364 to Krainiouk et al. discloses
determining anomalous image regions based on the difference between
the gradient of an image at a set of grid points and the local mean
of the image gradient. This technique generates few false positives
in "noisy" regions of an image such as those representing leaves in
a tree, or pebbles on a beach. U.S. Pat. No. 6,125,213 to Morimoto
discloses detecting potential defect or "trash" regions within an
image based on a comparison of the quadratic differential value of
a pixel with a pre-determined threshold value. In addition,
Morimoto discloses correcting "trash" regions within an image by
successively interpolating from the outside of the "trash" region
to the inside of this region--although this does not need to be
performed by the subsystem 130-2. U.S. Pat. No. 6,266,054 to Lawton
et al discloses automating the removal of narrow elongated
distortions from a digital image utilizing the characteristics of
image regions bordering the distortion. US patent application
2003/0039402 and WIPO patent application WO-03/019473 both to
Robins et al. disclose detecting defective pixels by applying a
median filter to an image and subtracting the result from the
original image to obtain a difference image. This is used to
construct at least one defect map and as such provide a measure of
the effect of dust on an image supplied to the subsystem 130-2.
[0085] U.S. Pat. No. 6,035,072 to Read discloses mapping defects or
dirt, which affect an image acquisition device. A plurality of
images are processed and stationary components which are common
between images are detected and assigned a high probability of
being a defect. Additional techniques which are employed to modify
defect probability include median filtering, sample area detection
and dynamic adjustment of scores. This dynamic defect detection
process allows defect compensation, defect correction and alerting
an operator of the likelihood of defects, but from the point of
view of the preferred embodiment, it is the map which is produced
which indicates to the prefilter 135 the degree to which the
supplied images are affected by dust and/or defects.
[0086] Additional subsystems of the image analysis prefilter are a
white balance analyzer 130-3, a color balance analyzer 130-4, and a
gamma/luminance analyzer 130-5. In the embodiment, each of these
provides, for example, an indicator of the degree to which each of
these characteristics deviates from optimal and by which the
supplied image might be corrected. Those skilled in the art will
realize that such techniques are practiced in a digital camera as
part of corrective image processing based on acquisition settings
110. Prior art techniques which can be employed in embodiments of
the present invention also exist for post-processing of an acquired
image to enhance its appearance. Some representative examples are
now described:
[0087] U.S. Pat. No. 6,249,315 to Holm teaches how a spatially
blurred and sub-sampled version of an original image can be used to
obtain statistical characteristics of a scene or original image. In
Holm, this information is combined with the tone reproduction
curves and other characteristics of an output device or media to
provide an enhancement strategy for digital images, whereas in the
preferred embodiment, an analysis prefilter employing the technique
of Holm preferably provides the color characteristics of the
supplied image to the prefilter 135.
[0088] U.S. Pat. No. 6,268,939 to Klassen et al. teaches correcting
luminance and chrominance data in digital color images.
Specifically, Klassen is concerned with optimizing the
transformations between device dependent and device independent
color spaces by applying subsampling of the luminance and
chrominance data.
[0089] U.S. Pat. No. 6,192,149 to Eschback et al. discloses
improving the quality of a printed image by automatically
determining the image gamma and then adjusting the gamma of a
printer to correspond to that of the image. Although Eschback is
concerned with enhancing the printed quality of a digital image and
not the digital image itself, if does teach a means for
automatically determining the gamma of a digital image and as such
can be used in an analysis pre-filter in embodiments of the present
invention. U.S. Pat. No. 6,101,271 to Yamashita et al. discloses
implementing a gradation correction to an RGB image signal which
allows image brightness to be adjusted without affecting the image
hue and saturation
[0090] A further subsystem of the image analysis prefilter is an
image texture analyzer 130-6 which allows texture information to be
gathered from the acquired and processed main image. This
information can be useful both in determining different regions
within an image and, when combined with information derived from
other image analysis filters such as the blur analyzer 130-1 or a
noise analyzer 130-7 it can enable automatically enhancement of an
image by applying deblurring or denoising techniques. US patent
application 2002/0051571 to Jackway et al discloses texture
analysis for digital images. US patent application 2002/0090133 to
Kim et al discloses measuring color-texture distances within a
digital images and thus offering improved segmentation for regions
within digital images.
[0091] A further subsystem of the image analysis prefilter is the
noise analyzer 130-7 which produces a measure of the effect of
noise on the image supplied to the subsystem 130-7. A further
illustrative subsystem of the image analysis prefilter 130 is an
object/region analyzer 130-8 which allows localized analysis of
image regions. One particular region which will invariably be found
in an image with eye-defects is a human face region. The detection
of a face region in an image with eye-defects is simplified as
described in US patent application 2004/0119851 to Kaku. Again, an
analysis pre-filter employing Kaku would therefore provide
indicators of where faces regions are to be found in a supplied
image to the pre-filter 135.
[0092] The last illustrative subsystem of the image analysis
prefilter 130 is a face recognition analyzer 130-9 which includes a
database of pre-determined data obtained from training performed on
a personal image collection (not shown) loaded onto the digital
camera in order to recognize a person associated with a determined
region preferably acquired by the analyzer 130-8 and to provide an
indicator of the person or person(s) whose faces have been
recognized in an image. Alternatively, the face recognition
analyzer 130-9 may provide an indicator of the types of any faces
recognized in the image provided to the pre-filter 130-9, for
example, a child or adult face, or African, Asian or Caucasian
face.
[0093] In one embodiment, the analyser 130-9 comprises a set of
classifiers which enable multiple sets of face (and/or non-face)
data to be combined to provide improved recognition of persons
found in an image. The types of classifiers used can be based on
skin color, age characteristics, eye-shape and/or eye-brow
thickness, the person's hair and/or clothing, poses associated with
a person and/or whether or not a person may be wearing makeup, such
as eye-shadow or lipstick, or glasses, as preferably obtained from
the training performed on the personal image collection.
[0094] One particular advantage of employing a face recognition
analyzer 130-9 as an element of the image analysis prefilter is
that it enables additional image processing modules to perform face
and peripheral region analysis which will enable a determination of
known persons within an image. A more detailed description of the
preferred person recognizer 135-2a is provided in co-pending
application Ser. No. 11/027,001, filed Dec. 29, 2004, and hereby
incorporated by reference. For the person recognizer 135-2a to
function more effectively an additional database component
containing classifier signatures associated with known persons is
preferably included. This database will typically be derived from a
personal collection of images maintained by the owner of a digital
camera and, in most typical embodiments, these will be stored
off-camera. Further details on the creation and management of
exemplary embodiments of such image collections and associated
off-camera and in-camera databases is given in co-pending
application Ser. No. 10/764,339, filed Jan. 22, 2004 and Ser. No.
11/027,001, filed Dec. 29, 2004 which are hereby incorporated by
reference.
[0095] The image analysis prefilter may also incorporate a module
to separate background and foreground regions of an image (not
shown). Such a module is described in co-pending application
entitled "Foreground/Background Segmentation in Digital Images With
Differential Exposure Calculations, serial number not yet assigned
(FN-122), filed Aug. 30, 2005, hereby incorporated by reference,
and may be advantageously employed to reduce the area of an image
to which a redeye filter is applied, thus speeding up the execution
time. In such a case the image is not necessarily corrected, or the
filter chain is not necessarily adapted but the method of
application of the filter chain to the image is altered.
[0096] Turning now to the image compensation prefilter 135. In the
present embodiment, a combination of image correction analyzer
135-2 and a redeye subfilter database 135-3 p1 (i) interpret the
results of the image analysis performed by the image analysis
prefilter 130; (ii) if corrective image processing is active,
determine an optimal correction strategy for application to the
acquired, processed image, or a subsampled copy thereof, (iii) if
adaption of the redeye filter chain is implemented, determine any
parameter/filter conflicts and further determines an optimal
adaption of the redeye filter chain (described later); and (iv) if
both corrective image processing and filter adaption are active,
determine an optimal combination of each.
[0097] For example, if the analyzer 130-9 has recognized one or
more persons or types of persons in an image, a customized redeye
filter set stored as a set of rules in the database 135-3 may be
applied to the image. To understand how such customization can
improve the performance of a redeye filter we cite some examples of
known aspects of the redeye phenomenon which are person
specific.
[0098] For example, children and babies are particularly
susceptible to redeye. They are also more prone to certain types of
redeye, e.g. "bright-eye" where the eye is almost completely white
with only a reddish periphery, which can be often more difficult to
analyze and correct.
[0099] In addition, racial or ethnic characteristics can cause
differences in the color characteristics of the redeye phenomenon.
For example, Asian people often exhibit a dull reddish or even
"brownish" form of redeyem while persons of Indian descent often
exhibit redeye effects with a distinctly "purplish" hue. The extent
of these characteristics may vary somewhat from individual to
individual.
[0100] As such, knowledge of the type of person in an image can be
used by the analyzer 135-2 to determine the filters, the order of
the filters and/or the filter parameters to be applied to an image.
For example, the filter parameters may be changed on the basis of
skin color in that a distinctive set of prototype values could be
available for each person; or age characteristics, to enable a
higher tolerance of certain color and/or luminance-based filters;
eye-shape and/or eye-brow thickness which are person specific;
and/or whether or not a person is wearing glasses, which can
introduce strong glints resulting in detection errors for standard
filter sets. Similarly, the filter order may be changed depending,
on the `identity` of the person in the image, i.e. whether or not
the person is wearing makeup and/or glasses. For example, if a
person is wearing eye shadow and/or lipstick, certain skin filters
might not be applied. Instead, alternative filters could be used to
determine a uniform color/texture in place of the normal skin
filter.
[0101] The actual corrective image processing 135-1 will typically
be implemented as a library of image processing algorithms which
may be applied in a variety of sequences and combinations to be
determined by the image correction analyzer 135-2. In many digital
cameras some of these algorithms will have partial or full hardware
support thus improving the performance of the compensation
prefilter 135.
[0102] It was already remarked that the analysis prefilter 130 can
operate on a subsampled copy of the main image 170-3. In the same
way the detection phase of the redeye filter 90 can be applied to a
subsampled copy of the main image 170-3, although not necessarily
of the same resolution. Thus where corrective image processing is
used by the image compensation prefilter it will also be applied to
a subsampled copy of the main image 170-3. This has significant
benefits with respect to computation speed and computing resources,
making it particularly advantageous for in-camera embodiments.
[0103] The image correction analyzer 135-2 may not always be able
to determine an optimal correction strategy for an acquired,
processed image due to conflicts between image processing
algorithms, or between the filter adaptions required for the redeye
filter chain. In other instances, where a strategy can be
determined but the image correction analyzer 135-2 may be aware
that the strategy is marginal and may not improve image quality it
may be desirable to obtain user input. Thus the image correction
analyzer 135-2 may generate a user indication 140 and in certain
embodiments may also employ additional user interaction to assist
in the image correction and redeye filter processes.
[0104] FIG. 2a to FIG. 2e illustrate several alternative
embodiments of the present invention which are described as
follows: [0105] (i) In FIG. 2(a) an acquired, processed main image,
or alternatively a subsampled copy thereof, is initially loaded,
step 201 to respective sub-systems of the analysis prefilter 130,
step 202. These produce their measurements and a determination is
made if any of the image quality characteristics lie within or
outside acceptable thresholds is made by the image correction
analyser 135-2, step 204. If image quality is within acceptable
limits for each of the image characteristics analyzed then the
redeye filter 90 can be applied normally and no corrective image
processing is required. However, if certain image characteristics
do lie outside acceptable tolerances then additional analysis is
performed by the analyser 135-2 to determine if corrective image
processing can be applied 206. If some of the analyzed image
characteristics lie too far outside acceptable thresholds, or if a
disadvantageous combination of image characteristics is determined,
it may not be possible to correct the image reliably prior to
applying the redeye filter. Thus the filter 90 can be disabled 220,
a user indication 140 can be provided and processing is completed
for this particular image 224, without performing the red eye
correction or performing the process with lower probability of
success. However, if the image can be repaired, 206-YES, the image
is corrected step 208, prior to executing the red eye algorithm 90.
In the preferred embodiment, the process of correcting the image,
208 may be performed on the full resolution image, or alternatively
on a subsampled image or a copy of the image. The exact nature and
possibilities for such corrections, 208, whether locally or
globally are described later. In any case, the corrected image
needs only be stored temporarily and can be discarded after red-eye
processing is complete, 209. It should be noted that performing the
pre-filtering, 208 on the image, does not means that the actual
red-eye detection and reduction algorithm, 90 has to be modified to
account for possible variability. Nonetheless, as image quality
supplied to the filter 90 is improved, the red eye algorithm can
use tighter parameters and more well defined restrictions as to the
nature of the red eye features that are to be identified so
producing improved results. [0106] (ii) FIG. 2(b) corresponds with
FIG. 2(a) except that it includes an additional determining step,
240 which follows the determination that corrective image
processing is possible, 206. This additional step determines if the
corrective image processing to be applied to the image can be
provided by a globally applied transformation of the image pixels.
The most popular global transformations are matrix multiplication
or lookup table transformations. For example, the analysis provided
by filters 130-3 . . . 130-5 may indicate to the analyser 135-2
that the principle cause of image non-optimality is a reddish color
cast. In this case, a simple transformation of the red image
component, R.fwdarw.R' is sufficient to compensate for the image
non-optimality. Another example will be an image that is under
exposed and a tone reproduction curve (TRC) needs to be corrected.
Global transformations have the advantage of being relatively
computationally efficient and with a potential to be highly
optimized. In addition, such transformations may be performed
within the redeye filter 90 itself, for example, as part of the
pixel locator and region segmentation process 92 described in more
detail later in relation to FIGS. 3 and 5, so reducing the overhead
involved in performing this correction. For the moment, it is
sufficient to say that in step 242, a pixel transformation within
the pixel locator and region segmentor 92 of the red-eye filter is
configured. It will also been seen that the steps 240, 242 may be
performed as an alternative to other corrections step 208, in
parallel with other corrections or in series with other corrections
prior to execution of the red-eye filter 90. [0107] (iii) In FIG.
2(c) instead of corrective image processing to compensate for a
non-optimally acquired image, the analyser 135-2 adapts the redeye
filter chain to provide image compensation for the redeye detection
process. Steps 202 and 204 are identical with previous embodiments.
However, when an image characteristic lies outside acceptable
tolerances, the analyser 135-2 determines the redeye subfilters
affected by said out-of-tolerance image characteristic, step 250.
Typically this determining step will involve the image correction
analyzer 135-2 obtaining the relevant data from an in-camera data
repository such as the redeye subfilter database 135-3. After the
affected subfilters have been determined 250, the next step is to
determine if subfilter compensation is possible 252. This will
depend on the different image characteristics which are outside
acceptable thresholds and the relevant sets of redeye subfilters
affected by each out-of-tolerance image characteristic. If filter
chain adaption is possible then the filter chain is modified 254
and the redeye filter is applied 90. If subfilter compensation is
not possible due to filter, or parameter-based conflicts then steps
220, 140, and 224 are performed as in the previous embodiments. The
subfilter determining process is further described in FIG. 4(b) and
an overview of the redeye subfilter matrix is given in FIG. 3.
[0108] The following example illustrates the concept of applying
the results of the analysis stage to modify the filter chain of the
correction process and the red eye detection process as opposed to
modification of the image pixels. It is assumed that a pixel
{R.sub.0,G.sub.0,B.sub.0} after the needed correction, step 208, is
transformed to pixel value {R.sub.1,G.sub.1,B.sub.1} by a
transformation T:
T[{R.sub.0,G.sub.0,B.sub.0}]={R.sub.1,G.sub.1,B.sub.1}. For
illustrative purposes, we assume that the first stage of the red
eye detection algorithm, as defined in block 92 of FIG. 1(a) is
comparison to a known value, to determine if the pixel is, in
simplified terms, red or not. The value of the pixel in to compare
with is {R',G',B'}. However, the two steps above of correcting and
comparing may be combined simply by transforming the static value
of {R',G',B'} based on the inverse of the correction
transformation. Thus, the preliminary preparatory stage will be:
{R'',G'',B''}=T.sup.-1[{R',G',B'}] and the pixel by pixel
comparison, as adapted, step 254 to the necessary needed
transformations will comprise the following test:
IF{R.sub.0,G.sub.0,B.sub.0}.gtoreq.{R'',G'',B''}. By doing so, the
entire image is not corrected, but the comparison is similar to the
state as if the image was corrected. The complexity and number of
necessary steps compared to the original algorithm is exactly the
same, with the extra value that the image algorithm now is taking
into account the sub-optimal quality of the image.
Mathematically Speaking:
[0109]
T[{R.sub.0,G.sub.0,B.sub.0}].alpha.{R',G',B'}={R.sub.0,G.sub.0,B.su-
b-.0}.alpha.T.sup.-1[{R',G',B'}]={R.sub.0,G.sub.0,B.sub.0}.alpha.{R'',G'',-
B'-'}
[0110] Where .alpha. denotes the relationship between the
objects.
[0111] Of course, such adaptation may be more complex than the
simplified example above, and may include change of multiple values
in the algorithm or change in the order the various filters are
applied, or change in the weight of the various filters. However,
the improvement in performance may justify the added architectural
complexity.
[0112] (iv) FIG. 2(d) illustrates a combination of the embodiments
described in 2(b) and 2(c). This embodiment is identical to the
previous embodiments except that if subfilter compensation is not
possible 252 it incorporates two additional steps to determining if
corrective image processing can be applied 206 and if this is
possible a second step 208 to apply said corrective image
processing. Note that subfilter adaption is preferred to corrective
image processing as it requires practically no computational
resources, but only changes the input parameters of the subfilters
which comprise the redeye filter chain and the composition and
order-of-execution of the chain itself. However in certain
circumstances correction of the original acquired image by image
processing means may provide more reliable redeye detection, or be
desirable as an end in itself.
[0113] (v) FIG. 2(e) describes an alternative variation of the
algorithm. This is identical to the embodiment of FIG. 2(a) except
that after determining if corrective image processing is possible
206, corrective image processing is applied to both the main
acquired image 170-1 and a subsampled copy 170-3 thereof, step
208-1. A second additional step then saves the corrected acquired
image 170-2, in the main image store 170, step 209, and a user
indication 140 is generated to inform the camera user that an
improved image is available. Additional steps may be added to allow
the user to select between original 170-1 and corrected images
170-2 if so desired. In this embodiment, redeye detection 92, 94,
98 is applied to the corrected subsampled copy of the main acquired
image and the redeye correction 96 is applied to the corrected copy
of the main acquired image. In other embodiments corrective image
processing would not be applied to the full-sized main image 170-1
so that the redeye correction would be applied to the uncorrected
main image.
[0114] FIG. 3 shows the principle subfilter categories which exist
within the main redeye filter 90. While each of the component
filters will be referred to in sequence, it will be appreciated
that where appropriate more than one of these filters may be
applied at a given time and the decisions above to modify the
filter chain can include a decision not alone as to which filters
may be executed in a sequence, but also on which filters can be
applied in parallel sequences. As described above, the pixel
transformer filter 92-0 allows global pixel-level transformations
of images during color determining and pixel grouping operations.
Also, within the pixel locator and region segmenter 92 we find
pixel color filters 92-1 which perform the initial determining if a
pixel has a color indicative of a flash eye defect; a region
segmentor 92-2 which segments pixels into candidate redeye
groupings; regional color filters 92-3, color correlation filters
92-4, and color distribution filters 92-5 which operate on
candidate regions based these criteria. In addition the pixel
locator and region segmenter 92 contains two additional functional
blocks which do not contribute directly to the color determining
and segmentation operations but are nevertheless intertwined with
the operation of the pixel locator and region segmenter. The
resegmentation engine 92-6 is a functional block which is
particularly useful for analyzing difficult eye defects. It allows
the splitting 92-6a and regrouping 92-6b of borderline candidate
regions based on a variety of threshold criteria.
[0115] After candidate eye-defect groupings have been determined by
the segmenter 92, a shape analyzer 94 next applies a set of
subfilters to determine if a particular candidate grouping is
physically compatible with known eye-defects. Thus some basic
geometric filters are first applied 94-1 followed by additional
filters to determine region compactness 94-2 and boundary
continuity 94-3. Further determining is then performed based on
region size 94-4, and a series of additional filters then determine
if neighbouring features exist which are indicative of eye shape
94-5, eyebrows 94-6 and iris regions 94-7. In certain embodiments
of the present invention the redeye filter may additionally use
anthropometric data to assist in the accurate determining of such
features.
[0116] Now the remaining candidate regions are passed to a falsing
analyzer 98 which contains a range of subfilter groups which
eliminate candidate regions based on a range of criteria including
lips filters 98-1, face region filters 98-2, skin texture filters
98-3, eye-glint filters 98-4, white region filters 98-5, region
uniformity filters 98-6, skin color filters 98-7, and eye-region
falsing filters 98-8. Further to these standard filters a number of
specialized filters may also be included as part of the falsing
analyzer 98. In particular we mention a category of filter based on
the use of acquired preview images 98-9 which can determine if a
region was red prior to applying a flash. This particular filter
may also be incorporated as part of the initial region determining
process 92, as described in co-pending U.S. application Ser. No.
10/919,226 from August, 2004 entitled "Red-Eye Filter Method And
Apparatus" herein incorporated by reference. An additional category
of falsing filter employs image metadata determined from the camera
acquisition process 98-10. This category of filter can be
particularly advantageous when combined with anthropometric data as
described in PCT Application No. PCT/EP2004/008706. Finally an
additional category of filter is a user confirmation filter 98-11
which can be optionally used to request a final user input at the
end of the detection process. This filter can be activated or
disabled based on how sub-optimal the quality of an acquired image
is.
[0117] The pixel modifier 96 is essentially concerned with the
correction of confirmed redeye regions. Where an embodiment of the
invention incorporates a face recognition module 130-9 then the
pixel modifier may advantageously employ data from an in-camera
known person database (not shown) to indicate aspects of the eye
color of a person in the image. This can have great benefit as
certain types of flash eye-defects in an image can destroy
indications of original eye color.
[0118] In the preferred embodiment, an additional component of the
redeye filter 90 is a filter chain adapter 99. This component is
responsible for combining, and sequencing the subfilters of the
redeye filter 90 and for activating each filter with a set of input
parameters corresponding to the parameter list(s) 99-1 supplied
from the image compensation prefilter 135.
[0119] Finally, it is remarked in the context of FIG. 3 that
although the pixel locator & region segmenter 92, the shape
analyzer 94 and the falsing analyzer 98 are illustrated as separate
components it is not intended to exclude the possibility that
subfilters from these components may be applied in out-of-order
sequences. As an illustrative example, regions which pass all the
falsing filters except for the region uniformity filter 98-6 may be
returned to the resegmentation engine 92-6 to determine if the
region was incorrectly segmented. Thus a subfilter from the pixel
locator and region segmentor 92 may be used to add an additional
capability to the falsing analysis 98.
[0120] FIG. 4 shows in more detail the operation of the image
analysis 130 and image compensation prefilters 135. In this example
the operation of the compensation prefilter 135, and more
particularly the operation of the image correction analyzer 135-2
has been separated into two functional modes: FIG. 4(a) illustrates
the workflow for the determining and performing corrective image
processing (so corresponding generally to steps 206, 208 of FIGS.
2(a),(b),(d) and (e)) while FIG. 4(b) describes the determining and
performing filter chain adaption including determining if a single
chain, or a combination of multiple filter chains will compensate
for the non-optimal image characteristics determined by the image
analysis prefilter 130 (so corresponding generally to step 250,252
and 254 of FIGS. 2(c) and 2(d)). FIG. 4(c) illustrates an exemplary
embodiment of the workflow of the image analysis prefilter 130.
[0121] In FIG. 4(a) the image correction analyzer 135-2 first loads
an image characteristic list 401 obtained from the image analysis
prefilter 130. This list will allow the correction analyzer to
quickly determine if a simple image correction is required or if a
number of image characteristics will require correction 402. In the
case of a single characteristic the correction analyzer can
immediately apply the relevant corrective image processing 412
followed by some tests of the corrected image 414 to ensure that
image quality is at least not deteriorated by the applied
corrective technique. If these tests are passed 416 then the image
can be passed on to the redeye filter 90 for eye defect correction.
Otherwise, if corrective image processing has failed the sanity
tests 416 then an additional test may be made to determine if
filter chain adaption is possible 422. In this case the algorithm
will initiate the workflow described in FIG. 4(b) for determining
the required filter chain adaptions 450. If corrective image
processing has failed 416 and filter chain adaption is not possible
422 then the correction analyzer will disable the redeye filter for
this image 220, and provide a user indication to that effect 140
after which it will pass control back to the main in-camera
application 224. Note that in certain embodiments the user
indication may be interactive and may provide an option to allow
the normal redeye filter process to proceed on the uncorrected
image, or alternatively offer additional user-selectable choices
for additional image analysis and/or correction strategies.
[0122] Now returning to the determining step between single and
multiple image characteristics requiring correction 402 we now
describe the correction approach for multiple image
characteristics. Typically an image which was non-optimally
acquired will suffer from one major deficiency and a number of less
significant deficiencies. We will refer to these as primary and
secondary image deficiencies. The next step in the workflow process
is to determine the primary image deficiency 404. After this has
been successfully determined from the image characteristics list
the next step is to determine the interdependencies between this
primary correction required and said secondary image
characteristics. Typically there will be more than one approach to
correcting the primary image characteristic and the correction
analyzer must next determine the effects of these alternative
correction techniques on the secondary image characteristics 406
before correction can be initiated. If any of the secondary
characteristics are likely to deteriorate significantly and all
alternative correction technique for the primary image
characteristic are exhausted then the correction analyzer may
determine that these interdependencies cannot be resolved 408. In
the present embodiment an additional test is next made to determine
if filter chain adaption is possible 422. In this case the
algorithm will initiate the workflow described in FIG. 4(b) for
determining the required filter chain adaptions 450. If corrective
image processing has failed 416 and filter chain adaption is not
possible 422 then the correction analyzer will disable the redeye
filter for this image 220, and provide a user indication to that
effect 140 after which it will pass control back to the main
in-camera application 224.
[0123] Given that the secondary interdependencies can be resolved
408 the correction analyzer next proceeds to determine the image
processing chain 410. In certain embodiments this step may
incorporate the determining of additional corrective techniques
which can further enhance the primary correction technique which
has been determined. In such an embodiment the correction analyzer
will, essentially, loop back through steps 404, 406, and 408 for
each additional correction technique until it has optimized the
image processing chain. It is further remarked that the determining
of step 408 will require access to a relatively complex
knowledgebase 135-4. In the present embodiment this is implemented
as a series of look-up-tables (LUTs) which may be embedded in the
non-volatile memory of a digital camera. The content of the
knowledgebase is highly dependent on (i) the image characteristics
determined by the image analysis prefilter and (ii) the correction
techniques available to the compensation prefilter and (iii) the
camera within which the invention operates. Thus it will be evident
to those skilled in the art that the knowledgebase will differ
significantly from one embodiment to another. It is also desirable
that said knowledgebase can be easily updated by a camera
manufacturer and, to some extent, modified by an end-user. Thus
various embodiments would store, or allow updating of the
knowledgebase from (i) a compact flash or other memory card; (ii) a
USB link to a personal computer; (iii) a network connection for a
networked/wireless camera and (iv) from a mobile phone network for
a camera which incorporates the functionality of a mobile phone. In
other alternative embodiments, where the camera is networked, the
knowledgebase may reside on a remote server and may respond to
requests from the camera for the resolving of a certain set of
correction interdependencies.
[0124] An example of image characteristics determined by the image
analysis prefilter is a person or type of person recognised by the
analyzer 130-9. Once a person or type of person has been recognized
using the face recognition analyzer, 130-9, it is preferred to
determine whether a customized redeye filter set is available and
if it has been loaded onto the camera. If this data is not
available, or if a person could not be recognized from a detected
face, a generic filter set will be applied to the detected face
region. If a person is recognized, the redeye filter will be
modified according to a customised profile loaded on the camera and
stored in the database 135-3. In general, this profile is based on
an analysis of previous images of the recognised person or type of
person and is designed to optimise both the detection and
correction of redeye defects for the individual or type of
person.
[0125] In particular, certain types of flash eye defects may
completely destroy the iris color of an eye. This can generally not
be restored by conventional image processing. However, if a simple
model of a person's eye is available from the image correction
knowledge base 135-4 which incorporates the appropriate geometric,
dimensional and color information, then much improved systems and
methods of redeye correction can be provided.
[0126] Now once the corrective image processing chain has been
determined it is applied to the image 412 and a number of sanity
checks are applied 412 to ensure that the image quality is not
degraded by the correction process 416. If these tests fail then it
may be that the determined interdependencies were marginal or that
an alternative image processing strategy is still available 418. If
this is so then the image processing chain is modified 420 and
corrective image processing is reapplied 412. This loop may
continue until all alternative image processing chains have been
exhausted. It is further remarked that the entire image processing
chain may not be applied each time. For example, if the differences
between image processing chains is a single filter then a temporary
copy of the input image to that filter is kept and said filter is
simply reapplied with different parameter settings. If, however
step 418 determines that all corrective measures have been tried it
will next move to step 422 which determines if filter chain
adaption is possible. Now returning to step 416, if the corrective
image processing is applied successfully then the image is passed
on to the redeye filter 90.
[0127] FIG. 4(b) describes an alternative embodiment of the
correction analyzer 135-2 which determines if filter chain adaption
is possible and then modifies the redeye filter appropriately.
Initially the image characteristics list is loaded 401 and for each
characteristic a set of filters which require adaption is
determined 452. This is achieved through referencing the external
database 135-3 and the comments and discussion provided in the
context of the image correction knowledgebase 135-4 apply equally
here.
[0128] Now once the filter lists for each image characteristic have
been determined the correction analyzer must determine which
filters overlap a plurality of image characteristics 454 and,
additionally determine if there are conflicts between the filter
adaptions required for each of the plurality of image
characteristics 456. If such conflicts exist the correction
analyzer must next decide if they can be resolved 460. To provide a
simple illustrative example we consider two image characteristics
which both require an adaption of the threshold of the main redness
filter in order to compensate for the measured non-optimallity of
each. If the first characteristic requires a lowering of the
redness threshold by, say, 10% and the second characteristic
requires a lowering of the same threshold by, say 15% then the
correction analyzer must next determine from the knowledgebase the
result of compensating for the first characteristic with a lowered
threshold of 15% rather than the initially requested 10%. Such an
adjustment will normal be an inclusive one and the correction
analyzer may determine that the conflict can be resolved by
adapting the threshold of the main redness filter to 15%. However
it might also determine that the additional 5% reduction in said
threshold will lead to an unacceptable increase in false positives
during the redeye filtering process and that this particular
conflict cannot be simply resolved.
[0129] If such filter conflicts cannot be simply resolved an
alternative is to determine if they are separable 466. If they are
separable that implies that two distinct redeye filter processes
can be run with different filter chains and the results of the two
detection processes can be merged prior to correcting the defects.
In the case of the example provided above this implies that one
detection process would be run to compensate for a first image
characteristic with a threshold of 10% and a second detection
process will be run for the second image characteristic with a
threshold of 15%. The results of the two detection processes will
then be combined in either an exclusive or an inclusive manner
depending on the separability determination obtained from the
subfilter database 135-3. In embodiments where a face recognition
module 130-9 is employed, a separate detection process may be
determined and selectively applied to the image for each known
person.
[0130] Returning to step 460, we see that if filter conflicts can
be resolved, the correction analyzer will prepare a single filter
chain parameter list 462 which will then be loaded 464 to the
filter chain adapter 99 of the redeye filter 90 illustrated in FIG.
3. Alternatively, if filter conflicts cannot be resolved, but are
determined to be separable 466 the correction analyzer prepares a
number of parameter lists 468 for the filter chain adapter which
are then loaded 464 as in the previous case. The redeye filter is
then applied 90.
[0131] However, if filter conflicts cannot be resolved and are not
separable the correction analyzer will then make a determination if
image processing compensation might be possible 422. If so then the
image processing compensation workflow of FIG. 4(a) may be
additionally employed 400. If it is determined that image
processing compensation is not possible then the correction
analyzer will disable the redeye filter for this image 220, and
provide a user indication to that effect 140 after which it will
pass control back to the main in-camera application 224.
[0132] FIG. 4(c) describes the workflow of the image analysis
prefilter 130 illustrated in FIG. 1(b). This performs an image
processing analysis of at least one image characteristic according
to at least one of a plurality of image processing techniques.
Preferably, the output of this analysis should be a simple measure
of goodness of the analyzed image characteristic. For the purposes
of an exemplary discussion we suppose that said measure is a
percentage of the optimum for said characteristic. Thus 100%
represents perfect quality for the measured image characteristic;
values above 95% represent negligible image
distortions/imperfections in said characteristic; values above 85%
represent noticeable, but easily correctable
distortions/imperfections and values above 60% represent major
distortions/imperfections which require major image processing to
correct the image characteristic. Values below 60% imply that the
image is too badly distorted to be correctable.
[0133] The first step in this workflow is to load or, if it is
already loaded in memory, to access the image to be analyzed. The
analysis prefilter next analyzes a first characteristic of said
image 482 and determines a measure of goodness. Now if said
characteristic is above a first threshold (95%) 486 then it is
marked as not requiring corrective measures 487 in the
characteristic list. If it is below said first threshold, but above
a second threshold (85%) 488 then it is marked as requiring
secondary corrective measures 489. If it is below said second
threshold, but above a third threshold (60%) 490 then it is marked
as requiring primary corrective measures 491 and if below said
third threshold 492 it is marked as uncorrectable 493. Now it is
remarked that for some embodiments of the present invention which
combine corrective image processing with filter chain adaption
there may be two distinct sets of thresholds, one relating to the
correctability using image processing techniques and the second
relating to the degree of compensation possible using filter chain
adaption. We further remark that for image compensation through
filter chain adaption that certain filters may advantageously scale
their input parameters directly according to the measure of
goodness of certain image characteristics. As an illustrative
example consider the redness threshold of the main color filter
which, over certain ranges of values, may be scaled directly
according to a measure of excessive "redness" in the color balance
of a non-optimally acquired image. Thus, the image characteristic
list may additionally include the raw measure of goodness of each
image characteristic. In an alternative embodiment only the raw
measure of goodness will be exported from the image analysis
prefilter 130 and the threshold based determining of FIG. 4(c) will
be performed within the correction analyzer 135-2 in which case
threshold values may be determined from the image correction
knowledgebase 135-4.
[0134] Returning to 493 we note that images of such poor quality
may require a second image acquisition process to be initiated and
so it is implicit in 493 that for certain embodiments of the
present invention it may be desirable that an alarm/interrupt
indication is sent to the main camera application.
[0135] Now the main loop continues by determining if the currently
analyzed characteristic is the last image characteristic to be
analyzed 496. If not it returns to analyzing the next image
characteristic 482. If it is the last characteristic it then passes
the image characteristics list to the image compensation prefilter
494 and returns control to the main camera application 224. It
should be remarked that in certain embodiments that a plurality of
image characteristics may be grouped together and analyzed
concurrently, rather than on a one-by-one basis. This may be
preferable if several image characteristics have significant
overlap in the image processing steps required to evaluate them. It
may also be preferable where a hardware co-processor or DSP unit is
available as part of the camera hardware and it is desired to batch
run or parallelize the computing of image characteristics on such
hardware subsystems.
[0136] A third principle embodiment of the present invention has
already been briefly described. This is the use of a global
pixel-level transformation of the image within the redeye filter
itself and relies on the corrective image processing, as determined
by the correction analyzer 135-2, being implementable as a global
pixel-level transformation of the image. Those skilled in the art
will realize that such a requirement implies that certain of the
image analyzer elements which comprise the image analysis prefilter
130 are not relevant to this embodiment. For example dust analysis,
object/region analysis, noise analysis and certain forms of image
blur cannot be corrected by such transformations. However many
other image characteristics are susceptible to such
transformations. Further, we remark that this alternative
embodiment may be combined with the other two principle embodiments
of the invention to compliment each other.
[0137] In FIG. 5(a) we illustrate an exemplary embodiment of the
red pixel locating and red region segmenting workflow which occurs
within the redeye filter as steps 92-1 and 92-2. This workflow has
been modified to incorporate a global pixel-level transformation
92-0 of the image as an integral element of the color determining
and region grouping steps of the redeye filter. It is implicit in
this embodiment that the correction analyzer has determined that a
global pixel level transformation can achieve the required image
compensation. The image to be processed by the redeye filter is
first loaded 502 and the labeling LUT for the region grouping
process in initialized 504. Next the current pixel and pixel
neighbourhoods are initialized 506.
[0138] FIG. 5(b) shows a diagrammatic representation of a 4-pixel
neighborhood 562, shaded light gray in the figure and containing
the three upper pixels and the pixel to the left of the current
pixel 560, shaded dark gray in the figure. This 4-pixel
neighborhood is used in the labeling algorithm of this exemplary
embodiment. A look-up table, LUT, is defined to hold correspondence
labels.
[0139] Returning to step 506 we see that after initialization is
completed the next step for the workflow of FIG. 5(a) is to begin a
recursive iteration through all the pixels of an image in a
raster-scan from top-left to bottom-right. The first operation on
each pixel is to apply the global pixel transformation 508. It is
assumed that the loaded image is an RGB bitmap and the global pixel
transformation is of the form: P(R,G,B).fwdarw.P(R',G',B'), where
the red, green and blue values of the current pixel, P(R,G,B) are
mapped to a shifted set of color space values, P(R',G',B'). There
are a number of advantages in performing this corrective
transformation at the same time as the color determining and pixel
grouping. In particular it is easier to optimize the computational
performance of the algorithm which is important for in-camera
implementations. Following step 508 the workflow next determines if
the current pixel satisfies membership criteria for a candidate
redeye region 510. Essentially this implies that the current pixel
has color properties which are compatible with an eye defect; this
does not necessarily imply that the pixel is red as a range of
other colors can be associated with flash eye defects. If the
current pixel satisfies membership criteria for a segment 510,
i.e., if it is sufficiently "red", then the algorithm checks for
other "red" pixels in the 4-pixel neighborhood 512. If there are no
other "red" pixels, then the current pixel is assigned membership
of the current label 530. The LUT is then updated 532 and the
current label value is incremented 534. If there are other "red"
pixels in the 4-pixel neighborhood then the current pixel is given
membership in the segment with the lowest label value 514 and the
LUT is updated accordingly 516. After the current pixel has been
labeled as part of a "red" segment 512 or 530, or has been
categorized as "non-red" during step 510, a test is then performed
to determine if it is the last pixel in the image 518. If the
current pixel is the last pixel in the image then a final update of
the LUT is performed 540. Otherwise the next image pixel is
obtained by incrementing the current pixel pointer 520 and
returning to step 508 and is processed in the same manner. Once the
final image pixel is processed and the final LUT completed 540, all
of the pixels with segment membership are sorted into a
labeled-segment table of potential red-eye segments 542.
[0140] With regard to the exemplary details of corrective image
processing 135-1 which may be employed in the present invention we
remark that a broad range of techniques exist for automatic or
semi-automatic image correction and enhancement. For ease of
discussion we can group these into 6 main subcategories as follows:
[0106] (i) Contrast Normalization and Image Sharpening. [0107] (ii)
Image Color Adjustment and Tone Reproduction Scaling. [0108] (iii)
Exposure Adjustment and Digital Fill Flash [0109] (iv) Brightness
Adjustment with Color Space Matching; Image Auto-Gamma
determination with Image Enhancement. [0110] (v) In-Camera Image
Enhancement [0111] (vi) Face Based Image Enhancement
[0141] All Categories may be Global Correction or Local Region
Based.
(i) Contrast Normalization and Image Sharpening
[0142] U.S. Pat. No. 6,421,468 to Ratnakar et al. disclose
sharpening an image by transforming the image representation into a
frequency-domain representation and by selectively applying scaling
factors to certain frequency domain characteristics of an image.
The modified frequency domain representation is then
back-transformed into the spatial domain and provides a sharpened
version of the original image. U.S. Pat. No. 6,393,148 to Bhaskar
discloses automatic contrast enhancement of an image by increasing
the dynamic range of the tone levels within an image without
causing distortion or shifts to the color map of said image.
(ii) Color Adjustment and Tone Scaling of a Digital Image
[0143] US patent application 2002/0105662 to Patton et al.
discloses modifying a portion of an image in accordance with
colormetric parameters. More particularly it discloses the steps of
(i) identifying a region representing skin tone in an image; (ii)
displaying a plurality of renderings for said skin tone; (iii)
allowing a user to select one of said renderings and (iv) modifying
the skin tone regions in the images in accordance with the
rendering of said skin tone selected by the user. U.S. Pat. No.
6,438,264 to Gallagher et al. discloses compensating image color
when adjusting the contrast of a digital color image including the
steps of (i) receiving a tone scale function; (ii) calculating a
local slope of the tone scale function for each pixel of the
digital image; (iii) calculating a color saturation signal from the
digital color image and (iv) adjusting the color saturation signal
for each pixel of the color image based on the local tone scale
slope. The image enhancements of Gallagher et al. are applied to
the entire image and are based on a global tone scale function.
Thus this technique may be implemented as a global pixel-level
color space transformation. U.S. Pat. No. 6,249,315 to Holm teaches
how a spatially blurred and sub-sampled version of an original
image can be used to obtain statistical characteristics of a scene
or original image. This information is combined with the tone
reproduction curves and other characteristics of an output device
or media to provide an enhancement strategy for optimized output of
a digital image. All of this processing can be performed
automatically, although the Holm also allows for simple, intuitive
manual adjustment by a user.
(iii) Digital Fill Flash and Post-Acquisition Exposure
Adjustment
[0144] US patent application 2003/0052991 to Stavely et al.
discloses simulating fill flash in digital photography. In Stavely
a digital camera shoots a series of photographs of a scene at
various focal distances. These pictures are subsequently analyzed
to determine the distances to different objects in the scene. Then
regions of these pictures have their brightness selectively
adjusted based on the aforementioned distance calculations and are
then combined to form a single, photographic image. US patent
application 2001/0031142 to Whiteside is concerned with a scene
recognition method and a system using brightness and ranging
mapping. It uses auto-ranging and brightness measurements to adjust
image exposure to ensure that both background and foreground
objects are correctly illuminated in a digital image. Much of the
earlier prior art is focused on the application of corrections and
enhancement of the entire image, rather than on selected regions of
an image and thus discuss the correction of image exposure and tone
scale as opposed to fill flash. Example patents include U.S. Pat.
No. 6,473,199 to Gilman et al. which describes a method for
correcting for exposure in a digital image and includes providing a
plurality of exposure and tone scale correcting nonlinear
transforms and selecting the appropriate nonlinear transform from
the plurality of nonlinear transforms and transforming the digital
image to produce a new digital image which is corrected for
exposure and tone scale. U.S. Pat. No. 5,991,456 to Rahman et al.
describes a method of improving a digital image. The image is
initially represented by digital data indexed to represent
positions on a display. The digital data is indicative of an
intensity value Ii (x,y) for each position (x,y) in each i-th
spectral band. The intensity value for each position in each i-th
spectral band is adjusted to generate an adjusted intensity value
for each position in each i-th spectral band. Each surround
function Fn (x,y) is uniquely scaled to improve an aspect of the
digital image, e.g., dynamic range compression, color constancy,
and lightness rendition. For color images, a novel color
restoration step is added to give the image true-to-life color that
closely matches human observation. However these do not teach the
concept of regional analysis and regional adjustment of image
intensity or exposure levels. U.S. Pat. No. 5,818,975 to Goodwin et
al. teaches area selective exposure adjustment. Goodwin describes
how a digital image can have the dynamic range of its scene
brightness reduced to suit the available dynamic brightness range
of an output device by separating the scene into two regions--one
with a high brightness range and one with a low brightness range. A
brightness transform is derived for both regions to reduce the
brightness of the first region and to boost the brightness of the
second region, recombining both regions to reform an enhanced
version of the original image for the output device. This technique
is analogous to an early implementation of digital fill flash.
Another example is U.S. Pat. No. 5,724,456 to Boyack et al. which
teaches brightness adjustment of images using digital scene
analysis. Boyack partitions the image into blocks and larger groups
of blocks, known as sectors. It then determines an average
luminance block value. A difference is determined between the max
and min block values for each sector. If this difference exceeds a
pre-determined threshold the sector is marked active. A histogram
of weighted counts of active sectors against average luminance
sector values is plotted and the histogram is shifted to using a
pre-determined criteria so that the average luminance sector values
of interest will fall within a destination window corresponding to
the tonal reproduction capability of a destination application or
output device.
(iv) Brightness Adjustment; Color Space Matching; Auto-Gamma
[0145] Another area of image enhancement in the prior art relates
to brightness adjustment and color matching between color spaces.
For example U.S. Pat. No. 6,459,436 to Kumada et al. describes
transforming image date from device dependent color spaces to
device-independent Lab color spaces and back again. Image data is
initially captured in a color space representation which is
dependent on the input device. This is subsequently converted into
a device independent color space. Gamut mapping (hue restoration)
is performed in the device independent color space and the image
data may then be mapped back to a second device-dependent color
space. U.S. Pat. No. 6,268,939 to Klassen et al. is also concerned
correcting luminance and chrominance data in digital color images.
More specifically Klassen is concerned with optimizing the
transformations between device dependent and device independent
color spaces by applying subsampling of the luminance and
chrominance data. Another patent in this category is U.S. Pat. No.
6,192,149 to Eschback et al. which discloses improving the quality
of a printed image by automatically determining the image gamma and
then adjusting the gamma of a printer to correspond to that of the
image. Although Eschback is concerned with enhancing the printed
quality of a digital image and not the digital image itself, if
does teach a means for automatically determining the gamma of a
digital image. This information could be used to directly adjust
image gamma, or used as a basis for applying other enhancements to
the original digital image. U.S. Pat. No. 6,101,271 to Yamashita et
al. discloses implementing a gradation correction to an RGB image
signal which allows image brightness to be adjusted without
affecting the image hue and saturation.
(v) In-Camera Image Enhancement
[0146] U.S. Pat. No. 6,516,154 to Parulski et al. discloses
suggesting improvements to a digital image after it has been
captured by a camera. The user may crop, re-size or adjust color
balance before saving a picture; alternatively the user may choose
to re-take a picture using different settings on the camera. The
suggestion of improvements is made by the camera user-interface.
However Parulski does not teach the use of image analysis and
corrective image processing to automatically initiate in-camera
corrective actions upon an acquired digital image.
(vii) Face-Based Image Enhancement
[0147] In US patent application 20020172419, Lin et al., discloses
automatically improving the appearance of faces in images based on
automatically detecting such images in the digital image. Lin
describes modification of lightness contrast and color levels of
the image to produce better results.
[0148] Additional methods of face-based image enhancement are
described in co-pending U.S. application Ser. No. 11/024,046, which
is hereby incorporated by reference.
Further Embodiments
[0149] Any of the embodiments described herein may be combined in
whole or in part with any of the features described in the
following in alternative embodiments.
[0150] There is provided in certain embodiments a method and
apparatus for red-eye detection in an acquired digital image as
claimed in the appended claims.
[0151] Certain embodiments compensate for sub-optimally acquired
images where degradations in the acquired image may affect the
correct operation of redeye detection, prior to or in conjunction
with applying the detection and correction stage.
[0152] Certain embodiments improve the overall success rate and
reduces the false positive rate of red eye detection and reduction
by compensating for non-optimally acquired images by performing
image analysis on the acquired image and determining and applying
corrective image processing based on said image analysis prior to
or in conjunction with applying one or many redeye detection
filters to the acquired image. Such corrections or enhancements may
include applying global or local color space conversion, exposure
compensation, noise reduction, sharpening, blurring or tone
reproduction transformations.
[0153] In certain embodiments, image analysis is performed on a
sub-sampled copy of the main acquired image where possible,
enhancing the performance of this invention inside devices with
limited computational capability such as hand held devices and in
particular digital cameras or printers.
[0154] In certain embodiments, the pre-filtering process is
optimized by applying when possible, as determined from the image
analysis, the image transformations at the pixel level during the
redeye detection process thus compensating for non-optimally
acquired images without requiring that corrective image processing
be applied to the full resolution image.
[0155] In certain embodiments, the redeye filter chain is
configured for optimal performance based on image analysis of an
acquired image to enhance the execution red eye detection and
reduction process. Such configuration takes place in the form of
variable parameters for the algorithm and variable ordering and
selection of sub-filters in the process.
[0156] Certain embodiments operate uniformly on both pixels which
are members of a defect and its bounding region thus avoiding the
need to determine individually if pixels in the neighborhood of
said defect are members of the defect and to subsequently apply
correcting algorithms to such pixels on an individual basis.
[0157] Using certain embodiments, variables that could
significantly effect the success of the red-eye detection algorithm
such as noise, color shifts, incorrect exposure, blur, over
sharpening etc, may be pre-eliminated before performing the
detection process, thus improving the success rate.
[0158] Alternatively or in addition these variables may be
pre-accounted for by changing the parameters for the detection
process, thus improving the performance and the success rate.
[0159] An advantage provided herein is that by bringing images into
a known and better defined image quality, the criteria for
detection can be tightened and narrowed down, thus providing higher
accuracy both in the positive detection and reduction in the false
detection.
[0160] A further advantage provided herein is that by accounting
for the reasons for suboptimal image quality the parameters for the
detection and correction algorithm may be modified, thus providing
higher accuracy both in the positive detection and reduction in the
false detection without the need to modify the image.
[0161] An additional advantage provided herein is that
misclassification of pixels and regions belonging to defect areas
is reduced if not altogether avoided, which means a reduction of
undetected correct positives.
[0162] An additional advantage provided herein is that color
misclassification of pixels and regions belonging to non-defect
areas is reduced if not avoided, which means a reduction of false
positives.
[0163] A further advantage provided herein is that certain
embodiments can be implemented to run sufficiently fast and
accurately to allow individual images in a batch to be analyzed and
corrected in real-time prior to printing.
[0164] Yet a further advantage of preferred embodiments of the
present invention is that they have a sufficiently low requirement
for computing power and memory resources to allow it to be
implemented inside digital cameras as part of the post-acquisition
processing step.
[0165] Yet a further advantage provided herein is that certain
embodiments have a sufficiently low requirement for computing power
and memory resources to allow them to be implemented as a computer
program on a hand-held personal digital assistant (PDA), mobile
phone or other digital appliance suitable for picture display
[0166] A further advantage provided herein is that certain
embodiments are not limited in their detection of red-eye defects
by requirements for clearly defined skin regions matching a human
face.
[0167] A further advantage provided herein is the ability to
concatenate image quality transformations and red eye detection to
improve overall performance.
[0168] The present invention is not limited to the embodiments
described above herein, which may be amended or modified without
departing from the scope of the present invention as set forth in
the appended claims, and structural and functional equivalents
thereof.
[0169] In methods that may be performed according to preferred
embodiments herein and at may have been described above and/or
claimed below, the operations have been described in selected
typographical sequences. However, the sequences have been selected
and so ordered for typographical convenience and are not intended
to imply any particular order for performing the operations.
[0170] In addition, all references cited above herein, in addition
to the background and summary of the invention sections, are hereby
incorporated by reference into the detailed description of the
preferred embodiments as disclosing alternative embodiments and
components.
* * * * *
References