U.S. patent application number 11/403764 was filed with the patent office on 2007-08-02 for bi-illuminant dichromatic reflection model for image manipulation.
This patent application is currently assigned to Tandent Vision Science, Inc.. Invention is credited to Richard Mark Friedhoff, Bruce Allen Maxwell, Casey Arthur Smith.
Application Number | 20070176941 11/403764 |
Document ID | / |
Family ID | 38327925 |
Filed Date | 2007-08-02 |
United States Patent
Application |
20070176941 |
Kind Code |
A1 |
Maxwell; Bruce Allen ; et
al. |
August 2, 2007 |
Bi-illuminant dichromatic reflection model for image
manipulation
Abstract
In a first exemplary embodiment of the present invention, an
automated, computerized method for manipulating an image comprises
the steps of manipulating the image to provide an intensity
adjusted image, deriving a bi-illuminant dichromatic reflection
model representation of the image and utilizing the bi-illuminant
dichromatic reflection model representation to manipulate the
intensity adjusted image to generate a color correct intensity
adjusted image. In a preferred embodiment of the present invention,
the step of manipulating the image to provide an intensity adjusted
image is carried out by executing a gamma correction method.
Inventors: |
Maxwell; Bruce Allen;
(Springfield, PA) ; Friedhoff; Richard Mark; (New
York, NY) ; Smith; Casey Arthur; (Ithaca,
NY) |
Correspondence
Address: |
DAVIDSON, DAVIDSON & KAPPEL, LLC
485 SEVENTH AVENUE, 14TH FLOOR
NEW YORK
NY
10018
US
|
Assignee: |
Tandent Vision Science,
Inc.
San Francisco
CA
|
Family ID: |
38327925 |
Appl. No.: |
11/403764 |
Filed: |
April 13, 2006 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
11341751 |
Jan 27, 2006 |
|
|
|
11403764 |
Apr 13, 2006 |
|
|
|
Current U.S.
Class: |
345/589 |
Current CPC
Class: |
G06T 5/007 20130101;
H04N 1/62 20130101; G06T 2207/20012 20130101; G06T 5/008 20130101;
G06T 2207/10024 20130101; G06T 2207/20092 20130101; G06T 7/90
20170101; G06T 2207/20104 20130101; G06T 2207/20192 20130101; G06T
15/506 20130101 |
Class at
Publication: |
345/589 |
International
Class: |
G09G 5/02 20060101
G09G005/02 |
Claims
1. An automated, computerized method for manipulating an image,
comprising the steps of: manipulating the image to provide an
intensity adjusted image; deriving a bi-illuminant dichromatic
reflection model representation of the image; and utilizing the
bi-illuminant dichromatic reflection model representation to
manipulate the intensity adjusted image to generate a color correct
intensity adjusted image.
2. The method of claim 1 wherein the step of manipulating the image
to provide an intensity adjusted image is carried out by executing
a gamma correction method.
3. The method of claim 2 wherein the bi-illuminant dichromatic
reflection model representation comprises a spectral ratio for the
image.
4. The method of claim 3 wherein the image comprises pixels having
an original intensity adjusted by the gamma correction method.
5. The method of claim 4 wherein the step of utilizing the
bi-illuminant dichromatic reflection model representation to
manipulate the intensity adjusted image to generate a color correct
intensity adjusted image includes the step of calculating color
correct intensities for each pixel, as a function of ratios, R1
representing a minimum degree of brightening, R2 representing a
maximum degree of brightening.
6. The method of claim 3 wherein the step of utilizing the
bi-illuminant dichromatic reflection model representation to
manipulate the intensity adjusted image to generate a color correct
intensity adjusted image is carried out by utilizing the spectral
ratio to calculate color correct color values for fully lit and
dark versions of pixels of the image.
7. A computer system which comprises: a CPU; and a memory storing
an image file; the CPU arranged and configured to execute a routine
to manipulate the image to provide an intensity adjusted image,
derive a bi-illuminant dichromatic reflection model representation
of the image, and utilize the bi-illuminant dichromatic reflection
model representation to manipulate the intensity adjusted image to
generate a color correct intensity adjusted image.
8. The computer system of claim 7 wherein the CPU executes a
routine to manipulate the image to provide an intensity adjusted
image by executing a gamma correction method.
Description
[0001] This application is a continuation-in-part of application
Ser. No. 11/341,751, filed Jan. 27, 2006, entitled: "Bi-illuminant
Dichromatic Reflection Model For Image Manipulation," which is
hereby incorporated by reference.
BACKGROUND OF THE INVENTION
[0002] Many significant and commercially important uses of modem
computer technology relate to images. These include image
processing, image analysis and computer vision applications. A
challenge in the utilization of computers to accurately and
correctly perform operations relating to images is the development
of algorithms that truly reflect and represent physical phenomena
occurring in the visual world. For example, the ability of a
computer to correctly and accurately distinguish between a shadow
and a material object edge within an image has been a persistent
challenge to scientists. Edge detection is a fundamental task in
image processing because without accurate and correct detection of
the edges of physical objects, no other processing of the image is
possible. If a cast shadow is indistinguishable from the object
casting the shadow, it would not be possible for the computer to
recognize the object.
[0003] An early and conventional approach to object edge detection
involves an analysis of brightness boundaries in an image. In the
analysis it is assumed that a boundary caused by a material object
will be sharp, while a boundary caused by a shadow will be soft or
gradual due to the penumbra effect of shadows. While this approach
can be implemented by algorithms that can be accurately executed by
a computer, the results will often be incorrect. In the real world
there are many instances wherein shadows form sharp boundaries, and
conversely, material object edges form soft boundaries. Thus, when
utilizing conventional techniques for shadow and edge recognition,
there are significant possibilities for false positives and false
negatives for shadow recognition. That is, for example, a material
edge that imitates a shadow and is thus identified incorrectly by a
computer as a shadow or a sharp shadow boundary that is incorrectly
interpreted as an object boundary.
[0004] Once shadows and object edges are identified, a typical
computerized operation is manipulation of the image to, for
example, remove shadows from the image. Most scenes depicted in an
image have a dominant illuminant, defined as a direct or incident
illuminant. The incident illuminant causes shadows. The component
of radiance onto a surface in the scene that is not from the
incident illuminant, is referred to as an indirect or ambient
illuminant. It is the ambient illuminant that is present within a
shadow. While much of the energy of the ambient illuminant may come
from the incident illuminant, it has generally interacted with the
environment.
[0005] Typical solutions for manipulating images focus on the
incident illuminant. Models have been developed for computerized
image pixel manipulation based upon the assumption that the ambient
illumination has the same spectral characteristics as the incident
illumination or is non-existent. One such known solution is the
dichromatic reflection model, which describes the variations in
appearance caused by the combination of body and surface reflection
on a material. Body reflection is what is normally considered the
color of the material. The surface reflection is referred to as a
highlight or specularity of the material reflecting the illuminant.
The known dichromatic reflection model assumes a single incident
illuminant and does not account for a non-zero ambient illuminant.
Thus, results of image manipulation based upon the dichromatic
reflection model are often not color correct.
[0006] Other useful solutions include color spaces such as hue and
chromaticity, and other normalized color spaces that attempt to
factor out the effect of changing the intensity of the incident
illuminant on the intensity of the reflection from a surface.
However, these color models have met with limited success in
solving practical vision problems. Consequently, there is a growing
realization that simple models of illumination do not correctly
reflect the visual world, and thus, do not provide color correct
manipulations. Recent research has focused upon empirical models of
color change over a single material. This approach is not
restricted by any prior assumptions about the behavior of
illumination color or reflectance.
[0007] There is also significant amount of research attempting to
determine the complete illumination environment. These methods are
based upon multiple images of the same scene and/or knowledge of
the scene geometry. In one research project, the existence and
importance of complex indirect illumination is acknowledged.
However, the method requires both a known geometry of a scene and
two images. While these research efforts approach a solution that
can extract and represent illumination environments of significant
complexity, they cannot be used in environments such as, for
example, consumer photography, or with existing photos or in any
situation where either taking multiple images of a scene from
different points of view or inserting objects into a scene are not
readily possible or are unreasonable.
SUMMARY OF THE INVENTION
[0008] The present invention provides a bi-illuminant dichromatic
reflection model (BIDR model) as a representation of an image to
facilitate color correct image manipulation.
[0009] In a first exemplary embodiment of the present invention, an
automated, computerized method for manipulating an image comprises
the steps of manipulating the image to provide an intensity
adjusted image, deriving a bi-illuminant dichromatic reflection
model representation of the image and utilizing the bi-illuminant
dichromatic reflection model representation to manipulate the
intensity adjusted image to generate a color correct intensity
adjusted image. In a preferred embodiment of the present invention,
the step of manipulating the image to provide an intensity adjusted
image is carried out by executing a gamma correction method.
[0010] In a second exemplary embodiment of the present invention, a
computer system comprises a CPU and a memory storing an image file.
According to a feature of the present invention, the CPU is
arranged and configured to execute a routine to manipulate the
image to provide an intensity adjusted image, derive a
bi-illuminant dichromatic reflection model representation of the
image, and utilize the bi-illuminant dichromatic reflection model
representation to manipulate the intensity adjusted image to
generate a color correct intensity adjusted image. In a preferred
embodiment of the present invention, the CPU executes a routine to
manipulate the image to provide an intensity adjusted image by
executing a gamma correction method.
[0011] In accordance with yet further embodiments of the present
invention, computer systems are provided, which include one or more
computers configured (e.g., programmed) to perform the methods
described above. In accordance with other embodiments of the
present invention, computer readable media are provided which have
stored thereon computer executable process steps operable to
control a computer(s) to implement the embodiments described above.
The automated, computerized methods can be performed by a digital
computer, analog computer, optical sensor, state machine, sequencer
or any device or apparatus that can be designed or programed to
carry out the steps of the methods of the present invention.
BRIEF DESCRIPTION OF THE DRAWINGS
[0012] FIG. 1 is a block diagram of a computer system arranged and
configured to perform operations related to images.
[0013] FIG. 2 shows an n X m pixel array image file for an image
stored in the computer system of FIG. 1.
[0014] FIG. 3 is a flow chart for modifying a shadow in an image
with a manual input, according to a feature of the present
invention.
[0015] FIG. 4 is a graph in RGB color space showing colors for a
material, from a fully shaded color value to a fully lit color
value, as determined by execution of a simplified bi-illuminant
dichromatic reflection model according to a feature of the present
invention.
[0016] FIGS. 5a & b are graphs in RGB color space showing
colors for a material, from a fully shaded color value to a fully
lit color value, with error bounds forming a cylinder, as
determined by execution of the bi-illuminant dichromatic reflection
model according to a feature of the present invention.
[0017] FIG. 6a is a flow chart for a color correct brightness
adjustment to an intensity adjusted image, according to a feature
of the present invention.
[0018] FIG. 6b is a graph in RGB space showing an intensity
adjusted pixel value for the intensity adjusted image of FIG.
6a.
[0019] FIG. 6c is a flow chart for estimating a gamma (.gamma.)
value for the pixel of FIG. 6b using a dynamic range compression
algorithm.
[0020] FIG. 7 is a flow chart for removing direct or incident
illumination from an image area, according to a feature of the
present invention.
[0021] FIG. 8 is a flow chart for modifying apparent ambient
illumination of an image area, according to a feature of the
present invention.
[0022] FIG. 9 is a flow chart for modifying apparent direct or
incident illumination of an image area, according to a feature of
the present invention.
[0023] FIG. 10 is a flow chart for modifying apparent level of
direct or incident illumination of an image area, according to a
feature of the present invention.
[0024] FIG. 11 is a graph in RGB space showing a color correct
gamma correction, according to a feature of the present
invention.
[0025] FIG. 12 is a flow chart for a color correct gamma
correction, as shown in FIG. 11.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0026] Referring now to the drawings, and initially to FIG. 1,
there is shown a block diagram of a computer system 10 arranged and
configured to perform operations related to images. A CPU 12 is
coupled to a device such as, for example, a digital camera 14 via,
for example, a USB port. The digital camera 14 operates to download
images stored locally on the camera 14, to the CPU 12. The CPU 12
stores the downloaded images in a memory 16 as image files 18. The
image files 18 can be accessed by the CPU 12 for display on a
monitor 20, or for print out on a printer 22.
[0027] Alternatively, the CPU can be implemented as a
microprocessor embedded in a device such as, for example, the
digital camera 14 or a robot. The CPU can also be equipped with a
real time operating system for real time operations relating to
images, for example, in connection with a robotic operation or an
interactive operation with a user.
[0028] As shown in FIG. 2, each image file 18 comprises an n X m
pixel array. Each pixel, p, is a picture element corresponding to a
discrete portion of the overall image. All of the pixels together
define the image represented by the image file 18. Each pixel
comprises a digital value corresponding to a set of color bands,
for example, red, green and blue color components (RGB) of the
picture element. The present invention is applicable to any
multi-band image, where each band corresponds to a piece of the
electromagnetic spectrum. The present invention can also be
utilized in connection with a grayscale image (a single band). The
pixel array includes m columns of n rows each, starting with the
pixel p(1,1) and ending with the pixel p(n, m). When displaying or
printing an image, the CPU 12 retrieves the corresponding image
file 18 from the memory 16, and operates the monitor 20 or printer
22, as the case may be, as a function of the digital values of the
pixels in the image file 18, as is generally known.
[0029] In an image operation, the CPU 12 operates to analyze the
RGB values of the pixels of a stored image file 18 to achieve
various objectives, such as, for example, manipulation of the image
to modify a shadow or to make a color correct brightness
adjustment. A fundamental observation underlying a basic discovery
of the present invention, is that an image comprises two
components, material and illumination. All changes in an image are
caused by one or the other of these components.
[0030] In a first exemplary embodiment of the present invention,
shadowed and fully lit regions of an image are manually selected by
a user. What is visible to the human eye upon display on the
monitor 20 of a stored image file 18 by the CPU 12, is the pixel
color values caused by the interaction between specular and body
reflection properties of material objects in, for example, a scene
photographed by the digital camera 14 and illumination flux present
at the time the photograph was taken. As noted above, the
illumination flux comprises an ambient illuminant and an incident
illuminant. The incident illuminant is light that causes a shadow
and is found outside a shadow perimeter. The ambient illuminant is
light present on both the bright and dark sides of a shadow, but is
more perceptible within the dark region.
[0031] FIG. 3 is a flow chart for modifying a shadow in an image
with a manual input, according to a feature of the present
invention. In two input steps the user selects a point or region on
a fully lit (Bright) material of an image 18 selected by the user
(step 100) and a point or region on a fully shadowed (Dark) part of
the same material in the same image 18 (step 102). These steps 100,
102 can be implemented by an interactive clicking by the user on
the monitor 20 operating to display the subject image file 18. A
user can select such regions because human eye physiology is
capable of distinguishing between shadows and actual physical
objects.
[0032] In step 104, the CPU 12 operates on the user input
selections to calculate a spectral ratio, S=Dark/(Bright-Dark),
where Dark, for example, is a pixel in the fully shadowed material
of the region selected by the user, and Bright is a pixel in the
fully lit portion of that material selected by the user. A
normalized version of the spectral ratio can be used in the methods
according to the present invention. According to a feature of the
present invention, the spectral ratio is used to manipulate pixel
values for a color correct adjustment. Based upon the fundamental
observation of the present invention that an image comprises two
components, material and illumination, the computer system 10 can
be operated to differentiate between material aspects and
illumination flux through recognition of a spectral shift caused by
the interplay between the incident or direct illuminant and the
ambient illuminant. When one of material and illumination is known
in an image, the other can be readily deduced. The spectrum for the
incident illuminant and the ambient illuminant can be different
from one another. A spectral shift caused by a shadow, i.e., a
decrease of the intensity of the incident illuminant, will be
substantially invariant over different materials present in a scene
depicted in an image.
[0033] Pursuant to a feature of the present invention, this
spectral shift information is detected by determination of an
illuminant ratio, or a characteristic spectral ratio formed by the
interplay of the incident illuminant and the ambient illuminant. A
spectral ratio is a ratio based upon a difference in color or
intensities between two areas of a scene depicted in an image,
which may be caused by different materials, an illumination change
or both.
[0034] An automated, computerized method for determining a
characteristic spectral or illuminant ratio due to illumination
flux, for an image, is disclosed in co-pending application Ser. No.
11/341,742, filed on Jan. 27, 2006, entitled: "Method and System
For Identifying Illumination Flux In An Image," which is hereby
incorporated by reference. As disclosed in the co-pending
application Ser. No. 11/341,742, to improve the accuracy and
correctness of the characteristic ratio for an image, the spectral
ratio information for illumination boundaries is determined on a
local level, that is, an illuminant ratio is determined for each of
several preselected local areas of a scene depicted in an image. An
analysis of a boundary is then executed utilizing the spectral
ratio for the specific location of the boundary within the image.
The determination of locally relevant spectral ratios accommodates
complexities that may be encountered in a real world image, for
example, the interplay of several different sources of light in a
room, inter-reflections, and so on.
[0035] In order to facilitate an accurate and correct manipulation
of pixel values, to modify, for example, the pixel from a shaded
color (Dark) to a correct fully lit color (Bright), the present
invention recognizes the characteristic spectral ratio (illuminant
ratio) as an analytical representation of image properties and
characteristics pursuant to a bi-illuminant dichromatic reflection
model. The bi-illuminant dichromatic reflection model (BIDR)
combines terms relevant to the ambient illuminant with the
dichromatic reflection model described above. The two illuminants
of the BIDR model correspond to the incident illuminant and the
ambient illuminant. The BIDR model can be stated as follow:
I.sub.(x, y, z, .theta., .phi.,
.lamda.)=[m.sub.b(.theta..sub.i)c.sub.b(.lamda.)+m.sub.s(.theta..sub.i,
.phi..sub.i, .theta..sub.e,
.phi..sub.e)c.sub.s(.lamda.)]l.sub.d(.lamda.)+M.sub.a(.lamda.)c.sub.b(.la-
mda.)+M.sub.as(.theta..sub.e, .phi..sub.e,
.lamda.)c.sub.s(.lamda.), where: I.sub.(x, y, z, .theta., .phi.,
.lamda.) is the radiance of a surface point at (x, y, z) in the
direction .theta., .phi., for the wavelength .lamda., [0036]
m.sub.b(.theta..sub.i) is the spectrum independent body reflectance
based on geometric factors, [0037] c.sub.b(.lamda.) is the geometry
independent body reflectance of a surface for the wavelength
.lamda., [0038] m.sub.s(.theta..sub.i, .phi..sub.i, .theta..sub.e,
.phi..sub.e) is the spectrum independent surface reflectance based
on geometric factors, [0039] c.sub.s(.lamda.) is the geometry
independent surface reflectance of a surface for the wavelength
.lamda., [0040] l.sub.d(.lamda.) is the incident illuminant for the
wavelength .lamda.. [0041] .theta..sub.i is the incident tilt angle
of the illuminant onto a surface relative to the surface normal.
[0042] .phi..sub.i is the incident pan angle: the angle between the
incident illuminant and a fixed axis on the tangent plane of a
surface. [0043] .theta..sub.e is the exitent tilt angle: the angle
of an outgoing ray relative to the surface normal. [0044]
.phi..sub.e is the exitent pan angle: the angle between an outgoing
ray and a fixed axis on the tangent plane of a surface. All of the
above terms comprise the dichromatic reflection model. The
remaining terms of the BIDR model relate to the ambient illuminant,
where: [0045] M.sub.a(.lamda.) is the integral of the ambient
illuminant and geometric body reflectance over a hemisphere,
excluding the incident illuminant, and [0046]
M.sub.as(.theta..sub.e, .phi..sub.e, .lamda.) is the integral of
the ambient illuminant and geometric surface reflectance over the
hemisphere, excluding the incident illuminant.
[0047] In the dichromatic reflection model, the radiance, I.sub.(x,
y, z, .theta., .phi., .lamda.), for a surface at a point viewed
from the orientation by .theta., .phi. is expressed as the sum of
two bipartite terms multiplied by the illumination intensity and
color l.sub.d(.lamda.) to reflect the effect of the incident
illuminant. The first reflectance term explains body reflection.
m.sub.b(.theta..sub.i) is the amount of incident illuminant energy
reflected by the body reflection, referred to as diffuse
reflection, as a function of the geometric relationship of the
light source and the surface, expressed by the incident angle
.theta..sub.i. c.sub.b(.lamda.) is the fraction of the incident
illuminant energy reflected by the body reflection for each
wavelength .lamda. and describes what is considered the body color.
The second reflection term explains surface reflection.
m.sub.s(.theta..sub.i, .phi..sub.i, .theta..sub.e, .phi..sub.e) is
the amount of the incident energy reflected by the surface
reflection as a function of the geometry as expressed by the
incident angles .theta..sub.i, .phi..sub.i and exit angles
.theta..sub.e, .phi..sub.e. c.sub.s(.lamda.) is the fraction of the
incident energy reflected by the surface reflection for each
wavelength .lamda.. The power of the dichromatic reflection model
is that the model predicts that the appearance, or color, of a
uniform color dielectric surface under a single incident illuminant
falls within a plane defined by the two color vectors c.sub.b and
c.sub.s in RGB space, even with variation in the amount of incident
illuminant and specularities on the surface.
[0048] According to a feature of the present invention, the BIDR
model adds body and surface reflection terms for the ambient
illuminant, which, as noted above may have a significantly
different spectrum from the incident illuminant. Unlike the
incident illuminant, the ambient illuminant is radiated from the
whole hemisphere visible from the surface point.
[0049] Thus, M.sub.a(.lamda.) is the integral of the ambient body
reflection: .intg.m.sub.b(.theta..sub.i)l.sub.a(.theta..sub.i,
.phi..sub.i, .lamda.)d .theta..sub.i d .phi..sub.i.
[0050] M.sub.as(.theta..sub.e, .phi..sub.e, .lamda.):
.intg.m.sub.s(.theta..sub.i, .phi..sub.i, .theta..sub.e,
.phi..sub.e)l.sub.a(.theta..sub.i, .phi..sub.i, .lamda.)d
.theta..sub.i d .phi..sub.i. This function represents the integral
of the ambient illuminant and geometric surface reflectance over
the hemisphere, excluding the incident illuminant.
[0051] The above functions are an integration of all incoming
illumination other than incident illuminant, as integrated over
.theta..sub.i, .phi..sub.i.noteq..theta..sub.L, .phi..sub.L
(incident illuminant). The BIDR model according to the present
invention can accurately and correctly accommodate and predict
tremendous complexity in the appearance of a surface, even a
surface of uniform material. If, for example, the ambient
environment is both bright and changes quickly across a surface,
the ambient illuminant terms can cause deviations from the plane
predicted by the dichromatic reflection model. Indeed, unlike the
dichromatic reflection model, the BIDR model according to the
present invention predicts a plane in the RGB space that does not
necessarily pass through the zero point of the space, but in fact
most often exhibits an offset from the zero point that comprises
the color of the ambient illumination multiplied by the object
color. The offset results in color correct changes when a pixel
value is manipulated using the BIDR model of the present
invention.
[0052] Given that the ambient illuminant is often an order of
magnitude or more than the incident illuminant, the ambient
specularity or body surface reflectance M.sub.as(.theta..sub.e,
.phi..sub.e, .lamda.) term has a minimal effect on the appearance
of a surface and can be discarded without significantly affecting
the quality of the model. The BIDR model then reduces to three
terms: I.sub.(x, y, z, .theta., .phi.,
.lamda.)=[m.sub.b(.theta..sub.i)c.sub.b(.lamda.)+m.sub.s(.theta..sub.i,
.phi..sub.i, .theta..sub.e,
.phi..sub.e)c.sub.s(.lamda.)]l.sub.d(.lamda.)+M.sub.a(.lamda.)c.sub.b(.la-
mda.).
[0053] Inasmuch as the appearance of cast shadows on a surface is
also relevant to the uses of the BIDR model, a shadow term or
factor s.sub.x, y, z .epsilon. [0,1] can be added to multiply the
incident illuminant term: I.sub.(x, y, z, .theta., .phi.,
.lamda.)=[m.sub.b(.theta..sub.i)c.sub.b(.lamda.)+m.sub.s(.theta..sub.i,
.phi..sub.i, .theta..sub.e,
.phi..sub.e)c.sub.s(.lamda.)]l.sub.d(.lamda.)s.sub.x, y,
z+M.sub.a(.lamda.)c.sub.b(.lamda.).
[0054] A fully lit pixel will have a shadow factor, s=1, and a
fully shadowed pixel will have an s=0, with the only remaining term
in the model at s=0 being the ambient body reflection. The model
can be restated by substituting .gamma..sub.b, the product of the
shadow factor s.sub.x, y, z and the geometric factor
m.sub.b(.theta..sub.i), for those non-spectral factors in the body
reflection, and .gamma..sub.s, the product of the shadow factor
s.sub.x, y, z and the geometric factor m.sub.s(.theta..sub.i,
.phi..sub.i, .theta..sub.e, .phi..sub.e), for those non-spectral
factors in the surface reflection. The BIDR model is thus restated
as: I.sub.(x, y, z, .theta., .phi.,
.lamda.)=c.sub.b(.lamda.)l.sub.d(.lamda.).gamma..sub.b+c.sub.s(.lamda.)l.-
sub.d(.lamda.).gamma..sub.s+M.sub.a(.lamda.)c.sub.b(.lamda.).
[0055] In the absence of specularities in a material, the second
term of the BIDR model,
c.sub.s(.lamda.)l.sub.d(.lamda.).gamma..sub.s, is zero, and the
measured values of a surface will exist on a single theoretical
line in, for example, RGB space. With no specularity term, the BIDR
model is given by: I.sub.(x, y, z, .theta., .phi.,
.lamda.)=c.sub.b(.lamda.)l.sub.d(.lamda.).gamma..sub.b+M.sub.a(.lamda.)c.-
sub.b(.lamda.). FIG. 4 shows a graph in RGB color space for colors
for a material, from a fully shaded color value to a fully lit
color value, as determined by execution of a simplified
bi-illuminant dichromatic reflection model from .gamma..sub.b=0 to
.gamma..sub.b=1, according to a feature of the present invention.
As shown in FIG. 4, the model predicts that all of the measured
colors of a particular material in shadow, light and penumbra
extend along a line in RGB space (the cube shown in FIG. 4). An
adjustment to a pixel value along a line predicted according to the
BIDR model yields a color correct result. For a further discussion
regarding the generation of BIDR cylinder representations of an
image, reference should be made to co-pending application Ser. No.
11/341,753, filed Jan. 27, 2006, entitled "Method And System For
Identifying Illumination Fields In an Image," which is hereby
incorporated by reference.
[0056] In practice, a camera or other sensor used to record an
image typically has noise, and no material in an image is of a
completely uniform color. Accordingly, the appearance values of the
surface fall within a cylinder having a width determined by
variation in material and imaging sensor noise characteristics. The
cylinder representing body reflection, according to the incident
and ambient illuminants considered in the BIDR model of the present
invention, has a unique starting place for every intrinsic color in
the scene. The starting point is determined by the product of the
ambient illumination and the body color, and the slope of the
cylinder in RGB space is determined by the product of the incident
illumination and the body color.
[0057] FIGS. 5a & b are graphs in RGB color space showing
colors for a material, from a fully shaded color value to a fully
lit color value, with error bounds forming a cylinder. In FIG. 5a,
a cylinder is shown around the RGB line predicted by the BIDR model
(a BIDR cylinder), with the dimensions of the cylinder diameter
being a function of the noise of the recording device used to
record the image, and variations.
[0058] FIG. 5b, illustrates a cone shaped BIDR cylinder in
recognition that the same absolute distance at the fully shadowed
end represents a greater actual color difference than the same
absolute distance from the line at the fully lit end of the line.
This is due to the fact that the magnitude of the color band
intensities at the shadow end are a greater percentage of the
actual distance from the line in the shadow end than at the lit
end. For example, consider a color at the shadow end of the
cylinder of (RGB)=(5, 6, 8) and a color at the lit end of (100,
120, 160). If the CPU 12 modifies the color value for each 10 units
toward the color red from both color values, the result is a shadow
color value of (15, 6, 8) and a lit color of (10, 120, 160). At the
dark end, the color shifted from a distinctly blue color to a
distinctly red color, while at the lit end the color remains blue.
In order to compensate for the difference, an adjustment is made to
the cylinder to be narrower at the shadow end and wider at the lit
end such that a same magnitude of color change will have a
correspondingly accurate absolute distance from the line.
[0059] When stating the spectral ratio, S=Dark/(Bright-Dark), in
terms of the BIDR model,
S=M.sub.a(.lamda.)c.sub.b(.lamda.)/([c.sub.b(.lamda.)l.sub.d(.lamda.).gam-
ma..sub.b+M.sub.a(.lamda.)c.sub.b(.lamda.)]-M.sub.a(.lamda.)c.sub.b(.lamda-
.)). This equation reduces to
S=M.sub.a(.lamda.)/l.sub.d(.lamda.).gamma..sub.b. Inasmuch as
.gamma..sub.b is scalar, the relative proportions of different
color bands in the spectral ratio, for example, RGB values, are
constant. Thus, for a given pair of direct and ambient illuminants
in an image, the normalized spectral ratio is constant for all
Bright and Dark pairs that correspond to the same material when
Dark is assumed to have a .gamma..sub.b=0, and the Bright is
receiving some incident illuminant. The spectral ratio is therefore
a representation of image properties and characteristics that,
according to the BIDR model, can be utilized as a basis for
manipulation of pixel values for a color correct adjustment, for
example, as a prediction of color adjustments that fall within a
color correct cylinder in RGB space, as illustrated in FIG. 5b.
[0060] In addition, the BIDR model can be used as a basis for
representing pixel values. Rather than an RGB value, each pixel is
represented by a BIDR cylinder in RGB space and a .gamma. value, to
indicate where in a specific BIDR cylinder, for example, the
cylinder of FIG. 5b, the color of a specific pixel is located.
Moreover, a spectral ratio based upon the Dark and Bright pixels of
a specific BIDR cylinder can be calculated as a local
characteristic or illuminant ratio for the image, as discussed
above. The BIDR cylinder/.gamma./spectral ratio representation for
a pixel provides a comprehensive and unique form of pixel
representation that permits color correct manipulations of the
pixel.
[0061] In step 106, the user selects an area from within the
regions provided by the user in steps 100, 102, to be manipulated
or corrected, and the degree of adjustment. The degree of
adjustment is reflected in the value of .gamma..sub.b, with, for
example, .gamma..sub.b=0.5 being selected if the user desired a
half-shadow. In step 108, the CPU 12 uses the spectral ratio
calculated in step 104 to adjust the intensity and color of the
selected region to achieve the degree of adjustment selected by the
user.
[0062] FIG. 10 is a flow chart for modifying apparent level of
direct or incident illumination of an image area, according to a
feature of the present invention. The flow chart of FIG. 10 can be
utilized to carry out step 108 of FIG. 3. In the routine of FIG.
10, the CPU 12 is given, in addition to the spectral ratio
calculated by the CPU 12 from Dark, for example, a pixel in the
fully shadowed material of the region selected by the user, and
Bright, a pixel in the fully lit portion of that material selected
by the user (step 104), the set of pixels comprising the area
selected by the user (step 106), a gamma (.gamma.) map for the set
of pixels, and a new gamma map set at a level corresponding to the
degree of adjustment selected by the user, for example, a
.gamma..sub.b=0.5 if the user desired a half-shadow (step 114).
[0063] A method for generating an approximate gamma map for the
pixels of the image can comprise a dynamic range compression
algorithm as an input. Examples of such algorithms include a
Retinex based algorithm, marketed by Hewlett-Packard or a gamma
correction algorithm, an industry standard image compression
algorithm. Note that a gamma correction process is distinct from a
gamma map that indicates a gamma (.gamma. corresponding to
.gamma..sub.b of the BIDR model representation). For a further
discussion of gamma map generation, see the description of FIGS. 6b
and 6c, below.
[0064] In step 116, the CPU 12 operates to calculate a fully
shadowed (Dark) version of each pixel in the selected area,
Dark=P/(1+.gamma./S), where P is the color vector in RGB space of a
pixel in the selected area of the subject image 18, gamma (.gamma.)
is the value of gamma for that pixel from the gamma map, and S is
the spectral ratio calculated by the CPU 12 in step 104 of FIG. 3.
The use of the spectral ratio as a representation of the BIDR model
for the subject image 18 results in a color correct Dark
determination corresponding to the end point of the cylinder
predicted by the BIDR model for the pixel, as depicted in FIG. 5b.
Thus, the Dark value correctly corresponds to the
M.sub.a(.lamda.)c.sub.b(.lamda.) term for that pixel in the
selected area of the subject image 18.
[0065] In step 118, the CPU 12 calculates a color adjusted new
value for each pixel in the selected area as a function of the new
gamma map designated by the user: P.sub.new=Dark+.gamma..sub.new
(Dark/S), where .gamma..sub.new is the new value for the pixel from
the new gamma map designated by the user. In step 120, the CPU 12
outputs the image area with the modified incident illuminant (step
108 of FIG. 3).
[0066] Referring once again to FIG. 3, the CPU 12 can smooth,
feather or implement other known techniques to the modified image
(step 110) and then output the modified image, for example for
display on the monitor 20 or print out on the printer 22 (step
112).
[0067] FIG. 6a shows a flow chart for a color correct brightness
adjustment to an intensity adjusted image, according to a feature
of the present invention. The CPU 12 is given an original image
area, an intensity adjusted image area, a gamma map and a spectral
ratio S for the original image area (step 200). The intensity
adjusted image area can be the output of a known dynamic range
compression algorithm, such as, for example, the Retinex-based
dynamic range compression algorithm marketed by Hewlett-Packard or
a Foveon statistics-based dynamic range compression algorithm.
These algorithms brighten darker areas of an image for a more
pleasing appearance, by adjusting the intensity but not the color
of pixels: f*(R, G, B)=>(fR, fG, fB). The routine of FIG. 6a is
meant to shift the colors as well as the intensity for each pixel
for a color correct image adjustment. The gamma (.gamma.) map can
be estimated by ascertaining a difference between the intensity
adjusted image and the original image.
[0068] To that end, reference is made to FIGS. 6b and 6c. FIG. 6b
is a graph in RGB space showing an intensity adjusted pixel value
for the intensity adjusted image input to the CPU 12 in the routine
of FIG. 6a. The graph shows the RGB value of the original pixel P
and the intensity adjusted value for the pixel A. The graph also
shows the position of a fully shadowed version of the pixel Dark,
and a color correct fully lit version B, as predicted by the BIDR
model according to a feature of the present invention. As can be
clearly seen in the graph of FIG. 6b, the color of A is displaced
in RGB space from the color correct value for the intensity
adjusted pixel, B. The result of execution of the routine of FIG.
6a by the CPU 12 is an adjustment of A to the value of B, for a
color correct depiction of the scene in the image.
[0069] FIG. 6c is a flow chart for estimating a gamma (.gamma.)
value for the pixel of FIG. 6b using a dynamic range compression
algorithm. In step 400, the CPU 12 is provided with the input
information corresponding to the input to the CPU 12 in step 200 of
FIG. 6a, absent the gamma value to be determined by the routine of
FIG. 6c. In step 402, for each pixel in the image 18, the CPU 12
calculates a ratio of the scalar value of the intensity adjusted
pixel of the image divided by the scalar value of the original
pixel value P: R=.parallel.A.parallel./.parallel.P.parallel..
[0070] In step 404, for each pixel, the CPU 12 determines a Q, as
the value of the minimum of R or the value of
(1+.parallel.S.parallel.)/.parallel.S.parallel.. In step 406, for
each pixel, the CPU 12 calculates a gamma value:
.gamma.=[(1+.parallel.S.parallel.)/Q]-.parallel.S.parallel.. After
completing steps 402-406 for each pixel, the CPU outputs a gamma
map of the image, with a gamma value for each pixel. The gamma
value indicates the percent of incident or direct illuminant on the
pixel.
[0071] In step 202, the CPU 12 calculates a fully shadowed version
(Dark) for each pixel as performed by the CPU 12 in step 116 of
FIG. 10, utilizing the algorithm Dark=P/(1+.gamma./S) wherein
.gamma. is the estimated value in the map generated from execution
of the routine of FIG. 6c. In step 204, the CPU 12 uses the Dark
version of each pixel, the spectral ratio and the estimated .gamma.
to calculate the color and intensity of each pixel such that the
corrected pixel has an intensity that matches the intensity
adjusted image with a correct color. Initially the CPU 12
calculates a bright version of each pixel:
P.sub.bright=P.sub.original*(1+S)/(.gamma.+S). P.sub.bright is the
color correct version of P.sub.original. Next, the CPU 12
determines an RGB vector relative to intensity change for each
pixel: V=P.sub.bright-Dark.
[0072] Thereafter, the CPU 12 calculates a fraction representing
the difference between the goal intensity, that is, the intensity
of the intensity adjusted image, and the intensity of the
P.sub.bright: fraction=(P.sub.goal intensity-P.sub.bright
intensity)/V.sub.intensity. The adjustment required to the
intensity of P.sub.bright to reach the intensity of the intensity
adjusted image is Adjustment=fraction*V. The final pixel value is
then: P.sub.final=P.sub.bright+Adjustment.
[0073] After the CPU 12 completes step 204 for each pixel in the
image area, the CPU 12 outputs a color correct intensity adjusted
image (step 206).
[0074] FIG. 7 is a flow chart for removing direct or incident
illumination from an image area, according to a feature of the
present invention. In step 208, the CPU 12 is given an original
image area, which can be, for example, an area of an image file 18
selected by a user, a gamma map for the image area, and a
characteristic spectral ratio S for the image area.
[0075] In step 210, the CPU calculates a shadowed version (Dark) of
each pixel in the image area, again as performed by the CPU 12 in
step 116 of FIG. 10, utilizing the algorithm Dark=P/(1+.gamma./S).
Upon completion of step 210 for all pixels, the CPU 12 outputs an
image with all materials lit only by the ambient illuminant (step
212).
[0076] FIG. 8 shows a flow chart for modifying apparent ambient
illumination of an image area, according to a feature of the
present invention. In step 214, the CPU 12 is given an original
image area, a gamma map, a characteristic spectral ratio S for the
image area, and an ambient illuminant modifier vector V. The
ambient illuminant modifier vector is a vector in, for example, RGB
space that, when multiplied by the actual vector of a fully
shadowed pixel in the image (Dark), provides a modified version of
the pixel with a new apparent ambient illumination.
[0077] In step 216, the CPU 12 uses the spectral ratio and gamma
map to calculate a fully shadowed (Dark) version of each pixel in
the image area, again using the algorithm: Dark=P/(1+.gamma./S). In
steps 218, 220, the CPU 12 multiplies each Dark pixel by V to yield
Dark.sub.new and thereby obtain a new apparent ambient illuminant
(step 218), and also multiply S by V to obtain S.sub.new (step
220).
[0078] In step 222, The CPU 12, using S.sub.new and the gamma map,
calculates a new color for each pixel:
P.sub.new=Dark.sub.new+.gamma.(Dark.sub.new/S.sub.new). Upon
completion of step 222 for all pixels, the CPU 12 outputs an image
area with a modified apparent ambient illuminant (step 224).
[0079] FIG. 9 is a flow chart for modifying apparent direct or
incident illumination of an image area, according to a feature of
the present invention. In step 300, the CPU is given an original
image area, a gamma map and spectral ratio for the image area and
an incident or direct illuminant modifier vector V. The direct
illuminant modifier vector is a vector in, for example, RGB space
that can be used to modify the spectral ratio for use in modifying
the apparent direct illumination in the image area.
[0080] In step 302, the CPU 12 calculates a Dark version of each
pixel, as completed in the routines of the previous figures. In
step 304, a new spectral ratio is determined by the CPU 12 using
the vector V, S.sub.new=S*1/V. The new spectral ratio permits a
manipulation of pixel values to in effect, modify the apparent
incident illumination. In step 306, the CPU 12 uses S.sub.new and
the gamma map to calculate a new color for each pixel:
P.sub.new=Dark+.gamma.(Dark/S.sub.new). Upon completion of step 306
for all pixels, the CPU 12 outputs an image area with a modified
apparent direct or incident illuminant.
[0081] FIG. 11 is a graph in RGB space showing a color correct
gamma correction, according to a feature of the present invention.
The graph shows the RGB value of the original pixel C and the
intensity adjusted value for the pixel B.sub.1 after gamma
correction. The graph also shows the position of a fully shadowed
version of the pixel, Dark, and a color correct fully lit version
P, as contained within a BIDR cylinder, as predicted by the BIDR
model according to a feature of the present invention. As can be
clearly seen in the graph of FIG. 11, the color of B.sub.1 is
displaced in RGB space from the color correct value for the pixel
P. The pixel B.sub.2 is the color correct intensity adjusted value
for the pixel P (and thus of the original pixel C) after gamma
correction, according to the present invention.
[0082] Gamma correction is an industry standard method for
brightening an image for display on a monitor or in print.
According to the standard, an image is adjusted such that for each
pixel in the image, the intensity is measured on a scale of 0 to 1,
and the brightened intensity is equal to the original intensity
raised to a power of (1.0/gamma), where gamma is a value greater
than or equal to 1.0. The gamma used in the industry standard is
not to be confused with gamma used above in this document to
represent the fraction of incident light on a surface. To avoid
confusion, the brightening factor used herein can be defined as
lambda (.lamda.). Gamma correction brightens all pixels some amount
irrespective of illumination. Like other forms of dynamic range
compression, gamma correction brightens shadows without making them
more similar in color to the color correct unshaded counterparts.
As will be described, the concepts of the BIDR model of the present
invention can be applied to brighten shadows and adjust the color
to provide a color correct, and thus, more visually pleasing
image.
[0083] FIG. 12 is a flow chart for a color correct gamma
correction, as shown in FIG. 11. In step 600, the CPU 12 is
provided an original image, from an image file 18, a brightening
power, .lamda., and a normalized spectral ratio for the image, S.
In step 602, for each pixel C, the CPU 12 finds the brightened
intensity(B.sub.1) as intensity(C).sup.1.0/lambda, as in a standard
and normal gamma correction. Then, in step 604, the CPU 12
calculates two ratios. R.sub.1 is some small percentile (such as
the 5th, 10th, 20th, or 40th) value of
intensity(B.sub.1)/intensity(C). R.sub.2 is some large percentile
(such as 60th, 80th, or 95th) value of
intensity(B.sub.1)/intensity(C). R1 represents a minimum degree of
brightening, and pixels with a smaller ratio are assumed to be
fully lit; R2 represents a maximum degree of brightening, and
pixels with a larger ratio are assumed to be fully shadowed.
[0084] In step 606, the CPU 12 calculates the intensity of the DARK
pixel (intensity(D)) of the BIDR cylinder as
intensity(D)=intensity(B.sub.1)/R.sub.2. If
intensity(D)>intensity(C), then the value C is updated to be
C*intensity(D)/intensity(C). Next, in step 608, the CPU 12
calculates the intensity of the bright pixel P on the BIDR cylinder
C as intensity(C)/R.sub.1 if
intensity(B.sub.1)/intensity(C)>R.sub.1; otherwise, the
intensity(P) is set to intensity(C). The intensity(P) will be less
than the intensity(B.sub.1) because gamma correction brightens all
pixels, even pixels in full incident illumination. The ratio
R.sub.1 is a measurement of the excess brightening beyond full
incident illumination.
[0085] Knowing the normalized spectral ratio S, the original color
C, and the intensity(D), the CPU 12 calculates the dark color D in
step 610. The dark color D is the color such that D=(kSC)/(1+kS)
for some scalar k. There is one value of k that satisfies the
equation with the correct intensity for D. The value can be found
by a direct algebraic solution or by successive approximation.
[0086] With the dark color D, the intensity(P), and the normalized
spectral ratio S, the CPU 12 calculates the fully lit color P in
step 612. The color P is the color with the proper intensity such
that P=(D+DSk)/(Sk) for some scalar k. The scalar k can be found as
k=(D.sub.R/S.sub.R+D.sub.G/S.sub.G+D.sub.B/S.sub.B)/(3*intensity(P)-D.sub-
.R-D.sub.G-D.sub.B). In step 614, the CPU 12 brightens the color P
to have the same intensity as B.sub.1, resulting in the color
corrected brightened color B.sub.2. Following these steps for each
pixel in the image produces a gamma corrected image with
appropriately colored shadows (step 616).
[0087] In the preceding specification, the invention has been
described with reference to specific exemplary embodiments and
examples thereof. It will, however, be evident that various
modifications and changes may be made thereto without departing
from the broader spirit and scope of the invention as set forth in
the claims that follow. The specification and drawings are
accordingly to be regarded in an illustrative manner rather than a
restrictive sense.
* * * * *