U.S. patent application number 14/546627 was filed with the patent office on 2015-03-12 for generalized assorted pixel camera systems and methods.
The applicant listed for this patent is Sony Corporation, The Trustees of Columbia University in the City of New York. Invention is credited to Tomoo Mitsunaga, Shree K. Nayar, Fumihito Yasuma.
Application Number | 20150070562 14/546627 |
Document ID | / |
Family ID | 41114742 |
Filed Date | 2015-03-12 |
United States Patent
Application |
20150070562 |
Kind Code |
A1 |
Nayar; Shree K. ; et
al. |
March 12, 2015 |
GENERALIZED ASSORTED PIXEL CAMERA SYSTEMS AND METHODS
Abstract
Generalized assorted pixel camera systems and methods are
provided. In accordance with some embodiments, the generalized
assorted pixel camera systems include a color filter array, where
the color filter array includes a plurality of primary filters and
a plurality of secondary filters. Each filter has a particular
spectral response and each filter is formed on a corresponding
pixel of a plurality of pixels. Each of the plurality of primary
filters and the plurality of secondary filters enhances an
attribute of image quality and the information obtained using the
plurality of primary filters and the plurality of secondary filters
is used to balance spectral resolution, dynamic range, and spatial
resolution for generating an image of a plurality of image
types.
Inventors: |
Nayar; Shree K.; (New York,
NY) ; Yasuma; Fumihito; (Tokyo, JP) ;
Mitsunaga; Tomoo; (Kawasaki, JP) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
The Trustees of Columbia University in the City of New York
Sony Corporation |
New York
Tokyo |
NY |
US
JP |
|
|
Family ID: |
41114742 |
Appl. No.: |
14/546627 |
Filed: |
November 18, 2014 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
12736333 |
May 11, 2011 |
8934034 |
|
|
PCT/US09/38510 |
Mar 27, 2009 |
|
|
|
14546627 |
|
|
|
|
61072301 |
Mar 28, 2008 |
|
|
|
61194725 |
Sep 30, 2008 |
|
|
|
Current U.S.
Class: |
348/336 |
Current CPC
Class: |
H04N 5/238 20130101;
H04N 9/04559 20180801; H04N 9/045 20130101; H01L 27/14621 20130101;
H04N 5/332 20130101; H04N 9/64 20130101 |
Class at
Publication: |
348/336 |
International
Class: |
H04N 5/238 20060101
H04N005/238; H04N 9/64 20060101 H04N009/64 |
Claims
1. A color filter array, comprising: a plurality of first color
filters each having a first spectral response; a plurality of
second color filters each having a second spectral response that is
different from the first spectral response; a plurality of third
color filters each having a third spectral response that is
different from the first spectral response and different from the
second spectral response; and a plurality of fourth color filters
each having a spectral response that is highly correlated with at
least a portion of the first spectral response, the second spectral
response or the third spectral response, wherein the plurality of
fourth filters includes at least four filters that each have a
different spectral response; wherein each of the plurality of first
color filters are arranged such that the first color filter is
adjacent to a second color filter and a third color filter, each of
the plurality of second color filters are arranged such that the
second color filter is adjacent to two first color filters and a
fourth color filter, each of the plurality of third color filters
are arranged such that the third color filter is adjacent to two
first color filters and a fourth color filter, and each of the
plurality of fourth color filters is arranged such that the fourth
color filter is adjacent to two second color filters and two third
color filters, and spaced apart from another fourth color filter
having the same spectral response by at least three color filters
along two orthogonal directions of the color filter array.
2. The color filter array of claim 1, wherein the transmittance of
each fourth filter is lower than the transmittance of the plurality
of first color filters, the plurality of second filters or the
plurality of third color filters that has a spectral response that
is highly correlated with that fourth filter.
3. The color filter array of claim 1, wherein the transmittance of
each fourth filter is lower than the transmittance of the plurality
of first color filters, the transmittance of the plurality of
second color filters, and the transmittance of the plurality of
third color filters.
4. The color filter array of claim 1, wherein the plurality of
first color filters are blue filters, the plurality of second color
filters are green filters, and the plurality of third color filters
are red color filters.
5. The color filter array of claim 4, wherein the plurality of
fourth color filters comprise: a plurality of fifth color filters
having a fourth spectral response that is highly correlated with at
least a portion of the first spectral response and having a
transmittance that is lower than the transmittance of the plurality
of first color filters; a plurality of sixth color filters having a
fifth spectral response that is highly correlated with a portion of
the second spectral response and having a transmittance that is
lower than the transmittance of the plurality of second color
filters; a plurality of seventh filters having a sixth spectral
response that is highly correlated with a portion of the second
spectral response and having a transmittance that is lower than the
transmittance of the plurality of second color filters, wherein a
wavelength at which the sixth spectral response has peak
transmittance is substantially different a wavelength at which the
fifth spectral response has peak transmittance; and a plurality of
eighth color filters having an eighth spectral response that is
highly correlated with at least a portion of the third spectral
response and having a transmittance that is lower than the
transmittance of the third color filters.
6. A camera system, comprising: a color filter array, comprising: a
plurality of first color filters each having a first spectral
response; a plurality of second color filters each having a second
spectral response that is different from the first spectral
response; a plurality of third color filters each having a third
spectral response that is different from the first spectral
response and different from the second spectral response; and a
plurality of fourth color filters each having a spectral response
that is highly correlated with at least a portion of the first
spectral response, the second spectral response or the third
spectral response, wherein the plurality of fourth filters includes
at least four filters that each have a different spectral response;
wherein each of the plurality of first color filters are arranged
such that the first color filter is adjacent to a second color
filter and a third color filter, each of the plurality of second
color filters are arranged such that the second color filter is
adjacent to two first color filters and a fourth color filter, each
of the plurality of third color filters are arranged such that the
third color filter is adjacent to two first color filters and a
fourth color filter, and each of the plurality of fourth color
filters is arranged such that the fourth color filter is adjacent
to two second color filters and two third color filters, and spaced
apart from another fourth color filter having the same spectral
response by at least three color filters along two orthogonal
directions of the color filter array; and an image sensor
comprising a plurality of pixels, wherein the color filter array is
disposed in the camera system such that, during operation of the
camera system, each color filter of the color filter array
corresponds to a pixel of the plurality of pixels.
7. The camera system of claim 6, wherein the transmittance of each
fourth filter is lower than the transmittance of the plurality of
first color filters, the plurality of second filters or the
plurality of third color filters that has a spectral response that
is highly correlated with that fourth filter.
8. The camera system of claim 6, wherein the transmittance of each
fourth filter is lower than the transmittance of the plurality of
first color filters, the transmittance of the plurality of second
color filters, and the transmittance of the plurality of third
color filters.
9. The camera system of claim 8, wherein the plurality of first
color filters are blue filters, the plurality of second color
filters are green filters, and the plurality of third color filters
are red color filters.
10. The camera system of claim 9, wherein the plurality of fourth
color filters comprise: a plurality of fifth color filters having a
fourth spectral response that is highly correlated with at least a
portion of the first spectral response and having a transmittance
that is lower than the transmittance of the plurality of first
color filters; a plurality of sixth color filters having a fifth
spectral response that is highly correlated with a portion of the
second spectral response and having a transmittance that is lower
than the transmittance of the plurality of second color filters; a
plurality of seventh filters having a sixth spectral response that
is highly correlated with a portion of the second spectral response
and having a transmittance that is lower than the transmittance of
the plurality of second color filters, wherein a wavelength at
which the sixth spectral response has peak transmittance is
substantially different a wavelength at which the fifth spectral
response has peak transmittance; and a plurality of eighth color
filters having an eighth spectral response that is highly
correlated with at least a portion of the third spectral response
and having a transmittance that is lower than the transmittance of
the third color filters.
11. The camera system of claim 6, further comprising a processor
that is configured to generate a plurality of images based on image
data captured by the image sensor during a single exposure, wherein
each of the plurality of images is of a different image type of a
plurality of image types that includes at least two of: a
monochrome image, a high dynamic resolution monochrome image, a
tri-chromatic image, a high dynamic resolution tri-chromatic image,
and a multispectral image.
12. The camera system of claim 11, wherein the processor is further
configured to generate a monochrome image using primarily
information obtained using pixels corresponding to the plurality of
first color filters, the plurality of second color filters, and the
plurality of third color filters.
13. The camera system of claim 12, wherein the processor is further
configured to generate a low exposure monochrome image using
primarily information obtained using pixels corresponding to the
plurality of fourth color filters.
14. The camera system of claim 13, wherein the processor is further
configured to generate a high dynamic range image by combining
information from the low exposure monochrome image and the
monochrome image.
15. The camera system of claim 11, wherein the processor is further
configured to generate a tri-chromatic image using primarily
information obtained using pixels corresponding to the plurality of
first color filters, the plurality of second color filters, and the
plurality of third color filters.
16. The camera system of claim 15, wherein the processor is further
configured to generate the tri-chromatic image using a color
reproduction matrix and a linear transformation to combine the
information obtained using pixels corresponding to the plurality of
first color filters, the plurality of second color filters, and the
plurality of third color filters.
17. The camera system of claim 16, wherein the processor is further
configured to: generate a low exposure tri-chromatic image using
primarily information obtained using pixels corresponding to the
plurality of fourth color filters; and generate a high dynamic
range tri-chromatic image by combining information from the low
exposure tri-chromatic image and information from the tri-chromatic
image.
18. The camera system of claim 17, wherein the processor is further
configured to generate the low exposure tri-chromatic image by
estimating aliasing of the information obtained using pixels
corresponding to the plurality of fourth color filters using the
information obtained from pixels corresponding to the plurality of
first color filters, the plurality of second color filters, and the
plurality of third color filters.
19. The camera system of claim 11, wherein the processor is further
configured to: generate anti-aliased information by performing
anti-aliasing on the information obtained using pixels
corresponding to the plurality of fourth colored filters; and
generate a multispectral image using the anti-aliased information
and information obtained using pixels corresponding to the
plurality of first color filters, the plurality of second color
filters and the plurality of third color filters to reconstruct
spectral reflectance for the multispectral image.
Description
CROSS REFERENCE TO RELATED APPLICATION
[0001] This application is a continuation of U.S. patent
application Ser. No. 12/736,333, filed May 11, 2011, which is the
United States National Phase Application under 35 U.S.C. .sctn.371
of International Application No. PCT/US2009/038510, which claims
the benefit of U.S. Provisional Patent Application No. 61/072,301,
filed Mar. 28, 2008 and U.S. Provisional Patent Application No.
61/194,725, filed Sep. 30, 2008. Each of the above-referenced
patent applications is hereby incorporated by reference herein in
its entirety.
NOTICE CONCERNING COLOR DRAWINGS
[0002] It is noted that the patent or application file contains at
least one drawing executed in color. Copies of this patent or
patent application publication with color drawings will be provided
by the Office upon request and payment of the necessary fee.
[0003] Nonetheless, because some readers will not have the color
drawings available, the description will also endeavor to describe
the drawings and the images they depict in a color-neutral manner,
which may create apparent redundancies of description.
COPYRIGHT NOTICE
[0004] A portion of the disclosure of this patent document contains
material which is subject to copyright protection. The copyright
owner has no objection to the facsimile reproduction by anyone of
the patent document or the patent disclosure, as it appears in the
Patent and Trademark Office patent file or records, but otherwise
reserves all copyright rights whatsoever.
TECHNICAL FIELD
[0005] The disclosed subject matter relates to generalized assorted
pixel camera systems and methods.
BACKGROUND
[0006] Most digital cameras and camcorders have a single image
sensor, such as a charge coupled device (CCD) image sensor or a
complementary metal-oxide semiconductor (CMOS) image sensor. These
image sensors use a color filter array or mosaic, which is an
assortment of different spectral filters, formed in front of the
CCD or CMOS image sensor for color acquisition.
[0007] A commonly-used color filter array or mosaic is the Bayer
mosaic shown in FIG. 1. As shown, the Bayer mosaic includes color
filters of the three primary colors red (R), green (G), and blue
(B), where the green (G) color filters are arranged in a
checkerboard pattern and the red (R) and blue (B) color filters are
arranged in line sequence. One reason tri-chromatic filter arrays
are used is that tri-chromatic sensing is near-sufficient in terms
of colorimetric color reproducibility. It is also commonly assumed
that this pixel assortment is the only practical approach for
sensing color information with a semiconductor image sensor.
However, the Bayer mosaic is limited in its capacity because it
provides a limited amount of spectral information. That is, the
Bayer mosaic provides spectral information for the three colors red
(R), green (G), and blue (B). In addition, while interpolation and
other techniques are available to fill in missing spectral
information, these approaches typically provide a resulting image
showing color aliasing and other artifacts. For example, FIG. 2
shows the differences between a ground truth image 210 and an image
220, which suffers from color aliasing and other artifacts
resulting from a bicubic interpolation applied to signals captured
using the Bayer mosaic. Dashed region 222 identifies the portion of
image 220 that is affected by color aliasing and other
artifacts.
[0008] In recent years, new image sensing technologies have emerged
that use pixel assortments to enhance image sensing capabilities.
For high dynamic range (HDR) imaging, a mosaic of neutral density
filters with difference transmittances has been used. This approach
to high sensitivity imaging builds upon the standard Bayer mosaic
by using panchromatic pixels that collect a significantly larger
proportion of incident radiation.
[0009] Despite these advances, the previously described mosaics and
camera systems have limitations. For example, these mosaics and
camera systems are used to generate one specific type of output
image.
[0010] Accordingly, it is desirable to provide generalized assorted
pixel camera systems and methods that overcome these and other
deficiencies of the prior art.
SUMMARY
[0011] In accordance with various embodiments, generalized assorted
pixel camera mechanisms are provided. In some embodiments,
generalized assorted pixel camera systems and methods are provided
that use a color filter array or mosaic with a rich assortment of
color filters, such as the one shown in FIG. 4. A color filter
array is used for an imaging or camera system in which one of a
plurality of filters having different color separation
characteristics (or colors) is bonded to each pixel. Each of the
color filters in the color filter array can enhance a particular
attribute of image quality. These attributes include, for example,
color reproduction, spectral resolution, dynamic range, and
sensitivity. By using the information captured by each of the
filters in the color filter array, these generalized assorted pixel
camera mechanisms allow a user to create a variety of image types
(e.g., a monochrome image, a high dynamic range (HDR) monochrome
image, a tri-chromatic (RGB) image, a HDR RGB image, and/or a
multispectral image) from a single captured image.
[0012] In some embodiments, these mechanisms can provide an
approach for determining the spatial and spectral layout of the
color filter array, such as the one shown in FIG. 4. For example,
generalized assorted pixel camera systems and methods are provided
that use a cost or error approach to balance variables relating to
colorimetric and spectral color reproduction, dynamic range, and
signal-to-noise ratio (SNR).
[0013] In some embodiments, these mechanisms can provide a
demosaicing approach for reconstructing the variety of image types.
For example, generalized assorted pixel camera systems and methods
are provided that include submicron pixels and anti-aliasing
approaches for reconstructing under-sampled channels. In
particular, information from particular filters is used to remove
aliasing from the information captured by the remaining
filters.
[0014] It should be noted that these mechanisms can be used in a
variety of applications. For example, these mechanisms for
enhancing spatial and spectral layout of a color filter array can
be used in a generalized assorted pixel camera system. The camera
system can capture a single image and, using the information from
each of the filters in the color filter array, to balance or
trade-off spectral resolution, dynamic range, and spatial
resolution for generating images of multiple image types. These
image types can include, for example, a monochrome image, a high
dynamic range (HDR) monochrome image, a tri-chromatic (RGB) image,
a HDR RGB image, and/or a multispectral image) from a single
captured image.
[0015] In accordance with some embodiments, a color filter array is
provided, the array comprising: a plurality of primary filters and
a plurality of secondary filters, wherein each filter has a
particular spectral response and each filter is formed on a
corresponding pixel of a plurality of pixels; and wherein each of
the plurality of primary filters and the plurality of secondary
filters enhances an attribute of image quality and wherein the
information obtained using the plurality of primary filters and the
plurality of secondary filters is used to balance spatial
resolution and image quality for generating an image of a plurality
of image types.
[0016] In accordance with some embodiments, a method for generating
images is provided, the method comprising: providing a color filter
array, the color filter array comprising: a plurality of primary
filters and a plurality of secondary filters, wherein each filter
has a particular spectral response and each filter is formed on a
corresponding pixel of a plurality of pixels; and wherein each of
the plurality of primary filters and the plurality of secondary
filters enhances an attribute of image quality and wherein the
information obtained using the plurality of primary filters and the
plurality of secondary filters is used to balance spatial
resolution and image quality for generating an image of a plurality
of image types; capturing an image using the color filter array,
wherein information from the plurality of primary filters and the
plurality of secondary filters corresponding to the image is
obtained; and generating the image in a plurality of image types
using the information from the plurality of primary filters and the
plurality of secondary filters.
[0017] In accordance with some embodiments, a camera system is
provided, the system comprising: a color filter array, the color
filter array comprising: a plurality of primary filters and a
plurality of secondary filters, wherein each filter has a
particular spectral response and each filter is formed on a
corresponding pixel of a plurality of pixels; and wherein each of
the plurality of primary filters and the plurality of secondary
filters enhances an attribute of image quality and wherein the
information obtained using the plurality of primary filters and the
plurality of secondary filters is used to balance spatial
resolution and image quality for generating an image of a plurality
of image types.
[0018] In some embodiments, an image processing system is provided,
the system comprising: a processor that is configured to: receive
information corresponding to an image from a color filter array,
wherein the color filter array includes a plurality of primary
filters and a plurality of secondary filters, wherein each filter
has a particular spectral response and each filter is formed on a
corresponding pixel of a plurality of pixels and wherein each of
the plurality of primary filters and the plurality of secondary
filters enhances an attribute of image quality and wherein the
information obtained using the plurality of primary filters and the
plurality of secondary filters is used to balance spatial
resolution and image quality for generating an image of a plurality
of image types; and generate the image in a plurality of image
types using the information from the plurality of primary filters
and the plurality of secondary filters.
BRIEF DESCRIPTION OF THE DRAWINGS
[0019] FIG. 1 illustrates an example of a Bayer mosaic.
[0020] FIG. 2 illustrates the differences between a ground truth
image and an image showing color aliasing and other artifacts
resulting from a bicubic interpolation applied to signals captured
using the Bayer mosaic of FIG. 1 in accordance with some
embodiments of the disclosed subject matter.
[0021] FIG. 3 illustrates the Modulation Transfer Function (MTF)
calculated for various pixel sizes in accordance with some
embodiments of the disclosed subject matter.
[0022] FIG. 4 illustrates a color filter array or arrangement in
accordance with some embodiments of the disclosed subject
matter.
[0023] FIG. 5 illustrates the Nyquist or usable frequency region of
the color filter array shown in FIG. 4 and the optical resolution
limit for submicron pixels in accordance with some embodiments of
the disclosed subject matter.
[0024] FIG. 6 illustrates the spectral responses of the seven
enhanced filters (e.g., filters a, b, c, d, e, f, and g) in the
color filter array of FIG. 4 in accordance with some embodiments of
the disclosed subject matter.
[0025] FIG. 7 is a schematic diagram of a system for creating
multiple image types from a single captured image using a camera
system with a color filter array, such as the one shown in FIG. 4,
in accordance with some embodiments of the disclosed subject
matter.
[0026] FIG. 8 illustrates examples of low exposure RGB images
calculated from the secondary filters through the anti-aliasing
approach in accordance with some embodiments of the disclosed
subject matter.
[0027] FIG. 9 illustrates an original image of a Circular Zone
Plate (CZP) and multiple images generated from a single captured
image of the original image in accordance with some embodiments of
the disclosed subject matter.
[0028] FIG. 10 illustrates the Modulation Transfer Function (MTF)
calculated for the generated images shown in FIG. 9 in accordance
with some embodiments of the disclosed subject matter.
[0029] FIGS. 11 and 12 illustrate additional examples of images
generated from a single captured image in accordance with some
embodiments of the disclosed subject matter.
[0030] FIG. 13 illustrates an 8.times.8 color filter array that
includes five different color filters (Green (G), Red (R), Blue
(B), Yellow (Y), and Emerald (E)), where each color filter has two
exposures (a bright exposure and a dark exposure), in accordance
with some embodiments of the disclosed subject matter.
[0031] FIG. 14 illustrates the Nyquist or usable frequency region
of the color filter array shown in FIG. 13 in accordance with some
embodiments of the disclosed subject matter.
[0032] FIG. 15 illustrates a 9.times.9 color filter array that
includes five different color filters (Green (G), Red (R), Blue
(B), Yellow (Y), and Emerald (E)) in accordance with some
embodiments of the disclosed subject matter.
[0033] FIG. 16 illustrates the Nyquist or usable frequency region
of the color filter array shown in FIG. 15 in accordance with some
embodiments of the disclosed subject matter.
[0034] FIG. 17 illustrates a directional smoothing approach that
can be used to reduce aliasing effects for images generated using
the color filter arrays shown in FIGS. 13 and 15 in accordance with
some embodiments of the disclosed subject matter.
[0035] FIG. 18 illustrate additional examples of images generated
from a single captured image using a camera system having one of
the color filter arrays shown in FIGS. 13 and 15 in accordance with
some embodiments of the disclosed subject matter.
DETAILED DESCRIPTION
[0036] In accordance with various embodiments, generalized assorted
pixel camera mechanisms are provided. In some embodiments,
generalized assorted pixel camera systems and methods are provided
that use a color filter array or mosaic with a rich assortment of
color filters, such as the one shown in FIG. 4. A color filter
array is used for an imaging or camera system in which one of a
plurality of filters having different color separation
characteristics (or colors) is bonded to each pixel. Each of the
color filters in the color filter array can enhance a particular
attribute of image quality. These attributes include, for example,
color reproduction, spectral resolution, dynamic range, and
sensitivity. By using the information captured by each of the
filters in the color filter array, these generalized assorted pixel
camera mechanisms allow a user to create a variety of image types
(e.g., a monochrome image, a high dynamic range (HDR) monochrome
image, a tri-chromatic (RGB) image, a HDR RGB image, and/or a
multispectral image) from a single captured image.
[0037] In some embodiments, these mechanisms can provide an
approach for determining the spatial and spectral layout of the
color filter array, such as the one shown in FIG. 4. For example,
generalized assorted pixel camera systems and methods are provided
that use a cost or error approach to balance variables relating to
colorimetric and spectral color reproduction, dynamic range, and
signal-to-noise ratio (SNR).
[0038] In some embodiments, these mechanisms can provide a
demosaicing approach for reconstructing the variety of image types.
For example, generalized assorted pixel camera systems and methods
are provided that include submicron pixels and anti-aliasing
approaches for reconstructing under-sampled channels. In
particular, information from particular filters is used to remove
aliasing from the information captured by the remaining
filters.
[0039] It should be noted that these mechanisms can be used in a
variety of applications. For example, these mechanisms for
enhancing spatial and spectral layout of a color filter array can
be used in a generalized assorted pixel camera system. The camera
system can capture a single image and, using the information from
each of the filters in the color filter array, balance or trade-off
spectral resolution, dynamic range, and spatial resolution for
generating images of multiple image types. These image types can
include, for example, a monochrome image, a high dynamic range
(HDR) monochrome image, a tri-chromatic (RGB) image, a HDR RGB
image, and/or a multispectral image) from a single captured
image.
[0040] In some embodiments, generalized assorted pixel camera
mechanisms with an image sensor having submicron pixels are
provided. Generally speaking, it has been determined that the
resolution performance of an imaging sensor with submicron pixels
exceeds the optical resolution limit.
[0041] To fabricate such a camera system, it should be noted that
the resolution of an optical imaging system can be limited by
multiple factors, such as diffraction and aberration. While
aberrations can be corrected during lens design, diffraction is a
limitation that cannot be avoided. The two-dimensional diffraction
pattern of a lens with a circular aperture is generally referred to
as the Airy disk, where the width of the Airy disk determines the
maximum resolution limit of the system. This is generally defined
as:
I(.theta.)=I.sub.0{2J.sub.1(z)/z}.sup.2,
where I.sub.0 is the intensity in the center of the Airy
diffraction pattern, J.sub.1 is the Bessel function of the first
kind of order one, and .theta. is the angle of observation (i.e.,
the angle between the axis of the circular aperture and the line
between the aperture center and observation point). It should be
noted that z=.pi.q/.lamda.N, where q is the radial distance from
the optical axis in the observation plane, .lamda. is the
wavelength of the incident light, and N is the f-number of the
system. In the case of an ideal lens, this diffraction pattern is
the Point Spread Function (PSF) for an in-focus image and the
Fourier transformation of the PSF is used to characterize the
resolution of an optical imaging system. This quantity is generally
referred to as the Modulation Transfer Function (MTF). The MTF of
such an imaging system can be calculated directly from the
wavelength .lamda. of incident light and the f-number N. This is
denoted by MTF.sub.opt (.lamda., N)=F(I(.theta.)), where F(.cndot.)
denotes the Fourier transformation.
[0042] It should be noted that pixels generally have a rectangular
shape and their finite size contributes to the resolution
characteristics of the imaging system. The Modulation Transfer
Function (MTF) of an image sensor can be approximated as the
Fourier transformation of a rectangular function, which is
described by MTF.sub.sensor(p)=F(s(t)). The rectangular function
s(t) can be expressed as:
s ( t ) = { 1 t .ltoreq. p .zeta. 2 0 t > p .zeta. 2
##EQU00001##
where p is the pixel size and .zeta. is an aperture ratio, which is
generally assumed to be 1 due to the use of on-chip
microlenses.
[0043] It should also be noted that the total fundamental optical
resolution limit of a camera system (e.g., including the lens and
the sensor) can be described in the frequency domain as
MTF=MTF.sub.opt (.lamda., N)MTF.sub.sensor(p). To calculate this,
the values of .lamda.=555 nm (which generally corresponds to the
peak of the sensitivity of the human eye) and N=f/5.6 (which is a
pupil size generally used in, for example, consumer photography)
are used. With these values, the fundamental MTF is determined by
pixel size p.
[0044] The MTF for various pixel sizes is shown in FIG. 3. As
shown, the MTF is calculated for pixel sizes of 0.70 .mu.m
(represented by the leftmost curve 305), 1.00 .mu.m (represented by
curve 310), 1.25 .mu.m (represented by curve 315), and 1.75 .mu.m
(represented by the rightmost curve 320). The MTF with a pixel size
p=1.00 .mu.m is about 0.1 at 0.25 fs, where fs is the image
sensor's sampling frequency. It should be noted that the human eye
cannot recognize contrast when the MTF is less than 0.1. It should
also be noted that the optical resolution limit of an image sensor
with p=1.00 .mu.m pixel size is half of the image sensor's Nyquist
frequency. Accordingly, FIG. 3 shows that the resolution
performance of a sensor with submicron pixels exceeds the optical
resolution limit.
[0045] In some embodiments, generalized assorted pixel camera
systems and methods are provided that use a color filter array or
mosaic with a rich assortment of color filters. Again, as shown in
FIG. 3, for a 1.0 .mu.m pixel size, the combined MTF, due to
diffraction from the lens aperture and averaging by pixels, leads
to an optical resolution limit of about one-quarter of the sampling
frequency fs, where fs=1/.DELTA.s and .DELTA.s is the sampling
pitch. To exploit this, an exemplary color filter array or
arrangement 400 in accordance with some embodiments is shown in
FIG. 4. As described previously, a color filter array is used for
an imaging or camera system in which one of a plurality of filters
having different color separation characteristics (or colors) is
bonded to each pixel. It should be noted that the term "color"
generally refers to a filter or a pixel value of that color
obtained from the filter. As shown in FIG. 4, color filter array
400 includes primary filters (i.e., color filters a, b, and c) and
secondary filters (i.e., color filters d, e, f, and g).
[0046] The pixels marked a, b, and c in color filter array 400
(collectively referred to herein as "primary filters") capture
three different spectral images on a rectangular grid with sampling
pitch .DELTA.s.sub.a,b,c=2p. Accordingly, the Nyquist frequency for
a, b, and c is fn.sub.a,b,c=fS.sub.a,b,c/2=fs/4. It should be noted
that, due to diffraction, filters a, b, and c do not cause aliasing
because the optical resolution limit is one-quarter of the sampling
frequency fs. These aliasing-free pixels a, b, and c can be used to
reconstruct high resolution images, such as high resolution
monochrome images and high resolution RGB images.
[0047] The pixels marked d, e, f, and g in color filter array 400
(collectively referred to as "secondary filters") each sample the
incident image on rectangular grids through different spectral
filters. The sampling pitch for each of the secondary filters is
.DELTA.s.sub.d,e,f,g=4p and the Nyquist frequency is
fn.sub.d,e,f,g=fs.sub.d,e,f,g/2=fs/8.
[0048] To further illustrate the Nyquist frequencies of color
filter array 400, the Nyquist or usable frequency region 500 is
shown in FIG. 5. As shown, the Nyquist region of the primary
filters is located in a substantially square portion of the
frequency space indicated by dashed line 510 and the Nyquist region
of the secondary filters is located in a substantially square
portion of the frequency space indicated by a dashed line 520.
[0049] In addition, FIG. 5 also illustrates the optical resolution
limit for submicron pixels, as described above, that is shown by
shaded area 530. It should be noted that, as the Nyquist
frequencies for the secondary filters (indicated by dashed line
520) are lower than the optical resolution limit, the possibility
of aliasing artifacts is introduced. However, as shown herein,
these aliasing artifacts can be removed by using high frequency
information from the demosaiced image obtained using the primary
filters.
[0050] Using color filter array 400, a plurality of image
characteristics can be captured simultaneously. It should be noted,
however, that there may be a trade-off in the fidelity of each
characteristic. For example, monochrome and standard RGB images are
reconstructed at high resolution using the primary filters of color
filter array 400. For high dynamic range (HDR) images, the spectral
resolution is improved by using the secondary filters and
decreasing the spatial resolution.
[0051] In another example, high spatial resolution can be obtained
by sacrificing the dynamic range and the spectrum. That is, a
monochrome image has high spatial resolution. By sacrificing the
spatial resolution, quality of the spectrum is improved. By further
sacrifice of the resolution, dynamic range is expanded in addition
to the improvement of the spectrum.
[0052] In some embodiments, a cost or error function can be used to
enhance the filter spectra for the primary and secondary filters.
The cost function can incorporates several terms, such as the
quality of color reproduction (e.g., for a RGB image),
reconstruction of reflectance (e.g., for a multispectral image),
and dynamic range (e.g., for a HDR image).
[0053] The value x.sub.m measured at a pixel in the m.sup.th
channel, where m is one of the primary or secondary filters a, b,
c, d, e, f, or g, is given by the following equation:
x.sub.m=.intg..sub..lamda..sub.min.sup..lamda..sup.maxi(.lamda.)r(.lamda-
.)c.sub.m(.lamda.)d.lamda.,
where i(.lamda.) is the spectral power distribution of the
illumination, r(.lamda.) is the spectral reflectance of the scene
point, and c.sub.m(.lamda.) is the spectral response of the
camera's m.sup.th color channel. When the wavelength .lamda. is
sampled at equally-spaced L points, x.sub.m can be described by the
following discrete expression:
x m = l = 1 L i ( .lamda. l ) r ( .lamda. l ) c m ( .lamda. l )
##EQU00002##
Moreover, if the above-mentioned equation is rewritten in matrix
form, it can be described as X=C.sup.TIR, where X=[x.sub.a,
x.sub.b, . . . x.sub.g].sup.T, C=[c.sub.m(.lamda..sub.l)], I is a
diagonal matrix made up of the discrete illumination samples
i(.lamda..sub.l), and R=[r(.lamda..sub.l)].
[0054] In some embodiments, the color reproduction error
corresponding to the primary and secondary filters can be
determined. For example, to obtain HDR RGB images, a high exposure
RGB image can be reconstructed using the primary filters of color
filter array 400 and a lower exposure image can be reconstructed
using the secondary filters of color filter array 400. In some
embodiments, the spectral responses of the primary and secondary
filters are to yield the highest color reproduction. It should be
noted that a variety of color rating indicies can be used to
evaluate the color reproduction characteristics of a filter and
these indicies can use a cost function that minimizes the
difference in the color between the measured color of a reference
material and its known color.
[0055] In some embodiments, to calculate the difference of color,
the CIE 1931 XYZ color space (created by the International
Commission on Illumination), which is based on direct measurements
of human visual perception and serves as the basis of which many
other color spaces are defined, can be used. The calculation of
sRGB tristimulus values (which are employed in some digital cameras
or color monitors) from the CIE XYZ tristimulus values is a linear
transformation. The CIE XYZ tristimulus values can be defined as
Y=A.sup.TIR, where Y represents the true tristimulus values and A
is a matrix of CIE XYZ color matching functions [ x y z]. The
estimated CIE tristimulus values corresponding to the primary
filters ' can be expressed as an optimal linear transformation:
'=T'X', where X'=[x.sub.a, x.sub.b, x.sub.c].sup.T. The
transformation T' is determined so as to minimize the color
difference: min.parallel.Y-T'X'.parallel..sup.2. Similarly, the
estimated CIE tristimulus values corresponding to the secondary
filters '' can be expressed as ''=T''X'', where X''=[x.sub.d,
x.sub.e, x.sub.f, x.sub.g].sup.T.
[0056] It should be noted that the average magnitude of color
difference between the true color Y and the estimate over a set of
N real-world objects can be used as a metric to quantify the camera
system's color reproduction performance. The color reproduction
errors corresponding to the primary and secondary filters can then
be described by the following equations:
E ' ( C ) = min T ' n = 1 N Y n - T ' X n ' 2 ##EQU00003## E '' ( C
) = min T '' n = 1 N Y n - T '' X n '' 2 ##EQU00003.2##
[0057] In some embodiments, the error introduced by the
reconstruction of the spectral distribution can be determined. For
example, the spectral distribution can be reconstructed using a
linear model. Since the model is linear, the reconstruction is
efficient and stable. The linear model for the reconstruction can
be expressed as the set of orthogonal spectral basis functions
b.sub.k(.lamda.):
r(.lamda.)=.SIGMA..sub.k=1.sup.K.sigma..sub.kb.sub.k(.lamda.),
where .sigma..sub.k are scalar coefficients and K is the number of
basis functions. By substituting the above-described equation into
the cost function, the cost or error function can be described by
the following equation:
x m = k = 1 K .sigma. k .intg. .lamda. min .lamda. max b k (
.lamda. ) i ( .lamda. ) c m ( .lamda. ) .lamda. ##EQU00004##
[0058] These equations can be written as X=F.sigma., where F is a
M.times.K matrix:
F=.intg..sub..lamda..sub.min.sup..lamda..sup.maxb.sub.k(.lamda.)i(.lamda.-
)c.sub.m(.lamda.)d.lamda., M is the number of color filter channels
(for example, color filter array 400 of FIG. 4 has seven channels,
so, M=7), and .sigma.=[.sigma..sub.k]. The spectral distribution is
reconstructed by minimizing .parallel.F.sigma.-X.parallel..sup.2.
It should be noted that the spectral reflectance of most materials
is known to be smooth and is generally positive. Accordingly, the
reconstruction approach can be expressed as a constrained
minimization as follows: {circumflex over (.sigma.)}=arg
min.parallel.{tilde over (F)}.sigma.-{tilde over
(X)}.parallel..sup.2, subject to B.sigma..gtoreq.0, where {tilde
over (F)}=[F.sup.T.alpha.P.sup.T].sup.T,
P.sub.lk=.differential..sup.2b.sub.k(.lamda..sub.l)/.differential..lamda.-
.sup.2 is a smoothness constraint, .alpha. is a smoothness
parameter, 1.gtoreq.L, 1.gtoreq.k.gtoreq.K, {tilde over
(X)}=[X.sup.T0].sup.T, and B=[b.sub.k(.lamda..sub.l)]. This
regularized minimization can be solved using quadratic programming.
The multispectral image's mean squared reconstruction error R(C)
can then be expressed as:
R ( C ) = n = 1 N .sigma. n - .sigma. ^ n 2 ##EQU00005##
where .sigma..sub.n represents the actual coefficients of the
n.sup.th object and {circumflex over (.sigma.)}.sub.n are the
reconstructed coefficients. It should be noted that, in some
embodiments, the number of basis functions K is 8 and the
smoothness parameter .alpha. is set to 64.0.
[0059] In some embodiments, the cost function can include an
approach for balancing the extension of dynamic range with
signal-to-noise (SNR) ratio. As described previously, to achieve
HDR imaging, secondary filters (e.g., filters d, e, f, and g of
color filter array 400) have lower transmittances than the primary
filters (e.g., filters a, b, and c of color filter array 400). This
can cause deterioration of signal-to-noise ratio (SNR) for the
secondary filters. Such a trade-off can be controlled based on the
ratio of the exposures of the primary and secondary filters:
.beta.=e.sub.max/e.sub.min, where e.sub.max is the average exposure
of the primary filters and e.sub.min is the average exposure of the
secondary filters. Accordingly, .beta. can be determined by C from
the previously-mentioned equation X=C.sup.TIR, where the determined
value of .beta. can be used to valance the extension of dynamic
range versus the reduction of the signal-to-noise ratio.
[0060] In some embodiments, dynamic range can be defined as: DR=20
log.sub.10B.sub.full/N.sub.r, where V.sub.full represents the
full-well capacity of the detector (e.g., V.sub.full=3500e.sup.-)
and N.sub.r is the root mean square (RMS) of the read-noise of the
image sensor. The RMS of the read-noise of the detector can be
defined as N.sub.r= {square root over
(N.sub.shot.sup.2+N.sub.dark.sup.2)}. For example, N.sub.dark can
be set to 33e.sup.-. In some embodiments that use the color filter
array 400 of FIG. 4, it should be noted that N.sub.r does not
change, but the maximum detectable gray level becomes
.beta.V.sub.full. Accordingly, the dynamic range of a camera system
using color filter array 400 can be expressed as follows:
DR GAP = 20 log 10 .beta. V full N r ##EQU00006##
[0061] In some embodiments, the signal-to-noise ration can be
defined as: SNR=20 log.sub.10V/N, where V is the signal and N is
the noise. The signal corresponding to a secondary filter can be
express using the exposure .beta. as V''.sub.max=V'.sub.max/.beta.,
where V'.sub.max is a signal due to a primary filter. When the
signal due to the primary filter is not saturated, the signal due
to the secondary filter can be determined from the primary signal.
The signal-to-noise ratio for a secondary filter when the primary
signal is saturated is the worst-case signal-to-noise ratio for a
camera system using mosaic 400:
S N R GAP = 20 log 10 V full / .beta. N max ##EQU00007##
where N.sub.max= {square root over
(N''.sub.shot.sup.2+N.sub.dark.sup.2)} and N''.sub.shot= {square
root over (V.sub.full/.beta.)}.
[0062] Because the camera system has a high performance in
signal-to-noise ratio and dynamic range when SNR.sub.GAP and
DR.sub.GAP are large, the following cost function can be used:
D ( C ) = 1 DR GAP 1 S N R GAP ##EQU00008##
[0063] In some embodiments, each of the above-mentioned cost
functions can be combined to provide a total cost function. For
example, since each of the above-mentioned cost functions represent
a particular dimension of image quality, the total cost function
can be expressed as a weighted sum of the individual costs:
G=w.sub.1{E'+E''}+w.sub.2R+w.sub.3D
[0064] It should be noted that the weights (e.g., w.sub.1, w.sub.2,
and w.sub.3) can be determined according to the image quality
requirements of the application for which the camera system is used
or manufactured. For example, in some embodiments, w.sub.1=1.0,
w.sub.2=1.0, and w.sub.3=1.0 can be used for determining the total
cost function. It should also be noted that, since the filters have
positive spectral responses (C is to be positive), the enhancement
or optimization of C can be expressed as:
C = arg min C G , subject to C .gtoreq. 0 ##EQU00009##
[0065] In some embodiments, initial guesses can be assigned to the
filter spectral responses. That is, to find the seven spectral
response functions in C, initial guesses can be used along with an
optimization approach. In one example, the initial guesses for the
filter responses can be selected from a set of commercially
available optical band pass filters and on-chip filters. In another
example, commercial filters can be assigned to each of the seven
channels based on one or more of the above-mentioned cost functions
(e.g., assigning from a set of 177 commercial filters based on
color reproduction error). Accordingly, the primary filters
C'.sub.0 and secondary filters C''.sub.0 are determined such
that:
min C 0 ' E ( C 0 ' ) ( C 0 ' .di-elect cons. C 0 ) min C 0 '' E (
C 0 '' ) ( C 0 '' .di-elect cons. C 0 ) ##EQU00010##
where C.sub.0 is the set of commercial filters.
[0066] In response to assigning seven initial guesses to each of
the seven filters, an iterative application can be used to perform
a constrained non-linear minimization of
C = arg min C G . ##EQU00011##
[0067] For example, Mathworks.RTM. Matlab.RTM. or any other
suitable computing program can be used to determine the spectral
responses. Using Matlab.RTM., the FMINCON routine can be used to
find a minimum of a constrained non-linear multivariable function
as described above. However, any other suitable computer program
can be used to find the minimum of a constrained non-linear
multivariate function.
[0068] FIG. 6 illustrates the spectral responses of the seven
enhanced filters in color filter array 400 of FIG. 4. By using the
cost function to determine the spectral responses and as a result
of the color reproduction term in the cost function, it should be
noted that the primary filters a, b, and c represented by curves
605, 610, and 615, respectively, have spectral responses
substantially similar to red, green, and blue filters. Accordingly,
the primary filters can be used to obtain RGB images, which
essentially cover the entire visible light spectrum.
[0069] In addition, the spectra captured by the secondary filters
d, e, f, and g (represented by curves 620, 625, 630, and 635,
respectively), irrespective of their spectral responses, are to be
highly correlated with the images obtained using the primary
filters. Consequently, anti-aliasing of images produced by
secondary filters can be performed. Furthermore, due to the
characteristics of the cost function, the secondary filters have
lower exposures or transmittances than primary filters.
Accordingly, using the primary and secondary filters, high dynamic
range information can be obtained and, since the seven filters have
different spectra and sample the visible spectrum, their
reconstructed images can be used to obtain smooth estimates of the
complete spectral distribution of each scene point i.e., a
multispectral image.
[0070] As shown in Table 1 below, the errors in the color
reproduction and spectral reconstruction components of the total
cost function, the estimated dynamic range, and the signal-to-noise
ratio of the initial and enhanced set of seven filters of color
filter array 400. In addition, Table 1 also illustrates the errors
in the color reproduction and spectral reconstruction components of
the total cost function, the estimated dynamic range, and the
signal-to-noise ratio for the red, green, and blue filters in a
Bayer mosaic.
TABLE-US-00001 TABLE 1 Optimization accuracy Initial Enhanced Bayer
filters filters filters .DELTA.E'(C) 0.0497 0.0429 0.0490
.DELTA.E''(C) 0.0100 0.0055 N/A .DELTA.R(C) 0.0624 0.0610 0.0709
DR.sub.GAP 58.2970 62.9213 56.9020 SNR.sub.GAP 34.7069 32.3694
35.4098
[0071] It should be noted that, in response to enhancing the
spectral responses of the filters in the generalized assorted pixel
color filter array using a cost function, each of the errors in
Table 1 have been reduced. It should also be noted that the
deterioration of the signal-to-noise ratio is kept low at about 2.3
dB, while the dynamic range is improved by about 4.6 dB. It should
further be noted that the errors in color reproduction and spectral
reconstruction components of the total cost function are higher
with the Bayer mosaic.
[0072] FIG. 7 shows a schematic diagram of a system 700 for
creating multiple image types from a single captured image using a
camera system with a color filter array in accordance with some
embodiments of the disclosed subject matter.
[0073] As shown in FIG. 7, camera system 700 includes a color
filter array 710 that includes primary filters 712 and secondary
filters 714. As described previously, color filter array 710 can be
similar to color filter array 400 of FIG. 4, where primary filters
712 include three color filters a, b, and c and secondary filters
714 include four color filters d, e, f, and g. The primary filters
capture three different spectral images on a rectangular grid with
sampling pitch .DELTA.s.sub.a,b,c=2p, while the secondary filters
each sample the incident image on rectangular grids with sampling
pitch .DELTA.s.sub.d,e,f,g=4p through different spectral
filters.
[0074] As also shown in FIG. 7, information obtained from the
primary filters 712 and secondary filters 714 can be used to
generate multiple types of images, such as a monochrome image 720,
a high dynamic range (HDR) monochrome image 730, a tri-chromatic
(RGB) image 740, a HDR RGB image 760, and a multispectral image
770. In some embodiments, a multimodal demosaicing approach with
anti-aliasing is applied to generate high resolution images.
[0075] Referring back to the color filter array 400 of FIG. 4, note
that there is one color measurement at each pixel. The other colors
are estimated from information obtained by neighboring pixels in
order to, for example, reproduce high resolution output images
irrespective of the type of image (e.g., monochrome image, HDR
monochrome image, RGB image, HDR RGB image, multispectral image,
etc.). This approach is generally referred to as "demosaicing."
[0076] Denoting .LAMBDA..sub.m as the set of pixel locations, (i,
j), for channel m.epsilon.{a, b, c, d, e, f, g, a mask function for
each filter can be defined as:
W m ( i , j ) = { 1 ( i , j ) .di-elect cons. .LAMBDA. m 0
otherwise ##EQU00012##
In the color filter array 400 of FIG. 4 or color filter array 710
of FIG. 7, there are seven types of color channels--i.e., a, b, c,
d, e, f, and g. Accordingly, the observed data, y(i,j), can be
expressed as:
y ( i , j ) = m = a , b , c , d , e , f , g W m ( i , j ) x m ( i ,
j ) ##EQU00013##
where x.sub.m is the mth channel's full resolution image.
[0077] Referring back to FIG. 7, monochrome image 720 can be
generated from a single captured image by using information
obtained from primary filters 712. As described previously,
information captured by primary filters 712 do not suffer from
aliasing. Accordingly, at 722, missing information for one of the
primary filters 712 can be estimated using linear interpolation
from other primary filters 712 from color filter array 710.
[0078] Monochrome image 720 of a high resolution can be
reconstructed using information measured by primary filters. This
can be expressed as:
I.sub.M(i,j)={{circumflex over (x)}.sub.a(i,j)+{circumflex over
(x)}.sub.b(i,j)+{circumflex over (x)}.sub.c(i,j)}/3
where {circumflex over (x)}.sub.a(i,j), {circumflex over
(x)}.sub.b(i,j), {circumflex over (x)}.sub.c(i,j) are the full
resolution images obtained by interpolating pixels with the primary
filters (e.g., primary filters a, b, and c of FIG. 4). For
interpolation, a Finite Impulse Response (FIR) Filters F(i,j) can
be used and can be expressed as follows:
{circumflex over (x)}.sub.v(i,j)=W.sub.v(i,j)y(i,j)+
W.sub.v(i,j)[F(i,j)*y(i,j)]
where v=a, b, or c, * denotes convolution, and
W.sub.v(i,j)=1-W(i,j). For example, in some embodiments, the fir1
function in Mathworks.RTM. Matlab.RTM. can be used to find FIR
filters of size 30.times.30 that pass all frequencies, thereby
minimizing the loss of high frequencies due to interpolation.
[0079] In some embodiments, high dynamic range monochrome image 730
can be generated from a single captured image by using information
obtained from primary filters 712 and secondary filters 714. To
create a high dynamic range monochrome image (e.g., image 730), a
low exposure monochrome image 732 can be constructed. At 734, low
exposure monochrome image 732 is constructed using information from
secondary filters 714 (e.g., the four secondary filters d, e, f,
and g of FIG. 4). These secondary filters 714 have lower exposure
and collectively cover the whole visible spectrum.
[0080] For example, the monochrome values at pixels with filter a
(e.g., filter a of color filter array 400 shown in FIG. 4) can be
calculated. As shown in FIG. 4, color filter array 400 includes
four different secondary pixels (e.g., pixels d, e, f, and g)
arranged diagonally about each pixel a. Accordingly, the monochrome
value at each pixel a can be calculated as the average of the
measurements at the four neighboring secondary pixels and can be
expressed as:
W a ( i , j ) { Q D * y ( i , j ) } , where Q D = ( 1 4 0 1 4 0 0 0
1 4 0 1 4 ) ##EQU00014##
It should be noted that aliasing caused by half-pixel phase shifts
cancel out when adding four pixels in a diagonal neighborhood. The
values at pixel a are then interpolated to the other pixels to
yield the low exposure monochrome image 732 (ILEM), which can be
expressed as:
I.sub.LEM(i,j)=L(i,j)+W.sub.s{Q.sub.D*L(i,j)}+W.sub.b{Q.sub.H*L(i,j)}+W.-
sub.c{Q.sub.V*L(i,j)}
where:
W s ( i , j ) = { 1 ( i , j ) .di-elect cons. { d , e , f , g } 0
otherwise and Q H = Q V T = ( 0 0 0 1 2 0 1 2 0 0 0 )
##EQU00015##
[0081] After obtaining low exposure monochrome image 732, at 736, a
high dynamic range monochrome image 730 can be generated by
combining the monochrome images of different exposures and their
associated information e.g., the monochrome image 720 generated
using primary filters 712 and the low exposure monochrome image 732
generated using secondary filters 714.
[0082] In some embodiments, tri-chromatic (RGB) image 740 can be
generated from a single captured image by using information
obtained from primary filters 712. As described previously in FIG.
6, the primary filters 712 used in color filter array 710, such as
primary filters a, b, and c in FIG. 4, have spectral responses
similar to red, green, and blue filters. At 742 and 744,
tri-chromatic (RGB) image 740 can be constructed using color
reproduction matrix T' and H' (a linear transformation from CIE XYZ
tristimulus values to sRGB tristimulus values) to combine the
information in the {circumflex over (x)}.sub.a, {circumflex over
(x)}.sub.b, {circumflex over (x)}.sub.c images computed using the
primary filters. The RGB image can be expressed as:
I.sub.RGB(i,j)=HT'[{circumflex over (x)}.sub.a(i,j){circumflex over
(x)}.sub.b(i,j){circumflex over (x)}.sub.c(i,j)].sup.T
[0083] As described previously, to calculate the difference of
color for color reproduction of a RGB image, the CIE 1931 XYZ color
space (created by the International Commission on Illumination),
which is based on direct measurements of human visual perception
and serves as the basis of which many other color spaces are
defined, can be used. The calculation of sRGB tristimulus values
(which are employed in some digital cameras or color monitors) from
the CIE XYZ tristimulus values is a linear transformation. The CIE
XYZ tristimulus values can be defined as Y=A.sup.TIR, where Y
represents the true tristimulus values and A is a matrix of CIE XYZ
color matching functions [ x y z]. The estimated CIE tristimulus
values corresponding to the primary filters ' can be expressed as
an optimal linear transformation: '=T'X', where X'=[x.sub.a,
x.sub.b, x.sub.c].sup.T. The transformation T' is determined so as
to minimize the color difference: min
.parallel.Y-T'X'.parallel..sup.2.
[0084] In some embodiments, a HDR RGB image 760 can be generated
from a single captured image by using information obtained from
primary filters 712 and secondary filters 714 of color filter array
710. To create a high dynamic range tri-chromatic image (e.g.,
image 760), a low exposure tri-chromatic image 750 can be
constructed.
[0085] Full resolution secondary filter images--{circumflex over
(x)}.sub.d, {circumflex over (x)}.sub.e, {circumflex over
(x)}.sub.f, and {circumflex over (x)}.sub.g--can be respectively
computed using the d, e, f, and g pixels using bilinear
interpolation. However, this can result in severe aliasing. In some
embodiments, the aliasing of the secondary filter images can be
estimated using information from the primary filter
images--{circumflex over (x)}.sub.a, {circumflex over (x)}.sub.b,
{circumflex over (x)}.sub.c at 752. It should be noted that there
is a strong correlation between the spectra of primary filters 712
and secondary filters 714, as shown by the overlap in FIG. 6. For
example, when anti-aliasing the full resolution image {circumflex
over (x)}.sub.e that corresponds to filter e, it should be noted
that filter e has a strong correlation with that of filter a.
Accordingly, the interpolated full resolution filter a image
{circumflex over (x)}.sub.a at each filter e locations can be
sampled. These can then be used to calculate a full resolution
image for filter e, which can be expressed as:
.OMEGA.{W.sub.e(i,j){circumflex over (x)}.sub.a(i,j)}
where .OMEGA.(.cndot.) represents bilinear interpolation. Aliasing
can be inferred by subtracting the original {circumflex over
(x)}.sub.a image from the interpolated one. Then, to obtain the
final estimate of aliasing in channel e, the above-mentioned
difference can be scaled by .PSI..sub.ae, which is the ratio of the
filter transmittances of the a and e pixels, to take into account
the difference in exposures of a and e pixels. The estimated
aliasing .PSI..sub.ae can be expressed as follows:
.gamma..sub.e(i,j)=[.OMEGA.{W.sub.e(i,j){circumflex over
(x)}.sub.a(i,j)}-{circumflex over (x)}.sub.a(i,j)].psi..sub.ae
where:
.psi. ae = ( l = 1 L C e ) ( l = 1 L C a ) ##EQU00016##
Accordingly, the anti-aliased image {circumflex over (x)}.sub.e can
be calculated at 754 as:
{circumflex over
(x)}.sub.e(i,j)=.OMEGA.{W.sub.e(i,j)y(i,j)}|.gamma..sub.e(i,j)
In addition, other anti-aliased secondary can be similar
calculated.
[0086] FIG. 8 shows examples of low exposure RGB images calculated
from the secondary filters through the anti-aliasing approach. For
example, image 810 shows a low exposure RGB image calculated from
secondary filters 714 without anti-aliasing. It should be noted
that false color artifacts 812 caused by aliasing are present.
Image 820 shows the downsampled image
.OMEGA.{W.sub.e(i,j){circumflex over (x)}.sub.a(i,j)} calculated
using the pixels with primary filter a of primary filters 712.
Image 830 then shows the aliasing Y.sub.e(i,j) estimated using the
downsampled image 820 and the full resolution image for channel a.
It should be noted that the brightness of image 830 is enhanced for
visualization. Accordingly, image 840 is a lower exposure RGB image
obtained after anti-aliasing using image 830, which provides the
estimation of aliasing. Image 840 shows the efficacy of the
anti-aliasing approach, where false color artifacts (e.g.,
artifacts 812 in image 820) can be removed.
[0087] A low exposure RGB image can be obtained by multiplying the
secondary filter images by a color reproduction matrix at 756,
which can be expressed as:
I.sub.LERGB(i,j)=HT''[{circumflex over (x)}.sub.d(i,j){circumflex
over (x)}.sub.e(i,j){circumflex over (x)}.sub.f(i,j){circumflex
over (x)}.sub.g(i,j)].sup.T
where T'' is the color reproduction matrix and H is the linear
transformation from CIE XYZ to sRGB.
[0088] After obtaining low exposure RGB 750, at 758, a high dynamic
range RGB image 760 can be generated by combining the tri-chromatic
(RGB) images of different exposures and their associated
information e.g., the RGB image 740 and the low exposure RGB image
750.
[0089] In some embodiments, a multispectral image 770 can be
generated from a single captured image using information from
primary filters 712 and secondary filters 714 of color filter array
710. For multispectral image 770, the spectral reflectance of an
object can be reconstructed using images {circumflex over
(x)}.sub.a, {circumflex over (x)}.sub.b, {circumflex over
(x)}.sub.c and anti-aliased images {circumflex over (x)}.sub.d,
{circumflex over (x)}.sub.e, {circumflex over (x)}.sub.f, and
{circumflex over (x)}.sub.g at 772. In some embodiments, a HDR RGB
image 760 can be generated from a single captured image by using
information obtained from primary filters 712 and secondary filters
714 of color filter array 710.
[0090] As described previously, the spectral distribution is
reconstructed by minimizing the expression:
.parallel.F.sigma.-X.parallel..sup.2. In some embodiments, the
reconstruction approach can be expressed as a constrained
minimization as follows: {circumflex over (.sigma.)}=arg
min.parallel.{tilde over (F)}.sigma.-{tilde over
(X)}.parallel..sup.2, subject to B.sigma..gtoreq.0, where {tilde
over (F)}=[F.sup.T.alpha.P.sup.T].sup.T,
P.sub.lk=.differential..sup.2b.sub.k(.lamda..sub.l)/.differential..lamda.-
.sup.2 is a smoothness constraint, a is a smoothness parameter,
1.gtoreq.L, 1.gtoreq.k.gtoreq.K, {tilde over (X)}=[X.sup.T
0].sup.T, and B=[b.sub.k(.lamda..sub.1)]. This regularized
minimization can be solved using quadratic programming.
[0091] FIG. 9 shows an original image 910 of a Circular Zone Plate
(CZP) and multiple images 920, 930, 940, and 950 generated from a
single captured image of the original image 910 in accordance with
some embodiments. It should be noted that original image 910, which
serves as the ground truth, shows a CZP image calculated using a
diffraction-limited model of a lens with a f-number of 5.6 and a
1.0 .mu.m pixel size. Using a camera system with a generalized
assorted pixel color filter array or mosaic (e.g., color filter
array 400, color filter array 710, etc.) having multiple primary
filters and multiple secondary filters to capture an image,
multiple image types e.g., a demosaiced monochrome image 920, a
demosaiced tri-chromatic (RGB) image 930, a demosaiced and
anti-aliased low exposure monochrome image 940, and a demosaiced
and anti-aliased low exposure tri-chromatic (RGB) image 950 can be
generated.
[0092] FIG. 10 shows Modulation Transfer Function (MTF)
calculations for each image 910, 920, 930, 940, and 950. As shown,
curve 1010 is associated with original image 910, curve 1020 is
associated with monochrome image 920 and tri-chromatic (RGB) image
930, curve 1030 is associated with low exposure monochrome image
940, and curve 1040 is associated with low exposure tri-chromatic
(RGB) image 950. Note that curve 1010 for monochrome image 920 and
tri-chromatic (RGB) image 930, which were generated using primary
filters of the color filter array, is substantially similar to
curve 1020 associated with original image 910. Note also that the
low exposure monochrome image 940 has a MTF of about 0.1 at 0.1754
fs, while the low exposure tri-chromatic (RGB) image 950 has a MTF
of about 0.1 at 0.1647 fs. For standard monochrome and RGB, this
generally occurs at 0.2125 fs. This demonstrates that the camera
mechanisms that use a color filter array with multiple primary
filters and multiple secondary filters and a multimodal demosaicing
approach allows a user to control the trade-off between spatial
resolution and radiometric details of the recovered image.
[0093] Additional examples of images generated from a single
captured image are shown in FIGS. 11 and 12. Note that ground truth
images 1110 of FIGS. 11 and 1210 of FIG. 12 are calculated using a
diffraction-limited model of a lens with a f-number of 5.6 and a
1.0 .mu.m pixel size. Image 1120 of FIG. 11 and image 1220 of FIG.
12 show examples of raw images captured using a camera system
having a generalized assorted pixel color filter array or mosaic
with primary filters and secondary filters. Image 1130 of FIG. 11
and image 1230 of FIG. 12 show examples of demosaiced monochrome
images generated using raw image 1120 and 1220, respectively, and
the information obtained from the primary filters. Image 1140 of
FIG. 11 and image 1240 of FIG. 12 show examples of high dynamic
range monochrome images generated using raw image 1120 and 1220,
respectively, and the information obtained from the primary and
secondary filters. Image 1150 of FIG. 11 and image 1250 of FIG. 12
show examples of tri-chromatic (RGB) images generated using raw
image 1120 and 1220, respectively, and the information obtained
from the primary filters. Image 1160 of FIG. 11 and image 1260 of
FIG. 12 show examples of high dynamic range tri-chromatic (RGB)
images generated using raw image 1120 and 1220, respectively, and
the information obtained from the primary and secondary
filters.
[0094] It should be noted that the texture and color of saturated
regions in the monochrome and RGB images become visible in the
corresponding high dynamic range images. As also shown in FIGS. 11
and 12, more detail is shown in the high dynamic range monochrome
image than in the high dynamic range tri-chromatic (RGB) image.
[0095] In addition, FIGS. 11 and 12 show examples of multispectral
images generated using information obtained and calculated from
primary and secondary filters. For example, image 1170 of FIG. 11
and image 1270 of FIG. 12 shows 31-band multispectral images
(400-700 nm, at 10 nm intervals) of several static scenes capturing
by using a tunable filter and a cooled CCD camera. The
corresponding reconstructed spectral reflectance curves 1180 and
1280 show that the reconstructed spectral reflectance (identified
by the dashed line) is substantially similar to the spectral
reflectance of the ground truth image.
[0096] Alternatively, some camera systems can use a different
generalized assorted pixel color filter array to capture a single
image of a scene and control the trade-off between image
resolution, dynamic range, and spectral detail to generate images
of multiple image types.
[0097] For example, FIG. 13 shows an example of an 8.times.8 color
filter array 1300 that includes five different color filters--e.g.,
Green (G), Red (R), Blue (B), Yellow (Y), and Emerald (E). Each
color filter has two exposures e.g., a bright exposure and a dark
exposure, where color (C) denotes a bright pixel and color (C')
denotes a dark pixel. For example, pixel G denotes a bright green
pixel, while pixel G' denotes a dark green pixel.
[0098] As shown in FIG. 13, the bright and dark green channel
samples every two lines in the horizontal and vertical directions
and sample at every line in the diagonal direction. Accordingly,
the horizontal and vertical sampling frequency of bright and dark
green channel is f.sub.HV/2 and the diagonal sampling frequency of
bright and dark green channel is f.sub.D, where f.sub.HV is the
horizontal and vertical sampling frequency and f.sub.D is the
diagonal sampling frequency of the image sensor. In addition, the
horizontal and vertical Nyquist frequency of bright and dark green
channel is half of the sampling frequency or f.sub.HV/4, while the
diagonal Nyquist frequency of bright and dark green channel is
f.sub.D/2. Referring back to FIG. 13, bright and dark red, blue,
yellow, and emerald channels sample every four lines in the
horizontal and vertical directions and sample every two lines in
the diagonal direction. Accordingly, the horizontal and vertical
sampling frequency of bright and dark red, blue, yellow, and
emerald channels is f.sub.HV/4 and the corresponding diagonal
sampling frequency is f.sub.D/2. Thus, the horizontal and vertical
Nyquist frequency of bright and dark red, blue, yellow, and emerald
channels is f.sub.HV/8 and the corresponding diagonal Nyquist
frequency is f.sub.D/4.
[0099] To further illustrate the Nyquist frequencies of color
filter array 1300 of FIG. 13, the Nyquist or usable frequency
region 1400 in the frequency domain is shown in FIG. 14. As shown,
the Nyquist region of the bright and dark green channel is located
in the substantially square area identified by a full line 1410 and
the Nyquist region of the bright and dark red, blue, yellow, and
emerald channels is located in the substantially square area
identified by a dashed line 1420.
[0100] In another example of a color filter array in accordance
with some embodiments of the disclosed subject matter, FIG. 15
shows an example of a 9.times.9 color filter array 1500 that
includes five different color filters e.g., Green (G), Red (R),
Blue (B), Yellow (Y), and Emerald (E).
[0101] As shown in FIG. 15, the green channel samples every line in
the horizontal and vertical directions and samples at every line in
the diagonal direction. Accordingly, the horizontal and vertical
sampling frequency of dark green channels is f.sub.HV and the
diagonal sampling frequency of green channels is f.sub.D. In
addition, the horizontal and vertical Nyquist frequency of green
channels is half of the sampling frequency or f.sub.HV/2, while the
diagonal Nyquist frequency of green channels is f.sub.D/2.
Referring back to FIG. 15, red, blue, yellow, and emerald channels
sample every two lines in the horizontal and vertical directions
and sample every two lines in the diagonal direction. Accordingly,
the horizontal and vertical sampling frequency of red, blue,
yellow, and emerald channels is f.sub.HV/2 and the corresponding
diagonal sampling frequency is f.sub.D/2. Thus, the horizontal and
vertical Nyquist frequency of red, blue, yellow, and emerald
channels is f.sub.HD/4 and the corresponding diagonal Nyquist
frequency is f.sub.D/4.
[0102] The Nyquist frequencies of color filter array 1500 are
further illustrated in FIG. 16. As shown, FIG. 16 shows that the
Nyquist region of the green channel is located in the substantially
diamond area identified by a full line 1610 and the Nyquist region
of the red, blue, yellow, and emerald channels is located in the
substantially diamond area identified by a dashed line 1620.
[0103] Similarly, as described above, multiple image types can be
generated from a single captured image using a camera system with a
color filter array, such as color filter array 1300 of FIG. 13 or
color filter array 1500 of FIG. 15, in accordance with some
embodiments of the disclosed subject matter. For example, the
information obtained from the five color filters with two exposures
of color filter array 1300 of FIG. 13 can be used to generate a
monochrome image, a high dynamic range (HDR) monochrome image, a
tri-chromatic (RGB) image, a HDR RGB image, and a multispectral
image. In another example, the information obtained from the five
color filters of color filter array 1500 of FIG. 15 can be used to
generate multiple types of images, such as a monochrome image and a
tri-chromatic (RGB) image. As also described above, a demosaicing
approach with anti-aliasing can be applied to generate images of
multiple types.
[0104] In some embodiments, using one of color filter arrays 1300
or 1500, a linear regression model of local color distribution can
be used to reduce aliasing effects. For example, it has been
determined that there are strong inter-color correlations at small
local areas (e.g., on a color-changing edge). These local color
distributions in an image can be expressed by the following linear
regression model:
R ^ ij = V GR V GG ( G i , j - M G ) + M R ##EQU00017## where :
##EQU00017.2## M C .ident. exp [ C i , j i , j .di-elect cons.
.OMEGA. ] V C 1 C 2 .ident. exp [ ( C 1 i , j - M C 1 ) ( C 2 i , j
- M C 2 ) i , j .di-elect cons. .OMEGA. ] ##EQU00017.3##
[0105] It should be noted that a pixel at location (i,j) in color
filters arrays 1300 or 1500 can be represented by either
(R.sub.i,j, g.sub.i,j, b.sub.i,j, y.sub.i,j, e.sub.i,j),
(r.sub.i,j, G.sub.i,j, b.sub.i,j, y.sub.i,j, e.sub.i,j),
(r.sub.i,j, g.sub.i,j, B.sub.i,j, y.sub.i,j, e.sub.i,j),
(r.sub.i,j, g.sub.i,j, b.sub.i,j, Y.sub.i,j, e.sub.i,j), or
(r.sub.i,j, g.sub.i,j, b.sub.i,j, y.sub.i,j, E.sub.i,j), where
R.sub.i,j, B.sub.i,j, Y.sub.i,j, and E.sub.i,j denote the known
red, green, blue, yellow, and emerald components of the color
filter array and r.sub.i,j, g.sub.i,j, b.sub.i,j, y.sub.i,j,
e.sub.i,j denote the unknown components of the color filter array.
In addition, it should also be noted that the estimates of
r.sub.i,j, g.sub.i,j, b.sub.i,j, y.sub.i,j, e.sub.i,j are denoted
as {circumflex over (R)}.sub.i,j, G.sub.i,j, {circumflex over
(B)}.sub.i,j, .sub.i,j, and E.sub.i,j.
[0106] The resulting Fourier transforms of V.sub.GR and M.sub.R are
as follows:
M R F R .omega. ( 0 ) ##EQU00018## V GR F .intg. - .infin. .infin.
G .omega. ( .omega. ) R .omega. ( .omega. ) .omega.
##EQU00018.2##
Using these expressions, the aliasing of R can be estimated.
[0107] In some embodiments, using one of color filter arrays 1300
or 1500, directional smoothing can be used to reduce aliasing
effects. For example, to reduce the computational cost of
anti-aliasing, directional smoothing can be used when the local
statistics (e.g., V.sub.GG, V.sub.GR, M.sub.G, and M.sub.R) are
calculated. As shown in FIG. 17, one-dimensional smoothing along a
direction to the local area of the color filter array 1710 is
applied at 1720 to obtain one-dimensional signals of colors 1730.
Then, an anti-aliasing approach is applied to the one-dimensional
color signals 1730 at 1740. After anti-aliasing, color data for
each phase is obtained at 1750. Local statistics (e.g., V.sub.GG,
V.sub.GR, M.sub.G, and M.sub.R) can then be calculated at 1760
using the anti-aliased one-dimensional color data.
[0108] It should be noted that the directional smoothing approach
can be applied in any suitable direction. For example, the
smoothing approach can be applied in the horizontal, vertical,
right-ascending diagonal (), and right-descending diagonal
direction (). It should also be noted that the direction of
smoothing can be selected based at least in part on the direction
of the local texture (e.g., horizontal smoothing for horizontal
stripes).
[0109] In some embodiments, the directional smoothing approach for
several directions (e.g., horizontal, vertical, right-ascending
diagonal, and right-descending diagonal direction) is performed and
anti-aliasing, computing local statistics, and output color
interpolations are also performed for each direction. By measuring
magnitudes of the gradient and local color variance of the
anti-aliased one-dimensional signals, residual aliasing for each
direction can be evaluated. In some embodiments, the direction that
provides the smallest residual aliasing can be selected as the
suitable direction of the interpolation filter.
[0110] FIG. 18 shows a portion of an original image 1810 and
multiple images 1820, 1830, 1840, and 1850 generated from a single
captured image of the original image 1810 in accordance with some
embodiments. Using a camera system with a generalized assorted
pixel color filter array or mosaic, such as color filter array 1300
of FIG. 13, having five different color filters, each having two
exposures (e.g., a bright exposure and a dark exposure) to capture
an image and the anti-aliasing approach described above, multiple
image types e.g., a monochrome image 1820, a tri-chromatic (RGB)
image 1830, a high dynamic range (HDR) monochrome image 1840, and a
HDR RGB image 1850 can be generated. It should be noted that, by
sacrificing spatial resolution, the quality of the spectrum and the
dynamic range can be improved.
[0111] In some embodiments, hardware used in connection with the
camera mechanisms can include an image processor, an image capture
device (that includes a generalized assorted pixel color filter
array, such as the one in FIG. 4), and image storage. The image
processor can be any suitable device that can process images and
image-related data as described herein. For example, the image
processor can be a general purpose device such as a computer or a
special purpose device, such as a client, a server, an image
capture device (such as a camera, video recorder, scanner, mobile
telephone, personal data assistant, etc.), etc. It should be noted
that any of these general or special purpose devices can include
any suitable components such as a processor (which can be a
microprocessor, digital signal processor, a controller, etc.),
memory, communication interfaces, display controllers, input
devices, etc. The image capture device can be any suitable device
for capturing images and/or video, such as a portable camera, a
video camera or recorder, a computer camera, a scanner, a mobile
telephone, a personal data assistant, a closed-circuit television
camera, a security camera, an Internet Protocol camera, etc. The
image capture device can include the generalized assorted pixel
color filter array as described herein. The image storage can be
any suitable device for storing images such as memory (e.g.,
non-volatile memory), an interface to an external device (such as a
thumb drive, a memory stick, a network server, or other storage or
target device), a disk drive, a network drive, a database, a
server, etc.
[0112] Accordingly, generalized assorted pixel camera systems and
methods are provided.
[0113] Although the invention has been described and illustrated in
the foregoing illustrative embodiments, it is understood that the
present disclosure has been made only by way of example, and that
numerous changes in the details of implementation of the invention
can be made without departing from the spirit and scope of the
invention, which is only limited by the claims which follow.
Features of the disclosed embodiments can be combined and
rearranged in various ways.
* * * * *