U.S. patent application number 13/160775 was filed with the patent office on 2011-10-06 for advanced noise reduction in digital cameras.
This patent application is currently assigned to Zoran Corporation. Invention is credited to Noam Korem, Victor Pinto, Dudi Vakrat.
Application Number | 20110242371 13/160775 |
Document ID | / |
Family ID | 40072032 |
Filed Date | 2011-10-06 |
United States Patent
Application |
20110242371 |
Kind Code |
A1 |
Vakrat; Dudi ; et
al. |
October 6, 2011 |
ADVANCED NOISE REDUCTION IN DIGITAL CAMERAS
Abstract
A noise reduction apparatus is presented that includes groups of
one or more serially connected non-linear filter units. Each of the
filter unit groups are driven by decimated input image data at a
different level of decimation and the output of at least one of
these groups serves as one of a plurality of inputs to another
group driven at a different decimation level.
Inventors: |
Vakrat; Dudi; (Nethanya,
IL) ; Korem; Noam; (Timrat, IL) ; Pinto;
Victor; (Zychron-Yaakov, IL) |
Assignee: |
Zoran Corporation
Sunnyvale
CA
|
Family ID: |
40072032 |
Appl. No.: |
13/160775 |
Filed: |
June 15, 2011 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
11754202 |
May 25, 2007 |
7983503 |
|
|
13160775 |
|
|
|
|
Current U.S.
Class: |
348/241 ;
348/607; 348/E5.077; 348/E5.078 |
Current CPC
Class: |
G06T 2207/20192
20130101; H04N 5/217 20130101; H04N 5/23229 20130101; G06T 5/002
20130101 |
Class at
Publication: |
348/241 ;
348/607; 348/E05.077; 348/E05.078 |
International
Class: |
H04N 5/217 20110101
H04N005/217; H04N 5/21 20060101 H04N005/21 |
Claims
1. An image noise reduction apparatus comprising groups of one or
more serially connected non-linear filter units; wherein each of
said filter unit groups are driven by decimated input image data at
a different level of decimation; and wherein the output of at least
one of said groups serves as one of a plurality of inputs to
another group driven at a different decimation level.
2. The noise reduction apparatus of claim 1, wherein the filter
units are implemented in hardware.
3. The noise reduction apparatus of claim 1, wherein the filter
units are implemented in software.
4. The noise reduction apparatus of claim 1, wherein the filter
units are edge preserving filters.
5. The noise reduction apparatus of claim 1, wherein the filter
units are sigma class filters.
6. The noise reduction apparatus of claim 1, wherein each of said
filter units use the same filtering algorithm.
7. The noise reduction apparatus of claim 1, wherein the input
image data is image data from a digital camera.
8. The noise reduction apparatus of claim 1, wherein the input
image data is image data from a video camera.
9. The noise reduction apparatus of claim 1, further comprising: a
first non-linear filter element connectable to be driven by said
input image data, filter the image data, and determine an
indication of uniformity of the image data; and a blend module
connectable to receive the filtered image data from the first
non-linear filter element, the indication of uniformity, and an
interpolated output of one of said groups and form a combination of
the filtered image data from the first non-linear filter element
and the interpolated output of said one of said groups, wherein the
combination is responsive to the indication of uniformity.
10. The noise reduction apparatus of claim 9, wherein the
combination is a linear combination.
11. The noise reduction apparatus of claim 9, wherein the
indication of uniformity of the image data around a pixel is
determined for an N by N neighborhood of the pixel, where N is an
odd integer.
12. The noise reduction apparatus of claim 1, wherein said another
group driven at a different decimation level further includes a
blend module that forms a combination of the output of the last of
the serially connected filter units therein and an interpolated
version of said output of at least one of said groups.
13. A method of performing a noise reduction operation on input
image data, comprising: decimating the input image data to a
plurality of different levels of decimation; for each level of
decimation, filtering the decimated image data by a group of one or
more serially connected non-linear filter units; and providing the
output of at least of one said groups as one of a plurality of
inputs to another of the groups at a different level of
decimation.
14. The method of claim 13, wherein the filter units are
implemented in hardware.
15. The method of claim 13, wherein the filter units are
implemented in software.
16. The method of claim 13, wherein the filter units are edge
preserving filters.
17. The method of claim 13, wherein the filter units are sigma
class filters.
18. The method of claim 13, wherein each of said filter units use
the same filtering algorithm.
19. The method of claim 13, wherein the input image data is image
data from a digital camera.
20. The method of claim 13, wherein the input image data is image
data from a video camera.
21. The method of claim 13, further comprising: filtering the input
data image with a first non-linear filter element; determining an
indication of uniformity of the image data; and forming a
combination of the filtered image data from the first non-linear
filter element and the interpolated output of said one of said
groups, wherein the combination is responsive to the indication of
uniformity.
22. The method of claim 21, wherein the combination is a linear
combination.
23. The method of claim 21, wherein the indication of uniformity of
the image data around a pixel is determined for an N by N
neighborhood of the pixel, where N is an odd integer.
24. The method of claim 13, further comprising: interpolating said
output of at least of one said groups; and forming a combination of
the output of the last of the serially connected filter units of
said another of the groups at a different level and the
interpolated output of at least one of said groups.
Description
[0001] This is a continuation of U.S. patent application Ser. No.
11/754,202, filed on May 25, 2007, which is incorporated herein by
reference.
BACKGROUND
[0002] This application relates to image processing in digital
cameras and other electronic digital image acquisition devices, and
particularly to techniques of improving noise reduction techniques
for such images.
[0003] Images obtained by digital cameras and other imaging systems
contain random noise, which typically grows stronger as the ISO
sensitivity grows higher. Noise reduction in digital cameras is
becoming increasingly important and problematic because of several
trends in the digital camera market which result in lower Signal to
Noise Ratios (SNR), including the increasing of sensor resolution
by reducing the pixel size and the providing better image quality
at higher ISO sensitivities, which enables capture of images in low
light conditions.
[0004] Prior art approaches typically effect noise reduction by
either applying edge preserving filters on the image or suppressing
chromatic components. Applying edge-preserving filters on the
image, such as median filters, bilateral filters and others, are
well known in the art. The difficulty encountered with these
methods is that the size of the filter required for an effective
noise reduction grows in proportion to the amount of noise in the
image. However, the size of the filters is usually limited in order
to save hardware costs, and software implementations tend to incur
too much time and processing power to be practical. Suppressing the
chromatic components of the pixels to zero in dark or gray areas
reduces the chromatic component of the noise in these areas. The
difficulty encountered using this method is that it affects only
dark/gray areas, and it is also very likely to suppress real colors
in the image. A seminal article on aspects of noise reduction in
imagery and using sigma filters for this purpose is given in
"Digital Image Smoothing and the Sigma Filter", Lee, J. S.,
Computer Vision, Graphics, and Image Processing, 24, 255-269,
1983.
[0005] These various prior art methods tend to have a number of
shortcomings when it comes to implementation in digital cameras,
video, and other imaging systems. There will always be noise when
an image is captured in low light conditions. The noise level will
increase as the sensor pixel size is decreased due to sensor
resolution issues and due to a trend to reduce sensor cost.
Therefore, there is substantial room for improvements in digital
imaging systems, even when considering future changes in the
application environment.
SUMMARY
[0006] The described methods and corresponding apparatus provide
ways to achieve superior image quality as compared to previous
noise reduction approaches. A noise reduction apparatus is
presented that includes groups of one or more serially connected
non-linear filter units. Each of the filter unit groups are driven
by decimated input image data at a different level of decimation
and the output of at least one of these groups serves as one of a
plurality of inputs to another group driven at a different
decimation level.
[0007] Various aspects, advantages, features and embodiments of the
present invention are included in the following description of
exemplary examples thereof, which description should be taken in
conjunction with the accompanying drawings. All patents, patent
applications, articles, other publications, documents and things
referenced herein are hereby incorporated herein by this reference
in their entirety for all purposes. To the extent of any
inconsistency or conflict in the definition or use of terms between
any of the incorporated publications, documents or things and the
present application, those of the present application shall
prevail.
BRIEF DESCRIPTION OF THE DRAWINGS
[0008] FIG. 1 illustrates an example of an edge preserving noise
reduction filter building block;
[0009] FIG. 2 is a block diagram of one configuration ("increasing
depth" mode) of filter kernels;
[0010] FIG. 3 is a block diagram of a second configuration
("increasing width" mode) of filter kernels; and
[0011] FIG. 4 shows a generalized filter array combining the width
and depth modes.
DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS
Overview
[0012] In order to achieve the desired results, preferred
embodiments described here employ a matrix of small noise reduction
filters working on decimated (downscaled) representations of the
image being acquired and performing in concert to achieve
performance equivalent to a much larger filter. These offer a
comprehensive solution, which can be implemented in hardware (HW)
in the exemplary embodiments, and which includes decimation along
with small size non-linear low pass filters. The results obtained
can achieve quality such as that obtained by applying a much larger
filter, with only a minor penalty in motion image frame rates or
silicon area.
[0013] The exemplary embodiments provide a comprehensive solution
for digital cameras, video, and imaging systems that provides
effective noise reduction in an image acquired using a high gain
analog signal path between the imaging system's sensor and its
analog to digital (A/D) converter. Images acquired under these
conditions are often call "high ISO images". In this imaging
environment, the various aspects presented in the following provide
results equivalent in quality to those achieved by very large noise
reduction filters, without the cost in frame-rate and/or
DRAM-bandwidth that is usually incurred when such filters are
employed.
[0014] A particular advance in the state of the art can be stated
as the technique of using an array of filters (of any appropriate
type), at various levels of decimation (downscaling to lower
resolution), and the use of selective blending of resealed,
filtered image data, dependent on the uniformity of the local
region of the image being processed.
[0015] These techniques provide effective noise removal and higher
image quality for high ISO images, with minor costs in required
hardware, frame rates and DRAM bandwidth. It allows the use of
sensors and/or an analog front end with lower SNR qualities, such
as CMOS sensors, or sensors with higher resolution and smaller
pixel size. Further, it allows further increase in the ISO
sensitivities in camera products, and increases the ability of
cameras to capture images in low light conditions.
Exemplary Filter Arrangements
[0016] The described techniques are based on successive decimation
of the neighborhood of the current pixel, edge preserving (or, more
generally, non-linear) filtration of the decimated pixel data, and
interpolation and blending of the various decimated image data to
achieve superior noise reduction without incurring excessive cost
in terms of hardware, processing power or time. The implementation
described here is based on a hardware solution; however, it will be
clear to one skilled in the art that the same concepts can be
implemented in software, firmware, or a combination of these
running on an appropriate signal processor, with associated memory
and peripheral hardware.
[0017] A basic component of the described noise reduction system is
the Edge Preserving Noise Reduction Filter/Blend building block
101. FIG. 1 is a block diagram of this component. It consists of
two elements: The Noise Reduction Filter 111, and a Blend Module
121. The Noise Reduction Filter 111 selectively removes noise from
the image while preserving, as much as possible, the edge detail in
the image. Additionally, the filter creates a metric, called
"flatness_measure(x,y)", which is a measure of the relative amount
of uniformity in the current region of the image. In response to
this measure, the Blend Module 121 melds (at 127) the current noise
filtered pixel with image data impressed on its secondary input (at
the bottom of the module, labeled `upscaled_pix(x,y)`). The
exemplary embodiment forms a linear combination weighted (at 123
and 125) by the "flatness_measure(x,y)". The operation of an
exemplary Noise Reduction Filter is described in greater detail
later in this discussion. Those skilled in the art will recognize
that any Noise Reduction Filter 111 that preserves edges and also
provides a suitable metric of local image uniformity may be used in
place of the filter described herein.
[0018] At each pixel location in the image, the current pixel is
cleansed of noise by examination and processing of the current
region's pixel data (neighborhood). The neighborhood is defined in
the exemplary embodiments as the pixels in an N.times.N square of
data surrounding the current pixel, where N is a small odd integer
(five, for example).
[0019] The enhanced noise reduction operation of the system is
achieved by configuring an array of the filter kernels, using
decimation (downscaling), filtering, interpolation (upscaling) of
the filtered pixel data and blending the upscaled data with higher
resolution data. Several configurations are possible, with some
examples described with respect to FIGS. 2-4.
[0020] FIG. 2 details one such configuration, dubbed "Increasing
Depth" mode here. In this mode, the full scale pixel data is
decimated to several lower levels of resolutions (at 205, 215,
225), and at each level the decimated data is filtered and then
selectively blended (at 201, 211, 221) with interpolated (upscaled
at 207, 217, 227) data from the immediately lower resolution,
depending on the degree of edge activity detected at each
resolution. Any number of resolution levels can be used as
indicated by the dotted line extensions. The number of resolution
levels will be limited only by the selection of N, the size of the
neighborhood, as this determines the number of times the input can
be downscaled. (Additional detail on upscaling and downscaling can
found, for example, in U.S. Pat. No. 6,937,772, which uses them in
a differing filtering arrangement.) The amount of upscaling at a
given level's interpolation will typically be the same as the
amount of downscaling at that level's decimation.
[0021] FIG. 3 details another possible configuration, dubbed
"Increasing Width" mode here. In this mode, no blending is
performed along the bottom row, with only the Noise Reduction
Filter portion of the modules being used. The width mode may have
any number of instances of the filter kernels in series, as
indicated by the dotted line extensions. FIG. 3 shows two such
kernels, 201a and 201b, where any additional kernels would be
placed in between. The first of these, 201a, receives the input
decimated at 205 and the last (or, here second) kernel 201b then
has its input upscaled at 207 to serve as the second input of the
top layer filter/blend module 101. Because the Noise Reduction
Filters are edge preserving, repeated applications of the same
filter on the same data set will provide more and more refined
estimates of the noise free value of the current pixel. Note that
as the edge preserving filter stages are non-linear, using multiple
such filters serially will act differently than just a single such
stage with more aggressive filtering. In "Width" mode processing,
the upscaled_pix(x,y) input to the Filter/Blend module is unused
and unconnected.
[0022] More generally, the system can use both "Width" and "Depth"
modes simultaneously, as diagrammed in FIG. 4. This functionally
connects the filter elements as an array with a depth of several
rows, where each row after the first uses the "width" mode
arrangement. (In FIG. 4, as well in the other figures, the
arrangement of elements is schematic to show the functional
relationships among the various elements, whether these are
implemented in software or hardware. Thus, even in a hardware
implementation, the various elements need not be physically
arranged as shown.) In this generalized architecture, several
levels of decimation are provided, which, at each level, are
subjected to Noise Reduction filters without Blending ("Width" mode
processing), after which upscaled versions of the resulting image
data are blended with the immediately higher resolution image data
("Depth" mode processing). In this arrangement, each level has
multiple filter elements (201i, 211i, 221i) connected in series,
where the first of the series receives in input decimated (at 205,
215, 225) with respect to the level above, and supplies the output
of the last of the series (interpolated at 207, 217, 227) to the
last of the series in the level above. This matrix of filters is
limited only by the size of the neighborhood considered, and the
processing power and time available. Although the exemplary
embodiment shows the same number of elements in each row after the
first single filter row, more generally this need not be the
case.
[0023] Consequently, it can be seen that the noise reduction filter
will have groups of one or more non-linear filter modules connected
serially, with each of these groups driven by the input image data
decimated to a different level of downscaling. The outputs, for at
least one of the groups, will then serve as one of the inputs to
another of the groups at a different decimation level.
Filter/Blend Module Operation
[0024] For each Input Image pixel in x,y coordinates, Pin(x,y), an
exemplary embodiment of the Edge Preserving Noise Reduction
Filter/Blend Module performs a series of operations. The first of
these is using input image pixels in an N.times.N environment
around Pin(x,y) to perform edge preserving low pass filtering. The
result is Pin_filtered(x,y). N is typically a small integer number.
The pseudo-code description below gives the operation of an
exemplary filter.
[0025] Also using the input image in an N.times.N environment
around Pin(x,y), the uniformity metric which indicates the flatness
of the environment around Pin(x,y) is calculated. The result is
Flatness_measure(x,y). N is typically a small odd integer number
(five, for example).
[0026] In the instance that the Filter/Blend module also receives
an up-scaled decimated image ("Depth" mode operation), the module
blends Pin_filtered(x,y) with the equivalent pixel from the
Up-scaled Decimated Image, Upscaled_pix(x,y), according to the
uniformity metric, Flatness_measure(x,y), to get the Output Image
pixel, Pout(x,y):
Pout(x,y)=Flatness_measure(x,y)*Upscaled_pix(x,y)+(1-Flatness_measure(x,-
y))*Pin_filtered(x,y)
Flatness_measure(x,y) is a value within the range of [0,1]. Around
edges it tends toward 0, resulting in milder filtering. In flat
areas of the Input Image, it is pushed towards higher values closer
to 1, resulting in more aggressive filtering.
[0027] When the imaging system also incorporates dynamic range
compensation, such as described, for example, in U.S. patent
application Ser. No. 11/752,797, the resultant gain factors for the
pixels of the image data may also be taken into account during the
noise reduction process. Such a combined approach, and its
advantages, are described in a US patent application entitled
"Dynamic Range Compensation-Dependent Noise Reduction" by Dudi
Vakrat, filed concurrently with the present application, whose
teachings can be advantageously combined with the noise reduction
techniques described here.
An Exemplary Edge Preserving Noise Reduction Filter Module
[0028] This section presents a pseudo-code description of the
operation of an exemplary non linear filter module. The filter
described is of the class of sigma filters (see, for example,
"Digital Image Smoothing and the Sigma Filter", Lee, J. S.,
Computer Vision, Graphics, and Image Processing 24, 255-269, 1983);
however, those skilled in the art will recognize that any noise
reducing filter that also provides an estimate of the local
region's spatial uniformity may be used for this function.
[0029] In this description, the constant "T" is a threshold that
indicates the noise levels of the camera's sensor and changes
according to the ISO sensitivity for each possible shutter setting.
For a given camera and a given ISO setting, the camera manufacturer
can calculate T by, for example, capturing a color chart and then
determining the maximum value of abs(Pin[x,y]-Pin[x+n,y+m]) over
all the flat areas in the image (areas without edges or
transitions).
[0030] In order to execute an N.times.N filter (where N is an odd
integer) on each input image pixel in the stream Pin(x,y), an
exemplary embodiment performs the following:
TABLE-US-00001 { Sum = 0 Count = 0 for (m= -(N-1)/2;
m<=(N-1)/2;m++) { for (n= -(N-1)/2; n<= (N-1);n++) { if
(abs(Pin(x,y) - Pin(x+n,y+m)) < T) { Sum = Sum + Pin(x+n,y+m)
Count = Count + 1 } } } If (filtering is performed)
Pin_filtered(x,y) = Sum/Count Else Pin_filtered(x,y) = Pin(x,y)
Flatness_measure(x,y) = Count/(N.sup.2) }
Although not shown in the above, the algorithm may also incorporate
gain factors from a dynamic range compensation process on a pixel
by pixel basis, as described in the US patent application entitled
"Dynamic Range Compensation-Dependent Noise Reduction" by Dudi
Vakrat, filed concurrently with the present application, as noted
above.
Implementation
[0031] In one preferred embodiment of the invention, the matrix of
filter/blend operations is achieved by successive iterations of a
single Edge Preserving Noise Reduction Filter/Blend Module
described above with respect to FIG. 1. In order to implement the
generalized filter array described in FIG. 4 above, the system
could proceed with the following sequence of actions: [0032] 1.
Decimate the full-scale image successively to the depth of
resolution desired, storing each decimated image individually.
[0033] 2. Beginning at the lowest resolution, apply the Noise
Reduction Filter successively, without the blending function
("Width" mode processing), to the desired number of iterations.
[0034] 3. Upscale the current resolution to the next higher level
and store it. [0035] 4. At the next higher resolution, apply the
Noise Reduction Filter successively, without the blending function,
to the desired number of iterations. [0036] 5. Blend the resulting
image with the upscaled image from the next lower resolution level.
[0037] 6. Repeat steps 3, 4, and 5 until a full-scale image is
achieved.
CONCLUSION
[0038] Although the various aspects of the present invention have
been described with respect to exemplary embodiments thereof, it
will be understood that the present invention is entitled to
protection within the full scope of the appended claims.
* * * * *