U.S. patent application number 13/975386 was filed with the patent office on 2014-03-20 for image acquisition method and apparatus.
This patent application is currently assigned to DigitalOptics Corporation Europe Limited. The applicant listed for this patent is DigitalOptics Corporation Europe Limited. Invention is credited to Petronel Bigioi, Peter Corcoran, Alexandru Drimbarean, Corneliu Florea, Eran Steinberg, Adrian Zamfir.
Application Number | 20140078322 13/975386 |
Document ID | / |
Family ID | 38566236 |
Filed Date | 2014-03-20 |
United States Patent
Application |
20140078322 |
Kind Code |
A1 |
Corcoran; Peter ; et
al. |
March 20, 2014 |
Image Acquisition Method and Apparatus
Abstract
An image acquisition sensor of a digital image acquisition
apparatus is coupled to imaging optics for acquiring a sequence of
images. Images acquired by the sensor are stored. A motion detector
causes the sensor to cease capture of an image when the degree of
movement in acquiring the image exceeds a threshold. A controller
selectively transfers acquired images for storage. A motion
extractor determines motion parameters of a selected, stored image.
An image re-constructor corrects the selected image with associated
motion parameters. A selected plurality of images nominally of the
same scene are merged and corrected by the image re-constructor to
produce a high quality image of the scene.
Inventors: |
Corcoran; Peter;
(Claregalway, IE) ; Steinberg; Eran; (San
Francisco, CA) ; Bigioi; Petronel; (Galway, IE)
; Drimbarean; Alexandru; (Galway, IE) ; Zamfir;
Adrian; (Bucuresti, RO) ; Florea; Corneliu;
(Bucuresti, RO) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
DigitalOptics Corporation Europe Limited |
Galway |
|
IE |
|
|
Assignee: |
DigitalOptics Corporation Europe
Limited
Galway
IE
|
Family ID: |
38566236 |
Appl. No.: |
13/975386 |
Filed: |
August 26, 2013 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
12907921 |
Oct 19, 2010 |
8520082 |
|
|
13975386 |
|
|
|
|
11753098 |
May 24, 2007 |
8169486 |
|
|
12907921 |
|
|
|
|
60892880 |
Mar 5, 2007 |
|
|
|
60803980 |
Jun 5, 2006 |
|
|
|
Current U.S.
Class: |
348/208.2 |
Current CPC
Class: |
H04N 5/23248 20130101;
H04N 5/23277 20130101; H04N 5/23258 20130101; H04N 5/23251
20130101 |
Class at
Publication: |
348/208.2 |
International
Class: |
H04N 5/232 20060101
H04N005/232 |
Claims
1. A digital image acquisition apparatus, comprising: an image
acquisition sensor coupled to imaging optics for acquiring a
sequence of images; an image store for storing one or more of said
sequence of images acquired by said sensor; a dedicated motion
detector hardware unit for providing hardware-based sensor control
including causing said sensor to cease capture of an image when the
degree of movement of the apparatus in acquiring said image exceeds
a threshold, and for selectively transferring said one or more of
said images acquired by said sensor to said image store; and a
dedicated motion extractor hardware unit for providing
hardware-based determination of motion parameters of at least one a
selected image stored in said image store.
Description
PRIORITY
[0001] This application is a Continuation of U.S. patent
application Ser. No. 12/907,921, filed Oct. 19, 2010; which is a
continuation of U.S. patent application Ser. No. 11/753,098, filed
May 24, 2007, now U.S. Pat. No. 8,169,486; which claims the benefit
of priority under 35 USC .sctn.119 to U.S. provisional patent
application No. 60/803,980, filed Jun. 5, 2006; and to U.S.
provisional patent application No. 60/892,880, filed Mar. 5, 2007;
which are incorporated by reference.
[0002] The present invention relates to an image acquisition method
and apparatus, in particular, the invention addresses the problems
of acquisition device or subject movement during image
acquisition.
BACKGROUND OF THE INVENTION
[0003] The approach to restoring an acquired image which is
degraded or unclear either due to acquisition device or subject
movement during image acquisition, divides in two categories:
[0004] Deconvolution where an image degradation kernel, for
example, a point spread function (PSF) is known; and [0005] Blind
deconvolution where motion parameters are unknown.
[0006] Considering blind deconvolution (which is the most often
case in real situations), there are two main approaches: [0007]
identifying motion parameters, such as PSF separately from the
degraded image and using the motion parameters later with any one
of a number of image restoration processes; and [0008]
incorporating the identification procedure within the restoration
process. This involves simultaneously estimating the motion
parameters and the true image and it is usually done
iteratively.
[0009] The first blind deconvolution approach is usually based on
spectral analysis. Typically, this involves estimating the PSF
directly from the spectrum or Cepstrum of the degraded image. The
Cepstrum of an image is defined as the inverse Fourier transform of
the logarithm of the spectral power of the image. The PSF (point
spread function) of an image may be determined from the cepstrum,
where the PSF is approximately linear. It is also possible to
determine, with reasonable accuracy, the PSF of an image where the
PSF is moderately curvilinear. This corresponds to even motion of a
camera during exposure. It is known that a motion blur produces
spikes in the Cepstrum of the degraded image.
[0010] So, for example, FIG. 5a shows an image of a scene
comprising a white point on a black background which has been
blurred to produce the PSF shown. (In this case, the image and the
PSF are the same, however, it will be appreciated that for normal
images this is not the case.) FIG. 5b shows the log of the spectrum
of the image of FIG. 5a, and this includes periodic spikes in
values in the direction 44 of the PSF. The distance from the center
of spectrum to the nearest large spike value is equal to the PSF
size. FIG. 5c shows the Cepstrum of the image, where there is a
spike 40 at the centre and a sequence of spikes 42. The distance
between the center 40 and the first spike 42 is equal to the PSF
length.
[0011] Techniques, for example, as described at M. Cannon "Blind
Deconvolution of Spatially Invariant Image Blurs with Phase"
published in IEEE Transactions on Acoustics, Speech, and Signal
Processing, Vol. ASSP-24, NO. 1, February 1976 and refined by R. L.
Lagendijk, J. Biemond in "Iterative Identification and Restoration
of Images", Kluwer Academic Publishers, 1991 involve searching for
those spikes in a Cepstrum, estimating the orientation and
dimension of the PSF and, then, reconstructing the PSF from these
parameters. This approach is fast and straight-forward, however,
good results are usually generally achieved only for uniform and
linear motion or for out of focus images. This is because for
images subject to non-uniform or non-linear motion, the largest
spikes are not always most relevant for determining motion
parameters.
[0012] A second blind deconvolution approach involves iterative
methods, convergence algorithms, and error minimization techniques.
Usually, acceptable results are only obtained either by restricting
the image to a known, parametric form (an object of known shape on
a dark background as in the case of astronomy images) or by
providing information about the degradation model. These methods
usually suffer from convergence problems, numerical instability,
and extremely high computation time and strong artifacts.
[0013] A CMOS image sensor may be built which can capture multiple
images with short exposure times (SET images) as described in "A
Wide Dynamic Range CMOS Image Sensor with Multiple Short-Time
Exposures", Sasaki et al, IEEE Proceedings on Sensors, 2004, 24-27
Oct. 2004 Page(s):967-972 vol. 2.
[0014] Multiple blurred and/or undersampled images may be combined
to yield a single higher quality image of larger resolution as
described in "Restoration of a Single Superresolution Image from
Several Blurred, Noisy and Undersampled Measured Images", Elad et
al, IEEE Transactions on Image Processing, Vol. 6, No. 12, December
1997.
SUMMARY OF THE INVENTION
[0015] A digital image acquisition apparatus is provided. An image
acquisition sensor is coupled to imaging optics for acquiring a
sequence of images. An image store is for storing images acquired
by the sensor. A motion detector is for causing the sensor to cease
capture of an image when a degree of movement in acquiring the
image exceeds a threshold. A controller selectively transfers the
image acquired by the sensor to the image store. A motion extractor
determines motion parameters of a selected image stored in the
image store. An image re-constructor corrects a selected image with
associated motion parameters. An image merger is for merging
selected images nominally of the same scene and corrected by the
image re-constructor to produce a high quality image of the
scene.
[0016] The motion extractor may be configured to estimate a point
spread function (PSF) for the selected image. The motion extractor
may be configured to calculate a Cepstrum for the selected image,
identify one or more spikes in the Cepstrum, and select one of the
spikes in the Cepstrum as an end point for the PSF. The extractor
may be configured to calculate a negative Cepstrum, and to set
points in the negative Cepstrum having a value less than a
threshold to zero. The image store may include a temporary image
store, and the apparatus may also include a non-volatile memory.
The image merger may be configured to store the high quality image
in the non-volatile memory.
[0017] The motion detector may include a gyro-sensor or an
accelerometer, or both.
[0018] A further digital image acquisition apparatus is provided.
An image acquisition sensor is coupled to imaging optics for
acquiring a sequence of images. An image store is for storing
images acquired by said sensor. A motion detector causes the sensor
to cease capture of an image when the degree of movement in
acquiring the image exceeds a first threshold. One or more
controllers cause the sensor to restart capture when a degree of
movement is less than a given second threshold, and selectively
transfer images acquired by the sensor to the image store. A motion
extractor determines motion parameters of a selected image stored
in the image store. An image re-constructor corrects a selected
image with associated motion parameters. An image merger merges
selected images nominally of the same scene and corrected by the
image re-constructor to produce a high quality image of the
scene.
[0019] A first exposure timer may store an aggregate exposure time
of the sequence of images. The apparatus may be configured to
acquire the sequence of images until the aggregate exposure time of
at least a stored number of the sequence of images exceeds a
predetermined exposure time for the high quality image. A second
timer may store an exposure time for a single image. An image
quality analyzer may analyze a single image. The apparatus may be
configured to dispose of an image having a quality less than a
given threshold quality and/or having an exposure time less than a
threshold time.
[0020] The image merger may be configured to align the images prior
to merging them. The first and second thresholds may include
threshold amounts of motion energy.
[0021] An image capture method with motion elimination is also
provided. An optimal exposure time is determined for the image. A
sequence of consecutive exposures is performed, including:
[0022] (i) exposing intermediate images until either the optimal
exposure time is reached or motion is detected beyond an excessive
movement threshold; and
[0023] (ii) discarding images that have insufficient exposure times
or that exhibit excessive movement;
[0024] (iii) storing non-discarded intermediate images for further
image restoration, including:
[0025] (iv) performing motion de-blurring on non-discarded
intermediate images;
[0026] (v) calculating a signal to noise ratio and, based on the
calculating, performing exposure enhancement on the non-discarded
images;
[0027] (vi) performing registration between restored intermediate
images;
[0028] (vii) assigning a factor to each of the restored images
based on quality of restoration, signal to noise ratio or overall
exposure time, or combinations thereof; and
[0029] (viii) merging the restored images based on a weighted
contribution as defined by said factor.
[0030] An aggregate exposure time of a sequence of images may be
stored. The sequence of images may be acquired until the aggregate
exposure time of at least a stored number of images exceeds a
predetermined exposure time for a high quality image. An exposure
time may be stored for a single image, and/or an image quality may
be analyzed for a single image. An image may be disposed of that
has an exposure time less than a threshold time and/or a quality
less than a given threshold quality.
[0031] The merging may include aligning each restored image. A
threshold may include a threshold amount of motion energy.
BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Embodiments will now be described by way of example, with
reference to the accompanying drawings, in which:
[0033] FIG. 1 illustrates schematically a digital image acquisition
apparatus according to an embodiment.
[0034] FIGS. 2a-2b illustrate (a) a PSF for a single image and (b)
the PSFs for three corresponding SET images acquired according to
the an embodiment.
[0035] FIGS. 3a-3c illustrate how blurring of partially exposed
images can reduce the amount of motion blur in the image.
[0036] FIG. 4 illustrates the generation of a PSF for an image
acquired in accordance with an embodiment;
[0037] FIGS. 5a-5e illustrate sample images/PSFs and their
corresponding Cepstrums.
[0038] FIG. 6 illustrates an estimate of a PSF constructed
according to an embodiment.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0039] An image acquisition system is provided in accordance with
an embodiment which incorporates a motion sensor and utilizes
techniques to compensate for motion blur in an image.
[0040] One embodiment is a system that includes the following:
[0041] (1) the image acquisition apparatus comprises a imaging
sensor, which could be CCD, CMOS, etc., hereinafter referred to as
CMOS;
[0042] (2) a motion sensor (Gyroscopic, Accelerometer or a
combination thereof);
[0043] (3) a fast memory cache (to store intermediate images);
and
[0044] (4) a real-time subsystem for determining the motion (PSF)
of an image. Such determination may be done in various ways. One
preferred method is determining the PSF based on the image
cepstrum.
[0045] In addition, the system can include a correction component,
which may include: [0046] (a) a subsystem for performing image
restoration based on the motion PSF, [0047] (b) an image merging
subsystem to perform registration of multi-images and merging of
images or part of images [0048] (c) a CPU for directing the
operations of these subsystems.
[0049] In certain embodiments some of these subsystems may be
implemented in firmware and executed by the CPU. In alternative
embodiments it may be advantageous to implement some, or indeed all
of these subsystems as dedicated hardware units. Alternatively, the
correction stage may be done in an external system to the
acquisition system, such as a personal computer that the images are
downloaded to.
[0050] In one embodiment, the ceptrum may include the Fourier
transform of the log-magnitude spectrum:
fFt(ln(|fFt(window.signal)|)).
[0051] In a preferred embodiment the disclosed system is
implemented on a dual-CPU image acquisition system where one of the
CPUs is an ARM and the second is a dedicated DSP unit. The DSP unit
has hardware subsystems to execute complex arithmetical and Fourier
transform operations which provides computational advantages for
the PSF extraction.
[0052] Image Restoration and Image Merging Subsystems
[0053] In a preferred embodiment, when the acquisition subsystem is
activated to capture an image it executes the following
initialization steps: (i) the motion sensor and an associated rate
detector are activated; (ii) the cache memory is set to point to
the first image storage block; (iii) the other image processing
subsystems are reset and (iv) the image sensor is signaled to begin
an image acquisition cycle and (v) a count-down timer is
initialized with the desired exposure time, a count-up timer is set
to zero, and both are started.
[0054] In a given scene an exposure time is determined for optimal
exposure. This will be the time provided to the main exposure
timer. Another time period is the minimal-accepted-partially
exposed image. When an image is underexposed (the integration of
photons on the sensor is not complete) the signal to noise ratio is
reduced. Depending on the specific device, the minimal accepted
time is determined where sufficient data is available in the image
without the introduction of too much noise. This value is empirical
and relies on the specific configuration of the sensor acquisition
system.
[0055] The CMOS sensor proceeds to acquire an image by integrating
the light energy falling on each sensor pixel. If no motion is
detected, this continues until either the main exposure timer
counts down to zero, at which time a fully exposed image has been
acquired. However, in this aforementioned embodiment, the rate
detector can be triggered by the motion sensor. The rate detector
is set to a predetermined threshold. One example of such threshold
is one which indicates that the motion of the image acquisition
subsystem is about to exceed the threshold of even curvilinear
motion which will allow the PSF extractor to determine the PSF of
an acquired image. The motion sensor and rate detector can be
replaced by an accelerometer and detecting a +/-threshold level.
The decision of what triggers the cease of exposure can be made on
input form multiple sensor and or a forumale trading of non-linear
motion and exposure time.
[0056] When the rate detector is triggered then image acquisition
by the sensor is halted. At the same time the count-down timer is
halted and the value from the count-up timer is compared with a
minimum threshold value. If this value is above the minimum
threshold then a useful SET image was acquired and sensor read-out
to memory cache is initiated. The current SET image data may be
loaded into the first image storage location in the memory cache,
and the value of the count-up timer (exposure time) is stored in
association with the image. The sensor is then re-initialized for
another short-time image acquisition cycle, the count-up timer is
zeroed, both timers are restarted and a new image acquisition is
initiated.
[0057] If the count-up timer value is below the minimum threshold
then there was not sufficient time to acquire a valid short-time
exposure and data read-out form the sensor is not initiated. The
sensor is re-initialized for another short-time exposure, the value
in the count-up timer is added to the count-down timer (thus
restoring the time counted down during the acquisition cycle), the
count-up timer is re-initialized, then both timers are restarted
and a new image acquisition is initiated.
[0058] This process repeats itself until in total the exposure
exceeds the needed optimal integration time. If for example in the
second SET image reaches full term of exposure, it will then become
the final candidate, with no need to perform post processing
integration. If however, no single image exceeds the optimal
exposure time, an integration is performed. This cycle of acquiring
another short-time image continues until the count-down timer
reaches zero--in a practical embodiment the timer will actually go
below zero because the last short-time image which is acquired must
also have an exposure time greater than the minimum threshold for
the count-up timer. At this point there should be N short-time
images captured and stored in the memory cache. Each of these
short-time images will have been captured with an curvilinear
motion-PSF. The total sum of N may exceeds the optimal exposure
time, which in this case the "merging system will have more images
or more data to choose from overall.
[0059] After a sufficient exposure is acquired it is now possible
in a preferred embodiment to recombine the separate short-term
exposure images as follows:
[0060] (i) each image is processed by a PSF extractor which can
determine the linear or curvilinear form of the PSF which blurred
the image;
[0061] (ii) the image is next passed onto an image re-constructor
which also takes the extracted PSF as an input; this reconstructs
each short-time image in turn. Depending on the total exposure
time, this image may also go through exposure enhancement which
will increase its overall contribution to the final image. Of
course, the decision whether to boost up the exposure is a tradeoff
between the added exposure and the potential introduction of more
noise into the system. The decision is performed based on the
nature of the image data (highlight, shadows, original exposure
time) as well as the available SET of images altogether. In a
pathological example if only a single image is available that only
had 50% exposure time, it will need to be enhanced to 2.times.
exposure even at the risk of having some noise. If however, two
images exist each with 50% exposure time, and the restoration is
considered well, no exposure will be needed. Finally, the
motion-corrected and exposure corrected images are passed it onto;
and
[0062] (iii) the image merger; the image merger performs local and
global alignment of each short-term image using techniques which
are well-known to those skilled in the arts of super-resolution;
these techniques allow each deblurred short-time image to
contribute to the construction of a higher resolution main
image.
[0063] This approach has several advantages including:
[0064] (1) the number of SET is kept to a minimum; if the motion
throughout an exposure is constant linear or curvilinear motion
then only a single image need be captured;
[0065] (2) the decision of who at images are used to create the
Final image are determined post processing thus enabling more
flexibility in determining the best combination, where the motion
throughout an exposure is mostly regular, but some rapid deviations
appear in the middle the invention will effectively "skip over"
these rapid deviations and a useful image can still be obtained;
this would not be possible with a conventional image acquisition
system which employed super-resolution techniques because the SET
images are captured for a fixed time interval;
[0066] (3) where the image captured is of a time frame that is too
small, this portion can be discarded;
[0067] FIG. 1a shows an Original PSF and FIG. 1b shows partial
PSFs. Referring now to FIG. 1, which illustrates a digital image
acquisition apparatus 100 according to a preferred embodiment of
the present invention, the apparatus 100 comprises a CMOS imaging
sensor 105 coupled to camera optics 103 for acquiring an image.
[0068] The apparatus includes a CPU 115 for controlling the sensor
105 and the operations of sub-systems within the apparatus.
Connected to the CPU 115 are a motion sensor 109 and an image cache
130. Suitable motion sensors include a gyroscopic sensor (or a pair
of gyro sensors) that measures the angular velocity of the camera
around a given axis, for example, as produced by Analog Devices'
under the part number ADXRS401.
[0069] In FIG. 1, a subsystem 131 for estimating the motion
parameters of an acquired image and a subsystem 133 for performing
image restoration based on the motion parameters for the image are
shown coupled to the image cache 130. In the embodiment, the motion
parameters provided by the extractor sub-system 131 comprise an
estimated PSF calculated by the extractor 131 from the image
Cepstrum.
[0070] An image merging subsystem 135 connects to the output of the
image restoration sub-system 133 to produce a single image from a
sequence of one or more de-blurred images.
[0071] In certain embodiments some of these subsystems of the
apparatus 100 may be implemented in firmware and executed by the
CPU; whereas in alternative embodiments it may be advantageous to
implement some, or indeed all of these subsystems as dedicated
hardware units.
[0072] So for example, in a preferred embodiment, the apparatus 100
is implemented on a dual-CPU system where one of the CPUs is an ARM
Core and the second is a dedicated DSP unit. The DSP unit has
hardware subsystems to execute complex arithmetical and Fourier
transform operations, which provides computational advantages for
the PSF extraction 131, image restoration 133 and image merging 135
subsystems.
[0073] When the apparatus 100 is activated to capture an image, it
firstly executes the following initialization steps: [0074] (i) the
motion sensor 109 and an associated rate detector 108 are
activated; [0075] (ii) the cache memory 130 is set to point to a
first image storage block 130-1; [0076] (iii) the other image
processing subsystems are reset; [0077] (iv) the image sensor 105
is signaled to begin an image acquisition cycle; and [0078] (v) a
count-down timer 111 is initialized with the desired exposure time,
a count-up timer 112 is set to zero, and both are started.
[0079] The CMOS sensor 105 proceeds to acquire an image by
integrating the light energy falling on each sensor pixel; this
continues until either the main exposure timer counts 111 down to
zero, at which time a fully exposed image has been acquired, or
until the rate detector 108 is triggered by the motion sensor 109.
The rate detector is set to a predetermined threshold which
indicates that the motion of the image acquisition subsystem is
about to exceed the threshold of even curvilinear motion which
would prevent the PSF extractor 131 accurately estimating the PSF
of an acquired image.
[0080] In alternative implementations, the motion sensor 109 and
rate detector 108 can be replaced by an accelerometer (not shown)
and detecting a +/-threshold level. Indeed any suitable subsystem
for determining a degree of motion energy and comparing this with a
threshold of motion energy could be used.
[0081] When the rate detector 108 is triggered, then image
acquisition by the sensor 105 is halted; at the same time the
count-down timer 111 is halted and the value from the count-up
timer 112 is compared with a minimum threshold value. If this value
is above the minimum threshold then a useful short exposure time
(SET) image was acquired and sensor 105 read-out to memory cache
130 is initiated; the current SET image data is loaded into the
first image storage location in the memory cache, and the value of
the count-up timer (exposure time) is stored in association with
the SET image.
[0082] The sensor 105 is then re-initialized for another SET image
acquisition cycle, the count-up timer is zeroed, both timers are
restarted and a new image acquisition is initiated.
[0083] If the count-up timer 112 value is below the minimum
threshold, then there was not sufficient time to acquire a valid
SET image and data read-out from the sensor is not initiated. The
sensor is re-initialized for another short exposure time, the value
in the count-up timer 112 is added to the count-down timer 111
(thus restoring the time counted down during the acquisition
cycle), the count-up timer is re-initialized, then both timers are
restarted and a new image acquisition is initiated.
[0084] This cycle of acquiring another SET image 130-n continues
until the count-down timer 111 reaches zero. Practically, the timer
will actually go below zero because the last SET image which is
acquired must also have an exposure time greater than the minimum
threshold for the count-up timer 112. At this point, there should
be N short-time images captured and stored in the memory cache 130.
Each of these SET images will have been captured with a linear or
curvilinear motion-PSF.
[0085] FIGS. 2a-2b illustrates Point Spread Functions (PSF). FIG.
2(a) shows the PSF of a full image exposure interval; and FIG. 2(b)
shows how this is split into five SET-exposures by the motion
sensor. In FIG. 2, boxes (1) through (5) are shown, and:
(1) will be used and with the nature of the PSF it has high
probability of good restoration and also potential enhancement
using gain; (2) can be well restored; (3) will be discarded as too
short of an integration period; (4) will be discarded having a
non-curvliniar motion; and (5) can be used for the final image.
[0086] So for example, while a single image captured with a
full-exposure interval might have a PSF as shown in FIG. 2(a), a
sequence of 3 images captured according to the above embodiment,
might have respective PSFs as shown in FIG. 2(b). It will be seen
that the motion for each of these SET image PSFs more readily lends
the associated images to de-blurring than the more complete motion
of FIG. 2(a).
[0087] After a sufficient exposure is acquired, it is now possible
to recombine the separate SET images 130-1 . . . 130-N as follows:
[0088] (i) each image is processed by the PSF extractor 131 which
estimates the PSF which blurred the SET image; [0089] (ii) the
image is next passed onto the image re-constructor 133 which as
well as each SET image takes the corresponding estimated PSF as an
input; this reconstructs each SET image in turn and passes it onto
the image merger 135; [0090] (iii) the image merger 135 performs
local and global alignment of each SET image using techniques which
are well-known to those skilled in the art of super-resolution.
These techniques allow each de-blurred SET image to contribute to
the construction of a higher resolution main image which is then
stored in image store 140. The image merger may during merging
decide to discard an image where it is decided it is detrimental to
the final quality of the merged image; or alternatively various
images involved in the merging process can be weighted according to
their respective clarity. This approach has several benefits over
the prior art: [0091] (i) the number of SET images is kept to a
minimum; if the motion throughout an exposure is constant linear or
curvilinear motion then only a single image needs to be captured;
[0092] (ii) where the motion throughout an exposure is mostly
regular, but some rapid deviations appear in the middle, the
embodiment will effectively "skip over" these rapid deviations and
a useful image can still be obtained. This would not be possible
with a conventional image acquisition system which employed
super-resolution techniques, because the SET images are captured
for a fixed time interval. Although the embodiment above could be
implemented with a PSF extractor 131 based on conventional
techniques mentioned in the introduction, where a PSF involves
slightly curved or non-uniform motion, the largest spikes may not
always be most relevant for determining motion parameters, and so
conventional approaches for deriving the PSF even of SET images
such as shown in FIG. 2(b) may not provide satisfactory
results.
[0093] Thus, in a particular implementation of the present
invention, the PSF extractor 131 rather than seeking spikes in a
Cepstrum, seeks large regions around spikes in the Cepstrum of an
image using a region-growing algorithm. This is performed by
inspecting candidate spikes in the Cepstrum, using region growing
around these candidates and then discriminating between them.
Preferably, the candidate spike of the largest region surrounding a
candidate spike will be the point chosen as the last point of the
PSF.
[0094] It can be seen from FIGS. 3a-3c that blurring the partially
exposed image reduces the amount of motion blur in the image. FIG.
3(a) shows an original image. FIG. 3(b) illustrates blurring with
full PSF. FIG. 3(c) illustrates reconstructed image from 3 SET
images using individual PSFs.
[0095] Referring to FIG. 3, an SET image 130-1 . . . 130-N is
represented in the RGB space (multi-channel) or as a gray-scale
("one-channel"). The Cepstrum may be computed on each color channel
(in the case of multi-channel image) or only on one of them and so,
by default, the Cepstrum would have the size of the degraded image.
In the preferred embodiment, the Fourier transform is performed,
step 32 only on the green channel. It will also be seen that, for
processing simplicity, the results are negated to provide a
negative Cepstrum for later processing.
[0096] In variations of the embodiment, the Cepstrum may be
computed: [0097] on each channel and, afterwards, averaged; or
[0098] on the equivalent gray image.
[0099] After computing the negative Cepstrum, the blurred image 130
is not necessary for the extractor 131 and can be released from
memory or for other processes. It should also be seen that as the
Cepstrum is symmetrical towards its center (the continuous
component), only one half is required for further processing.
[0100] As discussed in the introduction, images which are degraded
by very large movements are difficult to restore. Experiments have
shown that if the true PSF is known, a restored image can have an
acceptable quality where the PSF is smaller than 10% of the image
size. The preferred embodiment ideally only operates on images
subject to minimal movement. Thus, the original image can either be
sub-sampled, preferably to 1/3 of its original size or once the
Cepstrum is computed, it can be sub-sampled before further
processing or indeed during further processing without having a
detrimental affect on the accuracy of the estimated PSF where
movement is not too severe. This can also be considered valid as
the blurring operation may be seen as a low-pass filtering of an
image (the PSF is indeed a low pass filter); and therefore there is
little benefit in looking for PSF information in the high frequency
domain.
[0101] The next step 34 involves thresholding the negative
Cepstrum. This assumes that only points in the negative Cepstrum
with intensities higher than a threshold (a certain percent of the
largest spike) are kept. All the other values are set to zero. This
step has, also, the effect of reducing noise. The value of the
threshold was experimentally set to 9% of the largest spike
value.
[0102] Pixel candidates are then sorted with the largest spike
(excluding the Cepstrum center) presented first as input to a
region-growing step 36, then the second spike and so on.
[0103] The region-growing step 36 has as main input a sequence of
candidate pixels (referred to by location) as well as the Cepstrum
and it returns as output the number of pixels in a region around
each candidate pixel. Alternatively, it could return the identities
of all pixels in a region for counting in another step, although
this is not necessary in the present embodiment. A region is
defined as a set of points with similar Cepstrum image values to
the candidate pixel value. In more detail, the region-growing step
36 operates as follows: [0104] 1. Set the candidate pixel as a
current pixel. [0105] 2. Inspect the neighbors of the current
pixel--up to 8 neighboring pixels may not already be counted in the
region for the candidate pixel or other regions. If the neighboring
pixel meets an acceptance condition, preferably that its value is
larger than 0.9 of the value of the candidate pixel value, then
include it in the region for the candidate pixel, exclude the pixel
from further regions, and increment the region size. [0106] 3. If a
maximum number of pixels, say 128, has been reached, exit [0107] 4.
After finished inspecting neighbors for the current pixel, if there
are still un-investigated pixels, set the first included pixel as
the current pixel and jump to step 2. [0108] 5. If there are no
more un-investigated adjacent pixels, exit.
[0109] As can be seen, each pixel may be included in only one
region. If the region-growing step 36 is applied to several
candidate pixels, then a point previously included in a region will
be skipped when investigating the next regions.
[0110] After comparison of the sizes of all grown regions, step 38,
the pixel chosen is the candidate pixel for the region with the
greatest number of pixels and this selected point is referred to as
the PSF "end point". The PSF "start point" is chosen the center of
the Cepstrum, point 40 in FIG. 4(b)(ii).
[0111] Referring to FIG. 5(d), where the negative Cepstrum has been
obtained from an image, FIG. 5(e) degraded with a non-linear PSF,
there are areas 46', 46'' with spikes (rather than a single spike)
which correspond to PSF turning points 48', 48'', and it is areas
such as these in normal images which the present implementation
attempts to identify in estimating the PSF for an SET image.
[0112] In a continuous space, the estimated PSF would be a
straight-line segment, such as the line 50 linking PSF start and
end points, as illustrated at FIG. 6. In the present embodiment,
the straight line is approximated in the discrete space of the
digital image, by pixels adjacent the straight-line 50 linking the
PSF start and end points, as illustrated at FIG. 6. Thus, all the
pixels adjacent the line 50 connecting the PSF start and end points
are selected as being part of the estimated PSF, step 41. For each
PSF pixel, intensity is computed by inverse proportionality with
the distance from its center to the line 50, step 43. After the
intensities of all pixels of the PSF are computed, a normalization
of these values is performed such that the sum of all non-zero
pixels of the PSF equals 1, step 45.
[0113] Using the approach above, it has been shown that if the type
of movement in acquiring the component SET images of an image is
linear or near linear, then the estimated PSF produced by the
extractor 131 as described above provides good estimate of the
actual PSF for deblurring.
[0114] As the curving of movement increases, during restoration,
ringing proportional to the degree of curving is introduced.
Similarly, if motion is linear but not uniform, restoration
introduces ringing which is proportional with the degree of
non-uniformity. The acceptable degree of ringing can be used to
tune the motion sensor 108 and rate detector 109 to produce the
required quality of restored image for the least number of SET
images.
[0115] Also, if this PSF extractor 131 is applied to images which
have been acquired with more than linear movement, for example,
night pictures having a long exposure time, although not useful for
deblurring, the estimated PSF provided by the extractor 131 can
provide a good start in the determination of the true PSF by an
iterative parametric blind deconvolution process (not shown) for
example based on Maximum Likelihood Estimation, as it is known that
the results of such processes fade if a wrong starting point is
chosen.
[0116] The above embodiment has been described in terms of a CMOS
imaging sensor 105. In alternative implementations, a CCD image
sensor or indeed any another suitable image sensor could be used.
For a CCD, which is typically used with a shutter and which might
normally not be considered suitable for providing the fine level of
control required by the present invention, progressive readout of
an image being acquired should be employed rather than opening and
closing the shutter for each SET image.
[0117] The present invention is not limited to the embodiments
described above herein, which may be amended or modified without
departing from the scope of the present invention as set forth in
the appended claims, and structural and functional equivalents
thereof.
[0118] In methods that may be performed according to preferred
embodiments herein and that may have been described above and/or
claimed below, the operations have been described in selected
typographical sequences. However, the sequences have been selected
and so ordered for typographical convenience and are not intended
to imply any particular order for performing the operations.
[0119] In addition, all references cited above herein, in addition
to the background and summary of the invention sections, as well as
US published patent applications nos. 2006/0204110, 2006/0098890,
2005/0068446, 2006/0039690, and 2006/0285754, and U.S. patent
applications Nos. 60/773,714, 60/803,980, and 60/821,956, which are
to be or are assigned to the same assignee, are all hereby
incorporated by reference into the detailed description of the
preferred embodiments as disclosing alternative embodiments and
components.
[0120] In addition, the following United States published patent
applications are hereby incorporated by reference for all purposes
including into the detailed description as disclosing alternative
embodiments:
US 2005/0219391--Luminance correction using two or more captured
images of same scene. US 2005/0201637--Composite image with motion
estimation from multiple images in a video sequence. US
2005/0057687--Adjusting spatial or temporal resolution of an image
by using a space or time sequence (claims are quite broad) US
2005/0047672--Ben-Ezra patent application; mainly useful for
supporting art; uses a hybrid imaging system with fast and slow
detectors (fast detector used to measure PSF). US
2005/0019000--Supporting art on super-resolution.
US 2006/0098237--Method and Apparatus for Initiating Subsequent
Exposures Based on a Determination of Motion Blurring Artifacts
(and 2006/0098890 and 2006/0098891).
[0121] The following provisional application is also incorporated
by reference: Ser. No. 60/773,714, filed Feb. 14, 2006 entitled
Image Blurring.
* * * * *