U.S. patent application number 11/467044 was filed with the patent office on 2007-03-08 for smear correction in a digital camera.
This patent application is currently assigned to TransChip, Inc.. Invention is credited to Eugene Fainstain, Yoav Lavi.
Application Number | 20070052822 11/467044 |
Document ID | / |
Family ID | 37829675 |
Filed Date | 2007-03-08 |
United States Patent
Application |
20070052822 |
Kind Code |
A1 |
Fainstain; Eugene ; et
al. |
March 8, 2007 |
Smear Correction In A Digital Camera
Abstract
Methods and apparatus to measure small shifts and rotations of
video images captured in short succession after a first reference
video image is captured, and to add the sequence of captures while
shifting or rotating back the shifted or rotated images, in order
to improve the SNR in low light conditions.
Inventors: |
Fainstain; Eugene; (Netanya,
IL) ; Lavi; Yoav; (Raanana, IL) |
Correspondence
Address: |
TOWNSEND AND TOWNSEND AND CREW, LLP
TWO EMBARCADERO CENTER
EIGHTH FLOOR
SAN FRANCISCO
CA
94111-3834
US
|
Assignee: |
TransChip, Inc.
Ramat - Gan
IL
|
Family ID: |
37829675 |
Appl. No.: |
11/467044 |
Filed: |
August 24, 2006 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
60711156 |
Aug 24, 2005 |
|
|
|
Current U.S.
Class: |
348/239 ;
348/E5.08 |
Current CPC
Class: |
H04N 5/3595
20130101 |
Class at
Publication: |
348/239 |
International
Class: |
H04N 5/262 20060101
H04N005/262 |
Claims
1. A method of capturing a plurality of images, comprising:
capturing a first image; capturing at least a second image;
comparing the second image to the first image, thereby detecting
that the second image is rotated and/or shifted with respect to the
first image; manipulating the second image to thereby correct the
shift and/or rotation; and outputting the first and second images.
Description
CROSS-REFERENCES TO RELATED APPLICATIONS
[0001] This application is a non-provisional of, and claims the
benefit of, co-pending, commonly assigned U.S. Provisional Patent
Application No. 60/711,156, entitled "METHODS AND APPARATUS FOR
SMEAR CORRECTION IN A DIGITAL CAMERA," filed on Aug. 24, 2005,
which application is incorporated herein by reference for all
purposes.
BACKGROUND OF THE INVENTION
[0002] Embodiments of the present invention relate generally to
capturing digital images. More specifically, embodiments of the
invention relate to systems and methods for smear correction in
digitally-captured images.
BRIEF DESCRIPTION OF THE DRAWINGS
[0003] A further understanding of the nature and advantages of the
present invention may be realized by reference to the remaining
portions of the specification and the drawings wherein like
reference numerals are used throughout the several drawings to
refer to similar components. Further, various components of the
same type may be distinguished by following the reference label by
a dash and a second label that distinguishes among the similar
components. If only the first reference label is used in the
specification, the description is applicable to any one of the
similar components having the same first reference label
irrespective of the second reference label.
[0004] FIG. 1 is a diagram graphically depicting embodiments of the
present invention.
[0005] FIG. 2 is a simplified block diagram of an embodiment of the
present invention, which embodiment can detect and correct shift
smear.
[0006] FIG. 3 depicts a simplified block diagram of an embodiment
of a differentiator for shift smear.
[0007] FIG. 4 depicts a simplified block diagram of an embodiment
of a Correlator/Estimator for shift-smear.
[0008] FIG. 5 depicts a De-smear circuit for shift-smear.
[0009] FIG. 6 is a diagram graphically depicting an embodiment of
the present invention relating to motion measurement.
[0010] FIG. 7 depicts a simplified block diagram of an embodiment
of the present invention, which embodiment can detect and correct
Rotation Smear.
[0011] FIG. 8 depicts a simplified block diagram of an embodiment
of a correlation part of the rotation estimation.
[0012] FIG. 9 depicts a simplified block diagram of an embodiment
of a Correlator/Estimator for Rotation smear.
[0013] FIG. 10 depicts a De-smear circuit for Rotation smear.
[0014] FIG. 11 is an embodiment of a JPEG-based de-smear
circuit.
DETAILED DESCRIPTION OF THE INVENTION
[0015] The ensuing description provides preferred exemplary
embodiment(s) only, and is not intended to limit the scope,
applicability or configuration of the invention. Rather, the
ensuing description of the preferred exemplary embodiment(s) will
provide those skilled in the art with an enabling description for
implementing a preferred exemplary embodiment of the invention. It
being understood that various changes may be made in the function
and arrangement of elements without departing from the spirit and
scope of the invention as set forth in the appended claims.
[0016] Specific details are given in the following description to
provide a thorough understanding of the embodiments. However, it
will be understood by one of ordinary skill in the art that the
embodiments may be practiced without these specific details. For
example, systems may be shown in block diagrams in order not to
obscure the embodiments in unnecessary detail. In other instances,
well-known processes, structures and techniques may be shown
without unnecessary detail in order to avoid obscuring the
embodiments.
[0017] Also, it is noted that the embodiments may be described as a
process which is depicted as a flowchart, a flow diagram, a data
flow diagram, a structure diagram, or a block diagram. Although a
flowchart may describe the operations as a sequential process, many
of the operations can be performed in parallel or concurrently. In
addition, the order of the operations may be re-arranged. A process
is terminated when its operations are completed, but could have
additional steps not included in the figure. A process may
correspond to a method, a function, a procedure, a subroutine, a
subprogram, etc. When a process corresponds to a function, its
termination corresponds to a return of the function to the calling
function or the main function.
[0018] Moreover, as disclosed herein, the term "storage medium" may
represent one or more devices for storing data, including read only
memory (ROM), random access memory (RAM), magnetic RAM, core
memory, magnetic disk storage mediums, optical storage mediums,
flash memory devices and/or other machine readable mediums for
storing information. The term "computer-readable medium" includes,
but is not limited to portable or fixed storage devices, optical
storage devices, wireless channels and various other mediums
capable of storing, containing or carrying instruction(s) and/or
data.
[0019] Furthermore, embodiments may be implemented by hardware,
software, firmware, middleware, microcode, hardware description
languages, or any combination thereof. When implemented in
software, firmware, middleware or microcode, the program code or
code segments to perform the necessary tasks may be stored in a
machine readable medium such as storage medium. A processor(s) may
perform the necessary tasks. A code segment may represent a
procedure, a function, a subprogram, a program, a routine, a
subroutine, a module, a software package, a class, or any
combination of instructions, data structures, or program
statements. A code segment may be coupled to another code segment
or a hardware circuit by passing and/or receiving information,
data, arguments, parameters, or memory contents. Information,
arguments, parameters, data, etc. may be passed, forwarded, or
transmitted via any suitable means including memory sharing,
message passing, token passing, network transmission, etc.
[0020] FIG. 1 is an illustration of an algorithm 100 in accordance
with embodiments of the present invention, and according to which
pan and/or shift movement in the x dimension are measured. It
should be noted that FIG. 1 illustrates the algorithm in a
graphical way, and is given for clarity rather than as a
description of an actual implementation.
[0021] In the Figure, pixel values of column groups 110 of the
reference image are summed by summing units 130, to yield a vector
of numbers, each representing the sum of pixels in a column group.
Column groups 110 are illustrated as comprising 8 columns each.
However, this is an arbitrary number, and any other number may be
used, according to the present invention.
[0022] Subtractors 150 subtract each such sum from the sum of the
column group to its right. This subtraction yields a
spatial-difference vector, which is stored in Spatial-Difference
Register 170.
[0023] Similarly, pixel values of column groups 120 of the shifted
image are summed by summing units 140, to yield a vector of
numbers, each representing the sum of pixels in a column group.
Again, column groups 120 comprise 8 columns each, in this specific
example.
[0024] Subtractors 160 subtract each such sum from the sum of the
column group of the reference image. This subtraction yields a
temporal-difference vector, which is stored in Temporal-Difference
Register 180.
[0025] The vectors stored in Spatial-Difference register 170 and
temporal Difference Register 180 are next correlated. The
correlation result is normalized, and will equal +1.0 if the
temporal shift is identical to the spatial shift, i.e. 8 pixels in
the depicted case where groups of eight columns are used. In the
general case, where the temporal shift and the spatial shift are
not necessarily equal, temporal shift can be calculated by
multiplying the correlation result by the number of columns in a
column group.
[0026] The above formula is valid for a shift range of 0 to +8 for
the case of eight columns in a column group, and 0 to +n in the
general case. Inaccuracies stem mainly from the edges, and are
usually negligible. Outside said range, large errors may occur, and
the results should only be used as out-of-range indicator.
[0027] To cover for wider range of numbers, the correlation
described above has to be recalculated for a shifted version of the
spatial difference vector; for example, to cover for the range of 0
to -8, a second correlation is calculated where elements of the
spatial diff register may be shifted by one relative to element of
the temporal diff register. Re-correlation may be done by
Normalized Correlation 190, reusing its internal computation
resources.
[0028] As would be evident to those ordinarily skilled in the art,
the same algorithm may be used for tilt and/or shift in movement in
the vertical dimension, when row groups rather than column groups
are summed.
[0029] FIG. 2 depicts a simplified block diagram of an embodiment
200 of the present invention, which can detect and correct shift
smear. It is assumed that the image is scanned row by row.
Embodiment 200 comprises, among others, column and row accumulation
circuits. Adder 210 and register 230 accumulate pixel values in
groups of rows. As scan order is such that groups of rows can be
summed successively, this part of the circuit is straight
forward--Adder 210 and Register 230 accumulate the pixel stream;
when a group of rows terminates, a Control Unit (not shown) stores
the contents of Register 230 in Vector RAMs 260 or 270, for further
processing, and clears Register 230 for the accumulation of the
next group of rows.
[0030] The Control Unit, which is not shown, is also in charge of
advancing the Vector RAMs address so that it will point to the next
element of the X vector after the next group of rows is
accumulated. In addition, it selects Reference Image Vector RAM 270
when the reference image is scanned, and Shifted Image Vector RAM
260 when the shifted image is scanned.
[0031] The accumulation of the values of groups of columns,
according to this specific embodiment, is different, because the
pixels to be summed for a column group are not contiguous in the
incoming pixel stream. Whenever a segment of pixels belonging to a
certain column group starts, the current value of the partial sum
for that group is read from one of Vector RAMs 280 or 290, through
MUX 240 and into register 250. When the reference image is scanned,
Reference Image vector RAM 290 is used, and when the shifted image
is scanned, Shifted Image vector RAM 280 is used. The pixels
belonging to that segment are then accumulated in Register 250,
when Adder 220 sums the old value with the new values, and MUX 240
transferring the result of Adder 220 to Register 250. When the
segment stops, the value in Register 250 is written to Vector RAM
280 or Vector Ram 290, replacing the previous partial sum for that
column group.
[0032] X vectors of the reference and shifted Images 270 and 260,
and Y vectors of the reference and shifted Images 290 and 280, are
input to Shift-Smear Differentiator Unit 300, which calculates the
spatial and temporal difference vectors, and stores them in its
Spatial diff-x 370a, Spatial diff-y 370b, Temporal diff-x 370c, and
Temporal diff-y 370d memory units.
[0033] Shift-Smear Correlator/Estimator 400 calculates the
horizontal and vertical shifts between the reference and the
shifted images according to the present invention, based on the
Spatial and Temporal diff memory units 370a, 370b, 370c, and
370d.
[0034] De-smear unit 500, accumulates pixels from a sequence of
exposures, correcting the shift smear, according to shift values
input from Correlator/Estimator 400. Said correction consists of
summing the reference image values with values of successive
shifted images, shifting the shifted images by the inverse of the
detected shift value while they are added.
[0035] FIG. 3 depicts a simplified block diagram of an embodiment
of a differentiator for shift smear 300. Inputs to differentiator
300 are shifted image X vector 260, Reference image X vector 270,
shifted image Y vector 280, and Reference image Y vector 280 (all
in FIG. 2).
[0036] The differentiation calculation may be done serially, using
a single subtractor 360. The time to do the calculations may be in
the vertical blank period, when a pixel stream is not generated. A
Control unit (not shown) will control the inputs to Subtractor 360,
using Multiplexer units 310, 320, 340, 350. The Control Unit will
also generate Read and Write signals, as well as address inputs,
for Difference Memory segments 370a, 370b, 370c and 370d, where,
respectively, Spatial X, Spatial Y, Temporal X and Temporal Y
difference vectors will be stored.
[0037] When Spatial-X difference of the Reference image is
calculated, MUX 310 will route Reference Image X Vector 270 output
to REG 330 and to MUX 350, and MUX 350 will route MUX 310 input to
the Plus input of Subtractor 360. MUX 340 will route REG 330 output
to the Minus input of Subtractor 360.
[0038] All Reference Image X Vector elements will be read from
memory 270, in an order which may be left to right. Register 330
may function as a delay unit, delaying its input value by one clock
cycle. Consequently, the output of Register 330 will be hold the
value of an X vector element corresponding to column group n while
the input to Register 330 is already asserted with the value
corresponding to column group n+1. As MUX 340 now routes Register
330 output to the minus input of Subtractor 360, Subtractor 360
will subtract the value of column group n, from column group n+1.
The results will be stored in the SPATIAL-X Difference Memory
370a.
[0039] Similarly, the SPATIAL-Y part may be calculated by setting
MUX 310 to route Reference Image Y Vector 290 on its input to
Register 330 and MUX 350, MUX 350 may be set to route MUX 310
output to the Plus input of Subtractor 360, and MUX 340 may route
the output of Register 330 to the Minus input of Subtractor 360.
Scanning of Y Vector elements may be done from top to bottom row
groups, and, consequently, Register 330 may output the value
corresponding to row group n while its input is asserted with the
value corresponding to row group n+1, so that Subtractor 360 will
generate the SPATIAL-Y difference results. The outputs from
Subtractor 360 are stored in the SPATIAL-Y Memory 370b.
[0040] For the calculation of the temporal difference vectors, MUX
350 may route the output of MUX 320 to Subtractor 360 Plus input,
and MUX 340 may route the output of MUX 310 to the Minus input of
Subtractor 360. Register 330 is not used. When Temporal-X
difference is calculated, MUX 310 will route Reference Image X
Vector memory 270 to MUX 340, which will route it to Subtractor 360
Minus input, while MUX 310 will route Shifted Image X Vector memory
260 to MUX 350, which will route it to Subtractor 360 Plus input.
Memories 270, 260 may be scanned so as to output all entries. The
outputs from Subtractor 360, representing the temporal difference
in the X dimension, will be stored in the TEMPORAL-X Difference
Memory 370c.
[0041] Similarly, When Temporal-Y difference is calculated, MUX
units 310,320,340 and 350 will route memory units 290, 280 to
Subtractor 360, and the subtraction results will be stored in the
TEMPORAL-Y Difference Memory 370d.
[0042] It should be noted that the sign of the spatial difference
will be positive of the value of a subsequently read element is
larger than the value of the previous element. Likewise, the sign
of the temporal difference will be positive if the value for the
Shifted Image is larger than that of the Reference Image. As would
be obvious to those ordinarily skilled in the art, a MUX unit could
be saved from embodiment 300 if the sign of the temporal or spatial
difference vector could be negated, a negation which could be taken
care of in further processing steps. Such an embodiment is not
shown for the sake of clarity, but it falls within the scope of the
present invention.
[0043] FIG. 4 depicts a simplified block diagram of an embodiment
of Correlator/Estimator 400 for shift-smear. The inputs to
Correlator/Estimator 400 are the SPATAIL-X, SPATAIL-Y, TEMPORAL-X
and TEMPORAL-Y difference memories 370a, 370b, 370c and 370d,
respectively (FIG. 3). Correlator/Estimator 400 may work in series,
correlating for the X dimension then, reusing the arithmetic units,
correlating for the Y dimension; yet further it would reuse the
arithmetic units to calculate correlation for another range; e.g. 0
to -8. Alternatively, the order may be reversed. Yet alternatively
some or all of the correlations can be done in parallel.
[0044] When X-dimension correlation is calculated, MUX units 410,
420 route the SPATIAL-X and SPATIAL-Y difference memories 370a,
370c, to their outputs.
[0045] Each correlation calculation of the X and Y dimensions
comprises, according to this specific embodiment, two cycles, where
each of the two cycles comprises two phases. The two cycles may be
used for two ranges of correlations, and the two phases of each
cycle may be used so that in the First Phase the accumulation of
the squares of the values of the Spatial difference vector is
calculated, and in the Second Phase the multiplication products of
elements of the Spatial Difference and the Temporal Difference
vectors are accumulated. The terms "First Phase" and "Second Phase"
are used throughout the foregoing description for the sake of
clarity, but, as would be obvious to those ordinarily skilled in
the art, the order of the phases may be changed, or the two phases
can be done in parallel, all in the scope of the present
invention.
[0046] In the First Phase of each of the two cycles, MUX 430 routes
the output of MUX 410 to its output and in the Second Phase MUX 430
routes the output of MUX 420 to its output. Multiplier 440
multiplies the output of MUX 410 with the output of MUX 430. Hence,
in the First Phase Multiplier 440 will calculate the square of the
spatial difference elements, and in the Second Phase it will
calculate the multiplications of spatial and temporal difference
elements.
[0047] Adder 450 and Register 460 accumulate the results of the
multiplications. Register 460 will be cleared at the beginning of
each Phase. After the First Phase its contents will be transferred
to register 470a (designated .SIGMA.S.sup.2) for the storage of the
accumulation of the squares of the Spatial vector elements. After
the Second Phase, the contents of Register 460 will be transferred
to register 470b (designated .SIGMA.S*T) for the storage of the
accumulation of the multiplications of Spatial and Temporal
elements. This process will repeat in the two cycles.
[0048] In each of the two cycles, divider 480 divides the sum of
the accumulation of the multiplications of Spatial and Temporal
elements (.SIGMA.S*T) by the accumulation of the square Spatial
vector elements (.SIGMA.S.sup.2), to yield the shift estimate for
that specific cycle. This may be done once for the X dimension and
once for the Y dimension. The estimate results for the first cycle,
referring to a first range of possible shifts (e.g. 0 to 8), may be
stored in Register 490b, while the estimate results for the second
cycle, referring to a second range of possible shifts (e.g. -8 to
0) may be stored in register 490a.
[0049] The disclosure of embodiments for the estimate of a shift in
a captured image relative to a reference image is now completed.
Other embodiments of the same functions under the present invention
include several group sizes, several spatial difference estimates,
which may or may not partly overlap, selection of captures in the
sequence which are not necessarily the first capture as a Reference
Image, and doing iterative X and Y dimension estimates, where the
results of an estimate in one dimension are used to select the
accumulation boundaries for the estimate in the other
dimensions.
[0050] In addition, as would be appreciated by those ordinarily
skilled in the art, trivial variations of the foregoing do not
depart from the scope of the present invention.
[0051] FIG. 5 depicts De-smear circuit 500 for shift-smear. When
the Reference image is captured, MUX 540 will direct the incoming
pixel stream to Frame Buffer 550, for storage. Subsequent images,
which may be shifted relative to the reference image, are stored in
Frame Buffer 510.
[0052] After a Shifted Image is stored in Frame Buffer 510, using
Shift Estimates from Correlator/Estimator 400, the Reference image
and the Shifted image may be accumulated, including shift smear
correction. Said correction consists of summing the reference image
values with values of successive shifted images, shifting the
shifted images by the inverse of the detected shift values, while
they are added. For example, if shift values of (X.sub.1,Y.sub.1),
(X.sub.2,Y.sub.2), . . . (X.sub.n, Y.sub.n) are detected for a
sequence of n shifted images following the Reference image, the
smear corrected sum will be calculated by summing -S(i,
j)=.SIGMA.C.sub.k (i-X.sub.k, j-Y.sub.k), where S(i,j) is pixel
(i,j) of the sum image, C.sub.k(i,j) is pixel (i,j) of the capture
k, C.sub.0 is the reference image and (X.sub.0,Y.sub.0)=(0,0).
[0053] Reference Image from Frame Buffer 550 enters adder 530,
where each pixel is added with a pixel of Shifted Image Frame
Buffer 510. Scan Control unit 520 controls the address generation
of both Reference Image Buffer 550 and Shifted Image Buffer 510, so
that an inverse shift will take place, and pixels corresponding to
the same point in the target object will be accumulated.
[0054] In another aspect of the present invention, rotation of the
camera can be detected and estimated, and rotation smear can be
corrected, including rotation which is not centered at the center
of the image.
[0055] As can be shown using basic geometry, when an image rotates
with an angle .beta.around center of rotation (C.sub.x,C.sub.y),
pixel (P.sub.x,P.sub.y) will shift horizontally and vertically.
Assuming .beta. is small, The horizontal shift will equal
approximately .DELTA.X=*(P.sub.y-C.sub.y) and the vertical shift
will equal approximately .DELTA.Y=.beta.*(P.sub.x-C.sub.x).
[0056] Vertical shift for a vertical stripe at the left side of the
screen, located -W pixels from the center of the screen will equal:
.DELTA.Y.sub.Left=.beta.*(-W-C.sub.x). Similarly, vertical shift at
a Right vertical stripe located +W pixels from the center of the
screen will equal:
.DELTA.Y.sub.right=.beta.*(W-C.sub.x). For horizontal stripes at
the Top and the Bottom of the screen, with distances H from the
center, horizontal shifts of .DELTA.X.sub.top=*(H-C.sub.y) and
.DELTA.X.sub.bottom=.beta.*(-H-C.sub.y) will be observed.
[0057] It follows that any rotation of the image around any center
can be characterized by the set (.DELTA.Y.sub.left,
.DELTA.Y.sub.right, .DELTA.X.sub.top, .DELTA.X.sub.bottom). By
adding linear X-dimension shift values to the X numbers of the set,
and linear Y-dimension shift values to the Y numbers of the set,
any combination of shift and rotate can be defined.
[0058] Rotation around an arbitrary point in the image which is not
necessarily the center of the image is equivalent to a combination
of rotation around the center and a shift. Consequently, detection
and correction of non-centered rotation covers both centered shift
and rotation, and is fully defined by the set (.DELTA.Y.sub.left,
.DELTA.Y.sub.right, .DELTA.X.sub.top, .DELTA.X.sub.bottom). In the
foregoing the term Rotation, Rotated Image and Rotation Smear imply
a combination of centered rotation and shift.
[0059] FIG. 6 is an illustration of an algorithm 600 in accordance
with the present invention and according to which the rotation is
measured. It should be noted that FIG. 6 illustrates the algorithm
in a graphical way, and is given for clarity rather than as an
example for an actual embodiment.
[0060] Algorithm 600 processes information from Sensor Array 610,
which could be both the Reference Image and the Rotated Image.
[0061] Sensor Array 610 comprises zone 620 at its top half where
column groups are accumulated, zone 630 at its right half where row
groups are accumulated, zone 640 at its bottom half where column
groups are accumulated, and zone 650 at its left half where row
groups are accumulated. The zones may be non-overlapping or
partially overlapping. The accumulation vectors are stores at Top
Column Accumulator 660, Right Rows Accumulator 670, Bottom Column
Accumulator 680 and Left Rows Accumulator 690.
[0062] FIG. 7 depicts a simplified block diagram of an embodiment
1200 of the present invention, which can detect and correct
Rotation Smear. FIG. 7 and FIG. 2 share several subunits, as can be
observed from identical numbering of some of the units in FIG.
7.
[0063] It is assumed that the image is scanned row by row. Circuit
1200 comprises column accumulation circuits for the left and right
halves, and row accumulation circuits for the top and bottom halves
of the temporal and the reference image. Adder 210 and register 230
accumulate pixel values in groups of rows for the top and the
bottom halves of the images. As scan order is such that groups of
rows are summed successively, this part of the circuit is straight
forward--Adder 210 and Register 230 accumulate the pixel stream;
when a group of rows terminates, a Control Unit (not shown) stores
the contents of Register 230 in Vector RAMs 260a, 260b, 270a or
270b, for further processing, and clears Register 230 for the
accumulation of the next group of rows.
[0064] Said Control Unit is also in charge of advancing the Vector
RAMs addresses so that they will point to the next element of the X
vector after the next group of rows is accumulated. In addition, it
selects Reference Image Bottom X Vector RAM 270b, Reference Image
Top X Vector RAM 270a, Rotated Image Bottom X Vector RAM 260b,
Rotated Image Top X Vector RAM 260a, according to the zone and the
image being scanned.
[0065] The accumulation of the values of Left and Right groups of
columns, according to this specific embodiment, is different,
because the pixels to be summed for a column group are not
contiguous in the incoming pixel stream. Whenever a segment of
pixels belonging to a certain column group starts, the current
value of the partial sum for that group is read from one of Vector
RAMs 280a, 280b, 290a, 290b through MUX 1240 and into register 250.
When the Reference image is scanned, Reference Image vector RAMs
290a or 290b are used, and when the rotated image is scanned,
Rotated Image vector RAMs 280a or 280b are used.
[0066] Pixels belonging to a specific segment are accumulated in
Register 250, with Adder 220 summing the old value with the new
values, and MUX 1240 transferring the result of Adder 220 to
Register 250. When the segment stops, the value in Register 250 is
written to Vector RAM 280a, 280b, 290a or 290b, replacing the
previous partial sum for that column group.
[0067] X vectors of the reference and rotated Images 270a, 270b,
260a and 260b, and Y vectors of the reference and rotated Images
290a, 290b, 280a and 280b, are input to Rotation Differentiator
Unit 1300, which calculates the spatial and temporal difference
vectors, and stores them in its Spatial-diff registers 1310a,
1310b, 1310c, and 1310d; and Temporal-diff registers 1310e, 1310f,
1310g, and 1310f
[0068] Rotation Correlator/Estimator 1400 calculates the horizontal
and vertical shifts, of the Top, Bottom, Right and left halves,
between the reference and the rotated images, according to the
present invention, based on the Spatial and Temporal diff units
1310a through 1310h.
[0069] Rotation Correlator/Estimator 1400 may successively
correlate for different ranges of shift values, for example 0 to 8
and 0 to -8, in each Top, Right, Bottom and Left. Alternatively, it
may correlate for the different ranges in parallel, or in a
combination of parallel and serial.
[0070] De-smear unit 1500 accumulates images from a sequence of
exposures, correcting the rotation-smear, according Left, Right,
Top and Bottom values input from Correlator/Estimator 1400. Said
correction consists of summing the Reference image values with
values of successive images which are rotated relative to the
Reference image, rotating the rotated images by the inverse of the
detected rotation while they are added.
[0071] FIG. 8 depicts a simplified block diagram of an embodiment
1300 of the correlation part of the rotation estimation. It should
be noted that FIG. 8 is similar to FIG. 3, and that in both cases
similar correlations are calculated. For the rotation estimation,
four rather than 2 correlations are calculated, top, right, bottom
and left portions of the image. Horizontal shift is evaluated for
the top and bottom halves, and vertical shift is evaluated for the
left and right halves.
[0072] Differences Memories 1370a to 1370h, store the spatial and
temporal difference value, for the top, right, bottom and left
portions of the image.
[0073] FIG. 9 depicts a simplified block diagram of an embodiment
of Correlator/Estimator 1400 for Rotation smear. The inputs to
Correlator/Estimator 1400 are the Right Spatial, Left Spatial, Top
Spatial, Bottom Spatial, Right Temporal, Left Temporal, Top
Temporal and Bottom Temporal difference memories 1310a through
1310h, respectively (FIG. 7). Correlator/Estimator 1400 may work in
series, correlating for two or more shift ranges, and in each range
correlating for the Left portions of the images, and then, reusing
the arithmetic units, correlating for Right, Top and Bottom halves
of the images. Alternatively, the order may be reversed. Yet
alternatively correlation for the four portions and/or the two or
more shift ranges can be done in parallel, or in a combination of
serial and parallel calculation.
[0074] When Right, Left Top or Bottom correlation is calculated,
MUX units 1410, 1420 route the corresponding inputs from
differential memories 1310a through 1310h (FIG. 7) to their
outputs.
[0075] In each correlation calculation cycle, calculations for
different shift ranges of Top, Right, Bottom and Left will take
place, where ranges, for example, could be 0 to 8 and 0 to 8.
[0076] Each such correlation cycle, may comprise two phases, where
in the First Phase the accumulation of the squares of the values of
the Spatial Difference vector is calculated, and in the Second
Phase the multiplication products of elements of the Spatial
Difference and the Temporal Difference vectors are accumulated,
where in each phase correlation for Top, Right, Bottom and Left
portions is done. The terms "First Phase" and "Second Phase" are
used throughout the foregoing description for the sake of clarity,
but, as would be obvious to those ordinarily skilled in the art,
the order of the phases may be changed, or the two phases can be
done in parallel, all in the scope of the present invention.
[0077] In the First Phase MUX 1430 routes the output of MUX 1410 to
its output, and in the Second Phase MUX 1430 routes the output of
MUX 1420 to its output. Multiplier 1440 multiplies the output of
MUX 1410 with the output of MUX 1430. Hence, in the First Phase
Multiplier 1440 will calculate the square of the Spatial difference
elements, and in the Second Phase it will calculate the
multiplications of Spatial and Temporal difference elements.
[0078] Adder 1450 and Register 1460 accumulate the results of the
multiplications. Register 1460 will be cleared at the beginning of
each Phase. After the First Phase its contents will be transferred
to register 1470b (designated .SIGMA.S.sup.2) for the storage of
the accumulation of the squares of the Spatial vector elements.
After the Second Phase, the contents of Register 1460 will be
transferred to register 1470a (designated .SIGMA.S*T) for the
storage of the accumulation of the multiplications of Spatial and
Temporal elements.
[0079] Divider 1480 divides the sum of the accumulation of the
multiplications of Spatial and Temporal elements (.SIGMA.S*T) by
the accumulation of the square Spatial vector elements
(.SIGMA.S.sup.2), to yield the shift estimate. This may be done
once for each of the Left, Right, Top and Bottom halves, and
repeated for the two or more cycles of shift ranges
[0080] In the first cycle the result of the division is stored in
Register 1490b, and in the second cycles it is store din register
1490a. Thus registers 1490a, 1490b will hold shift results for two
different ranges. For more than two ranges more estimate registers
(not shown) will be employed.
[0081] The disclosure of embodiments for the shift estimates of a
rotation in a captured image relative to a reference image is now
completed. Other embodiments of the same functions include several
group sizes, and selection of captures in the sequence which are
not necessarily the first capture, for the Reference Image.
[0082] In addition, as would be appreciated by those ordinarily
skilled in the art, trivial variations of the foregoing do not
depart from the scope of the present invention.
[0083] FIG. 10 depicts De-smear circuit 1500 for Rotation smear. It
should be noted that De-smear 1500 is similar to De-Smear 500 (FIG.
5), and uses the same sub-units, except for the Scan Control 1520,
which, unlike Scan Control 520, can handle rotations.
[0084] When the Reference image is captured, MUX 540 may direct the
incoming pixel stream to Frame Buffer 550, for storage. Subsequent
images, which may be rotated relative to the reference image, may
be stored in Frame Buffer 510.
[0085] After a Rotated Image is stored in Frame Buffer 510, using
Top, Right, Bottom and Left shift Estimates from
Correlator/Estimator 1400, the Reference image and the rotated
image may be accumulated, including rotation smear correction. Said
correction consists of summing the reference image values with
values of said rotated image, rotating the rotated image by the
inverse of the measured rotation, while they are added. The process
described above then repeats for subsequent rotated images.
[0086] For example, if shift measurement values of
(T.sub.1,B.sub.1,L.sub.1,R.sub.1),
(T.sub.2,B.sub.2,L.sub.2,R.sub.2), . . .
(T.sub.n,B.sub.n,L.sub.n,R.sub.n) are detected for a sequence of n
captures following the reference image, the smear corrected sum for
pixel (ij) will be calculated by summing: S(i,
j)=.SIGMA.C.sub.k((i-(T.sub.k+B.sub.k)/2-(j/H)*(T.sub.k-B.sub.k)/2),(j-(L-
.sub.k+R.sub.k)/2-(i/W)*(L.sub.k-R.sub.k)/2)) , where S(i,j) is
pixel (i,j) of the sum image, C.sub.k(i,j) is pixel (i,j) of the
capture k, C.sub.0 is the reference image, H is half of the
vertical distance between the geometric centers of Top and Bottom,
in pixels, W is half of the horizontal distance between the
geometric centers of Left and Right, in pixels, and
(T.sub.0,B.sub.0,L.sub.0,R.sub.0),=(0,0,0,0).
[0087] In another aspect of the invention, it is possible to
implement Shift De-smear 500 and Rotation De-smear 1500 using JPEG
or similar compression, saving the needs for frame buffers 510, 550
(FIG. 5 and FIG. 10).
[0088] Circuit 2500 of FIG. 11 is an embodiment of such JPEG-based
de-smear. Incoming pixel stream for the reference and for the
subsequent shifted or rotated images first enters JPEG conversion
block 2510a where it is converted to JPEG or other compressed image
format, and output to Rotated JPEG Frame Buffer 2540b, and, through
MUX 2530, to Reference Image JPEG Frame Buffer 2540a. When the
Reference image is scanned, MUX 2530 routes JPEG 2510a output to
Reference Image JPEG Frame Buffer 2540a, where it is stored. When
subsequent rotated images are scanned, they will be stored in
Rotated JPEG Frame Buffer 2540b.
[0089] When Rotated images are to be added to the Reference image,
which may be done, for example, during Vertical Blank period, the
JPEG representation of both the Reference and the Rotated images
are first converted to x-y representation. This is done by DE-JPEG
2550afor the Reference image, and by De-JPEG 2550b for the Rotated
image. DE-JPEG may be done "on the fly", without the need for a
frame buffer.
[0090] Outputs from DE-JPEG units 2550a, 2550b, enter Stripe
Buffers 2560a, 2560b; and
[0091] output from Stripe Buffer 2560b enters Stripe Buffer 2560c.
Stripe Buffer 2560a may be implemented as a First-In-First-Out
(FIFO) memory. Stripe Buffers 2560b, 2560c may be implemented as
FIFO memory with additional random-access read capability. Each
Stripe Buffer 2560a, 2560b and 2560c should contain at least n
lines, where n is the maximum expected vertical displacement of any
pixel between the Reference Image and the Shifted or Rotated
Image.
[0092] Output from Stripe buffer 2560a corresponds to De-JPEG
conversion of the Reference Image, delayed by n lines. It is input
to Adder 2570, which adds each pixel with the corresponding pixel
of the Rotated Image, read by Random Read unit 2580. The result of
Adder 2570 corresponds to the sum of the Reference image and the
Rotated image, with the latter de-rotated prior to the addition.
Said sum is JPEG-converted by JPEG unit 2510b, and fed to MUX 2530.
MUX 2530, when image summation is done, will route the output of
JPEG 2510b to JPEG Frame Buffer 2540a.
[0093] This sequence of events will repeat for subsequent Rotated
images, where the JPEG conversion of subsequent images are stored
in JPEG Frame Buffer 2540b, and the partial sum of the images is
stored in JPEG Frame Buffer 2540a. For each subsequent image, JPEG
frame Buffer 2540a will be updated by the JPEG conversion of the
sum of the De-JPEG conversion of its last value, with the
de-rotated subsequent image, where the latter is JPEG converted
into JPEG Frame Buffer 2540b and then De-JPEG converted for the
summation.
[0094] De-Rotate Calculator 2520 gets Shift Estimates from
Correlator/Estimator 1400 (FIG. 7). For every pixel (x,y) of the
reference Image, it calculates the corresponding pixel of the
rotated image, similarly to Scan Control 1520 (FIG. 10) and
according to the same formula: S(i,
j)=.SIGMA.C.sub.k((i-(T.sub.k+B.sub.k)/2-(j/H)*(T.sub.k-B.sub.k)/2),(j-(L-
.sub.k+R.sub.k)/2-(i/W)*(L.sub.k-R.sub.k)/2)) (The reader is
referred to description of FIG. 10 above).
[0095] The x and y coordinates of each pixel in the rotated image
to be added are indicated by De-Rotate calculator 2520 to
Random-Read unit 2580, which translates the coordinates to an
address in Stripe Buffers 2560b, 2560c, where the corresponding
pixel of the rotated or shifted image is located. It then reads the
corresponding pixel and outputs it to Adder 2570.
[0096] As would be obvious to those ordinarily skilled in the art,
resources in 2500 could be shared and time multiplexed. For
example, a single JPEG unit could replace both 2510a and 2510b, and
a single De-JPEG unit could replace both units 2550a, 2550b, all
within the scope of the present invention.
[0097] While the principles of the disclosure have been described
above in connection with specific apparatuses and methods, it is to
be clearly understood that this description is made only by way of
example and not as limitation on the scope of the invention.
* * * * *