U.S. patent application number 14/304625 was filed with the patent office on 2015-12-17 for super-resolution from handheld camera.
The applicant listed for this patent is Samsung Electronics Co., Ltd.. Invention is credited to Anthony BOTZAS, Candice Hellen Brown ELLIOTT, Michel Francis HIGGINS.
Application Number | 20150363922 14/304625 |
Document ID | / |
Family ID | 54836573 |
Filed Date | 2015-12-17 |
United States Patent
Application |
20150363922 |
Kind Code |
A1 |
ELLIOTT; Candice Hellen Brown ;
et al. |
December 17, 2015 |
SUPER-RESOLUTION FROM HANDHELD CAMERA
Abstract
An apparatus and method for increasing the resolution of an
image are provided. The method includes capturing a plurality of
frames of an image, determining a reference frame from among the
plurality of frames, iteratively determining an offset of each of
the plurality of frames to the reference frame until unity scaling
is reached, and determining a pixel value for insertion between
pixels of the reference frame.
Inventors: |
ELLIOTT; Candice Hellen Brown;
(Santa Rosa, CA) ; BOTZAS; Anthony; (San Jose,
CA) ; HIGGINS; Michel Francis; (Duncans Mills,
CA) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Samsung Electronics Co., Ltd. |
Suwon-si |
|
KR |
|
|
Family ID: |
54836573 |
Appl. No.: |
14/304625 |
Filed: |
June 13, 2014 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
62011311 |
Jun 12, 2014 |
|
|
|
Current U.S.
Class: |
382/254 |
Current CPC
Class: |
G06T 2207/20208
20130101; G09G 5/10 20130101; H04N 5/35563 20130101; G09G 5/02
20130101; G06T 2207/20192 20130101; G06T 5/001 20130101; G06T
2207/10016 20130101; G06T 2207/20201 20130101; G06T 2207/20221
20130101; G06T 2207/20182 20130101; H04N 9/07 20130101; G06T 5/002
20130101; H04N 9/045 20130101; G06T 5/003 20130101; G06T 5/50
20130101; G06K 9/4652 20130101; G06T 2207/10024 20130101; G06T
3/4015 20130101 |
International
Class: |
G06T 5/50 20060101
G06T005/50; G06T 7/00 20060101 G06T007/00 |
Claims
1. A method for increasing the resolution of an image, the method
comprising: capturing a plurality of frames of an image;
determining a reference frame from among the plurality of frames;
iteratively determining an offset of each of the plurality of
frames to the reference frame until unity scaling is reached; and
determining a pixel value for insertion between pixels of the
reference frame.
2. The method of claim 1, wherein the iteratively determining the
offset of each of the plurality of frames comprises down scaling
each of the plurality of frames and the reference frame.
3. The method of claim 2, wherein the down scaling is performed by
powers of two.
4. The method of claim 2, wherein the down scaling is performed by
powers of three.
5. The method of claim 1, wherein the determining of the reference
frame comprises determining the number N of the plurality of frames
and selecting the N/2 frame as the reference frame.
6. The method of claim 1, wherein the determining of the pixel
value for insertion comprises up scaling each of the plurality of
frames and the reference frame.
7. The method of claim 1, wherein the determining of the pixel
value for insertion comprises re-sampling each of the plurality of
images by linear interpolation and comparing the re-sampled image
to the reference image at unity scaling.
8. The method of claim 7, wherein the re-sampling is performed by
at least one of 1/2 pixel and 1/4 pixel.
9. The method of claim 8, further comprising summing the values of
each pixel for each of the plurality of frames and determining an
average value for each pixel.
10. The method of claim 1, further comprising distorting each of
the plurality of frames in relation to the reference frame.
11. The method of claim 10, wherein the distorting of each of the
plurality of frames comprises determining a fractional offset at
each corner of an intersection of the frame and the reference
frame.
12. An apparatus for increasing the resolution of an image, the
apparatus comprising: a camera unit configured to capture a
plurality of frames of an image; and a control unit configured to
determine a reference frame from among the plurality of frames, to
iteratively determine an offset of each of the plurality of frames
to the reference frame until unity scaling is reached, and to
determine a pixel value for insertion between pixels of the
reference frame.
13. The apparatus of claim 12, wherein the control unit is
configured to iteratively determine the offset of each of the
plurality of frames by down scaling each of the plurality of frames
and the reference frame.
14. The apparatus of claim 13, wherein the down scaling is
performed by powers of two.
15. The apparatus of claim 13, wherein the down scaling is
performed by powers of three.
16. The apparatus of claim 12, wherein the control unit is
configured to determine the reference frame by determining the
number N of the plurality of frames and selecting the N/2 frame as
the reference frame.
17. The apparatus of claim 12, wherein the control unit is
configured to determine the pixel value for insertion by up scaling
each of the plurality of frames and the reference frame.
18. The apparatus of claim 12, wherein the control unit is
configured to determine the pixel value for insertion by
re-sampling each of the plurality of images by linear interpolation
and comparing the re-sampled image to the reference image at unity
scaling.
19. The apparatus of claim 18, wherein the re-sampling is performed
by at least one of 1/2 pixel and 1/4 pixel.
20. The apparatus of claim 19, wherein the control unit is further
configured to sum the values of each pixel for each of the
plurality of frames and determine an average value for each
pixel.
21. The apparatus of claim 12, wherein the control unit is further
configured to distort each of the plurality of frames in relation
to the reference frame.
22. The apparatus of claim 21, wherein the control unit is
configured to distort each of the plurality of frames by
determining a fractional offset at each corner of an intersection
of the frame and the reference frame.
Description
CROSS REFERENCE TO RELATED APPLICATION(S)
[0001] This application claims the benefit under 35 USC
.sctn.119(e) of U.S. Provisional Application No. 62/011,311, filed
Jun. 12, 2014, the entire disclosure of which is hereby
incorporated by reference.
TECHNICAL FIELD
[0002] The present disclosure relates to an apparatus and method
for providing an image. More particularly, the present disclosure
relates to an apparatus and method for providing an image having
increased-resolution.
BACKGROUND
[0003] Mobile terminals were first developed to provide wireless
communication between users. As technology has advanced, mobile
terminals now provide many additional features beyond the simple
telephone conversation. For example, mobile terminals are now able
to provide advanced functions such as an alarm, a Short Messaging
Service (SMS), a Multimedia Messaging Service (MMS), E-mail, games,
short range communication, an image capturing function using a
mounted digital camera, a multimedia function for providing audio
and video content, a scheduling function, and many more. With the
plurality of features now provided, a mobile terminal has
effectively become a necessity of daily life for most people.
[0004] As is known in the art, an image may be captured by a
digital camera mounted on the mobile terminal. For example, when a
user selects an image capturing function, a Graphical User
Interface (GUI) may be displayed, allowing the user to select a
capturing button of the GUI to ultimately capture a desired image.
When capturing an image, an image sensor of the digital camera is
controlled to receive information on a plurality of photosites.
However, the number of photosites may be limited and thus provide
an image having a low resolution. Accordingly, there is a need for
an improved apparatus and method for providing an image having
super-resolution using an existing image sensor.
[0005] The above information is presented as background information
only to assist with an understanding of the present disclosure. No
determination has been made, and no assertion is made, as to
whether any of the above might be applicable as prior art with
regard to the present disclosure
SUMMARY
[0006] Aspects of the present disclosure are to address at least
the above-mentioned problems and/or disadvantages and to provide at
least the advantages described below. Accordingly, an aspect of the
present disclosure is to provide an apparatus and method for
providing an image having super-resolution.
[0007] In accordance with an aspect of the present disclosure, a
method for increasing the resolution of an image is provided. The
method includes capturing a plurality of frames of an image,
determining a reference frame from among the plurality of frames,
iteratively determining an offset of each of the plurality of
frames to the reference frame until unity scaling is reached, and
determining a pixel value for insertion between pixels of the
reference frame.
[0008] In accordance with another aspect of the present disclosure,
an apparatus for increasing the resolution of an image is provided.
The apparatus includes a camera unit configured to capture a
plurality of frames of an image, and a control unit configured to
determine a reference frame from among the plurality of frames, to
iteratively determine an offset of each of the plurality of frames
to the reference frame until unity scaling is reached, and to
determine a pixel value for insertion between pixels of the
reference frame.
[0009] Other aspects, advantages, and salient features of the
disclosure will become apparent to those skilled in the art from
the following detailed description, which, taken in conjunction
with the annexed drawings, discloses various embodiments of the
disclosure.
BRIEF DESCRIPTION OF THE DRAWINGS
[0010] The above and other aspects, features, and advantages of
various embodiments of the present disclosure will be more apparent
from the following description taken in conjunction with the
accompanying drawings, in which:
[0011] FIGS. 1A-1C illustrate the motion of a camera as a user
attempts to take a still shot of a stationary object according to
an embodiment of the present invention;
[0012] FIGS. 2A-2D illustrate a method of improving the resolution
of an image for display by overlapping frames of a captured image
according to an embodiment of the present disclosure;
[0013] FIGS. 3A-3C illustrate a process of capturing and aligning
two low resolution images according to an embodiment of the present
disclosure;
[0014] FIG. 4 illustrates a progressive approximation for aligning
image captures according to an embodiment of the present
disclosure;
[0015] FIGS. 5A and 5B illustrate two iterations of a progressive
approximation method according to an embodiment of the present
disclosure;
[0016] FIG. 6 illustrates a result of four iterations of a
progressive approximation according to an embodiment of the present
disclosure;
[0017] FIGS. 7A and 7B illustrate an eighth iteration and ninth
iteration to determine fractional pixel offsets according to an
embodiment of the present disclosure;
[0018] FIGS. 8A-8C illustrate two image captures requiring
additional morphing techniques according to an embodiment of the
present disclosure;
[0019] FIGS. 9A-9C illustrate a hybrid global and local
registration algorithm for linearly distorted frames according to
an embodiment of the present disclosure;
[0020] FIG. 10 illustrates a progressive approximation for aligning
image captures according to another embodiment of the present
disclosure;
[0021] FIG. 11 is a flowchart illustrating a method for performing
a progressive approximation for aligning image captures according
to an embodiment of the present disclosure;
[0022] FIG. 12 is a block diagram schematically illustrating a
configuration of an electronic device according to an embodiment of
the present disclosure; and
[0023] FIG. 13 is a block diagram of an applications processor
configured to perform a progressive approximation for a
super-resolution image according to an embodiment of the present
disclosure.
[0024] Throughout the drawings, it should be noted that like
reference numbers are used to depict the same or similar elements,
features, and structures.
DETAILED DESCRIPTION
[0025] Detailed descriptions of various aspects of the present
disclosure will be discussed below with reference to the attached
drawings. The descriptions are set forth as examples only, and
shall not limit the scope of the present disclosure.
[0026] The detailed description with reference to the accompanying
drawings is provided to assist in a comprehensive understanding of
various embodiments of the disclosure as defined by the claims and
their equivalents. It includes various specific details to assist
in that understanding but these are to be regarded as merely
exemplary. Accordingly, those of ordinary skill in the art will
recognize that various changes and modifications of the embodiments
described herein can be made without departing from the scope and
spirit of the disclosure. In addition, descriptions of well-known
functions and constructions are omitted for clarity and
conciseness.
[0027] The terms and words used in the following description and
claims are not limited to the bibliographical meanings, but, are
merely used by the inventor to enable a clear and consistent
understanding of the disclosure. Accordingly, it should be apparent
to those skilled in the art that the following description of
various embodiments of the present disclosure are provided for
illustration purpose only and not for the purpose of limiting the
disclosure as defined by the appended claims and their
equivalents.
[0028] It is to be understood that the singular forms "a," "an,"
and "the" include plural referents unless the context clearly
dictates otherwise. Thus, for example, reference to "a component
surface" includes reference to one or more of such surfaces.
[0029] By the term "substantially" it is meant that the recited
characteristic, parameter, or value need not be achieved exactly,
but that deviations or variations, including for example,
tolerances, measurement error, measurement accuracy limitations and
other factors known to those of skill in the art, may occur in
amounts that do not preclude the effect the characteristic was
intended to provide.
[0030] Unless defined differently, all terms used in the present
disclosure, including technical or scientific terms, have meanings
that are understood generally by a person having ordinary skill in
the art. Ordinary terms that may be defined in a dictionary should
be understood to have the meaning consistent with their context,
and unless clearly defined in the present disclosure, should not be
interpreted to be excessively idealistic or formalistic.
[0031] According to various embodiments of the present disclosure,
an electronic device may include communication functionality. For
example, an electronic device may be a smart phone, a tablet
Personal Computer (PC), a mobile phone, a video phone, an e-book
reader, a desktop PC, a laptop PC, a netbook PC, a Personal Digital
Assistant (PDA), a Portable Multimedia Player (PMP), an MP3 player,
a mobile medical device, a camera, a wearable device (e.g., a
Head-Mounted Device (HMD), electronic clothes, electronic braces,
an electronic necklace, an electronic appcessory, an electronic
tattoo, or a smart watch), and/or the like.
[0032] According to various embodiments of the present disclosure,
an electronic device may be a smart home appliance with
communication functionality. A smart home appliance may be, for
example, a television, a Digital Video Disk (DVD) player, an audio,
a refrigerator, an air conditioner, a vacuum cleaner, an oven, a
microwave oven, a washer, a dryer, an air purifier, a set-top box,
a TV box (e.g., Samsung HomeSync.TM., Apple TV.TM., or Google
TV.TM.), a gaming console, an electronic dictionary, an electronic
key, a camcorder, an electronic picture frame, and/or the like.
[0033] According to various embodiments of the present disclosure,
an electronic device may be a medical device (e.g., Magnetic
Resonance Angiography (MRA) device, a Magnetic Resonance Imaging
(MRI) device, Computed Tomography (CT) device, an imaging device,
or an ultrasonic device), a navigation device, a Global Positioning
System (GPS) receiver, an Event Data Recorder (EDR), a Flight Data
Recorder (FDR), an automotive infotainment device, a naval
electronic device (e.g., naval navigation device, gyroscope, or
compass), an avionic electronic device, a security device, an
industrial or consumer robot, and/or the like.
[0034] According to various embodiments of the present disclosure,
an electronic device may be furniture, part of a
building/structure, an electronic board, electronic signature
receiving device, a projector, various measuring devices (e.g.,
water, electricity, gas or electro-magnetic wave measuring
devices), and/or the like that include communication
functionality.
[0035] According to various embodiments of the present disclosure,
an electronic device may be any combination of the foregoing
devices. In addition, it will be apparent to one having ordinary
skill in the art that an electronic device according to various
embodiments of the present disclosure is not limited to the
foregoing devices.
[0036] The term "super resolution" refers to a process of the
related art that is performed to enhance an image using a sequence
of captured frames of the image abstracted from a video. In the
related art, motion within the video sequence, either through
panning, or motion of objects within the sequence, is used to
introduce non-redundant additional samples of the image. This
sequence of images, or samples, is composited into a single static
image wherein the non-redundant samples are used to fill in a
higher order mosaic of potential sample point locations. From this,
a higher resolution image (i.e., an image having super-resolution)
may be generated.
[0037] As part of the present invention, it has been discovered
that when a human being attempts to hold a camera still for a
photograph, the user naturally "shakes" or moves the mobile
terminal in a random pattern around the desired framing position.
An aspect of this invention takes advantage of this natural
movement.
[0038] FIGS. 1A-1C illustrate the motion of a camera as a user
attempts to take a still shot of a stationary object according to
an embodiment of the present invention.
[0039] Referring to FIG. 1A, a mobile terminal was held in portrait
orientation and placed in burst mode to capture a still image. The
camera captured 20 shots of the subject of the image over a 1.5 sec
duration, the location of each shot denoted by a corresponding
numeral in FIG. 1A (i.e., the first shot was obtained while the
camera was at location "shot #1"). As evidenced from FIG. 1A, the
mobile terminal moved over a horizontal distance of 82 pixels and
appeared to have a random motion.
[0040] Referring to FIG. 1B, a mobile terminal was held in
landscape orientation and placed in burst mode to capture a still
image. Again, the camera captured 20 shots of the subject of the
image over a 1.5 sec duration, the location of each shot denoted by
a corresponding numeral in FIG. 1B (i.e., the first shot was obtain
while the camera was at location "shot #1"). As evidenced from FIG.
1B, the mobile terminal moved over a horizontal distance equal to
50 pixels of the camera and again appeared to have a random
motion.
[0041] Referring to FIG. 1C, a mobile terminal was placed in
continuous shooting mode for 5 sec. During that time, the locations
of 10 shots of the camera were considered and are illustrated in
FIG. 1C. As again evidenced, the motion of the camera appears to be
random.
[0042] As evidenced by FIGS. 1A-1C, when a user attempts to take a
still photo of a subject in burst mode, each successive shot or
exposure is slightly offset from the previous shot due to the
natural movement of the user. Being offset, the real world image
focused upon the image sensor may land in different phases on
photo-sites of the camera, thus taking a sample of the image at
different points with each exposure. When properly aligned with one
another, the multiple offset samples from each exposure effectively
form a virtual camera with a larger number of photo-sites. In other
words, a camera with higher resolution.
[0043] FIGS. 2A-2D illustrate a method of improving the resolution
of an image for display by overlapping frames of a captured image
according to an embodiment of the present disclosure.
[0044] Referring to FIGS. 2A-2D, it is assumed that a user is
attempting to capture an image of a stationary object that is
represented by the signal 201. In FIG. 2A, a first frame (i.e.,
frame 0) is captured by the digital camera. As is known in the art,
the digital camera used to capture the stationary image includes a
plurality of photo-sensors that are respectively located at set
positions (i.e., photo-sites) within the digital camera. In FIGS.
2A and 2B, the locations of the photo-sites corresponding to the
photo-sensors are illustrated by reference numerals 210-1, 210-2,
210-3, 210-4 and 210-5. Of course, it is understood that only five
photo-sites are illustrated for sake of brevity and convenience of
description and not by way of limitation. As illustrated in FIG.
2A, each photo-sensor located at photo-sites 210-1.about.210-5
samples the stationary object signal 201 to produce an output 220-0
corresponding to the stationary object 201 at that location.
[0045] In FIG. 2B, a second frame (i.e., frame 1) is captured by
the digital camera and each photo-sensor located at photo-sites
210-1.about.210-5 again samples the stationary object signal 201.
However, in FIG. 2B, while the location of the stationary object
remains the same, as evidenced by the stationary object signal 201,
the digital camera itself has moved as illustrated by offset 240.
In this case, when each photo-sensor located at photo-sites
210-1.about.210-4 again samples the stationary object signal 201,
the photo-sites produce an output 220-1 corresponding to the
stationary object 201, now at the offset location. Here, it is
assumed that the offset 240 is less than the pitch of photo-sites
within the digital camera.
[0046] FIG. 2C illustrates an overlapping of the samples 220-0 from
frame 0 and the samples 220-1 from frame 1. As can be seen in FIG.
2C, the overlapped samples 220-0 and 220-1 are illustrated relative
to the stationary object signal 201. That is, the overlapped
samples 220-0 are offset from corresponding samples 220-1 by offset
240.
[0047] In FIG. 2D, the samples 220-0 and 220-1 are averaged and the
results are graphed in comparison to the stationary object signal
201. As can be seen in FIG. 2D, the average of samples 220-0 and
220-1 produces a more accurate representation of the stationary
object signal 201 as compared to either the samples 220-0 of frame
0 or the sample 220-1 of frame 1, effectively doubling the
resolution of the digital camera and producing a higher quality
image.
[0048] FIGS. 3A-3C illustrate a process of capturing and aligning
two low resolution images according to an embodiment of the present
disclosure.
[0049] The example illustrated in FIGS. 3A-3C is substantially
similar to that illustrated in FIGS. 2A-2D. However, in FIGS.
3A-3C, an entire image is illustrated and considered, rather than
only a few photo-sites.
[0050] Referring to FIG. 3A, a first image 301A is captured as part
of a succession of image captures. In this example, the first image
301A of FIG. 3A is considered the reference image.
[0051] Referring to FIG. 3B, a second image 301B of the same
subject is captured as a second in the succession of image
captures. As discussed above, due to a natural movement of the
user, the second image 301B is offset from the first image 301A as
seen by the differences in proximity of the subject of the image to
the border of the image in each capture.
[0052] Referring to FIG. 3C, when the second image 301B is overlaid
on the first image 301A, an offset 310 can be seen. That is, when
the second image 301B of FIG. 3B is overlaid and aligned on the
first image 301A of FIG. 3A (i.e., the reference image), the images
are mismatched by the offset 310.
[0053] While only two image captures are illustrated in the
examples of FIGS. 2A-2D and FIGS. 3A-3C, it is to be understood
that this is merely for convenience and ease of description. In
implementation, the disclosure is not so limited and there may be
any number of image captures (i.e., frames) to be aligned for high
resolution composition. Furthermore, that the first image (e.g.,
301A), being the first image captured, is considered the reference
image is also only by way of example. In implementation, any of the
plurality of captured images may be considered the reference image.
Both of these aspects of the present disclosure will be discussed
in more detail below.
[0054] A first aspect of the present disclosure is to provide an
apparatus and method for aligning (i.e., registering) one image
relative to another image. In that regard, a progressive
approximation method is provided. In more detail, an apparatus and
method are provided that use a multi-resolution successive
approximation of the alignment. This method has the advantage of
being fast as well as robust in the presence of noise and repeating
structures in images that may "fool" other alignment methods. In
yet another embodiment of the present disclosure, the alignment
method includes a morphing routine to correct for changes due to
camera angle. This is important for handheld camera "snapshots"
that are likely to be taken at fairly close range.
[0055] FIG. 4 illustrates a progressive approximation for aligning
image captures according to an embodiment of the present
disclosure.
[0056] Referring to FIG. 4, a first image capture, denoted in FIG.
4 as "Image Capture A" and a second image capture, denoted as
"Image Capture B" are shown to illustrate an example of the
progressive approximation method of the present disclosure.
[0057] In the example of FIG. 4, both the Image Capture A and the
Image Capture B are first reduced in scale to 1/64 of their
original size wherein each pixel of each reduced image corresponds
to 64 pixels of the original image. After the first size reduction,
a rough displacement of Image Capture B in relation to Image
Capture A at 1/64 scale can be calculated. The calculated
displacement is a rough displacement based on the low resolution of
the reduced size images. After the first calculation is completed,
the images are again reduced in scale from their original size,
this time to 1/32 of their original size. Based on the first
calculated displacement, a more precise second displacement can be
calculated. This process continues until the images are returned to
their original scale, which, in this example, requires six
iterations. Notably, by doubling the scale with every iteration,
the accuracy of the offset calculation (i.e., the calculated
displacement) is also doubled.
[0058] FIGS. 5A and 5B illustrate two iterations of a progressive
approximation method according to an embodiment of the present
disclosure.
[0059] Referring to FIG. 5A, a first iteration of a progressive
approximation method compares a first image capture (Image A) with
a second image capture (Image B) to determine a rough displacement
of the second image capture with respect to the first image
capture. As illustrated in FIG. 5A, Image A includes a plurality of
low resolution pixels 501, as does Image B. To determine the rough
displacement in the first iteration, all pixels of Image B are
shifted in each of nine possible shift offsets 510 and each shifted
image is compared to Image A. Again, based on the reduction in size
of Image A and Image B, one pixel 501 corresponds to 64 pixels of
the original image. Hence, at that low scale, a shift of one pixel
is substantial when considering movement in the original image.
[0060] The nine possible shift offsets 510 in Image B include an
upper left offset (ul), an up offset (up), an upper right offset
(ur), a left offset (lf), a center or no offset (cn), a right
offset (rt), a lower left offset (ll), a down offset (dn), and a
lower right offset (lr). The first iteration determines which of
the nine possible shift offsets 510 will minimize the differences
between pixel values of Image A and Image B. In an embodiment of
the present disclosure, the difference in pixel values at each
offset is determined according to Equation (1).
cn = All Pixels ABS ( A - B no shift ) up = All Pixels ABS ( A - B
shifted up ) ur = All Pixels ABS ( A - B shifted upper right ) rt =
All Pixels ABS ( A - B shifted right ) ul = All Pixels ABS ( A - B
shifted upper left ) Equation ( 1 ) ##EQU00001##
[0061] In Equation (1), it is assumed that the offset determination
is performed with the full Red, Green, and Blue (RGB) data set when
comparing the pixel values of Image A and Image B. However, as an
alternative, the algorithms may be run on the green channels only
as a method of saving power or processing resources. In that case,
the offsets from the green channels can be used to composite all
channels of the pixels. The results of each determination of
Equation (1) are compared using Equation (2) to determine a value
of X.sub.offset and Y.sub.offset. offset.
MIN = min ( ul , up , ur , lf , cn , rt , ll , dn lr ) if cn = MIN
, then X offset = 0 , Y offset = 0 if up = MIN , then X offset = 0
, Y offset = - 1 if ur = MIN , then X offset = 1 , Y offset = - 1
if rt = MIN , then X offset = 1 , Y offset = 0 if ul = MIN , then X
offset = 1 , Y offset = 1 Equation ( 2 ) ##EQU00002##
[0062] Based on the results of Equation (2), a first displacement
(i.e., an X.sub.offset and a Y.sub.offset) of Image B with respect
to Image A can be determined. That is, although all pixels of Image
B are displaced according to the nine shift offsets, only one of
the shifted images is chosen based on the result of Equation (2).
Again, the displacement value is based on a scaled down image
(e.g., a 1/64.sup.th image) such that the displacement value is
only a rough approximation. To further refine the determination of
the displacement, a second and subsequent iterations are needed. In
the example of FIG. 5A, it is assumed that the smallest difference
between Image B and Image A occurs when Image B is shifted to the
right offset (i.e., rt) such that the X.sub.offset=1 and the
Y.sub.offset=0.
[0063] Referring to FIG. 5B, a second iteration of the progressive
approximation method is illustrated. In the second iteration, the
X.sub.offset and Y.sub.offset results from the first iteration are
used as a starting point and a displacement in each of nine
possible shift offsets 530 is again calculated. That is, the second
iteration continues from the previous offset location but the
resolution coordinates are now doubled to a scale of 1/32. Hence,
the previously determined X.sub.offset and Y.sub.offset based on a
1/64 scaling are doubled to reflect the current 1/32 scaling and
used as starting offsets for the second iteration. That is, the
starting X.sub.offset and Y.sub.offset for the second iteration are
determined using Equation (3).
X.sub.offset 1/32=X.sub.offset 1/64*2
Y.sub.offset 1/32=Y.sub.offset 1/64*2 Equation (3)
[0064] Using Equation (3) to determine the starting values for the
example of FIG. 5B, it is found that the starting value of
X.sub.offset=(1)*2=2 and the starting value of
Y.sub.offset=(0)*2=0. Hence, the Image B is further shifted by one
pixel coordinate in each of the nine possible shift offsets 530
from the starting location of (2,0), and Equation (1) and Equation
(2) are used to determine which offset of Image B results in a
smallest difference with Image A.
[0065] In the example of FIG. 5B, it is assumed that shifting Image
B to the lower left by one pixel results in the smallest difference
between Image A and Image B. According to Equation (2), a shift to
the lower left results in an X.sub.offset=-1 and a Y.sub.offset=1.
The offsets determined in the second iteration are added to the
offsets determined in the first iteration using the current
resolution coordinates. Hence, the X.sub.offset and Y.sub.offset
for the second iteration would be determined using Equation
(4).
X.sub.offset=X.sub.offset 1/64*2+new X.sub.offset
Y.sub.offset=Y.sub.offset 1/64*2+new Y.sub.offset Equation (4)
[0066] In the example of FIG. 5B, the result of Equation (4) would
be X.sub.offset=(1)*2+(-1)=1, and Y.sub.offset=(0)*2+(1)=1.
[0067] Although only two iterations are illustrated in FIGS. 5A and
5B, it is understood that the remaining iterations would follow
substantially the same process. That is, the values of X.sub.offset
and Y.sub.offset determined from the previous iteration and
modified by an equation similar to Equation (3) (i.e., multiplying
the value by 2) would be used as a starting point for further
shifting Image B by one pixel coordinate in each of nine possible
shift offsets. Equation (1) and Equation (2) would be used to
determine a minimum difference between Image A and Image B based on
the nine possible shifts, and an equation similar to Equation (4)
(i.e., previous offset*2+new offset) would be used to determine the
resultant offsets for the current iteration.
[0068] FIG. 6 illustrates a result of four iterations of a
progressive approximation according to an embodiment of the present
disclosure.
[0069] Referring to FIG. 6, a first iteration results in
displacement of the pixel coordinate towards the right offset
(i.e., rt), a second iteration results in displacement of the pixel
towards the lower left offset (i.e., ll), a third iteration results
in displacement of the pixel towards the upper offset (i.e., up)
and a fourth iteration again results in displacement of the pixel
towards the right offset (i.e., rt). Notably, with each iteration
illustrated in FIG. 6, the magnitude of the vector offset
decreases. That is, with each iteration, while the image is only
displaced by one pixel coordinate, the one pixel coordinate becomes
progressively smaller based on the increased scale of the image
(e.g., 1/64 to 1/32 to 1/16 to 1/8). Hence, starting with a 1/64
scale, an offset accurate to within 1 pixel of the native
resolution can be calculated after only seven iterations. Table 1
describes the relationship between the number of iterations, the
scale of the images, and a resultant error of the displacement
calculation in pixels.
TABLE-US-00001 TABLE 1 Iteration Scale Error 1 1/64 Offset
calculated to nearest 64 pixels; error = +/-32 pixels 2 1/32 Offset
calculated to nearest 32 pixels; error = +/-16 pixels 3 1/16 Offset
calculated to nearest 16 pixels; error = +/-8 pixels 4 1/8 Offset
calculated to nearest 8 pixels; error = +/-4 pixels 5 1/4 Offset
calculated to nearest 4 pixels; error = +/-2 pixels 6 1/2 Offset
calculated to nearest 2 pixels; error = +/-1 pixel 7 1 Offset
calculated to nearest pixel; error = +/-.5 pixels 8 2 Offset
calculated to nearest .5 pixels; error = +/-.25 pixels 9 4 Offset
calculated to nearest .25 pixels; error = +/-.125 pixels
[0070] To achieve a resultant image having super-resolution,
fractional pixel offsets are needed for insertion as samples
between pixels of the reference image. As can be seen in Table 1,
by the seventh iteration, the algorithm calculates a displacement
to the nearest pixel. To achieve the necessary fractional pixel
offsets, additional iterations are necessary. In one embodiment,
the algorithm is further designed to iterate two more times beyond
unity scaling. However, rather than using downscaled image data,
both Image A and Image B are `up-sampled` by a factor of two for
each iteration beyond unity scaling. This up-sampling is reflected
in Table 1.
[0071] In another embodiment, an optimization is performed in which
Image B is re-sampled by a 1/2 pixel in all directions and compared
to Image A at unity scale for the eighth iteration. For the final
iteration, Image B is re-sampled to the nearest 1/4 pixel in all
directions and again compared to Image A at unity scale.
[0072] FIGS. 7A and 7B illustrate an eighth iteration and ninth
iteration to determine fractional pixel offsets according to an
embodiment of the present disclosure.
[0073] Referring to FIG. 7A, a linear interpolation is performed on
Image B to re-sample the image by 1/2 pixel towards the right
(i.e., rt). Although not shown, the linear interpolation is
performed in each of the remaining 8 directions by 1/2 pixel. Once
each 1/2 pixel linear interpolation is completed, the resultant
re-sampled Image B that most closely matches Image A at unity scale
is selected. In the example of FIG. 7A, it is assumed that the 1/2
pixel linear interpolation towards the right offset (i.e., rt) most
closely matches Image A at unity scale.
[0074] Referring to FIG. 7B, based on the determination from FIG.
7A that the 1/2 pixel linear interpolation towards the right offset
(i.e., rt) most closely matched Image A at unity scale, Image B is
further re-sampled by 1/4 pixel linear interpolation in nine
possible directions from the right offset. That is, based on the
1/2 pixel re-sampling 701 that determined the right offset 703 most
closely matched Image A at unity scale, a 1/4 pixel re-sampling 705
is performed in nine possible offset directions, starting from the
right offset of iteration 8, and the closest match to Image A at
unity scale determines the final fractional offset.
[0075] Although two images are illustrated in the above examples
beginning with FIG. 4, this is merely for ease of discussion. In
implementation, any number of image captures may be used. However,
it has been discovered as part of the present disclosure that
several variables should be considered when determining the number
of image captures to consider. For example, if an increased
resolution of four times (i.e., 4.times.) the original image
resolution is desired, it would be necessary to populate a total of
16 frames (i.e., 4.times. horizontally and 4.times. vertically).
While the reference frame is available to populate one of the 16
frames, the super-resolution algorithm would ideally need 15 new
frames, each with unique fractional offsets relative to the
reference frame in order to populate all possible fractional states
in between the reference samples. To reliably populate all 16
fractional offset states in any given locality a total of 30 image
captures would be required. However, populating fewer than 16
fractional offset states still offers significant resolution
increases while requiring considerably fewer frame memory and
processing resources.
[0076] To better determine a number of image captures that should
be used to obtain a satisfactory image without imparting a
significant computation burden, A combinatorial model was
constructed to compute the cumulative probability of landing on a
certain number of unique fractional states (referred to as the
number of "Hits") within a certain number of captured frames. Every
new frame until the 4th frame reliably offers new resolution
information. Thus, four Hits (4=2.times. resolution) is highly
likely with only four frames, and seven captured frames reliably
produces six hits (6=2.4.times. resolution). To achieve greater
than 50% probability of 8 hits (8=2.8.times. resolution) at every
locality of the super-resolution image, one would need to invest at
least 10 frames of memory and processing.
[0077] Based on experimental results, it has been discovered that
it is generally efficient to populate half (or less than half) of
the available resolution states. Beyond this half way point, an
increasing number of redundancies occur resulting in progressively
less efficient use of available memory and processing resources.
Hence, for super-resolution system implementations with tight
constraints on frame memory and processing resources, it has been
found that 10 or fewer frame captures is sufficient.
[0078] It has further been discovered that the number of image
captures had an impact on noise reduction when performing the super
resolution algorithm according to embodiments of the present
disclosure. More specifically, it has been discovered that
super-imposing low-resolution frames has a significant noise
reduction benefit that increases proportionally by the square root
of the number of frames. Based on experimental results, it has been
discovered that when four frames are used for the super-resolution
algorithm, noise is reduced to 50% its original value. When 10
frames are used, noise is reduced to 31% its original value. And
when 25 frames are used, noise is reduced to 20% its original
value. In terms of Signal to Noise Ratio (SNR), SNR is increased
2.times. with four frames, 3.2.times. with 10 frames, and 5.times.
with 25 frames. Finally, in terms of SNR gain in (dB), four frames
increased SNR gain by 6 dB, 10 frames increased SNR gain by 10 dB
and 25 frames increased SNR gain by 14 dB.
[0079] Also for ease of discussion, the above examples beginning in
FIG. 4 illustrate a first iteration at a scale 1/64.sup.th the
original image. However, this too is not intended to be limiting.
Rather, it has been discovered as part of the present disclosure
that a down scale factor for the first iteration of the
registration algorithm should match the maximum offset of the frame
sequence (to the nearest power of 2). For example, in the
illustration of FIG. 1A, the maximum offset of the camera held in
the portrait orientation was 82 pixels. In that case, a down scale
factor of 64 (i.e., 1/64 scale) may be insufficient, although it
would be sufficient for the examples in FIGS. 1B and 1C in which
the maximum offsets were 49 pixels and 50 pixels, respectively.
Accordingly, while the down scale factor was to 64 (i.e., scaled to
1/64) in the above examples, it must be understood that this may be
easily adjusted based on many factors such as collection of more
product-specific user data, available resources and processing
costs, and the like.
[0080] As part of the present disclosure, it was discovered that
sequential image captures typically include perspective, scale,
rotation, and translation differences caused by random camera tilt
along various axes. Thus, an embodiment of the present disclosure
provides an image registration algorithm that not only aligns
(i.e., registers) sequential images but also transforms them to the
same camera perspective.
[0081] FIGS. 8A-8C illustrate two image captures requiring
additional morphing techniques according to an embodiment of the
present disclosure.
[0082] Referring to FIGS. 8A-8C, a subject 801 is shown in a first
image capture 810 in FIG. 8A, and the same subject 801 is shown in
a second, sequential image capture 820 in FIG. 8B. As illustrated
in FIG. 8C, the second image capture 820 has a linear distortion
with reference to the first image capture 810.
[0083] FIGS. 9A-9C illustrate a hybrid global and local
registration algorithm for linearly distorted frames according to
an embodiment of the present disclosure.
[0084] Referring to FIG. 9A, a second image capture 903 is
illustrated overlapping a first image capture 901 to determine a
rough offset and to define the rough intersection. In FIG. 9B, a
registration algorithm (with fractional offsets) is then run again
on four small local areas 905 (e.g., 128.times.128 pixels) at the
corners of the rough intersection. Finally, in FIG. 9C, the rough
global offset added to the four local offsets are used to define
four offset vectors corresponding to the center point of each
corner area 905. These four offset vectors are used to linearly
"distort" the second image capture 903 rather than simply
translating it, prior to composition of the super-resolution
image.
[0085] FIG. 10 illustrates a progressive approximation for aligning
image captures according to another embodiment of the present
disclosure.
[0086] Referring to FIG. 10, a first image capture, denoted in FIG.
10 as "Image Capture A" and a second image capture, denoted as
"Image Capture B" are shown to illustrate another example of a
progressive approximation method of the present disclosure. In the
example of FIG. 10, both the Image Capture A and the Image Capture
B are first reduced in scale to 1/81 of their original size. In
comparison to the embodiment illustrated in FIG. 4, the progressive
scaling has now been performed in powers three, rather than powers
of two (i.e., 1/64 scale, 1/32 scale, etc.). It has been discovered
that a first advantage of scaling in powers of three is that only
five iterations are required to reach unity scaling, as compared to
seven iterations when scaling by powers of two. Furthermore, when
scaling by powers of three, there is less redundancy in each search
state. However, it has been found that indexing when scaling by
powers of three is slightly more complex than when scaling by
powers of 2.
[0087] FIG. 11 is a flowchart illustrating a method for performing
a progressive approximation for aligning image captures according
to an embodiment of the present disclosure.
[0088] Referring to FIG. 11, data of N images captures is read in
to a memory, such as a buffer memory, for processing in operation
1101. In operation 1103, one of the N image captures is selected as
a reference image. As discussed above, each of the other image
captures will be compared to the reference image to determine its
offset. In an embodiment, a middle image capture of all the image
captures (e.g., image capture N/2) may be used as the reference
image capture. In operation 1105, a counter X is set to 1 and it is
determined in operation 1107 if the counter is less than the value
of N. The purpose of the counter in this instance is to ensure that
operations 1109 and 1111 are performed for each image capture with
the reference image, that is, each image pair. In operation 1109,
an offset of each image capture relative to the reference image
capture is determined. That is, as explained above, an iterative
algorithm is used to determine the offset of the image capture
relative to the reference image capture until unity scaling is
reached. In operation 1111, a fractional offset of each of four
corners of the image pair is determined. The fractional offsets
will be used for image morphing. In operation 1113, the counter is
increased by 1 and the process returns to operation 1107.
[0089] When it is determined in operation 1107 that all image pairs
have been considered, the process proceeds to operation 1115 at
which the counter is again set to 1. In operation 1117, it is
determined if the counter is less than or equal to the number of
image captures N. The purpose of the counter in this instance is to
ensure that operations 1119 to 1123 are performed for all image
captures. In operation 1119, morphing is performed of image capture
X according to the four corners determined in operation 1111. In
operation 1121, the image capture X is scaled up by a factor of
four to achieve a super-resolution for that image capture. In
operation 1123, the pixel sums for image capture X are read into a
sum buffer for later use in determining a final value for each
pixel. In operation 1125, the counter is increased by 1 and the
process returns to operation 1117.
[0090] When it is determined in operation 1117 that all image
captures have been considered, the process proceeds to operation
1127 at which the counter is set to the number of pixels. In
operation 1129, it is determined if the counter is equal to zero.
The purpose of the counter in this instance is to ensure that all
pixels of the image are considered. In operation 1131, an average
pixel value is determined based on the sum of pixel values
determined in operation 1123. In operation 1133, the counter is
increased by 1 and the process returns to operation 1129.
[0091] When it is determined in operation 1129 that all pixels have
been considered, the process proceeds to operation 1135 in which
image sharpening may be performed by an appropriate filter.
Notably, operation 1135 is optional.
[0092] FIG. 12 is a block diagram schematically illustrating a
configuration of an electronic device according to an embodiment of
the present disclosure.
[0093] Referring to FIG. 12, an electronic device 1200 may include
a control unit 1210, a storage unit 1220, an image processing unit
1230, a display unit 1240, an input unit 1250, a communication unit
1260, and a camera unit 1270.
[0094] According to various embodiments of the present disclosure,
the electronic device 1200 comprises at least one control unit
1210. The at least one control unit 1210 may be configured to
operatively control the electronic device 1200. For example, the at
least one control unit 1210 may control operation of the various
components or units included in the electronic device 1200. The at
least one control unit 1210 may transmit a signal to the various
components included in the electronic device 1200 and control a
signal flow between internal blocks of the electronic device 1200.
The at least one control unit 1210 may be or otherwise include at
least one processor. For example, the at least one control unit
1210 may include an Application Processor (AP), and/or the
like.
[0095] The storage unit 1220 may be configured to store user data,
and the like, as well a program which performs operating functions
according to various embodiments of the present disclosure. The
storage unit 1220 may include a non-transitory computer-readable
storage medium. As an example, the storage unit 1220 may store a
program for controlling general operation of an electronic device
1200, an Operating System (OS) which boots the electronic device
1200, and application program for performing other optional
functions such as a camera function, a sound replay function, an
image or video replay function, a signal strength measurement
function, a route generation function, image processing, and the
like. Further, the storage unit 1220 may store user data generated
according to a user of the terminal 1220, such as, for example, a
text message, a game file, a music file, a movie file, and the
like. According to various embodiments of the present disclosure,
the storage unit 1220 may store an application or a plurality of
applications that individually or in combination operate a camera
unit 1270 to capture (e.g., contemporaneously) one or more images
of substantially the same viewpoint, and/or the like. According to
various embodiments of the present disclosure, the storage unit
1220 may store an application or a plurality of applications that
individually or in combination operate the image processing unit
1230 or the control unit 1210 to perform any of the functions,
operations or steps as described above. The storage unit 1220 may
store an application or a plurality of applications that
individually or in combination operate the control unit 1210 and
the communication unit 1260 to communicate with a counterpart
electronic device to receive one or more images from the
counterpart electronic device, and/or the like. The storage unit
1220 may store an application or a plurality of applications that
individually or in combination operate display unit 1240 to display
a graphical user interface, an image, a video, and/or the like.
[0096] The display unit 1240 displays information inputted by a
user or information to be provided to the user as well as various
menus of the electronic device 1200. For example, the display unit
1240 may provide various screens according to the user such as an
idle screen, a message writing screen, a calling screen, a route
planning screen, and the like. According to various embodiments of
the present disclosure, the display unit 1240 may display an
interface which the user may manipulate or otherwise enter inputs
via a touch screen to enter selection of the function relating to
the signal strength of the electronic device 1200. The display unit
1240 can be formed as a Liquid Crystal Display (LCD), an Organic
Light Emitting Diode (OLED), an Active Matrix Organic Light
Emitting Diode (AMOLED), and the like. However, various embodiments
of the present disclosure are not limited to these examples.
Further, the display unit 1240 can perform the function of the
input unit 1250 if the display unit 1240 is formed as a touch
screen.
[0097] The input unit 1250 may include input keys and function keys
for receiving user input. For example, the input unit 1250 may
include input keys and function keys for receiving an input of
numbers or various sets of letter information, setting various
functions, and controlling functions of the electronic device 1200.
For example, the input unit 1250 may include a calling key for
requesting a voice call, a video call request key for requesting a
video call, a termination key for requesting termination of a voice
call or a video call, a volume key for adjusting output volume of
an audio signal, a direction key, and the like. In particular,
according to various embodiments of the present disclosure, the
input unit 1250 may transmit to the at least one control unit 1210
signals related to the operation of a camera unit (not shown), to
selection of an image, to selection of a viewpoint, and/or the
like. Such an input unit 1250 may be formed by one or a combination
of input means such as a touch pad, a touchscreen, a button-type
key pad, a joystick, a wheel key, and the like.
[0098] The communication unit 1260 may be configured for
communicating with other electronic devices and/or networks.
According to various embodiments of the present disclosure, the
communication unit 1260 may be configured to communicate using
various communication protocols and various communication
transceivers. For example, the communication unit 1260 may be
configured to communicate via Bluetooth technology, NFC technology,
WiFi technology, 2G technology, 3G technology, LTE technology, or
another wireless technology, and/or the like.
[0099] The camera unit 1270 may be configured to capture one or a
plurality of images and provide the data of the captured one or
more images to the control unit 1210 for processing.
[0100] FIG. 13 is a block diagram of an applications processor
configured to perform a progressive approximation for a
super-resolution image according to an embodiment of the present
disclosure.
[0101] Referring to FIG. 13, an applications processor 1300 may
include components of a conventional architecture such as a
pre-processor 1310, a Fully Integrated Mobile Display (FIMD) 1320,
and a CREO 1330. According to an embodiment of the present
disclosure, the applications processor 130 may further include an
Image Signal Processor (ISP) 1340, a super-resolution multi-frame
processor 1350, and a frame memory 1360.
[0102] The pre-processor 1310 may be configured to receive an input
1301, such as data from an image sensor or digital camera. After
processing the received data (e.g., lens shading, addressing
flicker, etc.), the pre-processor provides the pre-processed data
to the ISP 1340. The ISP 1340 performs additional functions on the
data such as conversion from RGB format to Ycbcr format, white
balancing, color saturation enhancement, and the like. Moreover,
the ISP 1340 provides the received data to the super-resolution
multi-frame processor 1350 for performing any or all of the
functions as described above. Based on the additional memory needs
of the super-resolution multi-frame processor 1350, a frame memory
1360 may also be provided. Upon completion of the super-resolution
processing, the super-resolution multi-frame processor 1350
provides an output 1305 including an image having an enhanced
resolution. The output may be provided to an external storage, a
display unit, and the like. In a normal mode, the ISP 1340 provides
an output for further processing to the FIIMD 1320 and the CREO
1330 which may ultimately provide an output signal to a display
unit, such as display unit 1240 for FIG. 12.
[0103] In an alternative embodiment, the pre-processor 1310 may
output raw data directly to the super-resolution multi-frame
processor 1350. In that case, the super-resolution multi-frame
processor 1350 may output super raw data to the ISP 1340. However,
this option requires the ISP 1340 to operate on a much larger data
set.
[0104] It will be appreciated that various embodiments of the
present disclosure according to the claims and description in the
specification can be realized in the form of hardware, software or
a combination of hardware and software.
[0105] Any such software may be stored in a non-transitory computer
readable storage medium. The non-transitory computer readable
storage medium stores one or more programs (software modules), the
one or more programs comprising instructions, which when executed
by one or more processors in an electronic device, cause the
electronic device to perform a method of the present
disclosure.
[0106] Any such software may be stored in the form of volatile or
non-volatile storage such as, for example, a storage device like a
Read Only Memory (ROM), whether erasable or rewritable or not, or
in the form of memory such as, for example, Random Access Memory
(RAM), memory chips, device or integrated circuits or on an
optically or magnetically readable medium such as, for example, a
Compact Disk (CD), Digital Versatile Disc (DVD), magnetic disk or
magnetic tape or the like. It will be appreciated that the storage
devices and storage media are various embodiments of non-transitory
machine-readable storage that are suitable for storing a program or
programs comprising instructions that, when executed, implement
various embodiments of the present disclosure. Accordingly, various
embodiments provide a program comprising code for implementing
apparatus or a method as claimed in any one of the claims of this
specification and a non-transitory machine-readable storage storing
such a program.
[0107] While the disclosure has been shown and described with
reference to various embodiments thereof, it will be understood by
those skilled in the art that various changes in form and details
may be made therein without departing from the spirit and scope of
the disclosure as defined by the appended claims and their
equivalents. Various embodiments of the present disclosure are
described as examples only and are noted intended to limit the
scope of the present disclosure. Accordingly, the scope of the
present disclosure should be understood as to include any and all
modifications that may be made without departing from the technical
spirit of the present disclosure.
* * * * *