U.S. patent application number 16/448662 was filed with the patent office on 2019-10-17 for method and apparatus for generating high dynamic range image.
The applicant listed for this patent is Huawei Technologies Co., Ltd.. Invention is credited to Fahd BOUZARAA, Ibrahim HALFAOUI, Onay URFALIOGLU.
Application Number | 20190318460 16/448662 |
Document ID | / |
Family ID | 57680271 |
Filed Date | 2019-10-17 |
![](/patent/app/20190318460/US20190318460A1-20191017-D00000.png)
![](/patent/app/20190318460/US20190318460A1-20191017-D00001.png)
![](/patent/app/20190318460/US20190318460A1-20191017-D00002.png)
![](/patent/app/20190318460/US20190318460A1-20191017-D00003.png)
![](/patent/app/20190318460/US20190318460A1-20191017-D00004.png)
![](/patent/app/20190318460/US20190318460A1-20191017-D00005.png)
![](/patent/app/20190318460/US20190318460A1-20191017-D00006.png)
![](/patent/app/20190318460/US20190318460A1-20191017-D00007.png)
![](/patent/app/20190318460/US20190318460A1-20191017-D00008.png)
![](/patent/app/20190318460/US20190318460A1-20191017-M00001.png)
![](/patent/app/20190318460/US20190318460A1-20191017-M00002.png)
View All Diagrams
United States Patent
Application |
20190318460 |
Kind Code |
A1 |
BOUZARAA; Fahd ; et
al. |
October 17, 2019 |
METHOD AND APPARATUS FOR GENERATING HIGH DYNAMIC RANGE IMAGE
Abstract
A method and an apparatus for generating a High Dynamic Range,
HDR, image are proposed. The method comprises obtaining a set of
two or more input images, the two or more input images including a
reference image and one or more non-reference images; for each of
the one or more non-reference images, performing an image analysis
which comprises, for each region of a plurality of regions of the
non-reference image, assessing whether the region of the
non-reference image and a corresponding region of the reference
image show the same image content and declaring the region of the
non-reference image as valid or as invalid based on the assessment;
and generating the HDR image by fusing the reference image and the
one or more non-reference images, wherein the fusing comprises, for
each of the one or more non-reference images, disregarding the
invalid regions of the respective non-reference image.
Inventors: |
BOUZARAA; Fahd; (Munich,
DE) ; URFALIOGLU; Onay; (Munich, DE) ;
HALFAOUI; Ibrahim; (Munich, DE) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Huawei Technologies Co., Ltd. |
Shenzhen |
|
CN |
|
|
Family ID: |
57680271 |
Appl. No.: |
16/448662 |
Filed: |
June 21, 2019 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
PCT/EP2016/082388 |
Dec 22, 2016 |
|
|
|
16448662 |
|
|
|
|
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
G06T 5/10 20130101; H04N
5/2355 20130101; G06T 5/008 20130101; G06T 2207/20221 20130101;
G06T 2207/20208 20130101; G06T 5/50 20130101; G06T 2207/20224
20130101 |
International
Class: |
G06T 5/00 20060101
G06T005/00; G06T 5/50 20060101 G06T005/50; G06T 5/10 20060101
G06T005/10 |
Claims
1. A method for generating a High Dynamic Range, HDR, image,
comprising: obtaining a set of two or more input images, the two or
more input images including a reference image and one or more
non-reference images; for each of the one or more non-reference
images, performing an image analysis which comprises, for each
region of a plurality of regions of the non-reference image,
assessing whether the region of the non-reference image and a
corresponding region of the reference image show the same image
content and declaring the region of the non-reference image as
valid or as invalid based on the assessment; and generating the HDR
image by fusing the reference image and the one or more
non-reference images, wherein the fusing comprises, for each of the
one or more non-reference images, disregarding the invalid regions
of the respective non-reference image; wherein the image analysis
comprises: generating a difference image of the non-reference image
by subtracting the non-reference image from the reference image or
vice versa; and generating a contrast-enhanced difference image by
applying a contrast enhancing transformation to the difference
image; wherein the assessing is on the basis of the
contrast-enhanced difference image.
2. The method of claim 1, wherein applying the contrast enhancing
transformation to the difference image comprises, for each region
of a plurality of regions of the difference image, applying a
sigmoid function to an intensity value of the region.
3. The method of claim 1, wherein the assessing whether the region
of the non-reference image and a corresponding region of the
reference image show the same image content comprises: comparing an
intensity value of a corresponding region of the difference image
against a threshold.
4. The method of claim 2, further comprising adapting the contrast
enhancing transformation according to a characteristic of the
reference image and the non-reference image.
5. The method of claim 4, wherein the characteristic includes a
color difference between the reference image and the non-reference
image.
6. The method of claim 3, wherein the method further includes
determining the threshold based on the non-reference image.
7. The method of claim 6, wherein the determining the threshold
includes: generating a histogram of the non-reference image,
wherein the histogram includes multiple bins and each of the
multiple bins covers a same range of intensity and has a bin
center; calculating a decrease of each bin of the multiple bins,
wherein the decrease is a difference between numbers of pixels
respectively at centers of two adjacent bins of the multiple bins;
identifying a bin from the multiple bins, wherein decrease of the
identified bin is larger than decrease of any non-identified bin of
the multiple bins; and calculating the threshold according to an
intensity of a point in the middle of two bin centers of the
identified bin and its next bin.
8. The method of claim 1, wherein the image analysis further
comprises performing a morphology operation on the invalid regions
of the respective non-reference image.
9. The method of claim 8, the morphology operation includes:
counting the invalid pixels inside a first window in the
contrast-enhanced image, wherein the first window is centered on an
invalid pixel; and declaring the invalid pixel on which the first
window is centered as valid or invalid according to the counted
number of invalid pixels inside the first window; and/or defining a
second window next to an invalid pixel in the contrast-enhanced
image; and declaring every pixel inside the second window as
invalid if a central pixel of the second window is an invalid
pixel.
10. An apparatus for generating a High Dynamic Range, HDR, image,
comprising a processor configured to perform the steps of claim
1.
11. An apparatus for generating a High Dynamic Range, HDR, image,
comprising: an obtaining unit configured to obtain a set of two or
more input images, the two or more input images including a
reference image and one or more non-reference images; an analysis
unit configured to perform, for each of the one or more
non-reference images, an image analysis which comprises, for each
region of a plurality of regions of the non-reference image,
assessing whether the region of the non-reference image and a
corresponding region of the reference image show the same image
content and declaring the region of the non-reference image as
valid or as invalid based on the assessment; and a generating unit
configured to generate the HDR image by fusing the reference image
and the one or more non-reference images, wherein the fusing
comprises, for each of the one or more non-reference images,
disregarding the invalid regions of the respective non-reference
image; wherein the analysis unit is configured to perform the image
analysis by: generating a difference image of the non-reference
image by subtracting the non-reference image from the reference
image or vice versa; and generating a contrast-enhanced difference
image by applying a contrast enhancing transformation to the
difference image; wherein the assessing is on the basis of the
contrast-enhanced difference image.
12. A computer program with program code means for performing the
method according to claim 1 if the program is executed on a
computer or on a digital signal processor.
13. A computer program product having a computer readable medium
with stored program code means for performing the method according
to claim 1 if the program is executed on a computer or by a digital
signal processor.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application is a continuation of International
Application No. PCT/EP2016/082388, filed on Dec. 22, 2016, the
disclosure of which is hereby incorporated by reference in its
entirety.
TECHNICAL FIELD
[0002] The present application refers to an apparatus and a method
for generating a high dynamic range image from at least two input
images.
[0003] Particularly, the present application refers to an apparatus
and a method which generates a high dynamic range image and which
involves detecting dynamic pixels in one or more of the input
images.
BACKGROUND
[0004] High Dynamic Range Imaging (HDRI) and Exposure Fusion (EF)
are methods of choice to computationally extend the dynamic range
of images depicting real world scenes. Unfortunately, those methods
are still prone to certain artifacts. Among others, the so-called
Ghost Effect is the most critical HDR limitation when it comes to
dealing with motion (camera or scene motion) in input Low Dynamic
Range (LDR) images.
[0005] In case of Exposure Fusion (EF), the input images are merged
using weighting maps which evaluate the saturation, exposedness and
contrast of the LDRs. This technique is based on the assumption
that the input-LDRs are aligned (static scene). However, real world
scenes are mostly dynamic and contain moving objects. This results
in Ghost Effects, where objects appear in several locations in the
final image.
[0006] This problem becomes more challenging when the input image
stack contains only a few images with large color differences,
which is the case in the mobile phone domain. To address this
issue, a de-ghosting step is required to preserve the quality of
the final HDR images.
[0007] There exists a lineup of deghosting methods based on
motion-maps which indicate the location of corresponding dynamic
pixels. These methods perform generally well when the input stack
offers a large number of differently exposed LDRs. In case of two
input images with large illumination difference, these methods
generally fail.
SUMMARY
[0008] The object of the present application is therefore to
provide a robust de-ghosting approach that performs efficiently in
many cases, in particular when only a few (e.g., two or three)
differently exposed images are available as input, also when these
images exhibit large illumination variations.
[0009] The above object is achieved by the solutions provided in
the enclosed independent claims. Advantageous implementations are
defined in the respective dependent claims.
[0010] A first aspect of the present application provides a method
for generating a High Dynamic Range (HDR) image, comprising:
[0011] obtaining a set of two or more input images, the two or more
input images including a reference image and one or more
non-reference images;
[0012] for each of the one or more non-reference images, performing
an image analysis which comprises, for each region of a plurality
of regions of the non-reference image, assessing whether that
region of the non-reference image and a corresponding region of the
reference image show the same image content and declaring that
region of the non-reference image as valid or as invalid based on
that assessment; and
[0013] generating the HDR image by fusing the reference image and
the one or more non-reference images, wherein the fusing comprises,
for each of the one or more non-reference images, disregarding the
invalid regions of the respective non-reference image.
[0014] The image analysis notably comprises:
[0015] generating a difference image of the non-reference image by
subtracting the non-reference image from the reference image or
vice versa (i.e., subtracting the reference image from the
non-reference image); and
[0016] generating a contrast-enhanced difference image by applying
a contrast enhancing transformation to the difference image.
[0017] The assessing is done on the basis of the contrast-enhanced
difference image.
[0018] A second aspect of the present invention provides an
apparatus for generating an HDR image, comprising a processor,
wherein the processor is configured to:
[0019] obtain a set of two or more input images, the two or more
input images including a reference image and one or more
non-reference images;
[0020] for each of the one or more non-reference images, perform an
image analysis which comprises, for each region of a plurality of
regions of the non-reference image, assessing whether that region
of the non-reference image and a corresponding region of the
reference image show the same image content and declaring that
region of the non-reference image as valid or as invalid based on
that assessment; and
[0021] generate the HDR image by fusing the reference image and
each of the one or more non-reference images, wherein the fusing
comprises, for each of the one or more non-reference images,
disregarding the invalid regions of the respective non-reference
image.
[0022] The image analysis further comprises:
[0023] generating a difference image of the non-reference image by
subtracting the non-reference image from the reference image or
vice versa; and
[0024] generating a contrast-enhanced difference image by applying
a contrast enhancing transformation to the difference image;
[0025] wherein the processor is configured to perform the assessing
on the basis of the contrast-enhanced difference image.
[0026] A third aspect of the present invention provides a computer
program with program code means for performing the method according
to the first aspect if the program is executed on a computer or a
digital signal processor is provided.
[0027] A fourth aspect of the present invention provides a computer
program product having a computer readable medium with stored
program code means for performing the method according to the first
aspect if the program is executed on a computer or a digital signal
processor is provided.
BRIEF DESCRIPTION OF THE DRAWINGS
[0028] FIG. 1 is a schematic flowchart of method for generating a
HDR image according to an embodiment of the present
application;
[0029] FIG. 2a is an example of the aforementioned embodiment;
[0030] FIG. 2b is an another example of the aforementioned
embodiment;
[0031] FIG. 3 is a flowchart of generating a motion map according
to an embodiment of the present application;
[0032] FIG. 4a is an example of a final motion map M;
[0033] FIG. 4b is another example of a final motion map M;
[0034] FIG. 5 schematically shows an example of an apparatus for
generating a HDR image;
[0035] FIG. 6 schematically shows an example of another apparatus
for generating a HDR image.
DESCRIPTION OF THE EMBODIMENTS
[0036] Illustrative embodiments of method, apparatus, and program
product for generating a HDR (High Dynamic Range) image are
described with reference to the figures. Although this description
provides a detailed example of possible implementations, it should
be noted that the details are intended to be exemplary and in no
way limit the scope of the application.
[0037] Moreover, an embodiment/example may refer to other
embodiments/examples. For example, any description including but
not limited to terminology, element, process, explanation and/or
technical advantage mentioned in one embodiment/example is
applicative to the other embodiments/examples.
[0038] For convenience of illustration, abbreviations and terms
listed below may be used in the following embodiments of this
invention as examples instead of limitations.
[0039] HM--Histogram Matching
[0040] HDR--High Dynamic Range
[0041] HDRI--High Dynamic Range Imaging
[0042] LDR--Low Dynamic Range
[0043] OF--Optical Flow
[0044] CRF--Camera Response Function
[0045] EF--Exposure Fusion
[0046] image--a visual representation of a real world or synthetic
scene by a digital camera; also referred to as picture.
[0047] pixel--the smallest addressable picture/image element.
[0048] window--a rectangular block of pixels extracted from an
image.
[0049] color mapping--(also known as color calibration, color
matching) is the operation of mapping the colors of an image
(source image) to the colors of another image (reference
image).
[0050] Image Histogram--Graphical illustration of the distribution
of the pixel color intensities of an image.
[0051] reference image--LDR image which belongs to the input stack
of images. The final HDR image is a modified version of the
reference image which has a wider dynamic range.
[0052] exposure--describes the amount of light gathered by the
capturing device (camera . . . ). A low-exposed image appears to be
dark and a high-exposed image appears to be bright.
[0053] dynamic Pixel--image pixel which belongs to a different part
of the captured scene, in comparison to a pixel with the same pixel
coordinates inside the reference image. Dynamic pixels typically
belong to motion objects. A dynamic pixel may also be called motion
pixel.
[0054] motion map--binary map which indicates the locations of
dynamic pixels in the corresponding non-reference image, in
comparison to the reference image.
[0055] Ghost-effect--type of image noise (artifact) in a HDR image
which results from merging several non-aligned input images
(non-aligned due to camera or scene motion). Motion-related objects
are depicted multiple times in the HDR, which create the ghost
effect.
[0056] The flowchart in FIG. 1 schematically illustrates a method
for generating a HDR image according to an embodiment. The method
comprises steps 102 to 103.
[0057] Step 101, obtaining a set of two or more input images, the
two or more input images including a reference image and one or
more non-reference images.
[0058] The input images are differently exposed images of a
scene.
[0059] The input images may be obtained by receiving from a device
or network accessible to an apparatus that carries out this method.
The input images may also be obtained by generating by the
apparatus. For example, the apparatus may generate the input images
using its camera.
[0060] Step 102, for each of the one or more non-reference images,
performing an image analysis which comprises, for each region of a
plurality of regions of the non-reference image, assessing whether
that region of the non-reference image and a corresponding region
of the reference image show the same image content and declaring
that region of the non-reference image as valid or as invalid based
on that assessment.
[0061] A region may be a pixel or a group of pixels.
[0062] Two regions of two images show the same image content if a
region in a first image and a region in a second image that is
geometrically identical to the first image correspond to each
other. Two regions correspond to each other if the regions are
identical in shape, size, and position relative to the image
corners.
[0063] The image analysis may further comprise the following
sub-steps: (102a) generating a difference image of the
non-reference image by subtracting the non-reference image from the
reference image or by subtracting the non-reference image from the
reference image; and (102b) generating a contrast-enhanced
difference image by applying a contrast enhancing transformation to
the difference image. In this case, the assessing above is on the
basis of the contrast-enhanced difference image.
[0064] In sub-step 102b, the process of applying the contrast
enhancing transformation comprises: for each region of a plurality
of regions of the difference image, applying a sigmoid function to
an intensity value of the region. The sigmoid function may, for
example, be a logistic function.
[0065] An example of assessing whether that region of the
non-reference image and a corresponding region of the reference
image show the same image content comprises: comparing an intensity
value of a corresponding region of the difference image against a
threshold.
[0066] Optionally, prior to performing the image analysis, the
following step may be performed:
[0067] For each of the one or more non-reference images, performing
an exposure transformation of the non-reference image or an
exposure transformation of the reference image prior to performing
the image analysis, to reduce an overall color or brightness
difference between the non-reference image and the reference
image.
[0068] Optionally, the contrast enhancing transformation may be
adapted according to a control parameter, such as a characteristic
of the reference image and the non-reference image. The
characteristic may include a color difference between the reference
image and the non-reference image.
[0069] Step 103, generating the HDR image by fusing the reference
image and each of the one or more non-reference images. The fusing
comprises, for each of the one or more non-reference images,
disregarding the invalid regions of the respective non-reference
image.
[0070] The disregarding may be performed, for example, by assigning
a weight of zero to every invalid region.
[0071] The threshold in step 102 may be determined based on the
non-reference image by following sub-steps 102i-102iv:
[0072] 102i, generating a histogram of the non-reference image,
wherein the histogram includes multiple bins and each of the
multiple bins covers a same range of intensity and has a bin
center;
[0073] 102ii, calculating a decrease of each bin of the multiple
bins, wherein the decrease is a difference between numbers of
pixels respectively at centers of two adjacent bins of the multiple
bins;
[0074] 102iii, identifying a bin from the multiple bins, wherein
decrease of the identified bin is larger than decrease of any
non-identified bin of the multiple bins; and
[0075] 102iv, calculating the threshold according to intensity of a
point in the middle of two bin centers of the identified bin and
its next bin.
[0076] Optionally, the image analysis in step 102 may further
comprise a morphology operation on the invalid regions of the
respective non-reference image. The morphology operation may
include:
[0077] counting the invalid pixels inside a first window in the
contrast-enhanced image, wherein the first window is centered on an
invalid pixel; and
[0078] declaring the invalid pixel on which the first window is
centered as valid or invalid according to the counted number of
invalid pixels inside the first window;
[0079] and/or
[0080] defining a second window next to an invalid pixel in the
contrast-enhanced image; and
[0081] declaring every pixel inside the second window as invalid if
a central pixel of the second window is an invalid pixel.
[0082] A dynamic pixel is an invalid pixel. Correspondingly, a
static pixel is a valid pixel.
[0083] This method provides for HDR de-ghosting based on a simple
yet very accurate algorithm for image analysis. The approach allows
for greater color difference (different exposures) as well as a
small stack of input images.
[0084] FIG. 2a illustrates an example of the aforementioned
embodiment. In this example, a pair of input images is processed.
It should be noted that this example will also work with more than
two input images.
[0085] Step 201, obtaining a pair of LDR images b and d.
[0086] Image b is a bright LDR, i.e. a LDR generated with
long-exposure. Image d is a dark LDR, i.e. a LDR generated with
short-exposure.
[0087] The images b and d exhibit scene differences and/or content
differences. Scene differences are generally related to
camera-motion or the nature of the capturing setup
(Stereo/Multi-camera setup). Content differences are caused by
moving objects.
[0088] The input images may be obtained in different ways as
described in the aforementioned step 101. In case of using a
camera, the input images can be captured simultaneously by using a
stereo/Multi-camera setup or sequentially by the same camera with
additional temporal dimension.
[0089] Step 202, detecting dynamic pixels in the input images to
generate a motion map M.
[0090] The dynamic pixels in the input images are related to scene
and/or camera motion.
[0091] The motion map is a binary mask, composed of zeros and ones.
Zero indicates a dynamic pixel, and one indicates a non-motion
(static) pixel. A dynamic pixel is invalid and a static pixel is
valid.
[0092] The process of detecting and generating the motion map is an
example of the image analysis in the aforementioned step 102.
Accordingly, the motion map in step 202 is an example of the result
obtained after performing the sub-step of declaring the respective
region of the non-reference image as valid (non-motion or static)
or as invalid (dynamic or motion) in the aforementioned step
102.
[0093] Step 203, designating one of images b and d as a reference
image.
[0094] In case of more than two input images, the reference image
is designated before step 202. In this case, step 203 can be
omitted since it has been already done before step 202.
[0095] Step 204, generating a final HDR image based on the motion
map and the reference image by using a modified version of Exposure
Fusion.
[0096] The final HDR represents a version of the reference LDR with
an extended dynamic range.
[0097] The motion map M is used to modify the exposure fusion
algorithm by including it (or motion maps in case of more than 2
LDRs) into the weighting maps (Wi(p) below) of the input
images:
W.sub.i(p)=(C.sub.i(p)).sup..omega..sup.C.times.(S.sub.i(p)).sup..omega.-
.sup.S.times.(E.sub.i(p)).sup..omega..sup.E.times.M.sub.i(p)
where Ci(p) is the contrast map for image i at pixel p, Si(p) is
the saturation map, Ei(p) is the exposedness map. The parameters
wc, ws and we represent the corresponding power values. Mi(p) is
the previously computed motion map of the image i. Image i may be
Image b or d. In the case of two input images, the motion map
corresponding to the designated reference image is composed of ones
("1") since the motion map of the reference image indicate that all
pixels are valid pixels. A valid pixel is a static pixel
(non-motion pixel) and is indicated by "1". Otherwise, "0"
indicates a dynamic pixel (motion pixel).
[0098] The computed motion map M is assigned to the non-reference
image. In the case of more than two input images, the weighting
maps of the non-reference images are set to zero for motion related
areas according to the equation, so that these pixels are excluded
from the final HDR image.
[0099] As mentioned above, step 203 may be performed before step
202 if there are more than two input images. Once the reference
image is selected, the algorithm in steps 202 and 204 applies on
every pair of reference and non-reference LDRs. In case of N input
images, the algorithm computes (N-1) motion maps, which will be
integrated, again as described earlier, into the final weighting
maps during the exposure fusion stage. These steps are summarized
in FIG. 2b.
[0100] FIG. 3 is a schematic flowchart of step 202 according to
another embodiment of the present application. The underlying idea
of the proposed motion detection algorithm is to explore the
difference image between image b and image d. There may be a large
color difference between Image b and image d that needs to be
reduced.
[0101] Step 301, generating a transformed image h based on images b
and d.
[0102] Image h has the same color properties as image b while has
the same content of image d, color difference between images b and
d is reduced.
[0103] Image h may be generated by using color mapping, such as
Histogram Matching (HM). HM is a low-complexity algorithm which
matches the Cumulative Distribution Function (CDF) of a source
image to the CDF of a target image with the desired color
distribution. In this embodiment, image d is considered as the
source image since it contains generally less saturated areas than
image b.
[0104] In case of two input images, step 301 does not influence the
choice of the reference image as describe Step 203 since the
transformed image h can be either assigned to image d or image
b.
[0105] Step 302, warping image b to image h to obtain a warped
image w.
[0106] Thereby, the content difference between images b and d is
reduced.
[0107] In this step, an optional global motion registration step
may be applied if the input images contain camera-related motion
(translation and rotation).
[0108] This step is based on the computation of a Homography matrix
H. To this end, SURF (Speeded Up Robust Features) may be used to
detect and extract features in image b and image h. Moreover,
RANSAC (Random sample consensus) may be used for the matching step.
Alternative features detection, extraction and matching techniques
might be deployed as well. Finally, image b is warped to the view
of image h using the computed matrix H, resulting in an image
w.
[0109] If no camera motion is detected, image w is a copy of image
b.
[0110] Step 303, determining an initial difference image I1_diff on
down-sampled versions of image w and image h. The initial
difference image I1_diff is an example of the difference image in
the aforementioned sub-step 101a.
[0111] The down-sampling step reduces color mapping noise and
artifacts. During this step, a Gaussian Pyramid may be used as a
low-pass filter. Additionally, the down-sampling decreases the
computational cost of the algorithm. Empirically, it is sufficient
to down-sample to 1 or 2 levels. The difference image is computed
according to following formulation (1):
I1.sub.diff(i,j)=|D(I.sub.h(i,j))-D(I.sub.w(i,j))| (1)
[0112] In the formulation (1), D represents the down-sampling
operator and (i, j) pixel coordinates.
[0113] The difference values of I1_diff can be classified into two
different classes:
[0114] (i) Difference values from motion related objects. These
values are generally large and less frequent.
[0115] (ii) Difference values originating from the inherent color
difference between image h and image w. These values are typically
smaller and more frequent.
[0116] Step 304, determining a final difference image I2_diff by
applying a contrast enhancing transformation to the initial
difference image I1_diff. The final difference image I2_diff is an
example of the contrast-enhanced difference image in the
aforementioned sub-step 101b.
[0117] Step 304 can accurately distinguish between the previously
mentioned difference values. To determine the final difference
image, contrast enhancing transformation is applied to the initial
difference image I1_diff. The contrast enhancing transformation may
be performed by using a sigmoid function, for example the following
logistic function (2):
I 2 diff ( i , j ) = 1 1 + k 1 e - k 2 ( I diff ( i , j ) - 0.5 ) (
2 ) ##EQU00001##
Where k1 and k2 are control parameters which can be set empirically
according to the characteristics of the scene (e.g. number of the
input images and/or color difference between the input images). For
example, k1 may be set to 0.09 and k2 may be set to 12. The digit
`0.5` is an example of control parameter k3 and may be replaced
with other value. The control parameters k1, k2 and k3 may be set
manually or automatically.
[0118] The values of k1, k2 and k3 are configured to ensure that 0
is mapped to 0 and 1 is mapped to 1. Therefore the manipulation
using the logistic function is a mapping from [0,1] to [0,1] or at
least approximates 0 and 1.
[0119] For example, value of k1 may be increased if the exposure
ratio between the input images is quite high. In case of high
exposure ratio, noisy pixels are created during the color mapping
stage using HM, which implies that the algorithm used in the color
mapping stage probably detected false positives. By increasing k1
in step 304, less dynamic pixels are detected and thus the noisy
pixels are removed. Therefore, accuracy of the whole process is
improved.
[0120] The logistic function allows for manipulating/enhancing the
contrast of the difference image, so that large difference values
corresponding to motion pixels are enhanced in comparison to
smaller difference values. This allows for better classification of
the difference values, through the accurate estimation of the
classification threshold(s). The impact of this step is s shown in
FIG. 4a and FIG. 4b.
[0121] Step 305, determining a threshold T_c based on the final
difference image I2_diff.
[0122] The threshold is an example of the threshold in the
aforementioned step 102 and may be determined by following
sub-steps 305a-305d:
[0123] 305a, generating a histogram of the final difference image
I2_diff, wherein the histogram includes multiple bins and each of
the multiple bins covers a same range of intensity and has a bin
center.
[0124] 305b, calculating a decrease of each bin of the multiple
bins, wherein the decrease is a difference between numbers of
pixels respectively at centers of two adjacent bins of the multiple
bins.
[0125] 305c, identifying a bin from the multiple bins, wherein
decrease of the identified bin is larger than decrease of any
non-identified bin of the multiple bins. This sub-step may be
described as following formulation (3).
T c = arg max T c i N p ( T c i ) - N p ( T c i + 1 ) , i = 0 , , B
- 2 ( 3 ) ##EQU00002##
[0126] In formulation (3), N.sub.p(T.sub.c.sup.i) is the number of
pixels around the bin center T.sub.c.sup.i of the bin number i out
of B bins. B is the total number of the bins. The value of B may be
10 so that each bin approximately covers a range of 25
intensities.
[0127] 305d, calculating the threshold according to intensity of a
point in the middle of two bin centers of the identified bin and
its next bin. Therefore, the threshold T_c is equal to
N w .ltoreq. w 2 2 : ##EQU00003##
[0128] The threshold T_c may be determined for each color channel.
These thresholds enable to distinguish between motion-related
difference values and HM-related values.
[0129] Step 306, generating an initial binary motion map M1
according to the threshold and the final difference image
I2_diff.
[0130] Accordingly, a pixel of the final difference image I2_diff
is marked as dynamic pixel (motion pixel) if at least one
difference value in I2_diff of a color channel c is larger than the
corresponding threshold T_c. This results in an initial binary
motion map M1, which indicates the location of the
motion-pixels.
[0131] Step 307, applying morphological operation on the initial
binary motion map M1 to generate a final motion map M. The final
motion map M here is an example of the motion map M in the
aforementioned step 202.
[0132] This operation aim at removing possible detection noise
(wrongly detected pixels) and enhance the shape and filling of
motion objects in the final motion map. The operation may comprise
any one or both of following sub-steps 307a-307b:
[0133] 307a, counting the number the invalid pixels Nw inside a
first window in the initial motion map M1, wherein the first window
is centered on an invalid pixel (motion pixel), and declaring the
invalid pixel (motion pixel) on which the first window is centered
as valid or invalid according to the counted number of invalid
pixels inside the first window.
[0134] The size of the first window may be set to 3 or 5. The
processing of the declaring may be done as following:
[0135] if
N w > w 2 2 : ##EQU00004##
the motion pixel will be discarded, that is, is not marked as
motion pixel.
[0136] if
T c i + 1 - T c i 2 . ##EQU00005##
the motion pixel is confirmed as a motion pixel.
[0137] 307b, defining a second window next to an invalid pixel in
the initial motion map M1, and
[0138] declaring every pixel inside the second window as invalid if
a central pixel of the second window is an invalid pixel. Likewise,
the size of the second window may also be set to 3 or 5.
[0139] The morphological operation enables to fill-up possible
missing motion-pixels inside motion objects, and thus improves the
shape of these objects in the final motion map M.
[0140] Examples of the final motion map M are shown in FIGS.
4a-4b.
[0141] FIG. 4a illustrates the visual impact of the logistic
function-based manipulation of the difference image on the final
motion map. Picture (a) shows a ground truth motion map. Picture
(b) shows a motion map without manipulation. Picture (c) shows a
motion map with the proposed manipulation.
[0142] In FIG. 4b, picture (c) shows a final motion map obtained
based on input images (a) and (b). In picture (c), black indicates
dynamic Pixels and white indicates static pixels.
[0143] FIG. 5 shows an embodiment of apparatus for generating an
HDR image. The apparatus generally comprises a processor that is
configured to perform the steps 101-103. In an example, the
processor is configured to perform the aforementioned steps
201-204. In particular, the processor is configured to perform step
202 as described in step 301-307.
[0144] FIG. 6 shows another embodiment of apparatus for generating
a HDR image. The apparatus comprises:
[0145] an obtaining unit 601 configured to perform the
aforementioned step 101;
[0146] an analysis unit 602 configured to perform the
aforementioned step 102; and
[0147] a generating unit 603 configured to perform the
aforementioned step 103.
[0148] In an example, the analysis unit 602 is configured to
perform the aforementioned steps 202-203. In particular, the
analysis unit 602 is configured to perform step 202 as described in
step 301-307.
[0149] The subject matter described above can be implemented in
software in combination with hardware and/or firmware. For example,
the subject matter described herein may be implemented in software
executed by one or more processors. In one exemplary
implementation, the subject matter described herein may be
implemented using a non-transitory computer readable medium having
stored thereon computer executable instructions that when executed
by the processor of a computer control the computer to perform
steps. Exemplary computer readable media suitable for implementing
the subject matter described herein include non-transitory computer
readable media, such as disk memory devices, chip memory devices,
programmable logic devices, and application specific integrated
circuits. In addition, a computer readable medium that implements
the subject matter described herein may be located on a single
device or computing platform or may be distributed across multiple
devices or computing platforms.
* * * * *