U.S. patent application number 15/441145 was filed with the patent office on 2018-07-12 for image processing apparatus and method.
The applicant listed for this patent is Postech Academy-Industry Foundation. Invention is credited to Seung Yong LEE, Hyeong Seok SON.
Application Number | 20180197278 15/441145 |
Document ID | / |
Family ID | 62781925 |
Filed Date | 2018-07-12 |
United States Patent
Application |
20180197278 |
Kind Code |
A1 |
LEE; Seung Yong ; et
al. |
July 12, 2018 |
IMAGE PROCESSING APPARATUS AND METHOD
Abstract
Disclosed are an image processing apparatus and an image
processing method. The image processing method performed in the
image processing apparatus comprises receiving a blurred image;
generating an intermediate image from the blurred image, the
intermediate image including artifacts with less sensitivity to
changes in a blur kernel than the blurred image; and generating a
first corrected image by removing the artifacts of the intermediate
image using an artifact removal model. Therefore, it is made
possible to learn how to remove artifacts of the intermediate image
without the image processing apparatus learning a direct
deconvolution method.
Inventors: |
LEE; Seung Yong; (Pohang-si,
KR) ; SON; Hyeong Seok; (Pohang-si, KR) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Postech Academy-Industry Foundation |
Pohang-si |
|
KR |
|
|
Family ID: |
62781925 |
Appl. No.: |
15/441145 |
Filed: |
February 23, 2017 |
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
G06K 9/4609 20130101;
G06T 2207/20081 20130101; G06K 9/40 20130101; G06T 2207/10024
20130101; G06T 5/003 20130101; G06T 5/20 20130101; G06T 2207/20084
20130101 |
International
Class: |
G06T 5/00 20060101
G06T005/00; G06K 9/66 20060101 G06K009/66; G06K 9/46 20060101
G06K009/46; G06T 5/20 20060101 G06T005/20 |
Foreign Application Data
Date |
Code |
Application Number |
Jan 12, 2017 |
KR |
10-2017-0005150 |
Claims
1. A machine learning method in which an image processing apparatus
learns image processing, the method comprising: receiving a clear
test image; generating an intermediate image from a blurred test
image modified from the clear test image, the intermediate image
including artifacts with less sensitivity to changes in a blur
kernel than the blurred test image; and generating an artifact
removal model that removes artifacts of an image from a result of
machine learning performed by using the intermediate image and the
clear test image.
2. The machine leaning method according to claim 1, further
comprising generating the blurred test image from a sum of a
convolution of the clear test image and the blur kernel and a
noise.
3. The machine leaning method according to claim 1, wherein the
intermediate image is obtained by applying a Wiener filter to the
blurred test image.
4. The machine leaning method according to claim 1, wherein the
artifact removal model is generated from the result of the machine
learning using a convolutional neural network.
5. The machine leaning method according to claim 4, wherein the
convolutional neural network includes at least one convolution
layer for generating a plurality of characteristic maps using a
plurality of convolution filters.
6. The machine leaning method according to claim 4, wherein the
convolutional neural network includes at least one node for summing
an output result of an intermediate layer of the convolutional
neural network and an output result of a layer preceding the
intermediate layer.
7. The machine leaning method according to claim 4, wherein the
convolutional neural network includes a node for summing an output
result of a last layer of the convolutional neural network and data
of the intermediate image input to the convolutional neural
network.
8. The machine leaning method according to claim 3, wherein, in the
generating the artifact removal model, a back propagation process
of the convolutional neural network is performed in consideration
of a regularization term obtained from a gradient of an output
result of the convolutional neural network.
9. An image processing method performed in an image processing
apparatus, the method comprising: receiving a blurred image;
generating an intermediate image from the blurred image, the
intermediate image including artifacts with less sensitivity to
changes in a blur kernel than the blurred image; and generating a
first corrected image by removing the artifacts of the intermediate
image using an artifact removal model.
10. The image processing method according to claim 9, wherein the
intermediate image is obtained by applying a Wiener filter to the
blurred test image.
11. The image processing method according to claim 9, wherein the
artifact removal model removes the artifacts of the intermediate
image by using a convolutional neural network.
12. The image processing method according to claim 11, wherein the
convolutional neural network includes at least one convolution
layer for generating a plurality of characteristic maps using a
plurality of convolution filters.
13. The image processing method according to claim 11, wherein the
convolutional neural network includes at least one node for summing
an output result of an intermediate layer of the convolutional
neural network and an output result of a layer preceding the
intermediate layer.
14. The image processing method according to claim 11, wherein the
convolutional neural network includes a node for summing an output
result of a last layer of the convolutional neural network and data
of the intermediate image input to the convolutional neural
network.
15. The image processing method according to claim 9, further
comprising generating a second corrected image which corrects
detail of the first corrected image, wherein the second corrected
image satisfies a condition of minimizing a linear sum of a squared
error of the blurred image and a convolution of the second
corrected image and the blur kernel, and a squared error of the
first corrected image and the second corrected image.
16. An image processing apparatus deconvoluting a blurred image,
the apparatus comprising: a processor; and a memory storing at
least one instruction executed by the processor, wherein the at
least one instruction is configured to generate, from the blurred
image, an intermediate image including artifacts with less
sensitivity to changes in a blur kernel than the blurred image; and
generate a first corrected image by removing the artifacts of the
intermediate image using an artifact removal model.
17. The image processing apparatus according to claim 16, wherein
the at least one instruction is further configured to receive a
clear test image; generate, from a blurred test image modified from
the clear test image, an intermediate training image with less
sensitivity to changes in a blur kernel than the blurred test
image; and generate the artifact removal model from a result of
machine learning performed by using the intermediate training image
and the clear test image.
18. The image processing apparatus according to claim 16, wherein
the at least one instruction is further configured to remove the
artifacts of the intermediate image by using a convolutional neural
network, and the convolutional neural network includes a node for
summing an output result of a last layer of the convolutional
neural network and data of the intermediate image input to the
convolutional neural network.
19. The image processing apparatus according to claim 16, wherein
the at least one instruction is further configured to remove the
artifacts of the intermediate image by using a convolutional neural
network, and the convolutional neural network includes at least one
node for summing an output result of an intermediate layer of the
convolutional neural network and an output result of a layer
preceding the intermediate layer.
20. The image processing apparatus according to claim 16, wherein
the at least one instruction is further configured to generate a
second corrected image which corrects detail of the first corrected
image, wherein the second corrected image satisfies a condition of
minimizing a linear sum of a squared error of the blurred image and
a convolution of the second corrected image and the blur kernel,
and a squared error of the first corrected image and the second
corrected image.
Description
CLAIM FOR PRIORITY
[0001] This application claims priority to Korean Patent
Application No. 2017-0005150 filed on Jan. 12, 2017 in the Korean
Intellectual Property Office (KIPO), the entire contents of which
are hereby incorporated by reference.
BACKGROUND
1. Technical Field
[0002] The present disclosure relates to image processing apparatus
and method, and more particularly, to image processing apparatus
and method for correcting blurred images.
2. Related Art
[0003] A blurred image may be photographed by shaking or trembling
of a photographing apparatus during image photographing. At this
time, a blur kernel indicates a shaking path of the photographing
apparatus. It is also called non-blind deconvolution to obtain a
clear image when the blurred image and the blur kernel
corresponding to the blurred image are given.
[0004] Meanwhile, since most of the blurred images include noise,
it is a very difficult problem to solve the deconvolution problem
in consideration of noise. In recent years, as deep learning
techniques have evolved, attempts have been made to solve the
deconvolution problem using deep learning techniques.
[0005] However, since the deconvolution method is largely changed
according to influences of the noise and changes of the blur kernel
as described above, the method using the deep learning techniques
has also been limited in accuracy and calculation speed.
[0006] Accordingly, there is a demand for a technology capable of
performing the deconvolution at a high speed irrespective of the
type of the current blur kernel.
SUMMARY
[0007] Accordingly, exemplary embodiments of the present disclosure
are provided to substantially obviate one or more problems due to
limitations and disadvantages of the related art.
[0008] Exemplary embodiments of the present disclosure provide
image processing method and apparatus. More specifically, the
exemplary embodiments of the present disclosure provide image
processing apparatus and method for obtaining clear images from
blurred images.
[0009] Technical objects of the present disclosure are not limited
to the aforementioned technical objects and other technical objects
which are not mentioned will be apparently appreciated by those
skilled in the art from the following description.
[0010] In order to achieve the above-described objective, an aspect
of the present disclosure provides a machine learning method in
which an image processing apparatus learns image processing. The
method may comprise receiving a clear test image; generating an
intermediate image from a blurred test image obtained from the
clear test image, the intermediate image including artifacts with
less sensitivity to changes in a blur kernel than the blurred test
image; and generating an artifact removal model that removes
artifacts of an image from a result of machine learning performed
by using the intermediate image and the clear test image.
[0011] The method may further comprise generating the blurred test
image from a sum of a convolution of the clear test image and the
blur kernel and a noise.
[0012] The intermediate image may be obtained by applying a Wiener
filter to the blurred test image.
[0013] The artifact removal model may be generated from the result
of the machine learning using a convolutional neural network.
[0014] The convolutional neural network may include at least one
convolution layer for generating a plurality of characteristic maps
using a plurality of convolution filters.
[0015] The convolutional neural network may include at least one
node for summing an output result of an intermediate layer of the
convolutional neural network and an output result of a layer
preceding the intermediate layer.
[0016] The convolutional neural network may include a node for
summing an output result of a last layer of the convolutional
neural network and data of the intermediate image input to the
convolutional neural network.
[0017] In the generating the artifact removal model, a back
propagation process of the convolutional neural network may be
performed in consideration of a regularization term obtained from a
gradient of an output result of the convolutional neural
network.
[0018] In order to achieve the above-described objective, another
aspect of the present disclosure provides an image processing
method performed in an image processing apparatus. The method may
comprise receiving a blurred image; generating an intermediate
image from the blurred image, the intermediate image including
artifacts with less sensitivity to changes in a blur kernel than
the blurred image; and generating a first corrected image by
removing the artifacts of the intermediate image using an artifact
removal model.
[0019] The intermediate image may be obtained by applying a Wiener
filter to the blurred test image.
[0020] The artifact removal model may remove the artifacts of the
intermediate image by using a convolutional neural network.
[0021] The convolutional neural network may include at least one
convolution layer for generating a plurality of characteristic maps
using a plurality of convolution filters.
[0022] The convolutional neural network may include at least one
node for summing an output result of an intermediate layer of the
convolutional neural network and an output result of a layer
preceding the intermediate layer.
[0023] The convolutional neural network may include a node for
summing an output result of a last layer of the convolutional
neural network and data of the intermediate image input to the
convolutional neural network.
[0024] The method may further comprise generating a second
corrected image which corrects detail of the first corrected image,
wherein the second corrected image satisfies a condition of
minimizing a linear sum of a squared error of the blurred image and
a convolution of the second corrected image and the blur kernel,
and a squared error of the first corrected image and the second
corrected image.
[0025] In order to achieve the above-described objective, yet
another aspect of the present disclosure provides an image
processing apparatus deconvoluting a blurred image. The apparatus
may comprise a processor; and a memory storing at least one
instruction executed by the processor, wherein the at least one
instruction is configured to generate, from the blurred image, an
intermediate image including artifacts with less sensitivity to
changes in a blur kernel than the blurred image; and generate a
first corrected image by removing the artifacts of the intermediate
image using an artifact removal model.
[0026] The at least one instruction may be further configured to
receive a clear test image; generate, from a blurred test image
modified from the clear test image, an intermediate training image
with less sensitivity to changes in a blur kernel than the blurred
test image; and generate the artifact removal model from a result
of machine learning performed by using the intermediate training
image and the clear test image.
[0027] The at least one instruction may be further configured to
remove the artifacts of the intermediate image by using a
convolutional neural network, and the convolutional neural network
includes a node for summing an output result of a last layer of the
convolutional neural network and data of the intermediate image
input to the convolutional neural network.
[0028] The at least one instruction may be further configured to
remove the artifacts of the intermediate image by using a
convolutional neural network, and the convolutional neural network
includes at least one node for summing an output result of an
intermediate layer of the convolutional neural network and an
output result of a layer preceding the intermediate layer.
[0029] The at least one instruction may be further configured to
generate a second corrected image which corrects detail of the
first corrected image, wherein the second corrected image satisfies
a condition of minimizing a linear sum of a squared error of the
blurred image and a convolution of the second corrected image and
the blur kernel, and a squared error of the first corrected image
and the second corrected image.
[0030] According to the disclosed embodiments, it is made possible
to learn how to remove artifacts of an intermediate image without
an image processing apparatus learning a direct deconvolution
method. This feature facilitates machine learning of the image
processing apparatus and can apply the results of the machine
learning to images blurred by various blur kernels. Further, the
structure of the convolutional neural network can be improved, and
the accuracy of artifact removal can be enhanced. In addition, the
quality of the deconvolution can be improved by correcting the
detail of the image from which the artifacts are removed.
BRIEF DESCRIPTION OF DRAWINGS
[0031] Exemplary embodiments of the present disclosure will become
more apparent by describing in detail exemplary embodiments of the
present disclosure with reference to the accompanying drawings, in
which:
[0032] FIG. 1 is a block diagram illustrating an image processing
apparatus according to an exemplary embodiment of the present
disclosure;
[0033] FIG. 2 is a flow chart explaining a machine learning method
of an image processing apparatus according to an exemplary
embodiment of the present disclosure;
[0034] FIGS. 3A and 3B illustrate a comparison between a clear test
image and a blurred test image;
[0035] FIG. 4 is a conceptual diagram illustrating an example of a
convolutional neural network;
[0036] FIG. 5 is a conceptual diagram illustrating another example
of a convolutional neural network;
[0037] FIG. 6 is a conceptual diagram illustrating yet another
example of a convolutional neural network;
[0038] FIG. 7 is a conceptual diagram illustrating yet another
example of a convolutional neural network;
[0039] FIG. 8 is a flowchart explaining an example of a method of
processing an image by an image processing apparatus;
[0040] FIG. 9 is a flowchart illustrating another example of a
method of processing an image by an image processing apparatus;
[0041] FIG. 10 is a conceptual diagram illustrating a process of
deconvoluting a blurred image by an image processing apparatus;
[0042] FIG. 11 is a table showing comparison between artifact
removal performances according to convolutional neural network
structures;
[0043] FIGS. 12A to 12E illustrate images for explaining
performance of an image processing method according to an exemplary
embodiment of the present disclosure; and
[0044] FIGS. 13A to 13E illustrate other images for explaining
performance of an image processing method according to an exemplary
embodiment of the present disclosure.
DESCRIPTION OF EXEMPLARY EMBODIMENTS
[0045] The following detailed description is provided to assist the
reader in gaining a comprehensive understanding of the methods,
apparatuses, and/or systems described herein. However, various
changes, modifications, and equivalents of the systems, apparatuses
and/or methods described herein will be apparent to one of ordinary
skill in the art. Also, descriptions of functions and constructions
that are well known to one of ordinary skill in the art may be
omitted for increased clarity and conciseness.
[0046] Throughout the drawings and the detailed description, the
same reference numerals refer to the same elements. The drawings
may not be to scale, and the relative size, proportions, and
depiction of elements in the drawings may be exaggerated for
clarity, illustration, and convenience.
[0047] The features described herein may be embodied in different
forms, and are not to be construed as being limited to the examples
described herein. Rather, the examples described herein have been
provided so that this disclosure will be thorough and complete, and
will convey the full scope of the disclosure to one of ordinary
skill in the art.
[0048] Combinations of respective blocks in an accompanying block
diagram and respective operations in a flowchart may be performed
by computer program instructions. These computer program
instructions can be mounted on a processor of a general purpose
computer, a special purpose computer, or other programmable data
processing equipment, and thus the instructions performed by the
processor of the computer or other programmable data processing
equipment generate a means for performing functions described in
the respective blocks of the block diagram or the respective
operations of the flowchart. To implement functions in a specific
way, these computer program instructions can be stored in a
computer-usable or computer-readable memory capable of aiming for a
computer or other programmable data processing equipment, so that
the instructions stored in the computer-usable or computer-readable
memory can also produce a manufactured item including an
instruction means for performing functions described in the
respective blocks of the block diagram or the respective operations
of the flowchart.
[0049] In addition, each block or operation may indicate a part of
a module, a segment or a code including one or more executable
instructions for executing specific logical function(s). It should
be noted that mentioned functions described in blocks or operations
can be executed out of order in some alternative embodiments. For
example, two consecutively shown blocks or operations can be
performed substantially at the same time, or can be performed in a
reverse order according to the corresponding functions.
[0050] Hereinafter, exemplary embodiments according to the present
disclosure will be described in detail by referring to accompanying
drawings. However, the exemplary embodiments according to the
present disclosure may be changed into various forms, and thus the
scope of the present disclosure is not limited to the exemplary
embodiments which will be described. The exemplary embodiments are
provided to assist the one of ordinary skill in the art. in gaining
a comprehensive understanding of the methods, apparatuses, and/or
systems described herein.
[0051] In the present specification, a `blurred image` may mean an
image whose quality has deteriorated due to shaking of a
photographing apparatus during photographing. An image processing
apparatus according to exemplary embodiments of the present
disclosure receives a blurred image and corrects the blurred image
to obtain a clear image. Also, `artifact` refers to the noise that
appears in an image by artificial image processing. For example,
the noise contained in the blurred image may be amplified during
image processing of the blurred image, and the amplified noise may
correspond to the artifact. A `test image` means an image used by
the image processing apparatus as training data for learning a deep
learning algorithm. Test images may be included in a database for
learning. The test image may include a clear test image. In case
that the test image contains only a clear test image, the image
processing apparatus may produce a blurred test image that is sharp
but artificially blurred. The image processing apparatus may
utilize the clear test image and the blurred test image for
supervised learning. As another example, both clear test images and
blurred test images may be provided as training images and input to
the image processing apparatus.
[0052] The image processing apparatus may learn how to correct a
blurred test image by using the blurred test images and the clear
test images.
[0053] FIG. 1 is a block diagram illustrating an image processing
apparatus according to an exemplary embodiment of the present
disclosure.
[0054] Referring to FIG. 1, an image processing apparatus 100
according to an exemplary embodiment may include at least one
processor 110, a memory 120, a storage device 125, and the
like.
[0055] The processor 110 may execute program instructions stored in
the memory 120 and/or the storage device 125. The processor 110 may
refer to a central processing unit (CPU), a graphics processing
unit (GPU), or a dedicated processor on which the methods according
to the present disclosure are performed. The memory 120 and the
storage device 125 may be composed of a volatile storage medium
and/or a non-volatile storage medium. For example, the memory 120
may be comprised of a read only memory (ROM) and/or a random access
memory (RAM).
[0056] The memory 120 may be storing at least one instruction
executed by the processor 110. The memory 120 may store
instructions for performing an image processing method learned by
the processor 110. The processor 110 may update instructions stored
in the memory 120 by machine learning. The machine learning
performed by the processor 110 may be performed through a
supervised learning method.
[0057] The processor 110 may receive an image from an image
providing unit 10 in accordance with the instruction stored in the
memory 120. Here, the image providing unit 10 may be a
photographing apparatus. The image providing unit 10 may transmit
the photographed image to the processor 110. As another example,
the image providing unit 10 may receive an image from a user and
transmit the received image to the processor 110.
[0058] The processor 110 may process the image received from the
image providing unit 10 to improve the quality of the image. For
example, the processor 110 may remove blur and noise from the
image.
[0059] A blurred image may be expressed by a below equation 1.
y=x*v+n [Equation 1]
[0060] In the equation 1, `y` denotes a blurred image, `x` denotes
a clear image, and `n` denotes noise. Also, `v` means a blur
kernel, and means a convolution operator. The blur kernel v may be
determined by a shaking path of the photographing apparatus that
photographed the blurred image y.
[0061] Referring to the equation 1, the blurred image y may be
represented by a sum of the noise n and a convolution operation
result of the clear image x and the blur kernel v. The image
processing apparatus 100 may obtain the clear image x from the blur
kernel v and the blurred image y, and this process is called a
deconvolution as described above. However, deconvolution has two
difficulties. First, due to the noise n included in the blurred
image, there may be very many solutions of the clear image x for a
single blur kernel v. Therefore, the considerable amount of time
and calculation may be required to calculate the clear image x
directly from the blurred image y. Second, the blurred image y may
vary according to the blur kernel v. It may be difficult to learn a
general deconvolution method because the image processing process
should vary according to the blur kernel v. For example, even if a
deconvolution model is generated from an image blurred by a
specific blur kernel, applying the generated model to an image
blurred by another blur kernel may make it difficult to achieve the
desired result.
[0062] Hereinafter, a process in which the image processing
apparatus 100 according to the exemplary embodiment of the present
disclosure learns an image processing method will be described.
[0063] FIG. 2 is a flow chart explaining a machine learning method
of an image processing apparatus according to an exemplary
embodiment of the present disclosure.
[0064] Referring to FIG. 2, in a step S110, the image processing
apparatus 100 may receive a clear test image. In a step S115, the
processor 110 may artificially process a clear test image to
generate a blurred test image. That is, the processor 110 may
convolute the blur kernel to the clear test image and add noise to
create the blurred test image, and the processor 110 may generate
the blurred test image using Gaussian random noise.
[0065] In the step S115, the processor 110 may generate the blurred
test image using various types of blur kernels. The processor 110
may generate a plurality of blurred test images, which are used for
machine learning.
[0066] The processor 110 may store the clear test image and the
blurred test image in the memory 120. The processor 110 may use the
clear test image and the blurred test image stored in the memory
120 as training data.
[0067] Although FIG. 2 illustrates an example in which a blurred
test image is artificially generated by the processor 110 of the
image processing apparatus 100, various embodiments of the present
disclosure are not limited thereto. For example, the image
processing apparatus 100 may receive both a clear test image and a
blurred test image as learning data. In this case, the step S115
may be omitted.
[0068] FIGS. 3A and 3B illustrate a comparison between a clear test
image and a blurred test image.
[0069] FIG. 3A shows a blurred test image including blurring
effects and noises, and FIG. 3B shows a clear test image with no
blurring effects and noises. Blurred test images and clear test
images may be used for machine learning of artificial neural
networks.
[0070] The processor 110 may use the artificial neural network to
process the blurred test image and compare the processing result
with the clear test image. The processor 110 may compare the
processing result of the artificial neural network with the clear
test image and train the artificial neural network so that the
processing result becomes close to the clear test image. The
artificial neural network may include a convolutional neural
network which will be described later.
[0071] Referring again to FIG. 2, in a step 120, the processor 110
may generate an intermediate image containing artifacts from the
blurred test image.
[0072] The processor 110 may generate the intermediate image by an
image processing process that is relatively simple in operation
process than a complete deconvolution process. The processor may
generate the intermediate image from the blur kernel of the blurred
test image and the blurred test image.
[0073] The intermediate image may be an image with almost no
blurring caused by the blur kernel in the blurred test image.
However, the intermediate image may include artifacts of amplified
noises included in the blurred test image. The artifacts may be
generated in the process of generating the intermediate image.
[0074] As described above, the blurred test image may be changed
sensitively as the blur kernel changes. This may make it difficult
for the processor 110 to directly deconvolute the blurred test
image. However, the process in which the processor 110 generates
the intermediate image including the artifacts may be relatively
simple as compared to the deconvolution process. Also, the
artifacts contained in the intermediate image may be less sensitive
to changes in the blur kernel than the blurred test image. For
example, when the blur kernel is changed in the same way, the
change in the artifacts included in the intermediate image may be
smaller than the change in the blurred test image.
[0075] In a step S130, the processor 110 may learn an artifact
removal model for to removing the artifacts from the intermediate
image by comparing the intermediate image with the clear test
image. The artifact removal model may be applied to remove
artifacts of other images containing the artifacts. The processor
110 may generate the artifact removal model by training the
artificial neural network by machine learning. The processor 110
may then use the generated artifact removal model to remove
artifacts contained in any other images.
[0076] As shown in FIG. 2, the processor 110 may learn how to
remove the artifacts of the intermediate image which has been
processed from the blurred image, without learning how to
deconvolute the blurred image directly. Since the artifacts are
less sensitive to changes in the blur kernel, the processor 110 may
not have to learn blurred images by all of the blur kernels.
[0077] The artifact removal model may be utilized to deconvolute
the blurred images by various blur kernels. For example, the
processor 110 may generate an intermediate image from a blurred
image to be deconvoluted and then obtain a clear image by removing
artifacts of the intermediate image using a previously learned
artifact removal model. The process by which the processor 110
utilizes the artifact removal model to obtain a clear image will be
described in detail later.
[0078] For example, the processor 110 may generate the intermediate
image by applying a Wiener Filter to the blurred test image. The
Wiener filter is a filter designed to minimize the mean square
error of the outputs of the filter and expected desirable outputs
of the filter, and is used as a technical means for image
correction. The processor 110 may use the Wiener filter to create
the intermediate image that contains artifacts.
[0079] However, the above-described example is merely illustrative,
and various embodiments are not limited thereto. For example, the
processor 110 may use L2 regularization deconvolution (C J Schuler,
H C Burger, S. Harmeling, and B. Scholkopf, `A machine learning
approach for non-blind image deconvolution`, In Proc. CVPR, 2013)
may be used to generate the intermediate image.
[0080] The processor 110 may create an artifact removal model using
a convolutional neural network. The processor 110 may use the
convolutional neural network to learn how to remove artifacts from
the intermediate image. The artifact removal model that the
processor 110 has learned may be used to remove artifacts contained
in other images.
[0081] FIG. 4 is a conceptual diagram illustrating an example of a
convolutional neural network.
[0082] Referring to FIG. 4, the convolutional neural network may
comprise convolution layers CL1, CL2, CL3, CL4, CL5, and CL6 for
generating a plurality of characteristic maps using convolutional
filters, and activation layers AL1, AL2, AL3, and AL4 that adjust
weights of the convolution filters.
[0083] The convolution layers CL1, CL2, CL3, CL4, CL5, and CL6 may
each include a plurality of nodes. The first convolution layer CL1
may generate characteristic maps using different convolution
filters. The nodes of the first convolution layer CL1 may receive
the intermediate image containing artifacts and convolute the
intermediate image to generate characteristic maps by using the
convolution filters. The characteristic maps generated by the nodes
of the first convolution layer CL1 may be input to the first
activation layer AL1.
[0084] The first activation layer AL1 may impart nonlinear
characteristics to the output result of the first convolution layer
CL1. The convolutional network may learn non-linear functions or
parameters using the activation layers AL1, AL2, AL3, and AL4. The
activation layers AL1, AL2, AL3, and AL4 may utilize an activation
function. The activation function may include, but is not limited
to, a sigmoid function, a Rectified Linear Unit (RELU) function,
and the like.
[0085] The convolutional neural network may determine the weights
of nodes included in each of the convolutional layers. The nodes
included in each of the convolutional layers may generate
characteristic maps using different convolutional filters. The
convolutional neural network may adjust the weights of the
convolution filters to generate characteristic maps by adjusting
the weights of the nodes.
[0086] The processor 110 may change the weights of the nodes
included in each of the convolution layers by supervised learning.
For example, the processor 110 may compare the output result using
the input intermediate image with the clear test image, and use the
result as feedback information. The convolutional neural network
may change the weights of the nodes of each of the convolutional
layers based on the feedback information so that the output of the
convolutional neural network approaches the clear test image. For
example, the processor 110 may change the weights of the nodes
included in the sixth convolution layer CL6 based on the feedback
information. As the weights of the nodes included in the sixth
convolution layer CL6 are changed, the feedback information may be
changed. The processor 110 may change the weights of the nodes
included in the fifth convolution layer CL5 using the changed
feedback information. The processor 110 may change the weights of
the nodes included in the fourth convolution layer CL4 using the
changed feedback information while changing the weights of the
nodes included in the fifth convolution layer CL5. In the same
manner, the processor 110 may sequentially change weights of nodes
included in the third convolution layer CL3, the second convolution
layer CL2, and the first convolution layer CL1.
[0087] The processor 110 may repeat the process of changing the
weights of the nodes included in the convolution layers CL1, CL2,
CL3, CL4, CL5, and CL6 using the intermediate images generated from
the plurality of blurred test images. As described above, the
process in which the processor 110 changes the weights of the nodes
included in the convolution layers CL1, CL2, CL3, CL4, CL5, and CL6
may be referred to as a `back propagation process`.
[0088] The processor 110 may learn the convolutional neural network
through the back propagation process. The processor 110 may
generate an artifact removal model from the machine learning
results of the convolutional neural network.
[0089] The structure of the convolutional neural network utilized
by the processor 110 is not limited to the exemplary embodiment
shown in FIG. 4.
[0090] FIG. 5 is a conceptual diagram illustrating another example
of a convolutional neural network.
[0091] Referring to FIG. 5, a convolutional neural network may
include nodes N1, N2, N3, and N4 for summing an output result of an
intermediate layer with an output result of a layer ahead of the
intermediate layer. For example, the first node N1 may sum the
output result of the second convolution layer CL2 and the output
result of the first convolution layer CL1 preceding the second
convolution layer CL2. The result of the summing of the first node
N1 may be input to the second activation layer AL2.
[0092] As shown in FIG. 4, when the number of layers is simply
increased, a residual occurring between the layers may not be
utilized as feedback information. Accordingly, when the
convolutional neural network processes the intermediate image, if
distortion occurs in any one of the layers, distortion may
accumulate and result in erroneous output. On the contrary, as
shown in FIG. 5, when a convolutional neural network includes a
node for summing an output result of an intermediate layer and an
output result of a layer ahead of the intermediate layer, a
residual occurring between the layers may be reflected to the
feedback information. Therefore, the distortion occurring in the
layers may be suppressed, and the output result of the
convolutional neural network can be more accurate.
[0093] FIG. 6 is a conceptual diagram illustrating yet another
example of a convolutional neural network.
[0094] Referring to FIG. 6, a convolutional neural network may
include a node Nf for summing an output result of a last layer and
data of the intermediate image input to the convolutional neural
network. The node Nf sums up the output result of the last layer
and the data of the intermediate image so that the image output
from the convolutional neural network may preserve overall color
fidelity of the intermediate image. This makes the output of the
convolutional neural network more accurate.
[0095] FIG. 7 is a conceptual diagram illustrating yet another
example of a convolutional neural network.
[0096] Referring to FIG. 7, a convolutional neural network may
include nodes N1, N2, N3, and N4 for summing an output result of an
intermediate layer with an output result of a layer ahead of the
intermediate layer. In addition, the convolutional neural network
may include a node Nf that sums an output result of a last layer
with data of the intermediate image input to the convolutional
neural network. The nodes N1, N2, N3, and N4 may reflect residuals
occurring between layers, thereby suppressing a distortion
phenomenon that may occur in the intermediate layers. Also, the
node Nf may re-sum the data of the intermediate image input to the
convolutional neural network to the final result so that the image
output from the convolutional neural network can maintain overall
color fidelity of the intermediate image.
[0097] The processor 110 may calculate a mean squared error (MSE)
between the output result of the convolutional neural network and
the clear test image, and utilize it as feedback information. The
processor 110 may perform a back propagation process using the
feedback information. The processor 110 may sequentially change the
weights of the nodes included in the layers CL6 to CL1 of the
convolutional neural network using the feedback information.
[0098] However, if the processor 110 uses only the mean square
error as the feedback information, an overfitting phenomenon may
occur. The overfitting phenomenon is a phenomenon in which machine
learning is overly concentrated on the data used in learning,
resulting in a loss of generality. For example, when the
overfitting phenomenon occurs, the convolutional neural network may
perform fairly well in removing artifacts from the images used in
training, but artifacts contained in other images may not be
effectively removed.
[0099] In order to prevent such the phenomenon, the processor 110
may perform a back propagation process considering further a
regularization term.
[0100] The feedback information used by the processor 110 in the
back propagation process may be expressed by a below equation
2.
FB=1/2.parallel.x-f(y).parallel..sup.2+.alpha.R(f(y)) [Equation
2]
[0101] In the equation 2, `FB` denotes a feedback term used in the
back propagation process, `x` denotes a clear test image, `y`
denotes a blurred test image, `f(y)` denotes an output of the
convolutional neural network obtained when the intermediate image
generated from the blurred test image is input to the convolutional
neural network, `R(f(y))` denotes a regularization term, and
`.alpha.` denotes the weight of the regularization term.
[0102] The first term of the equation 2 corresponds to the
above-mentioned MSE. When a is set to 0, the processor 110 may
perform the back propagation process by considering only the MSE.
The processor 110 may perform the back propagation process by
considering only the MSE, but may further consider a regularization
term by setting a to a value greater than zero.
[0103] The regularization term shown in the equation 2 may be
defined by a below equation 3.
R(f(y))=.parallel..gradient.f(y).parallel..sup.p [Equation 3]
[0104] In the equation 3, the `y` denotes the blurred test image,
and the `f(y)` denotes the output result of the convolutional
neural network when the intermediate image generated from the
blurred test image y is input to the convolutional neural network.
The `R(f(y))` denotes the regularization term, and the
`.gradient.f` denotes a gradient of T. `p` is a parameter
indicating the degree of scarcity of the gradient and may have a
value of about 0.8. However, the presented value is merely
illustrative, and the value of p may be set differently depending
on the developer.
[0105] The regularization term R(f(y)) shown in the equation 3 may
make the gradient of the output image of the convolutional neural
network sparse. Generally, a clear image may have a rarity of
gradients. The processor 110 may reflect the regularization term
R(f(y)) to the back propagation process so that the output image of
the convolutional neural network has the general characteristics of
a clear image.
[0106] The processor 110 may change the weights of the nodes of the
convolution layer in the convolutional neural network using the
regularization term R(f(y)). Through this, the convolutional neural
network may output images with gradients that are rare, even if an
uneducated type of image is input to the convolutional neural
network. It is made possible to suppress the above-mentioned
overfitting phenomenon by making the gradient of the output image
of the convolutional neural network scarce.
[0107] Since an actual image is discrete in pixel units while the
gradient operation in the equation 3 is an operation defined for
the continuous function, it may be necessary to redefine the
gradient operation in the equation 3. Illustratively, the gradient
operation of the equation 3 may be defined according to a below
equation 4.
.differential. .gradient. f p .differential. f i , j = k , l
.di-elect cons. { - 1 , + 1 } p f i , j - f i + k , j + l p - 1 [
Equation 4 ] ##EQU00001##
[0108] In the equation 4, denotes a gradient of the image f, and
`p` is a parameter indicating the degree of scarcity of the
gradient, and may have a value of about 0.8. However, the presented
values is merely illustrative, and the value of p may be set
differently depending on the developer. Also, `i` and `j` are
indexes indicating the position of the pixels.
[0109] Referring to the equation 4, the gradient of the image f at
a pixel (i, j) may be defined as a difference between a pixel value
at the coordinate (i, j) and a pixel value at the coordinate (i+k,
j+k) adjacent to the coordinate (i, j).
[0110] The machine learning method of the image processing
apparatus 100 has been described above. According to the
above-described embodiments, the image processing apparatus 100 may
generate an intermediate image including artifacts from a blurred
test image, and learn an artifact removal method for removing the
artifacts of the intermediate image. According to the embodiments,
the image processing apparatus 100 may learn a method of
eliminating the artifacts of the intermediate image without
learning a method of directly correcting the blurred image into a
clear image. Thus, the machine learning process is facilitated, and
the machine learning result may be used for deconvolution of images
blurred by various blur kernels.
[0111] Hereinafter, a method of processing an image by the image
processing apparatus 100 using the above-described machine learning
result will be described. The image processing apparatus 100 may
receive a blurred image and output a clear image.
[0112] FIG. 8 is a flowchart explaining an example of a method of
processing an image by an image processing apparatus.
[0113] Referring to FIG. 8, in a step S210, the image processing
apparatus 100 may receive a blurred image from the image providing
unit 10. The image processing apparatus 100 may also receive a blur
kernel of the blurred image.
[0114] In a step S220, the processor 110 may generate an
intermediate image from the blurred image and the blur kernel of
the blurred image. The intermediate image may include artifacts.
The artifacts may be less sensitive to changes of the blur kernel
than those of the blurred image. The processor 110 may generate the
intermediate image from the blurred image using a Wiener filter. As
another example, the processor 110 may generate the intermediate
image using L2 regularization deconvolution or the like.
[0115] In a step S230, the processor 110 may generate a first
corrected image by removing the artifacts of the intermediate image
using a previously learned artifact removal model. The first
corrected image may have better image quality than the intermediate
image. The processor 110 may remove the artifacts of the
intermediate image using the convolutional neural network. The
processor 110 may remove the artifacts of the intermediate image
using at least one of the convolutional neural networks described
with reference to FIGS. 4 to 7.
[0116] The convolutional neural network may include nodes N1, N2,
N3, and N4 for summing the output result of the intermediate layer
with the output result of the layer ahead of the intermediate
layer.
[0117] The convolutional neural network may include a node Nf for
summing the output of the last layer of the convolutional neural
network and the data of the intermediate image input to the
convolutional neural network.
[0118] The processor 110 may terminate the deconvolution process in
the step S230. As another example, the processor 110 may further
increase deconvolution quality through additional post-processing
which will be described later.
[0119] FIG. 9 is a flowchart illustrating another example of a
method of processing an image by an image processing apparatus.
[0120] In the following description of the embodiment illustrated
in FIG. 9, the description overlapping with that of FIG. 8 is
omitted.
[0121] Referring to FIG. 9, in a step S240, the processor 110 may
generate a second corrected image that has corrected the details of
the first corrected image.
[0122] The processor 110 may generate the second corrected image
such that a linear sum of a squared error between the second
corrected image and the blurred image and a squared error between
the first corrected image and the second corrected image is
minimized.
[0123] The second corrected image may be determined by a below
equation 5.
arg min x y 1 - v * x 2 + .lamda. y 2 - x 2 [ Equation 5 ]
##EQU00002##
[0124] In the equation 5, `x` denotes the second corrected image,
`y.sub.1` denotes the blurred image input to the image processing
apparatus 100, `v` denotes the blur kernel, and `y.sub.2` denotes
the first corrected image output from the convolutional neural
network. `.lamda.` is a parameter that controls the ratio of the
two squared error terms and is a real number greater than zero.
[0125] In the equation 5, .parallel.y.sub.1-v*x.parallel..sup.2
represents a squared error (referred to as `first squared error`)
between the convolution of the second corrected image x and the
blur kernel v and the blurred image y.sub.1. The smaller the first
squared error, the closer the convolution of the second corrected
image x and the blur kernel v is to the blurred image y.sub.1. That
is, the smaller the first squared error, the more the blur effect
may be removed in the second corrected image.
[0126] In the equation 5, .parallel.y.sub.2-x.parallel..sup.2
represents a square error (referred to as `second squared error`)
between the first corrected image y.sub.1 and the second corrected
image y.sub.2. The smaller the second squared error, the better the
output of the convolutional network is preserved in the second
corrected image.
[0127] The processor 110 may improve the quality of the second
corrected image by generating the second corrected image such that
the linear sum of the two squared errors is minimized.
[0128] The equation 5 may be solved more easily in the Fourier
domain. The equation 5 may be expressed in the Fourier domain as
shown in a below equation 6.
X ^ = V _ .circle-w/dot. Y 1 + .lamda. Y 2 V 2 + .lamda. [ Equation
6 ] ##EQU00003##
[0129] In the equation 6, `X`, `V`, `Y.sub.1`, and `Y.sub.2`
represent the results of Fourier transform of x, v, y.sub.1, and
y.sub.2 in the equation 5. `V` represents a conjugate of V, and
`{circumflex over (X)}` represents X satisfying the condition of
the equation 5. Also, `.circle-w/dot.` denotes an element-wise
multiplication operator.
[0130] The processor 110 may increase the accuracy of the
deconvolution by obtaining the second corrected image from the
equation 6.
[0131] FIG. 10 is a conceptual diagram illustrating a process of
deconvoluting a blurred image by an image processing apparatus.
[0132] In a step S210, the image processing apparatus 100 may
receive a blurred image I1 for which restoration is required.
[0133] In the step S220, the processor 110 may generate an
intermediate image 12 from the blurred image I1. The intermediate
image 12 is clearer than the blurred image I1, but additional image
processing may be required since it includes artifacts.
[0134] In a step S230, the processor 110 may generate a first
corrected image 13 using an artifact removal model. The first
corrected image 13 may be sharper than the intermediate image 12 by
the artifact removal.
[0135] In a step S240, the processor 110 may generate a second
corrected image 14 using the first corrected image 13 and the
blurred image I1. The second corrected image 14 may be a result of
the correction of the details of the first corrected image 13, and
the second corrected image 14 may be clearer and of better quality
than the first corrected image 13.
[0136] The artifact removal model may remove the artifacts of the
intermediate image by the convolutional neural network described
with reference to FIGS. 4 to 7.
[0137] FIG. 11 is a table showing comparison between artifact
removal performances according to convolutional neural network
structures.
[0138] Referring to FIG. 11, when the artifacts are removed using
the convolutional neural network shown in FIG. 4, the peak
signal-to-noise ratio (PSNR) of the output image of the
convolutional neural network is measured to be about 27.26 dB.
[0139] In the case that the convolutional neural network includes
nodes N1, N2, N3, and N4 for summing the output result of the
intermediate layer and the output result of the layer preceding the
intermediate layer as shown in FIG. 5, the PSNR is measured to be
about 29.42 dB. That is, the convolutional neural network shown in
FIG. 5 may perform better than the convolutional neural network
shown in FIG. 4.
[0140] In the case that the convolutional neural network includes
the node Nf for summing the output result of the last layer and the
input intermediate image data as shown in FIG. 6, the PSNR is
measured to be about 29.58 dB. That is, the convolutional neural
network shown in FIG. 6 may perform better than the convolutional
neural networks shown in FIG. 4 and FIG. 5.
[0141] In the case that the convolutional neural network includes
the nodes N1, N2, N3, and N4 for summing the output result of the
intermediate layer and the output result of the layer preceding the
intermediate layer, and the node Nf for summing the output result
of the last layer and the input intermediate image data as shown in
FIG. 7, the PSNR is measured to be about 29.85B. That is, the
convolutional neural network shown in FIG. 7 may perform better
than the convolutional neural networks shown in FIGS. 4 to 6.
[0142] FIGS. 12A to 12E illustrate images for explaining
performance of an image processing method according to an exemplary
embodiment of the present disclosure.
[0143] FIG. 12A shows a blurred image, and FIG. 12E shows a clear
original image taken in an environment where there is little
fluctuation and noises.
[0144] FIG. 12B shows a result of deconvoluting the blurred image
by using a method proposed by U. Schmidt et al. in a thesis `a
discriminative non-blind deblurring` published in CVPR of 2013.
[0145] FIG. 12C shows a result of deconvoluting the blurred image
by using a method proposed by C. J. Schuler et al., in a thesis `a
machine learning approach for non-blind image deconvolution`
published in CVPR of 2013.
[0146] FIG. 12D shows a result of deconvoluting the blurred image
by using the image processing method according to an exemplary
embodiment of the present disclosure. The result shown in FIG. 12D
is obtained by using the convolutional neural network illustrated
in FIG. 7.
[0147] The left box in FIG. 12A, 12B, 12C, 12D, or 12E is an
enlarged view of a part of each image, and the box in the lower
right corner of FIG. 12A illustrates the blur kernel that produces
blurring effects on the blurred image.
[0148] Referring to FIGS. 12A to 12D, in the case of FIG. 12B, the
PSNR is only 29.20 dB although the sharpness is clearer than FIG.
12A, and the blurring effect may not be completely erased. Also,
FIG. 12C is clearer than FIG. 12A, its PSNR is only 27.35 dB, and
the blurring effect may not be completely erased.
[0149] Meanwhile, in the case of FIG. 12D obtained by the
embodiment of the present disclosure, there is almost no difference
from the clear original image shown in FIG. 12E as the blurring
effect is almost eliminated. Also, its PSNR is 30.20 dB, and thus
it shows remarkably improved quality as compared to the cases of
FIGS. 12B and 12C.
[0150] FIGS. 13A to 13E illustrate other images for explaining
performance of an image processing method according to an exemplary
embodiment of the present disclosure.
[0151] FIG. 13A shows a blurred image, and FIG. 13E shows a clear
original image taken in an environment where there is little
fluctuation and noises.
[0152] FIG. 13B shows a result of deconvoluting the blurred image
by using the method proposed by U. Schmidt et al., and FIG. 13C
shows a result of deconvoluting the blurred image by using the
method proposed by C. J. Schuler et al.
[0153] FIG. 13D shows a result of deconvoluting the blurred image
by using the image processing method according to an exemplary
embodiment of the present disclosure. The result shown in FIG. 13D
is obtained by using the convolutional neural network illustrated
in FIG. 7.
[0154] The left box in FIG. 13A, 13B, 13C, 13D, or 13E is an
enlarged view of a part of each image, and the box in the lower
right corner of FIG. 13A illustrates the blur kernel that produces
blurring effects on the blurred image.
[0155] Referring to FIGS. 13A to 13D, in the case of FIG. 13B, the
PSNR is only 25.46 dB even though the sharpness is clearer than
FIG. 13A, and the blurring effect may not be completely erased.
Also, although FIG. 13C is clearer than FIG. 13A, its PSNR is only
25.83 dB, and the blurring effect may not be completely erased.
[0156] Meanwhile, in the case of FIG. 13D obtained by the
embodiment of the present disclosure, there is almost no difference
from the clear original image shown in FIG. 13E as the blurring
effect is almost eliminated. Also, its PSNR is 27.85 dB, and thus
it shows remarkably improved quality as compared to the cases of
FIGS. 13B and 13C.
[0157] In the above description, the image processing apparatus
100, the machine learning method of the image processing apparatus
100, and the image processing method of the image processing
apparatus 100 according to the exemplary embodiments of the present
disclosure were explained with reference to FIGS. 1 to 13.
[0158] According to the disclosed embodiments, it is made possible
to learn how to remove artifacts of an intermediate image without
the image processing apparatus learning the direct deconvolution
method. This feature facilitates machine learning of the image
processing apparatus and can apply the results of the machine
learning to images blurred by various blur kernels. Further, the
structure of the convolutional neural network can be improved, and
the accuracy of artifact removal can be enhanced. Further, the
quality of the deconvolution can be improved by correcting the
detail of the image from which the artifacts are removed.
[0159] The methods according to forms of the present disclosure may
be implemented as program instructions executable by a variety of
computers and recorded on a computer readable medium. The computer
readable medium may include a program instruction, a data file, a
data structure, or a combination thereof. The program instructions
recorded on the computer readable medium may be designed and
configured specifically for the present disclosure or can be
publicly known and available to those who are skilled in the field
of computer software. Examples of the computer readable medium may
include a hardware device such as ROM, RAM, and flash memory, which
are specifically configured to store and execute the program
instructions. Examples of the program instructions include machine
codes made by, for example, a compiler, as well as high-level
language codes executable by a computer, using an interpreter. The
above exemplary hardware device can be configured to operate as at
least one software module in order to perform the operation of the
present disclosure, and vice versa.
[0160] While the embodiments of the present disclosure and their
advantages have been described in detail above, it should be
understood that various changes, substitutions and alterations may
be made herein without departing from the scope of the
disclosure.
* * * * *