U.S. patent application number 15/518404 was filed with the patent office on 2018-07-12 for image upsampling system, training method thereof and image upsampling method.
This patent application is currently assigned to BOE TECHNOLOGY GROUP CO., LTD.. The applicant listed for this patent is BOE TECHNOLOGY GROUP CO., LTD.. Invention is credited to Jianmin HE, Pablo NAVARRETE MICHELINI, Lijie ZHANG.
Application Number | 20180197037 15/518404 |
Document ID | / |
Family ID | 54668007 |
Filed Date | 2018-07-12 |
United States Patent
Application |
20180197037 |
Kind Code |
A1 |
NAVARRETE MICHELINI; Pablo ;
et al. |
July 12, 2018 |
IMAGE UPSAMPLING SYSTEM, TRAINING METHOD THEREOF AND IMAGE
UPSAMPLING METHOD
Abstract
An image upsampling system, a training method thereof and an
image upsampling method are provided, the feature images of an
image are obtained by using the convolutional network, upsampling
processing is performed on the images with the muxer layer to
synthesize every n.times.n feature images in the input signal into
a feature image with the resolution amplified by n.times.n times,
in the upsampling procedure with the muxer layer, information of
respective feature images in the input signal is recorded in the
generated feature image(s) without loss; and thus, every time when
the image passes through a muxer layer with an upsampling multiple
of n, the image resolution can be increased by n.times.n times.
Inventors: |
NAVARRETE MICHELINI; Pablo;
(Beijing, CN) ; ZHANG; Lijie; (Beijing, CN)
; HE; Jianmin; (Beijing, CN) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
BOE TECHNOLOGY GROUP CO., LTD. |
Beijing |
|
CN |
|
|
Assignee: |
BOE TECHNOLOGY GROUP CO.,
LTD.
Beijing
CN
|
Family ID: |
54668007 |
Appl. No.: |
15/518404 |
Filed: |
March 2, 2016 |
PCT Filed: |
March 2, 2016 |
PCT NO: |
PCT/CN2016/075338 |
371 Date: |
April 11, 2017 |
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
G06K 9/4628 20130101;
G06T 3/4053 20130101; G06N 3/0454 20130101; G06N 3/08 20130101;
G06T 3/40 20130101; G06K 9/6289 20130101; G06N 3/04 20130101 |
International
Class: |
G06K 9/46 20060101
G06K009/46; G06N 3/04 20060101 G06N003/04; G06N 3/08 20060101
G06N003/08; G06K 9/62 20060101 G06K009/62; G06T 3/40 20060101
G06T003/40 |
Foreign Application Data
Date |
Code |
Application Number |
Sep 17, 2015 |
CN |
201510595656.7 |
Claims
1: An image upsampling system, comprising: at least one first
convolutional network and at least one muxer layer that are
cascaded; wherein an signal input end of the image upsampling
system is connected with a signal input end of a first
convolutional network in the at least one first convolutional
network, and a signal output end of the image upsampling system is
connected with a signal output end of a last muxer layer in the at
least one muxer layer; a signal input end of every muxer layer in
the at least one muxer layer is connected with a signal output end
of a first convolutional network located in a stage prior to the
muxer layer in the at least one first convolutional network, or
connected with a signal output end of another muxer layer located
in a stage prior to the muxer layer in the at least one muxer
layer; the first convolutional network is configured for converting
an image input to its signal input end into a plurality of feature
images and outputting the feature images to the signal input end of
the muxer layer connected therewith; the muxer layer is configured
for synthesizing every n.times.n feature images in the feature
images input to its signal input end into a feature image whose
resolution is n.times.n times that of the input feature image and
outputting the same; and a number of feature images input to the
muxer layer is a multiple of n.times.n, n being an integer greater
than one.
2: The image upsampling system according to claim 1, wherein a
number of muxer layers is two or three.
3: The image upsampling system according to claim 1, wherein, a
signal input end of each muxer layer is respectively connected with
a signal output end of one corresponding first convolutional
network in the at least one first convolutional network.
4: The image upsampling system according to claim 1, wherein, in
the case where there are provided a plurality of muxer layers, the
muxer layers have a same upsampling multiple.
5: The image upsampling system according to claim 1, wherein, the
muxer layer has an upsampling multiple which is a prime number.
6: The image upsampling system according to claim 5, wherein, the
muxer layer has an upsampling multiple which is 2.
7: The image upsampling system according to claim 1, wherein, the
muxer layer is a self-adaptive interpolation filter.
8: The image upsampling system according to claim 1, further
comprising: a second convolutional network, whose signal input end
is connected with a signal output end of the last muxer layer in
the at least one muxer layer, and whose signal output end is
connected with the signal output end of the image upsampling
system; wherein the second convolutional network is configured for
optimizing picture quality of the feature images output by the
muxer layer.
9: The image upsampling system according to claim 8, wherein, the
first convolutional network and the second convolutional network
include at least one convolution layer composed of a plurality of
filter units.
10: A display device, comprising the image upsampling system
according to claim 1.
11: A training method of the image upsampling system according to
claim 1, comprising: initializing respective parameters in the
image upsampling system; by using an original image signal as an
output signal of the image upsampling system and using an image
signal obtained by down-sampling the original image signal as an
input signal of the image upsampling system, adjusting the
respective parameters in the image upsampling system to allow the
down-sampled image signal subjected to upsampling processing with
the adjusted respective parameters to be the same as the original
image signal.
12: The training method according to claim 11, wherein,
initializing of the respective parameters in the image upsampling
system includes: initializing weights W.sub.ij of respective filter
units of respective convolution layers of the first convolutional
network and the second convolutional network in the image
upsampling system according to a formula below: W ij = { 1 / ( m )
( i , j ) are preset anchor pixel 0 other pixel } ##EQU00005##
where m represents the number of feature images input to the filter
unit; and initializing the biases of respective filter units to
0.
13: The training method according to claim 11, wherein,
initializing of respective parameters in the image upsampling
system includes: initializing the weights W.sub.ij of respective
filter units of respective convolution layers of the first
convolutional network and the second convolutional network in the
image upsampling system according to a formula below: W ij = W ij '
+ uniform ( - 1 , 1 ) m ; ##EQU00006## W ij ' = { 1 / ( m ) ( i , j
) are preset anchor pixel 0 other pixel } ##EQU00006.2## where m
represents the number of feature images input to the filter unit,
and uniform (-1,1) represents a random number selected between
(-1,1); and initializing the biases of respective filter units to
0.
14: A method for performing image upsampling with the image
upsampling system according to claim 1, comprising: converting, by
a first convolutional network, an input image input to the first
convolutional network into a plurality of feature images and
outputting the feature images; synthesizing, by a muxer layer,
every n.times.n feature images in the feature images input to the
muxer layer into a feature image with a resolution amplified by
n.times.n times as larger as the input feature images, and
outputting the same; the number of feature images input to the
muxer layer being a multiple of n.times.n, and n being an integer
greater than 1.
15: The image upsampling system according to claim 2, further
comprising: a second convolutional network, whose signal input end
is connected with a signal output end of the last muxer layer in
the at least one muxer layer, and whose signal output end is
connected with the signal output end of the image upsampling
system; wherein the second convolutional network is configured for
optimizing picture quality of the feature images output by the
muxer layer.
16: The image upsampling system according to claim 15, wherein, the
first convolutional network and the second convolutional network
include at least one convolution layer composed of a plurality of
filter units.
17: The image upsampling system according to claim 3, further
comprising: a second convolutional network, whose signal input end
is connected with a signal output end of the last muxer layer in
the at least one muxer layer, and whose signal output end is
connected with the signal output end of the image upsampling
system; wherein the second convolutional network is configured for
optimizing picture quality of the feature images output by the
muxer layer.
18: The image upsampling system according to claim 17, wherein, the
first convolutional network and the second convolutional network
include at least one convolution layer composed of a plurality of
filter units.
19: The image upsampling system according to claim 4, further
comprising: a second convolutional network, whose signal input end
is connected with a signal output end of the last muxer layer in
the at least one muxer layer, and whose signal output end is
connected with the signal output end of the image upsampling
system; wherein the second convolutional network is configured for
optimizing picture quality of the feature images output by the
muxer layer.
20: The image upsampling system according to claim 19, wherein, the
first convolutional network and the second convolutional network
include at least one convolution layer composed of a plurality of
filter units.
Description
TECHNICAL FIELD
[0001] The present disclosure relates to an image signal processing
technology, and more particularly, to an image upsampling system, a
training method thereof, and an image upsampling method.
BACKGROUND
[0002] At present, in an image signal processing procedure,
generally, the resolution of an image can be raised by means of the
standard methods such as a bicubic upsampling process and a linear
upsampling process (for enlarging image resolution). As illustrated
in FIG. 1, a 2.times. upsampling mode is shown, four different
filters: filter F1, filter F2, filter F3 and filter F4, are applied
to all the pixels (plus adjacent pixels) of an input image, each
filter generates a quarter of the pixels of the output image, and
the process may be understood as applying four filters
(convolution) to the input image, and then interleaving or
multiplexing to create a single output image that doubles the width
and the height of the image.
[0003] However, data computation amount of the current image
upsampling system is relatively large, and the upsampling multiple
(times) cannot be flexibly adjusted.
SUMMARY
[0004] In view of the above, embodiments of the present disclosure
provide an image upsampling system, a training method thereof and
an image upsampling method, for implementing high-quality
upsampling of image resolution based on a convolutional neural
network, to reduce a computation amount of upsampling, and to
improve flexibility of adjusting the upsampling multiple.
[0005] An aspect of the present disclosure provides an image
upsampling system, comprising: at least one first convolutional
network and at least one muxer layer that are cascaded; an signal
input end of the image upsampling system is connected with a signal
input end of a first convolutional network in the at least one
first convolutional network, and a signal output end of the image
upsampling system is connected with a signal output end of a last
muxer layer in the at least one muxer layer; a signal input end of
every muxer layer in the at least one muxer layer is connected with
a signal output end of a first convolutional network located in a
stage prior to the muxer layer in the at least one first
convolutional network, or connected with a signal output end of
another muxer layer located in a stage prior to the muxer layer in
the at least one muxer layer; the first convolutional network is
configured for converting an image input to its signal input end
into a plurality of feature images and outputting the feature
images to the signal input end of the muxer layer connected
therewith; the muxer layer is configured for synthesizing every
n.times.n feature images in the feature images input to its signal
input end into a feature image whose resolution is n.times.n times
that of the input feature image and outputting the same; and a
number of feature images input to the muxer layer is a multiple of
n.times.n, n being an integer greater than one.
[0006] According to an embodiment of the present disclosure, a
number of muxer layers is two or three.
[0007] According to an embodiment of the present disclosure, a
signal input end of each muxer layer is respectively connected with
a signal output end of one corresponding first convolutional
network in the at least one first convolutional network.
[0008] According to an embodiment of the present disclosure, in the
case where there are provided a plurality of muxer layers, the
muxer layers have a same upsampling multiple.
[0009] According to an embodiment of the present disclosure, the
muxer layer has an upsampling multiple which is a prime number.
[0010] According to an embodiment of the present disclosure, the
muxer layer has an upsampling multiple which is 2.
[0011] According to an embodiment of the present disclosure, the
muxer layer is a self-adaptive interpolation filter.
[0012] According to an embodiment of the present disclosure, the
image upsampling system further comprises: a second convolutional
network, whose signal input end is connected with a signal output
end of the last muxer layer in the at least one muxer layer, and
whose signal output end is connected with the signal output end of
the image upsampling system; the second convolutional network is
configured for optimizing picture quality of the feature images
output by the muxer layer.
[0013] According to an embodiment of the present disclosure, the
first convolutional network and the second convolutional network
include at least one convolution layer composed of a plurality of
filter units.
[0014] Another aspect of the present disclosure further provides a
display device, comprising any one of the above image upsampling
systems.
[0015] Still another aspect of the present disclosure further
provides a training method of any of one the image upsampling
systems, comprising: initializing respective parameters in the
image upsampling system; by using an original image signal as an
output signal of the image upsampling system and using an image
signal obtained by down-sampling the original image signal as an
input signal of the image upsampling system, adjusting the
respective parameters in the image upsampling system to allow the
down-sampled image signal subjected to upsampling processing with
the adjusted respective parameters to be the same as the original
image signal.
[0016] According to an embodiment of the present disclosure, in the
above training method, initializing of the respective parameters in
the image upsampling system includes: initializing weights Wij of
respective filter units of respective convolution layers of the
first convolutional network and the second convolutional network in
the image upsampling system according to a formula below:
W ij = { 1 / ( m ) ( i , j ) are preset anchor pixel 0 other pixel
} ##EQU00001##
where m represents the number of feature images input to the filter
unit; and initializing the biases of respective filter units to
0.
[0017] According to an embodiment of the present disclosure, in the
above training method, initializing of the respective parameters in
the image upsampling system includes: initializing the weights Wij
of respective filter units of respective convolution layers of the
first convolutional network and the second convolutional network in
the image upsampling system according to a formula below:
W ij = W ij ' + uniform ( - 1 , 1 ) m ##EQU00002## W ij ' = { 1 / (
m ) ( i , j ) are preset anchor pixel 0 other pixel }
##EQU00002.2##
where m represents the number of feature images input to the filter
unit, and uniform (-1,1) represents a random number selected
between (-1,1); and initializing the biases of respective filter
units to 0.
[0018] Further still another aspect of the present disclosure
further provides a, method for performing image upsampling with any
one of the above image upsampling systems, comprising: converting,
by a first convolutional network, an input image input to the first
convolutional network into a plurality of feature images and
outputting the feature images; synthesizing, by a muxer layer,
every n.times.n feature images in the feature images input to the
muxer layer into a feature image with a resolution amplified by
n.times.n times as larger as the input feature images, and
outputting the same; the number of feature images input to the
muxer layer being a multiple of n.times.n, and n being an integer
greater than 1.
[0019] According to the embodiments of the present disclosure, in
the image upsampling system, the training method thereof and the
image upsampling method, a feature images of an image are obtained
by a convolutional network, upsampling processing is performed on
the image with a muxer layer to synthesize every n.times.n feature
images in an input signal into a feature image with the resolution
amplified by n.times.n times; in the upsampling procedure with the
muxer layer, information of respective input feature images is
recorded in the generated feature image without loss; and thus,
every time when the image passes through a muxer layer which can
upsample by a multiple of n.times., the image resolution can be
increased by n.times.n times. In addition, in the image upsampling
system, more than one muxer layers can be provided for successive
processes of upsampling, each muxer layer can execute an upsampling
function of a single multiple, so that the system can flexibly
adjust the upsampling multiple(s) according to needs, implement a
universal upsampling system with respect to different upsampling
multiples. Further, because the number of feature images output by
each muxer layer is reduced while the muxer layer amplifies the
resolution of the feature image(s) by n.times.n times, an signal
amount input to a muxer layer or a first convolutional network of a
next cascaded stage can be reduced so as to simplify the
computation amount of upsampling.
BRIEF DESCRIPTION OF THE DRAWINGS
[0020] FIG. 1 is a schematic diagram of a 2.times. upsampling in
the state of art;
[0021] FIG. 2a to FIG. 2e are respectively structural schematic
diagrams of an image upsampling system provided by an embodiment of
the present disclosure;
[0022] FIG. 3 is a schematic diagram of upsampling by a muxer layer
in an image upsampling system provided by an embodiment of the
present disclosure;
[0023] FIG. 4 is a structural schematic diagram of a convolutional
network in an image upsampling system provided by an embodiment of
the present disclosure; and
[0024] FIG. 5 is a schematic flow chart of a training method of an
image upsampling system provided by an embodiment of the present
disclosure.
DETAILED DESCRIPTION
[0025] A convolutional neural network is a type of artificial
neural network, and has become a research hotspot in the fields
such as speech analysis and image recognition at present. The
weight sharing network structure of a convolutional neural network
renders the convolutional neural network more similar to a
biological neural network, and it can reduce the complexity of a
network model and reduces the number of weights. This advantage is
more obvious when the input of the network is a multi-dimensional
image, and the image can be directly used as the network input,
which avoids the complicated process such as feature extraction and
data reconstruction in a traditional identification algorithm. The
convolutional neural network is a multi-layer sensor specially
designed to identify a two-dimensional shape. This type of network
structure can be applied to translation, scaling, tilting, or other
forms of transformation.
[0026] Based on a convolutional neural network, an embodiment of
the present disclosure provides an image upsampling system, a
method for training the image upsampling system, and a method for
performing upsampling on an input image according to the trained
image upsampling system. The image upsampling system performs
upsampling by using the convolutional neural network, and, under
the premise of ensuring substantially no loss of image information,
can effectively converts a low-resolution image into a
high-resolution image.
[0027] Hereinafter, specific implementing modes of the image
upsampling system, the training method thereof and the image
upsampling method provided by the embodiments of the present
disclosure will be described in detail in connection with the
drawings.
[0028] An image upsampling system provided by an embodiment of the
present disclosure, as illustrated in FIG. 2a to FIG. 2d,
comprises: at least one first convolutional network (CN) and at
least one muxer layer (ML) cascaded; an signal input end of the
image upsampling system is connected with a signal input end of one
first convolutional network, and a signal output end of the image
upsampling system is connected with a signal output end of one
muxer layer.
[0029] A signal input end of a muxer layer is connected with a
signal output end of one first convolutional network or connected
with a signal output end of another muxer layer.
[0030] The first convolutional network is configured for converting
an image input to its signal input end into a plurality of feature
images and outputting to the signal input end of the muxer layer.
The muxer layer is configured for synthesizing every n.times.n
feature images in the feature images that are input to its signal
input end into a feature image whose resolution is n.times.n times
the resolution of the input feature images, and outputting the
synthesized feature image; and the number of feature images input
to the muxer layer is a multiple of n.times.n, where n is an
integer greater than one.
[0031] In the above-described image upsampling system provided by
the embodiment of the present disclosure, image upsampling
processing is performed with a muxer layer to synthesize every
n.times.n feature images input to the muxer layer into one feature
image with the resolution amplified by n.times.n times, and in the
upsampling procedure with the muxer layer, the information of each
input feature image is still recorded in the resultant feature
image without loss; and thus, every time when the image passes
through a muxer layer with an upsampling multiple (factor) of n,
the image resolution can be increased by n.times.n times. In
addition, in the image upsampling system, more than one muxer
layers may be provided for successively upsampling, the muxer
layers can respectively perform an upsampling function of an
individual multiple, so that the system can flexibly adjust the
upsampling multiple(s) according to needs and realize a universal
image upsampling system with respect to different upsampling
multiples. Further, because the number of feature images output by
each muxer layer is reduced while the muxer layer amplifies the
resolution of the feature image by n.times.n times, the signals
amount input to the muxer layer or first convolutional network in a
next cascaded stage can be reduced so as to alleviate the
computation amount of upsampling.
[0032] It should be noted that, if the system comprises a plurality
of muxer layers with an amplifying multiple of n times, then after
the image is upsampled by the system, the image resolution can be
improved by (n.times.n).times.(n.times.n) times.
[0033] For example, if the system comprises two muxer layers with
an upsampling multiple of 2.times. (two times), after the image
passes through the two muxer layers, the resolution is improved by
4.times.4 times; and if the system comprises three muxer layers
with an upsampling multiple of 2.times., after the image passes
through the three muxer layers, the resolution is improved by
4.times.4.times.4 times.
[0034] When specifically implemented, according to the required
upsampling multiple, there may be provided a variety of specific
implementing modes of the above-described image upsampling system
provided by the embodiment of the present disclosure. For example,
according to the required upsampling multiple, in the
above-described image upsampling system provided by the embodiment
of the present disclosure, one muxer layer may be provided as
illustrated in FIG. 2a, or two muxer layers may be provided as
illustrated in FIG. 2b and FIG. 2c, or three muxer layers may be
provided as illustrated in FIG. 2d.
[0035] Specifically, generally, when an upsampling process by an
upsampling multiple such as 2.times., 3.times. or 5.times. is
needed, the above-described image upsampling system provided by an
embodiment of the present disclosure, as illustrated in FIG. 2a,
comprises one first convolutional network and one muxer layer which
can upsample feature images by times of 2.times., 3.times. or
5.times.. When an upsampling multiple of 4.times. is needed, the
above-described image upsampling system provided by the embodiment
of the present disclosure, as illustrated in FIG. 2b and FIG. 2c,
may comprise two muxer layers which can upsample by 2.times. times.
When an upsampling multiple of 8.times. is needed, the
above-described image upsampling system provided by the embodiment
of the present disclosure, as illustrated in FIG. 2d, may comprise
three muxer layers which can upsample by 2.times. times. In this
way, the greater the required multiple, the larger the number of
muxer layers correspondingly, and the greater the data computation
amount performed by the corresponding system. Thus, preferably, in
the above-described image upsampling system provided by the
embodiment of the present disclosure, two or three muxer layers are
generally provided, to perform two or three processes of
upsampling.
[0036] Further, when a plurality of muxer layers are provided in
the above-described image upsampling system provided by the
embodiment of the present disclosure, in order that when each muxer
layer performs upsampling, high-quality feature images can be
synthesized into a high-quality feature image whose resolution is
n.times.n times the resolution of the input feature images, as
illustrated in FIG. 2c and FIG. 2d, generally, the signal input end
of the muxer layer is connected with the signal output end of the
first convolutional network, and thus firstly, the feature images
are obtained by the first convolutional network, then the obtained
feature images are input to the signal input end of the
corresponding muxer layer, that is, in the image upsampling system,
the first convolutional network and the muxer layer are arranged in
pair(s).
[0037] Further, in the case where a plurality of muxer layers are
provided in the above-described image upsampling system provided by
an embodiment of the present disclosure, the upsampling multiples
of the muxer layers may be the same or different from each other.
Usually, when there are provided a plurality of muxer layers, the
upsampling multiple n of the respective muxer layer is generally
elected to be the same. Moreover, the smaller the upsampling
multiple of each muxer layer, the smaller the computation amount,
and the better the upsampling effect. Thus, when the required
upsampling multiple is larger, generally upsampling is performed
many times, and the upsampling multiple n of each muxer layer is
generally elected to be a prime number such as 2, 3, 5 or 7.
Preferably, the upsampling multiple n of each muxer layer is set to
be 2.
[0038] Further, the above-described image upsampling system
provided by an embodiment of the present disclosure, as illustrated
in FIG. 2e, further comprises: a second convolutional network,
whose signal input end is connected with the signal output end of
an muxer layer, and whose signal output end is connected with the
signal output end of the image upsampling system; the second
convolutional network is configured for optimizing the picture
quality of the feature image output by a muxer layer. Before the
muxer layer of the last stage outputs a finally upsampled feature
image, the second convolutional network can be configured for
enhancing the picture quality of output picture according to actual
needs, to improve quality of the output image.
[0039] Specifically, in the above-described image upsampling system
provided by an embodiment of the present disclosure, both the first
convolutional network and the second convolutional network may
include at least one convolution layer composed of a plurality of
filter units. The number of convolution layers included in the
first convolutional network and the second convolutional network
may be set according to needs. And the number of filter units
included in each convolution layer may be the same or different.
Generally, in order to facilitate the system to optimize its
parameter(s), the number of convolution layers in each
convolutional network is generally set to be no more than 10.
[0040] Hereinafter, the above-described image upsampling system
provided by an embodiment of the present disclosure will be
described below with reference to the structure shown in FIG. 2e,
and with a case where two 2.times. muxer layers are configured for
performing 4.times. upsampling as an example.
[0041] Specifically, the first convolutional network at the first
stage is connected with the signal input end of the image
upsampling system and is composed of four convolution layers, each
convolution layer includes 128 filter units, and each filter unit
is composed of 3.times.3 filters; the filter at position [1,1] is
set as the center pixel. After the input image passes through the
first convolution layer, 128 feature images are generated and
output to a next convolution layer, until the last convolution
layer outputs 128 feature images to the muxer layer of a next
(second) stage.
[0042] After the muxer layer of the second stage receives the 128
feature images output by the first convolutional network of the
first stage, it synthesizes every four input feature images into
one feature image of the pixel resolution 4 times as larger as the
input feature images (2.times. upsampling), i.e., after the 128
input feature images pass through the muxer layer of the second
stage, 32 feature images are output to the first convolutional
network of the next (third) stage.
[0043] The first convolutional network of the third stage is
composed of four convolution layers, each convolution layer
includes 32 filter units, and each filter unit is composed of
3.times.3 filters; the filter at position [1,1] is set as the
center pixel. After the 32 feature images input from the muxer
layer of the second stage pass through the first convolution layer,
32 feature images are generated and output to a next convolution
layer, until the last convolution layer outputs 32 feature images
to the muxer layer of the next (fourth) stage.
[0044] After the muxer layer of the fourth stage receives the 32
feature images output by the first convolutional network of the
third stage, it synthesizes every 4 input feature images into one
feature image of pixel resolution of 4 times as larger as the input
feature images (2.times. upsampling), i.e., after the 32 input
feature images pass through the fourth stage of muxer layer, 8
feature images are output to the second convolutional network of
the next (fifth) stage.
[0045] The second convolutional network of the fifth stage is
composed of four convolution layers, the first two convolution
layers include 8 filter units, the third convolution layer includes
4 filter units, the fourth convolution layer includes 1 filter
unit, and each filter unit is composed of 3.times.3 filters; the
filter at position [1,1] is set as the center pixel. After the
image of the input signal passes through the first convolution
layer, 8 feature images are generated and output to the second
convolution layer, after they pass through the second convolution
layer, 8 feature images are generated and input to the third
convolution layer, after they pass through the third convolution
layer, 4 feature images are input to the fourth convolution layer,
and finally, the fourth convolution layer outputs one feature image
to the output end of the image upsampling system.
[0046] In the above-described procedure, each muxer layer is
substantially equivalent to a self-adaptive interpolation filter,
and as illustrated in FIG. 3, pixel values of every four feature
images in the input feature image are combined in a staggered
manner, to generate a feature image of a quadrupling pixel. As
illustrated in FIG. 3, a working principle of the muxer layer is
arranging pixel values of respective identical pixel-point
positions in 4 input feature image in a matrix in an output feature
image, so that in the upsampling procedure, no pixel information in
the feature image will be modified (lost or added).
[0047] In the above-described procedure, either the first
convolutional network or the second convolutional network can be a
neural network structure which uses images as input and output,
each neural network structure includes a plurality of convolution
layers, and each convolution layer is composed of a plurality of
filters. Hereinafter, the working principle will be briefly
introduced by means of a neural network structure of two
convolution layers in FIG. 4 as an example.
[0048] On the left side of FIG. 4, there are four input images,
which generate three feature images after passing through
respective filters of the first convolution layer, generate two
feature images after passing through respective filters of the
second convolution layer and then are output. Therein, each box
marked with a scalar weight W.sub.ij.sup.k is equivalent to a
filter (for example, a filter having 3.times.3 or 5.times.5 core),
and a bias b.sub.j.sup.k represents picture increment added to
convolution output; k represents the serial number of the
convolution layer, and i and j represent an input serial image
number and an output image serial number, respectively.
[0049] In the process of a system operation, the numerical values
of the scalar weight W.sub.ij.sup.k and the bias b.sub.j.sup.k are
relatively fixed, and before the system operation, it is necessary
to train the system by using a series of standard input and output
images, and depend on an application to adjust to adapt to some
optimization criteria. Therefore, before operation of the
above-described image upsampling system provided by an embodiment
of the present disclosure, a series of training needs to be
performed; and based on a same inventive concept, an embodiment of
the present disclosure further provides a training method of the
above-described image upsampling system, as illustrated in FIG. 5,
comprising the following operations:
[0050] S501: initializing respective parameters in the image
upsampling system; because a muxer layer does not introduce any
parameter, the respective parameters in the image upsampling system
are actually the parameters of all convolutional networks.
[0051] S502: using an original image signal as an output signal of
the image upsampling system, using an image signal obtained by
down-sampling the original image signal as an input signal of the
image upsampling system, and adjusting respective parameters in the
image upsampling system to allow the result from the down-sampled
image signal subjected to upsampling processing with the adjusted
respective parameters to be the same as the original image signal.
Thereafter, the adjusted respective parameters are used as the
upsampling parameter of the upsampling system, to up-sample the
low-resolution image.
[0052] In step S501, with respect to the initializing respective
parameters in the image upsampling system, a traditional
initializing mode may be used, and weights Wij of respective filter
units of the respective convolution layers of all the convolutional
networks are set to be a small random numbers, and all the biases
are initialized to be 0. The traditional initializing way will not
invoke any problem when applied to small-multiple (such as
2.times.) upsampling, but may invoke some problems when applied to
high-multiple (such as 4.times.) upsampling in connection with
several convolutional networks, and thus, the above-described
training method provided by an embodiment of the present disclosure
further provides two new ways with respect to the initializing
respective parameters in the image upsampling system, specifically
as follows:
[0053] First way: initializing biases of respective filter units to
be zero (0); and initializing weights Wij of respective filter
units of respective convolution layers of the first convolutional
network and the second convolutional network in the image
upsampling system according to a formula below:
W ij = { 1 / ( m ) ( i , j ) are preset anchor pixel 0 other pixel
} ##EQU00003##
[0054] where m represents the number of feature images input to the
filter unit.
[0055] Second way: initializing the biases of respective filter
units to be 0; and initializing the weights Wij of respective
filter units of respective convolution layers of the first
convolutional network and the second convolutional network in the
image upsampling system according to a formula below:
W ij = W ij ' + uniform ( - 1 , 1 ) m ; ##EQU00004## W ij ' = { 1 /
( m ) ( i , j ) are preset anchor pixel 0 other pixel }
##EQU00004.2##
[0056] where m represents the number of feature images input to the
filter unit; and uniform (-1,1) represents a random number selected
between (-1,1).
[0057] In the second initializing way, as compared with the first
initializing way, a small uniformly distributed noise value is
added to the weights W.sub.ij of respective filter units, which
facilitates the image upsampling system to have an ability of
identifying noise after training.
[0058] Based on the same inventive concept, an embodiment of the
present disclosure further provides a method for performing image
upsampling with the above-described image upsampling system;
because the principle for the method to solve the problem is
similar to that of an image upsampling system as described above,
for implementation of the method, the implementation of the system
may be referred to, and will not be repeated here.
[0059] An embodiment of the present disclosure provides a method
for performing image upsampling with an image upsampling system,
comprising: converting, by a first convolutional network, an input
image input to the first convolutional network into a plurality of
feature images having a specific feature and outputting the same;
and synthesizing, by a muxer layer, every n.times.n feature images
in the feature images input to the muxer layer into a feature image
with the resolution amplified by n.times.n times as larger as the
resolution of the input feature images, and outputting the same,
wherein the number of feature images input to the muxer layer is a
multiple of n.times.n, and n is an integer greater than 1.
[0060] Specifically, in the case where there are provided a
plurality of muxer layers in the system, each muxer layer, after
receiving the feature images, will perform upsampling processing on
the feature images, and output the result to a next muxer layer,
the next muxer layer performs upsampling processing on the received
feature image(s), until the last muxer layer outputs a final
up-sampled image.
[0061] The image upsampling system as described in the embodiment
of the present disclosure may be implemented by a set of central
processing units (CPU), and may also be implemented by a set of
graphics processing unit (GPU), or may also be implemented by a
field programmable gate array (FPGA).
[0062] Based on a same inventive concept, an embodiment of the
present disclosure further provides a display device, comprising
the above-described image upsampling system provided by the
embodiment of the present disclosure; the display device can be: a
mobile phone, a tablet computer, a television, a monitor, a laptop
computer, a digital photo frame, a wearable device, a navigator and
any products or parts with a display function. For implementation
of the display device, the embodiment of the above-described image
upsampling system may be referred to, and will not be repeated
here.
[0063] In the image upsampling system, the training method thereof
and the image upsampling method provided by some embodiments of the
present disclosure, the feature images of an image are obtained by
using the convolutional network, upsampling processing is performed
on the images with the muxer layer to synthesize every n.times.n
feature images in the input signal into a feature image with the
resolution amplified by n.times.n times, in the upsampling
procedure with the muxer layer, information of respective feature
images in the input signal is recorded in the generated feature
image(s) without loss; and thus, every time when the image passes
through a muxer layer with an upsampling multiple of n, the image
resolution can be increased by n.times.n times. In addition, in the
image upsampling system, more than one muxer layer may be provided
for successive processes of upsampling, each muxer layer can
execute an upsampling function of a single multiple so that the
system may flexibly adjust the upsampling multiple according to
needs, and implement a universal upsampling system for different
upsampling multiples. Further, because the number of feature images
output by each muxer layer is reduced while the muxer layer
amplifies the resolution of the feature images by n.times.n times,
the signal amount input to the muxer layer or first convolutional
network of a next cascaded stage can be reduced so as to simplify
the computation amount of upsampling.
[0064] It is evident that one person skilled in the art can make
various changes or modifications to the present invention without
departure from the spirit and scope of the invention. Thus, if such
changes and modifications to the present disclosure are within the
scope of the claims of the present disclosure and equivalent
thereof, the present disclosure also intends to include all such
changes and modifications within its scope.
[0065] This application claims priority of Chinese Patent
Application No. 201510595656.7 filed on Sep. 17, 2015, entitled
"IMAGE UPSAMPLING SYSTEM, TRAINING METHOD THEREOF AND IMAGE
UPSAMPLING METHOD", the disclosure of which is incorporated herein
by reference in its entirety as part of the present
application.
* * * * *