U.S. patent application number 11/972794 was filed with the patent office on 2008-07-17 for method for image prediction of multi-view video codec and computer readable recording medium therefor.
This patent application is currently assigned to KOREA ELECTRONICS TECHNOLOGY INSTITUTE. Invention is credited to Byeong-Ho Choi, Yo-Sung Ho, Je-Woo Kim, Yong-Hwan Kim, Ji-Ho Park, Hwa-Seon Shin.
Application Number | 20080170753 11/972794 |
Document ID | / |
Family ID | 39617819 |
Filed Date | 2008-07-17 |
United States Patent
Application |
20080170753 |
Kind Code |
A1 |
Park; Ji-Ho ; et
al. |
July 17, 2008 |
Method for Image Prediction of Multi-View Video Codec and Computer
Readable Recording Medium Therefor
Abstract
Provided are a method for image prediction of a multi-view video
codec capable of improving coding efficiency, and a computer
readable recording medium therefor. The method for image prediction
of a multi-view video codec includes partitioning an image to a
plurality of base blocks, acquiring information of reference images
which are temporally different, acquiring information of reference
images which have different views, and predicting a target block
based on the acquired information. Accordingly, an image that is
most similar to an image of a view to be currently compressed is
generated using multiple images of different views, so that coding
efficiency can be improved.
Inventors: |
Park; Ji-Ho; (Seoul, KR)
; Choi; Byeong-Ho; (Gyeonggi-do, KR) ; Kim;
Yong-Hwan; (Gyeonggi-do, KR) ; Kim; Je-Woo;
(Gyeonggi-do, KR) ; Shin; Hwa-Seon; (Gyeonggi-do,
KR) ; Ho; Yo-Sung; (Gwangju, KR) |
Correspondence
Address: |
EDELL, SHAPIRO & FINNAN, LLC
1901 RESEARCH BOULEVARD, SUITE 400
ROCKVILLE
MD
20850
US
|
Assignee: |
KOREA ELECTRONICS TECHNOLOGY
INSTITUTE
Gyeonggi-do
KR
|
Family ID: |
39617819 |
Appl. No.: |
11/972794 |
Filed: |
January 11, 2008 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
60884495 |
Jan 11, 2007 |
|
|
|
60886759 |
Jan 26, 2007 |
|
|
|
Current U.S.
Class: |
382/103 ;
382/238 |
Current CPC
Class: |
H04N 19/50 20141101;
H04N 19/105 20141101; H04N 19/61 20141101; H04N 19/176 20141101;
H04N 13/161 20180501; H04N 19/597 20141101 |
Class at
Publication: |
382/103 ;
382/238 |
International
Class: |
G06K 9/00 20060101
G06K009/00 |
Claims
1. A method for image prediction of a multi-view video codec, the
method comprising: partitioning an image to a plurality of base
blocks; acquiring information of reference images which are
temporally different; acquiring information of reference images
which have different views; and predicting a target block based on
the acquired information.
2. The method of claim 1, wherein the information of the reference
images which are temporally different includes motion information
and residual information associated with the referenced image.
3. The method of claim 2, wherein the predicting of the target
block comprises minimizing the residual information to minimize
residual information of the target block, using residual
information of an image having a different view.
4. The method of claim 3, further comprising adding report
information for decoding at the time of encoding, the reporting
information reporting the use of the residual information of the
image having a different view.
5. The method of claim 4, wherein the report information is
contained in a macroblock or higher layer.
6. The method of claim 4, wherein the residual information of the
image having a different view is residual information of a view
that is most similar to that of the target block among residual
information pieces present in different views.
7. The method of claim 4, wherein the residual information of the
image having the different value is optimum residual information
selected among residual information pieces present in different
views.
8. The method of claim 6, further comprising adding detailed
information for decoding at the time of encoding, the detailed
information reporting the view of the residual information being
used.
9. The method of claim 8, wherein the detailed information is
contained in a macroblock or higher layer.
10. The method of claim 4, wherein the residual information of the
image having a different view is combined residual information
obtained by combining at least two optimum residual information
pieces selected among residual information pieces present in
multiple images having different views.
11. The method of claim 10, wherein the combined residual
information is information generated as a mean vale of the at least
two selected residual information pieces.
12. The method of claim 10, wherein the combined residual
information is information generated by applying a filter to a mean
value of the at least two selected residual information pieces.
13. The method of claim 1, further comprising, before the
predicting of the target block: selecting a location obtained by
computation using a global disparity as a reference residual block
of the target block.
14. The method of claim 1, wherein the information of the reference
images which have different views includes motion information and
residual information associated with the referenced image.
15. The method of claim 14, wherein the predicting of the target
block comprises minimizing the residual information to minimize
residual information of the target block, using residual
information of a temporally different image.
16. The method of claim 15, further comprising adding report
information for decoding at the time of encoding, the report
information reporting the use of the residual information of the
temporally different image.
17. The method of claim 16, wherein the report information is
contained in a macroblock or higher layer.
18. The method of claim 16, wherein the residual information of the
temporally different image is residual information that is most
similar to the target block in terms of time among residual
information pieces present on multiple different times.
19. The method of claim 16, wherein the residual information of the
temporally different image is optimum residual information selected
among residual information pieces present on multiple different
times.
20. The method of claim 18, further comprising adding detailed
information for decoding at the time of encoding, the detailed
information reporting the time of the residual information being
used.
21. The method of claim 20, wherein the detailed information is
contained in a macroblock or higher layer.
22. The method of claim 16, wherein the residual information of the
temporally different image is combined residual information
obtained by combining at least two optimum residual information
pieces selected among residual information pieces present on
multiple different times.
23. The method of claim 22, wherein the combined residual
information is information generated as a mean value of the at
least two selected residual information pieces.
24. The method of claim 22, wherein the combined residual
information is information generated by applying a filter to a mean
value of the at least two selected residual information pieces.
25. A computer-readable recording medium storing a program for
executing image prediction of a multi-view video codec, the image
prediction comprising: partitioning a target image to a plurality
of base blocks; acquiring information of reference images which are
temporally different; acquiring information of reference images
which have different views; and predicting a target block based on
the acquired information.
26. The method of claim 7, further comprising adding detailed
information for decoding at the time of encoding, the detailed
information reporting the view of the residual information being
used.
27. The method of claim 19, further comprising adding detailed
information for decoding at the time of encoding, the detailed
information reporting the time of the residual information being
used.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application claims priority under 35 U.S.C. .sctn.119
to U.S. Provisional Application Nos. 60/884,495 and 60/886,759,
filed on Jan. 11, 2007 and Jan. 26, 2007, the disclosure of which
is incorporated herein by reference in its entirety.
BACKGROUND OF THE INVENTION
[0002] 1. Field of the Invention
[0003] The present disclosure relates to multi-view video coding,
and more particularly, to a method for image prediction of a
multi-view video codec and a computer readable recording medium
therefore.
[0004] 2. Description of the Related Art
[0005] A multi-view video codec is a video codec that outputs a
plurality of images captured from different views as one or
multiple compressed bitstreams. The multi-view video codec performs
compression using a correlation between different views, i.e.,
inter-view correlation, as well as compression using
spatial/temporal correlations, which is one of characteristics of a
related-art video codec. In the multi-view video codec, a technique
for effectively compressing images of different views must be
added. However, in most cases, the related-art technique that uses
the spatial/temporal correlation is still being used. In actuality,
there is almost no technique that uses the inter-view
correlation.
[0006] In the related art multi-view video codec employing the
existing H.264 standard, temporal prediction is conducted to remove
temporal redundancy between images, thereby improving coding
efficiency. However, the related art multi-view video codec
performs coding by using an image of a different view simply as a
reference image, failing to perform inter-view motion compensation
in due consideration of a disparity, which is a displacement value
between views. Thus, improvement of coding efficiency is limited
because no technique is used for removing inter-view redundancy
between images.
SUMMARY
[0007] Therefore, an object of the present invention is to provide
a method for image prediction of a multi-view video codec capable
of improving coding efficiency, and a computer readable recording
medium therefor.
[0008] Another object of the present invention is to provide a
method for image prediction of a multi-view video codec capable of
effectively using images of multiple different views, and a
computer readable recording medium therefor.
[0009] Another object of the present invention is to provide a
method for image prediction of a multi-view video codec capable of
proposing a new prediction scheme suitable for multi-view video
coding, and a computer readable recording medium therefor.
[0010] To achieve these and other advantages and in accordance with
the purpose(s) of the present invention as embodied and broadly
described herein, a method for image prediction of a multi-view
video codec in accordance with an aspect of the present invention
includes: partitioning an image to a plurality of base blocks;
acquiring information of reference images which are temporally
different; acquiring information of reference images which have
different views; and predicting a target block based on the
acquired information.
[0011] To achieve these and other advantages and in accordance with
the purpose(s) of the present invention, there is provided a
computer-readable recording medium storing a program for executing
image prediction of a multi-view video codec in accordance with
another aspect of the present invention, wherein the image
prediction includes partitioning a target image to a plurality of
base blocks; acquiring information of reference images which are
temporally different; acquiring information of reference images
which have different views; and predicting a target block based on
the acquired information.
[0012] The foregoing and other objects, features, aspects and
advantages of the present invention will become more apparent from
the following detailed description of the present invention when
taken in conjunction with the accompanying drawings.
BRIEF DESCRIPTION OF THE DRAWINGS
[0013] The accompanying drawings, which are included to provide a
further understanding of the invention and are incorporated in and
constitute a part of this specification, illustrate embodiments of
the invention and together with the description serve to explain
the principles of the invention.
[0014] FIG. 1 is a flowchart illustrating a method for image
prediction of a multi-view video codec according to an embodiment
of the present invention;
[0015] FIG. 2 illustrates an image reference structure in the
method for image prediction of a multi-view video codec according
to the embodiment of the present invention;
[0016] FIG. 3 illustrates encoding/decoding using residual
information of a macroblock according to an embodiment of the
present invention; and
[0017] FIG. 4 illustrates a weighting value reference model
according to an embodiment of the present invention.
DETAILED DESCRIPTION OF EMBODIMENTS
[0018] In a codec that supports multi-view, prediction can be
performed based on images of multiple different views, besides
using a technique employed in the related-art two-dimensional
codec. The prediction based on images of multiple different views
must be concerned with how to obtain the most similar image to an
image of a view to be currently compressed, using multiple images
of different views. Even if a multi-view codec (MVC) is also a
codec that supports multi-view, a technique using images having
multiple different views has not yet been employed therein. For
this reason, a technique for effectively compressing an image of a
view to be currently compressed, based on images of different views
will be described according to embodiments of the present
invention.
[0019] A method for image prediction of a multi-view video codec
according to exemplary embodiments of the present invention can be
realized as programs and stored in a computer-readable recording
medium that can execute the programs. A computer or an equivalent
digital device operates the programs for execution of the method
for image prediction of a multi-view video codec according to the
exemplary embodiments of the present invention. Thus, description
will be made on the assumption that the operations are performed by
a control means (hereinafter, referred to as a `control unit`) of
an arithmetic device.
[0020] Hereinafter, specific embodiments will be described in
detail with reference to the accompanying drawings. Like reference
numerals refer to like elements throughout. In some embodiments,
well-known processes, well-known device structures, and well-known
techniques will not be described in detail to avoid ambiguous
interpretation of the present invention.
[0021] FIG. 1 is a flowchart of a method for image prediction of a
multi-view video codec according to an embodiment of the present
invention, and FIG. 2 illustrates an image reference structure
according to the embodiment of the present invention. In operation
S110, a control unit partitions a target image to a number of base
blocks suitably for block-unit compression.
[0022] As illustrated in FIG. 2, two prediction methods may be used
for a block 210 to be currently encoded/decoded (hereinafter,
referred to as a target block). One of the prediction methods
computes a temporal difference value with a reference image to
obtain motion information, reference-image information and a
difference value between images. The other method uses an image of
a different view as a reference image to obtain motion information,
and reference image information. In the drawing, reference blocks
220 and 230 are images that the target block 210 references, and
residual blocks 240 and 250 are residual images that the target
block references 210. The residual means a difference between an
original image and an encoded image, which is resulted from
quantization of an encoding process.
[0023] In operation S120, the control unit acquires information of
reference images that are temporally different (hereinafter,
referred to as temporally different reference images). That is, a
temporal difference value with a reference image is computed to
acquire motion information and reference-image information. When
the target block 210 uses a temporally different image as a
reference image, the target image can acquire motion information
and a residual associated with the referenced image.
[0024] The residual information does not include a temporal
correlation but include an inter-view correlation. Thus,
information quantity can be reduced by just addition or subtraction
with residual information of an image having a different view. In
this case, for a decoding operation, information reporting the use
of the residual information of the different-view image may be
added at the time of encoding. The information reporting the use of
the residual information may be contained in macroblock layer or a
higher layer.
[0025] Even if residual information pieces of multiple images
having different views are present, residual information of an
image that is most similar to the target block 210 in terms of view
may be used. Also, optimum residual information may be selected
among residual information pieces of multiple images having
different views. In this case, information that reports view
information of the image of the residual information being used may
be added. For example, information reporting the use of residual
information of VIEW3 may be added. The information reporting the
use of the residual information of the image having a specific view
may be contained in a macroblock or a higher layer.
[0026] Also, new residual information may be used, which is
obtained by combining optimum residual information pieces among the
residual information pieces of images having different views. For
example, a mean value of the residual information pieces between at
least two selected images having two different views may be used as
the residual information, or new residual information may be
generated by applying a low-pass filter (LPF) such as a deblocking
filter to the mean value obtained in the aforementioned manner.
[0027] In operation S130, the control unit acquires information of
reference images having different views. That is, a difference
value with a reference image having a different view is computed to
acquire motion information and reference-image information. When
the target block 210 uses an image having a different view as a
reference image, the target image can obtain motion information and
residual information with respect to the referenced image.
[0028] According to the embodiment of the present invention, to
minimize this residual information, a method of using residual
information present in a temporally different image is provided so
as to minimize residual information to be currently
encoded/decoded. Here, the residual information does not include an
inter-view (spatial) correlation but include a temporal
correlation. The image quantity can be reduced by just addition or
subtraction with the residual of the temporally different image. In
this case, for decoding, information reporting the use of the
residual information of the temporally different image can be added
at the time of encoding. The information reporting the use of the
residual may be contained in a macroblock layer or a higher
layer.
[0029] Even if residual information pieces of multiple temporally
different images are present, residual information of an image that
is most similar to the target block 210 in terms of time may be
used. Also, optimum residual information may be selected among the
residual information pieces of the temporally different images. In
this case, information reporting the time of the image of the
residual information may be added. The information reporting the
use of the residual information of the image of the specific time
may be contained in a macroblock or a higher layer.
[0030] Also, new residual information may be used, which is
obtained by combining optimum residual information pieces among the
residual information pieces of the multiple temporally different
images. For example, the mean value of the residual information
pieces between two temporally different images may be used as the
residual information, or new residual information may be generated
by applying a filter to the mean value obtained in the
aforementioned manner.
[0031] A location obtained by computation using a global disparity
may be used as a reference residual block of the target block
210.
[0032] To report the application of the above techniques,
information of the following exemplary format may be used:
[0033] {"ResPred is equal to 1" specifies that the current slice
uses view residual prediction method. "ResPred is equal to 0"
specifies that the current slice does not use view residual
prediction method. When ResPred is not present, ResPred is inferred
as 0. It can be located in slice layer or higher layer (SPS
(Sequence Parameter Set), SPSE (SPS Extension), PPS (Picture
Parameter Set), PPSE (PPS Extension), slice header, slice header
extension, SEI (Supplementary Enhancement Information)).
[0034] "ResPredFlag is equal to 1" specifies that the residual
signal is predicted by the reconstructed residual signal of
neighbor view or temporally collocated macroblock, if possible,
with the considering of the global disparity. "ResPredFlag is equal
to 0" specifies any residual signal is not predicted. When
ViewPredFlag is not present, ViewPredFlag is inferred as 0.}
[0035] In operation S140, the control unit predicts the target
block based on the acquired information pieces. That is, those
information pieces are contained as prediction information of a
corresponding image at the time of encoding, and the prediction is
performed based on the prediction information at the time of
decoding. This will now be described with reference to FIG. 3 in
more detail.
[0036] FIG. 3 illustrates encoding/decoding using residual
information of a macroblock according to the embodiment of the
present invention. Referring to FIG. 3, if a disparity
corresponding to a target macroblock 310 indicates a reference
macroblock 320, residual information at time (t) is generated using
motion information of the reference macroblock, and a difference
value with residual information generated by using motion
information of a target macroblock is encoded/decoded. In this
case, information reporting the use of the residual information may
be contained in a residual data layer. Also, a value transmitted to
a slice header or a value computed by a time interval with a
transmitted slice may be used as a disparity.
[0037] According to another embodiment, a target block 210 may be
expressed as the following Equation (1):
Recon=Pred+Res (1)
where Pred denotes a reference image of a specific size, which is
most similar to a target block 210 of FIG. 2 in a temporal/spatial
domain and can be represented by motion information, and Res
denotes residual information indicating a difference between a
reference image and a target block 210.
[0038] According to the embodiment of the present invention, to
minimize this residual information, a method of using residual
information present in an image having a different view is
proposed, thereby reducing the residual information being currently
encoded/decoded. A video codec can be implemented such that Pred is
properly selected to minimize Res. As Pred in the multi-view codec,
an image that is proper in terms of view or time may be used. Pred
may be defined by the following Equation (2):
Pred=F(Pred'+Res') (2)
[0039] That is, Pred is obtained by applying a proper filter, e.g.,
an LPF such as a deblocking filter in H.264, to a value obtained by
adding a residual to a certain reference image.
[0040] When Equation (2) is applied to Equation (1), the following
Equation (3) can be obtained:
Recon=F(Pred'+Res')+Res'' (3)
where Pred' and Res' are a reference image and a residual of an
image that the target block 210 references, respectively. A
combination of Pred' and Res' that are properly induced is used as
a reference image of a current image, i.e., a target image, and
residual information therebetween is minimized.
[0041] If Equation (3) is rearranged with respect to the terms
Pred' and Res' by distributing F, F(Pred') is represented by Pred,
and Res is represented by F(Res')+Res''. Thus, a gain is obtained
by transmitting Res'' instead of Res as in the related art.
[0042] If Pred acquires a reference image in a temporal domain, the
term Res' is obtained from a view domain, whereas if Pred acquires
a reference image in a view domain, the term Res' is obtained from
a temporal domain. F( ), which is a filter suitable for the
obtained term Res' may be additionally used. For example, the
simplest filter having a filter coefficient {1/2, 1/2} may be used,
or a filter such as 1/20{1, -4. 20, 20, -4, 1} may be used.
[0043] To report the application of the above techniques,
information of the following exemplary format may be used:
[0044] {"NewPred is equal to 1" specifies that the current slice
uses new view prediction method according to the present invention.
"NewPred is equal to 0" specifies that the current slice does not
use new view prediction method according to the present invention.
When NewPred is not present, NewPred is inferred as 0. It can be
located in slice layer or higher layer (SPS, PPS, Slice header,
slice header extension, SEI).
[0045] "ResPredFlag is equal to 1" specifies that the predictor of
the current macroblock is derivate as following,
[0046] If the current macroblock is coded by inter mode (temporal
direction), residual signal of neighbor view(s) and reference block
of the current macroblock(s) are used as the predictor (Deblocking
filter can be applied on the predictor).
[0047] Otherwise (the current macroblock is coded by inter-view
mode (view direction)), residual signal of collocated block(s) and
reference block of the current macroblock(s) is used as the
predictor (Deblocking filter can be applied on the predictor.)
[0048] if possible, with the considering of the global disparity.
"ResPredFlag is equal to 0" specifies any residual signal is not
predicted. When ViewPredFlag is not present, ViewPredFlag is
inferred as 0.}
[0049] According to another embodiment of the present invention, a
weighting value or a weighted prediction value may be established.
For example, if an image has a certain temporal characteristic such
as image fading in/out, a weighting value of an image of a
different view may be used. The weighting value means information
indicating an extent to which image brightness or chrominance
signals change over time as illustrated in FIG. 4. FIG. 4
illustrates a weighting-value reference model according to the
embodiment of the present invention.
[0050] In general, even if images have different views, they may
have similar temporal characteristics. That is, in the case where
images are gradually brightened, an encoder may send a proper
weighting value to a decoder so that the decoder can collectively
apply the weighting value to the images of the different views.
[0051] If a different light source is used for each view, it may be
difficult to collectively apply the weighting value. In this case,
a new weighting value must be used.
[0052] As the simplest implementation method, weighting information
may be defined for each view. However, this method may be
inefficient because multiple redundant information pieces may be
transmitted.
[0053] According to the embodiment of the present invention, in
order to reduce redundant information and overcome a limitation
caused by using different light sources, a weighting value of a
specific view such as a BaseView or VIEW0 of FIG. 4 is shared, and
information reporting whether weighting values of different views
are used as they are (hereinafter, referred to as weighting
information) is used.
[0054] For example, as illustrated in FIG. 4, images of VIEW1 may
contain weighting information reporting the use of a weighting
value of BaseView (View0), and images of View2 may contain
weighting information reporting the use of their own weighting
values without using the weighting value of BaseView.
[0055] The weighting information is inserted in a bit stream to
prevent mutual mis-operation between an encoder and a decoder. The
weighting information may be contained in a slice header, a slice
header extension or a higher layer such as PPS, PPSE, SPS, SPSE or
SEI.
[0056] To report the application of the above techniques,
information of the following exemplary format may be used:
[0057] {"baseview_pred_weight_table_flag is equal to 1" specifies
that the variables for weighted prediction are inferred. When
baseview_pred_weight_table_flag is not present, it shall be
inferred as follows:
[0058] If baseViewFlag (which indicates whether baseview or not) is
equal to 1, base_pred_weight_table_flag shall be inferred to be
equal to 0.
[0059] Otherwise, baseview_pred_weight_table_flag shall be inferred
to be equal to 1.}
[0060] The method for image prediction of a multi-view video codec
and the computer readable recording medium therefor according to
the embodiments of the present invention, an image that is most
similar to an image of a view to be currently compressed is
generated by using inter-view prediction, i.e., using images of
multiple different views, thereby improving coding efficiency.
[0061] The methods for image prediction of a multi-view video codec
according to the exemplary embodiments can be realized as programs
and stored in a computer-readable recording medium that can execute
the programs. Examples of the computer-readable recording medium
include CD-ROM, RAM, ROM, floppy disks, hard disks, magneto-optical
disks and the like.
[0062] As the present invention may be embodied in several forms
without departing from the spirit or essential characteristics
thereof, it should also be understood that the above-described
embodiments are not limited by any of the details of the foregoing
description, unless otherwise specified, but rather should be
construed broadly within its spirit and scope as defined in the
appended claims, and therefore all changes and modifications that
fall within the metes and bounds of the claims, or equivalents of
such metes and bounds are therefore intended to be embraced by the
appended claims.
* * * * *