U.S. patent application number 13/232613 was filed with the patent office on 2012-03-15 for method and apparatus for hierarchical picture encoding and decoding.
This patent application is currently assigned to SAMSUNG ELECTRONICS CO., LTD.. Invention is credited to Woong-Il CHOI, Dae-Hee KIM, Min-Woo PARK.
Application Number | 20120063517 13/232613 |
Document ID | / |
Family ID | 45806720 |
Filed Date | 2012-03-15 |
United States Patent
Application |
20120063517 |
Kind Code |
A1 |
CHOI; Woong-Il ; et
al. |
March 15, 2012 |
METHOD AND APPARATUS FOR HIERARCHICAL PICTURE ENCODING AND
DECODING
Abstract
A residual picture is decoded first by demultiplexing an input
enhancement layer bitstream into one or more bitstreams
corresponding to an order of quality layers. Next, a residual
picture is restored by independently decoding the bitstream for
each quality layer. The restored residual picture is stored in a
frame buffer. A finally restored residual picture is output by
adding, to the restored residual picture, a picture that has
undergone motion compensation in a reference frame within a same
enhancement layer.
Inventors: |
CHOI; Woong-Il;
(Hwaseong-si, KR) ; PARK; Min-Woo; (Yongin-si,
KR) ; KIM; Dae-Hee; (Suwon-si, KR) |
Assignee: |
SAMSUNG ELECTRONICS CO.,
LTD.
Suwon-si
KR
|
Family ID: |
45806720 |
Appl. No.: |
13/232613 |
Filed: |
September 14, 2011 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
61382594 |
Sep 14, 2010 |
|
|
|
Current U.S.
Class: |
375/240.16 ;
375/E7.125 |
Current CPC
Class: |
H04N 7/0117 20130101;
H04N 19/36 20141101; H04N 19/51 20141101; H04N 19/33 20141101 |
Class at
Publication: |
375/240.16 ;
375/E07.125 |
International
Class: |
H04N 7/26 20060101
H04N007/26 |
Claims
1. A method for encoding a residual picture of picture data in
hierarchical picture processing in which one picture includes a
basement layer and one or more enhancement layers, and in which a
picture of the enhancement layer is divided into one or more
quality layers, comprising: combining the residual picture with a
picture that has undergone motion compensation in a reference frame
within a same enhancement layer, and generating a bitstream for
each quality layer by independently encoding the combined picture
for each quality layer; storing, in associated frame buffers,
residual differential pictures generated in the process of
independently encoding the combined picture for each quality layer;
outputting a motion-compensated picture by performing motion
compensation on the pictures stored in the frame buffers; and
outputting one enhancement layer bitstream by multiplexing together
the bitstreams for the individual quality layers.
2. The method of claim 1, wherein: a picture corresponding to a
difference, between (1) a picture obtained by combining the
residual picture with the picture that has undergone motion
compensation in the reference frame, and (2) a residual
differential picture generated in a process of encoding a picture
for a low quality layer is used as an input signal in an encoding
process for each quality layer.
3. The method of claim 1, wherein the storing comprises storing, in
associated frame buffers, residual differential pictures generated
in an encoding process of a highest one of the one or more quality
layers and a lowest one of the one or more quality layers.
4. The method of claim 3, wherein the outputting of a
motion-compensated picture comprises outputting the
motion-compensated picture by performing motion compensation on all
of the residual differential pictures generated in the encoding
process of the highest one of the one or more quality layers and
the lowest one of the one or more quality layers.
5. An apparatus for encoding a residual picture of picture data in
hierarchical picture processing in which one picture includes a
basement layer and one or more enhancement layers, and in which a
picture of the enhancement layer is divided into one or more
quality layers, comprising: a quality layer encoder adapted to
combine the residual picture with a picture that has undergone
motion compensation in a reference frame within a same enhancement
layer, and adapted to generate a bitstream and a residual
differential picture for each quality layer by independently
encoding the combined picture for each quality layer; a frame
buffer adapted to store residual differential pictures generated in
the quality layer encoder; a motion compensator adapted to output a
motion-compensated picture by performing motion compensation on the
pictures stored in the frame buffer; and a multiplexer adapted to
output a single enhancement layer bitstream by multiplexing the
bitstreams for the individual quality layers.
6. The apparatus of claim 5, wherein the quality layer encoder
uses, as an input signal, a picture corresponding to a difference
between: (1) a picture obtained by combining the residual picture
with the picture that has undergone motion compensation in the
reference frame, and (2) a residual differential picture generated
in a process of encoding a picture for a lower quality layer.
7. The apparatus of claim 5, wherein the frame buffer stores
residual differential pictures generated in an encoding process of
a highest one of the one or more quality layers and a lowest one of
the one or more quality layers.
8. The apparatus of claim 7, wherein the motion compensator outputs
the motion-compensated picture by performing motion compensation on
all of residual differential pictures generated in the encoding
process of the highest one of the one or more quality layers and
the lowest one of the one or more quality layers.
9. A method for decoding a residual picture of picture data in
hierarchical picture processing in which one picture includes a
basement layer and one or more enhancement layers, and in which a
picture of the enhancement layer is divided into one or more
quality layers, comprising: demultiplexing an input enhancement
layer bitstream into one or more bitstreams corresponding to an
order of the one or more quality layers; restoring a residual
picture by decoding the bitstream for each quality layer
independently, and storing the restored residual picture in a frame
buffer; and outputting a finally restored residual picture by
adding a picture, that has undergone motion compensation in a
reference frame within a same enhancement layer, to the restored
residual picture.
10. The method of claim 9, wherein the outputting of a finally
restored residual picture comprises outputting a finally restored
residual picture by adding (1) a picture that has undergone motion
compensation in a reference frame within a same enhancement layer,
to (2) a picture obtained by adding up residual pictures restored
in each quality layer decoding process.
11. The method of claim 9, wherein the storing comprises: storing
in a first frame buffer a picture obtained by adding (1) the
picture that has undergone motion compensation in the reference
frame to (2) a residual picture restored in a lowest quality layer
decoding process, the first frame buffer corresponding to a lowest
one of the one or more quality layers; and storing the finally
restored residual picture in a second buffer corresponding to a
highest one of the one or more quality layer.
12. The method of claim 11, wherein the picture stored in at least
one of the first and second frame buffers is used to restore a
residual picture of a next frame after undergoing motion
compensation.
13. The method of claim 9, wherein the storing comprises: storing a
residual picture restored in a lowest quality layer decoding
process, in a first frame buffer corresponding to the lowest one of
the one or more quality layers; and outputting a picture obtained
by adding (1) the picture that has undergone motion compensation in
the reference frame to (2) the residual picture restored in a
highest quality layer decoding process, as the finally restored
residual picture, and storing the finally restored residual picture
in a second frame buffer corresponding to a highest one of the one
or more quality layers.
14. The method of claim 13, wherein the picture stored in at least
one of the first and second frame buffers is used to restore a
residual picture of a next frame after undergoing motion
compensation.
15. An apparatus for decoding a residual picture of picture data in
hierarchical picture processing in which one picture includes a
basement layer and one or more enhancement layers, and in which a
picture of the enhancement layer is divided into one or more
quality layers, comprising: a demultiplexer adapted to demultiplex
an input enhancement layer bitstream into one or more bitstreams
corresponding to an order of the one or more quality layers; a
quality layer decoder adapted to restore a residual picture by
independently decoding the bitstream for each quality layer; a
frame buffer storing the residual picture restored in the quality
layer decoder; and a motion compensator adapted to output a finally
restored residual picture by adding (1) a picture that has
undergone motion compensation in a reference frame within a same
enhancement layer, to (2) the restored residual picture.
16. The apparatus of claim 15, wherein the motion compensator
outputs a finally restored residual picture by adding (1) a picture
that has undergone motion compensation in a reference frame within
a same enhancement layer, to (2) a picture obtained by adding up
residual pictures restored in each decoding process of the one or
more quality layers.
17. The apparatus of claim 15, wherein the frame buffer comprises:
a first frame buffer storing a picture obtained by adding (1) the
picture that has undergone motion compensation in the reference
frame to (2) a residual picture restored in a lowest quality layer
decoder; and a second frame buffer storing the finally restored
residual picture.
18. The apparatus of claim 17, wherein the motion compensator
performs motion compensation on the picture stored in at least one
of the first and second frame buffers, and uses the
motion-compensated picture to restore a residual picture of a next
frame.
19. The apparatus of claim 15, wherein the frame buffer comprises:
a first frame buffer storing a residual picture restored in a
lowest quality layer decoder; and a second frame buffer storing, as
the finally restored residual picture, a picture obtained by adding
(1) the picture that has undergone motion compensation in the
reference frame to (2) the residual picture restored in a highest
quality layer decoder.
20. The apparatus of claim 19, wherein the motion compensator
performs motion compensation on the picture stored in at least one
of the first and second frame buffers, and uses the
motion-compensated picture to restore a residual picture of a next
frame.
21. A residual encoder, comprising: a plurality of sequenced
quality layer encoders, including at least a first quality layer
encoder and a last quality layer encoder; a selective motion
compensator that outputs a motion-compensated picture; wherein: the
first quality layer encoder generates a first residual differential
picture based on a difference between a residual picture and the
motion-compensated picture, the first residual differential picture
being output to the selective motion compensator and to a
subsequent one of the plurality of sequenced quality layer
encoders; each subsequent one of the plurality of sequenced quality
layer encoders generates a respective residual differential picture
based on a difference between the residual picture and the picture
output by an immediately preceding one of the plurality of
sequenced quality layer encoders, the respective residual
differential picture being output to the selective motion
compensator and to another subsequent one of the plurality of
sequenced quality layer encoders; and the last quality layer
encoder generates a last residual differential picture based on a
difference between a residual picture and the respective residual
differential picture received from the immediately preceding one of
the plurality of sequenced quality layer encoders, the last
residual differential picture being output to the selective motion
compensator.
22. The residual encoder as set forth in claim 21, further
comprising frame buffers storing only the first and the last
residual differential pictures.
23. The residual encoder as set forth in claim 21, further
comprising frame buffers storing a respective residual differential
picture corresponding to each of the plurality of sequenced quality
layer encoders.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application claims the benefit under 35 U.S.C.
.sctn.119(a) of U.S. Provisional Application No. 61/382,594, filed
in the United States Patent and Trademark Office on Sep. 14, 2010,
which is incorporated by reference, herein, in its entirety.
FIELD
[0002] The inventive concept relates generally to hierarchical (or
layered) picture encoding/decoding technology and, more
particularly, to a hierarchical picture encoding/decoding method
and apparatus for refining picture quality using residual pictures
in video compression codec processing videos.
BACKGROUND
[0003] Generally, in video compression technology, pictures (or
images) are processed in units of macro blocks having M.times.N
pixels per block. In video processing, macro blocks are encoded and
decoded in either an intra mode or an inter mode. The term `macro
block,` as used herein, may be understood to refer to a set of
pixel blocks that each have a specific size. A `frame` includes
multiple macro blocks. The typical video compression technology
using macro blocks may include the use of compression standards
such as MPEG and H.26x.
[0004] The basic concept of video compression is to remove
spatially and temporally redundant data from the original picture
data. The intra mode is a scheme for removing spatial redundancy.
In other words, this mode is a scheme for removing redundancy,
between pixels in macro blocks having a specific size, in the
current frame. The inter mode is a scheme for removing temporal
redundancy. In other words, this mode is a scheme for estimating a
difference between macro blocks of a current frame and
corresponding macro blocks of a previous or of a following frame
(referred to as a "reference" frame) by estimating motions between
corresponding macro blocks in the two adjacent frames. The motion
estimation is a process of searching for macro blocks in the
reference frame, which are similar to macro blocks to be encoded in
the current frame. In picture encoding, motion compensation is
performed using the macro blocks in the reference frame, which were
searched for by the motion estimation process and were found. A
picture encoder entropy-encodes a difference between the "found"
macro blocks in the reference frame and macro blocks in the current
frame, along with a motion vector representing the location of the
reference frame during its transmission.
[0005] Typically, in motion compensation, a residual
encoder/decoder calculates multiple residual differential pictures
for the multiple quality layers constituting its input residual
picture, and also performs motion compensation on the residual
differential pictures independently, causing an increase in its
complexity and encoding/decoding time.
SUMMARY
[0006] An aspect of the exemplary embodiments is to provide a
method and apparatus for reducing complexity and refining picture
quality during picture encoding, especially during residual
encoding in hierarchical picture processing technology.
[0007] Another aspect of the exemplary embodiments is to provide a
method and apparatus for reducing complexity and refining picture
quality during picture decoding, especially during residual
decoding in hierarchical picture processing technology.
[0008] In accordance with one exemplary embodiment, a residual
picture of picture data in hierarchical picture processing is
encoded, and this includes a basement layer and at least one
enhancement layer. Also, a picture of the enhancement layer is
divided into one or more quality layers. The method includes
combining the residual picture with a picture that has already
undergone motion compensation in a reference frame within a same
enhancement layer. A bitstream is generated for each quality layer
by encoding the combined picture independently for each quality
layer. Residual differential pictures generated in the process of
encoding the combined picture are stored, in associated frame
buffers, independently for each quality layer. A motion-compensated
picture is output by selectively performing motion compensation on
the pictures stored in the frame buffers; and outputting an
enhancement layer bitstream by multiplexing the bitstreams for the
individual quality layers.
[0009] In accordance with another exemplary embodiment, an
apparatus encodes a residual picture of picture data in
hierarchical picture processing. In the apparatus, one picture
includes a basement layer and at least one enhancement layer. A
picture of the enhancement layer is divided into one or more
quality layers. The apparatus includes a quality layer encoder for
combining the residual picture with a picture that has already
undergone motion compensation, in a reference frame within the same
enhancement layer. The encoder also generates a bitstream and a
residual differential picture for each quality layer. It does this
by encoding the combined picture independently for each quality
layer. The apparatus also has a frame buffer for storing the
residual differential pictures generated by the quality layer
encoder. The apparatus also has a motion compensator for outputting
a motion-compensated picture by selectively performing motion
compensation on the pictures stored in the frame buffer.
Additionally, there is a multiplexer for outputting an enhancement
layer bitstream by multiplexing the bitstreams for the individual
quality layers.
[0010] In accordance with further exemplary embodiment, there is
provided a method for decoding a residual picture of picture data,
in hierarchical picture processing, in which one picture includes a
basement layer and at least one enhancement layer. Also, a picture
of the enhancement layer is divided into one or more quality
layers. The method includes demultiplexing an input enhancement
layer bitstream into one or more bitstreams. The bitstreams
correspond to an order of the quality layers. In this method, a
residual picture is restored by decoding the bitstream
independently for each quality layer. The restored residual picture
is stored in a frame buffer. The finally restored residual picture
is output by adding to the restored residual picture a picture that
has undergone motion compensation in a reference frame within the
same enhancement layer.
[0011] In accordance with yet another exemplary embodiment, an
apparatus decodes a residual picture of picture data in
hierarchical picture processing in which one picture includes a
basement layer and at least one enhancement layer, and a picture of
the enhancement layer is divided into one or more quality layers.
The apparatus includes a demultiplexer for demultiplexing an input
enhancement layer bitstream into one or more bitstreams
corresponding to an order of the quality layers. The apparatus also
includes a quality layer decoder for restoring a residual picture
by decoding the bitstream for each the quality layer independently.
A frame buffer stores the residual picture that was restored in the
equality layer decoder. A motion compensator outputs the finally
restored residual picture by adding, to the restored residual
picture, a picture that has undergone motion compensation in a
reference frame within the same enhancement layer.
BRIEF DESCRIPTION OF THE DRAWINGS
[0012] The above and other aspects, features and advantages of
certain exemplary embodiments will be more apparent from the
following description taken in conjunction with the accompanying
drawings, in which:
[0013] FIG. 1 illustrates a structure of a hierarchical picture
encoder;
[0014] FIG. 2 illustrates a structure of a hierarchical picture
decoder;
[0015] FIG. 3 illustrates a structure of a residual encoder
according to an exemplary embodiment;
[0016] FIG. 4 illustrates a structure of a quality layer encoder
according to an exemplary embodiment;
[0017] FIG. 5 illustrates a structure of a selective motion
compensator according to an exemplary embodiment;
[0018] FIG. 6 illustrates a structure of a residual encoder
according to another exemplary embodiment;
[0019] FIG. 7 illustrates a structure of a residual decoder in a
hierarchical picture decoder according to an exemplary
embodiment;
[0020] FIG. 8 illustrates a structure of a residual decoder in a
hierarchical picture decoder according to another exemplary
embodiment;
[0021] FIG. 9 illustrates a residual encoding process according to
an exemplary embodiment; and
[0022] FIG. 10 illustrates a residual decoding process according to
an exemplary embodiment.
DETAILED DESCRIPTION
[0023] Exemplary embodiments will now be described in detail, with
reference to the accompanying drawings. In the following
description, specific details (such as detailed configuration and
particular components) are provided merely to assist the overall
understanding of the exemplary embodiments. Therefore, it should be
apparent to those skilled in the art that various changes and
modifications of the embodiments described herein can be made
without departing from the scope and spirit of the inventive
concept. In addition, descriptions of well-known functions and
constructions have been omitted for clarity and conciseness, and to
avoid obscuring the inventive concept.
[0024] The inventive concept includes, among others, a method and
apparatus for performing residual encoding and/or decoding on
multiple quality layers, constituting an enhancement layer
bitstream/picture in a hierarchical picture encoding/decoding
process.
[0025] First, the terms used herein will be defined, in brief, as
follows.
[0026] Quality refinement: a process for refining the quality of
residual samples reconstructed using refined data.
[0027] Quality layers: one or more layers used in the quality
refinement process.
[0028] Quality basement layer: a layer representing the
lowest-quality picture among reconstructed pictures, among multiple
quality layers in one picture.
[0029] Quality enhancement layer: a layer representing a
high-quality picture among reconstructed pictures, among multiple
quality layers in one picture.
[0030] A general hierarchical picture encoding/decoding apparatus
will now be described.
[0031] FIG. 1 illustrates the structure of a hierarchical picture
encoder.
[0032] Referring to FIG. 1, the picture encoder operates according
to a hierarchical structure including a basement layer and one or
more enhancement layers, accepts an input picture for encoding, and
outputs a basement layer bitstream and an enhancement layer
bitstream. A picture of the basement layer and a picture of the
enhancement layer may be pictures having different resolutions,
sizes and/or view points.
[0033] It is assumed, in the example of FIG. 1, that the input
picture and the picture processed in the enhancement layer are high
resolution pictures, are pictures having a large size, and/or are
pictures representing one view point. It is also assumed that the
picture processed in the basement layer is a low resolution
picture, a picture having a small size, and/or a picture
representing another view point. A format down-converter 101
down-converts the input picture into a picture having a format the
same as that of the basement layer. For example, the down-converter
101 performs a down-conversion in terms of the size (or frame rate)
of the input picture or a view point of a stereoscopic picture. A
basement layer encoder 103 outputs a basement layer bitstream
generated by encoding the input basement layer picture according to
an existing encoding scheme using, e.g., video codecs such as VC-1,
H.264, MPEG-4 Part 2 Visual, MPEG-2 Part 2 Video, AVS, and
JPEG2000. In the process of encoding the basement layer picture,
the basement layer encoder 103 outputs a basement layer picture
reconstructed to a format up-converter 105.
[0034] The format up-converter 105 is configured to perform an
up-conversion in terms of, for example, the size (or the frame
rate) or a view point of an input picture, and may be considered to
perform a process that is a reverse of the process of the format
down-converter 101. The format up-converter 105 up-converts the
reconstructed basement layer picture into a picture having the same
format as that of the enhancement layer. The input picture that is
input to the format down-converter 101 is also input to a
subtractor 107. The subtractor 107 outputs residual data obtained
by subtracting the up-converted picture, output by format
up-converter 105, from the input picture. A residual
mapping/scaling unit 109 converts the residual data into a residual
picture. The residual picture is input to residual encoder 111,
which outputs an enhancement layer bitstream by performing residual
encoding on the input residual picture.
[0035] A method for encoding a residual picture to gradually refine
the quality in one enhancement layer will now be described.
[0036] The residual encoder 111 includes a quality layer encoder
(not shown) and a motion compensator (not shown). The quality layer
encoder performs encoding on quality layers independently so as to
encode the residual picture in multiple quality layers.
[0037] FIG. 2 illustrates the structure of a hierarchical picture
decoder.
[0038] Referring to FIG. 2, the picture decoder operates according
to a hierarchical structure including a basement layer and one or
more enhancement layers, and outputs a reconstructed basement layer
picture and a reconstructed enhancement layer picture. It does this
by decoding the basement layer bitstream and the enhancement layer
bitstream which were respectively encoded by the picture encoder in
FIG. 1. The picture of the basement layer and the picture of the
enhancement layer may be pictures having different resolutions,
sizes and/or view points.
[0039] It is assumed, in the example of FIG. 2, that the input
picture and the picture processed in the enhancement layer are high
resolution pictures, are pictures having a large size, and/or are
pictures representing one view point. It is also assumed that the
picture processed in the basement layer is a low resolution
picture, a picture having a small size, and/or a picture
representing another view point. A basement layer decoder 201
outputs a reconstructed basement layer picture. It does this by
decoding the input basement layer bitstream according to a decoding
scheme that corresponds to the video codec that was used in the
basement layer encoder 103 in FIG. 1. The basement layer picture
that is reconstructed by the basement layer decoder 201 is itself
output, and is also outputted to a format up-converter 203. The
format up-converter 203 up-converts the reconstructed basement
layer picture into a picture that is the same format as the
enhancement layer. A residual decoder 205 inputs the enhancement
layer bitstream, and outputs a residual picture. This residual
picture is generated by carrying out residual decoding on the input
enhancement layer bitstream. The residual picture is next converted
into residual data by a residual mapping/scaling unit 207. The
residual data is next added to the format up-converted picture by
an adder 209, and the result is the generation of a reconstructed
enhancement layer picture.
[0040] The residual decoder 205 includes a quality layer decoder
(not shown) and a motion compensator. The quality layer decoder
performs decoding on the quality layers independently so as to
reconstruct the residual picture in multiple quality layers.
[0041] FIG. 3 illustrates the structure of a residual encoder
according to an exemplary embodiment. Although the residual encoder
in FIG. 3 has three quality layers, this is merely for the sake of
providing a teaching example; the number of quality layers is
subject to change.
[0042] Referring to FIG. 3, the residual encoder includes three
quality layer encoders 301, 303 and 305, the number of which
corresponds to the number of quality layers. The residual encoder
also includes a selective motion compensator 307.
[0043] A residual picture input to the residual encoder is encoded
into a first bitstream and a first residual differential picture by
the first quality layer encoder 301. The second quality layer
encoder 303 encodes a picture, corresponding to a difference
between the residual picture and the first residual differential
picture, into a second bitstream and a second residual differential
picture. The third quality layer encoder 305 encodes a picture,
corresponding to a difference between the residual picture and the
second residual differential picture, into a third bitstream and a
third residual differential picture. The first to third residual
differential pictures are inputs of the selective motion
compensator 307. The selective motion compensator 307 carries out
motion compensation, depending on a motion vector, and outputs a
motion-compensated picture. A picture corresponding to a difference
between the residual picture and the motion-compensated picture is
input back to the first quality layer encoder 301. The first to
third bitstreams are integrated (or multiplexed) into one bitstream
(i.e., an enhancement layer bitstream) by a multiplexer (not
shown), and then transmitted to a decoder over a transport
channel.
[0044] FIG. 4 illustrates the structure of a quality layer encoder
according to an exemplary embodiment.
[0045] Referring to FIG. 4, the quality layer encoder includes a
transform encoder 401, a quantizer 403, a variable-length encoder
405, a de-quantizer 407, and an inverse-transform encoder 409. The
transform encoder 401 calculates a transform coefficient by
transform-encoding a differential picture received from a quality
layer encoder on an N.times.N block basis. The quantizer 403
quantizes the transform coefficient for an N.times.N block, and the
variable-length encoder 405 generates a bitstream from the
quantized level values. The level values quantized by the quantizer
403 restore the transform coefficient by means of the de-quantizer
407. The inverse-transform encoder 409 outputs a reconstructed
differential picture. It does this by restoring the picture, which
is accomplished by performing inverse-transform for the restored
coefficient value.
[0046] In order to achieve gradual quality refinement for a
residual picture, as for residual differential pictures
reconstructed by the quality layer encoders in FIG. 3, the quality
of the residual picture is progressively refined from the first
differential picture on to the third differential picture. The
quality layer encoder in FIG. 4 may achieve such a gradual quality
refinement by adjusting a quantization parameter of the quantizer
403. For example, the quality layer encoder may restore a
differential picture whose quality is gradually refined by
progressively reducing the quantization parameter of the quantizer
403 in order of the first quality layer to the third quality
layer.
[0047] FIG. 5 illustrates the structure of a selective motion
compensator according to an exemplary embodiment. Although the
selective motion compensator in FIG. 5 has three quality layers,
this is merely for the sake of providing a teaching example; the
number of quality layers is subject to change.
[0048] Referring to FIG. 5, the selective motion compensator
includes three frame buffers 501, 503 and 505, the number of which
corresponds to the number of quality layers, and a motion
compensator 507. The residual differential pictures, restored by
their associated quality layer encoders, are added to the
motion-compensated picture output from motion compensator 507. Then
they are stored in their associated frame buffers 501, 503 and 505.
The restored residual pictures stored in the frame buffers 501, 503
and 505 are selectively input to the motion compensator 507 under
control of the encoder, and used for motion compensation. The
encoder transmits, in a header, information indicating with which
frame buffer it has performed motion compensation, so that a
decoder can subsequently perform the same operation.
[0049] FIG. 6 illustrates a structure of a residual encoder
according to another exemplary embodiment. Although the residual
encoder in FIG. 6 has three quality layers, this is merely for the
sake of providing a teaching example; the number of quality layers
is subject to change.
[0050] Referring to FIG. 6, quality layer encoders 601, 603 and 605
for the quality layers may use the structure illustrated in FIGS. 3
and 4, except that only the residual differential pictures output
from the first quality layer encoder 601 for the first quality
layer, and the third quality layer encoder 603 for the last quality
layer, are stored in first and third frame buffers 607 and 609,
respectively, and they are all used for motion compensation of a
motion compensator 611. In a residual encoder with multiple quality
layers, frame buffers are always provided only for reconstructed
differential pictures of the lowest quality layer and the highest
quality layer. In other words, in FIG. 6, the residual encoder with
two or more quality layers always has just two frame buffers,
regardless of the number of quality layers.
[0051] FIG. 7 illustrates the structure of a residual decoder in a
hierarchical picture decoder according to an exemplary embodiment.
Although the residual decoder in FIG. 7 has three quality layers,
this is merely for the sake of providing a teaching example; the
number of quality layers is subject to change.
[0052] Referring to FIG. 7, first to third bitstreams (or
enhancement layer bitstreams being input to the residual decoder)
are input to corresponding first to third quality layer decoders
710, 720 and 730 for their associated quality layers. The first to
third quality layer decoders 710, 720 and 730 include
variable-length decoders 711, 721 and 731, de-quantizers 713, 723
and 733, and inverse-transform encoders 715, 725 and 735,
respectively.
[0053] The first bitstream is converted, in the first
variable-length decoder 711, into quantized level values using
variable-length decoding. Next, it is converted into a transform
coefficient for an N.times.N block, in the first de-quantizer 713,
using a de-quantization operation. The transform coefficient is
inverse-transformed on an N.times.N block basis, in the first
inverse-transform encoder 715, and then added to a
motion-compensated picture from motion compensator 740. The result
is stored in a first frame buffer 750.
[0054] The second bitstream is converted, in the second
variable-length decoder 721, into quantized level values using
variable-length decoding Next, it is converted into a transform
coefficient for an N.times.N block , in the second de-quantizer
723, using a de-quantization operation. The transform coefficient
is inverse-transformed on an N.times.N block basis, in the second
inverse-transform encoder 725, and then added to the sum of the
picture restored in the first quality layer decoder 710 and the
motion-compensated picture from motion compensator 740. The result
is delivered to the third quality layer decoder 730.
[0055] The third bitstream is converted, in the third
variable-length decoder 731, into quantized level values using
variable-length decoding. Next, it is converted into a transform
coefficient for an N.times.N block, in the third de-quantizer 733,
using a de-quantization operation. The transform coefficient is
inverse-transform encoded on an N.times.N block basis in the third
inverse-transform encoder 735, and then added to a sum of the
pictures restored in the first and second quality layer decoders
710 and 720 and of the motion-compensated picture from motion
compensator 740. The result is stored in a third frame buffer 760
The picture stored in the third frame buffer 760 is output as a
residual picture finally restored in the residual decoder.
[0056] The pictures stored in the first and third frame buffers 750
and 760 are used for motion compensation in the motion compensator
740.
[0057] FIG. 8 illustrates the structure of a residual decoder of a
hierarchical picture decoder according to another exemplary
embodiment. Although the residual decoder in FIG. 8 has three
quality layers, this is merely by way of example, and the number of
quality layers is subject to change.
[0058] Referring to FIG. 8, first to third quality layer decoders
810, 820 and 830 are substantially the same as those in FIG. 7, in
terms of the structure and operation, except for an operation of
output stages of the first to third quality layer decoders 810, 820
and 830. In other words, a picture output from a first
inverse-transform encoder 815 in the first quality layer decoder
810 is stored intact in a first frame buffer 850; a picture output
from a second inverse-transform encoder 825 in the second quality
layer decoder 820 is added to the picture output from the first
inverse-transform encoder 815, and delivered to the third quality
layer decoder 830. A picture output from a third inverse-transform
encoder 835 in the third quality layer decoder 830 is added to a
sum of the picture output from the second quality layer decoder 820
and the motion-compensated picture from a motion compensator 840,
and the result is stored in a third frame buffer 860. The picture
stored in the third frame buffer 860 is output as a residual
picture finally restored in the residual decoder.
[0059] In other words, when the quality refinement between pictures
is applied in the hierarchical picture decoder as in the exemplary
embodiment, a residual block is restored by adding a
motion-compensated block to an inverse-transform encoded block in a
quality basement layer (or a first quality layer), and the restored
residual samples constitute a picture of the quality basement
layer. In addition, inverse-transform blocks of quality enhancement
layers (e.g., second and third quality layers in FIGS. 7 and 8)
connected to the quality basement layer are used to refine the
quality of a inverse-transform block of the quality basement layer.
The inverse-transform block whose quality was finally refined is
added to the motion-compensated picture in the motion compensator.
After the quality refinement process, the quality enhancement layer
picture is restored by the restored residual samples.
[0060] FIG. 9 illustrates a residual encoding process according to
an exemplary embodiment.
[0061] Referring to FIG. 9, when a residual picture is input to a
residual encoder in step 901, the residual picture is added to a
picture that has undergone motion compensation in a reference frame
within the same enhancement layer, and then input to quality layer
encoders in step 903. In step 905, bitstreams are independently
generated for quality layers in their associated quality layer
encoders, and residual pictures are restored and stored in frame
buffers. A residual picture restored in a high quality layer
encoder is used to generate a bitstream in the next lower quality
layer encoder. Frame buffers may be provided either for all quality
layers, or for only the highest and lowest quality layers. In step
907, the pictures stored in the frame buffers undergo motion
compensation, and then are used for generating bitstreams of the
next frame. In accordance with an exemplary embodiment, one of the
pictures stored in all frame buffers may selectively undergo motion
compensation. Alternatively, both of the pictures stored in the
frame buffers corresponding to the highest and the lowest quality
layers may undergo motion compensation. In step 909, the bitstreams
generated in their associated quality layer encoders are
multiplexed into one enhancement layer bitstream, and then
transmitted to a decoder.
[0062] FIG. 10 illustrates a residual decoding process according to
an exemplary embodiment.
[0063] Referring to FIG. 10, in step 1001, an enhancement layer
bitstream input to a residual decoder is demultiplexed into
multiple bitstreams, the number of which corresponds to the number
of quality layers. In step 1003, residual pictures for individual
quality layers are restored by independently decoding the
demultiplexed bitstreams for the quality layers by means of quality
layer encoders. In step 1005, the restored residual pictures for
individual quality layers are stored in associated frame buffers
for restoration of residual pictures of the next frame, and then
undergo motion compensation. In step 1007, the finally restored
residual picture is output by adding the motion-compensated picture
of a reference frame within the same enhancement layer to the
restored residual pictures for individual quality layers. The
restored residual pictures are reconstructed as an enhancement
layer picture by being combined with the basement layer picture. In
accordance with an exemplary embodiment, when the finally restored
residual picture is made by combining the residual pictures
independently restored for quality layers, the picture that has
undergone motion compensation in the highest quality layer decoding
process may be reflected, or the picture that has undergone motion
compensation in the lowest quality layer decoding process may be
reflected. In the quality layer decoders, the restored residual
pictures are used for restoring a residual picture in a high
quality layer decoder.
[0064] As is apparent from the foregoing description, according to
the exemplary embodiments, multiple residual pictures have to
undergo motion compensation only once, thus ensuring a
lower-complexity residual encoder/decoder.
[0065] According to the exemplary embodiments, the drift occurs not
in the original picture but in the residual picture, and this
reduces the degradation in picture quality due to drift, and
thereby ensures effective, scalable encoding.
[0066] While the inventive concept has been shown and described
with reference to certain exemplary embodiments thereof, it will be
understood by those skilled in the art that various changes in form
and details may be made therein without departing from the spirit
and scope of the inventive concept as defined by the appended
claims and their equivalents.
* * * * *