U.S. patent application number 16/249063 was filed with the patent office on 2019-05-16 for apparatus and method of sample adaptive offset for luma and chroma components.
The applicant listed for this patent is HFI Innovation Inc.. Invention is credited to Ching-Yeh CHEN, Chih-Ming FU, Yu-Wen HUANG, Shaw-Min LEI, Chia-Yang TSAI.
Application Number | 20190149846 16/249063 |
Document ID | / |
Family ID | 47174900 |
Filed Date | 2019-05-16 |
United States Patent
Application |
20190149846 |
Kind Code |
A1 |
FU; Chih-Ming ; et
al. |
May 16, 2019 |
APPARATUS AND METHOD OF SAMPLE ADAPTIVE OFFSET FOR LUMA AND CHROMA
COMPONENTS
Abstract
A method and apparatus for processing reconstructed video using
in-loop filter in a video coding system are disclosed. The method
uses chroma in-loop filter indication to indicate whether chroma
components are processed by in-loop filter when the luma in-loop
filter indication indicates that in-loop filter processing is
applied to the luma component. An additional flag may be used to
indicate whether the in-loop filter processing is applied to an
entire picture using same in-loop filter information or each block
of the picture using individual in-loop filter information. Various
embodiments according to the present invention to increase
efficiency are disclosed, wherein various aspects of in-loop filter
information are taken into consideration for efficient coding such
as the property of quadtree-based partition, boundary conditions of
a block, in-loop filter information sharing between luma and chroma
components, indexing to a set of in-loop filter information, and
prediction of in-loop filter information.
Inventors: |
FU; Chih-Ming; (Hsinchu
City, TW) ; CHEN; Ching-Yeh; (Taipei City, TW)
; TSAI; Chia-Yang; (New Taipei City, TW) ; HUANG;
Yu-Wen; (Taipei City, TW) ; LEI; Shaw-Min;
(Zhubei City, TW) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
HFI Innovation Inc. |
Zhubei City |
|
TW |
|
|
Family ID: |
47174900 |
Appl. No.: |
16/249063 |
Filed: |
January 16, 2019 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
15015537 |
Feb 4, 2016 |
|
|
|
16249063 |
|
|
|
|
13311953 |
Dec 6, 2011 |
|
|
|
15015537 |
|
|
|
|
61503870 |
Jul 1, 2011 |
|
|
|
61498949 |
Jun 20, 2011 |
|
|
|
61486504 |
May 16, 2011 |
|
|
|
Current U.S.
Class: |
375/240.02 |
Current CPC
Class: |
H04N 19/463 20141101;
H04N 19/157 20141101; H04N 19/117 20141101; H04N 19/70 20141101;
H04N 19/186 20141101; H04N 19/132 20141101; H04N 19/196 20141101;
H04N 19/176 20141101; H04N 19/96 20141101; H04N 19/82 20141101 |
International
Class: |
H04N 19/82 20060101
H04N019/82; H04N 19/176 20060101 H04N019/176; H04N 19/132 20060101
H04N019/132; H04N 19/157 20060101 H04N019/157; H04N 19/186 20060101
H04N019/186; H04N 19/70 20060101 H04N019/70; H04N 19/117 20060101
H04N019/117; H04N 19/96 20060101 H04N019/96; H04N 19/196 20060101
H04N019/196; H04N 19/463 20060101 H04N019/463 |
Claims
1. A method for processing reconstructed video using in-loop filter
in a video decoder, wherein a picture area of the reconstructed
video is partitioned into blocks and the in-loop filter is applied
to the blocks, the method comprising: deriving reconstructed video
data comprising reconstructed block; determining if a current
reconstructed block is a new partition based on a merge flag;
receiving in-loop filter information responsive to the current
reconstructed block being a new partition; deriving the in-loop
filter information from a target block responsive to the current
reconstructed block not being said new partition, wherein the
current reconstructed block is merged with the target block
selected from two candidate blocks corresponding to two neighboring
blocks of the current reconstructed block, and the target block is
selected from said two neighboring blocks according to a second
flag; and applying in-loop filter processing to the current
reconstructed block using the in-loop filter information.
2. The method of claim 1, wherein deriving the in-loop filter
information is performed based on said merge flag responsive to an
existence of more than one neighboring block; and wherein said
deriving the in-loop filter information is inferred responsive to
an existence of only one neighboring block.
3. The method of claim 1, further comprising: applying
quadtree-based partitioning to the picture area, wherein said
quadtree-based partitioning comprises recursively partitioning a
region into four sub-regions until a smallest unit is reached.
4. The method of claim 3, wherein the smallest unit comprises a
largest coding unit (LCU).
5. The method of claim 3, wherein at least one of said one or more
candidate blocks is eliminated from merging with the current
reconstructed block according to a quadtree-partition property and
merging information of said one or more candidate blocks.
6. An apparatus for processing reconstructed video using in-loop
filter in a video decoder, wherein a picture area of the
reconstructed video is partitioned into blocks and the in-loop
filter is applied to the blocks, the apparatus comprising
processing circuitry configured to: derive reconstructed video data
comprising reconstructed block; determine if a current
reconstructed block is a new partition based on a merge flag;
receive in-loop filter information responsive to the current
reconstructed block being a new partition; derive the in-loop
filter information from a target block responsive to the current
reconstructed block not being said new partition, wherein the
current reconstructed block is merged with the target block
selected from two candidate blocks corresponding to two neighboring
blocks of the current reconstructed block, and the target block is
selected from said two neighboring blocks according to a second
flag; and apply in-loop filter processing to the current
reconstructed block using the in-loop filter information.
Description
CROSS REFERENCE TO RELATED APPLICATIONS
[0001] This application is a divisional application of co-pending
U.S. patent application Ser. No. 15/015,537, filed Feb. 4, 2016,
which is a divisional application of co-pending U.S. patent
application Ser. No. 13/311,953, filed Dec. 6, 2011, entitled
"Apparatus and Method of Sample Adaptive Offset for Luma and Chroma
Components", which claims priority to U.S. Provisional Patent
Application, No. 61/486,504, filed May 16, 2011, entitled "Sample
Adaptive Offset for Luma and Chroma Components", U.S. Provisional
Patent Application, No. 61/498,949, filed Jun. 20, 2011, entitled
"LCU-based Syntax for Sample Adaptive Offset", and U.S. Provisional
Patent Application, No. 61/503,870, filed Jul. 1, 2011, entitled
"LCU-based Syntax for Sample Adaptive Offset". The present
invention is also related to U.S. Non-Provisional patent
application Ser. No. 13/158,427, entitled "Apparatus and Method of
Sample Adaptive Offset for Video Coding", filed on Jun. 12, 2011.
The U.S. Provisional Patent Applications and U.S. Non-Provisional
Patent Application are hereby incorporated by reference in their
entireties.
FIELD OF THE INVENTION
[0002] The present invention relates to video processing. In
particular, the present invention relates to apparatus and method
for adaptive in-loop filtering including sample adaptive offset
compensation and adaptive loop filter.
BACKGROUND
[0003] In a video coding system, the video data are subject to
various processing such as prediction, transform, quantization,
deblocking, and adaptive loop filtering. Along the processing path
in the video coding system, certain characteristics of the
processed video data may be altered from the original video data
due to the operations applied to video data. For example, the mean
value of the processed video may be shifted. Intensity shift may
cause visual impairment or artifacts, which is especially more
noticeable when the intensity shift varies from frame to frame.
Therefore, the pixel intensity shift has to be carefully
compensated or restored to reduce the artifacts. Some intensity
offset schemes have been used in the field. For example, an
intensity offset scheme, termed as sample adaptive offset (SAO),
classifies each pixel in the processed video data into one of
multiple categories according to a context selected. The
conventional SAO scheme is only applied to the luma component. It
is desirable to extend SAO processing to the chroma components as
well. The SAO scheme usually requires incorporating SAO information
in the video bitstream, such as partition information to divide a
picture or slice into blocks and the SAO offset values for each
block so that a decoder can operate properly. The SAO information
may take up a noticeable portion of the bitrate of compressed video
and it is desirable to develop efficient coding to incorporate the
SAO information. Besides SAO, adaptive loop filter (ALF) is another
type of in-loop filter often applied to the reconstructed video to
improve video quality. Similarly, it is desirable to apply ALF to
the chroma component as well to improve video quality. Again, ALF
information such as partition information and filter parameters has
to be incorporated in the video bitstream so that a decoder can
operate properly. Therefore, it is also desirable to develop
efficient coding to incorporate the ALF information in the video
bitstream.
BRIEF SUMMARY OF THE INVENTION
[0004] One embodiment, among others, is a method for processing
reconstructed video using in-loop filter in a video decoder,
wherein a picture area of the reconstructed video is partitioned
into blocks and the in-loop filter is applied to the blocks. The
method comprises deriving reconstructed video data comprising
reconstructed block; determining if a current reconstructed block
is a new partition based on a merge flag; receiving in-loop filter
information responsive to the current reconstructed block being a
new partition; deriving the in-loop filter information from a
target block responsive to the current reconstructed block not
being said new partition, wherein the current reconstructed block
is merged with the target block selected from two candidate blocks
corresponding to two neighboring blocks of the current
reconstructed block, and the target block is selected from said two
neighboring blocks according to a second flag; and applying in-loop
filter processing to the current reconstructed block using the
in-loop filter information.
[0005] Another embodiment is a method for processing reconstructed
video using Sample Adaptive Offset in a video encoder. The method
comprises deriving reconstructed video data comprising luma
component and chroma components; incorporating chroma Sample
Adaptive Offset indication in a video bitstream if luma Sample
Adaptive Offset indication indicates that Sample Adaptive Offset
processing is applied to the luma component; incorporating chroma
Sample Adaptive Offset information in the video bitstream if the
chroma Sample Adaptive Offset indication indicates that the Sample
Adaptive Offset processing is applied to the chroma components; and
applying the Sample Adaptive Offset processing to the chroma
components according to the chroma Sample Adaptive Offset
information if the chroma Sample Adaptive Offset indication
indicates that the Sample Adaptive Offset processing is applied to
the chroma components, wherein the chroma Sample Adaptive Offset
information is shared between the chroma components.
[0006] Another embodiment is a method for processing reconstructed
video using Sample Adaptive Offset in a video decoder. The method
comprises deriving reconstructed video data from a video bitstream,
wherein the reconstructed video data comprises luma component and
chroma components; receiving chroma Sample Adaptive Offset
indication from the video bitstream if luma Sample Adaptive Offset
indication in the video bitstream indicates that Sample Adaptive
Offset processing is applied to the luma component; determining
chroma Sample Adaptive Offset information if the chroma Sample
Adaptive Offset indication indicates that the Sample Adaptive
Offset processing is applied to the chroma components; and applying
the Sample Adaptive Offset processing to the chroma components
according to the chroma Sample Adaptive Offset information if the
chroma Sample Adaptive Offset indication indicates that the Sample
Adaptive Offset processing is applied to the chroma components.
BRIEF DESCRIPTION OF THE DRAWINGS
[0007] FIG. 1 illustrates a system block diagram of an exemplary
video encoder incorporating a reconstruction loop, where the
in-loop filter processing includes deblocking filter (DF), sample
adaptive offset (SAO) and adaptive loop filter (ALF).
[0008] FIG. 2 illustrates a system block diagram of an exemplary
video decoder incorporating a reconstruction loop, where the
in-loop filter processing includes deblocking filter (DF), sample
adaptive offset (SAO) and adaptive loop filter (ALF).
[0009] FIG. 3 illustrates an example of sample adaptive offset
(SAO) coding for current block C using information from neighboring
blocks A, D, B and E.
[0010] FIG. 4A illustrates an example of quadtree-based picture
partition for sample adaptive offset (SAO) processing.
[0011] FIG. 4B illustrates an example of LCU-based picture
partition for sample adaptive offset (SAO) processing.
[0012] FIG. 5A illustrates an example of allowable quadtree
partition for block C, where blocks A and D are in the same
partition and block B is in a different partition.
[0013] FIG. 5B illustrates another example of allowable quadtree
partition for block C, where blocks A and D are in the same
partition and block B is in a different partition.
[0014] FIG. 5C illustrates an example of unallowable quadtree
partition for block C, where blocks A and D are in the same
partition and block B is in a different partition.
[0015] FIG. 6A illustrates an example of allowable quadtree
partition for block C, where blocks B and D are in the same
partition and block A is in a different partition.
[0016] FIG. 6B illustrates another example of allowable quadtree
partition for block C, where blocks B and D are in the same
partition and block A is in a different partition.
[0017] FIG. 6C illustrates an example of unallowable quadtree
partition for block C, where blocks B and D are in the same
partition and block A is in a different partition.
[0018] FIG. 7 illustrates an exemplary syntax design to incorporate
a flag in SPS to indicate whether SAO is enable or disabled for the
sequence.
[0019] FIG. 8 illustrates an exemplary syntax design for sao_param(
), where separate SAO information is allowed for the chroma
components.
[0020] FIG. 9 illustrates an exemplary syntax design for
sao_split_param( ), where syntax sao_split_param( ) includes
"component" parameter and "component" indicates either the luma
component or one of the chroma components.
[0021] FIG. 10 illustrates an exemplary syntax design for
sao_offset_param( ), where syntax sao_offset_param( ) includes
"component" as a parameter and "component" indicates either the
luma component or one of the chroma components.
[0022] FIG. 11 illustrates an example of quadtree-based picture
partition for sample adaptive offset (SAO) type determination.
[0023] FIG. 12A illustrates an example of picture-based sample
adaptive offset (SAO), where the entire picture uses same SAO
parameters.
[0024] FIG. 12B illustrates an example of LCU-based sample adaptive
offset (SAO), where each LCU uses its own SAO parameters.
[0025] FIG. 13 illustrates an example of using a run equal to two
for SAO information sharing of the first three LCUs.
[0026] FIG. 14 illustrates an example of using run signals and
merge-above flags to encode SAO information sharing.
[0027] FIG. 15 illustrates an example of using run signals, run
prediction and merge-above flags to encode SAO information
sharing.
DETAILED DESCRIPTION OF THE INVENTION
[0028] In High Efficiency Video Coding (HEVC), a technique named
Adaptive Offset (AO) is introduced to compensate the offset of
reconstructed video and AO is applied inside the reconstruction
loop. A method and system for offset compensation is disclosed in
U.S. Non-Provisional patent application Ser. No. 13/158,427,
entitled "Apparatus and Method of Sample Adaptive Offset for Video
Coding". The method and system classify each pixel into a category
and apply intensity shift compensation or restoration to processed
video data based on the category of each pixel. Besides adaptive
offset, Adaptive Loop Filter (ALF) has also been introduced in HEVC
to improve video quality. ALF applies spatial filter to
reconstructed video inside the reconstruction loop. Both AO and ALF
are considered as a type of in-loop filter in this disclosure.
[0029] The exemplary encoder shown in FIG. 1 represents a system
using intra/inter-prediction. Intra-prediction 110 is responsible
to provide prediction data based on video data in the same picture.
For inter-prediction, motion estimation (ME) and motion
compensation (MC) 112 is used to provide prediction data based on
video data from other picture or pictures. Switch 114 selects
intra-prediction or inter-prediction data and the selected
prediction data are supplied to adder 116 to form prediction
errors, also called residues. The prediction error is then
processed by transformation (T) 118 followed by quantization (Q)
120. The transformed and quantized residues are then coded by
entropy coding 122 to form a bitstream corresponding to the
compressed video data. The bitstream associated with the transform
coefficients is then packed with side information such as motion,
mode, and other information associated with the image area. The
side information may also be subject to entropy coding to reduce
required bandwidth. Accordingly the data associated with the side
information are provided to entropy coding 122 as shown in FIG. 1.
When an inter-prediction mode is used, a reference picture or
reference pictures have to be reconstructed at the encoder end.
Consequently, the transformed and quantized residues are processed
by inverse quantization (IQ) 124 and inverse transformation (IT)
126 to recover the residues. The residues are then added back to
prediction data 136 at reconstruction (REC) 128 to reconstruct
video data. The reconstructed video data may be stored in reference
picture buffer 134 and used for prediction of other frames. As it
is shown in FIG. 1, incoming video data undergo a series of
processing in the encoding system. The reconstructed video data
from REC 128 may be subject to intensity shift and other noises due
to the series of processing. Accordingly, deblocking filter 130,
sample adaptive offset (SAO) 131 and adaptive loop filter (ALF) 132
are applied to the reconstructed video data before the
reconstructed video data are stored in the reference picture buffer
134 in order to improve video quality. The adaptive offset
information and adaptive loop filter information may have to be
transmitted in the bitstream so that a decoder can properly recover
the required information in order to apply the adaptive offset and
adaptive loop filter. Therefore, adaptive offset information from
AO 131 and adaptive loop filter information from ALF 132 are
provided to entropy coding 122 for incorporation into the
bitstream. The encoder may need to access to the original video
data in order to derive AO information and ALF information. The
paths from the input to AO 131 and ALF 132 are not explicitly shown
in FIG. 1.
[0030] FIG. 2 illustrates a system block diagram of an exemplary
video decoder including deblocking filter and adaptive loop filter.
Since the encoder also contains a local decoder for reconstructing
the video data, some decoder components are already used in the
encoder except for the entropy decoder 222. Furthermore, only
motion compensation 212 is required for the decoder side. The
switch 214 selects intra-prediction or inter-prediction and the
selected prediction data are supplied to reconstruction (REC) 128
to be combined with recovered residues. Besides performing entropy
decoding on compressed video data, entropy decoding 222 is also
responsible for entropy decoding of side information and provides
the side information to respective blocks. For example, intra mode
information is provided to intra-prediction 110, inter mode
information is provided to motion compensation 212, adaptive offset
information is provided to SAO 131, adaptive loop filter
information is provided to ALF 132 and residues are provided to
inverse quantization 124. The residues are processed by IQ 124, IT
126 and subsequent reconstruction process to reconstruct the video
data. Again, reconstructed video data from REC 128 undergo a series
of processing including IQ 124 and IT 126 as shown in FIG. 2 and
are subject to intensity shift. The reconstructed video data are
further processed by deblocking filter 130, sample adaptive offset
131 and adaptive loop filter 132.
[0031] The in-loop filtering is only applied to the luma component
of reconstructed video according to the current HEVC standard. It
is beneficial to apply in-loop filtering to chroma components of
reconstructed video as well. The information associated with
in-loop filtering for the chroma components may be sizeable.
However, a chroma component typically results in much smaller
compressed data than the luma component. Therefore, it is desirable
to develop a method and apparatus for applying in-loop filtering to
the chroma components efficiently. Accordingly, an efficient method
and apparatus of SAO for chroma component are disclosed.
[0032] In one example incorporating an embodiment of the present
invention, an indication is provided for signaling whether in-loop
filtering is turned ON or not for chroma components when SAO for
the luma component is turned ON. If SAO for the luma component is
not turned ON, the SAO for the chroma components is also not turned
ON. Therefore, there is no need to provide the indication for
signaling whether in-loop filtering is turned ON or not for the
chroma components in this case. A example of pseudo codes for the
embodiment mentioned above is shown below: [0033] If SAO for luma
is turned ON; [0034] A flag is signaled to indicate whether SAO for
chroma is turned ON or not. [0035] Else; [0036] The flag is
signaled.
[0037] The flag to indicate if SAO for chroma is turned ON is
called chroma in-loop filter indication since it can be used for
SAO as well as ALF. SAO is one example of in-loop filter
processing, where the in-loop filter processing may be ALF. In
another example incorporating an embodiment of the present
invention, individual indications are provided for signaling
whether in-loop filtering is turned ON or not for chroma components
Cb and Cr when SAO for the luma component is turned ON. If SAO for
the luma component is not turned ON, the SAO for the two chroma
components is also not turned ON. Therefore, there is no need to
provide the individual indications for signaling whether in-loop
filtering is turned ON or not for the two chroma components in this
case. A example of pseudo codes for the embodiment mentioned above
is shown below: [0038] If SAO for luma is turned ON; [0039] A first
flag is signaled to indicate whether SAO for Cb is turned ON or
not; [0040] A second flag is signaled to indicate whether SAO for
Cr is turned ON or not. [0041] Else; [0042] Neither the first flag
nor the second flag is signaled.
[0043] As mentioned before, it is desirable to develop efficient
in-loop filtering method. For example, it is desired to reduce
information required to provide indication regarding whether SAO is
turned ON and SAO parameters if SAO is turned ON. Since neighboring
blocks often have similar characteristics, neighboring blocks may
be useful in reducing requiring SAO information. FIG. 3 illustrates
an example of utilizing neighboring block to reduce SAO
information. Block C is the current block being processed by SAO.
Blocks B, D, E and A are previously processed neighboring blocks
around C, as shown in FIG. 3. The block-based syntax represents the
parameters of current processing block. A block can be a coding
unit (CU), a largest coding unit (LCU), or multiple LCUs. A flag
can be used to indicate that the current block shares the SAO
parameters with neighboring blocks to reduce the rate. If the
processing order of blocks is raster scan, the parameters of blocks
D, B, E, and A are available when the parameters of block C are
encoded. When the block parameters are available from neighboring
blocks, these block parameters can be used to encode the current
block. The amount of data required to send the flag to indicate SAO
parameter sharing is usually much less than that for SAO
parameters. Therefore, efficient SAO is achieved. While SAO is used
as an example of in-loop filter to illustrate parameter sharing
based on neighboring blocks, the technique can also be applied to
other in-loop filter such as ALF.
[0044] In the current HEVC standard, the quadtree-based algorithm
can be used to adaptively divide a picture region into four
sub-regions to achieve better performance. In order to maintain the
coding gain of SAO, the encoding algorithm for the quadtree-based
SAO partition has to be efficiently designed. The SAO parameters
(SAOP) include SAO type index and offset values of the selected
type. An exemplary quadtree-based SAO partition is shown in FIGS.
4A and 4B. FIG. 4A represents a picture being partitioned using
quadtree partition, where each small square corresponds to an LCU.
The first partition (depth 0 partition) is indicated by split_0( ).
A value 0 implies no split and a value 1 indicates a split applied.
The picture consists of twelve LCUs as labeled by P1, P2, . . . ,
P12 in FIG. 4B. The depth-0 quadtree partition, split_0(1) splits
the picture into four regions: upper left, upper right, lower left
and lower right regions. Since the lower left and lower right
regions have only one row of blocks, no further quadtree partition
is applied. Therefore, depth-1 quadtree partition is only
considered for the upper left and upper right regions. The example
in FIG. 4A shows that the upper left region is not split as
indicated by split_1(0) and the upper right region is further split
into four regions as indicated by split_1(1). Accordingly, the
quadtree partition results in seven partitions labeled as P'0, . .
. , P'6 in FIG. 4A, where: [0045] SAOP of P1 is the same as SAOP
for P2, P5, and P6; [0046] SAOP of P9 is the same as SAOP for P10;
and [0047] SAOP of P11 is the same as SAOP for P12.
[0048] According to the partition information of SAO, each LCU can
be a new partition or merged with other LCUs. If the current LCU is
merged, several merge candidates can be selected. To illustrate an
exemplary syntax design to allow information sharing, only two
merge candidates are allowed for quad-tree partitioning of FIG. 3.
While two candidates are illustrated in the example, more
candidates from the neighboring blocks may be used to practice the
present invention. The syntax design is illustrated as follows:
[0049] If block C is not the first block of the picture, [0050] Use
one flag to indicate block C is a new partition. [0051] Else,
[0052] Block C is inferred as a new partition. [0053] If block C is
a new partition, [0054] Encode SAO parameters. [0055] Otherwise,
[0056] If a left neighbor and a top neighbor exist, [0057] Send a
mergeLeftFlag. [0058] If mergeLeftFlag is true, then block C is
merged with block A. [0059] Otherwise, block C is merged with block
B. [0060] Else, [0061] If a left neighbor exists, then block C is
merged with block A. [0062] Otherwise, block C is merged with block
B.
[0063] In another embodiment according to the present invention,
the relation with neighboring blocks (LCUs) and the properties of
quadtree partition are used to reduce the amount of data required
to transmit SAO related information. Furthermore, the boundary
condition of a picture region such as a slice may introduce some
redundancy in dependency among neighboring blocks and the boundary
condition can be used to reduce the amount of data required to
transmit SAO related information. The relation among neighboring
blocks may also introduce redundancy in dependency among
neighboring blocks and the relation among neighboring blocks may be
used to reduce the amount of data required to transmit SAO related
information.
[0064] An example of redundancy in dependency among neighboring
blocks is illustrated in FIGS. 5A-C. According to the property of
quadtree partition, if blocks D and A are in the same partition and
block B is in another partition, blocks A and C will be in
different partitions as shown in FIG. 5A and FIG. 5B. On the other
hand, the case shown in FIG. 5C is not allowed in quadtree
partition. Therefore, the merge-candidate in FIG. 5C is redundant
and there is no need to assign a code to represent the merge flag
corresponding to FIG. 5C. Exemplary pseudo codes to implement the
merge algorithm are shown as follows: [0065] If blocks A and D are
in the same partition and blocks B and D are in different
partitions, [0066] Send newPartitionFlag to indicate that block C
is a new partition. [0067] If newPartitionFlag is true, [0068]
Block C is a new partition as shown in FIG. 5A. [0069] Otherwise,
[0070] Block C is merged with block B without signaling as shown in
FIG. 5B
[0071] As shown in the above example, there are only two allowed
cases, i.e. block C is a new partition or block C is merged with
block B. Therefore, a single bit for newPartitionFlag is adequate
to identify the two cases. In another example, blocks D and B are
in the same partition and block A is in another partition, blocks B
and C will be in different partitions as shown in FIG. 6A and FIG.
6B. On the other hand, the case shown in FIG. 6C is not allowed
according to quadtree partition. Therefore, the merge-candidate
associated with the case in FIG. 6C is redundant and there is no
need to assign a code to represent the merge flag corresponding to
FIG. 6C. Exemplary pseudo codes to implement the merge algorithm
are shown as follows: [0072] If blocks B and D are in the same
partition and blocks A and D are in different partitions, [0073]
Send newPartitionFlag to indicate that block C is a new partition.
[0074] If newPartitionFlag is true, [0075] Block C is a new
partition as shown in FIG. 6A. [0076] Otherwise, [0077] Block C is
merged with block A without signaling as shown in FIG. 6B.
[0078] FIGS. 5A-C and FIG. 6A-C illustrate two examples of
utilizing redundancy in dependency among neighboring blocks to
further reduce transmitted data associated with SAO information for
the current block. There are many other conditions that the system
can take advantage of the redundancy in dependency among
neighboring blocks. For example, if blocks A, B and D are in the
same partition, then block C cannot be in another partition.
Therefore, block C must be in the same partition as A, B, and D and
there is no need to transmit an indication of SAO information
sharing. The LCU block in the slice boundary can be taken into
consideration to reduce the transmitted data associated with SAO
information for the current block. For example, if block A does not
exist, only one direction can be merged. If block B does not exist,
only one direction can be merged as well. If both blocks A and B do
not exist, there is no need to transmit a flag to indicate block C
as a new partition. To further reduce the number of transmitted
syntax elements, a flag can be used to indicate that current slice
uses only one SAO type without any LCU-based signaling. When the
slice is a single partition, the number of transmitted syntax
elements can also be reduced. While LCU is used as a unit of block
in the above examples, other block configurations (such as block
size and shape) may also be used. While slice is mentioned here as
an example of picture area that the blocks are grouped to share
common information, other picture areas such as group of slices and
a picture may also be used.
[0079] In addition, chroma and luma components may share the same
SAO information for color video data. The SAO information may also
be shared between chroma components. For example, chroma components
(Cb and Cr) may use the partition information of luma so that there
is no need to signal the partition information for the chroma
components. In another example, Cb and Cr may share the same SAO
parameters (SAOP) and therefore only one set of SAOP needs to be
transmitted for Cb and Cr to share. SAO syntax for luma can be used
for chroma components where the SAO syntax may include quadtree
syntax and LCU-based syntax.
[0080] The examples of utilizing redundancy in dependency among
neighboring blocks as shown in FIGS. 5A-C and FIG. 6A-C to reduce
transmitted data associated with SAO information can also be
applied to the chroma components. The SAOP including SAO type and
SAO offset values of the selected type can be coded before
partitioning information, and therefore an SAO parameter set
(SAOPS) can be formed. Accordingly, indexing can be used to
identify SAO parameters from the SAOPS for the current block where
the data transmitted for the index is typically less than the data
transmitted for the SAO parameters. When partition information is
encoded, the selection among SAOPS can be encoded at the same time.
The number of SAOPS can be increased dynamically. For example,
after a new SAOP is signaled, the number of SAOP in SAOPS will be
increased by one. To represent the number of SAOPS, the number of
bits can be dynamically adjusted to match the data range. For
example, three bits are required to represent SAOPS having five to
eight members. When a new SAOP is signaled, the number of SAOPS
will grow to nine and four bits will be needed to represent the
SAOPS having nine members.
[0081] If the processing of SAO refers to the data located in the
other slice, SAO will avoid fetching data from any other slice by
use a padding technique or change pattern to replace data from
other slices. To reduce data required for SAO information, SAO
parameters can be transmitted in a predicted form, such as the
difference between SAO parameters for a current block and the SAO
parameters for a neighboring block or neighboring blocks. Another
embodiment according to the present invention is to reduce SAO
parameters for chroma. For example, Edge-based Offset (EO)
classification classifies each pixel into four categories for the
luma component. The number of EO categories for the chroma
components can be reduced to two to reduce the transmitted data
associated with SAO information for the current block. The number
of bands for band offset (BO) classification is usually sixteen for
the luma component. In yet another example, the number of bands for
band offset (BO) classification may be reduced to eight for the
chroma components.
[0082] The example in FIG. 3 illustrates a case that current block
C has four merge candidates, i.e., blocks A, B, D and E. The number
of merge candidates can be reduced if the merge candidates are in
the same partition. Accordingly, the number of bits to indicate
which merge candidate is selected can be reduced or saved. If the
processing of SAO refers to the data located in the other slice,
SAO will avoid fetching data from any other slice and skip the
current processing pixel to avoid data from other slices. In
addition, a flag may be used to control whether the SAO processing
avoids fetching data from any other slice. The control flag
regarding whether the SAO processing avoids fetching data from any
other slice can be incorporated in a sequence level or a picture
level. The control flag regarding whether the SAO processing avoids
fetching data from any other slice can also be shared with the
non-crossing slice boundary flag of adaptive loop filter (ALF) or
deblocking filter (DF). In order to further reduce the transmitted
data associated with SAO information, the ON/OFF control of chroma
SAO depend on luma SAO ON/OFF information. The category of chroma
SAO can be a subset of luma SAO for a specific SAO type.
[0083] Exemplary syntax design incorporating various embodiments
according to the present invention is illustrated below. FIG. 7
illustrates an example of incorporating sao_used_flag in the
sequence level data, such as Sequence Parameter Set (SPS). When
sao_used_flag has a value 0, SAO is disabled for the sequence. When
sao_used_flag has a value 1, SAO is enabled for the sequence. An
exemplary syntax for SAO parameters is shown in FIG. 8, where the
sao_param( ) syntax can be incorporated in Adaptation Parameter Set
(APS), Picture Parameter Set (PPS) or slice header. The APS is
another picture-level header in addition to the PPS to accommodate
parameters that are likely to change from picture to picture. If
sao_flag indicates that the SAO is enabled, the syntax will include
split parameter sao_split_param(0, 0, 0, 0) and offset parameter
sao_offset_param(0, 0, 0, 0) for the luma component. Furthermore,
the syntax also includes SAO flag sao_flag_cb for the Cb component
and SAO flag sao_flag_cr for the Cr component. If sao_flag_cb
indicates that the SAO for the Cb component is enabled, the syntax
will include split parameter sao_split_param(0, 0, 0, 1) and offset
parameter sao_offset_param(0, 0, 0, 1) for chroma component Cb. If
sao_flag_cr indicates that the SAO for the Cr component is enabled,
the syntax will include split parameter sao_split_param(0, 0, 0, 2)
and offset parameter sao_offset_param(0, 0, 0, 2) for chroma
component Cr. FIG. 9 illustrates an exemplary syntax for
sao_split_param(rx, ry, Depth, component), where the syntax is
similar to a conventional sao_split_param ( ) except that an
additional parameter "component" is added, where "component" is
used to indicate the luma or one of the chroma components. FIG. 10
illustrates an exemplary syntax for sao_offset_param(rx, ry, Depth,
component), where the syntax is similar to a conventional
sao_offset_param ( ) except that an additional parameter
"component" is added. In sao_offset_param(rx, ry, Depth,
component), the syntax includes sao_type_idx
[component][Depth][ry][rx] if the split flag
sao_split_flag[component][Depth][ry][rx] indicates the region is
not further split. Syntax sao_type_idx [component][Depth][ry][rx]
specification is shown in Table 1.
TABLE-US-00001 TABLE 1 Number of categories, sample adaptive offset
type to nSaoLength sao_type_idx be used [sao_type_idx] 0 None 0 1
1-D 0-degree pattern edge offset 4 2 1-D 90-degree pattern edge 4
offset 3 1-D 135-degree pattern edge 4 offset 4 1-D 45-degree
pattern edge 4 offset 5 central bands band offset 16 6 side bands
band offset 16
[0084] The sample adaptive offset (SAO) adopted in HM-3.0 uses a
quadtree-based syntax, which divides a picture region into four
sub-regions using a split flag recursively, as shown in FIG. 11.
Each leaf region has its own SAO parameters (SAOP), where the SAOP
includes the information of SAO type and the offset values to be
applied for the region. FIG. 11 illustrates an example where the
picture is divided into seven leaf regions, 1110 through 1170,
where band offset (BO) type SAO is applied to leaf regions 1110 and
1150, edge offset (EO) type SAO is applied to leaf regions 1130,
1140 and 1160, and SAO is turned off for leaf regions 1120 and
1170. In order to improve the coding gain, a syntax design
incorporating an embodiment according to the present invention uses
a picture-level flag to switch between picture-based SAO and
block-based SAO, where the block may be an LCU or other block
sizes. FIG. 12A illustrates an example of picture-based SAO and
FIG. 12B illustrates a block-based SAO, where each region is one
LCU and there are fifteen LCUs in the picture. In picture-based
SAO, the entire picture shares one SAOP. It is also possible to use
slice-based SAO so that the entire slice or multiple slices share
one SAOP. In LCU-based SAO, each LCU has its own SAOP and SAOP1
through SAOP15 are used by the fifteen LCUs (LCU1 through LCU15)
respectively.
[0085] In another embodiment according to the present invention,
SAOP for each LCU may be shared by following LCUs. The number of
consecutive subsequent LCUs sharing the same SAOP may be indicated
by a run signal. FIG. 13 illustrates an example where SAOP1, SAOP2
and SAOP3 are the same. In other words, the SAOP of the first LCU
is SAOP1, and SAOP1 is used for the subsequent two LCUs. In this
case, a syntax "run=2" will be encoded to signal the number of
consecutive subsequent LCUs sharing the same SAOP. Since the SAOP
for the next two LCUs is not transmitted, the rate of encoding
their SAOPs can be saved. In yet another embodiment according to
the present invention, in addition to use a run signal, the LCU in
a following row according to the raster scan order may share the
SAOP of a current LCU. A merge-above flag may be used to indicate
the case that the current LCU shares the SAOP of the LCU above if
the above LCU is available. If the merge-above flag is set to "1",
the current LCU will use the SAOP of the LCU above. As shown in
FIG. 14, SAOP2 is shared by four LCUs, 1410 through 1440, where
"run=1" and "no merge-above" are used to indicate LCUs 1410 and
1420 share SAOP2 and they do not share SAOP with LCUs above.
Furthermore, "run=1" and "merge-above=1" are used to indicate LCUs
1430 and 1440 share SAOP2 and they also share SAOP with LCUs above.
On the other hand, both SAOP1 and SAOP3 are shared by two
subsequent LCUs and SAOP4 is shared by four subsequent LCUs.
Accordingly, the run signal for SAOP1, SAOP3 and SAOP4 are 2, 2 and
4 respectively. Since none of them shares SAOP with LCUs above, the
merge-above syntax has a value 0 for blocks associated SAOP1, SAOP3
and SAOP4.
[0086] In order to reduce the bitrate for the run signal, the run
signal of the above LCU can be used as a predictor for the run
signal of the current LCU. Instead of encoding the run signal
directly, the difference of the two run signals is encoded, where
the difference is denoted as d run as shown in FIG. 15. When the
above LCU is not the first LCU of an LCU group with a run value,
the run prediction value can be the run of the above LCU group
subtracted by the number of LCUs that are prior to the above LCU in
the same LCU group. The first LCU sharing SAOP3 has a run value of
2 and the first LCU above also has a run value of 2 (sharing
SAOP1). Accordingly, d run for the LCU sharing SAOP3 has a value of
0. The first LCU sharing SAOP4 has a run value of 4 and the first
LCU above also has a run value of 2 (sharing SAOP3). Accordingly, d
run for the LCU sharing SAOP4 has a value of 2. If the predictor of
a run is not available, the run may be encoded by using an unsigned
variable length code (U_VLC). If the predictor exists, the delta
run, d run may be encoded by using a signed variable length code
(S_VLC). The U_VLC and S_VLC can be k-th order exp-Golomb coding,
Golomb-Rice coding, or a binarization process of CABAC coding.
[0087] In one embodiment according to the present invention, a flag
may be used to indicate that all SAOPs in the current LCU row are
the same as those in the above LCU row. For example, a flag,
RepeatedRow, for each LCU row can be used to indicate that all
SAOPs in this LCU row are the same as those in the above LCU row.
If RepeatedRow flag is equal to 1, no more information needs to be
coded. For each LCU in the current LCU row, the related SAOP is
copied from the LCU in the above LCU row. If RepeatedRow flag is
equal to 0, the SAOPs of this LCU row are coded.
[0088] In another embodiment according to the present invention, a
flag may be used to signal whether RepeatedRow flag is used or not.
For example, the EnableRepeatedRow flag can be used to indicate
whether RepeatedRow flag is used or not. The EnableRepeatedRow flag
can be signaled at a slice or picture level. If EnableRepeatedRow
is equal to 0, the RepeatedRow flag is not coded for each LCU row.
If EnableRepeatedRow is equal to 1, the RepeatedRow flag is coded
for each LCU row.
[0089] In yet another embodiment according to the present
invention, the RepeatedRow flag at the first LCU row of a picture
or a slice can be saved. For the case of a picture having only one
slice, the RepeatedRow flag of the first LCU row can be saved. For
the case of one picture with multiple slices, if the SAO process is
slice-independent operation, the RepeatedRow flag of the first LCU
row in a slice can be saved; otherwise, the RepeatedRow flag will
be signaled. The method of saving RepeatedRow flag at the first LCU
row of one picture or one slice can also be applied to the case
where the EnableRepeatedRow flag is used.
[0090] To reduce transmitted data associated with SAOP, an
embodiment according to the present invention uses a run signal to
indicate that all of SAOPs in the following LCU rows are the same
as those in the above LCU row. For example, for N consecutive LCU
rows containing the same SAOP, the SAOP and a run signal equal to
N-1 are signaled at the first LCU row of the N consecutive repeated
LCU rows. The maximum and minimum runs of the repeated LCU rows in
one picture or slice can be derived and signaled at slice or
picture level. Based on the maximum and minimum values, the run
number can be coded using a fixed-length code word. The word length
of the fixed-length code can be determined according to the maximum
and minimum run values and thus can be adaptively changed at slice
or picture level.
[0091] In another embodiment according to the present invention,
the run number in the first LCU row of a picture or a slice is
coded. In the method of entropy coding of runs and delta-runs
mentioned earlier for the first LCU row of one picture or one
slice, if the SAOP is repeated for consecutive LCUs, a run is coded
to indicate the number of LCUs sharing the SAOP. If the predictor
of a run is not available, the run can be encoded by using unsigned
variable length code (U_VLC) or fixed-length code word. If the
fixed-length code is used, the word length can be coded adaptively
based on the image width, the coded runs, or the remaining LCU, or
the word length can be fixed based on the image width or be
signaled to the decoder. For example, an LCU row in a picture has N
LCUs and the LCU being SAO processed is the k-th LCU in the LCU
row, where k=0 . . . N-1. If a run needs to be coded, the maximum
number of run is N-1-k. The word length of the to-be-coded run is
floor(log 2(N-1-k)+1). In another example, the maximum and minimum
number of run in a slice or picture can be calculated first. Based
on the maximum and minimum value, the word length of the
fixed-length code can be derived and coded.
[0092] In yet another embodiment according to the present
invention, the information for the number of runs and delta-runs
can be incorporated at slice level. The number of runs, delta-runs
or the number of LCUs, NumSaoRun, is signaled at slice level. The
number of LCUs for the current coding SAOP can be specified using
the NumSaoRun flag. Furthermore, the number of runs and delta-runs
or the number of LCUs can be predicted using the number of LCUs in
one coding picture. The prediction equation is given by:
NumSaoRun=sao_num_run_info+NumTBsInPicture,
where NumTBsInPicture is the number of LCUs in one picture and
sao_num_run_info is the predicted residual value. Syntax
sao_num_run_info can be coded using a signed or unsigned
variable-length. Syntax sao_num_run_info may also be coded using a
signed or unsigned fixed-length code word.
[0093] Embodiment of in-loop filter according to the present
invention as described above may be implemented in various
hardware, software codes, or a combination of both. For example, an
embodiment of the present invention can be a circuit integrated
into a video compression chip or program codes integrated into
video compression software to perform the processing described
herein. An embodiment of the present invention may also be program
codes to be executed on a Digital Signal Processor (DSP) to perform
the processing described herein. The invention may also involve a
number of functions to be performed by a computer processor, a
digital signal processor, a microprocessor, or field programmable
gate array (FPGA). These processors can be configured to perform
particular tasks according to the invention, by executing
machine-readable software code or firmware code that defines the
particular methods embodied by the invention. The software code or
firmware codes may be developed in different programming languages
and different format or style. The software code may also be
compiled for different target platform. However, different code
formats, styles and languages of software codes and other means of
configuring code to perform the tasks in accordance with the
invention will not depart from the spirit and scope of the
invention.
[0094] The invention may be embodied in other specific forms
without departing from its spirit or essential characteristics. The
described examples are to be considered in all respects only as
illustrative and not restrictive. The scope of the invention is,
therefore, indicated by the appended claims rather than by the
foregoing description. All changes which come within the meaning
and range of equivalency of the claims are to be embraced within
their scope.
* * * * *