U.S. patent application number 16/821764 was filed with the patent office on 2020-07-09 for strong intra smoothing for in rext.
The applicant listed for this patent is SONY CORPORATION. Invention is credited to OHJI NAKAGAMI, TERUHIKO SUZUKI, ALI J. TABATABAI, JUN XU.
Application Number | 20200221136 16/821764 |
Document ID | / |
Family ID | 54007361 |
Filed Date | 2020-07-09 |
United States Patent
Application |
20200221136 |
Kind Code |
A1 |
NAKAGAMI; OHJI ; et
al. |
July 9, 2020 |
STRONG INTRA SMOOTHING FOR IN REXT
Abstract
Based on conditions or restrictions, Strong Intra Smoothing
(SIS) is enabled or disabled. SIS is not allowed for chroma when
the chroma format is 444, and SIS is not allowed for chroma when
the chroma format is 422 or 444.
Inventors: |
NAKAGAMI; OHJI; (TOKYO,
JP) ; SUZUKI; TERUHIKO; (TOKYO, JP) ;
TABATABAI; ALI J.; (CUPERTINO, CA) ; XU; JUN;
(SUNNYVALE, CA) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
SONY CORPORATION |
Tokyo |
|
JP |
|
|
Family ID: |
54007361 |
Appl. No.: |
16/821764 |
Filed: |
March 17, 2020 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
16246651 |
Jan 14, 2019 |
|
|
|
16821764 |
|
|
|
|
14635176 |
Mar 2, 2015 |
10230992 |
|
|
16246651 |
|
|
|
|
61971140 |
Mar 27, 2014 |
|
|
|
61954248 |
Mar 17, 2014 |
|
|
|
61947207 |
Mar 3, 2014 |
|
|
|
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
H04N 19/186 20141101;
H04N 19/117 20141101; H04N 19/179 20141101; H04N 19/157 20141101;
H04N 19/86 20141101 |
International
Class: |
H04N 19/86 20060101
H04N019/86; H04N 19/117 20060101 H04N019/117; H04N 19/157 20060101
H04N019/157; H04N 19/186 20060101 H04N019/186 |
Claims
1. An image processing device, comprising: a processor configured
to: encode an image such that non-application of strong intra
smoothing to chroma in a case that chroma format is 444; and encode
the image such that application of the strong intra smoothing to
the chroma in a case that all of the following conditions are true:
strong_intra_smoothing_enabled_flag is equal to 1, nTbS is equal to
32, cldx is equal to 0,
Abs(p[-1][-1]+p[nTbS*2-1][-1]-2*p[nTbS-1][-1]<(1<<(BitDepthY-5))-
,
Abs(p[-1][-1]+p[-1][nTbS*2-1]-2*p[-1][nTbS-1])<(1<<(BitDepthY-5-
)).
2. An image processing method, comprising: in an image processing
device: encoding an image such that non-application of strong intra
smoothing to chroma in a case that chroma format is 444; and
encoding the image such that application of the strong intra
smoothing to the chroma in a case that all of the following
conditions are true: strong_intra_smoothing_enabled_flag is equal
to 1, nTbS is equal to 32, cldx is equal to 0,
Abs(p[-1][-1]+p[nTbS*2-1][-1]-2*p[nTbS-1][-1]<(1<<(BitDepthY-5))-
,
Abs(p[-1][-1]+p[-1][nTbS*2-1]-2*p[-1][nTbS-1])<(1<<(BitDepthY-5-
)).
3. An image processing device, comprising: a processor configured
to: decode a bitstream in a state that strong intra smoothing is
disable in a case that chroma format is 444; and decode the
bitstream in a state that the strong intra smoothing is enable in a
case that all of the following conditions are true:
strong_intra_smoothing_enabled_flag is equal to 1, nTbS is equal to
32, cldx is equal to 0,
Abs(p[-1][-1]+p[nTbS*2-1][-1]-2*p[nTbS-1][-1]<(1<<(BitDepthY-5))-
,
Abs(p[-1][-1]+p[-1][nTbS*2-1]-2*p[-1][nTbS-1])<(1<<(BitDepthY-5-
)).
4. An image processing method, comprising: in an image processing
device: decoding a bitstream in a state that strong intra smoothing
is disable in a case that chroma format is 444; and decoding the
bitstream in a state that the strong intra smoothing is enable in a
case that all of the following conditions are true:
strong_intra_smoothing_enabled_flag is equal to 1, nTbS is equal to
32, cldx is equal to 0,
Abs(p[-1][-1]+p[nTbS*2-1][-1]-2*p[nTbS-1][-1]<(1<<(BitDepthY-5))-
,
Abs(p[-1][-1]+p[-1][nTbS*2-1]-2*p[-1][nTbS-1])<(1<<(BitDepthY-5-
)).
Description
CROSS-REFERENCE TO RELATED APPLICATION(S)
[0001] This application is a continuation of U.S. Ser. No.
16/246,651, filed Jan. 14, 2019, which is a continuation of U.S.
Ser. No. 14/635,176, filed Mar. 2, 2015, now U.S. Pat. No.
10,230,992, and also claims priority under 35 U.S.C. .sctn. 119(e)
of U.S. Provisional Application Ser. No. 61/971,140, filed Mar. 27,
2014; U.S. Provisional Application Ser. No. 61/954,248, filed Mar.
17, 2014; and U.S. Provisional Patent Application Ser. No.
61/947,207, filed Mar. 3, 2014, each of which are hereby
incorporated by reference in their entireties for all purposes.
FIELD OF THE INVENTION
[0002] The present invention relates to video coding. More
specifically, the present invention relates to high efficiency
video coding.
BACKGROUND OF THE INVENTION
[0003] In JCTVC-P1005_v1, Strong Intra Smoothing (SIS) is allowed
to apply to chroma when chroma format is 4:4:4. When
intra_smoothing_disabled_flag is equal to 0 and either cldx is
equal to 0 or ChromaArrayType is equal to 3, the filtering process
of neighboring samples specified is invoked with the sample array p
and the transform block size nTbS as inputs, and the output is
reassigned to the sample array p. In RExt6.0 SIS is only allowed
for luma. Thus, there is a mismatch on how to deal with SIS for the
chroma in 444 chroma format.
SUMMARY OF THE INVENTION
[0004] Based on conditions or restrictions, Strong Intra Smoothing
(SIS) is enabled or disabled. SIS is not allowed for chroma when
the chroma format is 444, and SIS is not allowed for chroma when
the chroma format is 422 or 444.
[0005] In one aspect, a method programmed in a non-transitory
memory of a device comprises acquiring video content and encoding
the video content without implementing strong intra smoothing
depending on the chroma format of the video content. The method
further comprises determining the chroma format is 444; and not
enabling strong intra smoothing. The method further comprises
determining the chroma format is 422 or 444; and not enabling
strong intra smoothing. A strong intra smoothing flag is equal to 1
and a image component index is equal to 0. The device is selected
from the group consisting of a personal computer, a laptop
computer, a computer workstation, a server, a mainframe computer, a
handheld computer, a personal digital assistant, a cellular/mobile
telephone, a smart appliance, a gaming console, a digital camera, a
digital camcorder, a camera phone, a smart phone, a portable music
player, a tablet computer, a mobile device, a video player, a video
disc writer/player, a television, a home entertainment system and
smart jewelry.
[0006] In another aspect, a system programmed in a non-transitory
memory of a device comprises a receiving device configured for
receiving encoded video content, wherein the encoded video content
was encoded without using strong intra smoothing depending on the
chroma format of the video content and a display device configured
for displaying the encoded video content. Encoding the encoded
video content included determining the chroma format is 444; and
not enabling strong intra smoothing. Encoding the encoded video
content included determining the chroma format is 422 or 444 and
not enabling strong intra smoothing. A strong intra smoothing flag
is equal to 1 and a image component index is equal to 0. The device
is selected from the group consisting of a personal computer, a
laptop computer, a computer workstation, a server, a mainframe
computer, a handheld computer, a personal digital assistant, a
cellular/mobile telephone, a smart appliance, a gaming console, a
digital camera, a digital camcorder, a camera phone, a smart phone,
a portable music player, a tablet computer, a mobile device, a
video player, a video disc writer/player, a television, a home
entertainment system and smart jewelry.
[0007] In another aspect, an apparatus comprises a display device,
a non-transitory memory for storing an application, the application
for receiving video content, wherein the video content was encoded
without using strong intra smoothing depending on the chroma format
of the video content and displaying the video content on the
display device and a processing component coupled to the memory,
the processing component configured for processing the application.
Encoding the encoded video content included determining the chroma
format is 444; and not enabling strong intra smoothing. Encoding
the encoded video content included determining the chroma format is
422 or 444; and not enabling strong intra smoothing. A strong intra
smoothing flag is equal to 1 and a image component index is equal
to 0. The apparatus is selected from the group consisting of a
personal computer, a laptop computer, a computer workstation, a
server, a mainframe computer, a handheld computer, a personal
digital assistant, a cellular/mobile telephone, a smart appliance,
a gaming console, a digital camera, a digital camcorder, a camera
phone, a smart phone, a portable music player, a tablet computer, a
mobile device, a video player, a video disc writer/player, a
television, a home entertainment system and smart jewelry.
BRIEF DESCRIPTION OF THE DRAWINGS
[0008] FIG. 1 illustrates a flowchart of an SIS decision method
according to some embodiments.
[0009] FIG. 2 illustrates a block diagram of an exemplary computing
device configured to implement the SIS decision method according to
some embodiments.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT
[0010] In High Efficiency Video Coding (HEVC) version 1, strong
intra smoothing (SIS) is applied to luma (420 format) to reduce
(e.g. smooth) some visual artifacts in intra prediction. In Range
Extension (RExt) of HEVC, more chroma formats (monochrome, 4:2:2
and 4:4:4) and higher bitdepths (12, 14, 16) are supported.
Implementations to extend the SIS in RExt to address SIS for 444
chroma format. Restrictions on applying SIS are also described.
[0011] SIS introduces blocking artifacts and coding efficiency loss
although SIS improves contouring artifacts. If the disabling SIS
conditions are not satisfied, then the appropriate flag is set to a
value other than 0 such as 1.
Filtering Process of Neighboring Samples
[0012] When filterFlag is equal to 1, the following applies: [0013]
The variable bilntFlag is derived as follows: If all of the
following conditions are true, bilntFlag is set equal to 1: [0014]
strong_intra_smoothing_enabled_flag is equal to 1 [0015] nTbS is
equal to 32 [0016] cldx is equal to 0 //cldx is the image component
index [0017]
Abs(p[-1][-1]+p[nTbS*2-1][-1]-2*p[nTbS-1][-1]<(1<<(BitDepthY-5))
[0018]
Abs(p[-1][-1]+p[-1][nTbS*2-1]-2*p[-1][nTbS-1])<(1<<(BitDe-
pthY-5)) Otherwise, bilntFlag is set equal to 0.
[0019] Shown herein is a comparison between SIS "on" and "off" for
chroma, based on both AHGS and AHG8 test sequences. The reference
software used is HM13.0+RExt6.0.
[0020] More specifically, SIS "on" means that SIS is allowed for
chroma when the chroma format is 4:4:4 and SIS is applied because
intra_smoothing_disabled_flag is set to 0 (false) in default
settings. Anchor: SIS "off" is exactly the HM13.0+RExt6.0.
[0021] In the simulation, AHGS test sequences and conditions apply.
Because the simulation is run in a cluster, encoding and decoding
times are not accurate. The results are summarized in Table 1.
Results show that allowing SIS for chroma when chroma format is
4:4:4 will degrade the coding performance of current reference
software under common test conditions.
TABLE-US-00001 TABLE 1 Summary of results for AHG5 conditions All
Intra Main-tier All Intra High-tier All Intra Super High-tier Y U V
Y U V Y U V RGB:4:4:4 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
YCbCr 4:4:4 0.4% 0.4% 0.4% 0.2% 0.2% 0.2% 0.1% 0.1% 0.0% YCbCr
4:2:2 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% Enc Time[%] 99%
100% 100% Dec Time[%] 101% 101% 99% Random Access Main-tier Random
Access High-tier Y U V Y U V RGB:4:4:4 0.0% 0.0% 0.0% 0.0% 0.0%
0.0% YCbCr 4:4:4 0.2% 0.3% 0.1% 0.1% 0.1% 0.0% YCbCr 4:2:2 0.0%
0.0% 0.0% 0.0% 0.0% 0.0% Enc Time[%] 100% 101% Dec Time[%] 99% 101%
Low delay B Main-tier Low Delay B High-tier Y U V Y U V RGB:4:4:4
0.0% 0.0% 0.0% 0.0% 0.0% 0.0% YCbCr 4:4:4 0.1% 0.4% 0.1% 0.1% 0.1%
0.0% YCbCr 4:2:2 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% Enc Time[%] 103%
101% Dec Time[%] 100% 99%
[0022] In the simulation, AHG8 test sequences and conditions apply.
Because the simulation is run in a cluster, encoding and decoding
times are not accurate. The results are summarized in Table 2.
Results show that allowing SIS for chroma when the chroma format is
4:4:4 will degrade the coding performance of current reference
software under common test conditions.
TABLE-US-00002 TABLE 2 Summary of results for AHG8 conditions All
Intra Main-tier All Intra High-tier All Intra Super High-tier Y U V
Y U V Y U V Class F 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
Class B 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% RGB:4:4:4 SC
0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% RGB:4:4:4 0.0% 0.0%
0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% Ani YCbCr 4:4:4 0.0% 0.0% 0.0%
0.0% 0.0% 0.0% 0.0% 0.0% 0.0% SC YCbCr 4:4:4 0.2% 0.2% 0.2% 0.1%
0.1% 0.1% 0.1% 0.0% 0.0% Ani RangExt 0.2% 0.2% 0.1% 0.1% 0.1% 0.0%
0.0% 0.0% 0.0% RGB4:4:4 SC 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
0.0% (op) YCbCr 4:4:4 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
SC (optional) Enc Time[%] Dec Time[%] Random Access Main-tier
Random Access High-tier Y U V Y U V Class F 0.0% 0.0% 0.0% 0.0%
0.0% 0.0% Class B 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% RGB:4:4:4 SC 0.0%
0.1% 0.0% 0.1% 0.1% 0.1% RGB:4:4:4 Anim 0.0% 0.0% 0.0% 0.0% 0.0%
0.0% YCbCr 4:4:4 SC 0.1% 0.1% 0.1% 0.1% 0.1% 0.0% YCbCr 4:4:4 Anim
0.1% 0.0% 0.0% 0.1% 0.0% 0.0% Rang Ext 0.1% 0.0% 0.0% 0.0% 0.0%
0.0% RGB4:4:4 SC 0.1% 0.1% 0.2% 0.0% 0.0% 0.0% (optional) YCbCr
4:4:4 SC 0.0% 0.0% 0.0% 0.0% -0.1% 0.0% (optional) Enc Time[%] Dec
Time[%] Low delay B Main-tier Low Delay B High-tier Y U V Y U V
Class F 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% Class B 0.0% 0.0% 0.0% 0.0%
0.0% 0.0% RGB:4:4:4 SC 0.1% 0.0% 0.1% 0.1% 0.0% 0.1% RGB:4:4:4 Anim
-0.1% -0.1% 0.0% -0.1% 0.0% 0.0% YCbCr 4:4:4 SC -0.1% 0.0% -0.1%
-0.1% 0.0% -0.2% YCbCr 4:4:4 Anim 0.0% 0.0% 0.1% 0.0% 0.0% 0.1%
Rang Ext 0.0% 0.1% 0.0% 0.0% 0.0% 0.0% RGB4:4:4 SC -0.7% -0.8%
-0.7% -0.2% -0.3% -0.3% (optional) YCbCr 4:4:4 SC -0.6% -0.7% -0.5%
-0.7% -0.8% -0.7% (optional) Enc Time[%] Dec Time[%]
[0023] FIG. 1 illustrates a flowchart of an SIS decision method
according to some embodiments. In the step 100, video content is
acquired, transmitted and/or received. In the step 102, the video
content is encoded. The video content is encoded using SIS with the
conditions/restrictions described herein. For example, SIS is not
allowed (e.g., disabled) for chroma when the chroma format is 444.
In another example, SIS is not allowed for chroma when the chroma
format is 422 or 444. In the step 104, the video content is
displayed. In some embodiments, fewer or additional steps are
implemented. For example, the encoded video content is decoded
after it is encoded. In some embodiments, the order of the steps is
modified. For example, the video content is encoded and then
transmitted.
[0024] FIG. 2 illustrates a block diagram of an exemplary computing
device configured to implement the SIS decision method according to
some embodiments. The computing device 200 is able to be used to
acquire, store, compute, process, communicate and/or display
information such as images and videos. In general, a hardware
structure suitable for implementing the computing device 200
includes a network interface 202, a memory 204, a processor 206,
I/O device(s) 208, a bus 210 and a storage device 212. The choice
of processor is not critical as long as a suitable processor with
sufficient speed is chosen. The memory 204 is able to be any
conventional computer memory known in the art. The storage device
212 is able to include a hard drive, CDROM, CDRW, DVD, DVDRW, High
Definition disc/drive, ultra-HD drive, flash memory card or any
other storage device. The computing device 200 is able to include
one or more network interfaces 202. An example of a network
interface includes a network card connected to an Ethernet or other
type of LAN. The I/O device(s) 208 are able to include one or more
of the following: keyboard, mouse, monitor, screen, printer, modem,
touchscreen, button interface and other devices. SIS decision
method application(s) 230 used to implement the SIS decision method
are likely to be stored in the storage device 212 and memory 204
and processed as applications are typically processed. More or
fewer components shown in FIG. 2 are able to be included in the
computing device 200. In some embodiments, SIS decision method
hardware 220 is included. Although the computing device 200 in FIG.
2 includes applications 230 and hardware 220 for the SIS decision
method, the SIS decision method is able to be implemented on a
computing device in hardware, firmware, software or any combination
thereof. For example, in some embodiments, the SIS decision method
applications 230 are programmed in a memory and executed using a
processor. In another example, in some embodiments, the SIS
decision method hardware 220 is programmed hardware logic including
gates specifically designed to implement the SIS decision
method.
[0025] In some embodiments, the SIS decision method application(s)
230 include several applications and/or modules. In some
embodiments, modules include one or more sub-modules as well. In
some embodiments, fewer or additional modules are able to be
included.
[0026] Examples of suitable computing devices include a personal
computer, a laptop computer, a computer workstation, a server, a
mainframe computer, a handheld computer, a personal digital
assistant, a cellular/mobile telephone, a smart appliance, a gaming
console, a digital camera, a digital camcorder, a camera phone, a
smart phone, a portable music player, a tablet computer, a mobile
device, a video player, a video disc writer/player (e.g., DVD
writer/player, high definition disc writer/player, ultra high
definition disc writer/player), a television, a home entertainment
system, smart jewelry (e.g., smart watch) or any other suitable
computing device.
[0027] To utilize the SIS decision method described herein, a
device such as a digital camera/camcorder is used to acquire
image/video content or a device such as a television is used to
display video content. The SIS decision method is automatically
used during the encoding process of the acquired or displayed video
content such as when the video content is transmitted. The SIS
decision method is able to be implemented with user assistance or
automatically without user involvement.
[0028] In operation, the SIS decision method determines when SIS is
implemented to ensure proper and efficient encoding of content.
Some Embodiments of Strong Intra Smoothing for in RExt
[0029] 1. A method programmed in a non-transitory memory of a
device comprising: [0030] a. acquiring video content; and [0031] b.
encoding the video content without implementing strong intra
smoothing depending on the chroma format of the video content.
[0032] 2. The method of clause 1 further comprising determining the
chroma format is 444; and not enabling strong intra smoothing.
[0033] 3. The method of clause 1 further comprising determining the
chroma format is 422 or 444; and not enabling strong intra
smoothing. [0034] 4. The method of clause 1 wherein a strong intra
smoothing flag is equal to 1 and a image component index is equal
to 0. [0035] 5. The method of clause 1 wherein the device is
selected from the group consisting of a personal computer, a laptop
computer, a computer workstation, a server, a mainframe computer, a
handheld computer, a personal digital assistant, a cellular/mobile
telephone, a smart appliance, a gaming console, a digital camera, a
digital camcorder, a camera phone, a smart phone, a portable music
player, a tablet computer, a mobile device, a video player, a video
disc writer/player, a television, a home entertainment system and
smart jewelry. [0036] 6. A system programmed in a non-transitory
memory of a device comprising: [0037] a. a receiving device
configured for receiving encoded video content, wherein the encoded
video content was encoded without using strong intra smoothing
depending on the chroma format of the video content; and [0038] b.
a display device configured for displaying the encoded video
content. [0039] 7. The system of clause 6 wherein encoding the
encoded video content included determining the chroma format is
444; and not enabling strong intra smoothing. [0040] 8. The system
of clause 6 wherein encoding the encoded video content included
determining the chroma format is 422 or 444; and not enabling
strong intra smoothing. [0041] 9. The system of clause 6 wherein a
strong intra smoothing flag is equal to 1 and a image component
index is equal to 0. [0042] 10. The system of clause 6 wherein the
device is selected from the group consisting of a personal
computer, a laptop computer, a computer workstation, a server, a
mainframe computer, a handheld computer, a personal digital
assistant, a cellular/mobile telephone, a smart appliance, a gaming
console, a digital camera, a digital camcorder, a camera phone, a
smart phone, a portable music player, a tablet computer, a mobile
device, a video player, a video disc writer/player, a television, a
home entertainment system and smart jewelry. [0043] 11. An
apparatus comprising: [0044] a. a display device; [0045] b. a
non-transitory memory for storing an application, the application
for: [0046] i. receiving video content, wherein the video content
was encoded without using strong intra smoothing depending on the
chroma format of the video content; and [0047] ii. displaying the
video content on the display device; and [0048] c. a processing
component coupled to the memory, the processing component
configured for processing the application. [0049] 12. The apparatus
of clause 11 wherein encoding the encoded video content included
determining the chroma format is 444; and not enabling strong intra
smoothing. [0050] 13. The apparatus of clause 11 encoding the
encoded video content included determining the chroma format is 422
or 444; and not enabling strong intra smoothing. [0051] 14. The
apparatus of clause 11 wherein a strong intra smoothing flag is
equal to 1 and a image component index is equal to 0. [0052] 15.
The apparatus of clause 11 wherein the apparatus is selected from
the group consisting of a personal computer, a laptop computer, a
computer workstation, a server, a mainframe computer, a handheld
computer, a personal digital assistant, a cellular/mobile
telephone, a smart appliance, a gaming console, a digital camera, a
digital camcorder, a camera phone, a smart phone, a portable music
player, a tablet computer, a mobile device, a video player, a video
disc writer/player, a television, a home entertainment system and
smart jewelry.
[0053] The present invention has been described in terms of
specific embodiments incorporating details to facilitate the
understanding of principles of construction and operation of the
invention. Such reference herein to specific embodiments and
details thereof is not intended to limit the scope of the claims
appended hereto. It will be readily apparent to one skilled in the
art that other various modifications may be made in the embodiment
chosen for illustration without departing from the spirit and scope
of the invention as defined by the claims.
* * * * *