U.S. patent application number 16/442581 was filed with the patent office on 2020-01-09 for application acceleration.
The applicant listed for this patent is Mellanox Technologies, Ltd.. Invention is credited to Noam Bloch, Ariel Naftali Cohen, Dotan David Levi, Kobi Pines, Assaf Weissman, Erez Yaacov.
Application Number | 20200014945 16/442581 |
Document ID | / |
Family ID | 69102407 |
Filed Date | 2020-01-09 |
![](/patent/app/20200014945/US20200014945A1-20200109-D00000.png)
![](/patent/app/20200014945/US20200014945A1-20200109-D00001.png)
![](/patent/app/20200014945/US20200014945A1-20200109-D00002.png)
![](/patent/app/20200014945/US20200014945A1-20200109-D00003.png)
![](/patent/app/20200014945/US20200014945A1-20200109-D00004.png)
![](/patent/app/20200014945/US20200014945A1-20200109-D00005.png)
![](/patent/app/20200014945/US20200014945A1-20200109-D00006.png)
![](/patent/app/20200014945/US20200014945A1-20200109-D00007.png)
![](/patent/app/20200014945/US20200014945A1-20200109-D00008.png)
![](/patent/app/20200014945/US20200014945A1-20200109-D00009.png)
![](/patent/app/20200014945/US20200014945A1-20200109-D00010.png)
View All Diagrams
United States Patent
Application |
20200014945 |
Kind Code |
A1 |
Levi; Dotan David ; et
al. |
January 9, 2020 |
Application acceleration
Abstract
A system including an acceleration device including input
circuitry configured, for each of a first plurality of video frames
to be encoded, to receive an input including at least one raw video
frame and at least one reference frame, and to divide each of the
first plurality of video frames to be encoded into a second
plurality of blocks, and similarity computation circuitry
configured, for each one of the first plurality of video frame to
be encoded: for each block of the second plurality of blocks, to
produce a score of result blocks based on similarity of each block
in each frame to be encoded to every block of the reference frame,
an AC energy coefficient, and a displacement vector. Related
apparatus and methods are also provided.
Inventors: |
Levi; Dotan David; (Kiryat
Motzkin, IL) ; Weissman; Assaf; (Moreshet, IL)
; Pines; Kobi; (Yahad, IL) ; Bloch; Noam;
(Bat Shlomo, IL) ; Yaacov; Erez; (Kibbutz Sarid,
IL) ; Cohen; Ariel Naftali; (Zichron Yaakov,
IL) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Mellanox Technologies, Ltd. |
Yokneam |
|
IL |
|
|
Family ID: |
69102407 |
Appl. No.: |
16/442581 |
Filed: |
June 17, 2019 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
16291023 |
Mar 4, 2019 |
|
|
|
16442581 |
|
|
|
|
62695063 |
Jul 8, 2018 |
|
|
|
62726446 |
Sep 4, 2018 |
|
|
|
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
H04N 19/105 20141101;
H04N 19/176 20141101; H04N 19/42 20141101; H04N 19/119 20141101;
H04N 19/43 20141101; H04N 19/132 20141101; H04N 19/96 20141101 |
International
Class: |
H04N 19/42 20060101
H04N019/42; H04N 19/105 20060101 H04N019/105; H04N 19/132 20060101
H04N019/132; H04N 19/119 20060101 H04N019/119; H04N 19/176 20060101
H04N019/176 |
Claims
1. A system comprising: an acceleration device comprising: input
circuitry configured, for each of a first plurality of video frames
to be encoded, to receive an input comprising at least one raw
video frame and at least one reference frame, and to divide each of
said first plurality of video frames to be encoded into a second
plurality of blocks; and similarity computation circuitry
configured, for each one of said first plurality of video frame to
be encoded: for each said block of said second plurality of blocks,
to produce: a score of result blocks based on similarity of each
said block in each frame to be encoded to every block of the
reference frame; an AC energy coefficient; and a displacement
vector.
2. The system according to claim 1 and wherein said at least one
raw video frame and said at least one reference frame are
identical.
3. The system according to claim 1 and wherein the score of result
blocks comprises a ranked list.
4. The system according to claim 1 and wherein the result blocks
are one of: fixed size; and variable size.
5. The system according to claim 1 and also comprising weighting
circuitry configured to weight at least some of the second
plurality of blocks.
6. The system according to claim 5 and wherein, for a given block B
of the second plurality of blocks, the weighting circuitry is
configured to weight the block B to produce a weighted block B' in
accordance with the following formula: B'=A*B+C1 where A and C1 are
scalars.
7. The system according to claim 1 and also comprising upsampling
circuitry configured to upsample at least some of said second
plurality of blocks, and wherein the score of results blocks is
based on similarity of each said block to at least one upsampled
block.
8. The system according to claim 1 and also comprising a second
component, and wherein said second component receives an output
from the acceleration device and produces, based at least in part
on said output received from the acceleration device, a second
component output in accordance with a coding standard.
9. The system according to claim 8, and wherein said second
component comprises a plurality of second components, each of said
plurality of second components producing a second component output
in accordance with a coding standard, the coding standard for one
of said plurality of second components being different from a
coding standard of others of said plurality of second
components.
10. The system according to claim 8 and wherein the second
component includes an aggregation component configured to aggregate
a plurality of adjacent blocks having equal displacement vectors
into a larger block.
11. The system according to claim 10 and wherein the larger block
has a displacement vector equal to a displacement vector of each of
the plurality of blocks having equal displacement vectors, and the
larger block has a score equal to a sum of scores of the plurality
of blocks having equal displacement vectors.
12. A method comprising: providing an acceleration device
comprising: input circuitry configured, for each of a first
plurality of video frames to be encoded, to receive an input
comprising at least one raw video frame and at least one reference
frame, and to divide each of said first plurality of video frames
to be encoded into a second plurality of blocks; and similarity
computation circuitry configured, for each one of said first
plurality of video frame to be encoded: for each said block of said
second plurality of blocks, to produce: a score of result blocks
based on similarity of each said block in each frame to be encoded
to every block of the reference frame; an AC energy coefficient and
a displacement vector; and providing said input to said
acceleration device, and producing said score of result blocks and
said displacement vector based on said input.
13. The method according to claim 13 and wherein said at least one
raw video frame and said at least one reference frame are
identical.
14. The method according to claim 12 and wherein the score of
result blocks comprises a ranked list.
15. The method according to claim 12 and wherein the result blocks
are one of: fixed size; and variable size.
16. The method according to claim 12 and also comprising weighting
at least some of the second plurality of blocks.
17. The method according to claim 16 and wherein, for a given block
B of the second plurality of blocks, the weighting weights the
block B to produce a weighted block B' in accordance with the
following formula: B'=A*B+1 where A and C1 are scalars.
18. The method according to claim 12 and also comprising upsampling
at least some of said second plurality of blocks, and wherein the
score of results blocks is based on similarity of each said block
to at least one upsampled block.
19. The method according to claim 12 and also comprising providing
a second component receiving an output from the acceleration device
and producing, based at least in part on said output received from
the acceleration device, a second component output in accordance
with a coding standard.
20. The method according to claim 18 and wherein said second
component comprises a plurality of second components, each of said
plurality of second components producing a second component output
in accordance with a coding standard, the coding standard for one
of said plurality of second components being different from a
coding standard of others of said plurality of second
components.
21. The method according to claim 19 and wherein the second
component includes an aggregation component configured to aggregate
a plurality of adjacent blocks having equal motion vectors into a
larger block.
22. The method according to claim 21 and wherein the larger block
has a displacement vector equal to a displacement vector of each of
the plurality of blocks having equal rank, and the larger block has
a score equal to a sum of scores of the plurality of blocks having
equal displacement vectors.
23. The system according to claim 1 and wherein said reference
frame comprises one of: a reconstructed reference frame; and an
original reference frame.
24. The method according to claim 12 and wherein said reference
frame comprises one of: a reconstructed reference frame; and an
original reference frame.
Description
PRIORITY CLAIM
[0001] The present application is a Continuation-in-Part of U.S.
patent application Ser. No. 16/291,023, of Levi et al, filed 4 Mar.
2019; and claims priority to US Provisional Patent Application
62/695,063 of Levi et al, filed 8 Jul. 2018, and to US Provisional
Patent Application 62/726,446 of Levi et al, filed 4 Sep. 2018.
FIELD OF THE INVENTION
[0002] The present invention relates generally to the fields of
computer architecture and application acceleration.
BACKGROUND
[0003] Video encoding is described herein as one example of a
workload which creates heavy stresses (by way of non-limiting
example, in the cloud and/or in the data center; the same applies
in other equipment and/or locations that are part of the video
flow). With respect to the particular non-limiting example of
preparing video for an edge computing device (by way of
non-limiting example, for a smart phone, a TV set etc.), data
center workload includes millions of video streams which have to be
encoded and re-encoded during every minute, in order to stream the
streams to multiple devices (generally having different video
codecs) in a very efficient way. Persons skilled in the art will
appreciate that adaptive streaming additionally requires the video
to be encoded to multiple bitrates.
SUMMARY OF THE INVENTION
[0004] Persons skilled in the art will appreciate the following:
[0005] 1. Providing solutions to issues mentioned in the present
application (including in the Background thereof) in exemplary
embodiments of the present invention have applicability to various
use cases, and not only to data center and/or cloud use cases.
[0006] 2. Solutions provided in exemplary embodiments of the
present invention have applicability to (by way of non-limiting
example): video compression; motion estimation; video deblocking
filter; Current Picture Referencing (CPR) and video transform.
Non-limiting examples of relevant systems to which exemplary
embodiments of the present invention may be applicable include, by
way of non-limiting example: HEVC/H.265 (high efficiency video
coding); AVC/H.264 (advanced video coding); VP9; AV-1 (AOMedia
Video 1); and VVC (versatile video coding). While, generally, uses
of exemplary embodiments of the present invention are described in
the context of motion estimation, such descriptions are not meant
to be limiting, and persons skilled in the art will appreciate, in
light of the description herein, how to provide solutions in at
least the other cases mentioned above in the present paragraph.
[0007] The load of a single encoding task itself is generally too
big for a single CPU. Reference is now made to FIG. 9, which is a
simplified graphical illustration depicting a relationship between
CPU power and various encoding tasks; to FIG. 10, which is a
simplified pictorial illustration depicting an approximate timeline
view of various past, present, and future encoding standards; and
to FIG. 11, which is a simplified tabular illustration depicting
complexity of various encoding standards. FIG. 9 shows that, for
quite some time, a single CPU has not been able to keep up with a
single encoding task; this problem, it is believed, will become
more severe in the future, as future standards are adopted and
become more widely used (see FIG. 10) and as the complexity of
those future standards is expected to be higher, as shown in FIG.
11.
[0008] In addition, the amount of information/data in every stream
is increasing, as more and more information/data is produced and
streamed into the edge devices at higher resolution. Reference is
now additionally made to FIG. 12, which is a simplified partly
pictorial, partly graphical illustration depicting relationships
between resolution and complexity. FIG. 12 show a very conservative
forecast of resolution mixture along the years, and also shows the
nominal complexity involved with the fact that there are more
pixels in every stream.
[0009] Reference is now additionally made to FIG. 13, which is a
simplified graphical illustration depicting relationships between
video complexity, CPU capability, and computational effort. FIG. 13
shows that the real video complexity (complexity per pixel X number
of pixels) is suffering from a growing gap between CPU capability
and the computational effort needed.
[0010] The consideration of the problem presented up to this point
does not include the major increase, which is expected to continue,
in the number of video streams needs to be simultaneously processed
in the data center.
[0011] Having considered the above, it is fair to ask why the art,
as known to the inventors of the present invention, does not yet
include any acceleration device offering video encoding
acceleration for the data center. Without limiting the generality
of the present invention, the term "device" may be used in the
present description to describe implementations of exemplary
embodiments of the present invention, as well as (in the preceding
sentence) apparent lacks in the known art. It is appreciated that,
in exemplary embodiments of the present invention, by way of
non-limiting example, implementation may take place in: an ASIC
[Application Specific Integrated Circuit]; an ASSP [Application
Specific Standard Part]; an SOC [System on a Chio], an FPGA [Field
Programable Gate Array]; in a GPU [graphics processing unit]; in
firmware; or in any appropriate combination of the preceding.
[0012] The inventors of the present invention believe that the
reasons that the art, as known to the inventors of the present
invention, does not yet include an appropriate video acceleration
as described above include:
[0013] 1. Technological Reason [0014] The huge diversity in the
video standards (as partially imposed by the situation depicted in
FIG. 10) which need to be supported, as well as the rich
feature-set needed in order to fully support any single encoder,
might make it appear, prior to the invention of exemplary
embodiments of the present invention, to be impossible to create a
generic acceleration device for accelerating video encoding in the
data center.
[0015] 2. Business Reason
[0016] In the last decade, a huge amount of technology and
knowledge was gained in the industry by encoder vendors (who
generally, but not necessarily, provide their encoders in software
[SW]; for purposes of ease of description, the term "SW" is used
throughout the present specification and claims to describe such
encoders, which might in fact be provided, generally by encoder
vendors in software, firmware, hardware, or any appropriate
combination thereof) that tuned the video encoding to "their best
sweet spot", representing what each encoder vendor believed to be
their competitive advantage. It is important in this context to
understand that video compression, by its nature as a lossy
compression, differs in various implementations in many aspects,
such as: performance; CPU load; latency; (which are also well known
in other, non-video workloads); but also in quality. For the aspect
of quality, there is no common and accepted objective metric that
quantifies and grades the quality; rather, multiple metrics (such
as, for example, PSNR, SSIM, and VMAF) are known, with no universal
agreement on which metric is appropriate. Thus, the accumulated
knowledge and reputation of each company somehow blocks any
potential acceleration device vendor that wishes to penetrate the
market of acceleration devices. In other words, if such a vendor
were to create a device such as, by way of on particular
non-limiting example, an ASIC implementing a video codec, the
vendor would find himself competing against the entire ecosystem.
Even within a given vendor, different quality measures might be
used for different use cases, thus leading to an incentive to
produce multiple such ASIC implementations.
[0017] The present invention, in exemplary embodiments thereof,
seeks to provide an improved video encoding, video compression,
motion estimation, Current Picture Referencing (CPR), and computer
architecture system in which part of the work (such as, by way of
non-limiting example, any one or more of the following: motion
estimation; Current Picture Referencing (CPR) transform;
deblocking; loop filter; and context-adaptive binary arithmetic
coding (CABAC) engine) is offloaded in such away that (in the
specific non-limiting example of an encoder): [0018] 1. Work will
be dramatically offloaded from the encoder; [0019] 2. The exemplary
embodiments of the present invention will be agnostic to the
specific encoder being implemented; and [0020] 3. A given encoder
vendor (by way of non-limiting example, a software encoder vendor)
will be enabled to run their own "secret sauce" and use one or more
exemplary embodiments of the present invention to provide
acceleration as a primitive operation. The appropriate
quality/performance can be chosen by a given encoder vendor for the
appropriate use case/s.
[0021] There is thus provided in accordance with an exemplary
embodiment of the present invention a system including an
acceleration device including input circuitry configured, for each
of a first plurality of video frames to be encoded, to receive an
input including at least one raw video frame and at least one
reference frame, and to divide each of the first plurality of video
frames to be encoded into a second plurality of blocks, and
similarity computation circuitry configured, for each one of the
first plurality of video frame to be encoded: for each block of the
second plurality of blocks, to produce a score of result blocks
based on similarity of each block in each frame to be encoded to
every block of the reference frame, an AC energy coefficient, and a
displacement vector.
[0022] Further in accordance with an exemplary embodiment of the
present invention the at least one raw video frame and the at least
one reference frame are identical.
[0023] Still further in accordance with an exemplary embodiment of
the present invention the score of result blocks includes a ranked
list.
[0024] Additionally in accordance with an exemplary embodiment of
the present invention the result blocks are one of fixed size, and
variable size.
[0025] Moreover in accordance with an exemplary embodiment of the
present invention the system also includes weighting circuitry
configured to weight at least some of the second plurality of
blocks.
[0026] Further in accordance with an exemplary embodiment of the
present invention, for a given block B of the second plurality of
blocks, the weighting circuitry is configured to weight the block B
to produce a weighted block B' in accordance with the following
formula: B'=A*B+C1, where A and C1 are scalars.
[0027] Still further in accordance with an exemplary embodiment of
the present invention the system also includes upsampling circuitry
configured to upsample at least some of the second plurality of
blocks, and the score of results blocks is based on similarity of
each block to at least one upsampled block.
[0028] Additionally in accordance with an exemplary embodiment of
the present invention the system also includes a second component,
and the second component receives an output from the acceleration
device and produces, based at least in part on the output received
from the acceleration device, a second component output in
accordance with a coding standard.
[0029] Moreover in accordance with an exemplary embodiment of the
present invention the second component includes a plurality of
second components, each of the plurality of second components
producing a second component output in accordance with a coding
standard, the coding standard for one of the plurality of second
components being different from a coding standard of others of the
plurality of second components.
[0030] Further in accordance with an exemplary embodiment of the
present invention the second component includes an aggregation
component configured to aggregate a plurality of adjacent blocks
having equal displacement vectors into a larger block.
[0031] Still further in accordance with an exemplary embodiment of
the present invention the larger block has a displacement vector
equal to a displacement vector of each of the plurality of blocks
having equal displacement vectors, and the larger block has a score
equal to a sum of scores of the plurality of blocks having equal
displacement vectors.
[0032] There is also provided in accordance with another exemplary
embodiment of the present invention a method including providing an
acceleration device including input circuitry configured, for each
of a first plurality of video frames to be encoded, to receive an
input including at least one raw video frame and at least one
reference frame, and to divide each of the first plurality of video
frames to be encoded into a second plurality of blocks, and
similarity computation circuitry configured, for each one of the
first plurality of video frame to be encoded: for each block of the
second plurality of blocks, to produce a score of result blocks
based on similarity of each block in each frame to be encoded to
every block of the reference frame, an AC energy coefficient and a
displacement vector, and providing the input to the acceleration
device, and producing the score of result blocks and the
displacement vector based on the input.
[0033] Further in accordance with an exemplary embodiment of the
present invention the at least one raw video frame and the at least
one reference frame are identical.
[0034] Still further in accordance with an exemplary embodiment of
the present invention the score of result blocks includes a ranked
list.
[0035] Additionally in accordance with an exemplary embodiment of
the present invention the result blocks are one of fixed size, and
variable size.
[0036] Moreover in accordance with an exemplary embodiment of the
present invention the method also includes weighting at least some
of the second plurality of blocks.
[0037] Further in accordance with an exemplary embodiment of the
present invention, for a given block B of the second plurality of
blocks, the weighting weights the block B to produce a weighted
block B' in accordance with the following formula: B' A*B+C1, where
A and C1 are scalars.
[0038] Still further in accordance with an exemplary embodiment of
the present invention the method also includes upsampling at least
some of the second plurality of blocks, and the score of results
blocks is based on similarity of each block to at least one
upsampled block.
[0039] Additionally in accordance with an exemplary embodiment of
the present invention the method also includes providing a second
component receiving an output from the acceleration device and
producing, based at least in part on the output received from the
acceleration device, a second component output in accordance with a
coding standard.
[0040] Further in accordance with an exemplary embodiment of the
present invention the second component includes a plurality of
second components, each of the plurality of second components
producing a second component output in accordance with a coding
standard, the coding standard for one of the plurality of second
components being different from a coding standard of others of the
plurality of second components.
[0041] Still further in accordance with an exemplary embodiment of
the present invention the second component includes an aggregation
component configured to aggregate a plurality of adjacent blocks
having equal motion vectors into a larger block.
[0042] Additionally in accordance with an exemplary embodiment of
the present invention the larger block has a displacement vector
equal to a displacement vector of each of the plurality of blocks
having equal rank, and the larger block has a score equal to a sum
of scores of the plurality of blocks having equal displacement
vectors.
[0043] Moreover in accordance with exemplary embodiments of the
present invention the reference frame includes one of a
reconstructed reference frame, and an original reference frame.
BRIEF DESCRIPTION OF THE DRAWINGS
[0044] The present invention will be understood and appreciated
more fully from the following detailed description, taken in
conjunction with the drawings in which:
[0045] FIG. 1A is a simplified tabular illustration depicting an
inter block in a video bit stream;
[0046] FIG. 1B is a simplified block diagram illustration of an
application acceleration system, constructed and operative in
accordance with an exemplary embodiment of the present
invention;
[0047] FIG. 2 is a simplified pictorial illustration depicting
H.264/AVC block partitioning;
[0048] FIG. 3 is a simplified pictorial illustration depicting HEVC
(H.265) block partitioning;
[0049] FIG. 4 is a simplified tabular illustration depicting an
exemplary order of block encoding, intended in exemplary cases to
be performed by an application acceleration system such as the
system of FIG. 1B;
[0050] FIG. 5 is a simplified tabular illustration depicting an
exemplary result of operations performed by an acceleration device
in accordance with an exemplary embodiment of the present
invention;
[0051] FIG. 6 is another simplified tabular illustration depicting
an exemplary result of operations performed by an acceleration
device in accordance with an exemplary embodiment of the present
invention;
[0052] FIG. 7 is a simplified pictorial illustration depicting a
"local minimum" problem that may be encountered by an exemplary
system in accordance with an exemplary embodiment of the present
invention;
[0053] FIG. 8 is a simplified pictorial illustration depicting an
operation on a low-resolution image, which operation may be carried
out by an exemplary system in accordance with an exemplary
embodiment of the present invention;
[0054] FIG. 9 is a simplified graphical illustration depicting a
relationship between CPU power and various encoding tasks;
[0055] FIG. 10 is a simplified pictorial illustration depicting an
approximate timeline view of various past, present, and future
encoding standards;
[0056] FIG. 11 is a simplified tabular illustration depicting
complexity of various encoding standards;
[0057] FIG. 12 is a simplified partly pictorial, partly graphical
illustration depicting relationships between resolution and
complexity;
[0058] FIG. 13 is a simplified graphical illustration depicting
relationships between video complexity, CPU capability, and
computational effort;
[0059] FIG. 14 is a simplified pictorial illustration depicting
various portions of video codec standards;
[0060] FIG. 15 is a simplified pictorial illustration depicting an
exemplary search area for motion vector prediction;
[0061] FIG. 16 is a simplified pictorial illustration depicting a
non-limiting example of pixel sub blocks;
[0062] FIG. 17A is a simplified partly pictorial, partly
block-diagram illustration depicting an exemplary system and
process, including motion estimation, for producing a
bitstream;
[0063] FIG. 17B is a simplified partly pictorial, partly
block-diagram illustration depicting an exemplary system and
process, including motion estimation, for producing a bitstream,
including use of an exemplary embodiment of the acceleration system
of FIG. 1B;
[0064] FIG. 18 is a simplified block diagram illustration depicting
a particular exemplary case of using the acceleration system of
FIG. 1B;
[0065] FIG. 19 is a simplified block diagram illustration depicting
another particular exemplary case of using the acceleration system
of FIG. 1B;
[0066] FIG. 20 is a simplified block diagram illustration depicting
still another particular exemplary case of using the acceleration
system of FIG. 1B;
[0067] FIG. 21 is a simplified block diagram illustration depicting
yet another particular exemplary case of using the acceleration
system of FIG. 1B;
[0068] FIG. 22 is a simplified partly pictorial, partly block
diagram illustration of an exemplary embodiment of a portion of the
application acceleration system of FIG. 1B;
[0069] FIG. 23 is a simplified tabular illustration depicting an
exemplary result of operations performed by an acceleration device
in accordance with another exemplary embodiment of the present
invention;
[0070] FIG. 24 is a simplified tabular illustration depicting an
exemplary result of operations performed by an acceleration device
in accordance with still another exemplary embodiment of the
present invention;
[0071] FIG. 25 is a simplified tabular illustration depicting an
exemplary result of operations performed by an acceleration device
in accordance with yet another exemplary embodiment of the present
invention;
[0072] FIGS. 26A and 26B are simplified tabular illustrations
useful in understanding still another exemplary embodiment of the
present invention;
[0073] FIG. 27A is a simplified partly pictorial, partly
block-diagram illustration depicting an exemplary system and
process, including motion estimation, for producing a bitstream;
and
[0074] FIG. 27B is a simplified partly pictorial, partly
block-diagram illustration depicting an exemplary system and
process, including motion estimation, for producing a bitstream,
including use of an exemplary embodiment of the acceleration system
of FIG. 1B.
DETAILED DESCRIPTION OF EMBODIMENTS
[0075] The following general discussion may be helpful in
understanding certain exemplary embodiments of the present
invention which are described herein.
[0076] Among hundreds of tools in a video compression standard, the
motion estimation portion/tool, (which is described in the standard
by the motion compensation procedure), is generally considered to
be the most demanding one when it comes to computational effort.
The preceding also applies a Current Picture Referencing (CPR)
tool.
[0077] Theoretically, motion estimation is not part of the video
standard, as illustrated in FIG. 14, which is a simplified
pictorial illustration depicting various portions of video codec
standards. However, a motion estimation procedure is involved in
comparing a block (termed herein a "reference block") against many
blocks in the reference frames, finding the block with greatest (or
in some cases close to greatest) similarity, and then performing a
motion compensation which comprises, as is well known, writing the
residual (as is known in the art) as well as the motion vectors
(also termed herein "displacement vectors") in the bitstream. In
some cases, a "factor" may be used to find a block which,
multiplied by some constant (factor), is close to a given block. In
other cases, two blocks from two different frames may be combined
via weighted summing, and then a block close to the average may be
found.
[0078] Typically, in many codec systems, the motion estimation and
the motion compensation are built as one unit, which means that the
motion estimation score function (which is the "similarity" to the
reference block), is the same as the compensation part, thus
allowing the codec to use the score of the best matching block as a
residual in the bitstream without further processing.
[0079] Reference is now made to FIG. 1A, which is a simplified
tabular illustration depicting an inter block, generally designated
101, in a video bit stream. A high level view of a inter block in a
video bitstream is shown in FIG. 1A.
[0080] A block header 103 specifies the block type (which can in
general be, by way of non-limiting example, inter or intra, the
particular non-limiting example shown in FIG. 1A being an inter
block with a single motion vector (MV) structure).
[0081] A motion vector 105 represents the distance between the top
left corner of the inter block to the top left corner of the
reference block, while residual bits 107 (which are, in certain
exemplary embodiments, represent the difference between the
reference block and the target block (a given block). Each one of
the sections shown in FIG. 1A may be compressed differently, as is
known in the art.
[0082] The portion/size of each section in FIG. 1A is dynamic, in
that each one of the three sections can be of variable size. A
goal, in exemplary embodiments of the present invention, is to
reduce the total number of bits in an entire bitstream, not just to
reduce the total number of bits in a given inter block.
[0083] Generally speaking, the part of the process of generating an
inter block which involves heavy computation is to read (in many
case, by way of non-limiting example) tens of blocks for every
reference block and to calculate the respective differences.
Performing this operation itself may consume approximately 50 times
more memory bandwidth than accessing the raw video itself. It is
also appreciated that the compute effort of the process of
generating an inter block may be very large, approximately, as well
as the compute effort which is estimated to be approximately 50
O(number of pixels).
[0084] The motion estimation part is generally responsible not only
for finding a block with the minimal residual relative to each
reference, but also for finding an optimal partitioning; by way of
non-limiting example, a 32.times.32 block with 5 bits residual will
consume many fewer bits in the bitstream than 4 8.times.8 blocks
with 0 bits residual in each of them. In this particular non
limiting example, the 32.times.32 block partitioning would be
considered optimal. Persons skilled in the art will appreciate that
the method of selecting the best matched block is dependent on the
details of the particular codec standard, including, by way of
non-limiting example, because different standards treat motion
vectors having large magnitude differently. Non-limiting examples
of "differently" in the preceding sentence include: different
partitioning; different sub-pixel interpolation; and different
compensation options which may be available. It is appreciated
that, in exemplary embodiments of the present invention, sub-pixels
are produced by upsampling, as is known in the art.
[0085] In addition to what has been stated above, the different
video standards differ from one another, with respect to motion
compensation, in at least the following parameters: [0086] 1.
Motion vector resolution:
[0087] Older standards allow only full pixel comparison (against
real blocks), while newer standards allow fractional sampling
interpolations. The different standards also differ in filters used
for the interpolation/s. Other features which differ between
different standards include the particulars of rounding and
clipping, as are well known in the art. [0088] 2. Motion
compensation residual function: calculates residual data in the
bitstream. [0089] 3. Block size being compensated. By way of
particular non-limiting example: In H.264/AVC block partitioning as
shown in FIG. 2 was allowed by the standard. [0090] By contrast, in
H.265/HEVC, the partitioning shown in FIG. 3 was introduced. [0091]
4. Weighted prediction. Weighted prediction was introduced in H.264
standard, in the Main and Extended profiles. The weighted
prediction tool allows scaling of a reference frame with a
multiplicative weighting factor, and also add an additive offset
(constant) to each pixel. The weighted prediction tool is generally
considered to be highly powerful in scenes of fade in/fade out.
When fades are uniformly applied across the entire picture, a
single weighting factor and offset are sufficient to efficiently
encode all macroblocks in a picture that are predicted from the
same reference picture. When multiple reference pictures are used,
the best weighting factor and offsets generally differ during a
fade for the different reference pictures, as brightness levels are
more different for more temporally distant pictures.
[0092] By way of particular non-limiting example: for single
directional prediction the following equation represents the motion
compensation with weighted prediction: [0093]
SampleP=Clip1(((SamplePW0+2LWD-1)>>LWD)+O0) where Clip1( ) is
an operator that clips to the range [0, 255], W0 and O0 are the
reference picture weighting factor and offset respectively, and LWD
is the log weight denominator rounding factor, SampleP is the list
0 initial predictor, and SampleP is the weighted predictor.
[0094] Persons skilled in the art will appreciate that the motion
estimation is generally performed against the weighted reference
frame (resulting from applying the above SampleP formula). Persons
skilled in the art will further appreciate that it is a reasonable
assumption that the compensation function will be different in
future codecs.
[0095] It appears reasonable to assume that future codec standards
will continue to differ in the points mentioned immediately
above.
[0096] Motion estimation procedures include "secret sauce" as
described above. In order to allow agnostic preparation which will
later allow motion estimation with the desired "secret sauce",
exemplary embodiments of the present invention will make many more
calculations than are known in systems which do not use an
acceleration device in accordance with exemplary embodiments of the
present invention, leaving open later decisions to be made by the
motion compensation/estimation software.
[0097] The following is a description of an exemplary embodiment of
a method useable in order to create a generic and agnostic
acceleration device, offloading motion estimation and Current
Picture Referencing (CPR) for a particular codec. By way of
non-limiting example, implementation of an appropriate acceleration
device may take place in: an ASIC [Application Specific Integrated
Circuit]; an ASSP [Application Specific Standard Part]; an SOC
[System on a Chip], an FPGA [Field Programable Gate Array]; in
firmware; in a GPU [graphics processing unit]; or in any
appropriate combination of the preceding. Implementations described
in the preceding sentence may also be referred to herein as
"circuitry", without limiting the generality of the foregoing. The
description will be followed by a detailed explanation of how the
acceleration device overcomes issues related to each of Motion
vector resolution; Motion compensation residual function; Block
size being compensated; and Weighted prediction, as mentioned
above. As mentioned above, motion estimation is described as one
particular non-limiting example.
[0098] Reference is now made to FIG. 1B, which is a simplified
block diagram illustration of an application acceleration system,
constructed and operative in accordance with an exemplary
embodiment of the present invention.
[0099] In the particular non-limiting example of FIG. 1B, the
application acceleration system, generally designated 110,
comprises a video acceleration system.
[0100] The video acceleration system 110 comprises a video
acceleration device 120; exemplary embodiments of the construction
and operation of the video acceleration device 120 are further
described herein. As described in further detail below, the video
acceleration device 120 is, in exemplary embodiments, configured to
produce a result map 140.
[0101] The result map 140 is provided as input to a further
component (often termed herein "SW", as described above); the
further component, in exemplary embodiments, comprises a motion
estimation/block partitioning/rate-distortion control unit 130. The
control until 130 may be, as implied by its full name, responsible
for: [0102] motion estimation; [0103] block partitioning; and
[0104] rate distortion (determining tradeoffs between bit rate and
distortion, for example)
[0105] In certain exemplary embodiments of the present invention,
it is appreciated that optimal performance may take place when:
high memory bandwidth is available; multiple queues are available
for managing memory access; and virtual memory address translation
is available at high performance and to multiple queues. One
non-limiting example of a commercially available system which
fulfills the previously-mentioned criteria for optimal performance
is the ConnectX-5, commercially available from Mellanox
Technologies Ltd. It is appreciated that the example of ConnextX-5
is provided as on particular example, and is not meant to be
limiting; other systems may alternatively be used.
[0106] The operation of the system of FIG. 1B, and particularly of
the video acceleration device 120 of FIG. 1B, is now described in
more detail.
Input to the System:
[0107] In certain exemplary embodiments of the present invention,
for each video frame being encoded (termed herein "target frame"),
the video acceleration device 120 reads previously decoded frames
(also known as reconstructed raw video), against which the target
frame is being compensated; by way of particular non-limiting
example, two previously decoded frames may be read. It is
appreciated that, in an exemplary embodiment using CPR, the video
acceleration device 120 may read/use the target frame twice, once
as a target frame and once as a reference frame. In addition and
optionally, a map of motion vector prediction may be provided; the
map of motion vector prediction shows a center of a search area for
each block. The particular example in the previous sentence is
non-limiting, it being appreciated that a center of search are may
be determined, including independently, for any given block.
Reference is now additionally made to FIG. 15, which is a
simplified pictorial illustration depicting an exemplary search
area for motion vector prediction. In FIG. 15, an exemplary map
(generally designated 1500) is shown. In the particular
non-limiting example of FIG. 15, the center of the search area for
a given block is designated 1510. In the particular non-limiting
example of FIG. 15, the search area comprises a plurality of search
blocks 1520, only some of which have been labeled with reference
number 1520 for ease of depiction. It is appreciated that,
optionally, other parameters may be configured and may help the
system "tune" to a specific encoder, specifying desired quality and
speed; such parameters may include, by way of non-limiting example:
accuracy of sub pixel interpolation; search area size; aggregation
level (maximum block size being aggregated); partitioning
information (whether partitioning may be, for example, only into
squares or also into rectangles); and block matching function to be
used (such as, by way of non-limiting example: SAD (Sum of Absolute
Difference); and SSE (Sum of Square Error)).
Output of the System:
[0108] Reference is now additionally made to FIG. 4, which is a
simplified tabular illustration depicting an exemplary order of
block encoding, intended in exemplary cases to be performed by an
application acceleration system such as the application
acceleration system 110 of FIG. 1B. In certain exemplary
embodiments of the present invention, the video acceleration device
120 outputs a list of the top ranked blocks, which have maximal
similarity to each and every one of the blocks being encoded; for
example, the current encoded frame is divided into small blocks
(such as, by way of non-limiting example, 8.times.8 blocks). This
is illustrated, including an exemplary non-limiting example of an
order of block encoding, in FIG. 4.
[0109] Reference is now additionally made to FIG. 5, which is a
simplified tabular illustration depicting an exemplary result of
operations performed by an acceleration device, such as the video
acceleration device 120 of FIG. 1B, in accordance with an exemplary
embodiment of the present invention. A result map 500 (which may
also be termed an "acceleration vector"), shown in FIG. 5, is an
example of the output (also termed herein a "result score board")
produced in exemplary embodiments of the present invention; the
result map in FIG. 5 is a particular non-limiting example of the
result map 140 of FIG. 1B.
[0110] The result map of FIG. 5 demonstrate the flexibility which
is provided to the SW when using embodiments of the acceleration
device described herein, in that (by way of non-limiting example)
block partitioning decisions can be carried out based on the data
comprised in the result map 500 of FIG. 5. Thus, by way of one
non-limiting example, if the SW is "naive", and uses the first
score for every block, it will produce four 8.times.8 blocks, with
a total residual of (5+0+0+0)=5 (see entries marked in bold
underscore in the result map 500).
[0111] Alternatively, in another non-limiting example, the SW may
choose to create one single 32.times.32 block (since there are 4
scores that has the same MV value, so that the corresponding blocks
can be combined) with a residual of: (6+3+1+1)=11 (see entries
marked in bold italics in the result map 500).
[0112] Similarly, the SW can choose to re-partition to bigger
blocks, for example when (by way of non-limiting example) based the
results of blocks: 4,5,6,7,16,17,18,19,20,21,22,23 which have
arrived from the video acceleration device 120 of FIG. 1, in order
to partition to 64.times.64 sizes.
[0113] Reference is now made to FIG. 17A, which is a simplified
partly pictorial, partly block-diagram illustration depicting an
exemplary system and process, including motion estimation, for
producing a bitstream, as is known in the field of video. In the
system of FIG. 17A, a plurality of target frames 1705 are input.
For a given block 1710 (of one or more target frames 1705, it being
appreciated that in general the process is carried out on a
plurality of target frames), a comparison operation is carried out
at comparison element 1715. The comparison operation is carried out
relative to a reference block 1720, produced/chosen by a motion
estimation unit 1725, based on a decoded picture input received
from a decoded picture buffer 1730.
[0114] The result of the comparison element 1715 is a residual
block 1735. The residual block 1735 undergoes a transform operation
at a transform unit 1740; quantization at a quantizing unit 1750;
and entropy encoding in an entropy unit 1755. The output the system
of FIG. 17A is a bitstream.
[0115] Meanwhile, quantized data from the quantizing unit 1750 is
dequantized an inverse quantizing unit 1760, and undergoes an
inverse transform operation at an inverse transform unit 1765, thus
producing a decoded residual block 1770. The decoded residual block
1770 is added to the reference block 1720 at element 1772, with a
result thereof being processed by loop filters 1775, and then sent
to the decoded picture buffer 1730, for further processing as
previously described.
[0116] Reference is now made to FIG. 17B, which is a simplified
partly pictorial, partly block-diagram illustration depicting an
exemplary system and process, including motion estimation, for
producing a bitstream, including use of an exemplary embodiment of
the acceleration system of FIG. 1B. In the exemplary system and
process of FIG. 17B, the motion estimation unit 1725 (FIG. 17A) has
been replaced with a block matching unit 1780, instantiated in a
video acceleration device such as the video acceleration device 120
of FIG. 1B. Other elements of FIG. 17B, having like numbers to
elements in FIG. 17A, may be similar thereto.
[0117] The block matching unit 1780 produces a result map 1782,
which may be similar to the result map 140 of FIG. 1B, and which is
sent to an RDO (rate distortion optimization unit) 1785, whose
function is to choose (in accordance with an applicable metric) a
best reference block and partition, with a goal of having bit
stream length at the end of the process match a target bitrate,
with maximal available quality. In accordance with the
above-mentioned discussion, the RDO 1785, along with the elements
which are common between FIGS. 17A and 17B, are SW elements as
explained above.
[0118] In order to elucidate further the above discussion of
certain goals of certain exemplary embodiments of the present
invention, the following describes a sense in which exemplary
embodiments of an acceleration device described herein is "codec
agnostic" or a "generic device"
Agnostic
[0119] (1)
[0120] In exemplary embodiments of the present invention, the
problem of motion vector resolution described above is overcome by
allowing device configuration, in the case of any particular coding
standard, to (by way of one non-limiting example) limit the search
to full pixels and half pixels, as appropriate for that standard.
The difference in the kernel coefficient of sub pixels described
above is overcome by allowing SW to configure the kernel
coefficient. This coefficient does not generally change "on the
fly"; even though the coefficient/coefficients are codec dependent,
they are fixed for the entire video stream. It is also appreciated
that video encodings may differ in motion vector resolution which
is allowed in describing the bit stream; by way of non-limiting
example, some may allow 1/4 pixel resolution, while some may allow
higher resolution (less than 1/4 pixel, such as, for example, 1/8
pixel). Encodings may also differ in a way in which sub pixel
samples are defined by interpolation of neighboring pixels using
known and fixed coefficients; the coefficients being also termed
herein "kernel". For each block, the acceleration device may
produce (by way of particular non-limiting example, in the case of
1/4 size pixels) sixteen sub-blocks, each of which represent
different fractional motion vectors. Reference is now made to FIG.
16, which is a simplified pictorial illustration depicting a
non-limiting example of pixel sub blocks. In FIG. 16, each shaded
rectangle (such as, by way of non-limiting example, rectangle
A.sub.1,0) represents the upper-left-hand corner of a pixel, while
the nearby non-shaded rectangles (in the example of shaded
rectangle A.sub.1,0, an additional fifteen rectangles (a.sub.1,0;
b.sub.1,0; c.sub.1,0; d.sub.1,0; e.sub.1,0; f.sub.1,0; g.sub.1,0;
h.sub.1,0; i.sub.1,0; j.sub.1,0; k.sub.10; n.sub.1,0; p.sub.1,0;
p.sub.1,0; and r.sub.1,0), for a total of sixteen rectangles
(sub-blocks). In general, in preferred embodiments of the present
invention, the kernel may be configured as an input. [0121] (2)
[0122] Different video encodings also differ in compensation,
meaning that the representation of the residual block 1735 of FIG.
17A and FIG. 17B, is done differently in different video encodings.
In exemplary embodiments of the present invention, the estimation
(which block is probably best) is separated from the compensation
(how to express the difference between a given block in a reference
frame and in a target frame), which in previous systems were
computed together. Thus, in distinction to previous systems, in
exemplary embodiments of the present invention the SW will
calculate the residual (what remains after compensation), but not
the estimation. In certain embodiments it may be that different
score functions are implemented in the acceleration device (such as
the video acceleration device 120 of FIG. 1B) thus allowing SW to
choose one of them to be used. The following is a non-limiting list
of exemplary score functions, which functions are known in the art:
SAD (Sum of Absolute Difference); and SSE (Sum of Square Error).
[0123] (3)
[0124] Some encoding standards allow compensating blocks against a
weighted image, meaning that a reference frame (previously decoded
frame) is multiplied by a factor, (rational number). Alternatively,
a sum of 2 reference frames may each be multiplied by a different
weighting factor. The acceleration device, in preferred embodiments
of the present invention, may either allow configuring a weighting
factor for each reference frame, or may receive as input an already
weighted frame.
[0125] Reference is now additionally made to FIG. 6, which is
another simplified tabular illustration depicting an exemplary
result of operations performed by an acceleration device in
accordance with an exemplary embodiment of the present invention.
By way of non-limiting example of a score board, consider a result
score board 600 as shown in FIG. 6; in FIG. 6, bold underscore and
bold italics are used similarly to their usage in FIG. 5.
[0126] "Smart SW" is able to see that two reference frames in the
result score board 600 have the same MV (1/4, -1/4), so the smart
SW can itself calculate the compensation of weighted prediction
between frame 0 and frame 1, and might thus get a better result
than the score board indicates. In the particular example shown,
since blocks 0, 1, 2, 3, all have a given score, those blocks can
be re-partitioned into one 16.times.16 block (this being only one
particular example, which could, for example, be expandable to
larger blocks). The reason for the possibility of achieving a
better result is that, by separating functions between an
acceleration device and SW as described herein, the SW can use the
varied results provided by the acceleration device to potentially
find the "bigger picture" and produce a better result.
[0127] Turning now to the partitioning issue as described
above:
[0128] It is believed that the flexible acceleration device output
allows the SW to do re-partitioning based on the acceleration
device result, as described immediately above.
[0129] In certain exemplary embodiments of the present invention,
the acceleration device may not necessarily stop the search when
reaching a threshold; by contrast, SW algorithms generally have a
minimal threshold that causes the SW to stop looking for
candidates, which means (by way of non-limiting example) that if
the SW found a block with a small residual in the first search try,
it will terminate the search process.
[0130] In the particular non-limiting case described above, since
the partitioning is done later in the process, and as described in
the example, in order to avoid inappropriate termination, the
acceleration device will complete the search and return a vector of
the best results found, in order to allow the SW to do re
partitioning. Re-partitioning is discussed in more detail above
with reference to FIG. 5.
[0131] Aggregation Towards Larger Blocks [0132] The acceleration
device also, in exemplary embodiments, provides matching scores for
adjacent blocks (blocks that are "upper" and "left", by way of
non-limiting example, relative to a given block) in order to allow
aggregation to take place efficiently. In exemplary embodiments,
the aggregation is done when adjacent blocks has the same
displacement vector, with a bigger block which is aggregated to
replace the adjacent blocks having a matching result which is the
sum of the score of the sub blocks, since the score function is
additive.
[0133] Avoiding Local Minimum [0134] Reference is now additionally
made to FIG. 7, which is a simplified pictorial illustration
depicting a "local minimum" problem that may be encountered by an
exemplary system in accordance with an exemplary embodiment of the
present invention When searching on small blocks, it may happen
that the search algorithm will "fall" into a local minimum, as
shown in FIG. 7. Local minimum in this context refers to a small
block which may have a very good match (even a perfect match) to a
certain region, even though there may exist a match using a much
bigger block. Exemplary techniques for overcoming a local minimum
problem, using a plurality of block sizes to do so, are discussed
below with reference to FIG. 8.
[0135] When dealing with small blocks, there is higher chance that
many of the small blocks will be similar and the present invention,
in exemplary embodiments thereof, will not find the entire area of
a larger object. In order to overcome this problem, the
acceleration device, in exemplary embodiments, performs a full and
hierarchical search over the following so-called predictors:
[0136] 1. Results of the collocated (the term "collocated" being
known in the video art) block/s in one or more previously decoded
frames. In exemplary embodiments, using such results is configured
by SW. Such results may comprise a P&A map, as described below.
[0137] 2. Result/s from adjacent block/s, as described above.
[0138] 3. The result around the global motion vector of the image
(that is, the global motion vector is used as a center of a search
area), using such result being configured by SW, which may be the
case when SW provides such a global motion vector. [0139] 4. The
result of a low resolution image search, as described below in more
detail, including with reference to FIGS. 19-21.
Low Resolution Image Search
[0140] Reference is now additionally made to FIG. 8, which is a
simplified pictorial illustration depicting an operation on a low
resolution image, which operation may be carried out by an
exemplary system in accordance with an exemplary embodiment of the
present invention. In the low resolution search technique,
resolution of the image is reduced (the picture is made smaller),
such as, by way of non-limiting example, by a factor of 4 in each
dimension. In an exemplary embodiment, searching and motion
estimation are then performed on the reduced resolution image, as
shown in FIG. 8. In the non-limiting example of reduction by a
factor of 4, in the reduced resolution image of FIG. 8 an 8.times.8
block represents a 64.times.64 block in the original image; persons
skilled in the art will appreciate that there is less noise in such
a reduced resolution image than in the original image. Thus, a
candidate anchor identified in the reduced resolution image will
allow us to look for a larger anchor in the full-resolution
partition.
[0141] It is appreciated that, in embodiments of the present
invention, the acceleration device returns the best result of each
anchor, and sometimes the second best result, and not the total
ranking score.
[0142] Reference is now made to FIG. 18, which is a simplified
block diagram illustration depicting a particular exemplary case of
using the acceleration system of FIG. 1B. FIG. 18 depicts an
exemplary case in which a single pass is executed in order to
obtain a result; in general, whether a single pass or multiple
passes are used is decided by SW. Other exemplary cases, in which
more than a single pass may be executed, are described below.
[0143] In FIG. 18, the video acceleration device 120 of FIG. 1B is
shown receiving the following inputs:
[0144] a first prediction and aggregation (P&A) map 1810;
[0145] a second P&A map 1820, which may refer to a second
reference frame (such as the second reference frame 1840 mentioned
below);
[0146] a first reference frame 1830;
[0147] a second reference frame 1840; and
[0148] a target frame 1850.
[0149] The first reference frame 1830, second reference frame 1840,
and the target frame 1850 will be understood in light of the above
discussion.
[0150] The first P&A map 1810 and the second P&A map 1820
may be similar in form to the result map 140 of FIG. 1B, or, by way
of very particular non-limiting example, to the result map 500 of
FIG. 5. In general, the first P&A map 1810 and the second
P&A map 1820: [0151] may be optional; [0152] are provided by
SW; and [0153] are provided with a deliberately poor score, since
it is believed that providing a poor score will lead to a better
result.
[0154] The video acceleration device 120 produces a result map 1860
which may be similar in form to the result map 140 of FIG. 1B, or,
by way of very particular non-limiting example, to the result map
500 of FIG. 5.
[0155] The above description of FIG. 18 may be further elaborated
on as follows. In general, one or more reference frames and one or
more target frames are received as input. A plurality of P&A
maps may be received; by way of one particular non-limiting
example, when two reference frames and two target frames are
received, up to four P&A maps may be received. In general, a
given P&A map refers to a reference frame paired with a target
frame, so that if there are two reference frames and two target
frames, there would be four P&A maps to cover the applicable
pairings. In a case where a particular P&A map is blank, this
may be an indication to the video acceleration device 120 to search
without any additional information as to where to search. In
general, a P&A map provides predication and aggregation points
to aid the video acceleration device 120 in searching related to
each target.
[0156] Reference is now made to FIG. 19, which is a simplified
block-diagram illustration depicting another particular exemplary
case of using the acceleration system of FIG. 1B. FIG. 19 depicts
an exemplary case in which a two passes executed in order to obtain
a result; in general, whether a single pass or multiple passes are
used is decided by SW. The two passes depicted in FIG. 19 use
downscaling, which is intended to assist in avoiding a local
minimum, as described above.
[0157] In FIG. 19, a full resolution target frame 1910 and a full
resolution reference frame 1920 are provided as input. Each of the
full resolution target frame 1910 and the full resolution reference
frame 1920 are downscaled at downscale units 1930 and 1940
respectively (which for sake of simplicity of depiction and
explanation are shown as separate units, it being appreciated that
alternatively a single downscale unit may be used to carry out
multiple downscale operations). In the particular non-limiting
example shown in FIG. 19 the downscale units 1930 and 1940 are
shown as downscaling by a factor of 1:8, but it is appreciated that
the example of 1:8 is not meant to be limiting.
[0158] The output of the downscale unit 1930 is a downscaled target
frame 1950. The output of the downscale unit 1940 is a downscaled
reference frame 1960. The downscaled target frame 1950 and the
downscaled reference frame 1960 are input into the video
acceleration device 120. Two instances of the video acceleration
device 120 are shown in FIG. 19 for ease of depiction and
acceleration, it being appreciated that only a single video
acceleration device 120 may be used,
[0159] By way of non-limiting example, an empty P&A map 1965
(see description of P&A maps above, with reference to FIG. 18)
is also input into the video acceleration device 120. The video
acceleration device 120 which receives the downscaled target frame
1950 and the downscaled reference frame 1960 produces a P&A
map(R) 1970, "R" designating that the map relates to a reference
frame, representing the best downscaled result found for the
downscaled target frame 1950 in the downscaled reference frame
1960.
[0160] Meanwhile, the full resolution target frame 1910 and the
full resolution reference frame 1920 are each provide as input to
the video acceleration device 120, which also receives the P&A
map(R) 1970, an which produces a second P&A map(R) 1975. It is
appreciated that, when a method such as that depicted in FIG. 19 is
used, so that in addition to a full search on an entire frame a
downscaled search is used, information is produced which may assist
in finding larger blocks (such as, by way of non-limiting example,
64.times.64 blocks). This may be because, in effect, the system has
"zoomed in" on each frame, so that more of each frame will be in an
effective search area.
[0161] Reference is now made to FIG. 20, which is a simplified
block-diagram illustration depicting still another particular
exemplary case of using the acceleration system of FIG. 1B, and to
FIG. 21, which is a simplified block-diagram illustration depicting
yet another particular exemplary case of using the acceleration
system of FIG. 1B. FIGS. 20 and 21 will be understood with
reference to the above description of FIG. 19. FIG. 20 depicts a
situation in which 1:4 and 1:8 downscaling are both used and are
combined, FIG. 21 depicts a situation in which both symmetrical and
asymmetrical downscaling are used and combined. In general, in the
situations depicted in FIGS. 20 and 21, the "quality" of output of
the video acceleration device 120 may be improved, with little
additional work expended in a few additional passes.
[0162] Reference is now made to FIG. 22, which is a simplified
partly pictorial, partly block diagram illustration of an exemplary
embodiment of a portion of the application acceleration system of
FIG. 1B. FIG. 22 depicts a particular exemplary non-limiting
embodiment of the video acceleration device 120 of FIG. 1B.
[0163] The video acceleration device 120 of FIG. 22 is shown
receiving as input: a P&A map 2210; a reference frame 2215; and
a target frame 2220. Each of the P&A map 2210, the reference
frame 2215, and the target frame 2220 may be similar to P&A
maps, reference frames, and target frames described above.
[0164] The video acceleration device 120 of FIG. 22 comprises the
following elements:
[0165] a result map buffer 2225;
[0166] a reference frame buffer 2230;
[0167] a target frame buffer 2235;
[0168] a block matching engine 2240;
[0169] a score board storage unit 2245; and
[0170] aggregation and ranking circuitry 2255.
[0171] The result map buffer 2260 is shown as storing a map 2260,
which may be the input P&A map 2210 or another map, as also
described below.
[0172] A non-limiting example of operation of the video
acceleration device 120 of FIG. 22 is now briefly described:
[0173] The target frame 2220 or a relevant portion thereof
(typically determined under SW control) is received by the video
acceleration device 120, and at least a relevant portion is stored
in the target frame buffer 2235. By way of a particular non
limiting example, the relevant portion could comprise a current
block of 8.times.8 pixels to be searched.
[0174] The reference frame 2215 or a relevant portion thereof is
received by the video acceleration device 120, and a relevant
search area (which may be a search area around the current block in
the target frame) is stored in the reference frame buffer 2230.
[0175] The block matching engine 2240 (which may comprise a
plurality of block matching engines, in order to execute more than
one operation in parallel) receives current block stored in the
target frame buffer 2235 and the relevant blocks stored in the
reference frame buffer 2230. The block matching engine 2240
determines a score (using, by way of non-limiting example as
described above, SAD or SSE), and writes the score to the score
board storage unit 2245, producing a score board 2250. Score boards
are described above; one particular non-limiting example is the
score board 500 of FIG. 5, described above.
[0176] In certain exemplary embodiments, the block matching engine
2240 may use the P&A map 2210 (which may be stored in the
result map buffer 2225, or elsewhere in the video acceleration
device 120) to "focus" score determination on blocks indicated in
the P&A map 2210, and blocks in proximity to those blocks.
[0177] The aggregation and ranking circuitry 2255 is configured, in
exemplary embodiments, to determine the best results from the score
board 2250, and also to determine large blocks by aggregation,
using (by way of non-limiting example) sums of values of adjacent
blocks, which blocks have the same displacement vector as a given
block, in order to produce an output score board/result map 2260.
While not shown in FIG. 22, the output score board/result map is
generally provided as output from the video acceleration device
120, as described above.
[0178] Reference is now made to FIG. 23, which is a simplified
tabular illustration depicting an exemplary result of operations
performed by an acceleration device in accordance with another
exemplary embodiment of the present invention. Referring back to
FIG. 5 and the description thereof, FIG. 5 described a non-limiting
example of system output which included information on a reference
frame. In another exemplary embodiment, an entire table (such as,
by way of a very particular non-limiting example, the table
depicted in FIG. 23, generally designated 2300), may refer to a
particular reference frame, it being appreciated that a plurality
of tables similar to the table 2300 may be provided, one for each
reference frame. Thus, in the case of the table 2300 of FIG. 23 (by
contrast to the table 500 of FIG. 5), the reference motion vector
(Ref MV) may comprise 2 comma-delimited numbers, comprising x and y
coordinates respectively (by way of non-limiting example).
[0179] The following description may apply, mutatis mutandis,
either to the case depicted and described with reference to the
table 500 of FIG. 5, or to the case depicted and described with
reference to the table 2300 of FIG. 23.
[0180] Referring back to FIG. 17A, each residual block 1735
undergoes a transform operation at the transform unit 1740. The
transform operation, in exemplary embodiments, converts the input
(such as each residual block 1735) from a spatial domain (energy
per location in an image) to a frequency domain. A non-limiting
example of such a transform is a discrete cosine transform (DCT).
Generally, the output of such a transform is a block having the
same dimensions as the input block. The top left element of such a
block is called a DC element; in particular, it is known in the art
of video encoding that the DC element is very close to the average
of the values of intensity of pixels in the block, in the spatial
domain. The other elements in the transform output are called AC
elements. It is well known in the art of video compression that the
human eye model is less sensitive to errors in the higher
frequencies, which are the last elements of the transform output,
than to lower frequencies. For this reason, the quantizing unit
1750 generally quantizes the last AC elements more than the first
AC elements, and much more than the DC element.
[0181] It is also known in the art of video compression that
residual blocks with less energy in the AC coefficients are
compressed better than other residual blocks. In other words, with
fewer bits in a bitstream a decoder will be able to reconstruct a
block which is closer to a source signal; in this context "closer"
may be, by non-limiting example, as measured by the PSNR metric, as
referred to above.
[0182] However, when doing motion estimation, or trying to find the
best block in a reference image against which to compensate, it is
known in the art that doing a transform to each candidate block in
order to estimate the rate distortion optimization score (RDO
score) of that candidate block, is extremely compute intensive, and
may in fact be practically impossible.
[0183] The following formula is believed to be a good estimation of
the energy residing in AC coefficients (a term used interchangeably
herein with "AC elements"): Given a Target block T and a candidate
reference block C, the energy of AC coefficients a residual block R
that will be created when compensating Block T from block C is: AC
in R.about.SAD(T,C)-|AVG(T)-AVG(C)|
[0184] where: [0185] SAD represents Sum of Absolute Difference, as
described above; [0186] AVG represents the average of pixel values
in a given block; .about.represents approximation; and [0187]
.parallel. represents absolute value.
[0188] Reference is now made to FIG. 24, which is a simplified
tabular illustration depicting an exemplary result of operations
performed by an acceleration device in accordance with still
another exemplary embodiment of the present invention. The table of
FIG. 24,generally designated 2400, may be similar to the table 2300
of FIG. 23, with the addition of a column for AC energy estimation,
the values of which may be computed in accordance with the formula
provided above for "AC in R". It is appreciated that such a column
may also be added mutatis mutandis, to the table 500 of FIG. 5.
[0189] Referring back to FIG. 22, the system of FIG. 22, with minor
variations, may be used to produce the table of FIG. 24, the
variations being as follows: [0190] The block matching engine 2240,
in additional to determining a score (using, by way of non-limiting
example as described above, SAD or SSE) as described above with
reference to FIG. 22, also determines the AC coefficient energy
using the "AC in R" formula described above. If SSE was used as a
score function, as described above with reference to FIG. 22, then
SAD in the "AC in R" formula described above is replaced by SSE.
When writing the score to the score board storage unit 2245,
producing a score board 2250, as described above with reference to
FIG. 22, score board will include: the block number MV, SAD/SSE and
AC energy.
[0191] Referring again to FIG. 24, choosing the "best" block to be
compensated against, from the encoder perspective, may be
accomplished as follows. The "best" block will be a block that will
introduce minimal quality loss, and still meet a "bit budget"; that
is, generally an encoder is supplied with a "bit budget" indicating
how many bits may be used in encoding. Thus, the encoder generally
maintains an accounting of bit use/quality loss. For example, the
best block may be determined using a formula such as:
Cost=MV_cost+Residual_cost*alpha
where:
[0192] MV_cost is the number of bits that the encoder needs in
order to encode a given motion vector (MV) in the bitstream;
[0193] Residual_cost is the cost in bits, for the encoder to encode
the residual coefficient in the bitstream; referring logically to
the "delta" between the 2 blocks (target Vs reference). It is
appreciated that the Residual_cost depends on the SAD result and on
the AC energy result, since each block is transformed, subtracted
from the reference block, and then quantized. The quantization
process implies that, when using low bitrates, where usually higher
quantizers are used, the cost of the residuals will impact less,
while the cost of bits used to represent the MV is constant. To
account for differences in quantization, the alpha parameter is
introduced, the alpha parameter being generally different for each
quantization parameter. For higher quantizers (lower bitrates) the
alpha value is smaller than it is for lower quantizers (higher
bitrates).
[0194] The discussion immediately above implies that an
acceleration device as described herein may be configured to output
just the overall cost, or to rank based on the Cost function above,
and thus to reduce the amount of data that the encoder needs to
analyze. In order to accomplish this, the encoder (or software
associated therewith) configures the alpha value or values in
advance of operation, and also configures, for every frame being
searched against, an average quantization parameter (QP) to the
acceleration device, and an alpha value in accordance
therewith.
[0195] Reference is now made to FIG. 25, which is a simplified
tabular illustration depicting an exemplary result of operations
performed by an acceleration device in accordance with yet another
exemplary embodiment of the present invention. The table of FIG.
25, generally designated 2500, may be similar to the table 2300 of
FIG. 23, except that additional information, which may be useful in
facilitating bi-directional prediction, has been added to the table
2500, relative to the table 2300 of FIG. 23.
[0196] Bi-directional prediction is available in certain video
coding standards, and allows prediction from two images; this means
that each block in a target image can be compensated against two
different blocks, one from each of two different images. In such a
case, the acceleration device may compare each block against a
block which is a weighted sum of two blocks from two different
images, in order to produce the table 2500. As is known in the art,
a weighting coefficient used in computing the weighted sum is
constant for a given target frame.
[0197] Prior to the score function as shown in table 2500 being
calculated (using, by way of particular non-limiting example, SAD
or SSE), an "imaginary" reference block may be assembled using the
following formula:
RefBlock .sub.(i,j))=W.sub.0* B0(i,j)+W.sub.1 * B1(i,j)
[0198] Where W.sub.0 and W.sub.1 are weights (generally supplied by
the encoder, and based on values in relevant video compression
standards);
[0199] (i,j) represents a location within a given block;
[0200] B0 represents a first actual block;
[0201] B1 represents a second actual block; and RefBlock represents
the computed "imaginary" reference block.
[0202] It is appreciated, that, generally, W0 and W1 are not
dependent on i and j.
[0203] The acceleration device may then perform cost calculations
on the "imaginary" reference block (using the block matching engine
2240 of FIG. 22, as described above) as if the "imaginary"
reference block were an ordinary reference block, but with two
different displacement vectors being output to the score board. The
displacement vectors may be in a separate table (in a case such as
that of FIG. 25), or a separate row (in a case such as that of FIG.
5).
[0204] Exemplary embodiments of the present invention which may be
useful with future codecs are now described. In such future codecs,
it is believed that it will be possible to copy the content of a
block from a previous encoded/decoded block. Such an ability is
limited to copying data from the current processed coding tree unit
(CTU) or an immediately previous CTU only. This CTU restriction
simplifies intra block copy (IBC) implementation which may be
useful for such future codecs, since copying from a reconstructed
buffer may be problematic in some systems; and the CTU restriction
eliminates the need to access a reconstructed buffer by allowing
addition of a separate, relatively small, buffer for IBC purpose
only.
[0205] Reference is now made to FIGS. 26A and 26B, which are
simplified tabular illustrations useful in understanding still
another exemplary embodiment of the present invention. FIG. 26A
shows an array of blocks, generally designated 2610. A current
block being processed 2620 is depicted, along with a plurality of
"valid" possible reference blocks 2630.
[0206] The inventors of the present invention believe that by
relaxing the restriction of which blocks are "valid" as depicted in
FIG. 26A, efficiencies may be achieved by allowing computations to
take place for future codecs using frame data already processed.
FIG. 26B shows an array of blocks for such a case, generally
designated 2640. A current block 2650 is depicted, along with a
plurality of "valid possible reference blocks 2660 and 2670.
[0207] Using the acceleration device in accordance with exemplary
embodiments of the present invention (in particular with reference
to FIGS. I7A and 17B, and the above explanation thereof),
acceleration is based on receiving a reconstructed buffer in frame
resolution, meaning that each frame can access only the previous
processed reconstructed frame and can not the current reconstructed
frame data. Since future codecs as described above use the
reconstructed buffer of the current frame, implementation of a
solution in accordance with the exemplary embodiment of FIGS. 17A
and 17B would be difficult or perhaps impossible, since a current
frame reconstructed image is generally not available during
operation of the system of FIGS. 17A and 17B
[0208] Reference is now made to FIG. 27A, which is a simplified
partly pictorial, partly block-diagram illustration depicting an
exemplary system and process, including motion estimation, for
producing a bitstream; and to FIG. 27B, which is a simplified
partly pictorial, partly block-diagram illustration depicting an
exemplary system and process, including motion estimation, for
producing a bitstream, including use of an exemplary embodiment of
the acceleration system of FIG. 1B, FIGS. 27A and 27B are similar
to FIGS. 17A and 17B, respectively, and like reference numbers have
been used therein.
[0209] Using the systems and processes of FIGS. 27A and 27B, a
motion vector (MV) search is run on the original frame rather than
on the reconstructed frame. This solution can be executed prior to
the encoding of the frame. A limitation of this solution may be
differences between the original frame data and the reconstructed
frame data, but the inventors of the present invention believe the
original frame and the reconstructed frame (and hence the
corresponding data) images are quite similar, especially for
encoding in high bitrate. The purpose of the MV search in this case
is mainly for finding the best vector rather than calculating the
cost of such a vector, since it can be assumed that the best MV is
the same for both the original and the reconstructed frames, while
the cost itself is believed to be more affected by the difference
of the frames. The "real" cost can be calculated by the encoder
rather than by the systems and processes of FIGS. 27A and 27B, and
since no additional search is required the cost operation is
relatively simple.
[0210] Additionally, in alternative exemplary embodiments of the
present invention, it is appreciated that a similar concept can be
used for regular MV search; by running the MV search on the
original frame data rather than on reconstructed frame data it is
possible to use the acceleration device on an entire video prior to
the encoder running, so encoding can be more efficient and parallel
implementation can be made simpler.
[0211] It is appreciated that software components of the present
invention may, if desired, be implemented in ROM (read only memory)
form. The software components may, generally, be implemented in
hardware, if desired, using conventional techniques. It is further
appreciated that the software components may be instantiated, for
example: as a computer program product or on a tangible medium. In
some cases, it may be possible to instantiate the software
components as a signal interpretable by an appropriate computer,
although such an instantiation may be excluded in certain
embodiments of the present invention.
[0212] It is appreciated that various features of the invention
which are, for clarity, described in the contexts of separate
embodiments may also be provided in combination in a single
embodiment. Conversely, various features of the invention which
are, for brevity, described in the context of a single embodiment
may also be provided separately or in any suitable
subcombination.
[0213] It will be appreciated by persons skilled in the art that
the present invention is not limited by what has been particularly
shown and described hereinabove. Rather the scope of the invention
is defined by the appended claims and equivalents thereof:
* * * * *