U.S. patent application number 13/421788 was filed with the patent office on 2017-06-08 for method of signalling motion information for efficient scalable video compression.
This patent application is currently assigned to UNISEARCH LIMITED. The applicant listed for this patent is Andrew Secker, David Taubman. Invention is credited to Andrew Secker, David Taubman.
Application Number | 20170163993 13/421788 |
Document ID | / |
Family ID | 28047329 |
Filed Date | 2017-06-08 |
United States Patent
Application |
20170163993 |
Kind Code |
A9 |
Taubman; David ; et
al. |
June 8, 2017 |
Method Of Signalling Motion Information For Efficient Scalable
Video Compression
Abstract
Embodiments of a method for incrementally coding and signaling
motion information for a video compression system involving a
motion adaptive transform and embedded coding of transformed video
samples using a computer are disclosed herein. In one such
embodiment, the method includes (a) storing computer-readable
instructions in the computer which, when executed, produce an
embedded motion field bit-stream, representing each, motion field
in coarse to fine fashion and (b) storing computer-readable
instructions in the computer which, when executed, interleave
contributions from said embedded motion field bit-stream with
successive contributions from said embedded coding of the
transformed video samples.
Inventors: |
Taubman; David; (Sydney,
AU) ; Secker; Andrew; (Sydney, AU) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Taubman; David
Secker; Andrew |
Sydney
Sydney |
|
AU
AU |
|
|
Assignee: |
UNISEARCH LIMITED
Sydney
AU
|
Prior
Publication: |
|
Document Identifier |
Publication Date |
|
US 20120230414 A1 |
September 13, 2012 |
|
|
Family ID: |
28047329 |
Appl. No.: |
13/421788 |
Filed: |
March 15, 2012 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
10528965 |
Feb 13, 2006 |
|
|
|
PCT/AU03/01233 |
Sep 19, 2003 |
|
|
|
13421788 |
|
|
|
|
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
H04N 19/19 20141101;
H04N 19/184 20141101; H04N 19/54 20141101; H04N 19/31 20141101;
H04N 19/46 20141101; H04N 19/62 20141101; H04N 19/147 20141101;
H04N 19/122 20141101; H04N 19/52 20141101; H04N 19/635
20141101 |
International
Class: |
H04N 7/32 20060101
H04N007/32 |
Foreign Application Data
Date |
Code |
Application Number |
Sep 20, 2002 |
AU |
2002951574 |
Claims
1. A method for incrementally coding and signaling motion
information for a video compression system involving a motion
adaptive transform and embedded coding of transformed video samples
using a computer, said method comprising the steps of: (a) storing
computer-readable instructions in the computer which, when
executed, produce an embedded motion field bit-stream, representing
each motion field in coarse to fine fashion; and (b) storing
computer-readable instructions in the computer which, when
executed, interleave successive contributions from said embedded
motion field bit-stream with successive contributions from said
embedded coding of the transformed video samples.
2. A system for incrementally coding and signaling motion
information for a video compression system involving a motion
adaptive transform and embedded coding of transformed video
samples, said system comprising: (a) means for producing an
embedded motion field bit-stream, representing each motion field in
coarse to fine fashion; and (b) means for interleaving successive
contributions from said embedded motion fields field bit-stream
with successive contributions from said embedded coding of the
transformed video samples.
3. A system for estimating and communicating motion information
required by a multi-frame encoding and decoding system which
involves a motion adaptive transform based on temporal lifting
steps, said system comprising: (a) means for estimating and
communicating motion parameters describing a first mapping from a
source frame onto a target frame within one of the lifting steps;
(b) means for inferring a second mapping within the encoding system
between either said source frame or said target frame, and another
frame, based on the estimated and signaled motion parameters
associated with said first mapping; and (c) means for inferring the
second mapping within the decoding system between either said
source frame or said target frame and another frame, based on the
communicated motion parameters associated with said first
mapping.
4. The system of claim 2, where the embedded motion field
bit-stream is obtained by applying embedded quantization and coding
techniques to motion field parameter values.
5. The system of claim 2, where the embedded motion field
bit-stream is obtained by coding node displacement parameters
associated with a triangular mesh motion model on a coarse to fine
grid, each successive segment of the embedded bit-stream providing
displacement parameters for node positions which lie on a finer
grid than the previous stage, all coarser grids of node positions
being subsets of all finer grids of node points.
6. The system of claim 5, where a coarse to fine motion
representation is obtained by first transforming the motion
parameters and then coding the transform coefficients using
embedded quantization and coding techniques.
7. The system of claim 6, where the motion parameters are
transformed by applying at least one of either spatial discrete
wavelet transforms or temporal transforms thereto.
8. The system of claim 7, wherein at least one of either the
spatial or temporal transforms are reversible integer-to-integer
transforms, suitable for lossless compression.
9. The system of claim 2, wherein the embedded motion bit-streams
are arranged into a sequence of quality layers, and the transformed
video samples are also encoded into embedded bit-streams which are
arranged into a separate sequence of quality layers.
10. The system of claim 2, where said interleaving of the
contributions from the embedded motion bit-streams and from the
transformed video samples is performed in a manner which minimizes
expected distortion in reconstructed video sequence at each of a
plurality of compressed video bit-rates.
11. The system of claim 10, where the measure of distortion is Mean
Squared Error.
12. The system of claim 10, where the measure of distortion is a
weighted sum of the Mean Squared Error contributions from different
spatial frequency bands, weighted according to perceptual relevance
factors.
13. The system of claim 10, where the distortion associated with
inaccurate representation of the motion parameters is determined
using an estimate of a spatial power spectrum of the video
source.
14. The system of claim 10, where the distortion associated with
inaccurate representation of the motion parameters is determined
using information about a spatial resolution at which the video
bit-stream is to be decompressed.
15. The system of claim 13, where the spatial power spectrum of the
video source is estimated using spatio-temporal video sample
subbands created during compression.
16. The system of claim 2, wherein proportions of contributions
from said embedded motion fields and said transformed video samples
in the embedded bit-stream are determined on the basis of a
plurality of tables associated with each frame, each table being
associated with a spatial resolution at which the video bit-stream
is to be decompressed.
17. The system of claim 16, wherein the embedded motion bit-streams
and the transformed video samples are each encoded as a series of
quality layers and the tables identify a number of motion quality
layers are to be included with each number of video sample quality
layers.
18. The system of claim 3, where said second mapping is the
reciprocal mapping from said target frame to said source frame, for
use within another one of the lifting steps.
19. The system of claim 18, where said reciprocal mapping is the
inverse of the first mapping.
20. The system of claim 19, where the motion parameters of said
first mapping correspond to a deformable triangular mesh motion
model.
21. The system of claim 20, where said reciprocal mapping is
inferred by inverting the affine transformations associated with
the triangular mesh used to represent said first mapping.
22. The system of claim 3, where the motion parameters of said
first mapping correspond to a block displacement motion model.
23. The system of claim 3, where said motion adaptive transform
involves multiple stages of temporal decomposition, corresponding
to different temporal frame rates.
24. The system of claim 23 where motion parameters at each temporal
resolution are deduced from original video frames.
25. The system of claim 23, wherein said second mapping is a
mapping between frames at a lower temporal resolution than said
first mapping, and said second mapping is inferred by compositing
the first mapping with at least one further mapping between frames
at the higher temporal resolution.
26. The system of claim 23, wherein said second mapping is a
mapping between frames at a higher temporal resolution than said
first mapping, and said second mapping is inferred by compositing
the first mapping with at least one further mapping at the higher
temporal resolution.
27. The system of claim 26, wherein said higher resolution is
double said lower resolution, and alternate mappings at the higher
temporal resolution are explicitly signaled to a decompressor, the
remaining mappings at the higher temporal resolution being replaced
by the mappings inferred by compositing the lower resolution
mappings with respective higher resolution mappings.
28. The system of claim 27, where said replaced mappings are used
within the lifting steps of said motion adaptive transform, in
place of the originally estimated mappings which were replaced.
29. The system of claim 27, where said replaced mappings are
refined with additional motion parameters, said refinement
parameters being signaled for use in decompression, and said
replaced and refined mappings being used within the lifting steps
of said motion adaptive transform, in place of the originally
estimated mappings which were replaced.
30. The system of claim 3, where inversion or composition of motion
transformations is accomplished by applying said motion
transformations to the node positions of a triangular mesh motion
model, the composited or inverted motion transformation being
subsequently applied by performing the affine transformations
associated with said mesh motion model.
31. The system of claim 30, where the source frame is partitioned
into a regular mesh and the inversion or composition operations are
applied to each node of the regular mesh to find a corresponding
location in the target frame, the composited or inverted motion
transformation being subsequently applied by performing the affine
transformations associated with said mesh motion model.
32. A non-transitory computer-readable storage medium with an
executable program stored thereon, wherein the program instructs
the computer to implement the method of claim 1.
33. A non-transitory computer-readable storage medium with an
executable program stored thereon, wherein the program instructs
the computer to implement the method of claim 2.
34. The method of claim 1, where the embedded motion field
bit-stream is obtained by applying embedded quantization and coding
techniques to motion field parameter values.
35. The method of claim 1, where the embedded motion field
bit-stream is obtained by coding node displacement parameters
associated with a triangular mesh motion model on a coarse to fine
grid, each successive segment of the embedded bit-stream providing
displacement parameters for node positions which lie on a finer
grid than the previous stage, all coarser grids of node positions
being subsets of all finer grids of node points.
36. The method of claim 35, where a coarse to fine motion
representation is obtained by first transforming the motion
parameters and then coding the transform coefficients using
embedded quantization and coding techniques.
37. The method of claim 36, where the motion parameters are
transformed by applying at least one of either spatial discrete
wavelet transforms or temporal transforms thereto.
38. The method of claim 37, wherein at least one of either the
spatial or temporal transforms are reversible integer-to-integer
transforms, suitable for lossless compression.
39. The method of claim 1, wherein the embedded motion bit-streams
are arranged into a sequence of quality layers, and the transformed
video samples are also encoded into embedded bit-streams which are
arranged into a separate sequence of quality layers.
40. The method of claim 1, where said interleaving of the
contributions from the embedded motion bit-streams and from the
transformed video samples is performed in a manner which minimizes
expected distortion in reconstructed video sequence at each of a
plurality of compressed video bit-rates.
41. The method of claim 40, where the measure of distortion is Mean
Squared Error.
42. The method of claim 40, where the measure of distortion is a
weighted sum of the Mean Squared Error contributions from different
spatial frequency bands, weighted according to perceptual relevance
factors.
43. The method of claim 40, where the distortion associated with
inaccurate representation of the motion parameters is determined
using an estimate of a spatial power spectrum of the video
source.
44. The method of claim 40, where the distortion associated with
inaccurate representation of the motion parameters is determined
using information about a spatial resolution at which the video
bit-stream is to be decompressed.
45. The method of claim 43, where the spatial power spectrum of the
video source is estimated using spatio-temporal video sample
subbands created during compression.
46. The method of claim 1, wherein proportions of contributions
from said embedded motion fields and said transformed video samples
in the embedded bit-stream are determined on the basis of a
plurality of tables associated with each frame, each table being
associated with a spatial resolution at which the video bit-stream
is to be decompressed.
47. The method of claim 46, wherein the embedded motion bit-streams
and the transformed video samples are each encoded as a series of
quality layers and the tables identify a number of motion quality
layers are to be included with each number of video sample quality
layers.
48. A method for estimating and communicating motion information
required by a multi-frame encoding and decoding system which
involves a motion adaptive transform based on temporal lifting
steps using a computer, said method comprising the steps of: (a)
storing computer-readable instructions in the computer which, when
executed, estimate and communicate motion parameters describing a
first mapping from a source frame onto a target frame within one of
the lifting steps; (b) storing computer-readable instructions in
the computer which, when executed, infer a second mapping within
the encoding system between either said source frame or said target
frame, and another frame, based on the estimated and communicated
motion parameters associated with said first mapping; and (c)
storing computer-readable instructions in the computer which, when
executed, infer the second mapping within the decoding system
between either said source frame or said target frame and another
frame, based on the communicated motion parameters associated with
said first mapping.
49. The method of claim 48, where said second mapping is the
reciprocal mapping from said target frame to said source frame, for
use within another one of the lifting steps.
50. The method of claim 49, where said reciprocal mapping is the
inverse of the first mapping.
51. The method of claim 48, where the motion parameters of said
first mapping correspond to a deformable triangular mesh motion
model.
52. The method of claim 51, where said reciprocal mapping is
inferred by inverting the affine transformations associated with
the triangular mesh used to represent said first mapping.
53. The method of claim 48, where the motion parameters of said
first mapping correspond to a block displacement motion model.
54. The method of claim 48, where said motion adaptive transform
involves multiple stages of temporal decomposition, corresponding
to different temporal frame rates.
55. The method of claim 54 where motion parameters at each temporal
resolution are deduced from original video frames.
56. The method of claim 54, wherein said second mapping is a
mapping between frames at a lower temporal resolution than said
first mapping, and said second mapping is inferred by compositing
the first mapping with at least one further mapping between frames
at the higher temporal resolution.
57. The method of claim 54, wherein said second mapping is a
mapping between frames at a higher temporal resolution than said
first mapping, and said second mapping is inferred by compositing
the first mapping with at least one further mapping at the higher
temporal resolution.
58. The method of claim 57, wherein said higher resolution is
double said lower resolution, and alternate mappings at the higher
temporal resolution are explicitly signaled to a decompressor, the
remaining mappings at the higher temporal resolution being replaced
by the mappings inferred by compositing the lower resolution
mappings with respective higher resolution mappings.
59. The method of claim 58, where said replaced mappings are used
within the lifting steps of said motion adaptive transform, in
place of the originally estimated mappings which were replaced.
60. The method of claim 58, where said replaced mappings are
refined with additional motion parameters, said refinement
parameters being signaled for use in decompression, and said
replaced and refined mappings being used within the lifting steps
of said motion adaptive transform, in place of the originally
estimated mappings which were replaced.
61. The method of claim 60, where inversion or composition of
motion transformations is accomplished by applying said motion
transformations to the node positions of a triangular mesh motion
model, the composited or inverted motion transformation being
subsequently applied by performing the affine transformations
associated with said mesh motion model.
62. The method of claim 61, where the source frame is partitioned
into a regular mesh and the inversion or composition operations are
applied to each node of the regular mesh to find a corresponding
location in the target frame, the composited or inverted motion
transformation being subsequently applied by performing the affine
transformations associated with said mesh motion model.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application is a continuation of U.S. patent
application Ser. No. 10/528,965, filed Feb. 13, 2006, which was the
National Stage of International Application No. PCT/AU03/01233,
filed Sep. 19, 2003 and claims priority to Australian Patent
Application No. 2002951574, filed Sep. 20, 2002, all of which are
hereby incorporated by reference in their entireties.
TECHNICAL FIELD
[0002] The present invention relates to efficient compression of
motion video sequences and, in preferred embodiments, to a method
for producing a fully scalable compressed representation of the
original video sequence while exploiting motion and other
patio-temporal redundancies in the source material. The invention
relates specifically to the representation and signalling of motion
information within a scalable compression framework which employs
motion adaptive wavelet lifting steps. Additionally, the present
invention relates to the estimation of motion parameters for
scalable video compression and to the successive refinement of
motion information by temporal resolution, spatial resolution or
precision of the parameters.
BACKGROUND
[0003] For the purpose of the present discussion, the term
"internet" will be used both in its familiar sense and also in its
generic sense to identify a network connection over any electronic
communications medium or collection of cooperating communications
systems.
[0004] Currently, most video content which is available over the
internet must be pre-loaded in a process which can take many
minutes over typical modem connections, after which the video
quality and duration can still be quite disappointing. In some
contexts video streaming is possible, where the video is
decompressed and rendered in real-time as it is being received;
however, this is limited to compressed bit-rates which are lower
than the capacity of the relevant network connections. The most
obvious way of addressing these problems would be to compress and
store the video content at a variety of different bit-rates, so
that individual clients could choose to browse the material at the
bit-rate and attendant quality most appropriate to their needs and
patience. Approaches of this type, however, do not represent
effective solutions to the video browsing problem. To see this,
suppose that the video is compressed at bit-rates of R, 2R, 3R, 4R
and 5R. Then storage must be found on the video server for all
these separate compressed bit-streams, which is clearly wasteful.
More importantly, if the quality associated with a low bit-rate
version of the video is found to be insufficient, a complete new
version must be downloaded at a higher bit-rate; this new
bit-stream must, take longer to download, which generally rules out
any possibility of video streaming.
[0005] To enable real solutions to the remote video browsing
problem, scalable compression techniques are essential. Scalable
compression refers to the generation of a bit-stream which contains
embedded subsets, each of which represents an efficient compression
of the original video with successively higher quality. Returning
to the simple example above, a scalable compressed video bit-stream
might contain embedded sub-sets with the bit-rates of R, 2R, 3R, 4R
and 5R, with comparable quality to non-scalable bit-streams, having
the same bit-rates. Because these subsets are all embedded within
one another, however, the storage required on the video server is
identical to that of the highest available bit-rate. More
importantly, if the quality associated with a low bit-rate version
of the video is found to be insufficient, only the incremental
contribution required to achieve the next higher level of quality
must be retrieved from the server. In a particular application, a
version at rate R might be streamed directly to the client in
real-time; if the quality is insufficient, the next rate-R
increment could be streamed to the client and added to the
previous, cached bit-stream to recover a higher quality rendition
in real time. This process could continue indefinitely without
sacrificing the ability to display the incrementally improving
video content in realtime as it is being received from the
server.
[0006] The above application could be extended in a number of
exciting ways. Firstly, if the scalable bit-stream also contains
distinct subsets corresponding to different intervals in time, then
a client could interactively choose to refine the quality
associated with specific time segments which are of the greatest
interest. Secondly, if the scalable bit-stream also contains
distinct subsets corresponding to different spatial regions, then
clients could interactively choose to refine the quality associated
with specific spatial regions over specific periods of time,
according to their level of interest. In a training video, for
example, a remote client could interactively "revisit" certain
segments of the video and continue to stream higher quality
information for these segments from the server, without incurring
any delay.
[0007] To satisfy the needs of applications such as that mentioned
above, low bit-rate subsets of the video must be visually
intelligible. In practice, this means that most of the available
bits will be devoted to a low bit-rate portion of the video are
likely to contribute to the reconstruction of the video at a
reduced frame rate, since attempting to recover the full frame rate
video over a low bit-rate channel will result in unacceptable
deterioration of the spatial details within each frame. In order to
achieve smooth quality scalability within a compressed video
sequence which also offers frame rate scalability, the details
required to recover higher frame rates must contribute to the
refinement of a model which involves motion sensitive temporal
interpolation.
[0008] Without temporal interpolation, missing frames cannot be
introduced into a low rate video sequence without first augmenting
their spatial fidelity to a level commensurate with the frames
already available, and this implies a large discontinuous jump in
the amount of information which must be provided to the decoder in
order to smoothly increase the reconstructed video quality.
Continuing this line of argument, we see that motion information is
important to highly scalable video compression; moreover, the
motion itself, must be represented in a manner which can be scaled,
according to the temporal resolution (frame rate), spatial
resolution and quality of the sample data.
Motion Adaptive Transforms Based on Wavelet Lifting
[0009] The present invention is best appreciated in the context of
an earlier invention, which is the subject of WO02/50772. This
earlier patent application describes a method for modifying the
individual lifting steps in a lifting implementation of a temporal
wavelet decomposition, so as to compensate for the effects of
motion. This method has the following advantageous properties: 1)
the motion sensitive transform may be perfectly inverted, in the
absence of any compression artefacts; 2) the low temporal
resolution subsets of the wavelet hierarchy offer high spatial
fidelity so that the transform allows excellent frame rate
scalability; 3) the high pass temporal detail subbands produced by
the transform have very low energy, allowing high compression
efficiency; 4) in the absence of motion, the transform reduces to a
regular wavelet decomposition along the temporal axis; and 5) in
the presence of locally translational motion, the transform is
equivalent to applying a regular wavelet decomposition along the
motion trajectories.
[0010] To assist in the present discussion, we briefly summarise
the key ideas behind this earlier invention. Any two-channel FIR
subband transform can be described as a finite sequence of lifting
steps [W. Sweldens, "The lifting scheme: A custom-design
construction of biorthogonal wavelets," Applied and Computational
Harmonic Analysis, vol 3, pp 196-2000, April 1996]. It is
instructive to begin with an example based upon the Haar wavelet
transform. Up to a scale factor, this transform may be realised in
the temporal domain, through a sequence of two lifting steps,
as
h k [ n ] = x 2 k + 1 - x 2 k [ n ] ##EQU00001## 1 k [ n ] = x 2 k
+ 1 2 h 2 k [ n ] ##EQU00001.2##
where x.sub.k[n].varies.x.sub.k[n.sub.1,n.sub.2] denotes the
samples of frame k from the original video sequence and
h.sub.k[n].varies.h.sub.k[n.sub.1,n.sub.2] and
l.sub.k[n].varies.l.sub.k[n.sub.1,n.sub.2] denote the high-pass and
low-pass subband frames.
[0011] l.sub.k[n] and h.sub.k[n] correspond to the scaled sum and
the difference of each original pair of flames. An example is shown
in FIG. 1A. Since-motion is ignored, ghosting artefacts are clearly
visible in the low-pass temporal subband, and the high-pass
subband, frame has substantial energy.
[0012] Now let W.sub.k1.fwdarw.k2 denote a motion-compensated
mapping of frame k1 onto the coordinate system of frame k2, so that
W.sub.k1.fwdarw.k2(x.sub.k1[n].apprxeq.x.sub.k2[n]) for all n. The
lifting steps are modified as follows.
h k [ n ] = x 2 k + 1 [ n ] - W 2 k .fwdarw. 2 k + 1 ( x 2 k ) [ n
] ( 1 ) l k [ n ] = x 2 k [ n ] + 1 2 W 2 k + 1 .fwdarw. 2 k ( h k
[ n ] ) ( 2 ) ##EQU00002##
Note that W.sub.2k.fwdarw.2k+1 and W.sub.2k+1.fwdarw.2k represent
forward and backward motion mappings, respectively. The high-pass
subband frames correspond to motion-compensated residuals. These
will be close to zero in regions where the motion is accurately
modelled. The result is shown in FIG. 1B.
[0013] The framework described above is readily extended to any
two-channel FIR subband transform, by motion-compensating the
relevant lifting steps.
[0014] We demonstrate this in the important case of the
biorthogonal 5/3 wavelet transform [D. Le Gall and A. Tabatabai,
"Sub-band coding of digital images using symmetric short kernal
filters and arithmetic coding techniques," IEEE International
Conference on Acoustics, Speech and Signal Processing, vol. 2, pp
761-764, April 1988]. As before, x.sub.2k[n] and x.sub.2k+1[n]
denote the even and odd indexed frames from the original sequence.
Without motion, the 5/3 transform may be implemented by
alternatively updating each of these two frame subsequences, based
on filtered versions of the other sub-sequence. The lifting steps
are
h k [ n ] = x 2 k + 1 [ n ] - 1 2 ( x 2 k [ n ] - x 2 k + 2 [ n ] )
##EQU00003## l k [ n ] = x 2 k [ n ] + 1 4 ( h k - 1 [ n ] )
##EQU00003.2##
[0015] As before, we introduce motion warping operators within each
lifting step, which yields the following
h k [ n ] = x 2 k + 1 [ n ] - 1 2 ( W 2 k .fwdarw. 2 k + 1 ( x 2 k
) [ n ] + W 2 k + 2 .fwdarw. 2 k + 1 ( x 2 k + 2 ) [ n ] ) ( 3 ) l
k [ n ] = x 2 k [ n ] + 1 4 ( W 2 k - 1 .fwdarw. 2 k ( h k - 1 ) [
n ] + W 2 k + 1 .fwdarw. 2 k + 1 ( h k ) [ n ] ) ( 4 )
##EQU00004##
[0016] FIG. 2 demonstrates the effect of these modified lifting
steps. The highpass frames are now essentially the residual from a
bidirectional motion compensated prediction of the odd-indexed
original frames. When the motion is adequately captured, these
high-pass frames have little energy and the low-pass frames have
excellent spatial fidelity.
Counting the Cost of Motion
[0017] In the example of the Haar transform, given above, two
separate motion mapping operators, W.sub.2k.fwdarw.2k+1 and
W.sub.2k+1.fwdarw.2k, are required to process every pair of frames,
x.sub.2k[n] and x.sub.2k+1[n]. Their respective motion parameters
must be transmitted to the decoder. To provide a larger number of
temporal resolution levels, the transform is re-applied to the
low-pass subband frames, lk[n], for which motion mapping operators
W.sub.4k.fwdarw.4k+2 and W.sub.4k+2.fwdarw.4k are required for
every four frames. Continuing in this way, an arbitrarily large
number of temporal resolutions may be obtained, using
2 2 + 2 4 + 2 8 + .2 ##EQU00005##
motion fields per original frame.
[0018] For the example of the 5/3 transform, also given above, four
motion mapping operators, W.sub.2k.fwdarw.2k+1,
W.sub.2k.fwdarw.2k-1, W.sub.2k+1.fwdarw.2k and W.sub.2k-1.fwdarw.2k
are required for every pair of frames (indexed by k), for just one
level of temporal decomposition. Continuing the transformation to
an arbitrarily large number of temporal resolutions involves
approximately 4 motion fields per original video frame.
[0019] The cost of estimating, coding and transmitting the above
motion fields can be substantial. Moreover, this cost may adversely
affect the scalability of the entire compression scheme, since it
is not immediately clear how to progressively refine the motion
fields without destroying the subjective properties of the
reconstructed video when the motion is represented with reduced
accuracy.
[0020] The previous invention clearly reveals the fact that any
number of motion modelling techniques are compatible with the
motion adaptive lifting transform, and also recommends the use of
continuously deformable motion models such as those associated
with, triangular or quadrilateral meshes (see, for example, Y.
Nakaya and H. Harashima, "Motion compensation based on spatial
transformations," IEE Trans. Circ. Syst. For Video Tech., Vol. 4,
pp 339-367, June 1994). However, no particular solution is
presented to the difficulties described above.
SUMMARY
[0021] Accordingly, in one aspect, the present invention provides a
method for incrementally coding and signalling motion information
for a video compression system involving a motion adaptive
transform and embedded coding of transformed video samples, said
method comprising the steps of: (a) producing an embedded
bit-stream, representing each motion field in coarse to fine
fashion; and (b) interleaving incremental contributions from said
embedded motion fields with incremental contributions from said
transformed video samples.
[0022] The present invention also provides a system for
incrementally coding and signalling motion information for a video
compression system involving a motion adaptive transform and
embedded coding of transformed video samples, said system
comprising: (a) means for producing an embedded bit-stream,
representing each motion field in coarse to fine fashion; and (b)
means for interleaving incremental contributions from said embedded
motion fields with incremental contributions from said transformed
video samples.
[0023] Thus, because each motion field is represented in coarse to
fine fashion and interleaved with the video data bit-stream, the
accuracy required for motion representation can be balanced with
the accuracy of the transformed sample values which may be
recovered from the bit-stream. Therefore, a fully scalable video
bit-stream may be progressively refined, both in regard to its
quantised sample representations and in regard to its motion
representation.
[0024] Preferably, the embedded motion field bit-stream is obtained
by applying embedded quantization and coding techniques to the
motion field parameter values.
[0025] Preferably, the embedded motion field bit-stream is obtained
by coding the node displacement parameters associated with a
triangular mesh motion model on a coarse to fine grid, each
successive segment of the embedded bit-stream providing
displacement parameters for node positions which lie on a finer
grid than the previous stage, all coarser grids of node positions
being subsets of all finer grids of node points.
[0026] Preferably, a coarse to fine motion representation is
obtained by first transforming the motion parameters and then
coding the transform coefficients using embedded quantization and
coding techniques.
[0027] Preferably, the motion parameters are transformed by
applying spatial discrete wavelet transforms and/or temporal
transforms thereto.
[0028] Preferably, the spatial and/or temporal transforms are
reversible integer-to-integer transforms, suitable for lossless
compression.
[0029] Preferably, the embedded motion bit-streams are arranged
into a sequence of quality layers, and the transformed video
samples are also encoded into embedded bit-streams which are
arranged into a separate sequence of quality layers.
[0030] Preferably, said interleaving of the contributions from the
embedded motion bit-streams and from the transformed video samples
is performed in a manner which minimizes the expected distortion in
the reconstructed video sequence at each of a plurality of
compressed video bit-rates.
[0031] Preferably, the measure of distortion is Mean Squared Error.
Preferably, the measure of distortion is a weighted sum of the Mean
Squared Error contributions from different spatial frequency bands,
weighted according to perceptual relevance factors.
[0032] Preferably, the distortion associated with inaccurate
representation of the motion parameters is determined using an
estimate of the spatial power spectrum of the video source.
[0033] Preferably, the distortion associated with inaccurate
representation of the motion parameters is determined using
information about the spatial resolution at which the video
bit-stream is to be decompressed.
[0034] Preferably, the power spectrum of the video source is
estimated using spatio-temporal video sample subbands created
during compression.
[0035] Preferably, the proportions of contributions from said
embedded motion fields and said transformed video samples in the
embedded bit-stream is determined on the basis of a plurality of
tables associated with each frame, each table being associated with
a spatial resolution at which the video bit-stream is to be
decompressed. In the embodiment wherein the embedded motion
bit-streams and the transformed video, samples are each encoded as
a series of quality layers, the tables identify the number of
motion quality layers which are to be included with each number of
video sample quality layers.
[0036] The preferred structure of the motion representation allows
rate-distortion optimal algorithms to balance the contributions of
motion information and sample accuracy, as it is being included
into an incrementally improving (or layered) compressed
representation. While rate-distortion optimisation strategies for
balancing motion and sample accuracy have been described in the
literature, those algorithms were applicable only to static
optimisation of a compressed bit-stream for a single target
bit-rate. The preferred embodiment of the present invention allows
for the rate-distortion optimised balancing of motion and sample
accuracy to be extended to scalable content in which the target
bit-rate cannot be known a priori.
[0037] According to a further aspect of the present invention, a
method for estimating and signalling motion information for a
motion adaptive transform based on temporal lifting steps,
comprises the steps of (a) estimating and signalling motion
parameters describing a first mapping from a source frame onto a
target frame within one of the lifting steps; and (b) inferring a
second mapping between either said source frame or said target
frame, and another frame, based on the estimated and signalled
motion parameters associated with said first mapping.
The present invention also provides a system for estimating and
signalling motion information for a motion adaptive transform based
on temporal lifting steps, said system comprising: (a) means for
estimating and, signalling motion, parameters describing a first
mapping from a source frame onto a target frame within one of the
lifting steps; and (b) means for inferring a second mapping between
either said source frame or said target frame, and another frame,
based on the estimated and signalled motion parameters associated
with said first mapping. Accordingly, the number of motion fields
which must be signalled to the decompressor can be reduced, as some
motion fields can be inferred from others.
[0038] For instance, in one embodiment said second mapping is the
reciprocal mapping from said target frame to said source frame, for
use within another one, of the lifting steps. Preferably, said
reciprocal mapping is the inverse of the first mapping.
[0039] Thus, the preferred embodiment provides a method for
estimating and representing only one of the motion fields in each
pair, W.sub.2k.fwdarw.2k+1 and W.sub.2k+1.fwdarw.2k, or
W.sub.2k.fwdarw.2k-1, and W.sub.2k-1.fwdarw.2k. Such pairs of
motion fields will be known here as "reciprocal pairs." This allows
the total amount of motion information to be reduced to one motion
field per frame for the Haar case, and 2 motion fields per frame
for the 5/3 case. It is found that collapsing reciprocal pairs to a
single motion field, from which the pair is recovered, actually
improves the properties of the motion adaptive transform, resulting
in increased compression efficiency, even when the benefits of
reduced motion cost are not taken into account.
[0040] In one embodiment, the motion parameters of said first
mapping correspond to deformable triangular mesh motion model.
Preferably, said reciprocal mapping is inferred by inverting the
affine transformations associated with the triangular mesh used to
represent said first mapping.
[0041] In another embodiment, the motion parameters of said first
mapping correspond to a block displacement motion model.
Preferably, said motion adaptive transform involves multiple stages
of temporal decomposition, corresponding to different temporal
frame rates.
[0042] Preferably, motion parameters at each temporal resolution
are deduced from original video frames.
[0043] In one embodiment said second mapping is a mapping between
frames at a lower temporal resolution than said first mapping, and
said second mapping is inferred by compositing the first mapping
with at least one further mapping between frames at the higher
temporal resolution.
[0044] This embodiment enables all of the required motion fields at
lower temporal resolutions (higher temporal displacements) to be
derived from an initial set of frame-to-frame motion fields. Thus,
the compressor need only estimate the motion between each
successive pair of frames, x.sub.k[n] and x.sub.k+1[n]. This
substantially reduces the cost in memory and computation of the
motion estimation task, without significantly altering the
compression efficiency or other properties of the motion adaptive
transform.
[0045] In another embodiment, said second mapping is a mapping
between frames at a higher temporal resolution than said first
mapping, and said second mapping is inferred by compositing the
first mapping with at least one further mapping at the higher
temporal resolution. For example, preferably the higher resolution
is double said lower resolution, and alternate mappings at the
higher temporal resolution are explicitly signalled to a
decompressor, the remaining mappings at the higher temporal
resolution being replaced by the mappings inferred by compositing
the lower resolution mappings with respective higher resolution
mappings. Preferably said replaced mappings are used within the
lifting steps of said motion adaptive transform, in place of the
originally estimated mappings which were replaced.
[0046] This further reduces the motion information to 1 motion
field per video frame, even for the 5/3 transform. The method of
this embodiment has the property that the motion representation is
temporally scalable. In particular, only one motion field must
barnacle available to the decoder for each video frame which it can
reconstruct, at any selected temporal resolution. This method
involves judicious compositing of the forward and backward motion
fields from different temporal resolution levels and is compatible
with the efficient motion estimation method described above, of
compositing motion fields at higher resolutions to obtain motion
fields at lower resolutions.
[0047] Preferably said replaced mappings are refined with
additional motion parameters, said refinement parameters being
signalled for use in decompression, and said replaced and refined
mappings being used within the lifting steps of said motion
adaptive transform, in place of the originally estimated mappings
which were replaced.
[0048] Preferably, inversion or composition of motion
transformations is accomplished by applying said motion
transformations to the node positions of a triangular mesh motion
model, the composited or inverted motion transformation being
subsequently applied by performing the affine transformations
associated with said mesh motion model.
[0049] Preferably, the source frame is partitioned into a regular
mesh and the inversion or composition operations are applied to
each node dale regular mesh to find a corresponding location in the
target frame, the composited or inverted motion transformation
being subsequently applied by performing the affine transformations
associated with said mesh motion model. This is a particularly
efficient computational method for performing the various motion
field transformations required by other aspects of the invention.
These methods are preferably replicated at both the compressor and
the decompressor, if the transform is to remain strictly
invertible.
BRIEF DESCRIPTION OF THE DRAWINGS
[0050] The various features, advantages and other uses of the
present apparatus will become more apparent by referring to the
following detailed description and drawing in which:
[0051] FIG. 1A is Embodiments of the invention will now be
described with reference to the accompanying drawings, in
which:
[0052] FIG. 1A illustrates the lifting steps for the Haar temporal
transform;
[0053] FIG. 1B illustrates a motion adaptive modification of the
lifting steps for the Haar temporal transform;
[0054] FIG. 2 illustrates the lifting steps for a motion adaptive
5/3 temporal transform;
[0055] FIG. 3 illustrates a triangular mesh motion model;
[0056] FIG. 4 illustrates schematically the compositing of two
motion fields at a higher temporal resolution to create one at a
lower resolution; and
[0057] FIG. 5 illustrates schematically the compositing of motion
fields in one embodiment of a temporally scalable motion
representation for the motion adaptive 5/3 lifting transform.
DETAILED DESCRIPTION
1st Aspect: Reciprocal Motion Fields
[0058] While the invention has been described in connection with
what is presently considered to be the most practical and preferred
embodiment, it is to be understood that the invention is not to be
limited to the disclosed embodiments but, on the contrary, is
intended to cover various modifications and equivalent arrangements
included within the spirit and scope of the appended claims, which
scope into be accorded the broadest interpretation, so as to
encompass all such modifications and equivalent structures as is
permitted under the law.
[0059] A natural strategy for estimating the reciprocal motion
fields, W.sub.2k.fwdarw.2k+1 and W.sub.2k+1.fwdarw.2k, would be to
determine the parameters for W.sub.2k.fwdarw.2k+1 which minimise
some measure (e.g., energy) of the mapping residual
x.sub.2k+1-W.sub.2k.fwdarw.2k+1(x.sub.2k) and to separately
determine the parameters for W.sub.2k+1.fwdarw.2k which minimise
some measure of its residual signal,
x.sub.2k-W.sub.2k+1.fwdarw.2k(x.sub.2k+1). In general, such a
procedure will lead to parameters for W.sub.2k.fwdarw.2k+1, which
cannot be deduced from those for W.sub.2k+1.fwdarw.k2 and
vice-versa, so that both sets of parameters must be sent to the
decoder.
[0060] It turns out that only one of the two motion fields must be
directly estimated. The other can then be deduced by "inverting"
the motion field which was actually estimated. Both the compressor
and the decompressor may perform this inversion so that only one
motion field must actually be transmitted.
[0061] True scene motion fields cannot generally be inverted, due
to the presence of occlusions and uncovered background. One would
expect, therefore, to degrade the properties of the motion adaptive
transform (e.g., compression performance, or quality of the low
temporal resolution frames) by replacing W.sub.2k.fwdarw.2k+1 with
an approximate inverse of W.sub.2k+1.fwdarw.2k or vice-versa.
[0062] It turns out, however, that the opposite is the case. Rather
than degrading the transform, representing each reciprocal pair
with only one motion field actually improves the compression
efficiency and the quality of the low temporal resolution
frames.
[0063] An explanation for the above phenomenon is given in. A.
Seeker and D. Taubman, "Lifting-based invertible motion adaptive
transform (LIMAT) framework for highly scalable video compression",
accepted to appear in IEEE Trans. Image Proc., 2003, a copy of
which is available at www.ee.unsw.edu.an/.about.taubman/. Briefly,
the excellent properties of the motion adaptive temporal lifting
transform are closely linked to the reciprocal relationship between
the pairs, W.sub.2k.fwdarw.2k+1 and W.sub.2k+1.fwdarw.2k, and
W.sub.2k.fwdarw.2k-1 and W.sub.2k-1.fwdarw.2k. If the frame warping
operations described by each pair are truly inverses of one
another, the motion adaptive transform is equivalent to a
one-dimensional DWT, applied along the underlying motion
trajectories. If they are not inverses of one another, this
desirable characteristic is lost, no matter how well they are able
to minimise motion compensated residuals.
[0064] According to the first aspect of the present invention, only
one motion field from each reciprocal pair should be directly
estimated and communicated to the decompressor. Unless otherwise
prohibited (e.g., by the later aspects of the invention), it is
mildly preferable to directly estimate and communicate the
parameters of the motion field which is used in the first
(predictive) lifting step. This is the lifting step described by
equations (1) and (3), for the Haar and 5/3 cases,
respectively.
Inversion of Triangular Mesh Motion Models
[0065] Where the motion is represented by a continuously deformable
triangular mesh [Y. Nakaya and H. Harashima, "Motion compensation
based on spatial transformations", IEEE Trans. Circ. Syst. For
Video Tech., vol. 4, pp 339-367, June 1994], the affine motion
which describes the deformation of each triangle in.
W.sub.2k.fwdarw.2k+1 or W.sub.2k.fwdarw.2k-1 may be directly
inverted to recover W.sub.2k+1.fwdarw.2k and W.sub.2k-1.fwdarw.2k,
respectively. A triangular-mesh model for motion field W.sub.k1+k2
involves a collection of node positions, {ti} in the target frame,
x.sub.k2 together with the locations, {si} of those same node
positions, as they appear in the source frame, x.sub.kl. Although
scene adaptive meshes have been described, in the preferred
embodiment of the invention the target node positions, {ti}, are
fixed, and the motion field is parameterized by the set of node
displacements, {si-ti}. The target frame, x.sub.k2, is partitioned
into a collection of disjoint triangles; whose vertices correspond
to the node positions. Since the partition must cover the target
frame, some of the target node positions must lie on the boundaries
of the frame. An example involving a rectangular grid of target
node vertices is shown in FIG. 3.
[0066] As suggested by the figure, it is convenient to write
{.DELTA..sub.j} for the set of target frame triangles. Let
t.sub.j,o, t.sub.j,1 and t.sub.j,2 denote the vertices of target
triangle .DELTA..sub.j. The triangular mesh then maps the source
triangle, .DELTA.'.sub.j, described by the vertices s.sub.j,o,
s.sub.j,2 and s.sub.j,2 onto target triangle .DELTA.'.sub.j, The
motion map itself is described by an affine transformation.
Specifically, for each location, t.epsilon..DELTA..sub.j, within
the target frame, the corresponding location, s, within the source
frame is given by the affine equation
s=A.sub.jt+b.sub.j
where t, s and b.sub.j are regarded as column vectors, A.sub.j is a
2.times.2 matrix; A.sub.j and b.sub.j may be deduced from the
motion parameters, using the fact that t.sub.j,i must map to
s.sub.j,i for each i=0, 1, 2. Of course, s does not generally lie
on an integer grid, and so the source frame must be interpolated,
using any of a number of well-known methods, to recover the value
of (W.sub.kl.fwdarw.k2(x.sub.kl))[t].
[0067] In the simplest case, whenever a target node position,
t.sub.i, lies on the boundary of frame) x.sub.k2, the corresponding
source node position, s.sub.i, is constrained to lie on the same
boundary of frame x.sub.kl, as depicted in FIG. 3. In this case,
the source triangles, .DELTA.'.sub.j, completely cover the source
frame and so each location, s, in frame x.sub.kl, may be associated
with one of the triangles, .DELTA..sup.i.sub.j and hence mapped
back onto the target frame though the inverse affine relation
t=A.sub.j.sup.-1(s-b.sub.j)
[0068] In this way, the value of (W.sub.k2.fwdarw.k1(x.sub.k2))[s]
may be found for each location, s, by interpolating frame x.sub.k2
to the location, t.
[0069] Constraining boundary nodes, t.sub.i, map to nodes, on the
same boundary, tends to produce unrealistic motion fields in the
neighbOurhOod of the frame boundaries, adversely affecting the
ability of the mesh to track true scene motion trajectories. For
this reason, the preferred embodiment of the invention does not
involve any such constraints. In this case, the source triangles,
.DELTA.'.sub.j will not generally cover frame x.sub.k1, and
inversion of the affine transformations yields values for
(W.sub.k2.fwdarw.k1(x.sub.k2))(s) only when s lies within one of
the source triangles, .DELTA.'.sub.j. For locations s which do not
belong to any of the source triangles, .DELTA.'.sub.j, any of a
number of policies may be described. As a simple example, the
nearest source triangle, .DELTA.'.sub.j, to s may be found and its
affine parameters used to find a location t in frame x.sub.k2.
[0070] An alternative approach is to first extrapolate the mesh to
one which is defined over a larger region than that required by the
forward motion field W.sub.k1.fwdarw.k2. So long as this region is
large enough to cover the source frame, each location s in frame
x.sub.k1 will belong to some source triangle within the
extrapolated mesh and the corresponding affine map can be inverted
to find the location t in frame x.sub.k2. In the preferred
embodiment of this approach, the node vector t.sub.e-s.sub.e at
each extrapolated node position n.sub.e in frame x.sub.k2, is
obtained by linear extrapolation of two node vectors,
t.sub.b-s.sub.b and t.sub.o-s.sub.o, having corresponding node
positions n.sub.b and n.sub.o. Here, the extrapolated node position
n.sub.e is outside the boundaries of frame x.sub.k2, n.sub.b is the
location of the nearest boundary node to n.sub.e, and
n.sub.o=2n.sub.b-n.sub.e is the mirror image of n, through the
boundary node, n.sub.b. The extrapolated node vectors are not
explicitly communicated to the decoder, since it extrapolates them
from the available interior node positions, following the same
procedure as the encoder.
"Inversion" of Block-Displacement Motion Models
[0071] Triangular mesh models are particularly suitable for the
recovery of a reverse motion field, W.sub.k2.fwdarw.k1, from its
forward counterpart W.sub.k1.fwdarw.k2. Most significantly, the
transformation between target locations, t, and source locations,
s, is continuous over the whole of the target frame. This is a
consequence of the fact that the affine transformation maps
straight lines to straight lines.
[0072] Block displacement models, however, are more popular for
video compression due to their relative computational simplicity. A
block displacement model consists of a partition of the target
frame into blocks, {B.sub.i}, and a corresponding set of
displacements, {.delta..sub.i}, identifying the locations of each
block within the source frame.
[0073] Unlike the triangular mesh, block displacement models
represent the motion field in a discontinuous (piecewise constant)
manner. As a result, they may not properly be inverted.
Nevertheless, when reciprocal pairs of motion maps,
W.sub.k1.fwdarw.k2 and W.sub.k2.fwdarw.k1, use block displacement
models, it is still preferable to estimate and transmit only one of
the two motion fields to the decoder, inferring the other through
an approximate inverse relationship. Since displacements are
usually small, it is often sufficient simply to reverse the sign of
the displacement vectors, {.delta..sub.i}, when forming
W.sub.k2.fwdarw.k1 from W.sub.k1.fwdarw.k2 or vice-versa.
2nd Aspect: Compositing of Simple Motion Fields.
[0074] For high energy compaction and low temporal resolution
frames with high fidelity, it is essential to have accurate motion
mappings for each level of a multi-resolution temporal subband
decomposition. The transform consists of a sequence of stages, each
of which produces a low- and a high-pass temporal subband sequence,
from its input sequence. Each stage in the temporal decomposition
is applied to the low-pass subband sequence produced by the
previous stage.
[0075] Since each stage of the temporal decomposition involves the
same steps, one might consider applying an identical estimation
strategy within each stage, estimating the relevant motion fields
from the frame sequence which appears at the input to that stage.
The problem with such a strategy is that estimation of the true
motion, based on subband frames, may be hampered by the existence
of unwanted artefacts such as ghosting. Such artefacts can arise as
a result of model failure or poor motion estimation, in previous
stages of the decomposition.
[0076] To avoid this difficulty, it is preferred to perform motion
estimation on the appropriate original frames instead of the input
frames to the decomposition stage in question. For example, in the
second stage of temporal decomposition it is more effective to
estimate the motion mapping W.sup.k1.fwdarw.k2.sup.(1) between
subband frames l.sub.k1.sub.(1)[n] and l.sub.k2.sub.(2)[n], by
using the corresponding original frames x.sub.k1[n] and
x.sub.k2[n]. Similarly, in the third stage, it is more effective to
estimate the motion mapping W.sub.k1.fwdarw.k2.sub.(2) between
subband frames l.sub.k1.sub.(2).sub.[n] and
l.sub.k2.sub.(2).sub.[n], by using the corresponding original
frames x.sub.4k1[n] and x.sub.4k2[n]. To clarify the notation being
used here, it is noted that the first stage of decomposition
employs motion mappings W.sub.k1.fwdarw.k2.sub.(o), producing low
and high-pass subband frames, l.sub.k.sub.(1)[n] and
h.sub.k.sub.(1)[n]
[0077] After several levels of subband decomposition, the temporal
displacement over which motion estimation must be performed will,
span many original frames. For example, in the fifth level of
decomposition the actual temporal displacement between neighbouring
subband frames is 16 times the original frame displacement. At a
typical frame rate of 30 frames per second (fps), this corresponds
to more than half a second of video.
[0078] Motion estimation is generally very difficult over large
temporal displacements due to the large possible-range of motion.
This complexity can be reduced by using knowledge of motion
mappings already obtained in previous levels of the decomposition.
For example, as described by equations (3) and (4), the first stage
of decomposition with the 5/3 kernel involves estimation of
W.sub.2k.fwdarw.2k+1.sub.(o) and W.sub.2k+2.fwdarw.2k+1.sub.(o).
These may be composited to form an initial approximation for
W.sub.k.fwdarw.k+1.sub.(1), which is required for the second stage
of decomposition. This is shown in FIG. 4, where the arrows
indicate the direction of the motion mapping. It is often
computationally simpler to create composite mappings from source
mappings that have the same temporal orientation, as suggested in
the figure. If necessary, the source mappings can be inverted to
achieve this. However, it is preferable to directly estimate source
mappings, having the same direction as the composite mapping.
[0079] The initial approximation, formed by motion, field
composition in the manner described above, can be refined based on
original video data, using motion estimation procedures well known
to those skilled in the art. It turns out, however, that the method
of compositing motion fields with a frame displacement of 1 to
produce motion fields corresponding to larger frame displacements
often produces highly accurate motion mappings; that do not need
any refinement. In some cases the composit mappings lead to
superior motion adaptive transforms than motion mappings formed by
direct estimation, or with the aid of refinement steps. The motion
field composition method described, here can be repeated throughout
the temporal decomposition hierarchy so that all the mappings for
the entire transform can be derived from the frame to frame motion
fields estimated in the first stage.
[0080] The composition method described above eliminates a
significant portion of the computational load associated with
direct estimation of the required motion fields. A total of one
motion mapping must be estimated for each original frame, having a
temporal displacement of only one frame. This is sufficient to
determine the complete set of motion mappings for the entire
transform.
[0081] This method is independent of the particular wavelet kernel
on which the lifting framework is based; however, the effectiveness
of the composition procedure does depend on the selected motion
model. An efficient method for performing the composition procedure
is described in the 4th aspect of this invention.
3rd Aspect: Efficient Temporally Scalable Motion Representation
[0082] An efficient temporally scalable motion representation
should satisfy two requirements. Firstly, at most one motion
mapping per video frame should be needed to reconstruct the video
at any temporal resolution. This is consistent with the above
observation that just one mapping per frame is sufficient to derive
all mappings for the entire transform.
[0083] Secondly, the above property should apply at each temporal
resolution available from the transform. In particular, this means
that the motion information must be temporally embedded, with each
successively higher temporal resolution requiring one extra motion
mapping per pair of reconstructed video frames. This property
allows the video content to be reconstructed at each available
temporal resolution, without recourse to redundant motion
information.
[0084] This aspect of the invention involves a temporally scalable
motion information hierarchy, based on the method of motion field
composition, as introduced in the description of the second aspect.
This representation achieves both of the objectives mentioned
above.
[0085] The motion information hierarchy described here is
particularly important for motion adaptive lifting structures that
are based on kernels longer than the simple Haar. Block transforms
such as the Haar require only the motion information between every
second pair of consecutive frames, at each stage of the
decomposition. Therefore an efficient temporally scalable motion
representation can be easily achieved by transmitting a single
motion mapping for every reciprocal pair.
[0086] It is generally preferable to use longer wavelet kernels
such as the 5/3. In fact, results given in A. Seeker and D.
Taubman, "Lifting-based invertible motion adaptive transform
(LIMAT) framework for highly scalable video compression", (accepted
to appear in IEEE Trans. Image Proc., 2003) reveal that this can
lead to considerable improvements in performance.
[0087] The motion representation for two stages of the 5/3
transform is given in FIG. 5. The mappings required to perform the
lifting steps are again shown as arrows, where the i.sup.th forward
mapping in the j.sup.th transform level is denoted F.sub.i.sup.j.
The term "forward mapping" is applied to those which approximate a
current frame by warping a previous frame. Likewise, backward
mappings, denoted B.sub.i.sup.j, correspond to warping a later
frame to spatially align it with a current frame. Observe that the
entire set of motion mappings depicted in FIG. 5 can be represented
using only F.sub.1.sub.2.sup.2 and B.sub.2.sub.1. Inverting
F.sub.1.sub.2 produces the backward mapping B.sub.1.sub.2. The
forward mapping F.sub.1.sub.1 is inferred by compositing the
upper-level forward mapping F.sup.1.sup.2 with the lower-level
backward mapping B.sub.2.sup.1. The remaining mappings
B.sub.2.sup.1 and F.sub.2.sup.1 are recovered by inverting,
F.sub.1.sup.1 and B.sup.2.sup.2, respectively.
[0088] For scenes with rapid motion, composited fields such as
F.sub.1.sup.1 in FIG. 5, may suffer from an accumulation of the
model failure regions present in the individual mappings. If so,
the compressor may correct this by transmitting an optional
refinement fields, possibly based on direct estimation using
original data.
[0089] As mentioned, the case for the Haar wavelet is much simpler.
Mappings F.sub.2.sup.1 Land B.sub.2.sup.1 are not required, so it
is sufficient to code mappings F.sub.1.sup.1 and F.sub.1.sup.2,
recovering the corresponding backward motion fields by inversion.
The methods described above can be applied recursively to any
number of transform stages, and the total number of required
mappings is upper bounded by one per original frame. Temporal
scalability is achieved since reversing a subset of the temporal
decomposition stages requires no motion information from higher
resolution levels.
[0090] Evidently, a motion mapping between any pair of frames can
be obtained by a combination of composition and inversion operators
involving the sequence of mappings F.sub.1.sup.2 and
B.sub.2i.sup.1. It follows that this motion representation strategy
is easily modified to encompass any wavelet kernel.
4th Aspect: Efficient Implementation of Motion Field
Transformations
[0091] A 4th aspect of the present invention describes an efficient
method for performing the motion field composition and inversion
transformations mentioned in previous aspects.
[0092] One possible way to represent a composited mapping is in
terms of a sequence of warpings through each individual mapping.
Motion compensation could be performed by warping the actual data
through each mapping in turn. However, this approach suffers from
the accumulation of spatial aliasing and other distortions that
typically accompany each warping, step.
[0093] A second problem with this approach is that errors due to
boundary approximations also accumulate over the sequence of
mappings. Boundary regions are prone to model failure, particularly
when the scene undergoes global motion such as camera panning.
[0094] To avoid these problems, each location in the target frame
of the composite motion field may be mapped back through the
various individual mappings to find its location in the source
frame of the composite motion field.
[0095] The preferred method, described here, however, is to
construct a triangular mesh model for the composit motion field,
deducing the displacements of the mesh node points by projecting
them through the various component motion mappings. The triangular
mesh model provides a continuous interpolation of the projected
node positions and can be represented compactly in internal memory
buffers. This method is particularly advantageous when used in
conjunction with triangular mesh models for all of the individual
motion mappings, since the frame warping machinery required to
perform the motion adaptive temporal transformation involves only
one type of operation--the affine transformation described
previously.
[0096] Motion field inversion, may be performed using a similar
strategy. The inverted motion mapping is represented using a
forward triangular mesh motion model, whose node displacements are
first found by tracing them through the inverse motion field. The
accuracy associated with both composit and inverse motion-fields
representations may be adjusted by modifying the size of the
triangular mesh grid. In the preferred embodiment of the invention,
the mesh node spacing used for representing composit and inverse
motion fields is no larger 8 frame pixels and no smaller than 4
frame pixels.
5th Aspect: Successive Refinement of Motion and Sample Accuracy
[0097] In order to provide for scalable video bit-streams which
span a wide range of bit-rates, from a few 10's of kilo-bits/s
(kb/s) to 10's of mega-bits/s (Mb/s), the accuracy with which
motion information is represented must also be scaled. Otherwise,
the cost of coding motion information would consume an undue
proportion (all or more) of the overall bit budget at low bit-rates
and would be insufficient to provide significant coding gain at
high bit-rates. In the 3rd aspect above, a method for providing
temporally scalable motion information has been described. In this
5th aspect, a method is described for further scaling the cost of
motion information, in a manner which is sensitive to both the
accuracy and the spatial resolution required of the reconstructed
video sequence.
[0098] During compression, an accurate motion representation is
determined and used to adapt the various lifting steps in the
motion adaptive transform. During decompression, however, it is not
necessary to receive exactly the same motion parameters which were
used during compression. The motion parameters are encoded using an
embedded quantisation and coding strategy. Such strategies are now
well known to those skilled in the art, being employed in scalable
image and video codecs such as those described in J. Shapiro,
"Embedded image coding using zerotrees of wavelet coefficients",
IEEE Trans. Sig. Proc., vol 41, pp 3445-3462, December 1993., D.
Taubman and A. Zakhor, "Multi-rate 3-d subband coding of video",
IEEE Trans. Image Proc., vol. 3, pp. 572-588, September 1994, A.
Said and W. Pearlman, "A new, fast and efficient image codee based
on set partitioning in hierarchical trees", IEEE Trans. Circ. Syst.
For Video tech., pp. 243-250, June 1996, D. Taubman, "High
performance scalable image compression with EBCOT", IEEE Trans.
Image Proc., vol. 9, pp. 1158-1170, July 2000. They allow the coded
bit-stream to provide a successively more accurate representation
of the information being coded. For the present purposes, this
information consists of the motion parameters themselves, and each
motion field, W.sub.k1.fwdarw.k2, is provided with its own embedded
bit-stream.
[0099] As an example of the way in which such an embedded motion
representation may be used, consider an interactive client-server
application, in which the client requests information for the video
at some particular spatial resolution and temporal resolution
(frame rate). Based on this information, the server determines the
distortion which will be introduced by approximating the relevant
motion information with only L.sub.q.sup.(M) bits from the
respective embedded bit-streams, where the available values for
L.sub.q.sup.(M) are determined by the particular embedded
quantisation and coding strategy which has been used. Let
D.sub.q.sup.(M) denote this distortion, measured in, terms of Mean
Squared Error (MSE), or a visually weighted MSE. The values
D.sub.q.sup.(M) may be estimated from the spatial-frequency power
spectrum of the relevant, frames. Most notably, D.sub.q.sup.(M)
depends not only on the accuracy with which the motion parameters
are represented by the L.sub.q.sup.(M) bits of embedded motion
information, but also on the spatial resolution of interest. At
lower spatial resolutions, less accuracy is required for the motion
information, since the magnitude of the phase shifts associated
with motion error are directly proportional to spatial
frequency.
[0100] Continuing the example, above, the server would also
estimate or know the distortion, D.sub.p.sup.(s), associated with
the first L.sub.p.sup.(s) bits of the embedded representation
generated during scalable coding of the sample values produced by
the motion adaptive transform. As already noted, scalable sample
data compression schemes are well known to those skilled in the art
Assuming an additive model for these two different distortion
contributions, the server balances the amount of information
delivered for the motion and sample data components, following the
usual Lagrangian policy. Specifically, given a total budget of
L.sup.max bits for both components, deduced from estimates of the
network transport rate, or by any other means the server finds the
largest values of p.sub..lamda. and q.sub..lamda. such that
- .DELTA. D p .lamda. ( S ) .DELTA. L p .lamda. ( S ) .gtoreq.
.lamda. and - .DELTA. D q .lamda. ( M ) .DELTA. L q .lamda. ( M )
.gtoreq. .lamda. . ( 5 ) ##EQU00006##
adjusting .lamda.>0 so that
L.sub.p.lamda..sup.(s)+L.sub.q.lamda..sup.(M) is as large as
possible, while not exceeding L.sup.max. Here,
.DELTA.D.sub.p.sup.(s)/.DELTA.L.sub.p.sup.(s) and
.DELTA.D.sup.q.sup.(M)/.DELTA.L.sub.q.sup.(M) are discrete
approximations to the distortion-length slope at the embedded
truncation points p (for sample data) and q (for motion data)
respectively.
[0101] The client-server application described above is only an
example. Similar techniques may be used to construct scalable
compressed video files which contain an embedded hierarchy of
progressively higher quality video, each level in the hierarchy
having its own balance between the amount of information
contributed from the embedded motion representation and the
embedded sample data representation.
[0102] The strategy described above, whereby an embedded motion
representation is produced by embedded quantisation and coding of
the individual motion parameters, may be extended to include
progressive refinement according to the density of the motion
parameters themselves. To see how this works, suppose that every
second row and every second column were dropped from the
rectangular grid of node positions in the triangular mesh of FIG.
3. In this coarse mesh, motion parameters would be sent only for
the remaining node positions and the coarse triangular mesh model
induced by this information would represent a coarse approximation
to the original motion model. Such approximations are readily
included within an embedded motion representation, from which an
appropriate distribution between the motion and sample data coding
costs may again be formed.
[0103] While rate-distortion optimisation strategies have
previously been described in the literature for balancing the costs
of motion and sample data information, this has not previously been
done in a scalable setting, where both the motion and the sample
data accuracy are progressively refined together.
[0104] While rate-distortion optimization strategies have
previously been de scribed in the literature for balancing the
costs of motion and sample data information, this has not
previously been done in a scalable setting, where both the motion
and the sample data accuracy are progressively refined together.
There are, in our opinion, two principle reasons why progressively
refined motion fields have not been investigated in the past for
video compression. Firstly, most existing video compression systems
(e.g., those described by international standards) employ motion
compensated predictive coding, so if the decoder were to use
different motion parameters to the encoder, their states would
progressively drift apart. This problem does not exist in the
context of motion adaptive wavelet transforms and, in particular,
those based on the motion compensated lifting paradigm taught in
W002/50772.
[0105] The second reason, why we believe others have not
investigated progressively refillable motion for scalable video
coding is that the motion information interacts in a complex manner
with the video sample data, making it more difficult to deduce the
impact of motion quantization on system performance. The invention
disclosed here, however, is inspired by the following interesting
observation. Although the interaction between motion errors and
video sample data errors is generally complex, at all
experimentally optimal combinations of the motion and sample data
accuracy, this relationship simplifies and may be approximately
modeled using linear methods. In the ensuing sub-sections, we teach
some specific methods for scalable motion coding and for optimally
balancing the distribution of motion information with video sample
information.
Scalable Motion Coding Methods
[0106] As mentioned above, a variety of methods for embedded coding
of data are well known to those skilled in the art. Amongst these
various methods, the authors' experimental investigations have
suggested particular preferred embodiments. Rather than coding the
motion parameters directly, it is preferable to first subject the
motion parameter fields to a spatial discrete wavelet transform
(DWT). That is, the horizontal components of each motion vector are
treated as a two dimensional image and the vertical components are
similarly treated as a two dimensional image; each image is
subjected to a spatial DWT and the transform coefficients are then
coded in place of the original motion vectors.
[0107] The use of a spatial wavelet transform is found to offer two
chief benefits over coding the motion parameters directly. Firstly,
the transform typically produces a large number, of near-zero
valued coefficients which can be quantized to zero with negligible
error and then efficiently encoded. Secondly, the DWT shapes the
quantization errors incurred when the motion representation is
scaled, and this shaping is found to significantly reduce the
reconstructed video distortion incurred at any given level of
motion quantization error. In the preferred embodiment, a
reversible (integer-to-integer) spatial DWT is used to allow exact
recovery of the originally estimated motion parameters from the
encoded transform coefficients, which is useful at high video
bit-rates. Reversible wavelet transforms are well-known to those
skilled in the art. One example is the reversible 5/3 spatial DWT
which forms part of the JPEG2000 image compression standard, IS
15444-1.
[0108] Temporal transformation of the motion parameter information
can have similar benefits to spatial transformation, and the
effects are found to be somewhat complementary. That is, both the
use of both a spatial DWT and a temporal transform together is
recommended. In one particular embodiment, each pair of temporally
adjacent motion fields is replaced by the sum and the difference of
the corresponding motion vectors. These sums and differences may be
interpreted as temporal low- and high-pass subbands.
[0109] Again, it is preferable to do this in a reversible manner
which is compatible with efficient lossless coding, since at high
video bit-rates it is best to preserve all of the estimated motion
information. For this reason, the operations of sum and difference
mentioned above should be replaced by the S-transform [V. I-leer
and H. E. Reinfelder, "A comparison of reversible methods for data
compression", Proc. SPIE conference, `Medical Imaging IV`, vol
1233, pp. 354-365, 1990].
[0110] As for the coding of motion transform coefficients, the
preferred embodiments are those which use techniques derived from
the general class of bit-plane coders. In particular, the highly
efficient and finely embedded fractional bit-plane coding
techniques which form part of the JPEG2000 image compression
standard are to be recommended. In general, each subband produced
by the motion parameter transform is partitioned into code-blocks,
and each code-block is encoded using a fractional bit-plane coder,
producing a separate finely embedded bit-stream for each motion
subband code-block.
[0111] In many cases, there are insufficient motion parameters to
justify dividing motion subbands into multiple code-blocks, but the
code-block partitioning principles enshrined in the JPEG2000
standard can be useful when compressing very large video frames,
each of which has a large number of motion vectors. It general,
then, the motion information is represented by a collection of
code-blocks; each of which has a finely embedded bit-stream which
may be truncated to any of a variety of coded lengths.
A Layered Framework for Joint Scaling, of Motion and Video Sample
Data
[0112] The EBCOT algorithm [D. Taubman, "High performance scalable
image compression with EBCOT", IEEE Trans. Image Proc., vol. 9, pp.
1158-1170, July 2000] represents an excellent framework for
converting a large number of embedded code-block bit-streams, each
with its own set of truncation points, into a global collection of
abstract "quality" layers. Each quality layer contains incremental
contributions from each code-blocks embedded bit-stream, where
these contributions are balanced in a manner which minimises the
distortion associated with the overall representation at the total
bit-rates associated with the quality layer. By arranging the
quality layers in sequence, one obtains a succession of truncation
points, at each of which the representation is as accurate as it
can be, relative to the size of the included quality layers.
[0113] Although the interaction between motion errors and video
sample errors is non-trivial, it turns out that for combinations of
motion and video sample bit-rates which are optimal, the
relationship between motion error and reconstructed video quality
is approximately linear. We may represent this linear relationship
as
D.sub.x,M.apprxeq..PSI..sub.R,SD.sub.M
[0114] where D.sub.M denotes mean squared error in the motion
vectors due to truncation of the embedded motion parameter
code-block bit-streams, and D.sub.x,M represents the total induced
squared error in the reconstructed video sequence. The scaling
factor, .PSI..sub.R,S, depends upon the spatial resolution at which
the video signal is to be reconstructed and also upon the accuracy
with which the video samples are represented. In preferred
embodiments of the present invention, motion parameter quality
layers are constructed from the embedded motion block bit-streams,
following the EBCOT paradigm.
[0115] In view of the above relationship, and noting that the
scaling factor, .PSI..sub.R,S, is substantially similar for all
motion coefficient subbands and code-blocks, the rate-distortion
optimality of the layered motion representation holds over a wide
range of spatial resolutions and levels of video sample
quantization error. This is extremely convenient, since it means
that the rate-distortion optimization problem expressed in equation
(5) can be solved once, while constructing the motion quality
layers, after which a video server or transcoder need only decide
how many motion layers are to be included in the video bit-stream
for a given spatial resolution and a given level of error in the
video sample data.
[0116] In preferred embodiments of the invention, the same layering
strategy of EBCOT is used to construct a separate set of
rate-distortion optimal quality layers for the video sample data.
These are obtained by subjecting the temporal subbands produced by
the motion-compensated temporal lifting steps to spatial wavelet
transform, partitioning the spatio-temporal video subbands into
their own code-blocks, and generating, embedded bit-streams for
each video sample code-block. The video sample quality layers then
consist of incremental contributions from the various video sample
code-blocks, such at the video sample distortion is as small as it
can be, relative to the total size of those quality layers. It
turns out, most conveniently, that the generation of
rate-distortion optimal video sample quality layers is
substantially independent of the spatial resolution (number of
resolution levels from the spatial video sample DWT which will be
sent to the decoder) and the temporal resolution (number of
temporal subbands produced by the motion compensated lifting steps
which will be sent to the decoder). It also turns out that the
optimality of the layer boundaries is approximately independent of
the level of motion distortion, at least for combinations of motion
and video sample bit-rates which are approximately optimal.
[0117] In summary, preferred embodiments of the invention produce a
single set of motion quality layers and a single set of video
sample quality layers. The layers are internally rate-distortion
optimal over the temporal interval within which they are formed.
Since video streams can have unbounded duration, we divide the time
scale into epochs known as "frame slots" In each frame slot, a
separate set of motion quality layers and video sample quality
layers is formed. The optimization problem associated with equation
(5) then reduces to that of balancing the number of motion quality
layers with the number of video sample quality layers which are
sent to a decoder within each frame slot. The solution to this
problem is dealt with below, but we note that it depends on the
parameter .PSI..sub.R,S which is a function of both the spatial,
resolution of interest to the decoder and the accuracy of the video
sample data. Equivalently, for any particular number of video
sample layers, p, the number of motion layers, q, which balances
the rate-distortion slopes of the motion and video sample
information is a function of both p and the spatial resolution of
interest.
Methods for Optimizing the Distribution of Motion and Video Sample
Data
[0118] In view of the preceding discussion, a complete
implementation of the preferred embodiment of the invention must
provide a means for deciding how many motion quality layers, q, are
to be included with a subset of the video bit-stream which includes
p video sample quality layers, given the spatial resolution R, at
which the video content is to be viewed. The preferred way to do
this is to include a collection of tables with each frame slot,
there being one table per spatial resolution which may be of
interest, where each table provides an entry for each number of
video sample quality layers, p, identifying the corresponding best
number of motion layers, q.sub.p. Depending upon the application,
there may be no need to send the table itself to a decoder.
[0119] A video server or transcoder, needing to meet a compressed
length constraint L.sub.max Within each frame slot, can use these
tables to determine p and q.sub.p which are jointly optimal, such
that the total length of the respective quality layers is as small
as possible, but no smaller than L.sub.max. It is then preferable
to discard data from the p.sup.th video sample quality layer, until
the length target L.sub.max is satisfied. This approach is
preferable to that of discarding motion data, since there is
generally more video sample data. One way to build the
aforementioned tables is to simply decompress the video at each
spatial resolution, using each combination of motion and sample
quality layers, q and p, so as to find the value of p.sub.q which
maximizes the ratio of distortion to total bit-rate in each frame
slot, for each p. Of course, this can be computationally expensive.
Nevertheless, this brute-force search strategy is computationally
feasible.
[0120] A preferred means to build the aforementioned tables is to
use the fact that these tables depend only on the linear scaling
factors, .PSI..sub.R,S. These scaling factors depend, in turn, on
the power spectra of the video frames which are reconstructed at
each level of video sample error, i.e., at each video sample
quality layer p. In the preferred embodiment of the invention,
these power spectra are estimated directly from the video sample
subband data during the compression process. We find, in practice,
that such estimation strategies can produce results almost as good
as the brute force search method described above, at a fraction of
the computational cost.
* * * * *
References