U.S. patent application number 12/543414 was filed with the patent office on 2010-12-23 for watermark placement in watermarking of time varying media signals.
Invention is credited to Mehmet U. Celik.
Application Number | 20100322463 12/543414 |
Document ID | / |
Family ID | 31720261 |
Filed Date | 2010-12-23 |
United States Patent
Application |
20100322463 |
Kind Code |
A1 |
Celik; Mehmet U. |
December 23, 2010 |
Watermark Placement in Watermarking of Time Varying Media
Signals
Abstract
A method of placing a watermark in a video stream estimates
motion between frames in the video stream, and computes a
representative motion for a frame. Before embedding the watermark
in that frame, the method spatially adjusts a digital watermark by
the representative motion. This method is particularly suited for
embedding a watermark in a video stream compressed using motion
estimation. In this case, the method uses the motion vectors in the
video stream to compute a dominant motion for a frame, and then
shifts the watermark by this dominant motion before embedding it in
the frame.
Inventors: |
Celik; Mehmet U.;
(Rochester, NY) |
Correspondence
Address: |
DIGIMARC CORPORATION
9405 SW GEMINI DRIVE
BEAVERTON
OR
97008
US
|
Family ID: |
31720261 |
Appl. No.: |
12/543414 |
Filed: |
August 18, 2009 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
10300921 |
Nov 19, 2002 |
7577841 |
|
|
12543414 |
|
|
|
|
60404038 |
Aug 15, 2002 |
|
|
|
Current U.S.
Class: |
382/100 |
Current CPC
Class: |
H04N 19/537 20141101;
H04N 19/61 20141101; G06T 2201/0053 20130101; G06T 1/0085 20130101;
H04N 19/51 20141101; H04N 19/467 20141101 |
Class at
Publication: |
382/100 |
International
Class: |
G06K 9/00 20060101
G06K009/00 |
Claims
1. A system for placing a watermark in a video stream comprising:
memory; a programmed computer in communication with the memory, the
programmed computer for estimating motion between frames in the
video stream, for computing a representative motion for at least
part of a frame to be embedded with a digital watermark, and for
spatially adjusting the digital watermark by the representative
motion, wherein the representative motion is derived from motion in
the video over time, and the representative motion used to
spatially adjust digital watermark is dependent on watermark
energy.
2. The system of claim 1 wherein the motion is derived from motion
data in a compressed video stream.
3. The system of claim 2 wherein the motion data comprises motion
vectors.
4. The system of claim 1 wherein spatially adjusting the digital
watermark comprises shifting the digital watermark in a direction
specified by the representative motion.
5. The system of claim 1 wherein the representative motion
comprises a dominant motion in the frame.
6. The system of claim 1 wherein the representative motion is
derived from statistics of motion data from a plurality of
locations in the frame.
7. The system of claim 1 wherein the spatially adjusted watermark
is embedded into the video such that the watermark is robust to the
spatial adjustment.
8. The system of claim 1 wherein the memory comprises a computer
readable medium having instructions executed by the programmable
computer.
9. A method of placing a watermark in a compressed data stream
comprised of frames of time varying data, where the data stream
includes frames compressed using motion estimation, the method
comprising: using a programmed computer to perform the acts of:
computing a representative motion for a frame based on motion data
in the compressed data stream; and spatially adjusting a digital
watermark to be embedded in the frame by the representative motion
of the frame; wherein the representative motion is derived from
motion in a frame, and the motion used to derive the representative
motion is weighted based on the watermark energy.
10. The method of claim 9 wherein the motion data comprises motion
vectors.
11. The method of claim 9 wherein spatially adjusting comprises
shifting the digital watermark.
12. The method of claim 9 wherein the representative motion
comprises a dominant motion derived from motion vector
statistics.
13. The method of claim 9 wherein the spatially adjusted watermark
is embedded into video frames so that the watermark is robust to
spatial adjustment and is detectable in frames of the video despite
spatial adjustment of the video.
14. The method of claim 9 wherein the representative motion is
derived from motion of blocks in a frame, and the motion of the
blocks used to derive the representative motion is weighted based
on the watermark energy to be carried in the blocks such that
blocks carrying more watermark signal are weighted more
heavily.
15. A computer readable storage medium on which is stored
instructions, which when executed by a computer, perform the method
of claim 9.
16. A device for placing a watermark in a compressed video data
stream comprised of frames of time varying data, where the data
stream includes frames compressed using motion estimation, the
device comprising: memory; a programmable computer in communication
with the memory, the programmable computer for extracting motion
vectors for a frame in the compressed video data stream, for
computing a representative motion for at least one part of the
frame based on the motion vectors in the compressed data stream,
and for spatially adjusting a digital watermark by the
representative motion before embedding the adjusted watermark in
the part of the frame; wherein the motion vectors are weighted
according to an amount of watermark energy to be embedded in blocks
of the video frame corresponding to the motion vectors.
17. The device of claim 16 wherein the representative motion is
derived from a statistical distribution of motion vectors for the
frame.
18. The device of claim 17 where a single, dominant motion is
computed for the part of the frame where the spatially adjusted
watermark is to be embedded.
19. The device of claim 16 wherein the motion vectors are weighted
according to an amount of watermark energy to be embedded in blocks
of the video frame corresponding to the motion vectors.
Description
RELATED APPLICATION DATA
[0001] This application is a continuation of U.S. application Ser.
No. 10/300,921, filed Nov. 19, 2002, which claims benefit of U.S.
Provisional Application 60/404,038 filed Aug. 15, 2002, which is
hereby incorporated by reference.
TECHNICAL FIELD
[0002] The invention relates to digital watermarking.
BACKGROUND AND SUMMARY
[0003] Digital watermarking is a process for modifying physical or
electronic media to embed a hidden machine-readable code into the
media. The media may be modified such that the embedded code is
imperceptible or nearly imperceptible to the user, yet may be
detected through an automated detection process. Most commonly,
digital watermarking is applied to media signals such as images,
audio signals, and video signals. However, it may also be applied
to other types of media objects, including documents (e.g., through
line, word or character shifting), software, multi-dimensional
graphics models, and surface textures of objects.
[0004] Digital watermarking systems typically have two primary
components: an encoder that embeds the watermark in a host media
signal, and a decoder that detects and reads the embedded watermark
from a signal suspected of containing a watermark (a suspect
signal). The encoder embeds a watermark by subtly altering the host
media signal. The reading component analyzes a suspect signal to
detect whether a watermark is present. In applications where the
watermark encodes information, the reader extracts this information
from the detected watermark.
[0005] Several particular watermarking techniques have been
developed. The reader is presumed to be familiar with the
literature in this field. Particular techniques for embedding and
detecting imperceptible watermarks in media signals are detailed in
the assignee's co-pending application Ser. No. 09/503,881 and U.S.
Pat. No. 6,122,403, which are hereby incorporated by reference.
[0006] These watermarking techniques can be applied to embed
auxiliary data in the compressed domain of multimedia signals. One
challenge, however, is that some compression methods use a
technique called "motion compensation" that may interfere with the
watermark signal or create artifacts in the host signal in which
the watermark is embedded. Motion compensation refers to a process
often used in compression where a signal from a particular frame is
predicted from one or more other frames. This technique enhances
compression efficiency because the compressed version need only
include a "key" frame (called the I frame), and predicted frames,
coded as the difference between the frame and a predicted version
of that frame.
[0007] "Drift compensation" refers to a method that prevents
accumulation of visual distortions due to additive watermarking in
compression schemes with motion compensation. Unless compensated
properly, watermark error propagates both temporally and spatially
as a result of motion compensation. Propagation of watermark signal
to the consecutive frames has two potential disadvantages: it
degrades the visual quality, and may interfere (most probably
deconstructive interference) with the consecutive watermark signals
reducing detection performance.
[0008] The invention provides a methods, systems and devices for
enhancing watermark embedding in time varying signals. One
particular implementation applies to digital watermarking of
compressed video stream in the MPEG compression format. However,
the invention is not limited to a particular compression format.
For example, the method may be applied to watermark embedding of
video in the spatial domain and uncompressed video.
[0009] One aspect of the invention is a system of placing a
watermark in a video stream. The system estimates motion between
frames in the video stream, and computes a representative motion
for a frame. The system spatially adjusts a digital watermark by
the representative motion before embedding it in the frame. This
approach is particularly suited for embedding a watermark in a
video stream compressed using motion estimation. The system can use
the motion data in the video stream to compute a dominant motion
for a frame, and then shift the watermark by this dominant motion
before embedding it in the frame. Additional aspects of the
invention include methods executed by programmable computing
devices, and device implementations.
[0010] Further features will become apparent with reference to the
following detailed description and accompanying drawings.
BRIEF DESCRIPTION OF THE DRAWINGS
[0011] FIG. 1 is a flow diagram illustrating an overview of method
of placing a watermark in a time varying signal.
[0012] FIG. 2 is a flow diagram of a particular implementation of
the method illustrated in FIG. 1.
DETAILED DESCRIPTION
[0013] This section describes a method for placing a watermark in
compressed video. The following assumptions are made as a context
for this method: [0014] 1. Though the method is generally
applicable to other compression formats, this example focuses on an
MPEG video sequence with Group of Picture (GOP) structure
comprising frames I . . . P . . . P . . . P . . . P . . . . At this
moment, we assume that B-VOP's are skipped during watermarking if
and when they exist in the GOP structure. [0015] 2. Watermark
detection is done per frame basis and the detection algorithm is
robust or invariant to spatial shifts. (Many watermarking
algorithms fit in this description.) Again this is not intended to
be a limitation on the watermark placement method. [0016] 3. In the
spatial domain, the watermark signal has white noise
characteristics. In particular, spatial shifts of the watermark
signal are uncorrelated. This assumption can be relaxed and a
limited correlation may be allowed. [0017] 4. In temporal the
direction, the watermark signal or at least a part of the signal
remains constant. For instance, the same watermark signal is
embedded in all frames, or a component of the watermark, e.g. a
template for synchronization, remains constant throughout the
frames. This assumption is applied for a GOP. An all-different
watermark may be used in different GOPs in the video stream. [0018]
5. During modification of the video samples (e.g., DCT-coefficients
in the compressed stream) to superimpose the watermark, the
resulting change in the bit-rate of the compressed video signal is
proportional to the energy of the additive signal, e.g., watermark.
Equivalently, under a fixed, or limited, bit-rate constraint, the
energy of the additive signal is constant. Again, this is not
intended as a limitation on the general method, yet applies to the
example implementation described in this section.
[0019] In general, a watermarking process with drift compensation
may be represented by the following equations. Superscript denotes
the frame number. The letter I refers to an intra-coded Video
Object Plane (VOP), and P refers to an inter-coded VOP with forward
prediction.
I.sup.k.sub.w=I.sup.k+W.sup.k
P.sup.k+1.sub.w=P.sup.k+1+W.sup.k+1-D.sup.k,
Where D.sup.k is the drift compensation component, which is a
motion compensated version of the cumulative error induced on the
previous frame, i.e. the difference between decoded original and
watermarked frames.
[0020] After drift compensation, each watermarked and reconstructed
frame will only differ from the original reconstructed frame by the
watermark amount.
R.sup.k=I.sup.k
R.sup.k+1=R.sup.k.sub.MC+P.sup.k+1=I.sup.k.sub.MC+P.sup.k+1
R.sup.k=I.sup.k+W.sup.k
R.sup.k+1=R.sup.k.sub.MC+P.sup.k+1+W.sup.k+1-D.sup.k
=R.sup.k.sub.MC+W.sup.k.sub.MC+P.sup.k+1+W.sup.k+1-W.sup.k.sub.MC
=I.sup.k.sub.MC+P.sup.k+1+W.sup.k+1
[0021] Note that, for each P-VOP the modification includes adding a
signal with two components, namely W.sup.k+1 and D.sup.k. In scenes
where there is a dominant motion, motion compensation in D.sup.k
translates to a spatial shift. Under assumption (3), we can say
that W.sup.k+1 and D.sup.k are uncorrelated, even if
W.sup.k=W.sup.k+1. As a result, the variance of the sum is expected
to be larger than the variance of W.sup.k+1. Under assumption (5),
this translates to a higher increase in bit-rate. Note that, if
W.sup.k=W.sup.k+1 and there is no motion, i.e. D.sup.k=W.sup.k,
then the additive modification to the signal is zero. Therefore,
there is no need for additional bits.
Method Overview
[0022] Above, we established the relation between the bit-rate
increase due to watermarking and the drift compensation. The method
explained here provides an improvement in terms of bit-rate and/or
watermark detection, when there is a dominant motion in the
sequence, e.g., panning.
[0023] FIG. 1 is a flow diagram illustrating an overview of method
of placing a watermark in a time varying signal. The method
estimates motion of the data between a reference frame and a frame
of interest (100). This motion estimation may be performed based on
pre-computed motion parameters in the data stream (e.g., motion
vectors or affine transform parameters from a video stream), or
based on motion parameters derived from motion estimation on the
frames at the time of watermark embedding. Next, the method
computes a representative motion for a particular embedding
location, such as the current frame (104). Finally, the method
geometrically transforms the digital watermark to be embedded in
the current frame. The digital watermark embedder then performs its
embedding operation on the frame to embed the watermark signal into
the video data of the current frame.
[0024] As detailed below, our particular implementation spatially
shifts the watermark signal in synchronization with the dominant
motion in the sequence. When the watermark signal, or part of it,
is constant (Assumption.4), the proposed method minimizes the
variance of the additive modification signal, W.sup.k+1-D.sup.k. As
a result, the same watermark detection strength is achieved with
minimum increase in the bit-rate of the compressed video.
Equivalently, this may translate to better detection for bit-rate
constrained applications.
[0025] As an additional advantage, the proposed method improves the
detection of the watermark in reconstructed B-VOP's, even if those
frames (or the error residuals) have not been watermarked. In
essence, we prevent possible de-constructive interference from two
reference frames on the B-VOP.
Example Implementation
[0026] We would like to determine a global spatial shift value that
will coincide with the motion of the most number of blocks of the
frame. The compressed domain implementation has a unique advantage
at this point. Motion vectors for each block (macro-block) of the
frame may be obtained by simply decoding the motion vector VLC's,
without employing a costly motion estimation algorithm. Under this
observation, pseudo-code of the method is given below. This method
is also illustrated in FIG. 2: [0027] 1. Decode motion VLC's to
obtain motion vectors (MV.x and MV.y) for each block (step 200).
[0028] 2. Compute the 2-D histogram of motion vectors (step 202)
[0029] a. For all blocks [0030] i. Hist[MV.x][MV.y]+=1; [0031] 3.
Find the maxima of the histogram, i.e. the bin with highest count.
[0032] 4. Define the coordinates of the maxima as the dominant
motion (DMV.x, DMV.y) (204). [0033] 5. Shift the watermark
W.sup.k+1 by DMV (206). [0034] a. W.sup.k+1(x,y)=W.sup.k+1(x+DMV.x,
y+DMV.y) [0035] 6. Execute the rest of the algorithm with updated
watermark signal.
Additional Features
[0036] The method described above predicts how the watermark in the
previous frame will propagate to the current frame. In order to
achieve this we have used the motion vectors for the image blocks,
and weigh them equally. We implicitly assume that same amount of
watermark has been added to all blocks. When and if a local scaling
is used for the watermark, the method may be implemented so as to
weigh different image blocks with respect to the watermark energy
they carry.
CONCLUDING REMARKS
[0037] Having described and illustrated the principles of the
technology with reference to specific implementations, it will be
recognized that the technology can be implemented in many other,
different, forms. To provide a comprehensive disclosure without
unduly lengthening the specification, applicants incorporate by
reference the patents and patent applications referenced above.
[0038] The methods, processes, and systems described above may be
implemented in hardware, software or a combination of hardware and
software. For example, the auxiliary data encoding processes may be
implemented in a programmable computer or a special purpose digital
circuit. Similarly, auxiliary data decoding may be implemented in
software, firmware, hardware, or combinations of software, firmware
and hardware. The methods and processes described above may be
implemented in programs executed from a system's memory (a computer
readable medium, such as an electronic, optical or magnetic storage
device).
[0039] The particular combinations of elements and features in the
above-detailed embodiments are exemplary only; the interchanging
and substitution of these teachings with other teachings in this
and the incorporated-by-reference patents/applications are also
contemplated.
* * * * *