U.S. patent application number 13/278513 was filed with the patent office on 2012-04-26 for correcting image capture distortion.
Invention is credited to Alastair M. Reed.
Application Number | 20120099758 13/278513 |
Document ID | / |
Family ID | 40088241 |
Filed Date | 2012-04-26 |
United States Patent
Application |
20120099758 |
Kind Code |
A1 |
Reed; Alastair M. |
April 26, 2012 |
Correcting Image Capture Distortion
Abstract
The present disclosure relates generally to improving
readability of machine-readable information, despite signal capture
distortion. One claim recites an apparatus comprising: memory for
storing input data; an electronic processor programmed for:
altering a digital watermark or a digital watermarking process to
pre-distort a digital watermark signal, wherein the altering is
intended to counteract or compensate for expected distortion so as
to enable machine-based detection of an embedded, pre-distorted
digital watermark signal despite the expected distortion, wherein
the expected distortion includes distortion associated with optical
capture of the input data; and embedding the pre-distorted digital
watermark signal in the input data. Of course, other claims and
combinations are provided as well.
Inventors: |
Reed; Alastair M.; (Lake
Oswego, OR) |
Family ID: |
40088241 |
Appl. No.: |
13/278513 |
Filed: |
October 21, 2011 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
12109573 |
Apr 25, 2008 |
|
|
|
13278513 |
|
|
|
|
60913987 |
Apr 25, 2007 |
|
|
|
Current U.S.
Class: |
382/100 |
Current CPC
Class: |
G06T 5/003 20130101;
G06T 1/0064 20130101; G06T 5/002 20130101; G06T 5/006 20130101 |
Class at
Publication: |
382/100 |
International
Class: |
G06K 9/00 20060101
G06K009/00 |
Claims
1. A method comprising: obtaining input data; using a programmed
processor, altering a digital watermark or a digital watermarking
process to pre-distort a digital watermark signal, wherein the
altering is intended to counteract or compensate for expected
distortion so as to enable machine-based detection of an embedded,
pre-distorted digital watermark signal despite the expected
distortion, wherein the expected distortion includes distortion
associated with optical capture of the input data; and embedding
the pre-distorted digital watermark signal in the input data.
2. The method of claim 1 further comprising printing or engraving
the input data including the pre-distorted digital watermark
signal.
3. The method of claim 1 wherein the input data comprises data
representing an image or video.
4. The method of claim 1 wherein the altering utilizes a model of
the distortion resulting from optical capture.
5. The method of claim 1 wherein the expected distortion includes
blurring resulting from optical capture of the printed or engraved
input data.
6. The method of claim 5 wherein the blurring is associated with
lens focal length.
7. The method of claim 5 wherein the blurring--left uncompensated
for--would result in a loss of high frequency information of the
digital watermark signal upon signal detection.
8. The method of claim 1 wherein said act of altering alters a
phase of the digital watermark signal relative to the expected
distortion.
9. The method of claim 8 wherein an altered phase of the digital
watermark signal causes the digital watermark signal to be less
readable in the absence of the expected distortion.
10. The method of claim 1 wherein expected distortion is modeled as
a convolution of the input data with a predetermined function.
11. The method of claim 10 wherein the predetermined function is
approximated by a Bessel function.
12. The method of claim 1 wherein the expected distortion is
associated with lens focal length.
13. A non-transitory computer readable medium comprising
instructions or circuitry to perform the method of claim 1.
14. An apparatus comprising: memory for storing input data; an
electronic processor programmed for: altering a digital watermark
or a digital watermarking process to pre-distort a digital
watermark signal, wherein the altering is intended to counteract or
compensate for expected distortion so as to enable machine-based
detection of an embedded, pre-distorted digital watermark signal
despite the expected distortion, wherein the expected distortion
includes distortion associated with optical capture of the input
data; and embedding the pre-distorted digital watermark signal in
the input data.
15. The apparatus of claim 14 in which the electronic processor
controls printing or engraving of the input data including the
pre-distorted digital watermark signal.
16. The apparatus of claim 14 wherein the input data comprises data
representing an image or video.
17. The apparatus of claim 14 wherein the altering utilizes a model
of the distortion resulting from optical capture.
18. The apparatus of claim 14 wherein the expected distortion
includes blurring resulting from optical capture of the printed or
engraved input data.
19. The apparatus of claim 18 wherein the blurring is associated
with lens focal length.
20. The apparatus of claim 18 wherein the blurring--left
uncompensated for--would result in a loss of high frequency
information of the digital watermark signal upon signal
detection.
21. The apparatus of claim 14 wherein the altering alters a phase
of the digital watermark signal relative to the expected
distortion.
22. The apparatus of claim 21 wherein an altered phase of the
digital watermark signal causes the digital watermark signal to be
less readable in the absence of the expected distortion.
23. The apparatus of claim 14 wherein expected distortion is
modeled as a convolution of the input data with a predetermined
function.
24. The apparatus of claim 23 wherein the predetermined function is
approximated by a Bessel function.
25. The apparatus of claim 14 wherein the expected distortion is
associated with lens focal length.
Description
RELATED APPLICATION DATA
[0001] This application is a division of U.S. patent application
Ser. No. 12/109,573, filed Apr. 25, 2008 (published as U.S.
2008-0298632 A1), which claims the benefit of U.S. Patent
Application No. 60/913,987, filed Apr. 25, 2007. Each of the above
patent documents is hereby incorporated by reference in its
entirety.
TECHNICAL FIELD
[0002] The present disclosure relates generally to data hiding and
digital watermarking. In some implementations the present
disclosure relates to correcting optical scan or other capture
distortion, e.g., such as lens blurring, focus errors and other
scanning anomalies.
BACKGROUND AND SUMMARY
[0003] Digital watermarking--a form of steganography--is a process
for modifying media content to embed a machine-readable code into
the content. The content may be modified such that the embedded
code is imperceptible or nearly imperceptible to the user, yet may
be detected through an automated detection process. Most commonly,
digital watermarking is applied to media such as images, audio
signals, and video signals. However, it may also be applied to
other types of data, including text documents (e.g., through line,
word or character shifting, background texturing, etc.), software,
multi-dimensional graphics models, and surface textures of
objects.
[0004] Digital watermarking systems typically have two primary
components: an embedding component (or encoder) that embeds a
watermark in media content, and a reading component (or detector)
that detects and reads the embedded watermark. The embedding
component embeds a watermark by altering data samples of the media
content in the spatial, temporal or some other domain (e.g.,
Fourier, Discrete Cosine or Wavelet transform domains). The reading
component analyzes target content to detect whether a watermark is
present. In applications where the watermark encodes information
(e.g., a plural-bit message), the reader extracts this information
from the detected watermark.
[0005] The present assignee's work in steganography, data hiding
and digital watermarking is reflected, e.g., in U.S. Pat. Nos.
5,862,260, 6,408,082, 6,614,914 and 7,027,614, which are each
hereby incorporated by reference. A great many other approaches are
familiar to those skilled in the art. The artisan is presumed to be
familiar with the full range of literature concerning
steganography, data hiding and digital watermarking.
[0006] In some cases digital watermarking and other
machine-readable indicia (e.g., barcodes, data glyphs, etc.) may be
detected from optical scan data, examples of which are disclosed
in, e.g., U.S. Pat. Nos. 5,978,773, 6,522,770, 6,681,028, 6,947,571
and 7,174,031, which are each hereby incorporated by reference.
[0007] Today's camera phones and other handheld readers present
expanded decoding opportunities--and challenges.
[0008] One challenge is providing handheld cameras (e.g., in a cell
phone or other mobile device) to an army of users, with nearly each
user having a different idea on proper focal length for image and
video capture.
[0009] In some cases a user may want to be close to a marked image
or video to, e.g., capture high spatial frequency content (see FIG.
1); but the close positioning often results in a captured image
that is slightly out of focus (or blurred)--which may hamper
detection of a machine-readable code contained or represented in
the image.
[0010] Thus, one inventive combination provides a method including:
obtaining input data; altering a digital watermark or a digital
watermarking process to pre-distort a digital watermark signal,
wherein the altering is intended to counteract or compensate for
expected distortion enabling machine-based detection of an
embedded, pre-distorted digital watermark signal despite the
expected distortion; and embedding the pre-distorted digital
watermark signal in the input data.
[0011] Another inventive combination provides a method including:
obtaining a plurality of different models representing different
expected distortion associated with a plurality of different data
captures, the different data captures each resulting in distortion
of a machine-readable signal; indexing the different models; upon
receiving a request, selecting a model associated with the request;
and providing the selected model.
[0012] Still another inventive combination provides a method
including: obtaining input data; altering a digital watermark or a
digital watermarking process to pre-distort a digital watermark
signal, wherein the altering is intended to counteract or
compensate for expected distortion due to image capture or an image
capture device, the altering enabling machine-based detection of an
embedded, pre-distorted digital watermark signal despite the
expected distortion; and embedding the pre-distorted digital
watermark signal in the input data.
[0013] Yet another inventive combination provides a method
including: obtaining input data, the input data representing
imagery captured with at least an optical lens, the input data
comprising test data and a machine-readable signal; evaluating
characteristics associated with the test data to determine
information regarding lens blurring of the input data associated
with the optical lens; adjusting the input data to compensate for
or to correct the lens blurring based at least in part on the
information; and analyzing the compensated for or corrected input
data to obtain the machine-readable signal.
[0014] Another inventive combination includes: obtaining input
data, the input data representing imagery or video, the input data
comprising test data and a machine-readable signal; determining
characteristics associated with the test data to determine
information regarding signal capture distortion of the input data;
based on at least the characteristics, determining an amount of
correction or counteracting to be applied to the input data;
applying a determined amount of correction or counteracting to the
input data; and analyzing corrected or counteracted input data to
obtain the machine-readable signal.
[0015] Further combinations, aspects, implementations, features,
embodiments and advantages will become even more apparent with
reference to the following detailed description, the accompanying
drawings and claims.
BRIEF DESCRIPTION OF THE DRAWINGS
[0016] FIG. 1 illustrates image capture with a cell phone including
an optical sensor.
[0017] FIG. 2A illustrates an example target image.
[0018] FIG. 2B illustrates examples of adverse effects on focus
(e.g., blurring) of the FIG. 2A example target image.
[0019] FIG. 3 illustrates a plot diagram.
[0020] FIG. 4 is a block diagram illustrating an embedding
process.
[0021] FIG. 5 is a diagram showing an approximation or estimate of
lens blurring.
[0022] FIG. 6 is a correction template or filter to compensate for
the blurring shown in FIG. 5.
[0023] FIG. 7 is a blurred signal (FIG. 5) after compensation by
the FIG. 6 correction template or filter.
[0024] FIG. 8A is a diagram showing test data in a spatial
frequency domain (with acceptable lens blur); FIG. 8B is another
diagram showing test data in a spatial frequency domain (with
unacceptable lens blur).
[0025] FIG. 9 shows a digital watermarked gray patch.
[0026] FIG. 10A shows the watermarked gray patch of FIG. 9 after
capture distortion (e.g., lens blurring).
[0027] FIG. 10B shows the FIG. 10A patch after compensation or
correction (e.g., using the FIG. 6 correction).
DETAILED DESCRIPTION
[0028] When an image is distorted (e.g., blurred, out of focus,
etc.) by an optical capture device (e.g., a cell phone camera, PDA,
digital camera, etc.) the loss of information from a target image
(FIG. 2A) can be in the form shown in FIG. 2B. For example, in some
cases there may be a loss of information in some high spatial
frequencies.
[0029] The distortion may result in some high spatial frequencies
of the target image being "out of phase" or otherwise
distorted.
[0030] In the case of a target image including or representing
machine-readable information, e.g., a barcode, data glyph or
digital watermark, some higher spatial frequency information
corresponding to at least some of the machine-readable information
is also out of phase, making reliable detection of the
machine-readable information more difficult.
[0031] Distortion such as blurring can be modeled or approximated
by a convolution of a target image with, e.g., a Bessel function,
as postulated by J. I. Yellott et al., "Correcting spurious
resolution in defocused images," SPIE 6492, pp. 649200-1-649200-12
(2007), hereby incorporated by reference. FIG. 3 illustrates a
portion of a Bessel function, where the negative areas (shown with
dotted boxes) correspond to a 90 degree phase shift.
[0032] Some examples are now provided illustrating and describing
compensation for distortion.
[0033] In a first implementation a target image includes a digital
watermark signal. But prior to being embedded in the target image,
the digital watermark signal is pre-distorted with a phase shift,
e.g., shifted 90 degrees relative to expected distortion. For
example, if the distortion is likely to occur in high frequency
areas such as discussed above with respect to convolving with the
function shown in FIG. 3, high frequency components of the digital
watermark are shifted 90 degrees. (An appropriate shift can be
achieved by, e.g., convolving a digital watermark signal with an
appropriate or corresponding Bessel function.) The pre-distorted
watermark is then embedded into an image and printed, engraved or
otherwise reproduced. Then, when expected distortion occurs (e.g.,
blurring due to image capture or focus errors), the high frequency
watermark components are distorted (e.g., convolved) into a more
readable form, enabling better watermark detection.
[0034] As another example, if a digital watermark includes signal
elements with values corresponding to [+1, 0, -1] at positions in a
high frequency area, the signal is preferably phase shifted to
compensate for excepted distortion. In the case of distortion
modeled by a convolution with a target image and the function shown
in FIG. 3, the signal is shifted 90 degrees resulting in signal
elements with values corresponding to [-1, 0, +1] at the positions
in the high frequency area. When this pre-distorted digital
watermark signal is subjected to blurring during image capture, the
pre-distorted signal elements are distorted again--but in an
expected manner--resulting in signal elements with values
corresponding to [+1, +1, 0 -1], which correspond to the original
signal elements. (The blurring shifts the signal essentially back
to its original, pre-distorted form.)
[0035] FIG. 4 illustrates a block diagram corresponding to one
implementation of pre-distortion.
[0036] An image 10 is obtained to be watermarked. The image 10 can
be of any form, e.g., a color image, grayscale, a photograph, a
graphic or artwork, video representation, etc. If the image 10 is
in analog form, e.g., a printed image, it can be optically scanned
or captured (e.g., with a digital camera or optical sensor) or
otherwise converted into a digital image.
[0037] A message 12 is provided (e.g., 1 or more bits). The message
12 is intended to be hidden within the image 10 with digital
watermarking or steganography. A digital watermarking signal is
generated by a digital watermark encoder 20 to represent or
otherwise carry the message 12. The digital watermark signal may be
an additive signal, e.g., one that is added (or subtracted,
multiplied, etc.) to the image 10, may represent instructions or
changes that should be made to the image 10 to carry the message,
e.g., based on a key, and/or may include changes or modifications
to frequency domain coefficients, and/or may include a random or
pseudo-random component. Of course, other digital watermarking
techniques may be used as well.
[0038] The digital watermark signal is pre-distorted to compensate
for expected distortion, e.g., by a distortion module 22.
Distortion module 22 may be incorporated into encoder 20 or encoder
20 may other wise communicate or cooperate with a distortion module
22. Distortion module 22 accesses or otherwise determines a
distortion model 24. Distortion model 24 provides a template, mask
or other instructions to be used by distortion module 22 when
pre-distorting the digital watermark signal. The distortion model
24 can be based on or tailored to, e.g., the type of image 10, the
watermark message 12, the expected distribution channel through
which image 10 will travel, the expected type of image capture,
optical lens system or a model of distortion introduced by such,
digital watermark encoder 20, human visual system (HVS), etc. In
one implementation, the distortion module 22 (or watermark encoder
20) communicates with a database, model library or a network remote
resource to access an appropriate model 24. The term "appropriate"
in this context implies that a model 24 is selected or obtained to
compensate for expected distortion. Of course, the distortion
module 22 may be pre-programmed with a default distortion model 24,
e.g., based on or tailored to expected distortion that a digital
watermark will most likely encounter.
[0039] Prior to embedding the digital watermark signal in the image
10, the distortion module 22 accesses a distortion model 24 and
pre-distorts a digital watermark signal to counteract or compensate
for expected distortion associated with optical scanning or
capture. Returning to an example discussed above, e.g., blurring or
other distortion that can be modeled with convolving image 10 with
a Bessel Function, the digital watermark is phase shifted in some
or all higher frequencies, e.g., 90 degrees, to compensate for the
expected distortion.
[0040] The pre-distorted digital watermark--representing or
carrying message 12--is embedded in image 10 to yield a watermarked
image 14. The watermarked image 14 can then be printed or otherwise
reproduced in an analog form.
[0041] A printed (or otherwise fixed) image 16 that includes a
pre-distorted digital watermark signal can then be optically
scanned or captured--which introduces expected distortion. But
since the digital watermark signal has been pre-distorted, the
expected distortion transforms (e.g., modeled as a convolution with
a determined function) the pre-distorted digital watermark signal
into a more readable form.
[0042] Optical data corresponding to the printed (or otherwise
fixed), watermarked image 16 is provided to a digital watermark
decoder to analyze the scan data and recover the message 12. Of
course, the digital watermark decoder may be co-located with an
image sensor or otherwise located on a handheld device (e.g., a
cell phone or personal digital assistant (PDA)). Still further, the
digital watermark decoder may be remotely located from an image
capture device.
[0043] Post-Capture Correction
[0044] In other embodiments, a digital watermark is provided
without the pre-distortion discussed above. Correction for lens
blur is applied after image capture. These post-capture correction
embodiments may allow for an even more imperceptible watermark,
relative to a pre-distortion watermark.
[0045] While post-capture creation may include some
post-processing, it may be more flexible relative to the
pre-distortion methods and systems discussed above, as many types
or amounts of capture distortion can be corrected.
[0046] A digital watermark is embedded in a host signal (e.g., an
image or video). The embedded host signal is provided onto a
surface (projected or rendered if video) or object (e.g., printed,
engraved, etc.) and then an optical scan or digital image is
captured of the surface or object.
[0047] Recall, here, the digital watermark has not been
pre-distorted as discussed above with respect to FIG. 4. So prior
to watermark detection (or as an initial stage of the watermark
detection) the captured image data is filtered or corrected. The
filter adjusts the captured image data to compensate for the
capture distortion (e.g., lens blur).
[0048] To determine an amount or level of compensation or
correction, one process quantifies or otherwise evaluates image
blur in the image or video data. Such an evaluation or
quantification helps to determine an appropriate amount of
correction to be applied to captured image and video.
[0049] In one example, test data can be introduced into an image or
video. Characteristics associated with the test data can be
evaluated in captured imagery to help determine an amount or level
of lens blur.
[0050] One example of test data is white noise (or other type of
noise, e.g., a pseudo- random pattern, noise in some predetermined
frequencies, etc.), which can be added to an image or video, e.g.,
as a part of or before/after digital watermark embedding. The white
noise is preferably imperceptible in the image or video, but of
course, there may be some applications where some perceptibility is
allowed. After image or video capture, the white noise can be
evaluated, e.g., in a spatial frequency domain as shown in FIGS. 8A
and 8B.
[0051] FIG. 8A illustrates a spatial frequency response for
captured imagery with acceptable capture distortion. The term
"acceptable" can be quantified by determining white noise signal
magnitude at a predetermined spatial frequency or range of
frequencies. If the signal magnitude is at or above a predetermined
magnitude for the predetermined spatial frequency (or range of
frequencies) the capture distortion can be deemed acceptable.
[0052] FIG. 8B illustrates a spatial frequency response for
captured imagery with unacceptable capture distortion. The term
"unacceptable" can be quantified by determining white noise signal
magnitude at a predetermined spatial frequency or range of
frequencies. If the signal magnitude is below a predetermined
magnitude for the predetermined spatial frequency (or range of
frequencies) the capture distortion can be deemed unacceptable.
[0053] The signal magnitude level can also be used to help
determine or estimate an amount or level of correction or
adjustment that is needed as well. For example, if the magnitude is
below (or between) a first pre-determined amount, a first level is
determined; and if the magnitude is below (or between) a second
pre-determined amount, a second level is determined An amount or
level of correction or adjustment corresponding to the first
pre-determined amount or the second pre-determined amount is then
selected.
[0054] Another example of test data are predetermined structures.
For example, a dark spot surrounded by a lighter (e.g., white)
color is provided in the spatial domain in an image or video. The
dark spot and surrounding lighter color are preferably subtle so as
not to detract from the image or video. A spatial frequency plot of
captured imagery including such test data is useful in evaluating
capture distortion such as lens blur. The dot structure is
approximated by a delta function in a spatial frequency domain. The
magnitude and frequency location of the delta function can be used
to determine an amount, type or level of capture distortion.
Appropriate correction can be determined and applied to captured
imagery if needed.
[0055] Once an estimate of the lens blur is determined, it can be
used to help correct capture distortion in captured imagery. For
example, the estimate may be mapped against pre-determined levels
of lens blurring, e.g., stored in a data structure associated with
or cooperating with a digital watermark detector. In one
implementation, estimated distortion is used to determine a level,
type or degree of lens blurring or other distortion. A
corresponding scale factor or other corresponding modification can
then be obtained, e.g., from a data structure. An obtained scale
factor or other modification can then be applied to a master
correction template or filter.
[0056] An approximation of one example lens blur is shown in FIG.
5, where pixel magnitude of the blur is illustrated. The
illustrated blur spans more than 15 pixels, which would severely
disrupt an image or video in the pixel area. A master correction
template or filter is shown in FIG. 6, e.g., for a 1-dimensional
signal. (Of course, other types of signals, e.g., 2-dimensional
signal can be similarly corrected.) The master correction template
or filter is adjusted, scaled or modified by a determined scale
factor or other corresponding modification, if needed. The
adjusted, scaled or modified template or filter is then applied to
the captured imagery to counteract, adjust or correct image blur
(FIG. 7). A digital watermark detector (or other machine-readable
symbology detector) analyzes the counteracted, adjusted or
corrected captured imagery to read or detect a digital watermark
(or other machine-readable code).
[0057] In another implementation, a correction template or filter
is generated upon evaluation of image blur. For example, based on
an evaluation of test data in a spatial frequency domain, a
correction filter is generated to counteract lens blur. A generated
correction template or filter is applied to captured imagery to
counteract, adjust or correct image blur. A digital watermark
detector (or other machine-readable symbology detector) analyzes
the counteracted, adjusted or corrected captured imagery to read or
detect a digital watermark (or other machine-readable code).
[0058] To further illustrate some of these processes, an original
watermark was created by embedding a Digimarc Mobile digital
watermark signal (provided by Digimarc Corporation headquartered in
Beaverton, Oregon) into a gray image patch. The digital watermark
signal is readily detectable from the watermarked gray image patch
(FIG. 9). Lens blur (e.g., approximated by FIG. 5) is applied to
the watermarked gray patch (FIG. 9), resulting in a blurred image
patch shown in FIG. 10A. The digital watermark is difficult (and
perhaps impossible) to read from the distorted FIG. 10A patch. But,
after compensation (e.g., as approximated in FIG. 6) is applied to
the distorted FIG. 10A image, e.g., as shown in FIG. 6, the
watermark is readily detected from the resulting image shown in
FIG. 10B. My tests show that the compensated for gray patch (FIG.
10B) reads about 5 dB better than the FIG. 10A patch and approaches
the detection results of a digital watermark embedded in the
original gray image patch (FIG. 9).
[0059] Concluding Remarks
[0060] Having described and illustrated the principles of the
technology with reference to specific implementations, it will be
recognized that the technology can be implemented in many other,
different, forms.
[0061] The methods, processes, components, modules, generators and
systems described above may be implemented in hardware, software or
a combination of hardware and software. For example, digital
watermark embedding or decoding processes may be implemented in a
programmable computer or a special purpose digital circuit.
Similarly, these watermarking processes may be implemented in
software, firmware, hardware, or combinations of software, firmware
and hardware.
[0062] The methods, components and processes described above may be
implemented in software programs (e.g., C, C++, Visual Basic, Java,
executable binary files, etc.) executed from a system's memory
(e.g., a computer readable medium, such as an electronic circuitry
and/or an optical or magnetic storage device).
[0063] The section headings are provided for the reader's
convenience. Features found under one heading can be combined with
features found under another heading. Of course, many other
combinations are possible given the above detailed and enabling
disclosure.
[0064] We have used the terms "image" and "imagery" above. Both of
these terms should be construed broadly enough to include images or
video.
[0065] The particular combinations of elements and features in the
above-detailed embodiments are exemplary only; the interchanging
and substitution of these teachings with other teachings in this
and the above-mentioned U.S. patent documents are also
contemplated.
* * * * *