U.S. patent number 9,224,184 [Application Number 13/804,413] was granted by the patent office on 2015-12-29 for methods and arrangements for identifying objects.
This patent grant is currently assigned to Digimarc Corporation. The grantee listed for this patent is Digimarc Corporation. Invention is credited to Yang Bai.
United States Patent |
9,224,184 |
Bai |
December 29, 2015 |
Methods and arrangements for identifying objects
Abstract
The present disclosure relates generally to digital watermarking
and grocery/retail store checkout. One claim recites a system
comprising: a 2D camera for capturing imagery of packaged items,
the packaged items including digital watermarking printed on
product packaging; one or more processors programmed for:
prioritizing at least some image areas from within at least one
captured imagery frame for digital watermark detection based on: i)
area brightness, and on ii) area frame location; and detecting
digital watermarks from one or more image areas prioritized from
the prioritizing image areas, in which the detecting digital
watermarks analyzes image areas in order of prioritization. Of
course, other features, combinations and claims are also
provided.
Inventors: |
Bai; Yang (Beaverton, OR) |
Applicant: |
Name |
City |
State |
Country |
Type |
Digimarc Corporation |
Beaverton |
OR |
US |
|
|
Assignee: |
Digimarc Corporation
(Beaverton, OR)
|
Family
ID: |
50485364 |
Appl.
No.: |
13/804,413 |
Filed: |
March 14, 2013 |
Prior Publication Data
|
|
|
|
Document
Identifier |
Publication Date |
|
US 20140112524 A1 |
Apr 24, 2014 |
|
Related U.S. Patent Documents
|
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
Issue Date |
|
|
61716591 |
Oct 21, 2012 |
|
|
|
|
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
G06T
7/10 (20170101); G06K 9/2054 (20130101); G06T
7/73 (20170101); G06K 9/3233 (20130101); G06T
7/194 (20170101); G06T 1/005 (20130101); G06T
2201/0065 (20130101); G06T 2201/0051 (20130101) |
Current International
Class: |
G06K
9/00 (20060101); G06T 1/00 (20060101); G06K
9/32 (20060101) |
References Cited
[Referenced By]
U.S. Patent Documents
Foreign Patent Documents
|
|
|
|
|
|
|
0789270 |
|
Aug 1997 |
|
EP |
|
WO0106755 |
|
Jan 2001 |
|
WO |
|
WO 2011/100048 |
|
Aug 2011 |
|
WO |
|
Other References
Alattar, Smart Images Using Digimarc's Watermarking Technology,
SPIE's 12.sup Symposium on Electronic Imaging, Jan. 25, 2000, vol.
3971, No. 25, 10 pages. cited by applicant .
Avcibas, et al., "Steganalysis of Watermarking Techniques Using
Image Quality Metrics", Proceedings of SPIE, Jan. 2001, vol. 4314,
pp. 523-531. cited by applicant .
Dautzenberg, "Watermarking Images," Department of Microelectronics
and Electrical Engineering, Trinity College Dublin, 47 pages, Oct.
1994. cited by applicant .
Hernandez et al., "Statistical Analysis of Watermarking Schemes for
Copyright Protection of Images," Proceedings of the IEEE, vol. 87,
No. 7, Jul. 1999. cited by applicant .
U.S. Appl. No. 60/125,349, filed Mar. 19, 1999, Alattar et al.
cited by applicant.
|
Primary Examiner: Harandi; Siamak
Attorney, Agent or Firm: Digimarc Corporation
Parent Case Text
RELATED APPLICATION DATA
This application claims the benefit of U.S. Patent Application No.
61/716,591, filed Oct. 21, 2013. This application is also related
to U.S. Provisional Patent Application No. 61/749,767, filed Jan.
7, 2013, and PCT Application No. PCT/US12/53201, filed Aug. 30,
2012 (attached as Appendix A) and published on Mar. 7, 2013 as
WO/2013/033442. Each of the above patent documents is hereby
incorporated herein by reference in its entirety. The above PCT
application is also included as part of this application as
Appendix A. Appendix A is hereby incorporated herein by reference
in its entirety.
Claims
We claim:
1. A system comprising: a 2D camera for capturing imagery of
packaged items moving along a path, the packaged items including
digital watermarking hidden on their packaging; one or more
processors programmed for: digital watermark detection of imagery
captured by the 2D camera, in which image areas in a captured
imagery frame are prioritized for digital watermark detection based
on: i) area brightness, and on ii) area frame location, in which
areas around the center of the imagery frame are more highly
prioritized relative to areas on the top or bottom of the imagery
frame.
2. A grocery checkout station comprising the system of claim 1.
3. The system of claim 2 in which the path comprises a conveyor
belt.
4. A system comprising: a 2D camera for capturing imagery of
packaged items moving along a path, the packaged items including
digital watermarking hidden on their packaging; one or more
processors programmed for: digital watermark detection of imagery
captured by the 2D camera, in which image areas in a captured
imagery frame are prioritized for digital watermark detection based
on: i) area brightness, and on ii) area frame location, in which at
least some of the image areas differ in image resolution.
5. A system comprising: a 2D camera for capturing imagery of
packaged items moving along a path, the packaged items including
digital watermarking hidden on their packaging; one or more
processors programmed for: digital watermark detection of imagery
captured by the 2D camera, in which image areas in a captured
imagery frame are prioritized for digital watermark detection based
on: i) area brightness, and on ii) area frame location, in which
image area prioritization is based, at least in part, on a training
set individualized according to a user's characteristics, in which
the user's characteristics include checkout speed and object
orientation.
6. A system comprising: a 2D camera for capturing imagery of
packaged items, the packaged items including digital watermarking
printed on product packaging; one or more processors programmed
for: prioritizing at least some image areas from within at least
one captured imagery frame for digital watermark detection based
on: i) area brightness, and on ii) area frame location; and
detecting digital watermarks from one or more image areas
prioritized from the prioritizing image areas, in which the
detecting digital watermarks analyzes image areas in order of
prioritization.
7. The system of claim 6 in which image areas around the center of
the captured imagery frame are more highly prioritized relative to
areas on the top or bottom of the captured imagery frame.
8. The system of claim 6 in which at least some image areas differ
in image resolution.
9. The system of claim 6 in which image area prioritization is
based, at least in part, on a training set.
10. The system of claim 9 in which the training set is
individualized according to a user's characteristics.
11. The system of claim 10 in which the user's characteristics
include checkout speed and object orientation.
12. The system of claim 6 in which the one or more processors are
programmed for terminating the detecting digital watermarks once a
digital watermark is found in a prioritized image area rather than
processing all prioritized image areas.
13. A grocery checkout station comprising the system of claim
6.
14. The system of claim 6 in which said one or more processors are
programmed for omitting images areas from within the at least one
captured imagery frame from prioritization.
15. The system of claim 6 in which image areas from multiple
captured imagery frames are prioritized for digital watermark
detection.
16. A computer program product being embodied in non-transitory
computer readable storage media and comprising processor executable
instructions for: obtaining captured imagery representing packaged
items, the packaged items including digital watermarking printed on
product packaging, the digital watermarking being represented in
the captured imagery; prioritizing at least some image areas from
within at least one captured imagery frame for digital watermark
detection based on: i) area brightness, and on ii) area frame
location; and detecting digital watermarks from one or more image
areas prioritized from the prioritizing image areas, in which the
detecting digital watermarks analyzes image areas in order of
prioritization.
17. The computer program product of claim 16 in which image areas
around the center of the captured imagery frame are more highly
prioritized relative to areas on the top or bottom of the captured
imagery frame.
18. The computer program product of claim 16 in which at least some
image areas differ in image resolution.
19. The computer program product of claim 16 in which image area
prioritization is based, at least in part, on a training set.
20. The computer program product of claim 19 in which the training
set is individualized according to a user's characteristics.
21. The computer program product of claim 20 in which the user's
characteristics include checkout speed and object orientation.
22. The computer program product of claim 16 comprising executable
instructions for terminating the detecting digital watermarks once
a digital watermark is found in a prioritized image area rather
than processing all prioritized image areas.
23. The computer program product of claim 16 comprising executable
instructions for omitting images areas from within the at least one
captured imagery frame from prioritization.
24. The computer program product of claim 16 in which image areas
from multiple captured imagery frames are prioritized for digital
watermark detection.
25. A system comprising: a 2D camera for capturing imagery of
packaged items, the packaged items including digital watermarking
printed on product packaging; means for prioritizing at least some
image areas from within at least one captured imagery frame for
digital watermark detection based on: i) area brightness, and on
ii) area frame location; and means for detecting digital watermarks
from one or more image areas prioritized from the prioritizing
image areas, in which the detecting digital watermarks analyzes
image areas in order of prioritization.
26. The system of claim 25 in which image areas around the center
of the captured imagery frame are more highly prioritized relative
to areas on the top or bottom of the captured imagery frame.
27. The system of claim 25 in which at least some image areas
differ in image resolution.
28. The system of claim 25 in which image area prioritization is
based, at least in part, on a training set.
29. The system of claim 28 in which the training set is
individualized according to a user's characteristics.
30. The system of claim 29 in which the user's characteristics
include checkout speed and object orientation.
31. The system of claim 25 in which the one or more processors are
programmed for terminating the detecting digital watermarks once a
digital watermark is found in a prioritized image area rather than
processing all prioritized image areas.
32. A grocery checkout station comprising the system of claim
25.
33. The system of claim 25 further comprising means for omitting
images areas from within the at least one captured imagery frame
from prioritization.
34. The system of claim 25 in which image areas from multiple
captured imagery frames are prioritized for digital watermark
detection.
Description
TECHNICAL FIELD
The present technology concerns object identification and is
suited, e.g., for identifying objects at supermarket checkouts and
other retail locations.
BACKGROUND AND SUMMARY
The widespread use of barcodes has greatly simplified supermarket
checkout. However, many problems persist, causing both
inconvenience for shoppers and added costs for retailers.
One of the difficulties is finding a barcode on a package. While
experienced supermarket clerks eventually learn barcode locations
for popular products, even the best clerks sometimes have
difficulty with less common products. For shoppers who use
self-service checkout stations, any product can be confounding.
Another issue concerns re-orienting a package so that its barcode
is in position for reading.
Digital watermarking can be placed on product packaging--preferably
over the majority of the package--to improve checkout speed.
Methods and systems for improving watermark detection from imagery
obtained from retail checkout cameras are described herein.
The foregoing and a great number of other features and advantages
of the present technology will be more readily apparent from the
following detailed description, which proceeds with reference to
the accompanying drawings.
BRIEF DESCRIPTION OF THE DRAWINGS
FIGS. 1A-1F show a small can moving along a conveyor at a retail
checkout station. The can enters the image frame in FIG. 1A and
leaves the frame in FIG. 1F. The can is moving right to left in the
images.
FIGS. 2A and 2B illustrate possible watermark detection block
patterns.
FIGS. 3A-3J show composite images of a coffee can moving along a
conveyor at a retail checkout station. Detection blocks are shown
in the upper right quadrant of each image composite.
FIGS. 4A and 4B show detection results from test scans.
FIG. 5 shows an example detection block prioritization in an image
frame.
DETAILED DESCRIPTION
In accordance with one aspect of this disclosure, the present
technology concerns a method for identifying items, e.g., by a
retail checkout system. A first such method involves moving an item
to be purchased along a path, such as by a conveyor. A first camera
arrangement captures first 2D imagery (e.g., image or video) data
depicting the item when the item is at a first position along the
path. Suitable 2D imaging scanners are provided, e.g., by DataLogic
ADC INC., located in Eugene, Oreg., USA.
The moving item preferably includes digital watermark printed or
carried on the product packaging. The digital watermarking may span
a substantial portion of the packaging extent. In regions where
there is no printing (e.g., white space), a yellow or other
unobtrusive watermark tint can optionally be applied. (Yellow
watermarking is particularly discussed, e.g., in application Ser.
No. 12/774,512, filed May 5, 2010, published as application no. US
2011-0274310 A1, and U.S. Pat. No. 6,345,104, each of which is
hereby incorporated herein by reference in its entirety.)
Digital watermarking patterns can be applied to items in a tiled
fashion, with a single square (or other shape) watermark pattern
being replicated across and down the item being watermarked. The
tiles are typically embedded with an upper left corner of a first
tile coincident with the upper left corner of the artwork. Tiles
are then placed across and down from this starting point. In other
cases, tiles are placed to coincide with or within specific spatial
areas on packaging.
Each watermark pattern may have an orientation. Common practice is
to embed the watermark tiles so that they are oriented in the same
manner as the artwork (i.e., with "up" in the artwork corresponding
to "up" in the watermark pattern). Differing watermark orientation,
however, can provide an indicator of different watermarked areas.
For example, a first watermarked area may include a first
orientation and a second watermarked area may include a second
orientation. Identification of the different watermark orientations
may help distinguish different spatial areas on the product
packaging.
Examples of watermarking are discussed, e.g., in assignee's U.S.
Patent Application No. 61/693,106; U.S. Pat. No. 8,199,969; and
published application no. US 2010-0150434 A1. Each of these patent
documents is hereby incorporated herein by reference in its
entirety.
Watermark embedding can be optimized too, e.g., as discussed in
assignee's U.S. provisional application No. 61/749,767.
Now back to some checkout scenarios. Recall from above that an item
to be purchased moves along a path, such as a conveyor. A first
camera arrangement captures image data depicting the item when the
item is at a first position along the path.
The next sections discuss a prioritization watermark detector
blocks within a captured image frame(s), e.g., across 30 frames or
more per second. A watermark detector "block" may correspond to an
image area (or specific image features) that the watermark detector
will analyze to determine whether a watermark signal is hidden
therein. For example, with reference to FIG. 5, six (6) areas of an
image are blocked out for analysis by a watermark detector. A
prioritization can determine which of these blocks will be analyzed
first (block 1), second (block 2) and so one through block 6. If a
watermark is found in block 1 or 2, the watermark detector can be
configured to stop looking at the other blocks 3-6. This is a
different scheme than some traditional watermark detectors have
taken. For example, some detectors would start watermark detection
analysis with a block in a corner of the image (shown with dashed
block in FIG. 5) and proceed to analyze blocks vertically across
and horizontally down the image block by block.
Sometimes, a digital watermark detector is fed an image of larger
resolution (e.g., 1024.times.1280 pixels) than what is covered by a
single watermark detector block (e.g., 256.times.256 pixels). If a
watermark detector is looking at image areas one detector block at
a time, then it may take the watermark detector multiple runs to
perform a watermark detection analysis across a whole image frame.
Given constraints on computational resources from the hardware
(e.g., embedded device, ARM processor, etc.), it may be difficult
to cover the whole area of every frame in a timely manner (e.g., as
packaged items are buzzing by on the conveyor past the camera).
Therefore, it may be beneficial to limit the number of blocks that
are analyzed by a watermark detector per image frame. For example,
a watermark detector may only select 3-21 blocks per image frame
for detection analysis.
This may not be an issue for well-marked large packages because
they fill a large portion of a field of view of a camera and, thus,
the chances of a single block detector being placed on a
watermarked area is high. On the other hand, small packages, like
cans and small boxes (e.g., a tea box), may only show up in a small
portion of the camera's field of view, as shown in FIGS. 1A-1F,
making the chance of a single block detector being placed on a well
watermarked area very low. FIGS. 1A-1F represent images from an
in-counter barcode scanner camera with a moving soft drink can,
e.g., of a small Red Bull can size, moving from right to left. FIG.
1A (the entering frame) and FIG. 1F (the leaving frame) are not
considered good candidates for watermark detection, because the can
occupies such a small space within the whole image frame.
During a normal checkout pace, and when the camera is running at a
speed of, e.g., 30 Frames Per Second (FPS), a typical small package
will show up in 2 to 4 frames with suitable watermark detection
presence, as shown in FIGS. 1B-1E. Since a small package covers a
small area of the camera's field of view, a strategy of reading a
watermark hidden in the small package's packaging from many
different detection blocks across the whole image frame may have
diminishing returns in terms of complexity vs. successful reads.
Possibly, a watermark detector may spend time looking for
watermarks in the background or on the package's boundary but not
on the package itself.
When dealing with a video stream, we have found that background
subtraction from the moving average of previous frames is a
computationally efficient and effective method to extract the fast
moving foreground objects. This method can separate static or slow
moving objects (classified as background) from fast moving objects
(classified as foreground), and place more single-block watermark
detector blocks on more meaningful areas (e.g., in terms
watermarked areas) of foreground objects.
Foreground detection can be configured to work as follows: 1.
Background(k+1)=alpha*Frame(k+1)+(1-alpha)*Background(k), 2.
Foreground(k+1)=Frame(k+1)-Background(k+1), if
Frame(k+1)-Background(k+1)>threshold, where indices k or k+1
represent the incoming frame's temporal axis, alpha is the learning
rate which controls how to update background from the incoming
frame, and the threshold can be set to help suppress noise from
illumination variations.
This process is computationally efficient because it may use
pixel-wise subtraction, addition and comparison. Also memory usage
is low since it does not save all previous frames but only a
weighted average of some of the more recent frames. By efficient
post-processing and clustering the results of each pixel, or groups
of pixels, approximate information about location/shape of the
foreground object can be obtained. Processing can be done in real
time, or near real time.
The location and/or shape of the object can be utilized to
constrain the area where watermark detector blocks should be
placed. Significant savings in the computational complexity can be
achieved without losing detection robustness.
Once the foreground region has been detected, we can assign
detector block locations in the imagery to enhance detection. For
example, the block patterns shown in FIGS. 2A and 2B can be placed
over the foreground area (e.g., over the can in FIG. 1C). There are
6 detector blocks in FIG. 2A and 11 blocks in FIG. 2B. The two (2)
darker blocks in FIG. 2B can be at a higher watermark resolution
than the nine (9) lighter colored blocks. In some detection
scenarios the patterns of these figures can be combined to identify
17 detection blocks per image frame. The imagery within each of
these 17 blocks can be analyzed by a watermark detector to
determine whether one or more watermarks are hidden therein. As
discussed above, the detector may optionally be configured to cease
detection of remaining blocks once a watermark is identified.
A first option uses a determined foreground region to trim down the
FIGS. 2A & 2B patterns. For example, in the case of a combined
17 detection blocks (e.g., combining FIGS. 2A & 2B's patterns
to yield 17 watermark detection blocks), a particular detection
block will only be used by a watermark detector if the block falls
inside (or, in some cases, overlapping with) a determined
foreground region. In the case of overlapping, a detector can set a
predetermined threshold, e.g., 75% or more of overlap with the
foreground region.
A second option is now discussed.
As a first action in the second option, the foreground region in
captured imagery can be expanded to a square region or other shaped
window, enclosing all (or most of) the foreground pixels. Then the
square foreground region can be divided into zones, e.g., equally
spaced zones. The foreground pixels (e.g., as determined by
incoming pixels minus background pixels) inside each zone can be
summed together. This summation is a representation of the
illumination of the foreground.
As a second action in the second option, two approaches can be used
to prioritize the placement of detecting blocks (e.g., areas in
which the watermark detector will look for watermark signal) inside
the square foreground region, because the number of single block
analysis areas may not be enough to cover the whole region.
The first approach is based on illumination (or brightness). The
foreground zones are ranked according to their illumination, with
those of a higher rank indicating a relatively better illumination
compared to those of lower rank. We would prefer not to place the
single block detectors on poor illuminated zones, so the watermark
detector is configured to avoid those poor illuminated zones. Also,
the watermark detector may be configured to discard or ignore zones
with high illumination values because they may indicate
over-saturated pixels from glare (e.g., caused by specular
reflection from the packaging by a scanner illumination).
The second approach is based on a geometric position of each zone.
In some cases the zones or areas near the top of the frame and near
the bottom of the frame detect poorly, due to over-saturated pixels
on the top and poor illuminated pixels on the bottom. So a
weighting can be assigned to each zone based on its geometric
locations within an image frame. For example, central zones may be
weighted more significantly than zones close to the frame boundary.
Or zones close to frame boundary may only be considered if no
watermark is initially found in central frame zones.
To combine the two approaches, we can add a normalized illumination
value of each zone with the weight of each zone from its geometric
position, and then do an ascending ranking of detection blocks.
Those zones with a higher value will have higher detection priority
in acquiring a single-block detector.
The above second option is illustrated in FIG. 3A-3J. The minimum
separation between selected detection blocks is set to a
predetermined pixel value, e.g., 64 pixels between each block, to
avoid choosing blocks with too much overlap (i.e. blocks that are
from similar image areas). These FIG. 3 images show 10 composed
frames of a coffee can imaged from an in-counter barcode scanner
camera. Each image includes the incoming frame (upper left
quadrant), detected square foreground region (lower left quadrant),
and up to 7 detection blocks overlaid on top of the incoming frame
(upper right quadrant).
We have performed some experiments to verify our process. The test
datasets we used are ad-hoc captures from non-professional checkers
simulating a supermarket checkout process. We have two datasets,
one labeled Scan-1 and the other Scan-2. The Scan-1 dataset
contains mostly small packages (e.g., small cans) and has 1025
frames of about 30 seconds recording from an in-counter barcode
scanner camera, and the Scan-2 dataset contains both small and
large packages and has 596 frames from the same camera. The Scan-2
dataset contains more frames with packages inside so it has more
frames detected as containing watermarking.
The results of using the first option, which uses a determined
foreground region to trim down the B17 pattern (including 17
detection blocks), are shown in Table 1. There are 168 frames and
53 frames detected as containing watermark from Scan-2 and Scan-1
datasets, respectively, using the fixed static B17 pattern. By
switching to the flexible foreground trimmed B17 (e.g., focuses on
blocks that fall within the foreground area), to get the same
detection rate, on average, only 10 frames are required for Scan-2,
and only 7 frames are required for Scan-1. Since Scan-1 contains
more small packages, and the benefits of using foreground detection
is more obvious on small packages, the savings in terms of number
of blocks per frame is more significant for Scan-1.
TABLE-US-00001 TABLE 1 Compares flexible pattern from foreground
with fixed static pattern in placing single block detectors. SCAN-2
168 frames detected with 168 frames detected w. fixed flexible
foreground placement detection pattern 2680/275 = 9.75 block/frame
17 block/frame SCAN-1 53 frames detected with 53 frames detected
with fixed flexible foreground placement detection pattern 978/162
= 6.04 block/frame 17 block/frame
The results of using the second option are shown in FIGS. 4A and
4B. The straight-dashed lines mark the number of detected frames
from Scan-2 and Scan-1 datasets using the static B17 pattern, 168
and 53, respectively. The curves indicate the number of detected
frames when choosing different number of blocks for placement
inside a foreground region. In general, when you have large set of
detector blocks, say, e.g., 17, then the flexible block pattern
give better detection results. And if you limit the number of
detector blocks, say, down to 9, the flexible block pattern still
gives a good detection rate with much reduced computational
cost.
In other cases, a "smart watermark detector," one that can train
itself based on user or cashier habits or preferences, is
preferred. For example, through a series of training check-out
runs, it can be determined that cashier 1 holds packaged items at a
certain angle, or at predetermined distances from the camera, or at
a certain swipe speed, or places items on a conveyor at certain
orientations. Other training information may include, e.g.,
proximity to the scanner, speed of scanning, production rotation
habits, professional vs. amateur checker speed or habits or
characteristics, etc. Or the detector may determine that they are
only getting watermark reads from certain block areas when a
certain checker checks out. All this information (or subsets of
this information) can be used to adjust the watermark detector,
e.g., by determine which blocks to prioritize in a detection
process. For example, it might be found that cashier 1 always
swipes items in front of the camera so that the packages are in the
top or bottom of the field of view. Whereas, above, these block
areas would typically be prioritized low. But if the detector knows
that cashier 1 is checking out then these areas can be more highly
prioritized. For example, these blocks are analyzed for
watermarking prior to blocks located in other areas of the field of
view. While this passage has focused on digital watermarking,
similar user-specific information can be used to adjust, e.g., an
image recognition or fingerprinting process.
A user's self-checkout habits--including how and at what speed they
present objects to the check-out camera--can be monitored and then
when they are determined they can be used to help configure a
watermark detector, e.g., by prioritize block selection for
watermark detection. A store loyalty card, customer ID or other
identifier can be associated with a database or record storing the
proper detector prioritization. That prioritization can then be
used to inform the watermark detector on how to better process
imagery for that person checking out. For example, a customer shows
their loyalty card or enters (e.g., types on a touch screen or
speaks to a voice recognition unit) prior to checking out. The
customer's checkout profile is identified and applied to the
watermark detector. For example, it might be determined that a
particular customer holds packages at a certain pose. Captured
imagery can be adjusted prior to watermark detection to adjust for
the pose using pose estimation and homography, e.g., as discussed
in assignee's U.S. patent application Ser. No. 13/789,126, filed
Mar. 7, 2013, which is hereby incorporated herein by reference in
its entirety.
Some checkout stations will continue to monitor barcodes even if
supplemental symbologies like watermarking are present during
checkout. In these cases please consider the flowing flow: 1.
Imagery is presented to a watermark detector. 2. The watermark
detector analyzes the imagery and detects a watermark. The
watermark may include a payload or index or other information. 3. A
process is invoked that utilizes that watermark information, index
or payload (or portions thereof) to create or obtain an image
overlay for captured imagery. The image overlay preferably includes
a barcode or other symbology that includes the watermark
information, or information obtained from utilizing the watermark
information. That way, if the same imagery that is analyzed for a
digital watermark is then feed to a barcode reader the graphic
overlay barcode will be easily recognizable even if the depicted
product packaging did not display a barcode.
One challenge may occur if two or more of the same packaged items
are within a single image frame. For example, 2 cans of diet
Mountain Dew might be pictured in the same frame. The watermark
detector finds a read, but in different, non-contiguous image
areas. In such cases a watermark payload may be used to look up a
spatial template. The spatial template is sized roughly to
represent a particular item (e.g., diet soda). The spatial template
is placed around a block area where watermarks were detected. If
watermarks (or watermark components like orientation components)
are located outside of the spatial template (or outside of a
predetermined area or tolerance) then there is a likelihood that
the image frame includes two or more watermarked objects.
The cashier can be warned to examine this area more carefully, or
the system may make a determination to independently ring up both
items.
In another implementation, the checkout camera includes or
cooperates with special illumination. The illumination projects
watermark orientation information on the packaging. The projected
illumination is captured along with the packaged items. The
projected orientation information is deciphered by the watermark
detector to help determine positioning information including
relative depth, orientation, etc. This information can be used in
watermark detection, or foreground/background decisions.
In still another implementation, watermarks are used to identify
certain areas on packaging. For example, a watermark signal (e.g.,
an orientation component) might be used to outline the nutrition
facts on a package. The watermarked area is then used to create a
spatial position on a reading device (in this case, e.g., a
smartphone like an iPhone or Android device). An augmented reality
display is overlaid on the watermarked area, e.g., as discussed in
assignee's U.S. patent application Ser. No. 13/789,126.
Concluding Remarks
This specification details a variety of embodiments. It should be
understood that the methods, elements and concepts detailed in
connection with one embodiment can be combined with the methods,
elements and concepts detailed in connection with other embodiments
and with those discussed in Appendix A. While some such
arrangements have been particularly described, many have not--due
to the large number of permutations and combinations. However,
implementation of all such combinations is straightforward to the
artisan from the provided teachings.
Although features and arrangements are described, in some cases,
individually, the inventors intend that they will also be used
together. Conversely, while certain methods and systems are
detailed as including multiple features, the inventors conceive
that--in other embodiments--the individual features thereof are
usable independently.
The present specification should be read in the context of the
cited references (with which the reader is presumed to be
familiar). Those references disclose technologies and teachings
that applicant intends be incorporated into certain embodiments of
the present technology, and into which the technologies and
teachings detailed herein be incorporated.
For example, with the documents cited in Appendix A, certain of the
cited references teach that a single image sensor can be used, in
conjunction with mirrors and other optics, to capture imagery of an
object from two or more different views (as opposed to using two or
more different cameras). Some such arrangements use
wavelength-selective optics (e.g., dichroic mirrors) so that three
different images can be projected onto common pixel elements of a
single sensor, allowing separate processing of the three different
images in different color channels. Other such arrangements use
mirrors to project images from different viewpoints onto different
rectangular sub-regions of a common sensor. Still further, other of
the prior art teaches that a color (RGB) image sensor can be used
to capture imagery of an object, yet object identification can
proceed using just pixels of a single color (e.g., whichever color
shows the highest variance in its histogram). The artisan will
recognize that these and all the other arrangements taught in the
cited art can utilize the methods and features detailed herein, and
vice versa.
To provide a comprehensive disclosure, while complying with the
statutory requirement of conciseness, applicants
incorporate-by-reference each of the documents referenced herein.
(Such materials are incorporated in their entireties, even if cited
above in connection with specific of their teachings.)
Although not particularly illustrated, it will be recognized that
the methods described above can be implemented using general
purpose (or special purpose) computers, e.g., comprising one or
more processors, multi-core processors, semiconductor memory, hard
disks, networking connections, and input-output devices, as are
conventional in the art. Software instructions for implementing the
above-detailed methods can be stored on tangible media associated
with such systems, e.g., disks and semiconductor memories.
While the focus in the above sections has been on digital watermark
recognition, many of the above techniques will also enhance other
object-identifying techniques such as a barcode, optical character
recognition, image recognition, fingerprinting. For example, the
foreground identification techniques above can be used to locate
frame areas to derive fingerprints.
In view of the wide variety of embodiments to which the principles
and features discussed above can be applied, it should be apparent
that the detailed embodiments are illustrative only, and should not
be taken as limiting the scope of the invention. Rather, we claim
as our invention all such modifications as may come within the
scope and spirit of the following claims and equivalents
thereof.
Concluding Remarks
This specification details a variety of embodiments. It should be
understood that the methods, elements and concepts detailed in
connection with one embodiment can be combined with the methods,
elements and concepts detailed in connection with other
embodiments. While some such arrangements have been particularly
described, many have not--due to the large number of permutations
and combinations. However, implementation of all such combinations
is straightforward to the artisan from the provided teachings.
Although features and arrangements are described, in some cases,
individually, the inventors intend that they will also be used
together. Conversely, while certain methods and systems are
detailed as including multiple features, the inventors conceive
that--in other embodiments--the individual features thereof are
usable independently.
The present specification should be read in the context of the
cited references (with which the reader is presumed to be
familiar). Those references disclose technologies and teachings
that applicant intends be incorporated into the certain embodiments
of the present technology, and into which the technologies and
teachings detailed herein be incorporated.
For example, certain of the cited references teach that a single
image sensor can be used, in conjunction with mirrors and other
optics, to capture imagery of an object from two or more different
views (as opposed to using two or more different cameras). Some
such arrangements use wavelength-selective optics (e.g., dichroic
mirrors) so that three different images can be projected onto
common pixel elements of a single sensor, allowing separate
processing of the three different images in different color
channels. Other such arrangements use mirrors to project images
from different viewpoints onto different rectangular sub-regions of
a common sensor. Still further, other of the prior art teaches that
a color (RGB) image sensor can be used to capture imagery of an
object, yet object identification can proceed using just pixels of
a single color (e.g., whichever color shows the highest variance in
its histogram). The artisan will recognize that these and all the
other arrangements taught in the cited art can utilize the methods
and features detailed herein, and vice versa.
To provide a comprehensive disclosure, while complying with the
statutory requirement of conciseness, applicants
incorporate-by-reference each of the documents referenced herein.
(Such materials are incorporated in their entireties, even if cited
above in connection with specific of their teachings.)
In view of the wide variety of embodiments to which the principles
and features discussed above can be applied, it should be apparent
that the detailed embodiments are illustrative only, and should not
be taken as limiting the scope of the invention. Rather, we claim
as our invention all such modifications as may come within the
scope and spirit of the following claims and equivalents
thereof.
* * * * *