U.S. patent application number 16/435292 was filed with the patent office on 2019-10-03 for concerning digital marking and reading of plastic items, useful in recycling.
The applicant listed for this patent is Digimarc Corporation. Invention is credited to Osama M. Alattar, Hugh L. Brunk, William Y. Conwell, Tomas Filler, Vojtech Holub, Ravi K. Sharma, Matthew M. Weaver.
Application Number | 20190306385 16/435292 |
Document ID | / |
Family ID | 68057400 |
Filed Date | 2019-10-03 |
![](/patent/app/20190306385/US20190306385A1-20191003-D00000.png)
![](/patent/app/20190306385/US20190306385A1-20191003-D00001.png)
![](/patent/app/20190306385/US20190306385A1-20191003-D00002.png)
![](/patent/app/20190306385/US20190306385A1-20191003-D00003.png)
![](/patent/app/20190306385/US20190306385A1-20191003-D00004.png)
![](/patent/app/20190306385/US20190306385A1-20191003-D00005.png)
![](/patent/app/20190306385/US20190306385A1-20191003-D00006.png)
![](/patent/app/20190306385/US20190306385A1-20191003-D00007.png)
![](/patent/app/20190306385/US20190306385A1-20191003-D00008.png)
![](/patent/app/20190306385/US20190306385A1-20191003-D00009.png)
![](/patent/app/20190306385/US20190306385A1-20191003-D00010.png)
View All Diagrams
United States Patent
Application |
20190306385 |
Kind Code |
A1 |
Sharma; Ravi K. ; et
al. |
October 3, 2019 |
CONCERNING DIGITAL MARKING AND READING OF PLASTIC ITEMS, USEFUL IN
RECYCLING
Abstract
A plastic item, such as a beverage bottle, conveys two distinct
digital watermarks, encoded using two distinct signaling protocols.
A first, printed label watermark conveys a retailing payload,
including a Global Trade Item Number (GTIN) used by a point-of-sale
scanner in a retail store to identify and price the item when
presented for checkout. A second, plastic texture watermark conveys
a recycling payload, including data identifying the composition of
the plastic. The use of two different signaling protocols assures
that a point-of-sale scanner will not spend its limited time and
computational resources working to decode the recycling watermark,
which lacks the data needed for retail checkout. In some
embodiments, a recycling apparatus makes advantageous use of both
types of watermarks to identify the plastic composition of the item
(e.g., relating GTIN to plastic type using an associated database),
thereby increasing the fraction of items that are correctly
identified for sorting and recycling. A great number of other
features and arrangements are also detailed.
Inventors: |
Sharma; Ravi K.; (Portland,
OR) ; Filler; Tomas; (Beaverton, OR) ; Holub;
Vojtech; (Beaverton, OR) ; Alattar; Osama M.;
(Tigard, OR) ; Brunk; Hugh L.; (Portland, OR)
; Weaver; Matthew M.; (Portland, OR) ; Conwell;
William Y.; (Portland, OR) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Digimarc Corporation |
Beaverton |
OR |
US |
|
|
Family ID: |
68057400 |
Appl. No.: |
16/435292 |
Filed: |
June 7, 2019 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
15823138 |
Nov 27, 2017 |
|
|
|
16435292 |
|
|
|
|
14611515 |
Feb 2, 2015 |
9832353 |
|
|
15823138 |
|
|
|
|
62854754 |
May 30, 2019 |
|
|
|
62845230 |
May 8, 2019 |
|
|
|
62836326 |
Apr 19, 2019 |
|
|
|
62830318 |
Apr 5, 2019 |
|
|
|
62818051 |
Mar 13, 2019 |
|
|
|
62814567 |
Mar 6, 2019 |
|
|
|
62812711 |
Mar 1, 2019 |
|
|
|
61934425 |
Jan 31, 2014 |
|
|
|
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
B42D 25/333 20141001;
H04N 21/44008 20130101; H04L 2209/608 20130101; H04N 21/23892
20130101; H04N 5/225 20130101; G06F 21/16 20130101; G09C 5/00
20130101; H04L 2209/34 20130101; B42D 25/324 20141001; H04N 21/8358
20130101 |
International
Class: |
H04N 5/225 20060101
H04N005/225; G06F 21/16 20060101 G06F021/16; H04N 21/44 20060101
H04N021/44; G09C 5/00 20060101 G09C005/00; H04N 21/2389 20060101
H04N021/2389; H04N 21/8358 20060101 H04N021/8358 |
Claims
1. A system for processing a material stream including plural
plastic objects, the system comprising: a conveyor belt; one or
more light sources positioned to illuminate plastic objects on the
conveyor belt; one or more cameras positioned to capture imagery
depicting plastic objects on the conveyor belt; watermark reading
means for decoding from said imagery (a) first digital watermark
information having a first signaling protocol, and (b) second
digital watermark information having a second, different, signaling
protocol; and a sorter responsive to decoded watermark
information.
2. The system of claim 1 that further includes clue detection means
for triggering the watermark reading means in response to a
promising excerpt of imagery.
3. A recycling system including plastic food and beverage
containers on a conveyor, and further including means for (a)
diverting certain of said containers from the conveyor based on
digital watermarks printed thereon, and (b) diverting certain of
said containers from the conveyor based on digital watermarks
formed by texturing on their plastic surfaces, wherein at least one
of said containers conveys a first digital watermark formed by
printing and a second digital watermark formed by plastic
texturing, said first and second digital watermarks having
different signaling protocols.
4. A method of sorting plastic items in a material stream,
including the acts: decoding first payload information from a
machine-readable indicia on a first item, the decoded first payload
information including a plastic recycling code indicating a type of
plastic used in the first item; sorting the first item from said
waste stream based on said plastic recycling code included in the
first payload information; decoding second payload information from
a machine-readable indicia on a second item, the decoded second
payload information omitting a plastic recycling code, but
including an identifier code; based on said identifier code,
obtaining a corresponding plastic recycling code from a database;
and sorting the second item from said waste stream based on the
plastic recycling code obtained from the database.
5. A container including a first digital watermark employing a
first signaling protocol, the payload of said first digital
watermark conveying data enabling a recycling system to determine a
type of plastic from which the container is manufactured,
characterized in that the container additionally includes a second
digital watermark employing a second signaling protocol different
than the first signaling protocol, the payload of said second
digital watermark also conveying data enabling said recycling
system to determine the type of plastic from which the container is
manufactured, wherein the first and second digital watermarks
differ in signaling protocol, yet both are useful for plastic
recycling.
6. The container of claim 5 in which the first digital watermark is
formed by 3D surface texturing, and the second digital watermark is
formed by ink printing, wherein the first and second digital
watermarks differ in formation and signaling protocol.
7. The container of claim 6 having front and back sides, wherein
the first digital watermark formed by 3D surface texturing is
present on both the front and back sides, and wherein the second
digital watermark formed by ink printing is present on both the
front and back sides.
8. The container of claim 6 in which the payload of said second
digital watermark also conveys additional data enabling a point of
sale apparatus to identify at least a name and a price for an item
sold in said container, the payload of the first digital watermark
lacking at least some of said additional data, wherein the first
and second digital watermarks differ in formation, payload and
signaling protocol.
9. The container of claim 5 in which the first digital watermark is
formed by 3D surface texturing, including plural corner
reflector-shaped indentations.
10. A recycling method that includes the acts: capturing imagery
depicting items in a material stream; extracting a first digital
watermark payload from imagery depicting a first item in said
material stream; based on payload data extracted from the first
digital watermark, determining that the first item is made with a
first type of reprocessable plastic; extracting a second digital
watermark payload from imagery depicting a second item in said
material stream; based on payload data extracted from the second
digital watermark, determining that the second item is made with a
second type of reprocessable plastic; and sorting said first and
second items from the material stream based on said determined
types of plastic.
11. The method of claim 10 in which the first type of reprocessable
plastic is polyethylene terephthalate, and the second type of
reprocessable plastic is high-density polyethylene.
12. The method of claim 10 that includes extracting the first
digital watermark payload from imagery depicting a label on the
first item.
13. The method of claim 10 that includes extracting the second
digital watermark payload from imagery depicting a 3D surface
texture on the second item.
14. The method of claim 10 in which the first item is a bottle.
15. The method of claim 10 in which the first digital watermark
payload extracted from the depiction of the first item includes a
fixed message portion and a variable message portion, and said
variable message portion includes plural fields, wherein one of
said fields is a global trade item number (GTIN) field.
16. The method of claim 10 in which the first watermark payload
includes a code identifying a plastic used in the first item, and
the second digital watermark payload includes linking data, wherein
the method further includes obtaining a code identifying a plastic
used in the second item from a database, through use of said
linking data.
17. The method of claim 10 in which the captured imagery comprises
image frames, and the method includes analyzing each of plural
pixel blocks in each of said image frames to find a clue suggesting
presence of watermark data, and undertaking further image analysis
when a clue is found, wherein the first digital watermark payload
is extracted as a consequence of finding a first clue, and the
second digital watermark payload is extracted as a consequence of
finding a second clue.
18. The method of claim 17 in which at least two of said plural
pixel blocks in an image frame partially overlap each other.
19. The method of claim 17 in which the first clue comprises
detecting a region of pixels each having a value above a threshold
value.
20. The method of claim 17 in which the first clue comprises
detecting an ensemble of spatial image frequencies corresponding to
a watermark reference signal.
21. The method of claim 17 in which the first clue comprises an
output from a classifier indicating that a pixel block likely
depicts a plastic item.
22. The method of claim 17 in which the first clue comprises an
output from a classifier indicating that a pixel block likely does
not depict a conveyor belt.
23. (canceled)
24. The method of claim 10 in which the material stream is moved by
a conveyor belt past a camera, wherein items on the conveyor belt
enter a camera frame field of view along a first edge of the camera
frame, and the method includes analyzing plural overlapping blocks
spanning said first edge of the camera frame for clues indicating
possible presence of watermark data, and undertaking further image
analysis when a clue is found.
25. The method of claim 10 in which the first digital watermark is
a watermark encoded according to a signaling protocol that
represents payload data using a tile of 128.times.128 elements, and
in which the second digital watermark is a watermark encoded
according to a second, different, signaling protocol, which
represents payload data using a tile of N.times.N elements, where
N<128.
26. (canceled)
27. The method of claim 10 that includes: assessing first and
second pixel patches in the captured imagery to determine whether
said patches likely depict a conveyor belt; watermark-processing
the first pixel patch as a consequence of said assessing
determining the first pixel patch likely does-not depict the
conveyor belt; and not watermark-processing the second pixel patch
as a consequence of said assessing determining that the second
pixel patch likely does depict the conveyor belt; wherein the first
digital watermark is extracted from imagery comprising the first
pixel patch.
28. The method of claim 10 that includes illuminating a first
region of the material stream with a first illumination source, and
illuminating a second region of the material stream with a second
illumination source of a type different than the first illumination
source, wherein the first digital watermark is extracted from
depiction of the first item when illuminated by the first
illumination source, and the second digital watermark is extracted
from depiction of the second item when illuminated by the second
illumination source.
29. The method of claim 28 in which the first illumination source
emits illumination of a first color, and the second illumination
source emits illumination of a second, different, color.
30. The method of claim 28 in which the first illumination source
emits illumination of a first polarization state, and the second
illumination source emits illumination of a second, different,
polarization state.
31. (canceled)
32. The method of claim 10 in which the material stream is moved by
a conveyor belt at a speed, and the captured imagery comprises an
image sequence captured at a frame rate, the sequence including
first, second and third frames, the method includes the acts:
analyzing plural image blocks in the first frame for a watermark
clue; detecting a watermark clue in a first image block of the
first frame; and identifying, in the second frame, one or more
second image blocks for analysis based on said conveyor belt speed
and said frame rate.
33. The method of claim 32 that further includes identifying in the
third frame one or more third image blocks for analysis based on
said conveyor belt speed and frame rate.
34. The method of claim 10 in which the first digital watermark
includes a first reference signal and the second digital watermark
includes a second reference signal, wherein the first reference
signal has a correlation 0.2>r>-0.2, when confusion-tested
with the second reference signal over a full range of affine
transformations, namely with scales of the first reference signal
ranging between 0.5 and 2.0 in increments of 0.02, and with
rotations of the first reference signal ranging between -90 degrees
through +90 degrees in increments of 1 degree, and with
translations of the first reference signal ranging across every
pixel of possible relative translation.
35-41. (canceled)
42. The method of claim 10 in which each of said first and second
items is marked with two different types of watermarks, namely (a)
a first type of watermark printed on either the item or on a label
applied to the item, said first type of watermark employing a first
signaling protocol, and (b) a second type of watermark formed as a
3D texture in the surface of the item, said second type of
watermark employing a second signaling protocol different than the
first signaling protocol; wherein the method includes applying
first and second different watermark reading algorithms to the
captured imagery, the first watermark reading algorithm being
adapted to read watermarks employing the first signaling protocol,
and the second watermark reading algorithm being adapted to read
watermarks employing the second signaling protocol; wherein the
method can sort the first and second items from the material stream
based on either the first or second types of watermarks that mark
said items, by payload data extracted by said respective first or
second watermark reading algorithms.
43. The method of claim 42 that further includes applying said
first and second different watermark reading algorithms to a common
patch of pixels, wherein said common patch of pixels is analyzed
both for the presence of the first and second digital
watermarks.
44. The method of claim 42 in which the first watermark reading
algorithm performs geometric synchronization using a first
reference signal, and the second watermark reading algorithm
performs geometric synchronization without using said first
reference signal.
45. The method of claim 44 in which the second watermark reading
algorithm performs geometric synchronization using a second
reference signal different than the first reference signal.
46. The method of claim 42 in which the first signaling protocol of
the first type of digital watermark represents the first payload by
elements at positions in an array of encoding locations of a first
size, namely 128.times.128 encoding locations, and the second
signaling protocol of the second type of digital watermark
represents the second payload by elements at positions in an array
of encoding locations of a second different than the first
size.
47. The method of claim 42 in which the first signaling protocol of
the first digital watermark produces encoded square block patterns
having a side dimension of 0.85 inches, and the second signaling
protocol of the second digital watermark produces encoded square
block patterns having a side dimension smaller than 0.85
inches.
48. The method of claim 42 in which the first signaling protocol of
the first digital watermark has a first payload capacity, and the
second signaling protocol of the second digital watermark has a
second, different, payload capacity.
49. The method of claim 42 in which the first watermark reading
algorithm performs a first type of error correction decoding, and
the second watermark reading algorithm performs a second type of
error correction decoding different than said first type of error
correction decoding.
50. The method of claim 42 in which the first watermark reading
algorithm employs a first key for descrambling, and the second
watermark reading algorithm employs a second, different, key for
descrambling.
51. The method of claim 42 in which the first watermark reading
algorithm employs a first spreading sequence for demodulation, and
the second watermark reading algorithm employs a second, different,
spreading sequence for demodulation.
52. The method of claim 42 in which the first watermark reading
algorithm employs a first scatter table, and the second watermark
reading algorithm employs a second, different, scatter table.
53. The method of claim 4 that includes decoding the first payload
information from the first item using a first digital watermark
reading algorithm, and decoding the second payload information from
the second item using a second digital watermark reading algorithm,
the method further including sorting the first item to a first
destination based on said plastic recycling code included in the
first payload information, and sorting the second item to a second
destination based on said plastic recycling code obtained from the
database, wherein: the second destination is the same as the first
destination; the first and second plastic items are two instances
of the same type of item; each of said items is marked with both a
first digital watermark and a second digital watermark, the first
watermark being formed by a first process and the second watermark
being formed by a second process different than the first process,
and the first watermark employing a first signaling protocol and
the second watermark employing a second signaling protocol
different than the first signaling protocol; and the first digital
watermark reading algorithm is different than the second digital
watermark reading algorithm, said difference including that the
first algorithm is adapted to read watermarks employing the first
signaling protocol, and the second algorithm is adapted to read
watermarks employing the second, different, signaling protocol;
wherein an item can be sorted from the material stream based on
reading of either the first or second watermarks, by said
respective first or second watermark reading algorithms.
54-65. (canceled)
66. The method of claim 10 in which the captured imagery includes a
frame of imagery, and the method includes: examining a plurality of
pixel blocks arrayed across said frame for clues indicating pixel
blocks that may depict watermark data, said plurality of pixel
blocks including a first pixel block, the pixel blocks included in
said plurality having a first pixel spacing; finding a watermark
clue in the first pixel block, and as a consequence analyzing the
first pixel block and also analyzing N pixel blocks around the
first pixel block, said N pixel blocks having a second pixel
spacing less than the first pixel spacing; and extracting the first
or second digital watermark payloads from one of said analyzed
pixel blocks.
67-74. (canceled)
75. The container of claim 5 including a printed label, the label
being printed to convey said first digital watermark and encoding a
payload P1 for sensing by a point of sale scanner in a retail
store, the first digital watermark including a first reference
signal that enables geometric registration of the first digital
watermark for decoding by said scanner, the first reference signal
comprising a first constellation of peaks in the 2D Fourier
magnitude domain, said container further including a surface shaped
to convey said second digital watermark and encoding a payload P2
that is different than P1, for sensing by a recycling apparatus,
the second digital watermark including a second reference signal
that enables geometric registration of the second digital watermark
for decoding, the second reference signal comprising a second
constellation of peaks in the 2D Fourier magnitude domain that is
different than the first constellation of peaks, wherein the first
digital watermark is encoded as a pattern of variations in local
luminance or chrominance, and the second digital watermark is
encoded as a 3D texture pattern; and wherein the different
constellation of peaks in the second reference signal prevents a
point of sale scanner that is attempting to read the payload P1 in
a retail store from being confused by the presence of the second
digital watermark.
76. The container of claim 75 wherein some but not all of the peaks
in the second constellation overlap with peaks in the first
constellation.
77. The container of claim 75 wherein none of the peaks in the
second constellation overlaps with a peak in the first
constellation.
78. The container of claim 77 wherein each of the peaks in the
first constellation lies on a different radial line in a 2D Fourier
magnitude plot of the first reference signal, and none of the peaks
in the second constellation lies on one of said radial lines.
79-103. (canceled)
104. The system of claim 1 in which the watermark reading means
comprises a first watermark extraction module for extracting first
digital data from patterns of a first format formed in the surfaces
of textured plastic objects, the first watermark extraction module
having an output coupled to the sorter, enabling the sorter to sort
the objects based on said first digital data, the watermark reading
means further comprising a second watermark extraction module for
extracting second digital data from patterns of a second,
different, format printed on said textured plastic objects, the
second watermark extraction module having an output coupled to said
sorter, enabling the sorter to sort the objects based also on the
second digital data, wherein the second watermark extraction module
is unable to extract digital data from patterns of the first
format.
105-119. (canceled)
Description
RELATED APPLICATION DATA
[0001] This application claims priority to provisional applications
62/854,754, filed May 30, 2019, 62/845,230, filed May 8, 2019,
62/836,326, filed Apr. 19, 2019, 62/830,318, filed Apr. 5, 2019,
62/818,051, filed Mar. 13, 2019, 62/814,567, filed Mar. 6, 2019,
and 62/812,711, filed Mar. 1, 2019.
[0002] This application is also a continuation-in-part of
application Ser. No. 15/823,138, filed Nov. 27, 2017 (published as
20180338068), which is a continuation of application Ser. No.
14/611,515, filed Feb. 2, 2015 (published as 20150302543), which
claims priority to provisional application 61/934,425, filed Jan.
31, 2014.
[0003] The disclosures of the above applications are incorporated
herein by reference.
BACKGROUND
[0004] There is a growing need to increase the percentage of
plastic items that are reused or recycled.
[0005] Applicant's document US20040156529 teaches that a plastic
surface of a 3D object can be textured by thermoplastic molding to
form a machine-readable digital watermark that conveys a plural-bit
payload.
[0006] Applicant's document US20040086151 teaches how a
digitally-watermarked 3D object can be produced using injection
molding (e.g., using vacuum forming or pressure forming).
[0007] Applicant's document US20020099943 teaches that an object
can convey two watermarks, one formed in the topology of the object
surface, and one formed by printing.
[0008] Applicant's document US20150016712 teaches that a 3D object
can be identified using watermark or image fingerprint data, and
this identification data can link to recycling information for the
object. For example, the identification data can link to a
recycling code indicating whether the object is formed of
polyethylene terephthalate, high-density polyethylene, polyvinyl
chloride, etc.
[0009] Applicant's document US20150302543 similarly teaches that
the payload of a watermark formed on a plastic object can convey,
or can link to, recycling codes for the object. Moreover, the '543
document teaches that a camera-equipped waste sorting apparatus can
sort incoming material streams based on decoded watermark data.
Document US20180345323, by FiliGrade B.V., also discloses sensing
recycling information from watermarked plastic bottles, and
separating a waste stream based on the decoded information.
[0010] Applicant's present work improves on the foregoing
technology, and provides many additional features and
advantages.
BRIEF DESCRIPTION OF THE DRAWINGS
[0011] FIGS. 1A-1Q illustrate a few different forms into which a
plastic surface can be shaped to convey digital watermark data.
[0012] FIG. 2A shows a form into which a plastic surface can be
shaped to convey trinary digital watermark data.
[0013] FIG. 2B shows another form into which a plastic surface can
be shaped to convey multi-state, or continuous tone, digital
watermark data.
[0014] FIGS. 3A, 3B, 3C and 3D show Voronoi, Delauney, traveling
salesman, and bricks patterns.
[0015] FIG. 4 shows peaks defining a reference signal for a printed
digital watermark, in the spatial frequency (Fourier magnitude)
domain.
[0016] FIG. 4A shows an enlargement from FIG. 4, showing radial
lines passing through the different peaks.
[0017] FIGS. 5A-5C show various reference signals that can be used
with a plastic texture digital watermark, tailored to avoid
interference with the printed watermark reference signal of FIG.
4.
[0018] FIGS. 6A-6C show spatial domain (pixel domain) counterparts
to the reference signals of FIGS. 5A-5C, respectively.
[0019] FIG. 7 shows a block of a "snakes" watermark pattern, with
which a mold for a plastic bottle can be shaped.
[0020] FIGS. 8, 9, 10 and 11 show different systems for capturing
imagery of items on a conveyor in a recycling apparatus.
[0021] FIG. 12 shows an arrangement providing variable
illumination, to effect contrast enhancement.
[0022] FIG. 13 details certain of the processing performed in an
illustrative recycling apparatus.
[0023] FIG. 14 provides still further details about the
illustrative recycling apparatus of FIG. 13.
[0024] FIG. 15 shows blocks arrayed in overlapping fashion across
an image frame, employed in certain embodiments of the
technology.
[0025] FIG. 16 shows a dense cluster of neighboring blocks that are
each analyzed when a watermark signal is found in a block, in
certain embodiments of the technology.
[0026] FIG. 17 shows the dense cluster of FIG. 16 in the context of
the image frame of FIG. 15, positioned where one of the blocks was
found to include a reference signal.
[0027] FIG. 18 shows a dense cluster of neighboring blocks that are
each analyzed when a glare region is detected in an image frame, in
certain embodiments of the technology.
[0028] FIG. 19 shows a sub-block histogram used in one method for
triggering block analysis.
[0029] FIG. 20 helps illustrate a different method for triggering
block analysis.
[0030] FIG. 21A shows blocks arrayed along an entry side of an
image frame, and a dense cluster of neighboring blocks positioned
where one of the entry blocks detected a watermark reference
signal, as may occur in certain embodiments of the technology.
[0031] FIG. 21B shows a frame captured shortly after the FIG. 21A
frame, with the cluster of analysis blocks having progressed down
the frame, in correspondence with conveyor movement.
[0032] FIG. 21C shows a frame captured shortly after the FIG. 21B
frame, showing detection of a watermark reference signal, which
spawns a second cluster of blocks.
[0033] FIG. 21D shows a frame captured shortly after the FIG. 21C
frame, showing the two clusters of analysis blocks having
progressed further down the frame, in accordance with conveyor
movement.
[0034] FIG. 22 illustrates intraframe combination of waxel
data.
[0035] FIG. 23 illustrates another example of intraframe
combination of waxel data.
[0036] FIG. 24 illustrates a camera/lighting arrangement employing
transmission of light through a plastic item to a camera.
[0037] FIG. 25 illustrates determining the center of gravity of an
object (here a transparent, clear, beverage bottle), by positions
of blocks in which watermark reference signals are detected.
INTRODUCTION
[0038] There is a growing need for high-reliability identification
of plastic items, e.g., for sorting waste streams.
[0039] Digital watermarks are advantageous for this purpose because
they can be applied to various types and shapes of materials.
Further, watermarks can be spread over a container and its labels
in ways that enhance readability even when the object is damaged,
soiled or partially occluded.
[0040] Digital watermarks provide 2D optical code signals that
enable machine vision in waste sorting systems, to ascertain the
types of materials in each object and sort the waste stream
accordingly. Encoded signals imparted into containers via 3D
printed molds, laser textured molds, and etched molds, as discussed
below, can be used to sort containers in various recycling
environments.
[0041] In accordance with one aspect of the present technology,
plastic items are encoded with two different watermarks. One
watermark is printed--typically by ink--on a label applied to the
item (or is printed on the item itself), and one is formed by 3D
texturing of the plastic surface.
[0042] The printed watermark commonly conveys a retail payload,
which is designed primarily for use by a point-of-sale terminal
scanner, e.g., containing or pointing to a product name, price,
weight, expiration date, package date, etc., so as to identify and
price an item at a retail checkout. The texture watermark commonly
includes a payload useful for recycling, e.g., containing or
pointing to data relating to the plastic. Each watermark typically
lacks some or all of the information conveyed by the other
watermark.
[0043] Importantly, in most embodiments the two watermarks (retail
and recycling) employ different signaling protocols. Applicant has
found that a typical point-of-sale retail scanner has only a very
short interval of time within which to read the retail watermark
before the next frame of imagery arrives for analysis. If retail
and recycling watermarks are depicted in the same image frame, they
must be quickly distinguishable--else the scanner may not succeed
in decoding the retail watermark before the next image frame
arrives. Part of the present specification teaches how the two
watermarks can be made readily distinguishable, so the retail
scanner doesn't waste precious milliseconds trying to decode a
recycling watermark, thereby helping ensure reliable retail
checkout operations.
[0044] The primary method by which retail and recycling watermarks
are made readily distinguishable is by use of different signaling
protocols (including, e.g., different reference signals, different
encoding protocols, and/or different output formats). By such
differences, a point-of-sale scanner can reliably discern a retail
watermark, while a recycling system can reliably discern a
recycling watermark, without risk of the point-of-sale scanner
accidentally expending effort trying to decode a payload from the
recycling watermark--leading to confusion.
[0045] Despite the difference in watermark signaling protocols, the
recycling system is desirably also configured with a watermark
processing module adapted to read the retail watermark (as well as
the recycling watermark), and to discern information from the
retail watermark usable for plastic recycling purposes (commonly by
reference to a database that associates retail watermark payload
data to plastic information). Thus regardless of which watermark is
read from an item by the recycling system, the system obtains
information to control proper item sorting by plastic type.
[0046] As noted, the two watermarks' signaling protocols can differ
in multiple manners, e.g., including the reference signals, and/or
the encoding algorithms used. The reference signal of each
watermark (sometimes termed a calibration signal, a synchronization
signal, a grid signal, or a registration signal) serves as a
synchronization component that enables the geometric pose of the
watermark, as depicted within captured imagery, to be discerned, so
that the payload can be extracted correctly. An exemplary reference
signal is a constellation of plural peaks in the spatial frequency
domain. A first of the two watermarks commonly includes a first
reference signal, and the other watermark commonly lacks this first
reference signal. (The latter watermark may include a different
reference signal, e.g., comprised of different frequencies of
peaks, different phases of peaks, and/or a different number of
peaks.)
[0047] The encoding algorithms can differ in the process by which
the data is encoded, and/or the format by which the encoded data is
expressed. For example, the printed watermark encoding algorithm
may employ a signaling protocol in which the resulting watermark
format is a square block that measures 0.85 inches on a side, with
a waxel resolution of 150 per inch, and conveying data structured
as a 128.times.128 array of element locations. In contrast, the
signaling protocol employed by the texture watermark encoding
algorithm may yield square blocks of a different size (typically
less than 0.85 inches on a side), with a waxel resolution different
than 150 per inch, and/or conveying data structured as other than a
128.times.128 array. The two different signaling protocols employed
in the two watermarks may have different payload capacities, e.g.,
one having a variable message portion that can convey 48 bits, and
one having a variable message portion that can convey just a half
or a third of this payload capacity.
[0048] The two encoding algorithms can additionally or
alternatively differ by error correction coding method used (if
any), the redundancy rate employed, the number of bits in the
signature string output by the error correction coder, the CRC
method used (if any), the scrambling key used to scramble the
signature string output from the error correction coder to yield a
scrambled signature string, the spreading key used to produce a
multitude of randomized "chips" from each bit of the scrambled
signature string, the scatter table data that defines spatial
placement of each of these "chips" in the output watermark pattern,
etc. The decoding algorithms can differ correspondingly.
[0049] The inability of one watermark reader (e.g., a point-of-sale
retail watermark reader) to read a watermark of the other type
(e.g., a recycling watermark) may be due to any of the just-noted
differences between the watermarks, e.g., concerning their
geometric reference signals, output formats, signaling protocols,
encoding/decoding algorithms, etc.
[0050] Each watermark payload typically includes fixed and variable
message portions. The fixed portion typically includes data
identifying the signaling protocol used. The variable message
portion commonly includes plural fields. For the printed retail
watermark, one field typically conveys a global trade item number
(GTIN), while other fields can convey Application Identifier codes
(e.g., indicating weight, expiration date, etc.) as defined by GS1.
Plastic identification information may be conveyed in a printed
retail watermark in the form of such an Application Identifier
code, although such an AI code is not presently part of the GS1
standard.
[0051] Some recycling systems employ two watermark readers, a first
reader configured to apply a first watermark reading algorithm
(e.g., to read a retail watermark employing a first signaling
protocol) and a second reader configured to apply a second,
different, reading algorithm (e.g., to read a recycling watermark
employing a second signaling protocol). Each such reader is unable
to read watermarks of the other type. Other recycling systems
employ a single reader, configured to read both types of
watermarks. Still other systems employ hybrid arrangements, with
certain components shared (e.g., performing a common FFT
operation), and other components dedicated to one type of watermark
or the other.
[0052] To assure reliable reading of the watermarks regardless of
position of an item in a waste stream, the watermarks are
preferably visible from multiple item viewpoints. For example, the
recycling, texture watermark is desirably formed on several
surfaces of each item, including front and back. Likewise, the
retail, printed watermark is desirably formed on opposing sides of
each item, e.g., on both front and back labels.
[0053] To apply watermark reading efforts most effectively, certain
embodiments of the present technology examine image pixel blocks
for clues that suggest presence of watermark data. Further
watermark analysis is undertaken only for image blocks where such a
clue is found. Many such clues are detailed, including detecting a
glare spot (a region of pixels each having a value above a
threshold value), detecting an ensemble of spatial image
frequencies corresponding to a watermark reference signal, a
classifier output indicating that a pixel block likely depicts a
plastic item, a classifier output indicating that a pixel block
likely does not depict a conveyor belt, a determination that pixels
from a majority of sub-blocks within a block have an average value
within 1, 2, 3 or 4 digital numbers of a histogram peak that is
based on previous imagery, detecting signals associated with
conveyor belt markings, detecting salt/pepper markings, and various
other techniques for distinguishing promising image blocks from
others. When a patch of promising imagery is identified, it is
typically analyzed for presence of both the retail and recycling
watermarks.
[0054] In some embodiments, when one block of imagery is found to
be promising, that determination also triggers examination of
plural nearby image blocks. An incoming frame of imagery may
initially be divided into blocks at a first density (e.g. with
first pixel spacing or overlap). When a promising block is found,
other blocks are examined at a greater density, e.g., spaced from
the promising block at a smaller pixel spacing, or with greater
overlap. Relatedly, when a promising block is found in one frame,
then a different location can be analyzed for watermark data in a
following frame--corresponding to predicted movement of the item
depicted in the promising block--based on conveyor speed and frame
capture rate.
[0055] Due to analysis of imagery for both types of watermarks,
sometimes two instances of the same type of object (e.g., two
identical 12 oz. Pepsi bottles) will be sorted based on reading of
two different watermarks. That is, the first bottle's plastic type
may be identified by its printed watermark, and the second bottle's
plastic type may be identified by its texture watermark. Both will
be diverted to the same recycling destination, despite the reading
of different watermarks.
DETAILED DESCRIPTION
[0056] Digital watermark information is read from image data
depicting plastic objects in a waste stream. The information can
indicate the type of plastic (e.g., polyethylene terephthalate,
high-density polyethylene, low-density polyethylene, polypropylene,
polycarbonate, etc.), or it can convey other information useful in
recycling. Diverters and other mechanisms in automated sorting
systems are controlled in accordance with such watermark
information, to direct plastic objects to appropriate destinations
for recycling or reuse.
[0057] Digital watermarks (hereafter watermarks) are printed on
packaging for many products, and commonly serve to encode a Global
Trade Item Number, or GTIN, (much like the ubiquitous 1D UPC
barcodes), but in a visually non-intrusive manner. A point of sale
scanner in a retail store can detect and decode the watermark data,
use it to look up the product's identity and price, and add same to
a shopper's checkout tally. The watermark data is typically
organized in square blocks that are redundantly tiled--edge to
edge--spanning some or all of the printing on the product. Because
the watermark data is spatially dispersed, the scanner can read the
data from different views of the product (e.g., from front and back
views of a drink bottle).
[0058] Most typically, watermark data is hidden as subtle
variations to the luminance and/or chrominance of pixels comprising
the artwork for a package. Sometimes a watermark can take the form
of an inconspicuous pattern of dots that may be spread, e.g.,
across an adhesive label applied to a plastic fresh food
container.
[0059] To hold costs down, point of sale scanners generally use
simple processors. Such scanners commonly devote most of their
effort to finding and decoding 1D barcodes; watermark reading is
sometimes an afterthought. A scanner that captures 30 frames per
second has only 33 milliseconds to process each frame, and devotes
most of that time to barcode reading. Only a few milliseconds are
available for watermark reading.
[0060] Watermark reading has two parts: finding a watermark, and
decoding the watermark.
[0061] In an illustrative implementation, finding the watermark
(sometimes termed watermark detection) involves analyzing a frame
of captured imagery to locate a known reference signal. The
reference signal can be a characteristic constellation of peaks in
the 2D Fourier magnitude domain (a.k.a. the spatial frequency
domain). In the spatial (pixel) domain, such a reference signal
takes the form of an ensemble of summed 2D sinusoids of different
spatial frequencies spanning a watermark block. FIG. 5A shows an
illustrative reference signal in the Fourier magnitude domain; FIG.
6A shows such the same reference signal in the spatial domain. The
frequencies are desirably integer-valued, so as to assure
continuity along edges of watermark blocks. When an object bearing
such a known reference signal is depicted in captured imagery, its
particular presentation reveals the scale, rotation, and
translation of the watermark payload data that is also present in
that imagery. The watermark payload data is encoded by watermark
elements ("waxels") that occupy locations in a 2D array--commonly
of size 128.times.128 elements. This array may span an area of,
e.g., 0.85 or 1.7 inches on a side, depending on whether the
watermark is formed at a resolution of 150 or 75 waxels per inch
(WPI). Such blocks are tiled in a repeating array across the
packaging, together with the reference signal.
[0062] Once the scale, rotation, and translation of the watermark
are known from analysis of the reference signal as depicted in
captured imagery, the watermark payload can be decoded. A decoder
samples the captured imagery at locations corresponding to the
originally-encoded 128.times.128 array of data, and uses these
sample values in decoding the original watermark payload.
(Convolutional coding is commonly used, e.g., to convert a 48-bit
payload to a string of 1024 data, which are then redundantly spread
among the 16,384 locations of the 128.times.128 element watermark
block.)
[0063] These and other details of watermark technology are familiar
to artisans, including from the patent documents identified
herein.
[0064] In one particular embodiment of the present technology, a
plastic container conveys two watermarks--one formed by label
printing, and a second formed by texturing of the plastic surface,
as by molding. (The label can comprise a substrate that is printed
and applied to the container, or it can comprise printing applied
directly to the container.)
[0065] Plastic can be molded in various ways, including blow
molding, injection molding, rotational molding, compression
molding, and thermoforming. In each such process, a heated plastic
resin is shaped in accordance with a mold. By shaping the surface
of the mold with a pattern, a reciprocal pattern is formed on the
surface of the resulting plastic product. If the pattern in the
mold is tailored (e.g., by CNC milling, laser etching, or 3D
printing) to have the shape of a watermark pattern (with variations
in luminance/chrominance transformed to variations in mold height,
depth, angle, reflectance, or local curvature), then the resulting
plastic product will have a surface texture corresponding to that
watermark. Such pattern in the plastic surface can be sensed by
optical methods detailed below.
[0066] FIGS. 1A-1Q are illustrations of representative surface
textures.
[0067] Most of the depicted textures are illustrated as a 2D
section through a 3D surface--showing modulation in just one
dimension. For clarity of illustration, the textures are shown on a
flat surface. Naturally, most plastic containers are curved in at
least one dimension.
[0068] Also for clarity's sake, most of the illustrations of FIGS.
1A-1Q show marks that have just two states. Particular examples
include the "sparse" dot marks detailed in patent publications
US20170024840, 20190139176 and application PCT/US19/19410. Other
two-state marks include line art patterns such as Voronoi,
Delaunay, traveling salesman, and bricks, as detailed in
applications PCT/US18/64516 and 16/435,164, filed Jun. 7, 2019, and
respectively shown in FIGS. 3A, 3B, 3C and 3D. (The Voronoi pattern
is achieved by forming a mesh of glints (here triangles) with their
vertices at locations corresponding to a sparse array of dots. The
Delaunay pattern is the dual of the Voronoi pattern, in which the
glints take the form of polygons of different numbers of sides. The
traveling salesman pattern is achieved by defining a traveling
salesman path that visits each dot in a sparse array of dots. The
bricks pattern is achieved by placing a vertical line segments at
dot positions in an array of sparse dots, and forming horizontal
lines at intermediate positions, thereby defining rectangular
glints.)
[0069] FIG. 1A is labeled to show the binary state of each waxel. A
"1" is here represented by a relatively-raised excerpt; a "0" is
represented by a nominal, baseline elevation of the plastic surface
(which may be viewed as relatively-depressed, compared to the
depicted "1" state). The nominal elevation of the plastic surface
is shown by the dashed line.
[0070] FIG. 1B is like FIG. 1A, but the sharp corners have been
rounded (e.g., by low pass filtering) to aid in release of the
shaped plastic from the mold. Such rounding can be used in any
embodiment, to smooth sharp angles.
[0071] FIG. 1C shows an embodiment in which the transitions between
states are sloped, and successive "1" values include a brief return
to the opposite, nominal surface level. FIG. 1D is a variant of
FIG. 1C. Sloping the transitions further aids in mold release, and
can aid in optical detection, depending on the illumination.
[0072] In some embodiments, the raised protrusions in FIGS. 1B, 1C
and 1D each can be a rounded hump, with only a small--or no--flat
area at its peak.
[0073] FIG. 1E shows that the "1" state can be characterized by a
surface that is non-parallel to the nominal flat surface that
characterizes the "0" state. FIG. 1F is a variant on FIG. 1E,
showing that the "1" state needn't be elevated; it can simply be
tilted.
[0074] FIG. 1G shows an arrangement in which the "1" and "0" states
are each tilted, in different directions, relative to the nominal
surface of the plastic. (The tilt directions may be 180 degrees
apart, as shown, or they may differ by 90 degrees.) Such tilts
cause light to be preferentially-reflected in different directions,
making the mark more conspicuous to the watermark reader.
[0075] While FIGS. 1A-1G have been described and illustrated as
comprising excerpts that rise above the nominal surface, it will be
recognized that such encoding can similarly (and perhaps more
commonly) comprise excerpts that are depressed below the nominal
surface. (Watermark encoding/reading is typically agnostic as to
polarity--up or down.) An example is in forming the lines used in
the patterns of FIGS. 3A-3D, and FIG. 7. Combinations of elevations
and depressions can also naturally be used.
[0076] FIG. 1H illustrates an advantageous dispersion phenomenon
associated with curved surfaces. For most placements of the camera
and light source, for a curved surface, the incident light (shown
by the larger arrows) will be reflected from the surface at a
diversity of angles (show by smaller arrows), and some of it will
be reflected towards the camera, creating a bright glint. In
contrast, for a flat surface, essentially all of the incident
illumination will be reflected in a single direction--most likely
away from the camera. Thus, flat surfaces will usually appear dark
to the camera, while curved surfaces will usually be characterized
by a bright glint. (If, by chance, the flat surfaces reflect
towards the camera, then an "inversion" occurs, with the flat
surfaces being brighter than the curved surfaces.)
[0077] FIG. 1I shows dispersion and focusing phenomena associated
with surfaces that have both bumps and pits. The bumps act as
described above--dispersing incident light over a wide range of
angles. In contrast, curved pits act as focusing elements. Compared
to the dispersion caused by the bumps, the focusing caused by the
pits causes a greater amount of light to be reflected in the
general direction of the light source. Assuming the camera is
relatively close to the light source (e.g., within ten degrees as
viewed from the illuminated surface), the pits will appear brighter
than the bumps, in camera-captured imagery. (The dashed line shows
the nominal plastic surface.)
[0078] (Here and elsewhere, it will be understood that the light
source and camera can be placed other than as indicated by the
Figures. They can be closely-spaced, e.g., within single-digit
degrees) or further spaced. The light can illuminate the surface
straight-down (90.degree. incidence), or obliquely, such as with
incidence angles of 80.degree., 60.degree., 30.degree. or
less.)
[0079] The FIG. 1I arrangement can be extended to three surface
features, bumps, pits and flats, as shown in FIG. 1J. The flat
surfaces will reflect as described in connection with FIG. 1H. The
FIG. 1J arrangement thus is an example of a surface that can be
used for trinary signal encoding, variously reflecting a medium
amount of light (i.e., the glints caused by the bumps), a larger
amount of light (i.e., the focused reflections caused by the pits),
and an extreme value (typically dark, but sometimes bright, caused
by the flat areas).
[0080] FIG. 1J also illustrates another aspect of surface shaping,
which can be used in any embodiment: the protrusions need not be
dimensionally-similar to the indentations. In this example, the
raised bumps are taller than the pits are deep. Relatedly, the
raised bumps have a smaller radius of curvature than the depressed
pits. The opposite can also be the case.
[0081] FIG. 1K illustrates that modulating the surface height can
have little--if any--effect on the reflected light pattern. What
often matters are the transitions in surface height, that is, the
derivative of the function that defines surface height.
[0082] In FIG. 1K, light incident at point B on the plastic surface
will reflect with the same direction and intensity as light
incident at point D. The two surfaces are at different elevations,
but are parallel. In contrast, light incident at point A will
reflect with a different intensity and direction than light
incident at point C. At point A, the surface derivative is negative
(the height diminishes with increasing movement to the right). At
point C, the surface derivative is positive. Assuming the camera is
located near the light source, virtually no incident light will
reflect back towards the camera from point A, whereas virtually all
of the incident light will reflect back towards the camera from
point C. The flat-bottomed pit shown in cross-section in FIG. 1K
thus has three reflective zones: one zone along the flat bottom,
one zone with a negative derivative, and one zone with a positive
derivative. With the light source positioned as shown (and the
camera nearby) a glint of reflection will be sensed by the camera
from the latter zone, with nil reflection from the first two.
[0083] Similar phenomena likewise arise from a flat-topped raised
bump, as shown in FIG. 1L. The left-most side of the bump has a
positive derivative, and reflects a glint of light back towards the
camera. The flat top does not reflect light back to the camera, nor
does the right-most side of the bump (with its negative
derivative).
[0084] (It will be understood that the described results depend on
the light source being positioned to the left side of the shaped
surface. If the light is positioned to the right side, some of the
results are inverted.)
[0085] Naturally, the shapes of the curves can be tailored to
optimize performance, e.g., favoring reflections in certain
directions relative to the incident light source.
[0086] Still another approach to surface texturing is to employ
retroreflective features, such as 3D corner reflector-shaped
indentations into the plastic. A 3D corner reflector has the
property that light is reflected back to its source, over a wide
range of incident angles. FIG. 1M shows this property in two
dimensions; it likewise extends to three dimensions.
[0087] Corner-shaped indentations can be formed in a plastic
surface where the watermark should have a bright appearance (e.g.,
a "1" state), and can be omitted where the watermark should have a
dark appearance (e.g., a "0" state). The deepest "point" of the
indentation can be rounded; what is important is that most of the
surface extents be perpendicular to each other.
[0088] FIG. 1N shows an excerpt from a square 128.times.128 waxel
watermark block, showing 16 waxels. Some are indented with a
retroreflective 3D corner reflector in accordance with the data
being encoded (e.g., representing a "1" signal); others are left
flat (e.g., representing a "0" signal). FIG. 1O shows an excerpt
employing triangular waxels, which organize in hexagonal arrays.
Again, some are indented with a retroreflective 3D corner reflector
in accordance with the data being encoded; others are not.
[0089] In variant embodiments, the two states of the signal tile
aren't expressed by corner reflector, or flat surface. Instead, a
corner reflector is formed at the location of every waxel. The two
states are distinguished by the treatment of the three orthogonal
surfaces (facets) defining the indented reflector. A "1" state is
characterized by smooth surfaces that reflect light with relatively
little scattering. A "0" state is characterized by textured (e.g.,
roughened or matte) surfaces that reflect light with relatively
more scattering. The first type of reflector is made to be
efficient; the second type of reflector is made to be inefficient.
Yet to human observers, the two features are essentially
indistinguishable, giving the surface a uniform-seeming texture.
FIG. 1P shows such an arrangement (with the roughened corner
reflectors shown by grey waxels).
[0090] If square waxels are formed in the plastic at a density of
75 per inch, then each waxel spans an area 0.0133 inches on a side.
So each corner reflector indentation has a width of this value or
less. With higher waxel densities, the dimensions get smaller.
[0091] Naturally, in retroreflective arrangements, the camera
should be positioned as close as practical to the light source,
desirably so that the angular distance between the two (as seen
from the conveyor) is less than ten degrees.
[0092] The texturing of some surfaces in the FIG. 1P arrangement
can be employed in other arrangements, including the other depicted
arrangements. That is, some areas of the plastic surface may be
roughened or matte-finished to increase scattering, while other
areas can be left smooth to provide more specular light reflection.
In some embodiments, the plastic surface has no depression or
elevation to encode watermark data. Rather, the encoding is
effected wholly by scatter-texturing of different regions, without
otherwise disturbing the nominal shape of the item.
[0093] FIG. 1Q shows, in a 3D view, a small excerpt of a planar
surface, marked with 3 sparse dots--here taking the form of
depressions into the surface.
[0094] While many of the illustrated surfaces can encode two
signaling states, and some can encode three states, more generally,
M-ary encoding can be used.
[0095] FIG. 2A shows another form of trinary encoding, in which the
signal is comprised of -1, 0 and 1 elements. A "-1" is represented
by a tilt in one direction, a "1" is represented by a tilt in
another direction, and a "0" is represented by a tilt midway
between the other two. Many other such forms can naturally be
devised, e.g., by including depressions from the nominal plastic
surface that mirror the protrusions of FIGS. 1A-1F. Quaternary
encoding can be realized using four different surface tilts, at
successive 90 degree angles. Quinary encoding can be realized by
using the four tilts of quadrature encoding, plus a fifth state
that is the plastic's nominal surface. M-ary encoding of still
higher orders can be realized by expanding the set of tilts.
[0096] (The surfaces in FIG. 2A, as in the other Figures, can be
roughened, e.g., with a matte or frosted finish, to scatter some
light in directions that are not purely reflective.)
[0097] In addition to M-ary encoding, the present technology is
also suited for use with so-called "continuous tone" watermarks,
which have a variety of intermediate states between two extrema.
Often the reference signal is continuously-valued (or valued by a
large set of quantized steps), and summation of such a reference
signal with a M-ary payload pattern representation yields a
continuous tone watermark. The continuous values of waxels in such
marks can be represented by the degree of localized surface height
or tilt. Such a mark is conceptually illustrated by FIG. 2B
[0098] The just-discussed patterns suggest that the shaping extends
to both surfaces of a plastic medium, e.g., top and bottom (or
inside bottle and outside). Sometimes, the shaping is of one
surface only (e.g., the outside), with the other surface being
smooth.
[0099] While plastic texturing using shaped molds is most common,
other forming approaches can be used. Laser or chemical etching is
an example, and results in a surface that is marked with
depressions, corresponding--by amplitude or tilt--to spatial
variations in a watermark signal. (Laser etching is well-suited for
serialization--in which each instance of an item is encoded
differently.)
[0100] In some embodiments, the plastic surface is locally treated
to render a matte or frosted finish, rather than a glossy finish.
In such case, the watermark itself can be formed as a pattern
comprised of matte and glossy waxels. Matte texture is achieved by
molding, or otherwise processing the surface, to achieve a measure
of surface roughness, e.g., vertical variation on the order of a
tenth or a half micrometer or larger.
[0101] In the exemplary embodiment, the plastic watermark is
tailored as to avoid confusion by the point of sale scanner. As
noted, such scanners have limited processing abilities, and limited
time to extract watermark identifiers. Several measures can be
taken to help prevent a point of sale scanner from attempting to
read a plastic watermark--an effort that wastes precious processing
time and may prevent the scanner from decoding a product GTIN from
a product label depicted in the same frame.
[0102] One measure to help avoid confusion by point of sale
scanners is to use a reference signal in the plastic watermark that
is unlikely to be mistaken for the reference signal used in the
printed label watermark. Such a reference signal can be developed
experimentally by randomly generating multiple candidate signals
(e.g., by picking a set of random peak locations in the spatial
frequency domain and assigning a random phase to each) and testing
each candidate to assess a likelihood that a point-of-sale
watermark reader will mistake such a signal for a printed label
watermark reference signal. The candidate reference signal with the
lowest likelihood of confusion is then used.
[0103] Another approach is similar, but involves mathematically
computing a theoretical confusion (correlation) metric indicating
similarity between different candidate random reference signals,
and the printed label reference signal, and picking the candidate
with the lowest correlation.
[0104] Applicant prefers the first approach since, as a sage once
quipped, in theory there is no difference between theory and
practice, but in practice there is.
[0105] The process for finding candidate reference signals for the
plastic watermark can be aided by applying different constraints in
the signal generation or selection process. One is that, desirably,
no peak in the plastic reference signal should be identical to any
peak in the printed label reference signal. Any randomly-generated
candidate plastic reference signal having such an attribute may be
discarded.
[0106] FIG. 4 shows the peaks of a printed label watermark
reference signal, in the 2D Fourier magnitude domain. The reference
signal of a plastic watermark should desirably have no peak
location in common.
[0107] Relatedly, in the printed label reference signal, each
frequency peak falls on a different radial line from the origin. A
few are shown in the enlargement of FIG. 4A. It is desirable that
no peak in the plastic reference signal be located on any of these
radial lines. (Depending on the scale with which watermarked
objects are viewed, the reference signal peaks move concentrically
towards and away from the origin, traveling on these radial
lines--risking confusion if both reference signals have a peak on
the same radial line.)
[0108] Applicant also prefers that no peak in the plastic reference
signal be on the vertical or horizontal axes, 31, 32, of the
spatial frequency plane. Many other features of the captured
imagery may have signal energy concentrated along these axes, so
peaks along such axes are best avoided.
[0109] The reference signal for printed labels is
quad-symmetric--mirror-imaged around the vertical and horizontal
frequency axes, and such arrangement may be used for the plastic
reference signal for reasons of detector efficiency. However, this
is not required, and a reference signal for the plastic watermark
that does not exhibit this attribute may be at less risk for
confusion.
[0110] While peaks along the vertical and horizontal axes are best
avoided, it is generally desirable that peaks for the plastic
reference signal be placed on radial lines at diverse angles. In
each quadrant of a quad-symmetric reference signal, a quarter to a
third of the peaks may be on different radial lines located within
30 degrees of the horizontal axis, a quarter to a third may be on
different radial lines located within 30 degrees of the vertical
axis, and a third to a half may be on different radial lines
located between these two ranges.
[0111] It is similarly desirable that the peaks for the plastic
reference signal be diverse in their distance from the origin. Low
frequency points are disfavored (e.g., below 20 or 25 cycles per
block), since scaling may cause them to shift to locations where
the watermark reading software is not looking for peaks (thus the
blank area in the center of FIG. 4), and reciprocally for high
frequencies (e.g., above 50 or 60 cycles per block). But within an
intermediate donut band (shown by the dashed circles in FIG. 4) a
spatial budget for allocating peaks can be used, as in the
preceding paragraph, to ensure a roughly uniform distribution.
[0112] Another measure to help avoid confusion by point of sale
scanners is to use a reference signal in the plastic watermark that
has a smaller number of peaks than the reference signal in the
printed label watermark. The fewer the number of peaks, the less
likely one will be mistaken for a peak in the printed label
watermark.
[0113] A corollary benefit is that in a plastic watermark reference
signal with fewer peaks, each can be encoded with more
energy--since the available signal energy budget is spread among
fewer features. A plastic reference signal comprised of fewer
strong peaks is less likely to result in confusion than a reference
signal comprised of more and weaker peaks.
[0114] A further guard against confusion of printed and plastic
watermarks is to form the marks at different scales. As noted,
printed watermarks are commonly formed at 75 or 150 waxels per inch
(i.e., watermark blocks of 1.7 or 0.85 inches square). Plastic
watermarks may be formed at a different resolution, such as 200,
250 or 300 waxels per inch (i.e., 0.64, 0.51 and 0.43 inches
square). So doing also aids in their detection from curved
container surfaces, since the apparent edge distortion of the
watermark pattern due to curvature diminishes with smaller
watermark block sizes.
[0115] One algorithm for generating a candidate plastic reference
signal is to take a Fourier magnitude plot of the label reference
signal with which confusion is to be avoided, and add two
circumscribing circles (as shown in FIG. 4) to define an annular
space within which all points should reside. Then add radial lines
extending from the center of the plot through each label reference
signal peak, to the outer circle, akin to FIG. 4A. Finally,
triangulate within the annular space, using the label reference
signal peaks as vertices, to define the largest triangles that
don't encompass any other peak. Then identify a point within the
annulus that is the most remote from the nearest straight line
(i.e., the radial lines, the triangulation lines, and the
horizontal and vertical axes) and add it to a set of candidate
points. Repeat until a desired number of points has been
identified.
[0116] Different candidate plastic reference signals can be tested
for potential confusion with the label reference signal by applying
different random distortions to each candidate signal--such as
tilt, rotation, and scaling, and additive Gaussian noise--and
determining how frequently the reference signal detection stage of
a point-of-sale watermark reader mistakes the distorted signal as
the reference signal for a label watermark. After each candidate
reference signal has been tested with several hundred different
distortions, one candidate signal will typically emerge as superior
to the others. (This signal may be examined, in the spatial domain,
by a human reviewer, to check that it has no
subjectively-objectionable attribute, but such review can also be
skipped.)
[0117] Several candidate plastic watermark reference signals are
shown in FIGS. 5A, 5B and 5C by their Fourier magnitude plots.
FIGS. 6A, 6B and 6C show their corresponding spatial domain
representations.
[0118] Confusion with the printed label watermark reference signal
tends to diminish with "flatness" of the spatial domain
representation of the plastic reference signal. Thus, in accordance
with another aspect of the present technology, each candidate
reference signal for the plastic watermark is varied, by trying
different phase assignments to different peaks in the Fourier
magnitude plot, to identify a set of phase assignments that
minimizes the standard deviation of pixels in the spatial domain
representation. This is a task well-suited to computer automation,
with a hundred thousand or a million different sets of phase
assignments being tried, to find the one set that yields the
spatial domain pattern with the smallest standard deviation.
[0119] While the reference signal patterns of FIGS. 5A-5C (and
6A-6C) were generated experimentally, e.g., using the techniques
detailed above, a check of their correlation with a printed label
watermark reference pattern finds a very small degree of
correlation r, with a maximum value of 0.2>r>-0.2 (and in
some instances 0.1>r>-0.1), when confusion-tested with the
printed label reference pattern over a full range of affine
transformations, i.e., at scales ranging between 0.5 and 2.0 in
increments of 0.02, and at rotations ranging between -90 degrees
through +90 degrees in increments of 1 degree, and at every pixel
of possible translation.
[0120] Correlation of two images f.sub.1 and f.sub.2, both of size
P.times.P pixels, can be expressed as:
r = 1 P 2 i = 1 P j = 1 P f 1 ^ ( x , y ) f 2 ^ ( x , y )
##EQU00001##
[0121] It will be understood that the detailed reference signals
are comprised of sinusoids of equal amplitude. In other
embodiments, the sinusoids can have different amplitudes, yielding
a more conspicuous "weave"-like pattern to their spatial domain
representations.
[0122] As noted, a reference signal is one of two elements of a
watermark signal, the other being an encoded representation of the
payload message. This representation can be generated by
convolutionally-encoding the symbols of the payload message to
generate a much longer string of symbols (e.g., 1024 bits), which
may be termed the signature. The signature may be randomized by
XORing with a scrambling key of the same length. Chips redundantly
representing each of the scrambled signature bits are randomly
spatially scattered among locations in a square array, e.g., of
128.times.128 (16,384) elements, to form a signature array.
[0123] A continuous-tone watermark can be generated by scaling the
spatial domain reference signal so its average pixel value is 128,
and then adding or subtracting to each component pixel value an
offset value that depends on whether the chip assigned to that
location is a 1 or a 0.
[0124] A sparse watermark can be generated by various methods,
which commonly involve producing an output pattern of spaced-apart
dots. Several methods are detailed in the cited documents, and are
discussed in the section entitled "Review of Exemplary Watermarking
Methods," below.
[0125] As noted above and illustrated by FIGS. 3A-3D, a sparse
pattern can be transformed into various bitonal line-based
representations. A further such pattern, termed snakes, is shown in
FIG. 7. It is generated from a continuous-tone watermark, per the
following algorithm executed using Adobe Photoshop and
Illustrator:
[0126] A monochrome (greyscale) block, at 300 DPI with a white
background, is filled with 50% grey, and encoded with a
continuous-tone watermark (reference and payload signals). The
image is then Photoshop-adjusted using the controls
Adjustment.fwdarw.Exposure: Default/Exposure: 1.05/Offset:
-0.075/Gamma Correction: 0.3. Next, filtering is applied using
Photoshop controls Filter.fwdarw.Blur.fwdarw.Gaussian Blur: Radius:
3 pixels; then
Filter.fwdarw.Stylize.fwdarw.Wind:Method=Wind/Direction: Right or
Left (doesn't matter). Then the image is thresholded by Photoshop
controls Image.fwdarw.Adjustment.fwdarw.Threshold: Threshold Level:
140 (+/-5). The resulting file is saved, and then opened in Adobe
Illustrator. The just-edited image is selected from inside the
layers. The "Image Trace" button in the main upper frame of the
Illustrator user interface is clicked, and after the preview
appears, the Image Trace Panel, next to the Drop Down frame showing
"default," is clicked. From the top line of the icons, the Outline
button is clicked. After the preview is presented, the "Expand"
button, next to the Drop Down frame showing "Tracing Result," is
clicked. This presents a UI enabling the size of the strokes in the
pattern to be made bolder or skinnier. Some bolding is applied to
produce a pattern like that of FIG. 7.
[0127] It will be seen that such a pattern is comprised of multiple
curved segments (many segments being compound-curved, i.e., with
multiple turns along their length), dispersed across an area, with
some segments crossing others, while others are independent--not
crossing others.
[0128] Turning now to the larger system, a recycling apparatus
according to one embodiment of the present technology employs one
or more cameras and lights sources to capture imagery depicting
watermarked plastic containers traveling on a conveyor in a waste
stream. Depending on implementation, the conveyor area imaged by a
camera system (i.e., its field of view) may be as small as about 2
by 3 inches, or as large as about 20 by 30 inches, or
larger--primarily dependent on camera sensor resolution and lens
focal length. In some implementations, multiple imaging systems are
employed to capture images that collectively span the width of the
conveyor. (A conveyor may be up to five feet or two meters in width
in a mass-feed system. Singulated-feed systems, in which items are
metered onto the conveyor one at a time, are narrower, e.g., 12
inches or 50 cm in width. Conveyor speeds of 1-5 meters/second are
common.)
[0129] FIG. 8 shows a simple arrangement in which a camera and a
light source are substantially co-located; that is, the
illumination comes from a position less than 10 degrees away from
projection of the camera's viewing axis onto the waste stream
conveyor (i.e., the camera target). In another arrangement, the
light source is positioned to obliquely illuminate the camera
target, i.e., the light source is oriented in a direction more than
50 degrees away from the orientation of the camera lens axis, as
shown in FIG. 9. In still another arrangement (FIG. 10), opposing
illumination is employed. That is, the axis of the light source has
an orientation more than 140 degrees away from the orientation of
the camera lens. In the latter arrangements, the surface texturing
can cause localized shadowing on the plastic surface, e.g., with
each plastic protrusion blocking light, causing an adjoining area
to be imaged with relatively less luminance than an area on which
the light is incident.
[0130] The positions of the camera and light source in each of
FIGS. 8-10 may be interchanged. In other embodiments, multiple
light sources can be used. Naturally, the exposure interval should
be short enough to avoid motion blur. A strobed light source aids
in avoiding blur. The light source can be as close to the conveyor
as the size of items passing underneath will allow, or it can be
spaced at a greater distance, two or four feet.
[0131] FIG. 11 shows an arrangement in which light sources of two
different colors, red and blue, illuminate the camera target from
opposite sides of the camera--at oblique angles (>50 degrees) in
this example. A green light source is co-located with the camera.
The camera in FIG. 11 is an RGB camera, including a 2D CMOS sensor
overlaid with a color filter in a Bayer pattern. Its raw output
includes red-filtered pixels, green-filtered pixels, and
blue-filtered pixels. Three distinct monochrome (greyscale) images
are thereby formed by corresponding pixels from the sensor
array--one depicting the waste stream in the red part of the
visible light spectrum, one depicting the waste stream in the blue
part of the spectrum, and one depicting the waste stream in the
green part of the spectrum.
[0132] The FIG. 11 arrangement shows the light sources arrayed
along the direction of conveyor (waste stream) movement. In an
alternative embodiment, the light sources are positioned
across--rather than in line with--the direction of conveyor
movement. In still other embodiments, a first pair of red/blue
light sources is positioned along the direction of conveyor
movement (as shown), and a second pair is positioned across the
direction of movement. The pairs of light sources are activated for
alternate frames of image capture by the camera (which may be
capturing frames, e.g., at 60 or 150 frames per second). One frame
is illuminated by in-line red/blue light sources, and the next
frame is illuminated by cross-wise red/blue light sources, etc.
Each frame is illuminated by the green light source.
[0133] Each of the resulting image frames is analyzed for watermark
data--looking both for printed label watermarks and plastic
watermarks. In some embodiments, a fourth image frame is produced
by computing a difference between red and blue pixel values in each
Bayer cell. The resulting difference values can be divided in half,
and summed with an offset value of 128, to ensure that elements of
the difference image are within a 0-255 range. This difference
image, too, is processed to decode any printed label or plastic
watermark that is present. Such arrangement in shown in FIG.
13.
[0134] While eight distinct watermark reading systems are shown in
FIG. 13, certain of the image processing may be consolidated, with
results shared between the label- and plastic-reading watermark
systems. For example, each red frame of data may be FFT'd by a
common FFT stage, and the results used in both label- and
plastic-reading (synchronization). However, if resolutions of the
two watermarks are different (e.g., 150 and 250 WPI), then entirely
separate processing paths may be preferred.
[0135] In one particular embodiment, images are captured with an
F/8 lens, with an exposure interval of 20 microseconds. The imaging
distance is set so that each captured pixel corresponds to an area
of about 1/150 inch in a focal zone positioned three inches above
the conveyor. Thus, each pixel corresponds to a single waxel at 150
WPI. The camera gain (or the distance from the light source to the
conveyor) is adjusted so that a pure white item on the conveyor is
depicted with captured 8-bit pixel values of 250.
[0136] The effective dynamic range of the imaging system can be
extended by employing zones of different illumination intensity. A
normal intensity zone can be illuminated as just-described, so that
a white target produces camera pixel values of 250. An adjoining,
high intensity zone can be illuminated with twice that intensity,
or greater. So-doing over-exposes light areas, but dark items can
then be resolved with better luminance gradations (i.e., contrast
enhancement). For example, watermark patterning on a dark printed
label that might appear as pixel values in the range of 5-10 in the
former illumination conditions can appear with an extended range,
such as 10-20 (or even 50-100), in the latter illumination.
[0137] In one particular implementation, such lighting variation is
a design parameter of the lens on a single light source. For
example, a linear array of LEDs may be equipped with a linear lens
that projects a pattern of varying intensity--with the high
intensity zone in the center, bordered by normal intensity zones on
either side. As the conveyor moves items through the projected
light, each point on the item first passes through the normal
intensity zone, and then passes through the high intensity zone,
and then passes through another normal intensity zone. Depending on
the conveyor speed, frame rate, and illumination area, each point
on the item may be imaged once, twice or more as it passes through
each of these zones.
[0138] In another arrangement, two or more different light sources
can be employed to provide higher- and lower-intensity light zones,
with similar effect.
[0139] In still another arrangement, depicted in FIG. 12, a linear
light source 120 (shown in side view) that is designed to output a
substantially homogenous intensity across its illumination area, is
tilted relative to the conveyor 122, so path lengths from the light
to different regions of the belt are different. In such case, the
fall-off of illumination with distance yields a gradation effect,
with a region 124 of the conveyor nearest the light being intensely
illuminated, and regions more removed 126 being progressively
less-intensely illuminated.
[0140] In a particular embodiment, a captured image frame spans
both the more-brightly, and more-dimly, illuminated regions on the
belt. In a first single frame, a bright area on an item will be
over-exposed, while a dark area will be contrast-enhanced. In
another single frame, the bright area will be properly-exposed,
while a dark area will be relatively under-exposed. The decoder
will tend to ignore the over-exposed region(s), as it does not have
the variation in pixel values that can serve as a clue to selection
of a patch for analysis, so no such patch will be analyzed. The
decoder will likewise tend to ignore the too-dark region(s), for
also lacking pixel variations. Thus, in a sequence of frames
depicting a single item as it passes through the variable
illumination, a darker region will tend to be analyzed from one
frame (when it falls in the brighter illumination) and not in
another (when it falls in the dimmer illumination). Similarly, a
lighter region will tend to be analyzed in one frame (when it falls
in the dimmer illumination) and not in another (when it falls in
the brighter illumination).
[0141] In another arrangement, red light and white light sources
are used. The red light source(s) and white light source(s) can
illuminate a common area, or they can illuminate overlapping or
adjacent areas. All such illuminated areas can be within the field
of view of a common imaging camera.
[0142] In still other arrangements, polarized light is used for
illumination. Additionally, or alternatively, one or more
polarization filters can be used at the image sensor, to attenuate
light of an orthogonal polarization.
[0143] In many applications, glare--a specular reflection of light
from a surface--is a hindrance. In certain embodiments of the
present technology, in contrast, such specular reflections can be
important in signaling watermark information. Rather than
filtering-out glare, polarizing filters can be used to accentuate
signal-conveying glare.
[0144] Some embodiments of the present technology employ novel
image sensors with polarizing filter arrays. An example is the Sony
Polarsens image sensor. The pixel array is overlaid with a
spatially-corresponding polarizer array, comprised of four
different angled polarizers (90.degree., 45.degree., 135.degree.
and 0.degree.). An image frame consisting just of data from the
90.degree. polarized sensors can be analyzed for watermark data.
Likewise for each of the other three polarization states. Moreover,
differences can be computed between, e.g., the 90.degree. "image"
and the 45.degree. "image," etc., and such difference images can
likewise be analyzed for watermark data.
[0145] This Sony sensor is available in a variety of
configurations. The IMX250MZR is illustrative. It is a monochrome
CMOS sensor, with 2464.times.2056 pixels. A color counterpart is
the Sony IMX250MYR.
[0146] Once a plastic item has been identified, it can be routed
from the conveyor into appropriate receptacles, or onto further
conveyors, by known means, such as solenoid-operated plungers,
stepper motor-controlled arms, forced air jets, etc. Exemplary
separation and sorting mechanisms are known to the artisan, e.g.,
from patent publications U.S. Pat. Nos. 5,209,355, 5,485,964,
5,615,778, US20040044436, US20070158245, US20080257793,
US20090152173, US20100282646, US20120168354 and US20170225199.
These mechanisms are here-referred to as "sorting diverters," or
simply "diverters" for short, and their operation is controlled in
accordance with the type of plastics identified.
[0147] FIG. 14 more particularly details some of the data that is
involved.
[0148] In the depicted embodiment, each plastic watermark conveys a
payload of 32 bits. This payload can be segmented into various
fields. One field identifies the type of plastic by class (e.g.,
ABS, EPS, HDLPE, HDPE, HIPS, LDPE, PA, PC, PC/ABS, PE, PET, PETG,
PLA, PMMA, POM, PP, PPO, PS, PVC, etc.). Another identifies a
sub-type of the plastic, e.g., by its average molecular weight,
solution viscosity value, or recommended solvent, or by whether the
plastic is used as a food container or a non-food container. A
third field identifies a color of the plastic. (Color can be
optically sensed, sometimes. However, plastic consumer packaging
increasingly includes a printed shrink sleeve over the container,
concealing its color.) A fourth field identifies a date the plastic
was manufactured, e.g., by month and year. A fifth identifies a
country of manufacture. A sixth identifies a manufacturing company.
More or fewer fields can naturally be used. Additional fields
include whether packaged food (vs. non-food), whether multi-layer
(vs. mono-layered), and whether compostable (vs. only recyclable).
Some fields convey an index or flag (yes/no) value. If needed, each
index value can be resolved into a literal text, date string, or
value (or range of values) by reference to a data structure, such
as a table or database.
[0149] In an illustrative embodiment, the sorting diverters are
responsive to the first three fields of data, serving to separate
the plastic by type, sub-type and color. All of the decoded
watermark data are logged to provide statistics about the waste
stream being processed.
[0150] The printed label payload typically conveys a longer
payload, e.g., 48 or 96 bits. The contents can vary by item, but
each usually starts with a GTIN, and may be followed by one or more
Application Identifier key value pairs (e.g., indicating expiration
date, lot code, item weight, etc.). In some arrangements, none of
the payload expresses the type of plastic used in the item
container.
[0151] To determine the plastic type, a data structure 121, such as
a table or database, can be used. It serves to associate item GTINs
with corresponding information about the plastic used for the item
containers. That is, the data structure is queried with a GTIN
identifier decoded from the printed label watermark payload, and
the system thereby accesses previously-stored data identifying the
plastic type, sub-type and color (if available) for the product
having that GTIN. This plastic material information is provided to
the logic controlling the sorting diverters, as is done with data
from the plastic watermark.
[0152] From the foregoing, it will be recognized that a technical
problem with the prior art was assuring reliable reading of a GTIN
label watermark on product packaging presented to a point of sale
scanner, within the limited time and processing constraints of such
environment. A technical effect of the detailed arrangement is
enabling such packaging to convey a second watermark, to facilitate
recycling, without impairing the reliable reading of the GTIN label
watermark at the point of sale scanner, due to differences in the
signaling protocols used in the two watermarks.
[0153] A further technical problem was assuring reliable optical
reading of watermark data from items in a fast-moving waste stream.
In some embodiments, reliability is enhanced by the imaging
apparatus used to capture depictions of items in the waste stream.
In some embodiments, reliability is enhanced by the shape of
texture markings applied to surfaces of plastic containers within
the waste stream.
[0154] It will be recognized that the present technology can be
utilized in waste sorting systems of the sort sold by Pellenc ST,
MSS Inc., Bulk Handling Systems, National Recovery Technologies
LLC, Rofin Australia PTY, Ltd., Green Machine Sales LLC,
EagleVizion, BT-Wolfgang Binder GmbH, RTT Steinert GmbH, S+S
Separation and Sorting Technology GmbH, and Tomra Systems ASA. The
optical sorting employed in such machines (e.g., based on near
infrared- or visible-spectroscopy, based on the different
absorption spectra of different plastics, etc.) can be replaced
with the present technology, or the present technology can be used
in conjunction with these other methods.
Block Analysis
[0155] In one illustrative embodiment, a conveyor belt is spanned
by an array of cameras, each of which provides image frames at rate
of 150 per second. Each frame measures 1280.times.1024 pixels, and
spans a field of view measuring about 8.times.6 inches on the
conveyor belt. Analysis blocks are arrayed across each captured
image, and each block is analyzed for watermark clues, such as a
watermark reference signal. If a watermark reference signal is
found, it is used to identify the pose of the watermarked object on
the conveyor belt (e.g., using the technology detailed in U.S. Pat.
Nos. 9,959,587 and 10,242,434). Using the pose information, the
image is re-sampled in the region where the reference signal was
detected, to extract waxel data, which is then provided to a
decoder which seeks to extract a watermark payload.
[0156] While this specification commonly refers to processing
blocks or patches of imagery measuring 128.times.128 pixels (or
waxels), applicant has discovered that the detailed arrangements
are often better served by processing smaller sets of data, such as
96.times.96, 88.times.88, 80.times.80, 64.times.64, etc. (Due to
the curvature and crushing of items found in waste streams, not
many planar surfaces are present. Yet geometric synchronization
typically proceeds on an assumption of planarity. This is believed
to be a reason that processing small patches of imagery can yield
superior results, i.e., so that the non-planar effects of physical
distortions are minimized.) Thus, the reader should understand that
references to 128.times.128 in connection with watermark reading
operations are exemplary only, and that smaller data sets are
contemplated and are often preferred. (Watermark encoding, in
contrast, may nonetheless be performed on the basis of a
128.times.128 block size, yet decoding can extract the watermark
payload from analysis of smaller image blocks. Or encoding can
likewise proceed based on smaller blocks.)
[0157] The analysis blocks arrayed across each image frame for
watermark reading may be uniformly or randomly spaced-apart, or
tiled edge-to-edge, or overlapping (e.g., with each block
overlapping its immediate neighbor by 20%-80%). FIG. 15 shows an
exemplary block pattern, in which a 1280.times.1024 image frame is
analyzed using 96.times.96 pixel blocks, with each block
overlapping its neighbors by 25%. A few blocks are shown in bold,
since the tiling pattern makes the boundaries of individual blocks
ambiguous.
[0158] In some embodiments, if a watermark reference signal or
other clue (e.g., as detailed below) is found in one of the
analysis blocks, then a more dense cluster of analysis blocks is
examined in that same neighborhood for reference signals and, if
successful, then analyzed for payload data. FIG. 16 shows an
example. The original block is shown in bold lines at the center.
Further blocks are arrayed around it with 75% overlap (omitting
block locations that were analyzed in the original array of block
locations). Again, a few of the blocks are shown in dashed-bold for
clarity's sake. FIG. 17 shows this area of more densely-placed
blocks in context, in the frame of FIG. 15--located where a
watermark reference signal or other clue was found among the
originally-examined blocks.
[0159] In some recycling systems, the conveyor belt is empty in
places, with no item present in part of the camera view. Clues
about the presence or absence of such gaps can be detected,
enabling processing resources to be applied to more promising
imagery. Similarly, watermark processing of captured imagery may be
triggered only when a quick assessment of the imagery finds a clue
indicating plastic may be present (or indicating that something
other than the conveyor belt is depicted).
[0160] Plastic is often characterized by regions of specular
reflection, or glare, as the plastic surface mirror-reflects
incident illumination towards the camera. This glare can be sensed
and can serve as a clue to activate (trigger) watermark processing.
For example, multiple blocks within an incoming sequence of image
frames (e.g., at 150 frames per second) can each be analyzed for a
2.times.2 pixel area in which the pixel magnitude is in the top 5%,
10% or 20% of the sensor's output range (or within similar
percentiles of historically-sensed pixels from previous blocks
depicting that region of the conveyor belt). Any frame meeting this
criterion is analyzed for watermark data. (Desirably, more than
just an image excerpt near the glare is analyzed, since the plastic
may extend far beyond such point.)
[0161] In one particular embodiment, no part of a frame is
processed until a glare pixel is detected. When this event occurs,
analysis of the full frame is not triggered. Rather, a 7.times.7
array of overlapping pixel blocks is positioned based on the glare
spot, and each of these blocks is analyzed for the presence of a
watermark reference signal. The blocks may overlap by 50% or more
of their width, i.e., more than the usual block overlap. FIG. 18
shows an example, in which blocks overlap by 75% of their width.
The glare spot is identified by the "+" mark in the middle of the
densely-overlapping blocks. Again, a few blocks are particularly
identified by bold dashed lines, since the boundaries of the
component blocks are not otherwise clear.
[0162] In addition, or alternatively, metrics other than glare are
employed for determining that imagery likely merits watermark
processing.
[0163] One method, termed a block trigger method, provides a clue
that helps discriminate empty and not-empty portions of conveyor
belt, based on comparison of incoming pixel values to historical
norms.
[0164] One particular block trigger algorithm compiles a histogram
of pixel values selected from sub-blocks within an analysis block
(such as within one of the bolded blocks in FIG. 15), over many
captured image frames. Each block may be 96.times.96 waxels
(pixels, at scale=1). The block is logically divided into a
4.times.4 array of sub-blocks, each 24 waxels on a side (i.e., 16
sub-blocks per block, each comprising 24{circumflex over ( )}2 or
576 pixels). Values from a randomly-selected but static 25 pixels
from each sub-block are averaged together to yield one average
pixel value per sub-block (i.e., having a value between 0 and 255
in 8-bit greyscale). A new such sub-block average pixel value is
produced for each frame.
[0165] 256 of these average pixel values for a particular sub-block
are eventually compiled in a histogram (i.e., over 256 frames).
These values show a tight peak corresponding to the average pixel
value for empty conveyor belt, at the belt location corresponding
to that particular sub-block (and with its particular
lighting).
[0166] When a new frame is captured, values are again computed for
the 16 sub-blocks within the block. Each value is judged against
the histogram for that block. If the new value is within a few
digital numbers (e.g., 1, 2, 3 or 4) of the pixel value where the
histogram shows its sharp peak, then this counts as a vote for the
conclusion that the sub-block imagery depicts an empty belt. The 16
votes thereby produced, for the 16-sub-blocks of the block, are
tallied. If a threshold number of votes (e.g., 11 of the 16)
conclude that the sub-block imagery depicts an empty belt, then the
block is concluded to depict an empty belt. In such case, analysis
of the block is skipped. Else, the block is analyzed for watermark
data.
[0167] This process is performed for all the blocks within the
camera view (e.g., all the blocks shown in FIG. 15), for each
frame.
[0168] (If imagery is captured at a finer resolution, i.e., with
more than 1 pixel per waxel, then the 25 values from each sub-block
can be determined by sub-sampling, e.g., by averaging values of
four or nine of the finer pixels near the 25 static locations.
Alternatively, the value of the single fine pixel closest to each
of the 25 static locations can be employed.)
[0169] FIG. 19 shows an illustrative histogram for one exemplary
sub-block, after 219 frames have been processed. The x-axis shows
average pixel values for different frames, computed for that
sub-block. The y-axis shows the count of frames having different
average pixel values for that sub-block ("bin count"). The
histogram peaks at 20. In relevant part, the associated bin counts
for different average pixel values are as follows:
TABLE-US-00001 Average Pixel Value Frame Count . . . . . . 17 4 18
2 19 42 20 53 21 35 22 4 23 2 . . . . . .
[0170] When the next frame of imagery is captured, if the average
value computed from the 25 static pixel locations in this sub-block
equals 18, 19, 20, 21 or 22 (i.e. the peak value of 20, +/-2), the
sub-block is regarded as depicting empty conveyor belt. If 10 other
of the 16 sub-blocks of that block agree, then this is taken as a
clue that the block depicts empty conveyor belt. As a consequence,
no watermark processing is performed on that block. Alternatively,
if such a consensus do not agree, then this serves as a clue that a
plastic item may be depicted by the block, and further processing
is triggered.
[0171] Each histogram is kept fresh by periodically discarding
data, to make room for more data. For example, when a frame counter
associated with a histogram indicates that 256 frames have been
processed, and 256 average values for the sub-block populate the
histogram, the histogram contents are thinned by half, to 128
values. This can be done by taking the bin count for each average
pixel value in the histogram and dividing by two (rounding down).
The frame counter is reset accordingly, i.e., to 128 frames. Counts
of average pixel values from the next 128 frames are then tallied
in the histogram, at which time the thinning repeats. By this
arrangement, historical pixel values decay exponentially in
significance, allowing the histogram to reflect most recent
data.
[0172] With each newly-captured frame, this block-trigger method
provides a clue as to whether to trigger a watermark reading
operation, for every block location in the frame. The average pixel
values derived from the newly-captured frame serve to update
corresponding histograms for use in evaluating blocks in following
image frames.
[0173] (It will be recognized that watermark reading can be
triggered if sufficient sub-blocks have average pixel values above
(brighter) and/or below (darker) than the histogram peak (i.e., the
recent belt luminance). That is, a plastic object may include
regions of dark, as well as light, pixels. Both help inform the
triggering decision.)
[0174] A related technique proceeds similarly, but is based on
statistics of color distribution, rather than luminance
distribution.
[0175] In a preferred implementation of the block trigger
algorithm, if further analysis of a block is triggered, and the
analysis detects a watermark reference signal (or decodes a
watermark payload) from that block, then the average sub-block
pixel value data for that block are not added to their respective
histograms (or, if earlier-added, such counts are removed). This
way, the histograms aren't tainted by data from imagery that is
known not to depict empty conveyor belt.
[0176] Many recycling systems set a limit on the number of image
blocks that can be analyzed during processing of each frame--a
processing budget. For example, the limit may be 200 blocks. A
fraction of this total, such as 50-75 blocks, may be reserved, for
analysis of blocks placed densely around any block from which a
watermark reference signal or other clue is detected (e.g., as
discussed above connection with FIGS. 15-17). If clues are detected
from several blocks, so that dense placement of further analysis
blocks will exceed the 200 block limit, then further blocks can be
allocated in accordance with the values of the clues (e.g., the
strength of detected watermark reference signals), with the blocks
that look most promising getting the largest allocation of
neighboring analysis blocks.
[0177] If not all 200 blocks are analyzed each frame, electricity
consumption drops, and heat output from the computer processor(s)
(heat output that often must be offset by air-conditioning) also
drops.
[0178] In a variant block trigger method, the full processing
budget (e.g., of 150 block analyses) is spent each frame. Some
block analyses are triggered as described above, i.e., because 11
of 16 sub-blocks (or, more generally, K of L sub-blocks) have
average pixel values (over 25, or more generally N, selected
pixels) within a few digital numbers of the peaks of their
respective histograms. Any remaining analysis blocks are then
allocated in accordance with the differences between the noted
average sub-block pixel values and peaks of respective histograms,
summed over all 16 sub-blocks of a block. Those blocks having the
smallest aggregate differences are triggered for watermark
analysis, until the full budget of 150 analysis blocks is
reached.
[0179] Some systems enable such variant method, automatically or
manually, when the load of material on the conveyor belt exceeds
some threshold. In extreme cases, the conveyor belt may be almost
wholly obscured--for intervals of hundreds of consecutive
frames--by covering objects. In this case, prominent peaks
associated with the background belt luminances will not emerge from
the histograms. But each histogram will nonetheless have a peak
somewhere. This variant block trigger method will employ the noted
procedure to allocate the full budget of analysis blocks to the
image frame. In effect, this results in a largely-random selection
of blocks for analysis. But since the belt is evidently crowded
with objects, this is not an unreasonable block selection
strategy.
[0180] Other clues for discerning that imagery likely merits
watermark processing employ image statistics, such as mean,
standard deviation, and/or variance.
[0181] FIG. 20 shows, by the large rectangle, an image frame field
of view that spans a conveyor. The dashed rectangle shows the
positioning of a linear LED light source that also spans the
conveyor. Due to orientation of the light source, or its lens (or
reflector), the illumination has a spatial intensity profile as
shown by the chart immediately to the left--showing maximum
intensity in the region of the lamp, falling off sharply in one
direction, and less-sharply in the other (on a scale of 0-100).
[0182] Arrayed across the direction of belt travel are plural
strips of image blocks, each, e.g., 128.times.128 pixels in size.
Only two rows are shown in the diagram, but similar strips span the
image frame. Adjoining strips are often illuminated differently,
due to the lamp's illumination profile.
[0183] (While the blocks are non-adjoining and non-overlapping in
FIG. 20, this is for illustrative clarity. In actual practice, the
blocks are commonly adjoining or overlapping.)
[0184] A metric is derived from each of these blocks in each strip,
and is used as a clue to determine similarity of the image block to
imagery depicting empty belt.
[0185] In an illustrative embodiment, a feature value f is computed
for each block and used to identify regions that merit further
watermark analysis. In general, f ( ) is a function of each pixel
in a block. An initialization phase is performed while the belt is
moving but empty, e.g., when the sorting system is initially
powered-on. Feature values f are calculated for each block over
multiple frames, and the values are grouped by strip. For each
strip, a population mean and standard deviation are estimated from
the corresponding group of sample feature values obtained over the
multiple frames.
[0186] When a new image frame is thereafter captured, feature
values are calculated for each block in the new frame. For each
feature value, a normalized feature value is calculated, using the
previously estimated mean and standard deviation values for the
strip containing the block from which the feature value was
calculated. The normalized feature value is calculated as
f N = f - .mu. .sigma. , ##EQU00002##
[0187] where .mu. and .sigma. are the estimated mean and standard
deviation, respectively. If a feature value has a Gaussian or
Generalized Gaussian distribution for regions containing the belt,
the normalized feature will have a monotonic decreasing probability
distribution; smaller values will be more likely than larger
values. These normalized feature values yield a first metric
indicating how similar (or dissimilar) the new block is compared to
empty belt data in that region. Smaller values indicate more
similarity to empty belt data. Larger values indicate more
dissimilarity. The largest values indicate blocks that are the best
candidates for watermark reading, as they are most dissimilar from
the empty belt.
[0188] The normalized feature metric values are sorted, largest to
smallest, to identify the blocks with the best prospects for
depicting watermarked objects. This establishes a priority order
for watermark reading. If the system processing budget permits
analysis of 150 blocks per frame, then data from the 150 blocks
having the highest first metrics are routed for watermark
processing.
[0189] Different underlying features f can be used, with different
resulting effectiveness in watermark processing. Illustrative
embodiments include block mean and block standard deviation.
[0190] The effectiveness of a specific feature in distinguishing
image blocks containing belt-only pixels from other image blocks
depends on the conditional distribution of the feature for these
two classes of blocks. For some non-belt image blocks, feature
f.sub.A may not be useful in distinguishing the block from belt
blocks, while feature f.sub.B may do a good job of distinguishing
the block. For other non-belt blocks, the situation may be
reversed, and f.sub.A might be the preferred feature. This leads to
an additional class of embodiments that make use of multiple
features.
[0191] In multiple feature embodiments, separate sets of mean and
standard deviation estimates are calculated for each feature in the
initialization phase, and a corresponding normalized feature value
is calculated for each feature of each block in a new image frame.
Normalized feature values are combined into a single metric value
using a combination function. The resulting combined metric values
are sorted, and the sorted list of metric values forms a priority
list for watermark processing.
[0192] One example of a combination function is the sum of the
normalized feature values. Other embodiments include more
complicated functions, e.g., derived from statistical analysis of
the normalized feature distributions for the two classes of belt
and non-belt blocks. Polynomial equations combining the feature
values are used in some implementations. Additional embodiments may
have combination functions that vary by image strip, to take
advantage of the fact that different image strips may result in
different normalized feature distributions.
[0193] It will be recognized that the just-described arrangements
always make full use of the full system processing budget. If the
system budget allows for analysis of 150 blocks per frame, then 150
blocks are analyzed each frame. (As before, there may be a reserve
budget of additional blocks that can be allocated based on
processing results from the first 150 blocks.)
[0194] The foregoing arrangements may be regarded as species of
classifiers, i.e., classifying whether imagery likely depicts a
belt (or glare from plastic) or not. Many other species of
classifiers can be used to provide clues by which watermark
processing can be controlled.
[0195] One such alternative employs a neural network, trained to
classify an image frame (or portion) as depicting either (a) only
belt, or (b) something other than only belt, by training the
network with a large corpus of labeled images, variously showing
images of one class or the other. Suitable networks and training
methods are detailed in patent publications 20160063359,
20170243085 and 20190019050, and in Krizhevsky, et al, Imagenet
classification with deep convolutional neural networks, Advances in
Neural Information Processing Systems 2012, pp. 1097-1105.
Additional information is detailed in copending application Ser.
No. 15/726,290, filed Oct. 5, 2017.
[0196] If the image, or image patch, is determined to likely depict
only conveyor belt, then no further analysis of such imagery is
undertaken. (Instead, freed processor cycles can be applied to
further-processing other imagery, e.g., by analyzing additional
blocks, by attempting decoding using different candidate affine
transforms, etc.).
[0197] A different arrangement provides clues classifying imagery
depicting empty conveyor belt, and distinguishing such imagery from
other imagery, by sensing characteristic belt markings. For
example, conveyor belts commonly have scars, stains, and other
striation patterns that are elongated in the axis of the belt
travel (the belt direction). Such markings, detected in imagery,
are predominantly of low frequency. Captured imagery can be
low-pass filtered to reduce high frequency noise, and the resulting
imagery can then be analyzed to assess the strengths of edges in
different directions (e.g., by a Canny or Sobel algorithm).
[0198] In a particular embodiment, a 128.times.128 block of imagery
is low-pass filtered, and then examined with a Canny edge detector
to assess the strength of gradients running along the belt
direction, and the strength of gradients running across the belt
direction (e.g., by summing gradient values in vertical and
horizontal image directions). If the patch depicts a belt, the sum
of the former gradients will be substantially larger than the sum
of the latter gradients. A logistic regressor is trained to respond
to the two strength values by classifying the image patch as either
depicting a belt or not. If it depicts a belt, no further analysis
on such block is undertaken; if it does not depict a belt, further
watermark analysis of the block can be commenced.
[0199] In other embodiments a simple ratio between the two
summed-gradient measures is computed, and this value is compared
against a threshold, to determine whether the image blocks depicts
the conveyor belt.
[0200] Sorting machines are sometimes equipped with laser systems
to detect the presence of objects. For example a laser beam may be
swept across over an expanse of the belt using a rotating mirror
arrangement, and serve to trigger detections along elements of a
linear photodetector array on the other side. So long as each of
the photodetectors detects the laser beam, the conveyor belt in the
swept region is known to be empty. Such check can used to suppress
analysis of captured image blocks.
[0201] A further type of clue that can trigger further watermark
analysis is based on a pepper-in-salt (or salt-in-pepper) pattern
metric, indicating a likelihood that a block depicts a sparse dot
watermark. Exemplary algorithms for computing such a metric are
described next.
[0202] An input image block is down-sampled, if necessary, so it is
at scale=1. That is, each waxel is depicted with the size of one
pixel. We are looking for dark pixels in a field of light--pixel
outliers. However, the image contrast may be large or small, and
the illumination may vary across the block. The computed metric
desirably should be robust to such variables. To do this, we
compute a measure that examines neighborhoods of pixels, and also
takes into account sensor acquisition noise.
[0203] The acquisition noise present in captured imagery is a
function of pixel values, with higher pixel values having higher
noise values. A polynomial function, or a reference table, can
provide a noise standard deviation value for each pixel value
between 0 and 255. To identify pixels with outlier values (e.g.,
pixels that are relatively darkest in a field of relatively lighter
pixels), a measure, sigma, is computed for a neighborhood of pixels
around a subject pixel at coordinates (i,j) and having a value X,
by the equation:
.sigma. ( i , j ) = x ( i , j ) - N ( i , j ) _ S ( N ( i , j ) _ )
##EQU00003##
where the N-bar term is the average pixel value over the
neighborhood, and the S term is the standard deviation of
acquisition noise for the neighborhood, based on the polynomial or
the reference table data. (The best neighborhood size can be
heuristically determined. Neighborhoods can be as small as
4.times.4, or as large as 64.times.64, for a 128.times.128 block
size. An 8.times.8 neighborhood is exemplary. Alternatively, the
neighborhood can be temporal, comprising the value of a single
pixel at coordinates (i,j) throughout a series of
previously-captured frames, e.g., dozens or hundreds.)
[0204] For pixels darker than the neighborhood average, this sigma
value will be negative. To be regarded as a sparse mark dot, we set
a darkness threshold that the sigma value must meet, e.g.,
.sigma..sub.ij<-3. We produce a filtered image block that
includes only those pixels for which the corresponding sigma value
meets this test. All other pixels are removed (e.g., set to white,
with pixel value=255).
[0205] The procedure so far identifies the darkest dots, but in so
doing also includes pixels forming the darkest edges (e.g.,
including dark text). To focus our watermark extraction efforts
only on sparse mark dots, we want to filter out pixels that are not
separated from other pixels (a form of morphological filtering).
Various techniques can be used for this task. A simple technique is
to visit every dark pixel, examine a 5.times.5 pixel region
centered on that image location, and count the number of dark
pixels in the region. If there are more than two dark pixels in the
5.times.5 region, the center pixel is removed (e.g., changed to
white). The resulting processed block is then comprised wholly of
isolated dark dots.
[0206] Finally, this processed block is examined to count the
number of dark dots remaining, within the block boundary. This
count serves as a metric indicating a likelihood that the block
contains a sparse watermark.
[0207] This metric may be compared to see if it exceeds an
empirically-determined threshold K (e.g., K=500) to identify frames
likely to depict sparse watermark data. Alternatively, blocks
within the frame can be ranked based on their associated sparse
metrics, and those blocks having the highest sparse metrics can
then be further analyzed for watermark data, up to the
block-processing budget limit.
[0208] Various simplifications and modifications can be made to
this particular algorithm. For example, a simpler procedure just
identifies a set of darkest pixels in the block. (For example, the
darkest 10% or 30% of pixels in the block can be identified.) The
procedure then applies the above-described morphological filtering
and counting operations to yield a sparse metric.
[0209] Another variant arrangement discriminates potential sparse
dots from not, by learning gleaned from previous image frames.
[0210] An exemplary learning process analyzes pixel values from a
sampling of, e.g., 10 blocks in each frame, for a series of past
frames. Each block is divided into sub-blocks, e.g., of 5.times.8
pixels. For each analyzed sub-block, both the mean pixel value and
the minimum pixel value are determined.
[0211] In some sub-blocks, the minimum pixel value will be that of
a dark sparse dot. Other sub-blocks don't include a sparse dot, so
the minimum pixel value will simply be the lowest-valued pixel
among image content that is not a sparse dot (e.g., background
imagery, an item marked with a continuous-tone watermark instead of
a sparse watermark, etc.).
[0212] From these collected statistics, we identify the largest of
the minimum pixel values (the "max-minimum"), for each associated
sub-block mean value. For example, if we consider all sub-blocks
that have a mean pixel value of 151, we may find that the largest
of the minimum pixel values encountered in the analyzed series of
frames is 145. Any pixel value larger than 145 is pretty-assuredly
not a sparse mark dot, in a sub-block having a mean value of 151.
This value, and other values similarly observed, can thus help
establish a threshold to discriminate potential sparse mark dots
(outliers) from not-potential sparse dots.
[0213] In one particular embodiment, we describe the set of all
such points by a best-fit line, characterized by a slope and
offset, e.g.:
.tau..sub.Outlier=0.96*.mu.-1.6
where .mu. is the mean pixel value for a sub-block.
[0214] Thereafter, when a new frame of imagery is received, we
compute the mean value of each 5.times.8 pixel sub-block, and
determine the applicable outlier threshold by the best-fit line
equation. Any pixels in the sub-block having values smaller than
this threshold are identified as candidate sparse dots. (E.g., if a
sub-block has a mean pixel value of 82, we treat all pixels in that
sub-block with pixel values of 77 and less as candidate sparse
dots.) We then apply the morphological filter across the block to
discard connected dots, as described earlier, and then count the
number of dots remaining in the block, yielding the sparse metric.
As before, this metric can be tested against a threshold to
identify blocks that merit watermark processing. Alternatively, all
blocks in the frame can be ranked according to this metric, and be
selected for processing on that basis--until the block-processing
budget is reached.
[0215] In alternative embodiments, the above-detailed processes can
be altered to generate a metric based on bright pixels in a dark
field (i.e., salt-in-pepper). One such alteration simply inverts
the image block, dark-for-light, before performing one of the
just-described algorithms.
[0216] Some recycling systems may look for multiple clues in
deciding which blocks to watermark-analyze. For example, block
trigger clues may first be obtained for all blocks in an image
frame, to identify blocks depicting only conveyor. The remaining
blocks can then each be assessed to determine a sparse metric, as
just-described, to evaluate which of the non-conveyor-depicting
blocks is most promising for watermark analysis.
[0217] As items are moved by the recycling system conveyor, they
pass the camera(s) in linear fashion, entering the field of view
from one side of the camera sensor, and departing from the other.
Once a clue indicating an image block depicting non-empty conveyor
belt has been sensed in one frame, e.g., as detailed above, then
imagery can be analyzed not only in the current frame, but imagery
depicting successively-displaced regions in the camera field of
view can also be analyzed in the N frames that follow. N is a
function of the camera frame rate, the belt speed, and the span of
the camera's field of view. For example, if the camera's field of
view is 15 inches, and the conveyor is moving 10 feet per second,
then any item on the conveyor should be in-view for an eighth of a
second, as it travels through the camera's field of view. If the
camera captures 60 frames per second, then N can be set for 6
(i.e., corresponding blocks in seven total frames are
analyzed).
[0218] In one particular embodiment, an array of overlapping
analysis blocks is positioned along the side of the camera field of
view at which objects first enter, and each of these blocks is
analyzed, every frame, for a watermark reference signal. If a
reference signal or other clue is found within any of these blocks,
such detection spawns further analysis of an overlapping cluster of
blocks, as described earlier, centered on the detecting block. This
cluster is progressively advanced across the field of view, frame
to frame, in accordance with the speed of the conveyor belt.
[0219] FIGS. 21A-21D illustrate such an arrangement. A line of
blocks is analyzed at the item-entry side of each image frame.
(Blocks in the interior of the frame may normally not be analyzed.)
A watermark reference signal or other clue is discerned in one of
these edge blocks (shown in bold), in which case a cluster of
overlapping blocks in that vicinity can be analyzed for watermark
reference signals. If a watermark reference signal is detected,
then analysis continues to attempt recovery of a watermark payload,
from waxel data obtained using affine parameters sleuthed from the
reference signal. Corresponding clusters of blocks are analyzed in
successive frames, at successive positions, until the detected
object is no longer in view of the camera.
[0220] If one of the blocks in the advancing cluster detects a
watermark reference signal or other clue (e.g., the bolded block in
FIG. 21C), then a supplemental cluster of analysis blocks (shown in
dotted lines) can be spawned--centered on the detecting block. This
supplemental cluster of blocks can likewise progress across the
field of view with the original cluster, in synchrony with the
conveyor movement. Meanwhile, the original band of blocks, arrayed
across the entry side of the camera field of view, continues to
examine each new image frame for watermark reference signals or
other clues.
Optimizations
[0221] As noted, conveyor belts on which plastic items are conveyed
for identification/sorting move at relatively high speeds. To
assure adequate illumination and depth of field, smaller apertures
and longer exposures are desired. This can lead to motion blur.
[0222] Some embodiments of the technology apply deblurring to
captured imagery before attempting watermark reading. Different
techniques can be employed, including deconvolution by inverse
filtering, by a Wiener filter, or by the Richardson-Lucy Algorithm.
A suitable point spread function (PSF) can be estimated for the 1D
motion using classical methods. (The PSF essentially characterizes
the amount of energy with which light from a single point in the
scene exposes each pixel of the camera during a static
exposure.)
[0223] More sophisticated methods can be employed, e.g., utilizing
flutter shutter techniques that sample the scene at different
times, for different intervals, and use the resulting imagery to
derive a more refined estimate of the unblurred scene. (See, e.g.,
patent publication US20090277962.)
[0224] In an illustrative embodiment, deblurring is performed in
the Fourier domain, with the Fourier transform of the image being
divided by the Fourier transform of the blur kernel. In other
embodiments, such an operation can be performed in the spatial
(pixel) domain.
[0225] In convolutional decoding of the watermark payload, list
decoding can be employed. Instead of outputting a single decoded
payload, list decoding outputs a list of possibilities, one of
which is correct. This allows for handling a greater number of
errors than is allowed with unique decoding. The plural listed
payloads can then be assessed using CRC data, or constraints within
the payload itself (e.g., the value of a certain field of the data
is known to be drawn from only a subset of possible values) to
identify the one correctly-decoded payload.
[0226] Instead of attempting to characterize the pose of a
128.times.128 waxel image patch, a smaller patch, e.g., 96.times.96
waxels, is desirably analyzed instead, as noted above. (If, as in a
preferred embodiment, the camera sensor, lens, and imaging distance
are chosen so that watermarked objects are depicted at a scale in
which each pixel roughly corresponds to the area of a single waxel,
then a 96.times.96 waxel patch corresponds a 96.times.96 pixel
patch.) An FFT of size 128.times.128 is performed on this patch, by
zero-padding, or by processing adjoining pixel imagery with a
square or gaussian window to focus on the central area. As
indicated, the methods detailed in U.S. Pat. Nos. 9,959,587 and
10,242,434 are used to characterize rotation and scaling.
Translation can then be determined using the phase deviation method
of the '587 patent. Collectively, rotation, scaling and translation
(the affine parameters) describe how the original watermark is
presented within the captured imagery.
[0227] The phase deviation method provides a metric indicating the
strength of the detected reference signal within the analyzed patch
of imagery, namely a sum of the phase deviations between the
measured and expected phases of each of the reference signals. If
this phase deviation metric is below a threshold value (lower
metrics being better), then the patch of imagery is concluded to
contain a readable watermark. An interpolation operation then
follows--sampling the imagery at points corresponding to waxel
locations, guided by the discerned affine parameters, to generate
data for payload decoding.
[0228] As noted, if one patch of imagery is concluded to contain a
readable watermark, then adjoining patches are checked to determine
whether they, too, contain readable watermarks, e.g., using the
procedure just-described. For each such patch, a corresponding set
of affine parameters is determined. (Typically, each patch is
characterized by a different set of affine parameters.) Again, an
interpolation operation follows, generating more waxel data for use
in payload decoding.
[0229] As before, the adjoining patches may be edge-adjoining, or
may be overlapped--by any number of waxels.
[0230] If image patches smaller than 128.times.128 are analyzed
(e.g., 96.times.96 or 64.times.64 waxels), then not all of the
128.times.128 waxel encoding locations may be depicted in each
patch (depending on scaling). Nonetheless, corresponding locations
are identified between the analyzed patches (using the affine
parameters), and their sampled waxel data are combined (e.g.,
averaged or summed). Combined tallies of waxel data for some or all
of the 128.times.128 encoding locations are thereby produced, and
are provided to a Viterbi decoder for extraction of the encoded
watermark payload.
[0231] This is shown schematically in FIG. 22. A reference signal
is detected in a small image patch 141 (here shown as only
8.times.8 waxels), which leads to search and discovery of reference
signals in adjoining small image patches 142 and 143. Each has a
different affine pose. The watermark signal block (not particularly
shown) spans an area larger than any patch.
[0232] For some waxels in the watermark signal block, such as waxel
144, interpolated data from a single image patch is provided to the
decoder. For other waxels, such as waxel 145, interpolated data is
available from each of two overlapping patches. These two
interpolated values are averaged (or summed) and provided to the
decoder. For still other waxels, such as waxel 146, data from three
patches is averaged (summed) and provided to the decoder. For yet
other waxels, such as waxel 147, no data is available for the
decoder.
[0233] Sometimes data for a particular waxel is available from two
different (but typically adjoining) 128.times.128 waxel watermark
blocks. FIG. 23 shows two such blocks, in solid lines. Also shown
are two 96.times.96 waxel patches, in dashed lines, that are
processed as described above. From the affine pose parameters
determined for such patches, it is known that the waxel denoted by
the circle in the left patch spatially corresponds to the waxel
denoted by the circle in the right patch. Both convey the same chip
of signature information. In this case, the two waxel values are
summed, for submission to the decoder.
[0234] The decoder processes whatever data is available and
produces an extracted payload (or a list of candidate
payloads).
[0235] In some embodiments, the waxel data contributed by each
image patch is weighted in accordance with the strength metric for
the associated reference signal. In other embodiments, different
metrics can be employed, e.g., as detailed in patent application
Ser. No. 16/011,092, filed Jun. 18, 2018 (where they are termed
Reference Pattern Strength, and Linear Reference Pattern Strength).
Alternatively, each waxel data can be weighted in accordance with a
corresponding message strength factor, as detailed in U.S. Pat. No.
7,286,685.
[0236] The just-described accumulation of waxel data from across
plural patches in an image frame may be termed intraframe signature
combination. Additionally, or alternatively, accumulation of waxel
data from the same or corresponding waxel locations across patches
depicted in different image frames can be used, which may be termed
interframe signature combination.
[0237] Once affine parameters for a patch are known (describing
appearance of a watermark in the patch), then payload reading can
proceed by payload correlation techniques instead of Viterbi
decoding. This is particularly useful if the number of distinct
payloads is small, e.g., on the order of dozens or hundreds. This
may be the case if the only payload of concern is plastic type
data, and there are only a limited number of plastic types that may
be encountered.
[0238] In one particular arrangement, a set of templates is
produced, each expressing the waxel encoding associated with one
particular type of plastic. Waxel elements that are common across
all plastic types (or across a substantial fraction, such as 30%)
can be omitted from the templates, to reduce chances for confusion.
The image data is correlated with the different templates to
identify the one pattern with which it most strongly corresponds.
Since the image was already determined to include a reference
signal (e.g., of a plastic texture watermark), then one of the
limited number of waxel patterns should be present, making
correlation a high reliability method of discerning the
payload.
[0239] Plastic bottles are increasingly not printed directly, but
rather are wrapped in plastic sleeves that are printed and
heat-shrunk to conform to the bottles. This poses a difficulty, as
the heat-shrink material common contracts primarily in one
direction (the circumferential direction). Any watermark pattern
printed on such sleeve is then differentially-scaled by the
heat-shrinking, posing an impediment to watermark reading.
[0240] To redress this problem, one or more of the "seed" linear
transforms (detailed in U.S. Pat. Nos. 9,959,587 and 10,242,434)
that serve as starting points for iterative searches to determine a
watemark's affine transform, are initialized to include a
differential scale component. This allows the iterative process to
more quickly reach a better estimate of affine distortion, when
detecting watermarks from heat-shrunk plastic sleeves.
[0241] Sometimes a watermark will not be read from a marked item,
and it will travel the conveyor without being identified as to
type. Meanwhile, other items are ejected off the conveyor, e.g.,
into bins for ABS, HDPE, PET, PETg, etc., leaving just unidentified
items.
[0242] These unidentified items may be collected into a bin of
their own, and reprocessed later. Failure to read is unusual, and
is commonly redressed by changing the pose at which the item is
presented to the illumination, and camera. By collecting such
items, and reprocessing, they will be presented at different poses
the second time through, and will likely then be identified.
[0243] Alternatively, instead of collecting and re-processing, the
items can be tumbled (e.g., dropped from one conveyor onto
another), or bumped/shuffled (e.g., the conveyor may travel through
a curtain of hanging obstacles) to change the items' poses, and a
second camera/illumination system can then gather additional
imagery for analysis.
[0244] In some embodiments, captured imagery is a function of light
transmission through the item, instead of simply light reflection
from the item. FIG. 24 schematically illustrates such an
arrangement, in which items briefly "fly" off one conveyor onto
another, presenting a camera view through the item, to one or more
light sources on the other side. Such approach can be used with any
of the camera/illumination systems described earlier. "Clues" based
on detection of conveyor belt can similarly be based on detection
of the null background of such a fly-by.
[0245] The direct least squares technique for determining scale and
rotation transformations, to characterize appearance of a watermark
in imagery, operates by successively winnowing and refining a large
set of candidate transforms, until only one remains. The cited
phase deviation process then follows, establishing the x- and
y-translation of the watermark pattern within the imagery. In
certain embodiments of the present technology, the direct least
squares technique does not winnow candidate transforms down to one;
instead, two or more top candidates are output. The phase deviation
process is applied to each, yielding multiple candidate affine
poses. The best pose is selected--the one yielding the smallest sum
of phase deviations between the measured and expected phases of
each of the reference signals. By such arrangement, fewer items are
left unread on their first pass through the apparatus, minimizing
the need for reprocessing.
[0246] Sometimes a pattern of reflected light from a textured
surface patch--especially with transparent plastic bottles--can
appear inverted: dark for light, and light for dark. Additionally,
a pattern may appear flipped (mirrored), as when a transparent
texture surface is read from the underside. Accordingly, after the
scale and rotation are established (whether by direct least
squares, correlation, or another approach), multiple versions of
the imagery are submitted for analysis by a process to determine x-
and y-translation (whether by phase deviation, or correlation). One
version is inverted--black for white (darker for lighter). Another
version is mirrored (left for right). Another is the original
imagery. Only one of these will properly synchronize with the known
phase properties of the reference signal peaks by which translation
is determined; no match will be found for the others. Again, such
measure helps maximize the number of plastic items that are read on
their first pass through the apparatus, minimizing the need for
reprocessing.
[0247] (In some embodiments, the decoder is configured to test the
polarity of the message chips encoded in each sub tile block (e.g.,
32 by 32 waxels with 16 sub-tiles per tile) to assess whether the
message chips are inverted. Such a decoder executes the test by
correlating the watermark signal for the sub-tile to check whether
it has a positive or negative correlation peak. A negative peak
indicates that the signal is inverted, and the decoder inverts
chips from such an inverted sub-tile prior to aggregating them with
chips of other sub-tiles. Correlation can be performed with known
or fixed parts of the watermark signal.)
[0248] Applicant has found it is sometimes advantageous to capture
frames with different imaging parameters--each depicting a common
area of belt. For example, a single camera may alternate between
shorter and longer exposure intervals, e.g., 20 and 100
microseconds, in successive frames. Or two cameras may capture
imagery of a common area of the belt--one with a relatively larger
aperture (e.g., f/4) and one with a relatively smaller aperture
(e.g., f/8). Or with different exposure intervals. The resulting
variation in captured imagery helps assure that the small
variations associated with watermark encoding are readily
detectable--despite the wide range of luminances reflected from the
items being imaged.
[0249] When plastic material is molded, a first surface of the
material is commonly adjacent a shaped mold surface, while a
second, opposite surface is not. This opposite surface may
nonetheless be shaped, as when a vacuum draws the first surface of
the material into the mold, and the second surface follows. But the
physical definition of the second surface is not as good; it lacks
high frequency detail. This second surface, however, may be the one
that is imaged by the camera (e.g., as may happen with a carbon
black plastic tray on which meat is packaged, which may be
presented either top-up or bottom-up to the camera). To redress
this problem, some or all captured frames (or excerpts) can be
processed to accentuate high frequency detail.
[0250] In one exemplary embodiment, if no reference signal is found
in an analyzed block, the block is processed by an unsharp mask
filter, and the analysis is repeated--in the expectation that such
processing will aid in detecting a watermark reference signal
depicted from the backside of molded plastic. In another exemplary
embodiment, if a reference signal is detected in a block, but
payload extraction fails, then the block is processed by an unsharp
mask filter, and the payload extraction operation is re-tried.
More on Plastic Shaping, Etc.
[0251] The following discussion further details technology for
encoding plastic containers and labels to carry machine-readable
indicia. Included are details for overcoming certain signal
distortions introduced in plastic container design and
production.
[0252] To review briefly, watermarks are optical codes that
typically comprise a 2D pattern of code signal elements, generally
in a square block, that can be tiled with other blocks,
edge-to-edge, to span a surface. Each square array can be regarded
as a "grid" of encoding locations. In some embodiments, each
location is marked to represent one of two data, e.g., "-1" or "1."
(The two data can be "0" and "1" in other embodiments.)
[0253] Applicant's earlier-cited document 20040156529 describes how
to apply a code signal by etching a mold with a pattern carrying
data. After the pattern carrying the desired data is determined,
the pattern is used to texture the surface of plastic by forming
the plastic in the mold. For an injection molding process, the mold
is etched by a computer-driven etching apparatus. Each cell in the
output grid (array) pattern corresponds, e.g., to a
250.times.250-micron patch on the mold. If the output grid pattern
for a particular cell has a value of "1," a depression is formed in
a corresponding patch on the mold surface. If the output grid
pattern in a cell has a value of "-1" (or "0") no depression is
formed. The depth of the depressions depends on aesthetic
considerations. Typical depressions have a depth less than a half
millimeter and may be on the order of the patch size (250 microns)
or less. The resulting pattern of mold-pitting is a physical
manifestation of the output grid pattern. When the mold is used to
form the surface of a product container, the negative of this
pattern is created, with each pit resulting in a raised point on
the container.
[0254] The size of the textured region depends on the patch size,
and the number of rows/columns in the output grid pattern. The
larger the textured region, the more "signal" is available for
decoding, and the less exacting can be the specifications of the
reading device. A textured region about one centimeter on a side
has been found to provide more than adequate signal. Smaller
textured regions (or larger) can be used, depending on the
application requirements.
[0255] Techniques other than computer-controlled etching
apparatuses can be used to shape the mold in accordance with the
output grid signal. A small computer-controlled milling machine can
be used. So can laser cutting devices.
[0256] While the foregoing approach contemplates that the container
is formed with the texture already on it, in other embodiments, the
container can be formed with a flat surface, and the texturing
applied later, as by a heated press mold, assuming the packaging
material is thermoplastic.
[0257] To enhance the "signal" conveyed by the texturing, surface
changes can be made corresponding to both "1" and "-1" values in
the output pattern grid (instead of just corresponding to the "1"
values, as described above). Thus, raised areas are formed in
patches corresponding to "1" valued output pattern cells, and pits
are formed corresponding to "-1" valued output pattern cells.
[0258] In other embodiments, the texturing can also be applied by
an additional layer of material applied to the container in the
desired output pattern after it has been formed. For example, a
viscous ink can be applied in a screen-printing process. The screen
has an opening where the corresponding cell of the output grid
pattern has a "1" value, and no opening otherwise. When the viscous
ink is applied through the screen, small patches of ink are
deposited where the screen had openings, but not elsewhere.
[0259] Patches larger than 250 microns may be employed in such
embodiments, depending on the resolution limitations of the
screen-printing process. The result is again a textured surface,
with the pattern of raised areas conveying the binary data
payload.
[0260] Various material other than ink can be applied to form the
textured layer on top of the container. Thermoplastics and epoxy
resins are just two alternatives.
[0261] In some such embodiments, techniques other than printing are
used to apply a textured layer to the container. For example,
various photolithographic techniques can be used. One technique
employs a photo-reactive polymer, which is applied to the surface
and then optically exposed through a mask corresponding to the
output grid pattern. The exposed polymer is developed, thereby
removing patches of material.
[0262] In still other embodiments, the output grid pattern is
printed onto the container surface in two contrasting colors (e.g.
black and white). Cells having a "1" value can be printed in one
color, and cells having a "-1" value can be printed in another. In
such embodiments, the binary payload is not discerned from a
pattern of textures, but rather from a pattern of contrasting
colors.
[0263] Applicant's other patent documents identified herein detail
other procedures for physically realizing 2D optical codes on
items, as further discussed in priority application 62/814,567.
[0264] To counteract signal distortion in the design and/or
manufacturing of plastic containers, a variety of methods can be
employed.
[0265] In a first embodiment, signal encoding is introduced to a
container mold during 3D printing of the mold. The mold's interior
surface, which contacts the outer surface of a container, is
printed to include a subtle texture, pattern, image or design. The
texture, pattern, image or design carries the encoded signal. For
example, a raw sparse watermark signal is generated, e.g., as
detailed in published documents US20170024840, US20190139176 and
pending application Ser. No. 16/405,621. We use the term "raw" here
to mean that the sparse watermark signal has not been combined with
a host image or surface. The raw sparse watermark is used as a
template to guide 3D printing the interior surface of the mold. The
mold's surface includes varied bumps and pits that collectively
(and often redundantly) carry the raw sparse watermark.
[0266] Let's look at the workflow. A 3-dimensional (3D) mold is
designed in CAD software, such as AutoCad, Photoshop, Solidworks,
Materialise, or many others. The CAD software defines the geometry
of the mold. For example, the mold may be shaped to produce a water
bottle, a yogurt cup, or other container. A 2D encoded signal
(e.g., a sparse watermark) is generated. At this point the 2D
watermark signal needs to be mapped onto the 3D interior surface of
the mold, preferably in a manner that minimizes distortion of the
encoded signal.
[0267] One approach to minimizing distortion utilizes 1-directional
predistortion based on relative size of an expected container.
We'll use an hour-glass shaped container as an example. The radius
at the middle of such container is smaller than the radius at the
top and bottom. If mapping a 2D rectangularly shaped watermark tile
to this container, there would be a potential for different scaling
in the middle of the container relative to the top and bottom.
Thus, a watermark tile may be stretched more in one spatial
dimension (x axis) than another (y axis). This type of distortion
is sometimes referred to as differential scale or shear. Consider
an example where an original watermark tile is a square. As a
result of differential scale, the square may be warped into a
parallelogram with unequal sides. Differential scale parameters
define the nature and extent of this stretching. Differential scale
can cause particular problems for a watermark detector. When
looking at an embedding tile with a square shape having x and y
coordinates, with equal x and y sides, the x dimension is reduced
in the middle of the container when applied while the y dimension
remains generally the same length. If the middle radius is about
0.75 relative to the top and bottom radius, the x coordinates
shrinks when mapped to the surface by about 0.75*x while the y
coordinates remain generally the same (1*y). This results in a
differential scale for the x and y coordinates, which is like
creating an angle of image capture of about 41 degrees, making
detection of the sparse watermark difficult.
[0268] One objective of a solution, on the encoding side, is to
create an encoded signal that, once rendered to the surface of a
mold, is in an orientation range that is detectable by a decoder.
For example, the signal is preferably within a scale, rotation and
translation state that a detector can find. Differential scale is
particularly hard to re-register for data extraction. To address
this differential scale problem, we strive to have the x and y
coordinates of the tile retain a similar dimension relative to one
another after mapping to a 3D surface. So, we pre-distort the tile
in 1-direction prior to embedding. In particular, we pre-distort
the tile in the y direction by a similar amount as is expected by
any x direction distortion. The result, after pre-distortion and
mapping, is similar dimensions in terms of x and y sides, albeit,
with resulting in smaller embedding tile. The y direction of
different tiles placed over the surface can be individually
determined by the relative size of a radius at each embedding
location. The pre-distortion will vary across the mold based on
where a tile is placed on the 3D surface. (This same distortion
correction process can be used for when applying labels on
containers, e.g., heat shrink wrap labels, onto curved containers.
The y direction of embedding tiles can be altered to include the
same expected scale as x direction after heat shrinking.)
[0269] Another approach to minimizing distortion utilizes so-called
UV texturing (or mapping). UV texturing utilizes polygons that make
up a 3D object to be textured by surface attributes from a 2D image
(e.g., a "UV texture map"). The texture map has coordinates U, V,
while a 3D objects has coordinates X, Y, Z. UV mapping assigns
pixels in the UV texture map to surface mappings on a polygon. This
can be achieved by copying a triangular piece of the UV texture map
and pasting it onto a triangle on the 3D object. UV texturing is an
alternative mapping system it that only maps into a texture space
rather than into the geometric space of the object. The rendering
computation uses the UV texture coordinates to determine how to
arrange the three-dimensional surface. UV texturing can be used to
convey a 2D sparse watermark (or other 2D encoded signal) onto a
surface of a mold. Here, the sparse watermark is used as the UV
texture map, which is used to texture the surface of the mold.
Differing grayscale levels within the UV texture map can be used to
indicate mold surface texture depth or height. The resulting mold's
textured surface carries the watermark signal.
[0270] Even further detection improvements can be achieved when
using the above 1-dimensional scaling or UV texture mapping prior
to 3D printing or laser texturing when used in combination with
advanced decoding techniques within a detector, as described in
assignee's patent documents U.S. Pat. No. 9,182,778 (e.g.,
including direct least squares methods for recovering geometric
transform in a detector), and U.S. Pat. No. 9,959,587 (e.g., using
direct least squares for perspective approximation, and using phase
estimation in coordinate update and correlation metrics), and U.S.
patent application Ser. No. 15/588,451, filed May 5, 2017 (e.g.,
using direct least squares to improve perspective distortion (tilt)
performance) and document U.S. Pat. No. 10,242,434 (e.g., a
detector using a hybrid of complementary methods for geometric
registration like Log polar for low tilt angles/weak signals and
direct least squares for higher tilt angles). Such detection
techniques will help recover signals that have been distorted
during mold creation and/or distorted during image capture of
containers manufactured using such molds. For example, mold
creation may introduce first distortion associated with 2D mapping
to a surface of a 3D object, and image capture may introduce tilt,
scale, and/or rotation from a camera angle relative to the
container.
[0271] In still further embodiments, we utilize distortion
resolving techniques discussed in applicant's patent documents to
compensate for mapping 2D signals into 3D molds. See, e.g., patent
documents U.S. Pat. Nos. 6,122,403, 6,614,914, 6,947,571,
7,065,228, 8,412,577, 8,477,990, 9,033,238, 9,182,778, 9,349,153,
9,367,770, 9,864,919, 9,959,587, 10,242,434, and just-cited U.S.
patent application Ser. No. 15/588,451, filed May 5, 2017.
[0272] Surface texture in a mold's interior surface is used to
create an encoded signal in a plastic container. The texture is
created by creating bumps and/or pits in the mold surface, which
result in pits or bumps on the container. For example, for a sparse
watermark tile, each embedding location corresponds, e.g., to a
n.times.m inch patch on the mold. If an embedding location has a
value of "1," a depression is formed in a corresponding patch on
the mold surface. If the embedding location has a value of "-1," no
depression (or a pit) is formed. Thus, raised areas are formed in
containers corresponding to "1" valued embedding location, and
unaltered areas (or pits) are formed corresponding to "-1" valued
embedding location. When an image of a marked container is
analyzed, the bumps and pits will have different reflective
properties. These differences can be analyzed to decode the encoded
signal.
[0273] Returning to the workflow, after defining the geometry of
the mold, generating a 2D encoded signal (e.g., a sparse
watermark), and mapping the watermark signal to the 3D interior
surface of the mold, we produce a corresponding a 3D printer file
format (e.g., STL, OBJ, AMF, or 3MF) to control a 3D printer. The
3D printer prints the mold, including the surface texture according
to the encoded signal pattern. Of course, a 3D printer needs to be
capable of printing at a resolution that accommodates the encoded
signal pattern. For example, if an encoded signal pattern
corresponds to 50 or 100 dots per inch, the printer would need to
be able to replicate such.
[0274] In other embodiments, instead of using a sparse watermark to
guide surface texturing, a neural network generated signal, or
Voronoi, Delaunay, or stipple half-toning-based signal can be used.
Such signals are discussed in, e.g., PCT/US18/64516 and U.S. patent
application Ser. No. 16/435,164, filed Jun. 7, 2019.
[0275] In other embodiments, encoding is introduced to a mold
surface with laser engraving, etching, embossing or ablation. The
machine-readable indicia (carried by surface topology changes
within a mold) is imparted to the plastic as a container is being
formed. Very fine texture patterns can be achieved with laser
engraving and tool etching. Recently, laser texturing for molds has
developed to the point where it can produce different depth levels.
Multiple different levels of depth can be utilized to carry
different signal information. For example, in signal value terms, a
first depth may represent a "1," while a second depth may represent
a "0," and a third depth may represent a "-1". Like above, UV
texture mapping and/or 1-directional predistortion can be used to
combat 2D to 3D transformations.
[0276] Another consideration for 3D printed molds, laser engraved
molds and etched molds, is that container surface signals must be
able to survive formation, not degrade the finished container
(e.g., create too thin of areas), and facilitate container release
from a mold (e.g., not get stuck in a mold). For example, if the
mold creates a bump or raised area on the container, the
corresponding indentation in the mold should be shaped to
facilitate container release from the mold. For example, if the
mold includes sharp and deep pits (corresponding to sharp and high
bumps on the container), the container may not release from the
mold. The mold pits bumps can be directionally shaped, e.g.,
teardrop-shaped (or roche moutonnee-shaped), in the direction of
the mold release. Or the pits can be shaped to match tool release
draft angles for a tool, material type and/or piece shape.
[0277] Similar considerations should be had for sintered metal or
ceramic parts where a watermark is carried by surface texture. The
watermark textured piece must release from a mold without
deformation before it's fired, and watermark texture will deform
along with the piece during sintering. Expected deformation can be
corrected for with predistortion of the watermark signal.
[0278] In still another embodiment, watermark texture may be formed
of linear striped ridges with stripe position modulation. A mold
including such linear ridges can be spatially moved, and/or
increased/decreased in size, to represent a watermark signal.
Mold Generation Particulars
[0279] One particular example takes a sparse watermark signal and
places shapes or structures at dot locations. Instead of marking
square dots, the 3D surface topology is preferably formed with
smoothed divots, pits or peaks, e.g., shaped with Gaussian or
sinusoidal like curves. Another example forms line art or elemental
features (like circles, lines, ovals, etc.) that coincide with
peaks and/or valleys of a continuous watermark signal. Another
example forms a 3D surface pattern of signal rich art designs
described in PCT/US18/64516 and U.S. application Ser. No.
16/435,164, filed Jun. 7, 2019, including but not limited to
Voronoi, stipple, Delaunay and traveling salesman patterns, to name
a few. In such examples, the topology is formed such that the cross
section of the pattern of surface peaks or depressions is smoothed
(e.g., in the form of a sinusoid or Gaussian cross section. The
cross section achievable is dependent on marking type (CNC milling,
laser marking, 3D printing) and should be designed to ensure proper
part release from the mold. The smoothing of contours should
address the latter objective.
[0280] In the following examples, design objectives for converting
2-dimensional data carrying signals into a mold are discussed.
[0281] In choosing a signaling type (e.g., continuous vs. binary
vs. sparse), various factors come into play, such as the type of
plastic being molded (PET, PP, PE, HDPE, LDPE, etc.), the type of
manufacturing process (e.g., blow molding, injection molding,
thermoforming, etc.), the type of mold (metal, ceramic, etc.), the
mold making process (etching, engraving, etc.), aesthetics, and
attributes of camera/lighting used for detection. In particular,
continuous signaling typically requires higher resolution both
spatially and in depth (of embossing, debossing, etching, etc.).
Binary signaling typically includes higher resolution spatially but
low resolution in terms of depth. Sparse binary signaling can be
performed when available spatial and depth resolution are both
low.
[0282] Another factor to be considered is a ratio between reference
(synchronization) signal strength, and message signature strength.
Ensuring sufficiently strong message signal strength relative to
the synchronization signal component enhances reliability in
recovery of the digital payload. For sparse and binary marks,
synchronization signal-to-message signal ratio can be determined
heuristically based on specified watermark resolution, image
resolution, dot size, dots per tile, and payload size. A different
set of heuristics can be created for different types of plastics,
molds, etc. For example, the properties of the plastic (e.g.,
intrinsic viscosity of sheet grade, bottle grade and film grade
PET) may determine how conducive the embossed plastic is to carry
spectral features (e.g., low vs. high frequencies) of a watermark
signal. Similar considerations also apply to continuous and binary
signaling
[0283] Still another factor to be considered is watermark signal
resolution. The resolution of a watermark signal in each signal
block (tile) should be sufficiently high to achieve desired
aesthetics, while enabling the watermark payload to be read from
small tiles so that object curvature introduces less geometric
deformation across each tile. In one example, a recommended
resolution is 200 watermark cells (waxels) per inch (WPI), or
higher. With a tile size of 128 by 128 waxels, the tile dimensions
for a 200 WPI tile are then 0.64 inch by 0.64 inch.
[0284] In addition to improved detection in objects with
non-rectangular geometries, higher resolution watermarks enable
improved detection from flattened, crushed, deformed, or shredded
objects as found in a recycling stream.
[0285] Decreasing the dot density within each watermark tile has
various benefits. For example, visibility of the signal pattern on
the molded object is less, meaning that it interferes less with the
visual quality and aesthetics of the object. For clear containers,
the signal pattern has less visual impact on the contents within
the container (e.g., the water in a clear plastic water bottle).
Moreover, since dots are translated into bumps or
pits/divots/recesses in an object surface, fewer dots means the dot
spacing is less, making it easier to form the corresponding shapes
in a mold. Techniques for making mold surface topology (e.g., bumps
or pits/divots/recesses) include, e.g., laser engraving, etching,
electrical discharge machining (e.g., so-called "spark erosion"),
Computer Numerical Control ("CNC") milling, or 3D printing. When
using a CNC milling bit, care can be taken to ensure adequate
resolution. Marking equipment with a larger marking width can be
used to remove surface material, leaving contoured bumps with a
diameter smaller than the bit width. Bit shapes can be varied to
achieve desired dot representation including, e.g., conical bits,
triangle bits, circle cross section, ball mill. Additionally,
divots may need not be deep, but can use intensity variation.
Fewer, more widely spaced bumps/divots also makes it more feasible
to do smoothing of the contours of the bumps and divots in the
mold.
[0286] Dot density can be expressed as a percentage of dots
relative to a tile with maximum percentage of dot coverage. The
maximum percentage of dot coverage in a watermark signal tile
comprising a binary pattern of dot or no dot per cell is 50%. This
means that half of the cells (waxels) in the tile are marked with
dots (e.g., dark values). Dot density should preferably be lower
for lower visibility, e.g., 10 to 35 (meaning 5 to 17.5% of the
tile marked with dots).
[0287] Dot size is mentioned above. Dot size is a parameter that
controls the size of the elemental dot structure within a sparse
signal. We indicate dot size at a specified image resolution in
dots per inch (DPI), e.g., 600 DPI, meaning 600 pixels per inch.
Dot size is an integer value indicating dimension of a dot along
one axis in pixels at a given image resolution. A dot size of 1
means that the dot is one pixel. A dot size of 2 means that the dot
is two pixels (e.g., arranged in a row, column, or along a diagonal
in a two-dimensional array of pixel coordinates). For example, a
dot size 1 or 2 at 600 DPI translates to a dot width of 42 or 84
microns. A depression having this dot width can be made with a
larger bit size (e.g., 257 microns) because the bit need only
partially be pressed into the surface of an aluminum mold.
[0288] Dots can be of different shapes. Although square dots can be
represented easily in an image in the form of pixels, there may be
more suitable shapes and structures for encoding the signal in
physical material such as plastic or metal. Examples include
circles, ovals, lines, etc. Smoother shapes may be easier to
produce (for example, due to ease of manufacturing the mold) than
shapes with sharp edges or corners.
[0289] Different types of plastics, molds and mold-making allow for
different depth of marking on the surface of plastic, e.g., deeper
or more superficial. Generally, when deeper marking is available,
lower dot density can be used, whereas when marking is superficial,
a higher density of dots is typically employed. Deeper marking is
more likely to survive workflow alterations such as surface
abrasions, flattening and crushing.
[0290] The image signal representation of a watermark tile provided
for conversion into a 3D surface topology for a mold, may be a
vector or rasterized image file, such as an SVG image format. Image
editing tools like Adobe Photoshop, design tools such as Adobe
Illustrator or signal processing software such as MATLAB from Math
Works, can be employed to generate the files, e.g., in conjunction
with watermarking tool plug-ins from applicant Digimarc Corp.
[0291] In the electronic image file, dots can be of different
shapes. Although square dots can be represented easily in an image
in the form of pixels, different shapes and structures are
generally more suitable encoding the signal in physical material
such as plastic or metal. Examples include circles, ovals, lines,
etc. Smoother shapes are easier to reproduce, for example, due to
ease of manufacturing the mold, than shapes with sharp edges or
corners. A vector representation enables the dots to be defined in
terms of a dot shape that is beneficial for aesthetics of the
finished, molded product and for performance of the mold.
Performance of the mold considerations are the tapering, smoothing
or contouring of the divots or bumps so that the molded part
releases from the mold. In a simple example, dots have a circle
shape, which facilitates shaping into the surface of an aluminum
mold, for example, using CNC equipment. The 3D structure of the
shape plays a role in ease of manufacturing (e.g., release) as well
as in the induced light intensity variations that carry the
modulations necessary for encoding the watermark signal. Forms of
signal rich art (e.g., as discussed in U.S. application Ser. No.
16/435,164, filed Jun. 7, 2019 and PCT/US18/64516) can be created
by selectively placing objects of a desired shape at dot locations,
and/or drawing vector art through dot locations so that the vector
art is highly correlated with the watermark signal at the dot
locations.
[0292] The resolution of the tile image (e.g., in DPI) determines
the granularity of the modulations that can be made in the
material. Use of higher resolution (e.g., 600 DPI) provides more
flexibility in designing the features (e.g., dots or other
structures) that can be embossed, debossed, etched, milled, eroded,
etc. Use of higher resolution also provides more flexibility in
signal formation, for example, in creating sparse binary marks by
allowing more leeway in choice of keep-out areas, dot shapes,
sizes, etc.
Review of Exemplary Watermarking Methods
[0293] In an exemplary watermarking method, a plural-symbol message
payload (e.g., 48 binary bits, which may represent a product's
Global Trade Identification Number (GTIN) or plastic recycling
information, together with 24 associated CRC bits) is applied to an
error correction coder. This coder transforms the symbols of the
message payload into a much longer array of encoded message
elements (e.g., binary or M-ary elements) using an error correction
method. (Suitable coding methods include block codes, BCH, Reed
Solomon, convolutional codes, turbo codes, etc.) The coder output
may comprise hundreds or thousands of binary bits, e.g., 1024,
which may be termed raw signature bits. These bits may be scrambled
by XORing with a scrambling key of the same length, yielding a
scrambled signature.
[0294] Each of these scrambled signature bits modulates a
pseudorandom noise modulation sequence (spreading carrier) of
length 16, e.g., by XORing. Each scrambled signature bit thus
yields a modulated carrier sequence of 16 "chips," producing an
enlarged scrambled payload sequence of 16,384 elements. This
sequence is mapped to elements of a square block having
128.times.128 embedding locations in accordance with data in a
scatter table, yielding a 2D payload signature pattern. (To each of
four 64.times.64 quadrants in the 128.times.128 block, the scatter
table assigns 4 chips for each scrambled signature bit.) Each
location in the 128.times.128 block is associated with either a
value of 0 and 1, or black and white--with about half of the
locations having each state. This bimodal signal is frequently
mapped to a larger bimodal signal centered at an eight-bit
greyscale value of 128, e.g., with values of 95 and 161. Each of
these embedding locations may correspond to a small region of
pixels, such as a 2.times.2 patch, termed a "bump," which yields a
watermark message block having dimensions of 256.times.256
pixels.
[0295] As noted, a synchronization component is commonly included
in a digital watermark, to help discern parameters of any affine
transform to which the watermark has been subjected prior to
decoding, so that the payload can be correctly decoded. A
particular synchronization component takes the form of a reference
signal comprised of a dozen or more magnitude peaks of sinusoids of
pseudorandom phase, in the Fourier domain. This signal is
transformed to the spatial domain in a 256.times.256 block size
(e.g., by an inverse Fast Fourier transform), corresponding to the
256.times.256 block to which the enlarged scrambled payload
sequence is mapped. The spatial domain reference signal, which may
comprise floating-point values between -1 and 1, can be scaled to a
range of -40 to 40 and combined with the 256.times.256 pixel
payload block to yield a final watermark signal block, e.g., having
values ranging from 55 (i.e., 95-40) to 201 (i.e., 161+40). This
signal can then be summed with host imagery, after first
scaling-down to render it inconspicuous.
[0296] If such a watermark signal block is printed at a spatial
resolution of 300 dots per inch (DPI), a printed block of about
0.85 inches square results. (Since the 0.85 inch side dimension
corresponds to 128 waxels, this works out to 150 waxels per inch.)
Such blocks can be tiled edge-to-edge for marking a larger
surface.
[0297] The just-described watermark signal may be termed a
"continuous tone" watermark signal. It is usually characterized by
multi-valued data, i.e., not being just on/off (or I/O, or
black/white)--thus the "continuous" moniker. Each pixel of the host
image (or of a region within the host image) is associated with one
corresponding element of the watermark signal. A majority of the
pixels in the image (or image region) are changed in value by
combination with their corresponding watermark elements. The
changes are typically both positive and negative, e.g., changing
the local luminance of the imagery up in one location, while
changing it down in another. And the changes may be different in
degree--some pixels are changed a relatively smaller amount, while
other pixels are changed a relatively larger amount. Typically, the
amplitude of the watermark signal is low enough that its presence
within the image escapes notice by casual viewers (i.e., it is
steganographic).
[0298] (Due to the highly redundant nature of the encoding, some
embodiments can disregard pixel changes in one direction or
another. For example, one such embodiment only changes pixel values
in a positive direction. Pixels that would normally be changed in a
negative direction are left unchanged.)
[0299] In a variant continuous tone watermark, the signal acts not
to change the local luminance of artwork pixels, but rather their
color. Such a watermark is termed a "chrominance" watermark
(instead of a "luminance" watermark). An example is detailed, e.g.,
in U.S. Pat. No. 9,245,308.
[0300] "Sparse" or "binary" watermarks are different from
continuous tone watermarks. They do not change a majority of pixel
values in the host image (or image region). Rather, they have a
print density (which may sometimes be set by the user) that results
in marking between about 5% and 45% of pixel locations in the
image. Adjustments are typically all made in the same direction,
e.g., reducing luminance. Sparse elements are typically bitonal,
e.g., being either white or black. Although sparse watermarks may
be formed on top of other imagery, they are usually presented in
regions of artwork that are blank or colored with a uniform tone.
In such cases a sparse marking may contrast with its background,
rendering the marking visible to casual viewers. Although sparse
marks may take the form of a field of seemingly-random dots, they
can also take the form of line structures, as detailed elsewhere.
As with continuous tone watermarks, sparse watermarks generally
take the form of signal blocks that are tiled across an area of
imagery.
[0301] A sparse watermark can be produced from a continuous-tone
watermark by thresholding. That is, the darkest elements of the
summed reference signal/payload signal blocks are copied into an
output signal block until a desired density of dots is
achieved.
[0302] Patent publication US20170024840 details various other forms
of sparse watermarks. In one embodiment, a signal generator starts
with two 128.times.128 inputs. One is a payload signal block, with
its locations filled with a binary (0/1, black/white) expanded
scrambled payload sequence, as described above. The other is a
spatial domain reference signal block, with each location assigned
a floating point number between -1 and 1. The darkest (most
negative) "x"% of these reference signal locations are identified,
and set to black; the others are set to white.
Spatially-corresponding elements of the two blocks are ANDed
together to find coincidences of black elements between the two
blocks. These elements are set to black in an output block; the
other elements are left white. By setting "x" higher or lower, the
output signal block can be made darker or lighter.
[0303] Publication US20190171856, International PCT application No.
PCT/US19/19410, filed Feb. 25, 2019, and U.S. application Ser. No.
16/405,621, filed May 7, 2019, detail additional sparse encoding
embodiments. One embodiment uses a reference signal generated at a
relatively higher resolution (e.g., 384.times.384 pixels), and a
payload signature spanning a relatively lower resolution array
(e.g., 128.times.128). The latter signal has just two values (i.e.,
it is bitonal); the former signal has more values (i.e., it is
multi-level, such as binary greyscale or comprised of
floating-point values). The payload signal is interpolated to the
higher resolution of the reference signal, and in the process is
converted from bitonal form to multi-level. The two signals are
combined at the higher resolution (e.g., by summing in a weighted
ratio), and a thresholding operation is applied to the result to
identify locations of extreme (e.g., dark) values. These locations
are marked to produce a sparse block (e.g., of 384.times.384). The
threshold level establishes the dot density of the resulting sparse
mark.
[0304] A different embodiment sorts samples in a block of a
reference signal by darkness, yielding a ranked list of the darkest
N locations (e.g., 1600 locations), each with a location (e.g.,
within a 128.times.128 element array). The darkest of these N
locations are always-marked in an output block (e.g., 400
locations), to ensure the reference signal is strongly expressed.
The others of the N locations are marked, or not, depending on
values of message signal data that are mapped to such locations
(e.g., by a scatter table in the encoder). Locations in the sparse
block that are not among the N darkest locations are never marked,
and they are consequently affirmatively ignored by the decoder. By
setting the number N larger or smaller, sparse marks with more or
fewer dots are produced. (This embodiment is termed the "fourth
embodiment" in earlier-cited application Ser. No. 16/405,621 and
PCT/US19/19410.)
[0305] In generating a sparse mark, a spacing constraint can be
applied to candidate mark locations to prevent clumping. The
spacing constraint may take the form of a keep-out zone that is
circular, elliptical, or of other (e.g., irregular) shape. The
keep-out zone may have two, or more, or less, axes of symmetry (or
none). Enforcement of the spacing constraint can employ an
associated data structure having one element for each location in
the tile. As dark marks are added to the output block,
corresponding data is stored in the data structure identifying
locations that--due to the spacing constraint--are no longer
available for possible marking.
[0306] In some embodiments, the reference signal can be tailored to
have a non-random appearance, by varying the relative amplitudes of
spatial frequency peaks, so that they are not all of equal
amplitude. Such variation of the reference signal has consequent
effects on the sparse signal appearance.
[0307] A sparse pattern can be rendered in various forms. Most
straight-forward is as a seemingly-random pattern of dots. But more
artistic renderings are possible, including those discussed and
illustrated above.
[0308] Other overt, artistic patterns conveying watermark data are
detailed in U.S. patent application Ser. No. 16/129,487. In one
detailed approach, a designer creates a candidate artwork design or
selects one from a library of designs. Vector art in the form of
lines or small, discrete print structures of desired shape work
well in this approach. A payload is input to a signal generator,
which generates a raw data signal in the form of two-dimensional
tile of data signal elements. The method then edits the artwork at
spatial locations according to the data signal elements at those
locations. When artwork with desired aesthetic quality and
robustness is produced, it is applied to an object, e.g., by laser
marking.
[0309] Other techniques for generating visible artwork bearing a
robust data signal are detailed in assignee's U.S. patent
application Ser. No. 16/212,125, filed Dec. 6, 2018, and
62/841,084, filed Apr. 30, 2019. In some embodiments, a neural
network is applied to imagery including a machine-readable code, to
transform its appearance while maintaining its machine readability.
One particular method trains a neural network with a style image
having various features. The trained network is then applied to an
input pattern that encodes a plural-symbol payload. The network
adapts features from the style image to express details of the
input pattern, to thereby produce an output image in which features
from the style image contribute to encoding of the plural-symbol
payload. This output image can then be used as a graphical
component in product packaging, such as a background, border, or
pattern fill. In some embodiments, the input pattern is a watermark
pattern, while in others it is a host image that has been
previously watermarked.
[0310] Other such techniques do not require a neural network.
Instead, a watermark signal block (i.e., reference plus message
signals) is decomposed into sub-blocks. A style image is then
analyzed to find sub-blocks having the highest correlation to each
of the watermark signal sub-blocks. Sub-blocks from the style image
are then mosaiced together to produce an output image that is
visually evocative of the style image, but has signal
characteristics mimicking the watermark signal block.
[0311] In addition to the references cited elsewhere, details
concerning watermark encoding and reading that can be included in
implementations of the present technology are disclosed in
applicant's previous patent filings, including US patent documents
U.S. Pat. Nos. 5,850,481, 6,122,403, 6,590,996, 6,614,914,
6,782,115, 6,947,571, 6,975,744, 6,985,600, 7,044,395, 7,065,228,
7,123,740, 7,130,087, 7,403,633, 7,763,179, 8,224,018, 8,300,274,
8,412,577, 8,477,990, 8,543,823, 9,033,238, 9,349,153, 9,367,770,
9,521,291, 9,600,754, 9,749,607, 9,754,341, 9,864,919, 10,113,910,
10,217,182 and US20160364623, and in pending U.S. application Ser.
No. 16/141,587, filed Sep. 25, 2018, Ser. No. 16/270,500, filed
Feb. 7, 2019, 62/814,567, filed Mar. 6, 2019, 62/820,755, filed
Mar. 19, 2019, 62/824,934, filed Mar. 27, 2019, and Ser. No.
16/405,621, filed May 7, 2019.
[0312] While the just-detailed technologies are often described in
the context of print watermarking, the same techniques can be used
for 3D texture/shape-based watermarking. The sparse dot and line
elements of binary marks can be represented by protrusions (or
depressions) in a 3D surface.
[0313] Similarly, the positive and negative variations in pixel
values associated with continuous-tone watermarks can be
represented by spatial variations in 3D surface height. In some
arrangements, the surface is changed in only one direction, e.g.,
by protrusions up from the surface. In other arrangements, the
surface may be changed in reciprocal directions, by both
protrusions up from, and depressions (pits) down into, the 3D
surface.
[0314] When the surface is changed in only one direction, one
implementation disregards negative variations in watermark values;
the surface is changed only by positive variations. Another
implementation disregards positive variations in the watermark
signal; the surface is changed only by negative variations. In both
such implementations, the surface changes may be either in the
positive, protrusion direction, or in the negative, depression
direction.
[0315] In still other implementations, the most negative variation
(extremum) of the continuous watermark signal maps to no change in
the surface, while increasingly-positive variations from this
extremum map to increasingly greater surface changes (either
protrusions or depressions). In yet other implementations, the most
positive variation of the continuous watermark signal maps to no
change in the surface, while increasingly-negative variations from
this value map to increasingly greater surface changes (again,
either protrusions or depressions).
[0316] When the surface is changed in two directions, negative
values of a continuous-tone watermark signal can map to depressions
into the surface (with depth depending on negative signal value),
while positive values of the watermarks signal can map to
protrusions from the surface (with height depending on positive
signal value). In other embodiments, the polarities can be
switched, with positive values of the watermark signal mapping to
depressions into the surface, and vice versa. The depths of the
deepest depressions, and the heights of the highest protrusions may
be equal, but they need not be. Ditto the average depression depth
and the average protrusion height. The depths/heights may be
asymmetrical--as if a DC offset was applied to the continuous-tone
watermark signal
[0317] In cases where the surface is changed in two directions,
both depressions and protrusions desirably convey watermark payload
information (unlike the arrangement of US20180345323, which teaches
that only one or the other conveys payload information).
CONCLUDING REMARKS
[0318] Having described and illustrated the principles of our
inventive work with reference to illustrative examples, it will be
recognized that the technology is not so limited. For example,
while the described embodiments involve imagery captured using
visible illumination, this is not essential. Other forms of
illumination, such as ultraviolet and infrared, can alternatively
be used.
[0319] Although plastic bottles are described as including both
printed watermarks and texture watermarks, it will be recognized
that certain of the present technologies constitute improvements to
texture watermarks--regardless of the presence or absence of
printed watermarks. Thus, for example, plastic bottles encoded with
recycling information using patterns like those shown in the
figures are improvements over prior art marking of plastic
containers with recycling data (which marking tends to be more
conspicuous and detract from package aesthetics). Similarly, other
improvements are applicable to watermarking generally, such as the
detailed clues for distinguishing empty conveyor from non-empty
conveyor.
[0320] Although the printed and textured watermarks in the detailed
embodiments employ reference signals comprising peaks of different
spatial frequencies, this is not essential to avoid confusion. In
other embodiments, both watermarks employ reference signals
comprising peaks of the same spatial frequencies. Or the reference
signal of one watermark can employ peaks at spatial frequencies
that are a subset of the peaks used in the reference signal of the
other watermark.
[0321] If the two watermarks' reference signals share some or all
spatial frequency peaks in common, the peaks in one reference
signal may be assigned phases different than the peaks in the other
reference signal, in order to avoid confusion. If differentiation
of the two watermarks by peak phases does not yield adequate
false-positive behavior, additional testing can be done. For
example, phases may be checked twice, on two different counterparts
of the captured image. These counterparts may be successive image
frames, or they can be a single image frame that is processed to
yield two images. For example, gaussian noise can be added to yield
a second image. Or a second image can be generated by discarding
even rows and columns of pixel data from the first image. Numerous
such possibilities exist. Only if the two phase-based
identifications of the watermark signal from the two counterpart
images agree is the result trusted.
[0322] In still other embodiments, the two watermarks employ
different scrambling keys, or different spreading keys, or
different scattering tables, to avoid confusion.
[0323] In embodiments in which the two watermarks' reference
signals employ spatial frequency peaks in common, the processing
arrangements can be simplified. For example, scale and rotation
synchronization for both watermarks can be performed by a common
processing stage, since such synchronization involves a common set
of reference signal peaks. Such methods are detailed, e.g., in
patent applications 62/834,260, filed Apr. 15, 2019, and
62/834,657, filed Apr. 16, 2019.
[0324] In one particular detector, a "supergrid" reference signal
construct is employed, including all peaks from both reference
signals. Scale and rotation of the input imagery is determined by
synchronizing with such a composite reference signal. Once such
synchronization is achieved, it is straightforward to determine
whether the input imagery includes one reference signal or the
other, e.g., by examining for peaks or phases that are unique to
one of the two reference signals.
[0325] Although the described embodiments employ a reference signal
comprised of peaks in the Fourier magnitude domain, it should be
recognized that reference signals can exhibit peaks in different
transform domains.
[0326] Relatedly, it is not necessary for a digital watermark
signal to include a distinct reference signal for geometrical
synchronization purposes. Sometimes the payload portion of the
watermark signal, itself, has known aspects or structure that
allows geometrical synchronization without reliance on a separate
reference signal.
[0327] While a GTIN payload data field from the label watermark in
the FIG. 13 embodiment is used to access corresponding plastic
type, etc., information from a database, this is not required.
Other fields of the label watermark can be used for this purpose,
e.g., detailing the various data types referenced herein. Indeed,
the use of a database in conjunction with label watermarks is not
essential; the payload can convey plastic data directly, such as in
one of the Application Identifier key value pairs.
[0328] Although the specification particularly describes 2D image
sensors, 2D sensors are not required. Image sensing can instead be
performed by a linear array sensor that captures line scan images
at a suitably-high rate.
[0329] While the surface shaping shown in some of the figures
employs primarily straight lines, this is, in part, a drafting
expedient. Commonly the surface texturing effects curved, tapered
shapes.
[0330] Reference was made to processing patches of captured imagery
of specified sizes in waxels. While the exact waxel-size of a patch
cannot be determined until its scale is assessed (e.g., using the
cited direct least squares method), the encoding scale of each
watermark that the system might encounter is known in advance, and
the imaging distance is fixed, so the scale-correspondence between
captured pixels and encoded waxels is roughly known, which is
adequate for the present purposes.
[0331] The image processing described herein is typically performed
on data that has previously been "oct-axis" (or "criss-cross")
filtered, as described in the cited references. In an illustrative
embodiment, oct-axis-filtered data can have any integer value in
the range of -8 to 8.
[0332] In some embodiments, after the affine parameters
characterizing item pose have been discerned, an estimate of the
reference signal is subtracted from the captured image, since the
reference signal then serves only a noise. Oct-axis processing can
then be applied to the remaining signal.
[0333] Although this specification has repeatedly referenced
plastic bottles, it will be recognized that the technology can be
used in conjunction with any items, e.g., trays, pouches, cups,
transport containers, etc.
[0334] For best diverter performance, a center of gravity of
detected plastic items is estimated, and this position is used in
controlling operation of the diverter mechanism (e.g., this
position is the target for a jet of compressed air). Each detection
of a watermark block serves as a data point in estimating the
center of gravity. In one example, the coordinates of the center of
each watermark block are determined in the coordinate system of the
image in which blocks are detected. These coordinates, in x- and
y-directions, are respectively averaged, to yield a center of
gravity for the object within that image frame. For example, in
FIG. 25, these coordinates average to indicate the position shown
by the bullseye. The spatial relationship between the camera field
of view and the diverter assembly is known, as is the speed of the
belt, enabling the diverter to be activated at an instant, and
aimed at a location, calculated to best divert the item from the
belt.
[0335] (If the belt is crowded with objects, watermark blocks can
be checked for payload consistencies before their locations are
averaged. If one watermark indicates one type of plastic, and a
nearby block indicates a different type of plastic, then they are
known to mark different items, and their coordinates should not
both be employed in a common average.)
[0336] There is a short interval of time between the moment an item
is imaged by the camera(s), and the moment the item is positioned
for diversion from the conveyor. This interval is generally
adequate to enable cloud processing. For example, captured imagery
(or derivatives of such imagery) can be transmitted to a remote
cloud computer service, such as Microsoft Azure, Google Cloud,
Amazon AWS. The cloud processor(s) can perform some or all of the
processing detailed herein, and return result data to the waste
processing system--in time to control the diverters
accordingly.
[0337] Likewise, in a waste stream in which some items literally
include plastic recycling codes in their payloads, while recycling
codes for other items must be obtained from a database (e.g., based
on a look-up from decoded GTIN identifiers), the short interval of
time prior to diversion allows time to consult a cloud database for
the needed recycling codes for the latter items.
[0338] It will be recognized that recycling systems employing
aspects of the present technology do not require a conveyor belt
per se. For examples, articles can be transported past the camera
system and to diverter systems otherwise, such as by rollers or by
free-fall. All such alternatives are intended to be included by the
term "conveyor belt."
[0339] While reference was made to processing captured imagery with
an unsharp mask filter, other filters (linear or non-linear) can
similarly be employed to emphasize high-frequency components of the
imagery (or, similarly, to de-emphasize low-frequency
components).
[0340] Although most of the detailed arrangements operate using
greyscale imagery, certain performance improvements (e.g., more
reliable identification of empty belt, and more reliable watermark
encoding) may be achieved by the greater-dimensionality of
multi-channel imagery. RGB sensors can be used. However, half of
the pixels in RGB sensors are typically green-filtered (due to
prevalence of the common Bayer color filter). Still better results
can be achieved with sensors that output four (or more) different
channels of data, such as R/G/B/ultraviolet. Or R/G/B/infrared. Or
R/G/B/polarized. Or R/G/B/white.
[0341] While described in the context of plastic items, it will be
recognized that many aspects of the present technology are
applicable with other items, such as items made of glass or
metal.
[0342] Similarly, while the technology has been described in the
context of digital watermarks, it will be recognized that any other
machine-readable marking can be used, such as DotCode and dot peen
markings (although certain benefits, such as readability from
different viewpoints, may be impaired). Document U.S. Pat. No.
8,727,220 teaches twenty different 2D codes that can be embossed or
molded into an outer surface of a plastic container.
[0343] As noted, image blocks in some embodiments are analyzed for
clues that suggest whether the blocks depict conveyor belt or not.
If not, further analysis is undertaken, such as analysis of a block
for a watermark reference signal. In other embodiments, blocks are
initially analyzed for the presence of a watermark reference
signal, and detection of such a reference signal serves as a clue.
Such reference signal detection triggers further analysis, such as
analysis of the block for payload data, and/or analysis of nearby
blocks, or spatially-displaced blocks in subsequent image frames,
for reference signals. (Typically, the type of reference signal
that is detected indicates whether the associated watermark is
likely of the printed or textured variety, so that a corresponding
decoding algorithm can be applied.)
[0344] As noted, the two watermarks contemplated by most
embodiments of the present technology differ in three respects:
form, payload, and signaling protocol. For the avoidance of doubt,
it should be understood that each of these attributes is distinct.
Two watermarks may differ in form (printed vs. textured), yet be
identical in signaling protocol and payload. Likewise, two
watermarks may differ in payload, yet be identical in form and
signaling protocol. Similarly, two watermarks may differ in
signaling protocol, yet be identical in form and payload.
(Signaling protocol encompasses all aspects of a watermark except
its form and payload, e.g., including reference signal, encoding
algorithm, output data format, payload length, syntax, etc.)
[0345] While reference is often made to watermark blocks that are
square in shape, it will be recognized that printed or textured
surfaces can likewise be tiled with watermark blocks of other
shapes. For example, a hexagonal honeycomb shape may be composed of
triangularly-shaped waxels.
[0346] Although the technology is described with reference to
detection of a watermark synchronization (reference) signal using a
direct least squares and phase deviation approach, other techniques
can also be used. One example is a coiled all-pose arrangement, as
detailed in WIPO patent publication WO2019/067730. Another option
is to use an impulse matched filter approach, (e.g., correlating
with a template comprised of peaks), as detailed in patent
documents U.S. Pat. Nos. 10,242,434 and 6,590,996.
[0347] It will be recognized that processing a surface to effect a
matte, or frosted, finish is a form of 3D surface
shaping/texturing, albeit on a very small scale. Generally, any
non-inked treatment that changes a surface's bidirectional
reflectance distribution function (BDRF) is regarded as a 3D
shaping/texturing operation herein.
[0348] Although the curved surfaces shown, e.g., in FIGS. 1H-1L,
are sections of spheres, and/or are symmetrical in cross-section,
neither is required. Sections of ellipsoids and more complex
(higher order) surfaces can more generally be employed. Some such
surfaces, when cross-sectioned by a plane normal to a nominal
surface of the plastic item, may have non-symmetrical shapes.
Indeed, some such surfaces are characterized in that they have no
cross-section, normal to the nominal plastic surface, that is
symmetrical.
[0349] It will be understood that the methods and algorithms
detailed above can be executed using computer devices employing one
or more processors, one or more memories (e.g. RAM), storage (e.g.,
a disk or flash memory), a user interface (which may include, e.g.,
a keypad, a TFT LCD or OLED display screen, touch or other gesture
sensors, together with software instructions for providing a
graphical user interface), interconnections between these elements
(e.g., buses), and a wired or wireless interface for communicating
with other devices.
[0350] The processes and system components detailed in this
specification can be implemented as instructions for computing
devices, including general purpose processor instructions for a
variety of programmable processors, such as microprocessors and
systems on a chip (e.g., the Intel Atom and i9 series, the ARM A8
and Cortex series, the Qualcomm Snapdragon, and the nVidia Tegra
4). Implementation can also employ a variety of specialized
processors, such as graphics processing units (GPUs, such as are
included in the nVidia Tegra series, and the Adreno 530--part of
the Qualcomm Snapdragon processor), and digital signal processors
(e.g., the Texas Instruments TMS320 and OMAP series devices, and
the ultra-low power Qualcomm Hexagon devices, such as the
QDSP6V5A), etc. These instructions can be implemented as software,
firmware, etc. These instructions can also be implemented in
various forms of processor circuitry, including programmable logic
devices, field programmable gate arrays (e.g., the Xilinx Virtex
series devices), field programmable object arrays, and application
specific circuits--including digital, analog and mixed
analog/digital circuitry. Execution of the instructions can be
distributed among processors and/or made parallel across processors
within a device or across a network of devices. Processing of data
can also be distributed among different processor and memory
devices. Cloud computing resources can be used as well. References
to "processors," "modules" or "components" should be understood to
refer to functionality, rather than requiring a particular form of
implementation.
[0351] Implementation can additionally, or alternatively, employ
special purpose electronic circuitry that has been custom-designed
and manufactured to perform some or all of the component acts, as
an application specific integrated circuit (ASIC).
[0352] Software instructions for implementing the detailed
functionality can be authored by artisans without undue
experimentation from the descriptions provided herein, e.g.,
written in C, C++, Visual Basic, Java, Python, Tcl, Perl, Scheme,
Ruby, etc., in conjunction with associated data.
[0353] Software and hardware configuration data/instructions are
commonly stored as instructions in one or more data structures
conveyed by tangible media, such as magnetic or optical discs,
memory cards, ROM, etc., which may be accessed across a network.
Some embodiments may be implemented as embedded systems--special
purpose computer systems in which operating system software and
application software are indistinguishable to the user (e.g., as is
commonly the case in basic cell phones). The functionality detailed
in this specification can be implemented in operating system
software, application software and/or as embedded system
software.
[0354] Different of the functionality can be implemented on
different devices. Different tasks can be performed exclusively by
one device or another, or execution can be distributed between
devices. In like fashion, description of data being stored on a
particular device is also exemplary; data can be stored anywhere:
local device, remote device, in the cloud, distributed, etc.
[0355] Other recycling arrangements are taught in patent documents
U.S. Pat. Nos. 4,644,151 5,965,858, 6,390,368, US20060070928,
US20140305851, US20140365381, US20170225199, US20180056336,
US20180065155, US20180349864, and US20190030571. Alternate
embodiments of the present technology employ features and
arrangements from these cited documents.
[0356] This specification has discussed various embodiments. It
should be understood that the methods, elements and concepts
detailed in connection with one embodiment can be combined with the
methods, elements and concepts detailed in connection with other
embodiments. While some such arrangements have been particularly
described, many have not--due to the number of permutations and
combinations. Applicant similarly recognizes and intends that the
methods, elements and concepts of this specification can be
combined, substituted and interchanged--not just among and between
themselves, but also with those known from the cited prior art.
Moreover, it will be recognized that the detailed technology can be
included with other technologies--current and upcoming--to
advantageous effect. Implementation of such combinations is
straightforward to the artisan from the teachings provided in this
disclosure.
[0357] While this disclosure has detailed particular ordering of
acts and particular combinations of elements, it will be recognized
that other contemplated methods may re-order acts (possibly
omitting some and adding others), and other contemplated
combinations may omit some elements and add others, etc.
[0358] Although disclosed as complete systems, sub-combinations of
the detailed arrangements are also separately contemplated (e.g.,
omitting various of the features of a complete system).
[0359] While certain aspects of the technology have been described
by reference to illustrative methods, it will be recognized that
apparatuses configured to perform the acts of such methods are also
contemplated as part of applicant's inventive work. Likewise, other
aspects have been described by reference to illustrative apparatus,
and the methodology performed by such apparatus is likewise within
the scope of the present technology. Still further, tangible
computer readable media containing instructions for configuring a
processor or other programmable system to perform such methods is
also expressly contemplated.
[0360] To provide a comprehensive disclosure, while complying with
the Patent Act's requirement of conciseness, applicant
incorporates-by-reference each of the documents referenced herein.
(Such materials are incorporated in their entireties, even if cited
above in connection with specific of their teachings.) These
references disclose technologies and teachings that applicant
intends be incorporated into the arrangements detailed herein, and
into which the technologies and teachings presently-detailed be
incorporated.
[0361] In view of the wide variety of embodiments to which the
principles and features discussed above can be applied, it should
be apparent that the detailed embodiments are illustrative only,
and should not be taken as limiting the scope of the invention.
* * * * *