U.S. patent application number 15/993883 was filed with the patent office on 2018-12-27 for system and assessment of reflective objects along a roadway.
The applicant listed for this patent is MANDLI COMMUNICATIONS, INC.. Invention is credited to ROBERT A. LAUMEYER, James E. Retterath.
Application Number | 20180372621 15/993883 |
Document ID | / |
Family ID | 27397401 |
Filed Date | 2018-12-27 |
View All Diagrams
United States Patent
Application |
20180372621 |
Kind Code |
A1 |
Retterath; James E. ; et
al. |
December 27, 2018 |
SYSTEM AND ASSESSMENT OF REFLECTIVE OBJECTS ALONG A ROADWAY
Abstract
A system for classifying different types of sheeting materials
of road signs depicted in a videostream compares estimated
retroreflectivity values against known minimum retroreflectivity
values for each of a plurality of colors. Once a road sign has been
identified in the videostream, the frames associated with that road
sign are analyzed to determine each of a plurality of colors
present on the road sign. An estimated retroreflectivity for each
of the plurality of colors present on the road sign is then
determined. By comparing the estimated retroreflectivity for each
of the plurality of colors against known minimum retroreflectivity
values for the corresponding color for different types of sheeting
materials, an accurate determination of the classification of the
sheeting material of the road sign is established. Preferably,
certain conditions of gross failure of the sheeting material are
filtered out before classification of the sheeting material is
determined.
Inventors: |
Retterath; James E.;
(Excelsior, MN) ; LAUMEYER; ROBERT A.; (Eden
Prairie, MN) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
MANDLI COMMUNICATIONS, INC. |
Fitchburg |
WI |
US |
|
|
Family ID: |
27397401 |
Appl. No.: |
15/993883 |
Filed: |
May 31, 2018 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
15595594 |
May 15, 2017 |
9989457 |
|
|
15993883 |
|
|
|
|
15148722 |
May 6, 2016 |
9671328 |
|
|
15595594 |
|
|
|
|
14512735 |
Oct 13, 2014 |
9335255 |
|
|
15148722 |
|
|
|
|
14025614 |
Sep 12, 2013 |
8860944 |
|
|
14512735 |
|
|
|
|
13205337 |
Aug 8, 2011 |
8660311 |
|
|
14025614 |
|
|
|
|
12419843 |
Apr 7, 2009 |
7995796 |
|
|
13205337 |
|
|
|
|
11056926 |
Feb 11, 2005 |
7515736 |
|
|
12419843 |
|
|
|
|
09928218 |
Aug 10, 2001 |
6891960 |
|
|
11056926 |
|
|
|
|
60224761 |
Aug 12, 2000 |
|
|
|
60296596 |
Jun 7, 2001 |
|
|
|
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
G06K 9/00818 20130101;
G08G 1/096758 20130101; G01N 2201/06113 20130101; G06K 9/2036
20130101; G06K 9/4652 20130101; G01N 2201/102 20130101; G01N 21/55
20130101; G01S 17/06 20130101; G08G 1/096783 20130101; G06K 9/00791
20130101; G01N 21/84 20130101; G01N 21/251 20130101; G06K 9/4661
20130101; G01N 2201/12 20130101 |
International
Class: |
G01N 21/25 20060101
G01N021/25; G06K 9/00 20060101 G06K009/00; G01N 21/55 20060101
G01N021/55; G06K 9/20 20060101 G06K009/20; G06K 9/46 20060101
G06K009/46; G08G 1/0967 20060101 G08G001/0967; G01S 17/06 20060101
G01S017/06; G01N 21/84 20060101 G01N021/84 |
Claims
1. A method for determining sheeting type of a reflective sheeting
material covering at least a portion of a surface of a road sign
comprising: using a light source to illuminate at least a portion
of the reflective sheeting material covering the road sign as the
light source is traversed along the roadway; collecting a plurality
of light intensity measurements reflected from the at least a
portion of the reflective sheeting material to generate an
as-measured retroreflectivity profile, the light intensity
measurements corresponding to exemplary angular displacements of
incident light from a reference position; and determining sheeting
type of the reflective sheeting material based on the as-measured
retroreflectivity profile.
2. The method of claim 1, wherein determining sheeting type of the
reflective sheeting material further comprises: providing a
database of retroreflectivity profiles, each retroreflectivity
profile having an associated known sheeting type; selecting a
retroreflectivity profile from the retroreflectivity profile pool
that best correlates to the as-measured retroreflectivity profile;
and determining sheeting type of the retroreflective sheeting
material based on the retroreflectivity profile associated with the
selected retroreflectivity profile.
3. The method of claim 2, wherein the known sheeting type is
selected from the group consisting of Type I (Engineer Grade), Type
III (High Intensity), and Type VII (Prismatic).
4. The method of claim 3, wherein the Prismatic is one of Type VIIa
(Diamond Grade VIP) and Type VIIb (Diamond Grade LDP).
5. The method of claim 2, wherein the providing step comprises
establishing a functional relationship between a plurality of
exemplary instances of retroreflectivity values corresponding to
each known sheeting type and a plurality of instances of one of an
observation angle and an entrance angle.
6. The method of claim 2, wherein the retroreflectivity profile is
a characteristic curve of retroreflectivity values represented as a
function of one of a monotonically varying observation angle and an
entrance angle.
7. The method of claim 2, wherein the as-measured retroreflectivity
profile is a curve of retroreflectivity values calculated from the
plurality of light intensity measurements plotted as a function of
the exemplary angular displacements of incident light from a
reference position.
8. The method of claim 7, wherein each of the exemplary angular
displacements of incident light comprises an observation angle
corresponding to one of a plurality of distances of the light
source from the road sign.
9. The method of claim 7, wherein each of the exemplary angular
displacements comprises an entrance angle corresponding to one of a
plurality of distances of the road sign from the road sign.
10. The method of claim 7, wherein each of the exemplary angular
displacements comprises an observation angle corresponding to one
of a plurality of distances of the light source from the road
sign.
11. The method of claim 2, wherein the selecting step comprises:
providing a computer model coding a function for obtaining a set of
coordinates from each retroreflectivity profile, each coordinate
comprising a pairing of a retroreflectivity value and an associated
angular displacement, wherein the function associates the
retroreflectivity value to the angular displacement; applying the
computer model to the retroreflectivity profiles in the
retroreflectivity pool to generate a set of reference coordinates
for each retroreflectivity profile, each set of reference
coordinates defining a characteristic curve having a characteristic
shape; applying the computer model to the as-measured
retroreflectivity profile to obtain a set of as-measured
coordinates defining an as-measured performance curve having an
as-measured shape; and selecting the characteristic curve having
the characteristic shape which best approximates the as-measured
shape.
12. The method of claim 11, wherein the selecting step comprises:
computing parameters to provide a best-fit curve to the set of
as-measured coordinates; scaling the characteristic curves to
obtain a scaled characteristic curve with a characteristic shape
such that the scaled characteristic curve is sized substantially
the same as the as-measured performance curve; and selecting the
characteristic curve as acceptable if a deviation of the
characteristic shape from the as-measured shape is within a
predefined threshold
13. The method of claim 12, wherein scaling the characteristic
curves comprises multiplying the set of reference coordinates for
each retroreflectivity profile by a positive scaling factor less
than unity to obtain a set of scaled reference coordinates defining
a scaled characteristic curve having a characteristic shape.
14. The method of claim 2, wherein the determining step comprises
the steps of: calculating a degree of correlation between the
as-measured retroreflectivity profile and the selected
retroreflectivity profile; selecting the sheeting type associated
with the selected retroreflectivity profile if the degree of
correlation lies between an acceptable range of predefined values;
and identifying the road sign as exhibiting a gross failure of the
reflective sheeting material covering at least a portion of the
surface of the road sign if the degree of correlation lies outside
the acceptable range of predefined values.
15. A method for determining sheeting type of a reflective sheeting
material covering a road sign comprising: means for illuminating at
least a portion of the reflective sheeting material covering the
road sign from a plurality of distances along the roadway; means
for collecting a plurality of light intensity measurements
reflected from the at least a portion of the reflective sheeting
material; means for generating an as-measured retroreflectivity
profile using the light intensity measurements corresponding to
exemplary angular displacements of incident light from a reference
position; means for providing a database of retroreflectivity
profiles with each retroreflectivity profile having an associated
known sheeting type; means for comparing the as-measured profile to
each retroreflectivity profile from the retroreflectivity profile
pool; means for selecting the retroreflectivity profile that best
correlates to the as-measured retroreflectivity profile; and means
for determining sheeting type of the reflective sheeting material
based on the known sheeting type associated with the selected
retroreflectivity profile.
16. The method of claim 15, wherein the means for determining
sheeting type further comprises: means for calculating a degree of
correlation between the as-measured retroreflectivity profile and
the selected retroreflectivity profile; means for determining if
the degree of correlation lies between an acceptable range of
predefined values; means for selecting the sheeting type associated
with the selected retroreflectivity profile if the degree of
correlation lies between an acceptable range of predefined values;
and means for classifying the road sign as exhibiting a gross
failure of the reflective sheeting material covering at least a
portion of the surface of the road sign if the degree of
correlation lies outside the acceptable range of predefined
values.
17. The method of claim 15, wherein the means for determining
sheeting type further comprises: means for computing parameters to
provide a best-fit curve to the as-measured retroreflectivity
profile, the best-fit curve having an as-measured shape; means for
scaling each retroreflectivity profile in the retroreflectivity
pool to obtain a scaled characteristic curve sized substantially
the same as the best-fit curve and having a characteristic shape;
and means for computing a deviation of the characteristic shape
from the as-measured shape; means for determining the sheeting type
by identifying the sheeting type associated with the characteristic
curve if the deviation of the characteristic shape from the
as-measured shape is within a predefined threshold; and means for
filtering the road sign as exhibiting a gross failure of the
reflective sheeting material covering at least a portion of the
surface of the road sign if the deviation lies outside the
predefined threshold.
18. A system determining sheeting type of a reflective sheeting
material covering a road sign comprising: at least one light source
mounted on a vehicle capable of traversing a roadway and suitable
for illuminating at least a portion of the road sign; at least one
light intensity sensor for receiving light reflected from the road
sign and computing the intensity of the received light; a
positioning system for determining a location of the vehicle from a
stationary frame of reference; a ranging system for determining a
distance of the road sign from the light source and an inclination
of a surface of the road sign relative to the light source; a
control system operably connected to the light source, intensity
sensor, positioning system and ranging system such that the
intensity sensor, ranging system and positioning system record
information associated with an area that includes at least one road
sign as the vehicle traverses along the roadway in response to
repeated illumination of the area by the light source; and a
computer processing system that utilizes the recorded information
to determine an as-measured retroreflectivity profile having an
as-measured shape and comprising the plurality of as-measured
retroreflectivity values for the sheeting type of the road sign and
comparing the as-measured shape with a characteristic shape of at
least one known retroreflectivity profile in a database of known
retroreflectivity profiles for each of a plurality of different
sheeting types to classify a sheeting material of the road sign as
one of the plurality of different sheeting types.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] The present application is a continuation of U.S.
application Ser. No. 15/595,594, filed May 15, 2017, now U.S. Pat.
No. 9,989,457, which is a continuation of U.S. application Ser. No.
15/148,722, filed May 6, 2016, now U.S. Pat. No. 9,671,328, which
is a continuation of U.S. application Ser. No. 14/512,735, filed
Oct. 13, 2014, now U.S. Pat. No. 9,335,255, which is a continuation
of U.S. application Ser. No. 14/025,614, filed Sep. 12, 3013, now
U.S. Pat. No. 8,860,944, which is a divisional of U.S. application
Ser. No. 13/205,337, filed Aug. 8, 2011, now U.S. Pat. No.
8,660,311, which is a divisional of U.S. application Ser. No.
12/419,843, filed Apr. 7, 2009, now U.S. Pat. No. 7,995,796, which
is a continuation of application Ser. No. 11/056,926, filed Feb.
11, 2005, now U.S. Pat. No. 7,515,736, which is a continuation of
application Ser. No. 09/928,218, filed Aug. 10, 2001, now U.S. Pat.
No. 6,891,960, which claims the benefit of the contents and filing
date accorded to two U.S. Provisional patent applications, the
first of which was filed on Aug. 12, 2000 as Application No.
60/224,761, and the second of which was filed on Jun. 7, 2001 as
Application No. 60/296,596. The application is related to U.S. Pat.
No. 6,674,878, issued Jan. 6, 2004, and U.S. Pat. No. 6,453,056,
issued Sep. 17, 2002, which is a continuation of U.S. Pat. No.
6,266,442, filed Oct. 23, 1998, all of which are hereby
incorporated by reference.
FIELD OF THE INVENTION
[0002] The present invention relates generally to the field of
automated object recognition. More specifically, the present
invention relates to a system for classifying different types of
sheeting materials of road signs depicted in a videostream.
BACKGROUND OF THE INVENTION
[0003] The goal of using an automated image identification system
to recognize road signs and traffic signs is well known. Various
techniques have been proposed for the recognition of road signs as
part of a real-time automated vehicle navigation system. Due to the
processing limitations imposed by a real-time environment, almost
all of these techniques have involved template matching of shape
and color. Given the wide variations in lighting and conditions,
few if any of these systems provide accurate results.
[0004] Another use of automated road sign recognition is for the
purpose of identifying and creating an accurate inventory of all
road signs and traffic signs along a given street or highway. In
one system as described in U.S. Pat. No. 6,266,442, entitled,
"METHOD AND APPARATUS FOR IDENTIFYING OBJECTS DEPICTED IN A
VIDEOSTREAM," issued Jul. 24, 2001 to Laumeyer et al., an
acquisition vehicle equipped with video cameras and position
identifying technology, such as global positioning satellite (GPS)
receivers, is systematically driven over the roads and streets in a
given area to produce a videostream tagged with location
information. The tagged videostream is analyzed by computer
software routines to perform object recognition of the desired
objects, the road signs in this case. The results of this analysis
are exported to an asset management database that stores attributes
of the road signs.
[0005] Road signs are manufactured from a sheeting material made up
of multiple layered films (one or more colored layers that are
fused with a layer that produces the reflectivity) that is adhered
to the sign face. There are different types of sheeting material
utilized in the road sign industry. Currently, specific attributes
about each road sign like retroreflectivity (measured in
candelas/lux/sq. meter) and sheeting type must be gathered manually
by sending personnel in the field to measure retroreflectivity with
a handheld device (like the Impulse RM retro-reflectometer from
Laser Technology, Inc.) and to visually determine the sheeting type
of each sign. Measurements of retroreflectivity and identification
of sheeting type are helpful in evaluating the visibility of a sign
and whether it has deteriorated due to a breakdown in the pigments
or reflective material in the sheeting material of the sign. The
retroreflectivity and sheeting type can also be used to produce a
predictive model of how the sign will perform into the future based
on the as-measured characteristics.
[0006] Generally, highway and street maintenance departments do not
systematically evaluate the deterioration of the reflective
materials used on road signs and markers. If inspections of road
signs or markers are performed, they are typically accomplished by
having inspectors manually position a handheld retroreflectometer
directly on the surface of a sign in order to determine a
retroreflectivity value for that sign. When there are a large
number of road signs or markers (sometimes referred to as traffic
control devices or TCDs) in a given jurisdiction, the task of
manually inspecting all of these road signs and markers can be time
consuming and expensive.
[0007] One technique for determining retroreflectivity, designated
as "R.sub.A" generally (and from time to time in this disclosure),
which does not require that a retroreflectometer be placed directly
on a sign is described in U.S. Pat. No. 6,212,480 entitled,
"APPARATUS AND METHOD FOR DETERMINING PRECISION REFLECTIVITY OF
HIGHWAY SIGNS AND OTHER REFLECTIVE OBJECTS UTILIZING AN OPTICAL
RANGE FINDER INSTRUMENT" issued Apr. 3, 2001 to Dunne. The Dunne
patent relates to a device commercialized by the assignee thereof
and marketed as the "Impulse RM" retro-reflectometer by Laser
Technology, Inc., of Englewood, Colo., USA. In use, handheld
devices fabricated according to the Dunne patent are manually
directed toward, or precisely at, a target object and then manually
"fired." Once fired, the handheld device bounces a laser off the
target object and measures the reflected laser energy that is then
used to determine a retroreflectivity.
[0008] There are several drawbacks of the handheld laser
arrangement described by the Dunne patent. The handheld device can
only measure a single color at a time and can only measure one
object at a time. The determination of retroreflectivity for a
given object is valid only for the actual location, or discrete
measurement point, along the roadway at which the measurement was
made by the human operator. In order to validate a measurement made
by such devices, the device must be taken back to the precise
location in the field where an original measurement occurred for a
valid comparison measurement to be made.
[0009] Another technique established for determining the
retroreflectivity of signs has been introduced by the Federal
Highway Administration (FHWA). The Sign Management and
Retroreflectivity Tracking System (SMARTS) is a vehicle that
contains one high intensity flash source (similar to the Honeywell
StrobeGuard.TM. SG-60 device), one color camera, two black and
white cameras, and a range-sensing device. The SMARTS vehicle
requires two people for proper operation--one driver and one system
operator to point the device at the target sign and arm the system.
The SMARTS travels down the road, and the system operator "locks
on" to a sign up ahead by rotating the camera and light assembly to
point at the sign. At a distance of 60 meters, the system triggers
the flash source to illuminate the sign surface, an image of which
is captured by one of the black and white cameras. A histogram is
produced of the sign's legend and background that is then used to
calculate retroreflectivity. A GPS system stores the location of
the vehicle along with the calculated retroreflectivity in a
computer database.
[0010] Like the handheld laser device of the Dunne patent, the
SMARTS device can only determine retroreflectivity for one sign at
a time and can only determine retroreflectivity for the discrete
point on the roadway 60 meters from the sign. Two people are
required to operate the vehicle and measurement system. The SMARTS
vehicle cannot make retroreflectivity determinations for signs on
both sides of the roadway in a single pass over the roadway, and
does not produce nighttime sign visibility information for lanes on
the roadway not traveled by the vehicle. Because the system
operator in the SMARTS vehicle must locate and track signs to be
measured while the vehicle is in motion, a high level of
operational skill is required and the likelihood that a sign will
be missed is significant. Most importantly for purposes of the
present invention, the SMARTS device makes no attempt to
automatically determine sheeting type of a sign.
[0011] There are an estimated 58 million individual TCDs that must
be monitored and maintained in the United States and new TCD
installations increase this number daily. For the reasons that have
been described, the existing techniques for determining
retroreflectivity do not lend themselves to increasing processing
throughput so as to more easily manage the monitoring and
maintenance of these TCDs. So called automated data collection
systems often require that normal traffic be stopped during data
collection because either the acquisition vehicle moved very slowly
or because the acquisition vehicle had to come to a full stop
before recording data about the roadside scene. Furthermore, a
human operator is required to point one or more measurement devices
at a sign of interest, perform data collection for that particular
sign and then set up the device for another particular sign of
interest. With such a large number of TCDs that must be monitored,
it would be desirable to provide an automated system for
determining the retroreflectivity of road signs and markers that
addresses these and other shortcomings of the existing techniques
to enable a higher processing throughput of an automated
determination of the retroreflectivity and sheeting classification
of road signs and markers.
SUMMARY OF THE INVENTION
[0012] The present invention is a system for classifying different
types of sheeting materials of road signs depicted in a
videostream. Once a road sign has been identified in the
videostream, the frames associated with that road sign are analyzed
to determine each of a plurality of colors present on the road
sign. An estimated retroreflectivity for each of the plurality of
colors present on the road sign is then determined. By comparing
the estimated retroreflectivity for each of the plurality of colors
against known minimum retroreflectivity values for the
corresponding color for different types of sheeting materials, an
accurate determination of the classification of the sheeting
material of the road sign is established. Preferably, certain
conditions of gross failure of the sheeting material are filtered
out before classification of the sheeting material is
determined.
[0013] In a preferred embodiment, a system for the automated
determination of retroreflectivity values for reflective surfaces
disposed along a roadway is utilized to establish the
retroreflectivity values for both foreground and background colors
of a road sign for the purpose of classifying the sheeting material
of that sign. An area along the roadway that includes at least one
reflective surface is repeatedly illuminated by a light source and
multiple light intensity values are measured over a field of view
which includes at least a portion of the area illuminated by the
light source. A computer processing system is used to identify a
portion of the light intensity values associated with a reflective
surface and analyze the portion of the light intensity values to
determine at least one retroreflectivity value for that reflective
surface. Color images of the area and locational information are
also generated by the system and are used together with a
characterization profile of the light source to enhance the
accuracy of the determination of retroreflectivity values.
[0014] In contrast to the existing techniques for determining
retroreflectivity that require an operator to target individual
signs from a known distance, a preferred embodiment of the present
invention can determine retroreflectivity without targeting
individual signs and can automatically determine sheeting
classification as a result. To overcome the limitations imposed by
the existing techniques, the preferred embodiment employs several
enhancements that are designed to improve the accuracy of
evaluating intensity measurements made over a view where the
reflective surfaces are not individually targeted and therefore
neither the distance to the reflective surface or the normal vector
to the reflective surface are known.
BRIEF DESCRIPTION OF THE DRAWINGS
[0015] FIG. 1 is a capture vehicle used in a preferred embodiment
of the present invention.
[0016] FIG. 2 is a data flow diagram of a preferred embodiment of
the present invention.
[0017] FIG. 3 depicts a block diagram of the systems, subsystems
and processes for capturing and processing roadside information
from a moving platform in order to compute retroreflectivity
according to the teaching of the present invention, wherein the
arrows connecting the blocks of the diagram illustrate the
connections between the systems and subsystems for computing
R.sub.A for each reflective asset recorded by the system of the
present invention.
[0018] FIG. 4 depicts in diagram form, a preferred configuration of
a sensor suite for use with a four-wheeled vehicle and the
interconnections and couplings between the physical subcomponents
of a system designed according to the present invention.
[0019] FIG. 5 is a plan view of a divided multi-lane vehicle
pathway and depicts how periodic light intensity measurements may
be made as a vehicle traverses the vehicle pathway over time and
the discrete locations where such periodic light intensity
measurements are performed by a data acquisition vehicle operating
in accordance with the present invention.
[0020] FIG. 6 depicts four digital frames of data as captured by
the intensity sensor at various discrete locations along the
vehicle pathway depicted in FIG. 5.
[0021] FIG. 7 depicts a flowchart showing the steps required to
convert intensity measurements into foreground and background
retroreflectivity for a single reflective asset.
[0022] FIG. 8 depicts a typical light source intensity profile over
the visible electromagnetic spectrum, which illustrates how
different wavelengths of electromagnetic radiation possess
different light intensities.
[0023] FIG. 9 depicts a preferred methodology for creating a
retroreflectivity profile for all lanes and locations adjacent a
vehicle pathway for a single reflective asset or sign which
retroreflectivity profile is based upon a single pass of a data
acquisition vehicle over the vehicle pathway.
[0024] FIG. 10 illustrates the facts that the normal vector of a
reflective asset and the sheeting type of such a reflective asset
create symmetry that may be used to determine retroreflectivity
values along all rays (or vectors) that have the same relative
angle to the normal vector of the reflective asset.
[0025] FIG. 11 depicts the concept of observation angle (i.e.,
angle between incident light from a light source and an human
observer (or light sensor), of the light as reflected from the face
of a reflective asset) in the context of a conventional passenger
vehicle traversing a vehicle pathway and where light from the
vehicle reflects from a stop sign to the vehicle operator (shown in
ghost).
[0026] FIG. 12 depicts the concept of entrance angle (i.e., angle
between incident light from a light source mounted to a vehicle and
a normal vector relative to the substantially flat face of a
reflective surface disposed adjacent a vehicle pathway).
[0027] FIG. 13 is a graphic representation of the retroreflectivity
performance for white sheeting for the three main types of sign
sheeting.
[0028] FIGS. 14A, 14A-1, 14A-2, 14B, 14B-1, and 14B-2 depict a
flowchart for a preferred embodiment of the threshold algorithm
used to determine sign sheeting type.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0029] Referring to FIG. 1, an acquisition vehicle 10 is equipped
with multiple video cameras 12 that generate a series of raw
videostreams 14 representative of a road or street over which the
acquisition vehicle 10 is traveling. A global positioning satellite
(GPS) receiver 16 supplies location information that is combined
with the raw videostream 14 by a processor 18 to generate a tagged
videostream 20.
[0030] In one embodiment as shown in FIG. 2, the tagged videostream
20 is analyzed by a computer processor 22 to identify each road
sign 30 and generate an asset management database 24 containing
attribute and location information associated with each road sign
30. In an alternate embodiment, the identification of road signs 30
and generation of asset management database 24 is accomplished by
the same processor 18 that tags the raw videostream 14. The details
of this process are set out in U.S. Pat. No. 6,453,056, issued Sep.
17, 2002 to Laumeyer et al., which is hereby incorporated by
reference.
[0031] Either concurrent with or subsequent to the identification
of road signs 30 and generation of asset management database 24,
the computer processor 22 evaluates that portion 32 of each video
frame or image that depicts a road sign 30 to determine a set of
color values 40, 42 for each of a plurality of colors on the road
sign 32. A retroreflectivity value is generated for each color
portion 34, 36 on each frame of the tagged videostream 20
containing the road sign 30 that represents a different color value
40, 42. Preferably, the values for retroreflectivity are determined
by measuring the intensity of the signal for each color portion 34,
36. These values are then analyzed over the number of frames
containing each color portion 34, 36 to arrive at the maximum
retroreflectivity value 44, 46 that corresponds to the color value
40, 42 for each color portion 34, 36 of the road sign 30. These
maximum retroreflectivity values 44, 46 will be the reference
values used in the threshold algorithm from which sign sheeting
classification will be determined.
[0032] There are three main types of sheeting recognized in the
industry: 1) Type I, commonly called Engineer Grade; 2) Type III,
commonly called High Intensity; and 3) Type VII, commonly called
Prismatic (Prismatic is sometimes divided into two groups--Type
VIIa called Diamond Grade VIP, and Type VIIb called Diamond Grade
LDP). In order to remove the manual element of determining sheeting
type and measuring retroreflectivity, the automated system of the
present invention must accurately distinguish between these
sheeting types. To accomplish this, the present invention utilizes
the fact that all signs use the same sheeting type for foreground
and background colors, that each road sign will have at least two
colors and that the reflectivity for each color for each type of
sheeting material has a relatively unique minimum initial
retroreflectivity value. Most signs also have either White, Yellow
or Orange colors as one of the colors on the sign. According to 3M,
a manufacturer of reflective sheeting material, each color of the
three sheeting types has a minimum initial retroreflectivity value.
The following table lists the minimum values for common colors of
each type:
TABLE-US-00001 Type I Type III Type VIIa Type VIIb Color
Min.R.sub.A Min.R.sub.A Min.R.sub.A Min.R.sub.A White 70 250 430
800 Yellow 50 170 350 660 Orange 25 100 200 360 Red 14.5 45 110 215
Green 9 45 45 80 Blue 4 20 20 43
[0033] This information is stored in a reflectivity/color database
50. The computer processor 22 accesses the database 50 using the
maximum reflectivity value 44, 46 that corresponds to the color
value 40, 42 for each color portion 34, 36 to determine the lowest
possible sheeting type for each color. If the sheeting type is
classified the same for all of the color portions 34, 36 for a
given road sign 30, then the sheeting class 52 is established as
that type and this information is preferably stored in the asset
management database 24 along with other attributes of the given
road sign 30. If there is a discrepancy in the classification of
sheeting material between different colors, a subsequent analysis
by the processor 22 using, for example, a neural network program,
to incorporate other determining factors, such as time of day,
shadow, direction that could affect the retroreflectivity of the
lighter colors (white, yellow and orange) more than the
retroreflectivity of the darker colors (red, green, blue). In
general, retroreflectivity values for lighter colors are weighted
more heavily in resolving any discrepancies in classification of
sheeting material.
[0034] The system described herein acquires many data points along
the desired roadway without specifically targeting any objects of
interest. For a roadside sign, the specific geometry of the sign
(its orientation with respect to the roadway) is not necessarily
known, nor is it required. The retroreflectivity points determined
along the roadway are generated for the "as placed" road sign. Road
signs will display their best retroreflectivity performance (have
the highest retroreflectivity values) at or near the normal vector
for the sign face. Since the geometry of the as-measured sign is
not known, the system chooses the highest retroreflectivity value
for that sign as the value that will be used in the threshold
algorithm for sign sheeting classification.
[0035] There are several factors that can cause retroreflectivity
readings for signs to actually be lower than the values for the
underlying sheeting type. For example, a sign that has a layer of
dirt on the face will produce lower retroreflectivity numbers than
usual. If these lower numbers are used in the threshold comparison
algorithm, an incorrect sheeting type may result. These systematic
errors can be removed by analyzing the retroreflectivity profile
for the sign.
[0036] Sign sheeting types vary by the intensity of light that is
reflected, but they also have reflective characteristics that give
them unique signatures. FIG. 13 shows the profiles for Type I, Type
III, and Type VII white sign sheeting types. Note the unique
profiles for the three sheeting types. The preferred embodiment of
the present invention, due to its ability to utilize multiple
retroreflectivity points, can determine sheeting type by matching
the as-measured profile with a uniformly reduced (where the entire
curve is multiplied by a scaling factor that is less than one)
characteristic curve that best correlates to the as-measured
profile. This correlation step will establish the "best fit"
performance curve based on the shape of the curve, without
consideration for the magnitude of the curve. This "uniform
reduction" of the sign sheeting performance curve allows the proper
sheeting type to be determined, thus overcoming the problem with
signs that have some type of surface anomaly. The performance curve
correlation described above can be used in one of two ways--either
as a validation of the proper determination of sheeting type from
the threshold algorithm or as one of the qualification criteria
prior to performing the threshold algorithm.
[0037] The sheeting types are all manufactured with multiple
layers. In order for the present invention to accurately compute
retroreflectivity for purposes of determining sheeting type of a
given road sign, it is also necessary for the system to recognize
gross sheeting failures like extreme color fade, de-lamination and
vapor fade (darkening of the daylight appearance of the sheeting
due to the corrosion of the metal sign backing). These gross
failures will impact R.sub.A measurements of the sheeting.
Preferably, the sheeting determination system described herein
tests for the absence of these gross failures prior to making any
R.sub.A measurements as part of the process of categorizing
sheeting type.
[0038] Retroreflectivity, designated as "R.sub.A" generally (and
from time to time in this disclosure), varies according to two key
parameters, observation angle and entrance angle. Observation angle
100 (See FIG. 11) is the angular displacement between a light
source 110 and a light sensor 120, as measured from an object face
surface 130. In the case of a vehicle 140 driven by vehicle
operator 145 moving along a highway 150, observation angle 100 is
defined by the distance of the vehicle 140 from a sign face surface
130, the placement of the light source (headlights) 110 on the
vehicle 140, and the position of the light sensor (eyes of the
vehicle operator) 120 of the vehicle 140.
[0039] Entrance angle 160 (See FIG. 12) is defined as the angular
displacement of the incident light 170 relative to the normal
vector 180 from the object face surface 130. Entrance angles are
impacted by the angular position 200 of a sign 190 relative to the
highway 150, the sign 190 lateral distance 210 from the highway
150, and the distance 220 of the vehicle 140 from the sign 190. The
inventors hereof are believed to be the first persons to
successfully decrease the complexity and increase the efficiency of
determination of R.sub.A in the field.
[0040] The method of automated determination of R.sub.A (See FIGS.
3 and 4) preferably utilizes a plurality of subsystems located
on/in a capture vehicle 225. These subsystems include a light
intensity measurement system 230, a vehicle positioning system 240,
a color image capture system 250 and a data recording system 260.
The light intensity measurement system 230 preferably includes a
high output light source 270, a light intensity sensor 280 and an
intensity measurement system control 290. A plurality of intensity
measurements 300 are generated by the intensity measurement system
230 in response to the repeated strobing of the high output light
source 270. The vehicle positioning system 240 preferably includes
a GPS receiver 310, an inertial navigation system 320, a distance
measuring instrument 330 and a master clock 340. A position
measurement 350 is generated by the vehicle positioning system 240.
The color image capture system 250 preferably includes a
stereoscopic camera pair 360, iris controls 370 and image capture
control 380. The image capture system 250 generates a digital
imagery stream 390.
[0041] The data required for the automated determination of R.sub.A
is accumulated while traversing a highway 150 with the capture
vehicle 225 (See FIGS. 5 and 6). The capture vehicle 225 is shown
on a 4-lane divided highway 400 with the capture vehicle 225
located in a proximate lane 410 to the stop sign 190. Preferably, a
series of reflected light intensity frames 420 are generated at a
constant measurement interval 430 as the capture vehicle travels
along the highway 150.
[0042] Characterization of sign 190 R.sub.A preferably utilizes the
data recording system 260 to create a single tagged videostream 440
from the reflected light intensity frames 420, position
measurements 350 and digital imagery 390 for each capture event 430
(See FIGS. 3, 5, 6, 7, and 8). A computer processor 450 identifies
an object of interest 460 in a portion of the intensity frame 420
and determines the object of interest attributes 465 associated
with that object of interest. Preferably, the objects of interest
are identified from the digital imagery stream 390 generated by the
color image capture system 250 in the manner as taught by U.S. Pat.
No. 6,266,442. Alternatively, other techniques known in the art for
isolating an object of interest in a videostream can be used.
Preferably, the computer processor 450 correlates the portion of an
image frame of the digital imagery stream 290 with a similar
portion of the intensity frame 420 containing the object of
interest 460.
[0043] For each object of interest 460, a background intensity
measurement 470 and a foreground intensity measurement 480 is
generated. Using an intensity algorithm 490, a light intensity
sensor characterization 275 and a look-up-table 475, the computer
processor 450 determines a background luminance value 500 and a
foreground luminance value 510. Based on the background luminance
value 500, the foreground luminance value 510, a characterization
of light source wavelength 540, the background sheeting color 505
and the foreground sheeting color 506 the computer processor 450
characterizes a background R.sub.A 520 and a foreground R.sub.A530
which are preferably reported separately for that object of
interest.
[0044] The automated determination of multiple R.sub.A values for a
given object of interest 460 allows for the extrapolation of
R.sub.A values at an unmeasured viewing point 550 for an object of
interest, such as a sign 190 (See FIGS. 9 and 10). In this example,
the unmeasured viewing point resides in a nontraversed lane 560.
The computer processor 450 defines an undetermined
retroreflectivity ray 570 for unmeasured viewing point 550. Using
interpolated values, the computer processor 450 determines an
R.sub.A value for unmeasured viewing point 550 and any point
located along undetermined retroreflectivity ray 570.
[0045] Pursuant to the teaching of the present invention, a method
and apparatus for determining retroreflectivity of relatively flat
surface portions of objects disposed adjacent a highway 150
traversed by a vehicle 140 are taught, enabled, and depicted. The
present invention may be utilized to detect and determine a
retroreflective surface of interest disposed in a scene of
non-retroreflective surfaces. That is, at least one object face
surface 130 which exhibits retroreflectivity over at least a
relatively narrow conical volume of magnitude of several degrees
from a normal vector 180 originating from said object face surface
130.
[0046] In accordance with the present invention, a determination of
the retroreflectivity of objects adjacent a highway 150 preferably
includes providing: (i) position measurements 350 of a capture
vehicle 225; (ii) precise position of the object of interest 460,
or sign 190; (iii) intensity measurements 300 from a high output
light source 270 and light intensity sensor 280 at measurement
intervals 430 along said highway 150. Thus, a single-pass along the
highway 150 by the capture vehicle 225 operating the light
intensity measurement system 230, vehicle positioning system 240,
image capture system 250 and data recording system 260 taught
herein eliminates many shortcomings of the prior art and allows a
single vehicle operator to conduct virtually continuous data
measurement and acquisition of objects of interest 460 disposed
adjacent a highway 150, at capture events 430 on said highway 150,
without disrupting or interrupting other vehicle traffic traversing
said highway 150.
[0047] FIG. 3 shows a block diagram of the on-board systems and the
desktop systems required to record a tagged videostream 440 and
create sign R.sub.A profiles 590 for various signs 190 along a
highway 150. The vehicle positioning system 240 contains all of the
equipment to precisely locate the capture vehicle 225. All location
information is synchronized with a master clock 340 preferably
associated with a computer processor 450, which allows other data
types to be merged with the vehicle location information at later
stages in the post-processing. All of the on-board systems utilize
the same master clock 340 information, thus allowing any events
(image capture system 250, intensity measurement 300, and trigger
controls 227, 228) to be correlated to the precise vehicle location
and attitude during real-time, near real-time, or post-processing
of the data acquired by the capture vehicle 225.
[0048] The image capture system 250 consists of at least one set of
stereoscopic cameras 360 that gather digital imagery along the
highway 150. Each capture event is combined with time stamp
information from the vehicle positioning system 240 which also
provides trigger control 227 for the image capture system 250 and
trigger control 228 for the light intensity measurement system 230.
These images and associated time stamps are later combined with
photogrammetry to create objects of interest 460 and their
associated attributes 465.
[0049] The light intensity measurement system 230 preferably
consists of at least one high output light source(s) 270 and the
associated light intensity sensor(s) 280. The precise control for
these items is contained within the light intensity measurement
system 230, and master time sequencing instrument 340 information
received from the vehicle positioning system 240 (or computer
processor 450) is combined to create a tagged videostream 440 so
precise vehicle information can be utilized during
post-processing.
[0050] The data recording system 260 is constantly monitoring
control information from the other three on-board systems and
records the necessary information. No post-processing is performed
in the data recording system 260. As computer power increases in
the future, one skilled in the art could produce a system whereby
most, if not all, of the post-processing functions were performed
in the capture vehicle 225, perhaps even in real time. The
inventors can imagine several uses for the production of real-time
information from the image capture system 250 in the future, but
the cost of obtaining such information with today's computing power
makes this option prohibitively expensive today.
[0051] The lower half of FIG. 3 shows the functional blocks for
data post-processing. There are two main functions--the creation of
objects of interest 460 and their associated attributes 465, and
the determination of retroreflectivity for each object of interest
460. There are many methods for creating objects of interest 460
from digital imagery, a few of which are discussed in this
disclosure. The specific steps required to compute R.sub.A are
outlined in the discussion below.
[0052] FIG. 4 shows a typical configuration within a capture
vehicle that is capable of producing data and imagery to create
digital representations of objects of interest 460 and objects of
interest retroreflectivity 466. The distance measuring instrument
(DMI) 330, GPS receiver 310 and inertial navigation system (INS)
320 constitute the vehicle positioning system 240. Not all of these
components are necessary to obtain the desired results, but better
precision, and therefore more meaningful data, are produced if all
three components are included.
[0053] The high output light source(s) 270 and light intensity
sensor(s) 280 constitute the light intensity measurement system
230. These components make it possible to gather on-the-fly
information for a desired highway 150 to allow the computation of
object of interest retroreflectivity 466, as well as create a full
3-D sign R.sub.A profile 590 for those same objects of interest
460.
[0054] The stereoscopic cameras 360 constitute the digital imagery
system 390 that allows for the creation of objects of interest 460
and their associated attributes 465 during post-processing. More
than one set of stereoscopic cameras 360 can be employed, thus
increasing the accuracy of positional measurements for objects of
interest 460. Other, non-stereoscopic imaging systems could also be
employed with little or no change to the vehicle positioning system
240 or to the light intensity measurement system 230.
[0055] FIG. 5 shows the top view of a 4-lane divided highway 400
with a stop sign 190. The capture vehicle 225 is traveling in the
proximate lane 410 to the stop sign 190 and makes intensity
measurements 300 at capture events 430 while traveling the depicted
route. The techniques described herein will allow a
retroreflectivity value for this stop sign 190 to be computed for
any point along the 4-lane divided highway 400, independent of
whether the intensity measurement 300 was made at that point and
also independent of whether the capture vehicle 225 actually drove
over that point.
[0056] It should be noted that intensity measurements 300 are made
continuously while the capture vehicle 225 is in motion, thus
requiring no prior knowledge of either the positions or the
existence of signs.
[0057] FIG. 6 shows some typical reflected light intensity frames
420 as captured by the light intensity sensor 280 at various
discrete locations along the roadway. These reflected light
intensity frames 420 are the result of the high output light source
270 being energized (or flashed) while each reflected light
intensity frame 420 is captured by one or more light intensity
sensors 280. Since most of the objects in the scene are not
reflective, and due to the high setting of the threshold range in
the light intensity sensor(s) 280, the reflected light intensity
frames 420 will actually show very few objects. For effective
luminance results throughout a wide range of retroreflective
materials, more than one light intensity sensor 280 may be required
in order to get enough levels of gray within the active part of the
visible spectrum. When multiple light intensity sensors 280 are
required or used, they may each have different threshold ranges and
each thus detect luminance values in different parts of the desired
luminance ranges.
[0058] In order to compute retroreflectivity (R.sub.A), one needs
to know the luminance of the reflected energy. Luminance (expressed
in candelas per square meter, or cd/m.sup.2) will vary according to
the intensity sensor characterization profile 275 of the light
intensity sensor(s) 280 and the color of the material from which
light is reflected.
[0059] Most roadway signs 190 contain text and/or symbols overlaid
on a background. To ensure maximum visibility during day and night
conditions, the colors of the foreground information (text and/or
symbols) are chosen to have maximum day and night contrast with the
background material. The techniques taught herein allow the
retroreflectivity of roadway signs 190 to be determined for both
foreground and background materials. Computing both the foreground
530 and background retroreflectivity 520 for each object of
interest 460 allows us to ensure that the proper nighttime contrast
is achieved for roadside assets. For example, a stop sign 190 with
a red background and white lettering can provide good daytime
contrast between the text and the sign background. But if these two
materials display very similar retroreflectivity characteristics,
their nighttime contrast will be minimal, thus rendering the sign
ineffective during nighttime conditions.
[0060] FIG. 7 shows a block diagram of the steps required to
transform intensity measurements 300 into foreground luminance
values 510 and background luminance values 500. First, a black and
white camera is preferably used as a light intensity sensor 280 to
maximize the sensitivity of intensity measurements 300 (intensity
will be determined from the gray value of the corresponding
pixels). Think of an intensity measurement 300 as intensity values
for N discrete points within the scene, where N corresponds to the
number of pixels in the light intensity sensor's 280 array. For a
light intensity sensor 280 that has a resolution of 640.times.480
pixels, there are 307,200 discrete intensity values for each
intensity sensor measurement 300. Although the preferred embodiment
utilizes an intensity sensor measurement 300 in the form of an
array of discrete pixel intensity values, preferably a single pixel
intensity value is selected and utilized for the automated
determination of a corresponding retroreflectivity value.
Alternatively, an average or other combination of a group of pixel
intensity values could be utilized for the automated determination
of a corresponding retroreflectivity value. Intensity values will
vary according to the color of the reflected light, since not all
colors of incoming light excite the light intensity sensor 280
pixels in the same way. By knowing the background or foreground
color of the object of interest 460 along with the light intensity
sensor's 280 ability to sense, or the light intensity sensor's 280
profile for a particular color, the intensity value 300 for a
particular color can be converted into a luminance value. Light
intensity sensor 280 characterization is essential for high
precision computations since N photons of a given particular color
(or wavelength) of light will represent a different gray value
(intensity level) in the sensor than N photons of another color (or
wavelength) of light. The look-up-table (LUT) 475 shown in FIG. 7
is a digital table stored in memory that uses the indexes of
intensity (a single gray level value from the intensity measurement
300) and sheeting color to determine the luminance. The light
intensity sensor characterization 275 is empirical information
about the light intensity sensor 280 that is used to create the LUT
475. The same LUT 475 is used for computing foreground 510 and
background luminance values 500.
[0061] The reader should note and appreciate that luminance is
strictly a measure of the reflected light, while retroreflectivity
(or R.sub.A, expressed in candelas/lux/m.sup.2) is a measure of the
reflected light with respect to the incident light for that object.
FIG. 7 shows the information needed to accurately convert luminance
to R.sub.A: sensor location, object location, light source
characterization, and color of reflective material. For less
precise R.sub.A computations, a subset of the aforementioned
characteristics can be utilized. For example, if a uniform light
source (equal intensity throughout the scene), columnated light
reflected from the surface of the object of interest 460, and a
known distance 220 between the object of interest 460 and the light
intensity sensor 280 are all assumed, then the sheeting color and
luminance value may be used to determine a rough approximation
(within 20%, for example) for R.sub.A.
[0062] To obtain the highest quality R.sub.A calculations, all of
the data shown in FIG. 7 should be utilized. The characterization
of light source angle defines the amount of light emitted from the
high output light source 270 throughout the source's field of view.
Due to the limitations of lamp design and their associated
reflectors, most semi-uniform light sources will have their
greatest intensity at or near the normal vector for the light
source. Since the high output light source(s) 270 are not aimed at
objects of interest 460, the part of the incident light beam that
is striking the object of interest 460 when the intensity
measurement 300 is captured must be determined. Light source angle
characterization is a process whereby empirical data from the light
is modeled to establish the light intensity for numerous discrete
vectors from the center of the light. When intensity values are
determined for a discrete point in the scene (from the object's
face surface 130), the light intensity sensor 280 location and
heading, as well as the object of interest 460 location, are used
to determine which light vector emanating from the light source was
responsible for the resulting intensity measurement. The
characterization of light source angle therefore, is a
look-up-table where an object of interest's 460 angular
displacement from the normal vector 180 for the high output light
source 270 is converted to a light intensity for the associated
vector.
[0063] Since the beam from the high output light source 270 is
diverging, objects of interest 460 farther from the origin of the
light will receive less incident radiation than those objects of
interest 460 closer to the light. The characterization of light
source angle is constructed at a few discrete distances from the
light. Simple geometry can be used to compute the incident
radiation (using an interpolation method for an actual distance
between two discrete distances in the characterization of light
source angle) hitting the actual object of interest 460 based on
the empirical data from the characterization of light source
angle.
[0064] The preferred high output light source 270 is a uniform
full-spectrum (visible spectrum) light. In practice, this light
source will not emit the same intensity for all wavelengths of
visible light. One variable of light source color characterization
that should be considered is the output profile of the light
throughout the visible spectrum. FIG. 8 shows a typical
full-spectrum light source output profile. Note that the intensity
in the blue area (400-500 nm) of the spectrum is stronger than in
the red area (600-700 nm). This profile specifies the amount of
light energy (number of photons) emitted for a given frequency.
Since R.sub.A depends on the intensity of the incident light, the
light source color characterization 540, light source angle
characterization 535, background sheeting color 505 and foreground
sheeting color 506 must be combined to determine how the background
luminance value 500 and foreground luminance value 510 is converted
to R.sub.A (i.e., what percent of the incident photons of the
foreground/background color were reflected back to the sensor).
[0065] The divergence pattern for the light source may have
different profiles for various portions of the visible spectrum. In
practice, a separate light source angle characterization profile
may be required for each possible foreground and background color
of any given object of interest 460.
[0066] A preferred high output light source 270 is of the type set
forth in the attached installation and operation guide entitled
"StrobeGuard.TM. High Intensity Obstruction Lighting System, Model
No. SG-60," manufactured by Honeywell, Inc. In order to create a
useful sign R.sub.A profile 590 for an object of interest 460,
intensity measurements 300 for frequent capture events 430 along a
highway 150 while the capture vehicle 225 is in motion. For
example, at vehicle speeds of 50 miles per hour, intensity
measurements 300 should be taken at a rate of at least two per
second. The StrobeGuard.TM. SG-60 model has a recharge time of
about 1.5 seconds between successive flash events. As a result, one
SG-60 light will not provide enough flash events per second to
allow an adequate number of intensity measurements 300. In order to
meet the requirements of two flash events per second for a capture
vehicle 225 traveling at 50 miles per hour, three of the
StrobeGuard.TM. SG-60 units would need to be fired in a
synchronized, round-robin pattern to obtain enough trigger
events.
[0067] The light intensity measurement system 230 described herein
attempts to remove observation angle 100 as an R.sub.A variable.
This is done by keeping the offset between the high output light
source(s) 270 and light intensity sensor(s) 280 as low as possible.
As mentioned previously, an R.sub.A profile of a simulated roadway
580 can be computed, even though the intensity was not measured at
every point and even though the capture vehicle 225 did not drive
over every point. First, it is critical that the geometry of
R.sub.A is understood. Reflective materials like sign sheeting are
designed to project near-columnated light back toward the light
source. If a perfectly columnated light being reflected from the
object of interest 460 being measured and a zero observation angle
are assumed, the R.sub.A values for all discrete locations along a
ray projected from the object will be identical.
[0068] FIG. 9 shows how to compute R.sub.A for any discrete
location along a 4-lane divided highway 400. The R.sub.A value for
the desired point will be based on the R.sub.A value that lies
along the pathway traveled by the data acquisition vehicle 225. To
compute this "reference R.sub.A value" (the R.sub.A value for a
discrete location on or along a vehicle path), an undetermined
retroreflectivity ray 570 is drawn from the desired location to the
face of the reflective asset. The discrete location where the
undetermined retroreflectivity ray 570 intersects the vehicle path
will be used as the reference R.sub.A value. Since the discrete
location on the vehicle path will always lie between two measured
locations where intensity measurements 300 were made, the reference
R.sub.A value is computed by interpolating the two closest (in
distance) R.sub.A values along the vehicle path. As used herein,
interpolate has the usual and typical meaning. It will be
understood that interpolation consistent with the present invention
can involve interpolation followed by extrapolation and shall also
include such other special mathematical expressions used or created
to account for border effects and effects at the lateral periphery
and at the furthest distance where R.sub.A may be reliably
determined by application of the teaching of this disclosure.
[0069] If perfectly columnated light is assumed, the value of
R.sub.A at the desired point will be the same as the reference
R.sub.A value. In practice, all sign 190 sheeting materials will
have some beam divergence for reflected light. This beam divergence
information can be used to adjust the computed R.sub.A value up (or
down) from the reference R.sub.A value for discrete locations
closer to (or farther from) the object's face surface 130.
[0070] While knowing the normal vector 180 to a sign 190 face is
not required, there are some advantages for planning and
maintenance purposes that make the information useful. Several ways
to compute the normal vector 180 for a sign 190 exist. First of
all, the "assumption" method requires that the normal vector 180
from the surface of the sign 190 is assumed to be parallel to the
capture vehicle pathway 410 at the nearest location of the capture
vehicle pathway 410 to the sign 190. Second, a scanning laser
operating in conjunction with an optical sensor and having a common
field of view may be used to more precisely resolve the normal
vector 180 from the object's face surface 130. Third, stereoscopic
cameras 360 may be employed in a useful, albeit very imprecise
manner of determining the normal vector 180. Fourth, the assumption
method and stereo imaging method may be combined whereby the normal
vector 180 is assumed to lie parallel to the vehicle pathway 410
unless the stereo imaging output renders the assumption false.
[0071] Of the methods listed above, the highest precision measuring
systems for determining the normal vector 180 consists of a scanned
laser and associated optical sensor. This combination yields
relative distance measurements between the capture vehicle 225 and
the object's face surface 130 that are more precise than optical
measurements with cameras. A laser scanner attached to the capture
vehicle 225 and directed toward a roadside scene populated with
retroreflective signs 130 generates a series of reflection points
to the optical sensor that appear as a horizontal segment of
points. The optical sensor must be fast enough (i.e., have adequate
data acquisition rates) to capture at least several individual
discrete measurements across the object's face surface 130 (or of
any other reflective asset). In general, two types of laser
scanners are suitable to be utilized according to the present
invention; namely, single-axis scanners and dual-axis scanners. A
preferred sensor is of the type set forth in the proposal entitled,
"Holometrics 3-D Vision Technology," as referenced in the
previously identified provisional patent application.
[0072] Since most all types of roadside signs 190 to be measured
are disposed at various elevations relative to the highway 150 and
the capture vehicle 225, a single-axis laser scanner cannot be
mounted such that the scanning laser beam covers only a single
elevation or constant height relative to the highway 150 and the
capture vehicle 225. Rather, the inventors hereof suggest that use
of a single-axis type laser scanner must either be mounted high on
the capture vehicle 225 with a downward facing trajectory, or be
mounted low on the capture vehicle 225 with an upward facing
scanning trajectory. These two mounting schemes for a single-axis
laser scanner help ensure the lateral scan will intersect with
virtually every object face surface 130 of all signs 190 or other
objects of interest 460 present in a roadside scene regardless of
the elevation or height or such signs relative to the roadway or to
the moving platform.
[0073] Dual-axis laser scanners 335 circumvent the varying sign
height problem inherently encountered if a single-axis laser
scanner is employed as the source of integrated energy when
practicing the teaching of the present invention. A dual-axis laser
scanner 335 operates by continuously moving the scanning beam scan
up and down at a relatively slow rate while sweeping the laser beam
laterally from side to side across the field of view at a
relatively more rapid rate.
[0074] In order to obtain the normal vector 180 for a sign 190 as
taught hereunder, only a select horizontal series of discrete
locations across the object's face surface 130 needs to be sensed
by the high-speed optical sensor. For each point in the horizontal
series of discrete locations recorded for a given sign 190 due to
the incident radiation provided by the scanning laser, as sensed by
the high speed optical sensor, the precise direction of the
incident laser is recorded, thus allowing both distance and
direction of the measured point to be determined.
[0075] Either of the scanning methods produces a massive number of
sensed discrete locations representing discrete reflections of the
incident laser radiation and each must be processed in order to
correlate each of the sensed discrete locations with the object's
face surface 130. Once the lateral series of discrete locations for
a sign 190 is determined, simple triangulation methods are used to
combine: (i) the vehicle location, (ii) vehicle heading vector, and
(iii) scanned sign point to ultimately determine the normal vector
180 for the object's face surface 130.
[0076] As stated earlier, knowing the sign's 190 normal vector 180
can expand the utilization of the present invention. The
retroreflective properties of sign 190 sheeting materials are
typically symmetrical about the vertical axis of the object's face
surface 130. Because of this symmetry, R.sub.A values (either
computed or extrapolated/interpolated values) will be identical for
rays that are symmetrical about the vertical axis.
[0077] FIG. 10 shows how the sign's 190 normal vector 180 can be
used to extrapolate more R.sub.A points. The R.sub.A value for
Point B is the same as the R.sub.A value for Point A since their
angle relative to the normal vector 180 is the same and since their
distance from the sign 190 is the same. If Point B has the same
relative angle (from the sign's 190 normal vector 180) as Point A,
but lies closer to (or farther from) the object's face surface 130,
the sign 190 material's beam divergence profile can be used to
adjust the R.sub.A value for Point B up (or down) from the value
obtained for Point A.
[0078] The image capture system 250 and light intensity measurement
system 230 are preferably free running, with measurements being
made periodically during capture vehicle 225 operation. There is no
requirement that these two systems be synchronized. In fact, these
systems could operate in completely different capture vehicles 225,
if necessary. When both systems are contained within the same
capture vehicle 225, the only constraint for simultaneous operation
is placed on the image capture system 250. Because of the intensity
of the high output light source 270 in the light intensity
measurement system 230, it is preferred that the image capture
system 250 not capture frames at the same instant that the high
output light source 270 is triggered. If images are actually
captured while the high output light source 270 is triggered, their
positional results would still be valid, but the colors displayed
would be inaccurate because of the high output light being directed
toward the (typically lower-thresholded) stereoscopic cameras
360.
[0079] One skilled in the art could completely eliminate any need
for the image capture system 250 to know the firing events of the
light intensity measurement system 230 by choosing sampling rates
for the two systems that do not share any harmonic frequencies. On
the rare occasions when the image capture system 250 captures
images while the high output light source 270 is energized (or
flashed), the skilled implementer could use time stamps to
determine when this system simultaneity occurred and discard the
imaging frames.
[0080] Referring now to FIGS. 14A and 14B, a preferred embodiment
of the flowchart for the sign sheeting threshold algorithm will be
described. In the preferred embodiment, certain assumptions are
made that simplify the threshold algorithm process. It will be
understood that additional assumptions could be made to further
simplify the process, or that the process could be further expanded
to allow certain of the assumptions to be avoided. In the preferred
embodiment, it is assumed that each road sign 30 has only two
colors of reflective sheeting and that the sheeting type is the
same for both the background sheeting color 505 and the foreground
sheeting color 506. It is also assumed that the retroreflectivity
for both the background R.sub.A 520 and the foreground R.sub.A 530
are non-zero values. As previously described, the algorithm assumes
that prefiltering has eliminated retroreflectivity values for road
signs 30 that demonstrate some type of gross failure of the
sheeting material, such as delamination, excessive surface wear,
extreme color fade, vapor fade, graffiti, or excessive dirt or
other obstructions that would prevent an accurate determination of
the retroreflectivity value. Such filtering is preferably
accomplished by image analysis of the color images using any number
of known image analysis techniques for characterizing anomalies in
images.
[0081] The sign sheeting threshold algorithm process is initiated
at step 600. At steps 610-624, the background sheeting color 505 is
compared to a series of known sign sheeting colors. If the
background color is yellow-green as determined at step 610, then
the sign sheeting type is classified as Type VII as step 628. If
the background color is white as determined at step 612, then the
background R.sub.A 520 is compared to the maximum retroreflectivity
values for different sheeting types at steps 630, 632. If the
background R.sub.A 520 is less than the maximum white
retroreflectivity value for sheeting Type I as determined at step
630, then the sign sheeting type is classified as Type I at step
634. Otherwise, if the background R.sub.A 520 is less than the
maximum white retroreflectivity value for sheeting Type III as
determined at step 632, then the sign sheeting type is classified
as Type III at step 636. If neither steps 630 or 632 are satisfied,
then the sign sheeting type is classified as Type VII at step 638.
A similar process is repeated for colors yellow at step 614 and
steps 640, 642, orange at step 616 and steps 644, 646, red at step
618 and steps 650, 652.
[0082] If the background color is either green, blue or brown, as
determined at steps 620, 622 and 624, then a second determination
is made at step 660 whether the foreground color 506 is white and
at step 670 whether the foreground color is yellow. If step 660 is
satisfied, then the foreground R.sub.A 520 is compared to the
maximum retroreflectivity values for different sheeting types at
steps 662, 664. If the foreground R.sub.A 530 is less than the
maximum white retroreflectivity value for sheeting Type I as
determined at step 662, then the sign sheeting type is classified
as Type I at step 666. Otherwise, if the foreground R.sub.A 530 is
less than the maximum white retroreflectivity value for sheeting
Type III as determined at step 664, then the sign sheeting type is
classified as Type III at step 668. If neither steps 662 or 664 are
satisfied, then the sign sheeting type is classified as Type VII at
step 669. A similar process is repeated for the yellow foreground
color at steps 672 and 674.
[0083] In the event that the background color 505 was not
identified in steps 610-624 or the foreground color 506 was not
identified in steps 660, 670, the sign sheeting type is considered
undetermined at step 680. It will be understood that various
options can be undertaken at step 680, including obtaining another
color image and set of retroreflectivity values for the given road
sign either with or without additional filtering or preprocessing
of the raw data, marking the image and data for further review by
an operator, discarding the information and marking the road sign
as needing manual investigation, marking the road sign as needing
replacement, or any combination of these or other operations.
[0084] Although the preferred embodiment of the threshold algorithm
has been described with respect to the use of maximum
retroreflectivity values, it will be understood that the threshold
algorithm could utilize either minimum or maximum retroreflectivity
values. Similarly, the combinations and orders of comparison of the
colors and foreground or background colors may be altered and
additional comparisons may be made to accommodate additional
sheeting types that may be developed. Preferably, the colors 505
and 506 are determined based on CIELUV color values as evaluated by
the image capture system 250. Alternatively, other equivalent color
value systems such as RGB could be used for the color values.
Preferably, the color values for the comparisons are a range of
values specified by the manufacturer of the sheeting material.
Alternatively, the color values for the comparison can be ranges of
values empirically established.
* * * * *