U.S. patent application number 09/852487 was filed with the patent office on 2002-05-09 for method and apparatus for traffic light violation prediction and control.
This patent application is currently assigned to Nestor Traffic Systems, Inc.. Invention is credited to Glier, Michael T., Hsieh, Steven S., Laird, Mark D., Reilly, Douglas L., Small, Steven I., Sybel, Randall T., Tinnemeier, Michael T..
Application Number | 20020054210 09/852487 |
Document ID | / |
Family ID | 26720719 |
Filed Date | 2002-05-09 |
United States Patent
Application |
20020054210 |
Kind Code |
A1 |
Glier, Michael T. ; et
al. |
May 9, 2002 |
Method and apparatus for traffic light violation prediction and
control
Abstract
A traffic sensor system for detecting and tracking vehicles is
described. The disclosed system may be employed as a traffic light
violation prediction system for a traffic signal, and as a
collision avoidance system. A video camera is employed to obtain a
video image of a section of a roadway. Motion is detected through
changes in luminance and edges in frames of the video image.
Predetermined sets of pixels ("tiles") in the frames are designated
to be in either an "active" state or an "inactive" state. A tile
becomes active when the luminance or edge values of the pixels of
the tile differ from the respective luminance or edge values of a
corresponding tile in a reference frame in accordance with
predetermined criteria. The tile becomes inactive when the
luminance or edge values of the pixels of the tile do not differ
from the corresponding reference frame tile in accordance with the
predetermined criteria. Shape and motion of groups of active tiles
("quanta") are analyzed with software and a neural network to
detect and track vehicles.
Inventors: |
Glier, Michael T.;
(Jamestown, RI) ; Reilly, Douglas L.;
(Narragansett, RI) ; Tinnemeier, Michael T.;
(Providence, RI) ; Small, Steven I.; (Medfield,
MA) ; Hsieh, Steven S.; (Canton, MA) ; Sybel,
Randall T.; (Randolph, VT) ; Laird, Mark D.;
(Milford, MA) |
Correspondence
Address: |
WEINGARTEN, SCHURGIN, GAGNEBIN & LEBOVICI LLP
TEN POST OFFICE SQUARE
BOSTON
MA
02109
US
|
Assignee: |
Nestor Traffic Systems,
Inc.
|
Family ID: |
26720719 |
Appl. No.: |
09/852487 |
Filed: |
May 10, 2001 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
09852487 |
May 10, 2001 |
|
|
|
09059151 |
Apr 13, 1998 |
|
|
|
60043690 |
Apr 14, 1997 |
|
|
|
Current U.S.
Class: |
348/149 ;
348/E7.087 |
Current CPC
Class: |
H04N 7/183 20130101;
G08G 1/164 20130101; G08G 1/04 20130101; G08G 1/08 20130101; G08G
1/087 20130101 |
Class at
Publication: |
348/149 |
International
Class: |
H04N 007/18 |
Claims
What is claimed is:
1. A traffic light violation prediction system for a traffic signal
having a at least a red phase and a green phase, comprising: at
least one image capturing device, said image capturing device
operative to provide image data of at least one vehicle approaching
said traffic signal; and a computation unit, operative in response
to said image capturing device and an indication of said current
traffic light phase, to determine whether said at least one vehicle
approaching said traffic signal will violate a red light phase of
said traffic signal.
2. The system of claim 1, wherein said image capturing device
comprises at least one video camera.
3. The system of claim 1, wherein said traffic signal has a yellow
light phase, and said computation unit is further responsive to a
time remaining in said yellow light phase.
4. The system of claim 1, wherein said computation unit is further
responsive to a current speed of said at least one vehicle
approaching said traffic intersection.
5. The system of claim 1, wherein said computation unit is further
responsive to a current acceleration of said at least one vehicle
approaching said traffic intersection.
6. The system of claim 1, wherein said computation unit is further
responsive to a current position of said at least one vehicle
approaching said traffic intersection.
7. The system of claim 1, wherein said computation unit is further
operable to compute a time remaining before said at least one
vehicle approaching said traffic intersection enters said traffic
intersection, responsive to a determination of a current
acceleration of said vehicle.
8. The system of claim 7, wherein said computation unit is further
operable to calculate a rate of deceleration required for said at
least one vehicle to stop within said time remaining before said
vehicle enters said traffic intersection.
9. A method for predicting a traffic light violation of a traffic
signal having at least a red phase and a green phase, comprising:
providing image data showing at least one vehicle approaching said
traffic signal; and determining, responsive to said image data and
an indication of a current traffic light phase, whether said at
least one vehicle approaching said traffic signal will violate a
red light phase of said traffic signal.
10. The method of claim 9, wherein said image data is generated by
at least one video camera.
11. The method of claim 9, wherein said determining is performed by
a computation unit comprising software executing on a
processor.
12. The method of claim 9, wherein said traffic light further
includes a yellow phase, and wherein said determining is further
responsive to a time remaining in said yellow phase.
13. The method of claim 9, wherein said determining further
includes the step of determining a current speed for said at least
one vehicle approaching said traffic intersection.
14. The method of claim 9, wherein said determining further
includes the step of determining a current acceleration for said
vehicle approaching said traffic intersection.
15. The method of claim 14, wherein said determining further
includes computing a time remaining before said vehicle approaching
said traffic intersection enters said traffic intersection,
responsive to said determination of said current acceleration of
said vehicle.
16. The method of claim 15, further comprising calculating, by said
computation unit, a deceleration required for said vehicle to stop
within said time remaining before said vehicle enters said traffic
intersection.
17. A collision avoidance system for a first traffic signal having
a current light phase equal to one of a red light phase and a green
light phase, and a second traffic signal having a current light
phase equal to one of a red light phase and a green light phase,
comprising: at least one image capturing device, for capturing a
plurality of images; said plurality of images showing at least one
vehicle approaching said first traffic signal; a computation unit,
responsive to said plurality of images and an indication of said
current first traffic signal light phase, for determining whether
said at least one vehicle approaching said first traffic signal
will violate said red light phase of said first traffic signal, and
for delaying an upcoming green traffic light phase of said second
traffic signal responsive to a determination that said at least one
vehicle approaching said first traffic signal will violate a red
phase of said first traffic signal.
18. The system of claim 17, wherein said image capturing device
comprises at least one video camera.
19. The system of claim 17, wherein said computation unit comprises
software executing on a processor.
20. The system of claim 17, wherein said computation unit is
further responsive to a time remaining in yellow light phase
input.
21. The system of claim 17, wherein said computation unit is
further operable to determine a current speed for said at least one
vehicle.
22. The system of claim 1, wherein said computation unit is further
operable to determine a current acceleration for said at least one
vehicle.
23. The system of claim 17, wherein said computation unit is
further operable to compute a time remaining before one of said at
least one vehicle enters said traffic intersection, responsive to
determination of a current acceleration of said vehicle.
24. The system of claim 23, wherein said computation unit is
further operable to calculate a deceleration required for said at
least one vehicle to stop within said time remaining before said
vehicle enters said traffic intersection.
25. A method of collision avoidance for a first traffic signal
having a current light phase equal to one of the set including at
least red and green and a second traffic signal having a current
light phase equal to one of the set including at least red and
green, comprising: capturing a plurality of images, said images
showing at least one vehicle approaching said first traffic signal,
said images derived from an output of a violation prediction image
capturing device; determining, responsive to said plurality of
images and indication of said current first traffic signal light
phase, whether said at least one vehicle approaching said first
traffic signal will violate a red light phase of said first traffic
signal; and delaying an upcoming green light phase of said second
traffic signal for a programmed time period responsive to a
determination that said at least one vehicle approaching said first
traffic signal will violate said red light phase of said first
traffic signal.
26. The method of claim 25, wherein said violation prediction image
capturing device comprises at least one video camera.
27. The method of claim 25, wherein said collision avoidance unit
comprises software executing on a processor.
28. The method of claim 25, further comprising: determining at
least one vehicle location associated with said at least one
vehicle; and wherein said determining whether said at least one
vehicle will violate said red light phase of said first traffic
signal is responsive to said at least one vehicle location.
29. The method of claim 25, further comprising: determining a time
remaining in a current yellow light phase; and wherein said
determining whether said at least one vehicle will violate said red
light phase of said first traffic signal is responsive to said time
remaining in said current yellow light phase.
30. The method of claim 25, further comprising: determining a
current speed for said at least one vehicle; and wherein said
determining whether said at least one vehicle will violate said red
light phase of said first traffic signal is responsive to said
current speed of said at least one vehicle.
31. The method of claim 25, wherein said determining whether said
at least one vehicle will violate said red light phase of said
first traffic signal further comprises determining a current
acceleration for said at least one vehicle.
32. The method of claim 25, wherein said determining whether said
at least one vehicle will violate said red light phase of said
first traffic signal further comprises computing a time remaining
before said at least one vehicle enters said traffic
intersection.
33. The method of claim 32, wherein said determining whether said
at least one vehicle will violate said red light phase of said
first traffic signal further comprises calculating a rate of
deceleration required for said at least one vehicle to stop within
said time remaining before said vehicle enters said traffic
intersection.
34. The method of claim 33, wherein said determining whether said
at least one vehicle will violate said red light phase of said
first traffic signal further comprises determining whether said
required deceleration is larger than a specified deceleration limit
value.
35. An apparatus for facilitating operation of a traffic light at
an intersection of first and second roadways, comprising: at least
one camera that provides first and second video frames that are
representative of a field of view of said camera at different
points in time; a video processing circuit that detects vehicles;
and a processor circuit that determines vehicle position in the
first and second video frames and vehicle velocity from the
difference in vehicle position, said processor circuit being
operative to delay the operation of said traffic light in response
to predetermined conditions associated with said first and second
video frames.
36. An apparatus for facilitating operation of a traffic light at
an intersection of first and second roadways, comprising: at least
one camera that provides first and second video frames that are
representative of a field of view of said camera at different
points in time; a video processing circuit that detects vehicles;
and a processor circuit that determines vehicle position in the
first and second video frames and vehicle velocity from the
difference in vehicle position, said processor circuit being
operative to generate a signal indicating that a vehicle is about
to pass through said intersection in violation of a red-light.
37. A method for controlling vehicular traffic at an intersection
having a first traffic light for controlling traffic approaching
said intersection from a first direction and a second traffic light
for controlling traffic approaching said intersection from a second
direction, wherein each of said traffic lights includes at least a
red phase and a green phase, said method comprising: obtaining a
plurality of images of a vehicle approaching said intersection from
said first direction; analyzing said plurality of images to predict
whether said vehicle will violate said red light phase of said
first traffic light; responsive to a determination said vehicle
will violate said red light phase of said first traffic light
generating a delay signal indicative of said predicted
violation.
38. The method of claim 37 further including delaying a change of
said second traffic signal from said red light phase to said green
light phase responsive to said delay signal.
39. The method of claim 37, wherein said obtaining said plurality
of images is performed by an image capturing device comprising at
least one video camera.
40. The method of claim 37, wherein analyzing said plurality of
images is performed by a collision avoidance unit comprising
software executing on a processor.
41. The method of claim 37, further comprising: determining at
least one vehicle location associated with said at least one
vehicle; and wherein said determining whether said at least one
vehicle will violate said red light phase of said first traffic
signal is responsive to said at least one vehicle location.
42. The method of claim 37, further comprising: determining a time
remaining in a current yellow light phase; and wherein said
determining whether said at least one vehicle will violate said red
light phase of said first traffic signal is responsive to said time
remaining in said current yellow light phase.
43. The method of claim 37, further comprising: determining a
current speed for said at least one vehicle; and wherein said
determining whether said at least one vehicle will violate said red
light phase of said first traffic signal is responsive to said
current speed of said at least one vehicle.
44. The method of claim 37, wherein said determining whether said
at least one vehicle will violate said red light phase of said
first traffic signal further comprises determining a current
acceleration for said at least one vehicle.
45. The method of claim 37, wherein said determining whether said
at least one vehicle will violate said red light phase of said
first traffic signal further comprises computing a time remaining
before said at least one vehicle enters said traffic
intersection.
46. The method of claim 45, wherein said determining whether said
at least one vehicle will violate said red light phase of said
first traffic signal further comprises calculating a rate of
deceleration required for said at least one vehicle to stop within
said time remaining before said vehicle enters said traffic
intersection.
47. The method of claim 46, wherein said determining whether said
at least one vehicle will violate said red light phase of said
first traffic signal further comprises determining whether said
required deceleration is larger than a specified deceleration limit
value.
48. An accident avoidance system for an intersection having a
traffic signal having a current light phase equal to one of a red
light phase and a green light phase, and a pedestrian crosswalk
passing through a path of traffic controlled by said first traffic
signal, comprising: at least one image capturing device, for
capturing a plurality of images of said pedestrian crosswalk, said
plurality of images showing at least one pedestrian within said
pedestrian crosswalk; and a computation unit, responsive to said
plurality of images and an indication of said current traffic
signal light phase, for determining whether said at least one
pedestrian in said crosswalk will exit said crosswalk prior to said
current traffic signal light phase transitioning from red to green,
and for delaying an upcoming green traffic light phase of said
traffic signal responsive to a determination that said at least one
pedestrian will not exit said crosswalk prior to said current
traffic signal light phase transitioning from red to green.
49. The system of claim 48, wherein said image capturing device
comprises at least one video camera.
50. The system of claim 48, wherein said computation unit is
further responsive to a current speed of said at least one
pedestrian within said pedestrian crosswalk.
51. The system of claim 48, wherein said computation unit is
further responsive to a current acceleration of said at least one
pedestrian within said pedestrian crosswalk.
52. The system of claim 48, wherein said computation unit is
further responsive to a current position of said at least one
pedestrian within said pedestrian crosswalk.
53. The system of claim 48, wherein said computation unit is
further operable to compute a time remaining before said current
traffic signal light phase transitions from red to green.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application is a Divisional of U.S. Pat. application
Ser. No. 09,059,151, filed Apr. 13, 1998, entitled TRAFFIC SENSOR,
which claims priority to U.S. Provisional Patent Application Ser.
No. 60/043,690, entitled TRAFFIC SENSOR, filed Apr. 14, 1997.
STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT
[0002] N/A
BACKGROUND OF THE INVENTION
[0003] The present invention is related to traffic monitoring
systems, and more particularly to a traffic monitoring system for
detecting, measuring and anticipating vehicle motion.
[0004] Systems for monitoring vehicular traffic are known. For
example, it is known to detect vehicles by employing inductive loop
sensors. At least one loop of wire or a similar conductive element
may be disposed beneath the surface of a roadway at a predetermined
location. Electromagnetic induction occurs when a vehicle occupies
the roadway above the loop. The induction can be detected via a
simple electronic circuit that is coupled with the loop. The
inductive loop and associated detection circuitry can be coupled
with an electronic counter circuit to count the number of vehicles
that pass over the loop. However, inductive loops are subjected to
harsh environmental conditions and consequently have a relatively
short expected lifespan.
[0005] It is also known to employ optical sensors to monitor
vehicular traffic. For example, traffic monitoring systems that
employ "machine vision" technology such as video cameras are known.
Machine vision traffic monitoring systems are generally mounted
above the surface of the roadway and have the potential for much
longer lifespan than inductive loop systems. Further, machine
vision traffic monitoring systems have the potential to provide
more information about traffic conditions than inductive loop
traffic monitoring systems. However, known machine vision traffic
monitoring systems have not achieved these potentials.
SUMMARY OF THE INVENTION
[0006] In accordance with the present invention, a traffic
monitoring station employs at least one video camera and a
computation unit to detect and track vehicles passing through the
field of view of the video camera. The disclosed system may be used
as a traffic light violation prediction system for a traffic
signal, and/or as a collision avoidance system.
[0007] In an illustrative embodiment, the camera provides a video
image of a section of roadway in the form of successive individual
video frames. Motion is detected through edge analysis and changes
in luminance relative to an edge reference frame and a luminance
reference frame. The frames are organized into a plurality of sets
of pixels. Each set of pixels ("tile") is in either an "active"
state or an "inactive" state. A tile becomes active when the
luminance or edge values of the pixels of the tile differ from the
luminance and edge values of the corresponding tiles in the
corresponding reference frames in accordance with predetermined
criteria. The tile becomes inactive when the luminance and edge
values of the pixels of the tile do not differ from the
corresponding reference frame tiles in accordance with the
predetermined criteria.
[0008] The reference frames, which represent the view of the camera
without moving vehicles, may be dynamically updated in response to
conditions in the field of view of the camera. The reference frames
are updated by combining each new frame with the respective
reference frames. The combining calculation is weighted in favor of
the reference frames to provide a gradual rate of change in the
reference frames. A previous frame may also be employed in a
"frame-to-frame" comparison with the new frame to detect motion.
The frame-to-frame comparison may provide improved results relative
to use of the reference frame in conditions of low light and
darkness.
[0009] Each object is represented by at least one group of
proximate active tiles ("quanta"). Individual quantum, each of
which contains a predetermined maximum number of tiles, are tracked
through successive video frames. The distance traveled by each
quantum is readily calculable from the change in position of the
quantum relative to stationary features in the field of view of the
camera. The time taken to travel the distance is readily calculable
since the period of time between successive frames is known.
Physical parameters such as velocity, acceleration and direction of
travel of the quantum are calculated based on change in quantum
position over time. Physical parameters that describe vehicle
motion are calculated by employing the physical parameters
calculated for the quanta. For example, the velocities calculated
for the quanta that comprise the vehicle may be combined and
averaged to ascertain the velocity of the vehicle.
[0010] The motion and shape of quanta are employed to delineate
vehicles from other objects. A plurality of segmenter algorithms is
employed to perform grouping, dividing and pattern matching
functions on the quanta. For example, some segmenter algorithms
employ pattern matching to facilitate identification of types of
vehicles, such as passenger automobiles and trucks. A physical
mapping of vehicle models may be employed to facilitate the proper
segmentation of vehicles. A list of possible new objects is
generated from the output of the segmenter algorithms. The list of
possible new objects is compared with a master list of objects, and
objects from the list of possible new objects that cannot be found
in the master list are designated as new objects. The object master
list is then updated by adding the new objects to the object master
list.
[0011] At least one feature extractor is employed to generate a
descriptive vector for each object. The descriptive vector is
provided to a neural network classification engine which classifies
and scores each object. The resultant score indicates the
probability of the object being a vehicle of a particular type.
Objects that produce a score that exceeds a predetermined threshold
are determined to be vehicles.
[0012] The traffic monitoring station may be employed to facilitate
traffic control in real time. Predetermined parameters that
describe vehicle motion may be employed to anticipate future
vehicle motion. Proactive action may then be taken to control
traffic in response to the anticipated motion of the vehicle. For
example, if on the basis of station determined values for vehicle
distance from the intersection, speed, acceleration, and vehicle
class (truck, car, etc.), the traffic monitoring station determines
that the vehicle will "run a red light," traversing an intersection
during a period of time when the traffic signal will be otherwise
be indicating "green" for vehicles entering the intersection from
another direction, the traffic monitoring station can delay the
green light for the other vehicles or cause some other actions to
be taken to reduce the likelihood of a collision. Such actions may
also include displaying the green light for the other vehicles in
an altered mode (e.g., flashing) or in some combination with
another signal light (e.g., yellow or red), or initiating an
audible alarm at the intersection until the danger has passed.
Further, the traffic monitoring station may track the offending
vehicle through the intersection and record a full motion video
movie of the event for vehicle identification and evidentiary
purposes.
BRIEF DESCRIPTION OF THE DRAWING
[0013] The foregoing features of this invention, as well as the
invention itself, may be more fully understood from the following
Detailed Description of the Invention, and Drawing, of which:
[0014] FIG. 1A is a perspective diagram of a traffic monitoring
station that illustrates configuration;
[0015] FIG. 1B is a side view diagram of a traffic monitoring
station that illustrates tilt angle;
[0016] FIG. 1C is a top view diagram of a traffic monitoring
station that illustrates pan angle;
[0017] FIG. 2 is a flow diagram that illustrates the vehicle
detection and tracking method of the traffic monitoring
station;
[0018] FIG. 3 is a diagram of a new frame that illustrates use of
tiles and quanta to identify and track objects;
[0019] FIG. 4 is a diagram of a reference frame;
[0020] FIG. 5 is a diagram that illustrates edge detect tile
comparison to determine tile activation;
[0021] FIG. 6 is a diagram that illustrates adjustment of segmenter
algorithm weighting;
[0022] FIG. 7 is a diagram that illustrates feature vector
generation by a feature extractor;
[0023] FIG. 8 is a diagram of the traffic monitoring station of
FIG. 1 that illustrates the processing module and network
connections;
[0024] FIG. 9 is a block diagram of the video capture card of FIG.
8;
[0025] FIG. 10A is a diagram that illustrates use of the new frame
for image stabilization;
[0026] FIG. 10B is a diagram that illustrates use of the reference
frame for image stabilization;
[0027] FIG. 11 is diagram of the field of view of a camera that
illustrates use of entry and exit zones;
[0028] FIG. 12 is a block diagram of traffic monitoring stations
networked through a graphic user interface;
[0029] FIG. 13 is a flow diagram that illustrates station to
station vehicle tracking; and
[0030] FIG. 14 is a diagram of an intersection that illustrates
traffic control based on data gathered by the monitoring
station.
DETAILED DESCRIPTION OF THE INVENTION
[0031] U.S. Provisional Patent Application Ser. No. 60/043,690,
entitled TRAFFIC SENSOR, filed Apr. 14, 1997, is hereby
incorporated herein by reference.
[0032] Referring to FIGS. 1A, 1B and 1C, a traffic monitoring
station 8 includes at least one camera 10 and a computation unit
12. The camera 10 is employed to acquire a video image of a section
of a roadway 14. The computation unit 12 is employed to analyze the
acquired video images to detect and track vehicles.
[0033] A three dimensional geometric representation of the site is
calculated from parameters entered by the user in order to
configure the traffic monitoring station 8 for operation. The
position of a selected reference feature 16 relative to the camera
10 is measured and entered into memory by employing a graphic user
interface. In particular, a distance Y along the ground between the
camera 10 and the reference feature 16 on a line that is parallel
with the lane markings 17 and a distance X along a line that is
perpendicular with the lane markings are measured and entered into
memory. The camera height H, lane widths of all lanes W1, W2, W3
and position of each lane in the field of view of the camera are
also entered into memory. The tilt angle 15 and pan angle 13 of the
camera are trigonometrically calculated from the user-entered
information, such as shown in Appendix A. The tilt angle 15 is the
angle between a line 2 directly out of the lens of the camera 10
and a line 6 that is parallel to the road. The pan angle 13 is the
angle between line 2 and a line 3 that is parallel to the lane
lines and passes directly under the camera 10. A value used for
scaling ("scaler") is calculated for facilitating distance
calculations. The scaler is a fixed factor for the entire image
that is used for conversion between real distances and pixel
displacements. Hence, the distance and direction from the camera to
any point in the field of view of the camera, and the distance and
direction between any two points in the field of view of the camera
can be determined.
[0034] Corrections for roadway grade and bank may also be
calculated during configuration. "Grade" refers to the change in
height of the roadway relative to the height of the camera within
the field of view of the camera. "Bank" refers to the difference in
height of portions of the roadway along a line perpendicular with
the lane markings. The user determines the grade and bank of the
roadway and enters the determined values into memory by employing a
graphic user interface. The grade and bank corrections are achieved
by translating the reference plane to match the specified grade and
bank.
[0035] Referring to FIGS. 2 and 3, operation of the traffic
monitoring station will now be described. A video frame 18 is
acquired from the camera as depicted in step 20. If an interlaced
camera is employed, the acquired frame is de-interlaced. If a
progressive scan camera is employed then de-interlacing is not
necessary. Image stabilization techniques may also be employed to
compensate for movement of the camera due to wind, vibration and
other environmental factors, as will be described below. The
acquired frame 18 is organized into tiles 22 as depicted in step
24. Each tile 22 is a region of predetermined dimensions. In the
illustrated embodiment, each frame contains 80 tiles per row and 60
tiles per column and the dimensions of each tile are 8 pixels by 8
pixels. Tile dimensions may be adjusted, may be non-square, and may
overlap other tiles.
[0036] Referring to FIGS. 2, 3 and 4, a list 26 of tiles in which
motion is detected ("active tiles") 38 is generated by employing
either or both of reference frames 28, 29 and a previously acquired
frame 30 in separate comparisons with the acquired frame 18. The
reference frame 28 represents the luminance of the image from the
camera in the absence of moving vehicles. The reference frame 29
represents edges detected in the image from the camera in the
absence of moving vehicles. In the illustrated embodiment, both the
reference frames 28, 29 and the previous frame 30 are employed. If
a color camera is employed, the chrominance (color) portion of each
tile 22 in the acquired frame is separated from the luminance
(black and white) portion prior to comparison.
[0037] As illustrated in FIG. 5, an edge detect comparison may be
employed to detect motion and activate tiles. For each tile 22 of
the new frame 18 (FIG. 3), the tile is organized into four
"quartiles" 32 of equal size. The pixel luminance values in each
quartile 32 are summed to provide a representative luminance value
for each quartile. In the illustrated embodiment, each pixel has a
luminance represented by a value from 0 to 255, where greater
values indicate greater luminance. The quartile having the maximum
representative luminance value is then identified and employed as a
baseline for analyzing the other quartiles. In particular, the
maximum luminance quartile 34 is designated to be in a first state,
illustrated as logic 1. The other quartiles in the tile are
designated to be in the first state if their representative
luminance value exceeds a threshold defined by a predetermined
percentage of the luminance value of the maximum luminance quartile
34 (lum.gtoreq..beta.lum.sub.max). .beta. ("the gain") can be fixed
at a specific level or may be allowed to vary based upon the
characteristics of the image. Quartiles with a representative value
that fails to exceed the threshold are designated to be in a second
state, illustrated by logic 0. Each quartile is then compared with
the corresponding quartile from the corresponding tile 36 from the
reference frame 29 (FIG. 4) and, separately, the previously
acquired frame. The tile 22 is designated as "active" if the
comparison indicates a difference in the state of more than one
quartile. If the comparison indicates a difference in the state of
one or fewer quartiles and at least one quartile of the tile is in
the second state, the tile is designated as "inactive."
[0038] In the case where each quartile 32 in the corresponding
tiles of the current frame and the reference frame are designated
to be in the first state a luminance activation technique is
employed. A luminance intensity value is determined by summing the
luminance of all pixels in the tile and dividing the sum by the
total number of pixels in the tile, i.e., computing the average
luminance. The average luminance of the tile is compared with the
average luminance of the tile 36 in the corresponding location of
the reference frame 28 and the previous frame to detect any
difference therebetween. In particular, the average luminance of
the reference tile is subtracted from the average luminance of the
new tile to produce a difference value and, if the magnitude of the
difference value exceeds a predetermined threshold, motion is
indicated and the tile is designated as "active." The model using
tiles, quartiles and pixels is isomorphic to a neural model of
several layers.
[0039] Referring again to FIGS. 3 and 4, the reference frames 28,
29 may be either static or dynamic. A static reference frame may be
generated by storing a video frame from the roadway or portion(s)
of the roadway when there are no moving objects in the field of
view of the camera. In the illustrated embodiment the reference
frames 28, 29 are dynamically updated in order to filter
differences between frames that are attributable to gradually
changing conditions such as shadows. The reference frames are
updated by combining each new frame 18 with the reference frames.
The combining calculation may be weighted in favor of the reference
frames to filter quickly occurring events, such as the passage of
vehicles, while incorporating slowly occurring events such as
shadows and changes in the ambient light level.
[0040] Referring to FIGS. 2 and 3, and Appendix B, active tiles 38
in the list 26 of active tiles are organized into sets of
proximately grouped active tiles ("quanta") 40 as depicted by step
42. The quanta 40 are employed to track moving objects such as
vehicles on successive frames. The distance traveled by each
quantum is calculated based upon the change in position of the
quantum from frame to frame. Matching and identifying of quantum is
facilitated by a "grab phase" and an "expansion phase" as depicted
by step 44. Each quantum has a shape. In the "grab phase," active
tiles are sought in a predicted position that is calculated for the
quantum in the new frame, within the shape defined by the quantum.
The predicted position is determined by the previously observed
velocity and direction of travel of the quantum. If any active
tiles are located within the quantum shape region in the predicted
position of the quantum in the new frame, the quantum is considered
found. If no active tiles are located in the quantum shape region
in the predicted position in the new frame, the quantum is
considered lost. In the "expansion phase," active tiles that are
adjacent to a found quantum and that have not been claimed by other
quanta are incorporated into the found quantum, thereby allowing
each quantum to change shape. Unclaimed active tiles are grouped
together to form new quanta unless the number of active tiles is
insufficient to form a quantum. If any of the quanta that have
changed shape now exceed a predetermined maximum size then these
"parent" known quantum are reorganized into a plurality of
"children" quantum. Each child quantum inherits the characteristics
of its parent quantum, such as velocity, acceleration and
direction.
[0041] The identified quanta are organized into objects as depicted
in step 46. The traffic sensor employs a plurality of segmenter
algorithms to organize the identified quanta into objects. Each
segmenter algorithm performs a grouping, dividing or pattern
matching function. For example, a "blob segmenter" groups quanta
that are connected. Some segmenter algorithms facilitate
identification of types of vehicles, such as passenger automobiles
and trucks. Some segmenter algorithms facilitate identification of
vehicle features such as headlights. Some segmenter algorithms
reorganize groups of quanta to facilitate identification of
features.
[0042] Referring to FIG. 6, the segmenter algorithms are employed
in accordance with a dynamic weighting technique to facilitate
operation under changing conditions. Five segmenter algorithms
designated by numbers 1-5 are employed in the illustrative example.
One segmenter algorithm is employed in each time slot. In
particular, the segmenter algorithm in the time slot designated by
an advancing pointer 48 is employed. When a segmenter algorithm
successfully detects and tracks an object that is determined to be
a vehicle by the neural network and is consistent across a
plurality of frames then that segmenter algorithm is granted an
additional time slot. Consequently, the segmenter algorithms that
are more successful under the prevailing conditions are weighted
more heavily than the unsuccessful segmenter algorithms. However,
each segmenter algorithm is assigned at least one permanent time
slot 50 in order to assure that each of the segmenter algorithms
remains active without regard to performance. Hence, operation
dynamically adjusts to changing conditions to maintain optimum
performance. It should be apparent that the number of segmenters,
and number and position of the time slot allocations may be altered
from the illustrative example.
[0043] A list of possible new objects represented by their
component quanta is generated by the segmenters as depicted by step
54. The list of possible new objects is compared with a master list
of objects, and any objects from the list of possible new objects
that cannot be found in the master list is designated as a new
object as depicted by step 56. The object master list is updated by
adding the new objects to the object master list as depicted in
step 57. The objects in the updated object master list are then
classified and scored as depicted in step 58.
[0044] Referring to FIGS. 2 and 7, the objects in the master list
are examined by employing at least one feature extractor 49 as
depicted by step 47. Each feature extractor produces a vector 51 of
predetermined length that describes an aspect of the object, such
as shape. The illustrated feature extractor overlays the object
with a 5.times.5 grid and generates a vector that describes the
shape of the object. Because the number of cells 53 in the grid
does not change, the representative vector 51 is relatively stable
when the size of the object (in number of pixels) changes, such as
when the object approaches or moves away from the camera. The
vector 51 is concatenated with vectors 55 provided from other
feature extractors, if any, to produce a larger vector 57 that
represents the object. Other grid patterns and combinations of
overlays may be used to achieve improved results based upon camera
position relative to the vehicles and other environmental
factors.
[0045] Masking using a vehicle template may be employed to remove
background information prior to feature extraction. The object is
then compared with templates 136 that depict the shape of known
types of vehicles such as cars, vans, trucks etc. When the best fit
match is determined, the center of the object, where the center of
the template is located in the match position, is marked and only
portions of the object that are within the template are employed
for generating the vectors.
[0046] The descriptive vectors generated by the feature extractors
are provided to a neural network classification engine that assigns
a score to each object. The score indicates the probability of the
object being a vehicle, including the type of vehicle, e.g.,
passenger automobile, van, truck. Objects that produce a score that
exceeds a predetermined threshold are determined to be vehicles of
the type indicated. If there are regions of overlap between objects
in the updated object master list, ownership of the quanta in those
regions is resolved in a competition phase as depicted in step 60.
Of the objects in competition for each quantum in the overlap
region, the object that was assigned the highest score by the
neural network obtains ownership of the quanta.
[0047] Physical characteristics relating to object motion, such as
velocity, acceleration, direction of travel and distance between
objects, are calculated in step 62. The calculations are based on
changes in position of a plurality of quanta from frame to frame.
In particular, vehicle velocity may be calculated as the average
velocity of the quanta of the vehicle, by the change in location of
a specific portion of the vehicle such as the center front, or by
other techniques. Similarly, vehicle acceleration may be calculated
as the change in vehicle velocity over time and vehicle direction
may be calculated by extrapolating from direction of travel of
quanta over a plurality of frames. The velocity, acceleration and
direction of travel of the quanta are calculated based on known
length and width dimensions of each pixel and the known period of
time between successive frames.
[0048] Referring to FIG. 8, the computation unit 12 includes at
least one video capture card 66. The video capture card 66 performs
initial processing on video signals received from the camera 10.
The computation unit 12 operates on the output of the video capture
card 66. The functions described with regard to the embodiment
illustrated in FIGS. 8 and 9 are implemented with a custom video
capture card 66. These functions may alternatively be implemented
with a commercially available frame grabber and software. In the
illustrative embodiment the computation unit 12 is a commercially
available IBM compatible computer that employs the Windows 95
operating system. The IBM compatible computer includes a Peripheral
Controller Interconnect ("PCI") bus interface.
[0049] Referring to FIG. 9, the video capture card 66 is operative
to process new video frames, establish and maintain the reference
frame, and compare the new frames with the reference frame in order
to ascertain luminance and/or edge differences therebetween that
are indicative of motion. A digitizer circuit 70 is employed to
convert the analog video signals from the camera. The camera may
provide analog video signals in either National Television
Standards Committee ("NTSC") format or Phase Alteration Line
("PAL") format. The chrominance portion, if any, of the video
signal is separated from the luminance portion of the video signal
by the digitizer circuit 70. The resulting digital signals are
provided to an image state machine 72 where the video signal is
de-interlaced, if necessary. In particular, the video signal is
de-interlaced unless a progressive scan camera is employed. The
output of the image state machine 72 is a succession of
de-interlaced video frames, each frame being 640 pixels by 480
pixels in size. The image state machine is coupled to a Random
Access Memory ("RAM") 74 that includes a ring of three buffers
where frame data is collected prior to transmission of the frames
over a digital bus 76 via pixel fetch circuitry 78.
[0050] Referring to FIGS. 9 and 10, image stabilization is employed
by the video control processor 99 to compensate for camera movement
due to environmental factors such as wind. Up to two anchor
features 69 that are identified by the user during configuration of
the traffic monitoring station are employed. The location of each
anchor 69 on the new frame 18 is determined, and the new frame is
adjusted accordingly. Each anchor 69 is located by matching a
template 162 to the image in the new frame. The template 162 is a
copy of the rectangular region of the reference frame 28 that
includes a representation of the anchor feature 69. A pixel by
pixel comparison is made between the template 162 and a selected
region of the new frame to determine whether a match has been found
based on average luminance difference. The selected region of the
new frame is adjusted until the best match is located (best
match=.sub.min.SIGMA..vert- line.New.sub.x,y-Ref.sub.x,y). The
first comparison may be made at the coordinates at which the anchor
69 is located in the reference frame, or at the coordinates at
which the anchor was located in the previous frame. If a
.sub.min.SIGMA. calculation that is less than or equal to the
.sub.min.SIGMA. calculation in the previous frame is found, the
location is determined to be a match, i.e., the anchor is found. If
a .sub.min.SIGMA. calculation that is less than or equal to the
.sub.min.SIGMA. calculation in the previous frame is not found, the
location of the selected region is adjusted until the best match is
located. The location of the selected region is adjustable within
an area of up to 8 pixels in any direction from the matching
coordinates of the previous frame. The selected region is shifted
in turn both vertically and horizontally by a distance of four
pixels to yield four .sub.min.SIGMA. calculation results. If the
lowest of the four results is lower than the result at the start
point, the selected region is moved to the place that yielded the
lowest result. If none of the results is lower than the result at
the start point, the selected region is shifted in turn both
vertically and horizontally by half the original distance, i.e., by
two pixels, to yield four new .sub.min.SIGMA. calculation results.
If the lowest of the four results is lower than the result at the
start point, the selected region is moved to the place that yielded
the lowest result. The distance may be halved again to one pixel to
yield four new .sub.min.SIGMA. calculation results. When the best
result is found, the anchor is considered found if the result
achieves a predetermined threshold of accuracy. If the best result
fails to achieve the predetermined threshold of accuracy, an edge
comparison is undertaken. The edge comparison is made between the
template and the region that defines the best .sub.min.SIGMA.
calculation results. If at least one vertical edge, at least one
horizontal edge, and at least 75% of all constituent edges are
matched, the anchor is considered found. Otherwise, the anchor is
considered not found.
[0051] The new frame 18 is adjusted to produce a stabilized frame
based upon how many anchors 69 were found, and where the anchors
were found. In the event that both anchors are found and the
anchors were found at the same coordinates as in the reference
frame, the camera did not move and no correction is necessary. If
both anchors moved by the same distance in the same direction, a
two-dimensional X-Y offset vector is calculated. If both anchors
moved in different directions, the camera may have zoomed and/or
rotated. A zoom is indicated when the anchors have moved either
towards or away from the center 164 of the image. For example, the
anchors appear larger and further from the center of the image when
the camera zooms in, and smaller and closer to the center of the
image when the camera zooms out. In the case of a camera that is
zoomed out, the image is "inflated" by periodically duplicating
pixels so that the anchors appear in the expected dimensions. In
the case of a camera that is zoomed in, the image is "deflated" by
periodically discarding pixels so that the anchors appear in the
size that is expected.
[0052] In the event that only one anchor was found, adjustment is
based upon the location of that one anchor. If the anchor did not
move, no correction is applied. If the previous frame was not
zoomed, it is assumed that the new frame is not zoomed. If the
previous frame was zoomed, it is assumed that the new frame is also
zoomed by the same amount. In the event that neither anchor is
found, the corrections that were calculated for the previous frame
are employed.
[0053] From the number of anchors found and their positions, the
video control processor 99 calculates a set of correction factors
as described above and sends them to the pixel fetch circuitry 78.
These correction factors include instructions for shifting the
frame horizontally, vertically, or both to correct for camera pan
and tilt motion, and/or directions for inflating or deflating the
frame to compensate for camera zoom motion. If no correction is
needed, the video control processor calculates a set of correction
factors which instructs the pixel fetch circuitry to do a simple
copy operation. The correction factors allow the pixel fetch
circuitry to select pixels from RAM 74 for transmission on the bus
76 in stabilized order. The pixels are collected into a stabilized
frame 84 for use by the computation unit 12 (FIG. 8).
[0054] Referring to FIG. 9, a differencing unit 82 employs the
contents of the reference frame buffer 80 and the incoming pixels
on the bus 76 to compare the reference frame with the stabilized
frame, pixel by pixel, in order to determine the differences. The
difference values are stored in the difference frame buffer 86. The
computation unit 12 may access the difference frames over the PCI
bus 94.
[0055] A tiling unit 88 is operative to organize the incoming
pixels on bus 76 into tiles 22 (FIG. 3). The tiles are stored in a
tile buffer 90 for use by the computation unit 12 (FIG. 8), which
may access the tiles via the PCI bus 94.
[0056] Referring to FIG. 11, user-defined zones may be employed to
facilitate operation where the view of the camera is partially
obstructed and where sections of roadway converge. An entry zone is
employed to designate an area of the video image in which new
objects may be formed. Objects are not allowed to form outside of
the entry zone. In the illustrated example, an overpass 100
partially obstructs the roadway being monitored. By placing an
entry zone 102 in front of the overpass 100, undesirable detection
and tracking of vehicles travelling on the overpass is avoided. A
second entry zone 104 is defined for a second section of roadway
within the view of the camera. Vehicles entering the roadway
through either entry zone 102 or entry zone 104 are tracked. An
exit zone 106 is employed to designate an area where individual
vehicles are "counted." Because of the perspective of the field of
view of the camera, more distant vehicles appear smaller and closer
together. To reduce the likelihood of multiple vehicles being
counted as a single vehicle, the number of vehicles included in the
vehicle count is determined in the exit zone 106, which is
proximate to the camera.
[0057] Referring now to FIG. 12, a plurality of traffic monitoring
stations 8 may be employed to monitor and share data from multiple
sections of roadway. Information gathered from different sections
of roadway may be shared via a computer network. The gathered
information may be displayed on a graphic user interface 108
located at a separate operations center 110. Video images 112 from
the camera are provided to the graphic user interface 108 through
flow manager software 114. The flow manager maintains near actual
time display of the video image through balance of video smoothness
and delay by controlling buffering of video data and adapting to
available bandwidth. Data resulting from statistical analysis of
the video image is provided to the graphic user interface 108 from
an analysis engine 116 that includes the tiling unit, segmenter
algorithms and neural network described above. The controller card
may be employed to transmit the data through an interface 118 to
the operations center 110, as well as optional transmission to
other traffic monitoring stations. The interface 118 may be shared
memory in the case of a standalone monitoring station/graphic user
interface combination or sockets in the case of an independent
monitoring station and graphic user interface. The operations
center 110 contains an integration tool set for post-processing the
traffic data. The tool set enables presentation of data in both
graphical and spreadsheet formats. The data may also be exported in
different formats for further analysis. The video may also be
displayed with an overlay representing vehicle position and
type.
[0058] Alarm parameters may be defined for the data generated by
the analysis engine 116. For example, an alarm may be set to
trigger if the average velocity of the vehicles passing through the
field of view of the camera drops below a predetermined limit.
Alarm calculations may be done by an alarm engine 122 in the
traffic monitoring station or at the graphic user interface. Alarm
conditions are defined via the graphic user interface.
[0059] Networked traffic monitoring stations may be employed to
identify and track individual vehicles to determine transit time
between stations. The shape of the vehicle represented by active
tiles is employed to distinguish individual vehicles. At a first
traffic control station, a rectangular region ("snippet") that
contains the active tiles that represent a vehicle is obtained as
depicted by step 132. Correction may be made to restore detail
obscured by inter-field distortion as depicted by step 134. The
snippet is then compared with templates 136 that depict the shape
of known types of vehicles such as cars, vans, trucks etc, as
depicted in step 138. When the best fit match is determined, the
center of the snippet, where the center of the template is located
in the match position, is marked as depicted by step 140. Further,
the size of the snippet may be reduced to the size of the matching
template. First and second signatures that respectively represent
image intensity and image edges are calculated from the snippet as
depicted by step 142. The signatures, matching template type,
vehicle speed and a vehicle lane indicator are then transmitted to
a second traffic monitoring station as depicted by step 144. The
second traffic monitoring station enters the information into a
list that is employed for comparison purposes as depicted in step
146. As depicted by step 148, information that represents vehicles
passing the second traffic monitoring station is calculated by
gathering snippets of vehicles and calculating signatures, a lane
indicator, speed and vehicle type in the same manner as described
with respect to the first traffic monitoring station. The
information is then compared with entries selected in step 149 from
the list by employing comparitor 150. In particular, entries that
are so recent that incredibly high speed would be required for the
vehicle to be passing the second traffic monitoring station are not
employed. Further, older entries that would indicate an incredibly
slow travel rate are discarded. The signatures may be accorded
greater weight in the comparison than the lane indicator and
vehicle type. Each comparison yields a score, and the highest score
152 is compared with a predetermined threshold score as depicted by
step 154. If the score does not exceed the threshold, the "match"
is disregarded as depicted by step 156. If the score exceeds the
threshold, the match is saved as depicted by step 158. At the end
of a predetermined interval of time, a ratio is calculated by
dividing the difference between the best score and the second best
score by the best score as depicted by step 160. If the ratio is
greater than or equal to a predetermined value, a vehicle match is
indicated. The transit time and average speed of the vehicle
between traffic monitoring stations is then reported to the graphic
user interface.
[0060] Inter-field distortion is a by-product of standard video
camera scanning technology. An NTSC format video camera will
alternately scan even or odd scan lines every 60th of a second. A
fast moving vehicle will move enough during the scan to "blur,"
seeming to partially appear in two different places at once.
Typically, the car will move about 1.5 ft during the scan (approx.
60 mph). Greater distortion is observed when the car travels at
higher velocity. Greater distortion is also observed when the
vehicle is nearer to the camera. The distortion compensating
algorithm is based on knowledge of the "camera parameters" and the
speed of the vehicle. The camera parameters enable mapping between
motion in the real world and motion in the image plane of the
camera. The algorithm predicts, based on the camera parameters and
the known speed of the vehicle, how much the vehicle has moved in
the real world (in the direction of travel). The movement of the
vehicle on the image plane is then calculated. In particular, the
number of scan lines and distance to the left or right on the image
is calculated. Correction is implemented by moving the odd scan
lines `back` to where the odd scan lines would have been if the car
had stayed still (where the car was when the even scan lines were
acquired). For example, to move 4 scan lines back, scan line n
would be copied back to scan line n-4, where n is any odd scan
line. The right/left movement is simply where the scan line is
positioned when copied back. An offset may be added or subtracted
to move the pixels back into the corrected position. For example,
scan line n may have an offset of 8 pixels when moved back to scan
line n-4, so pixel 0 in scan line n is copied to pixel 7 in scan
line n-4, etc. If the speed of a particular vehicle cannot be
determined, the average speed for that lane may be employed to
attempt the correction. Distortion correction is not necessary when
a progressive scan camera is employed.
[0061] Referring to FIG. 14, the traffic monitoring station may be
employed to facilitate traffic control. In the illustrated
embodiment, the traffic monitoring station is deployed such that an
intersection is within the field of view of the camera. Vehicle
detection can be employed to control traffic light cycles
independently for left and right turns, and non-turning traffic.
Such control, which would require multiple inductive loops, can be
exerted for a plurality of lanes with a single camera.
Predetermined parameters that describe vehicle motion are employed
to anticipate future vehicle motion, and proactive action may be
taken to control traffic in response to the anticipated motion of
the vehicle. For example, if the traffic monitoring station
determines that a vehicle 124 will "run" a red light signal 125 by
traversing an intersection 126 during a period of time when a
traffic signal 128 will be indicating "green" for a vehicle 130
entering the intersection from another direction, the traffic
monitoring station can provide a warning or control such as an
audible warning, flashing light and/or delayed green light for the
other vehicle 130 in order to reduce the likelihood of a collision.
Further, the traffic monitoring station may track the offending
vehicle 124 through the intersection 126 and use the tracking
information to control a separate camera to zoom in on the vehicle
and/or the vehicle license plate to record a single frame, multiple
frames or a full motion video movie of the event for vehicle
identification and evidentiary purposes. The cameras are
coordinated via shared reference features in a field of view
overlap area. Once the second camera acquires the target, the
second camera zooms in to record the license plate of the offending
vehicle. The traffic monitoring station could also be used to
detect other types of violations such as illegal lane changes,
speed violations, and tailgating. Additionally, the traffic
monitoring station can be employed to determine the optimal times
to cycle a traffic light based upon detected gaps in traffic and
lengths of queues of cars at the intersection.
[0062] The determination of whether the vehicle will run the red
light may be based upon the speed of the vehicle and distance of
the vehicle from the intersection. In particular, if the vehicle
speed exceeds a predetermined speed within a predetermined distance
from the intersection it may be inferred that the vehicle cannot or
is unlikely to stop before entering the intersection.
[0063] As described herein, the disclosed system employs video
tracking to detect vehicles that will not stop for a traffic light
that is changing to red. In an illustrative embodiment, the
disclosed system outputs a signal in response to detection of such
Non-Stopping Vehicles (NSV's), for example when they have completed
passage through the intersection and cross traffic can safely
proceed. With this information, a traffic controller can be
optionally programmed to delay the onset of the green light for
cross traffic or pedestrians until any detected red-light violating
vehicles have moved through the intersection. Through its ability
to anticipate red-light vehicle violation before the violation
occurs, the disclosed system may advantageously reduce the risk of
collisions and/or injuries at intersections.
[0064] The features of the presently disclosed system involve the
ability (1) to process a sequence of video images of oncoming
traffic to detect, classify and provide continuous tracking of
detected vehicles; (2) to calculate from image information real
world vehicle displacements with sufficient accuracy to support
measurement of vehicle speed and acceleration for all vehicles in
the camera field of view; (3) to provide continuous, real time
measurement of vehicle position, speed and acceleration; (4) to
develop and implement a decision model, using among its inputs,
measures of vehicle position, speed, acceleration and
classification in order to determine likelihood of a vehicle
stopping; and (5) to update the decision model in real time as a
result of changing values of vehicle parameters (e.g., position,
speed, acceleration) for each oncoming vehicle in the camera's
field of view. The disclosed system supports the use of cameras for
intersection monitoring/control and surveillance. The disclosed
system may further be applied to other safety considerations: the
detection of vehicles approaching toll booths, railway crossings or
other controlled roadway structures (lane reducers, etc.) at
excessive speeds, and/or the detection of vehicles exhibiting other
aspects of hazardous driving behavior.
[0065] The disclosed system includes innovative video-monitoring
capabilities to improve the safety of signalized intersections. A
significant risk exists for motorists who are given a green light
to proceed through an intersection when a vehicle oncoming from
another direction of travel elects to run the red-light for that
direction. Using the disclosed system, Non-Stopping Vehicles
(NSV's) can be detected by a sensor in advance of the onset of the
green light for the cross traffic, and the sensor can generate a
signal that specifically indicates a condition of a vehicle passing
or about to pass through an intersection in violation of a
red-light. This signal can then be used to delay the onset of the
green light phase of a traffic light for any cross traffic until
the NSV has moved through the intersection and it is safe for cross
traffic and/or pedestrians to proceed. In an illustrative
embodiment, such a sensor signal is not necessarily used to
dynamically lengthen, or otherwise change, the timing cycle for the
traffic light of the NSV. The timing cycle in the traffic light for
the NSV may be left unchanged and the NSV does indeed violate a
red-light.
[0066] The capabilities of the disclosed system are based on
monitoring and controlling traffic flow at an intersection through
approach detection, stop line and turn detection functionality. The
disclosed system accurately provides speed and position information
on vehicles within a field of view, and determines from the motion
characteristics of oncoming vehicles and a knowledge of the
intersection timing cycle, whether a vehicle is in the process of
running a red-light or has a high probability of running the
red-light. In one embodiment, this determination starts as soon as
the light for traffic in the given direction of interest cycles to
yellow. The accurate, early detection of an NSV provided by the
disclosed system is essential in order to produce a sensor signal
in advance of the normal cycle for activating the green light of
the cross traffic.
[0067] The disclosed system supports a number of options for
determining an "all clear" signal for an intersection. For example,
in intersections equipped with high slew rate PTZ cameras (>90
deg/sec.), the camera that images the NSV can be controlled by the
disclosed system to track the NSV through the intersection,
generating the all clear signal when the NSV has exited the
intersection. Alternatively, for intersections not equipped with
PTZ cameras, two alternative embodiments may be employed to
determine the "all clear" signal. In a first such alternative
embodiment, no single camera images the entire intersection area,
but the intersection is completely imaged in the field of view of
two or more cameras. In this first alternative embodiment, the
multiple cameras may be used to cooperatively track the target NSV
through the field of view of each camera until the NSV has
traversed the entire intersection. In a second alternative
embodiment, in which the intersection is not imaged either by a
single camera or by combined multiple cameras, a determination is
made, on the basis of the NSV's speed and acceleration as it exits
the camera's field of view, and from predetermined knowledge of the
spatial extent of the intersection that is not imaged either by a
single camera or by multiple cameras, of the time that will elapse
before the vehicle has moved completely through the intersection.
Based on such a determination, the all clear signal can be
generated once this predicted time has elapsed. This second
alternative embodiment requires the fewest resources in terms of
camera installation requirements.
[0068] In an exemplary embodiment, the present invention may, for
example, provide (1) wide area detection of vehicles entering the
approach zone of an intersection, (2) high-precision continuous
speed and acceleration profiles using 20 frame-per-second video
sampling; (3) single camera support for monitoring two approach
directions, (4) continuous tracking of vehicles as they cross lane
boundaries, and (5) cooperative vehicle tracking between multiple
cameras to ensure vehicle clearance through the intersection
area.
[0069] The disclosed system advantageously operates to anticipate
and predict traffic light violations before they occur, for example
using an approaching vehicle's speed. Accordingly, the disclosed
system can be used to intelligently delay a green light until the
offending vehicle is safely past the intersection. Along these same
lines, one or more video cameras may be used by the present system
because they meet key requirements of the intersection safety
application: the need for sensor information on which accurate
measurements of vehicle position, speed, acceleration and
classification can be made; the need for such information on all
vehicles approaching the intersection (not just for those that are
nearest to the intersection in each lane); and the need for
continuously updated information such that the individual vehicle
measurements can be continuously updated in real time.
[0070] Alternatively, other sensor technologies may be used as the
basis for sensor devices in the disclosed system, provided that
they can potentially satisfy the sensor requirements for the
intersection safety application. For example, the more advanced
synthetic aperture radar (SAR) systems may be used to provide a
radar-based image of the roadway. In similar fashion to video
processing, a radar-based image or images can be processed to
detect, track and classify target vehicles on a continuous
basis.
[0071] As a further advantageous feature, the present system
leverages the investment that municipalities make in deploying
video cameras at intersections for surveillance and/or traffic
monitoring and flow control purposes to provide significant safety
benefits as well. Because the disclosed system potentially supports
such dual use of video cameras, significant cost savings result.
Additionally, because of the compatibility of the disclosed system
with surveillance uses of the video cameras, it can be used for
active monitoring in a traffic control center, offering the
potential for integration with larger traffic management
systems.
[0072] Moreover, the features of the disclosed system can provide
intelligent green light delays based upon the detection of
pedestrians that are in the process of crossing an intersection at
a well defined crosswalk. In some cases, late-crossing pedestrians
are not visible to motorists who are waiting for a green light.
This may result from a motorist's view being obstructed by a truck
or bus in an adjacent lane. An embodiment of the disclosed system
could identify pedestrians or pedestrian groups, and communicate
information on expected time to crossing completion that could be
optionally used to delay activation of a green light until such
pedestrians were safely clear of the intersection. The disclosed
system may further be embodied to monitor other potential safety
hazards, such as: vehicles moving at excessive speed approaching
toll booths or other controlled roadway structures (lane reducers,
etc.); vehicles out of control on steep grades, hills or steep
downgrades leading to a traffic light or rail crossing.
Additionally, the disclosed system can use the tracking position
information it generates to control and move a high performance PTZ
camera mechanism and standard NTSC color camera to zoom in on and
identify a potential red-light violator for enforcement purposes.
In such a case, no additional pavement sensors or fixed cameras are
required for red-light violation monitoring.
[0073] The principles of the disclosed system may be applied to any
intersection monitoring system, whether video-based or not, that
meets required minimum operational specifications for the creation
of accurate vehicle count and approach speed measurements on a
real-time basis. Irrespective of the specific sensor technology
employed, controller logic must be employed to interface sensor
outputs to the intersection traffic lights. For example, some
existing controllers can be adapted to implement a green-light
delay based on outputs from the disclosed system.
[0074] Implementations and/or deployments of the disclosed system
could differ in operation from intersection to intersection. In
this regard, an implementation of the disclosed system could be
configured to take into account the different demographics of an
area as well as the nature of its traffic flow patterns. For
example, at rush hour, vehicles often "creep" into an intersection
on a yellow light, even when traffic is backed up, preventing their
transit through the intersection before the onset of a red light in
their direction. In such cases, the disclosed system may be
pre-configured to delay the activation of the cross traffic green
light only for a maximum period of time, in order to give cross
traffic the opportunity to flow, allowing them to navigate around
vehicles present in the intersection but which have not yet
traversed the intersection region.
[0075] Having described the embodiments consistent with the present
invention, other embodiments and variations consistent with the
present invention will be apparent to those skilled in the art.
Therefore, the invention should not be viewed as limited to the
disclosed embodiments but rather should be viewed as limited only
by the spirit and scope of the appended claims.
1 short PTLPivotSolve ( double XOffset, double YOffset, double Lane
Width, double Lanes, double LSlope, double RSlope, double DiffX,
double CameraHeight, double Grade, double Bank, double xPoint,
double yPoint, double RealX, double RealY, double *LambdaSols
double *panSols, double *TiltSols, double *Error, short ArraySize )
{ short SolCount; long x, y; double Xo, Yo, LinearDistance, yP;
double Horizon, New Horizon, dim; double BaseTilt, BaseLambda,
BasePan, PivotX, PivotY, Tilt, Pan, Lambda; CameraParams cp; double
Scaler = 240.0; double PI = 3.1415926535; double RadialDistance;
SolCount = 0; cp.Height = CameraHeight; cp.XSize = 640; cp.YSize =
480; cp.XCenter = 320; cp.YCenter = 240; cp.XOffset = (long)
XOffset; cp.YOffset = (long) YOffset; Grade = atan ( Grade / 100.0
); Bank = atan ( Bank / 100.0 ); cp.Grade = Grade; cp.sinG = sin (
Grade ); Yo = CameraHeight * cp.sinG; cp.cosG = cos ( Grade );
CameraHeight = CameraHeight * cp.cosG; cp.Bank = Bank; cp.sinB =
sin ( Bank ) ; Xo = CameraHeight * cp.sinB; cp.cosB = cos ( Bank );
CameraHeight = CameraHeight * cp.cosB; DiffX /= Scaler; dim = 1.0
/RSlope; dim-= 1.0 /LSlope; dim /= Lanes; Horizon =-DiffX / (Lanes
* dim); SolCount = 0; LinearDistance = sqrt ( ( RealX * RealX ) +
(RealY * RealY ) ); BaseTilt = atan ( CameraHeight /LinearDistance
); if ( _isnan ( BaseTilt ) ) { // Bogus solution return ( 0 ); //
No solutions } cp.st = sin(BaseTilt); cp.ct = cos(BaseTilt); yP =
(240 - yPoint - YOffset) / Scaler; NewHorizon = Horizon - ( yP);
BaseLambda = NewHorizon / tan (BaseTilt ); if (_isnan ( BaseLambda
) ) { return ( 0 ); } cp.Lambda = BaseLambda; RadialDistance=sqrt (
(LinearDistance*LinearDistance) + (CameraHeight *CameraHeight));
BasePan = atan (RealX / RadialDistance); if (_isnan ( BasePan ) ) {
return ( 0 ); } cp.sp = sin(BasePan); cp.cp = cos(BasePan); x = 640
- (long) xPoint; y - 480 - (long) yPoint; GetRealXY ( &cp, x,
y, &PivotX, &PivotY ); // Now get the real, relocated
camera parameters LinearDistance = sqrt (( PivotX * PivotX) + (
PivotY * PivotY)); RadialDistance=sqrt
((LinearDistance*LinearDistance) + (CameraHeight *CameraHeight));
Tilt = atan ( CameraHeight / LinearDistance ); if (_isnan ( Tilt )
) { // Bogus solution return ( 0 ); // No solutions } Lambda =
(Horizon / CameraHeight) *RadialDistance; if (_isnan ( Lambda ) ) {
// Bogus solution return ( 0 ); // No solutions } Pan = asin (
PivotX) / RadialDistance ); if (_isnan ( Pan ) ) { // Bogus
solution return ( 0 ); // No solutions } (*LambdaSols = Lambda;
*PanSols = Pan; *TiltSols = Tilt; *Error = 0.0; SolCount = 1;
return ( SolCount ); }
[0076]
* * * * *