U.S. patent application number 16/812450 was filed with the patent office on 2020-11-12 for multi-sensor target tracking using multiple hypothesis testing.
This patent application is currently assigned to PRINCETON SATELLITE SYSTEMS, INC.. The applicant listed for this patent is PRINCETON SATELLITE SYSTEMS, INC.. Invention is credited to Joseph MUELLER, Michael A. PALUSZEK, Stephanie THOMAS.
Application Number | 20200355782 16/812450 |
Document ID | / |
Family ID | 1000004989278 |
Filed Date | 2020-11-12 |
![](/patent/app/20200355782/US20200355782A1-20201112-D00000.png)
![](/patent/app/20200355782/US20200355782A1-20201112-D00001.png)
![](/patent/app/20200355782/US20200355782A1-20201112-D00002.png)
![](/patent/app/20200355782/US20200355782A1-20201112-D00003.png)
![](/patent/app/20200355782/US20200355782A1-20201112-D00004.png)
![](/patent/app/20200355782/US20200355782A1-20201112-M00001.png)
![](/patent/app/20200355782/US20200355782A1-20201112-M00002.png)
![](/patent/app/20200355782/US20200355782A1-20201112-M00003.png)
![](/patent/app/20200355782/US20200355782A1-20201112-M00004.png)
![](/patent/app/20200355782/US20200355782A1-20201112-M00005.png)
![](/patent/app/20200355782/US20200355782A1-20201112-M00006.png)
View All Diagrams
United States Patent
Application |
20200355782 |
Kind Code |
A1 |
PALUSZEK; Michael A. ; et
al. |
November 12, 2020 |
MULTI-SENSOR TARGET TRACKING USING MULTIPLE HYPOTHESIS TESTING
Abstract
The invention is for a multi-sensor multiple hypotheses testing
tracking system. The multi-hypothesis testing system associates
measurements from multiple sensors with tracks. Measurements are
incorporated using a Kalman Filter and the same filters are used to
propagate the trajectories of the tracks.
Inventors: |
PALUSZEK; Michael A.;
(Princeton, NJ) ; MUELLER; Joseph; (Champlin,
MN) ; THOMAS; Stephanie; (West Windsor, NJ) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
PRINCETON SATELLITE SYSTEMS, INC. |
Plainsboro |
NJ |
US |
|
|
Assignee: |
PRINCETON SATELLITE SYSTEMS,
INC.
Plainsboro
NJ
|
Family ID: |
1000004989278 |
Appl. No.: |
16/812450 |
Filed: |
March 9, 2020 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
16351668 |
Mar 13, 2019 |
10585163 |
|
|
16812450 |
|
|
|
|
14835094 |
Aug 25, 2015 |
10241191 |
|
|
16351668 |
|
|
|
|
62041454 |
Aug 25, 2014 |
|
|
|
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
G01S 11/12 20130101;
G01S 13/726 20130101; G01S 13/931 20130101; G01S 5/0294 20130101;
G01S 15/931 20130101; G01S 5/16 20130101; G01S 5/18 20130101; G01S
15/66 20130101 |
International
Class: |
G01S 5/02 20060101
G01S005/02; G01S 5/16 20060101 G01S005/16; G01S 5/18 20060101
G01S005/18; G01S 13/72 20060101 G01S013/72; G01S 11/12 20060101
G01S011/12; G01S 15/66 20060101 G01S015/66; G01S 15/931 20060101
G01S015/931; G01S 13/931 20060101 G01S013/931 |
Goverment Interests
GOVERNMENT RIGHTS IN INVENTION
[0002] This invention was partly developed under U.S. Air Force
Contracts FA9453-07-M-0110 and FA9453-08-C-0073.
Claims
1. A system for tracking a plurality of moving vehicles that
reduces a number of computations required, the system comprising: a
plurality of sensors that form a plurality of scans by measuring
positions of the plurality of moving vehicles over a plurality of
time intervals, wherein each of the plurality of scans is a
collection of measurements from the plurality of sensors taken in a
particular time interval; and a processor that is communicatively
coupled to the plurality of sensors; wherein the processor:
receives, from the plurality of sensors, a first collection of
measurements of a first scan measured during a first time interval,
wherein each measurement of the first collection of measurements is
a single measurement from one of the plurality of sensors,
generates a first group of tracks using a dynamic state transition
model, wherein the first group of tracks includes a track for each
of the first collection of measurements; forms a first hypothesis
for the first group of tracks, wherein the first hypothesis is a
set of compatible tracks, wherein two or more tracks are compatible
if each track describes a different target and do not share an
identical measurement from any of the plurality of scans; removes
one or more particular tracks from the first group of tracks that
are not included in the first hypothesis to form a first pruned
group by assigning a score to each track group; predicts using a
Kalman filter, expected positions of each of the plurality of
moving vehicles based on the first pruned group; updates the
dynamic state transition model to form an updated dynamic
transition model based on a plurality of covariances calculated for
the first pruned group, receives, from the plurality of sensors, a
second collection of measurements from a second scan during a
second time interval, forms a second hypothesis based on the
updated dynamic transition model, and tracks the plurality of
moving vehicles based on the second hypothesis.
2. The system of claim 1, wherein the plurality of sensors includes
a camera.
3. The system of claim 1, wherein the plurality of sensors includes
a radar device.
4. The system of claim 1, wherein the plurality of sensors includes
a range finder.
5. The system of claim 1, wherein the plurality of sensors is an
acoustic range finder.
6. The system of claim 1, wherein the Kalman filter is an unscented
Kalman filter.
7. The system of claim 1, wherein the Kalman filter is any
combination of a Kalman filter, an unscented Kalman filter, and an
extended Kalman filter.
8. A method for tracking a plurality of moving vehicles that
reduces a required number of computations, the method comprising:
receiving, from a plurality of sensors, a first collection of
measurements of a first scan measured during a first time interval,
wherein the plurality of sensors form a plurality of scans by
measuring positions of the plurality of moving vehicles over a
plurality of time intervals, wherein each of the plurality of scans
is a collection of measurements from the plurality of sensors taken
in a particular time interval; generating a first group of tracks
using a dynamic state transition model, wherein the first group of
tracks includes a track for each of the first collection of
measurements; forming a first hypothesis for the first group of
tracks, wherein the first hypothesis is a set of compatible tracks,
and two or more tracks are compatible if each track describes a
different target and do not share an identical measurement from any
of the plurality of scans; removing one or more particular tracks
from the first group of tracks that are not included in the first
hypothesis to form a first pruned group by assigning a score to
each track; predicting, using a Kalman filter, expected positions
of each of the plurality of moving vehicles based on the first
pruned group; updating the dynamic state transition model to form
an updated dynamic transition model based on a plurality of
covariances calculated for the first pruned group; receiving, from
the plurality of sensors, a second collection of measurements from
a second scan during a second time interval; forming a second
hypothesis based on the updated dynamic transition model; and
tracking the plurality of moving vehicles based on the second
hypothesis.
9. The method of claim 8, wherein the second hypothesis is formed
by appending one of the second collection of measurements to an
existing track in the first pruned group by generating a new track
that includes at least one of the first collection of measurements
associated with the existing track on a condition that the one of
the second collection of measurements describes an object described
by the existing track.
10. The method of claim 8, wherein the second hypothesis is formed
by generating a new track, independent of all of first collection
of measurements, using one of the second collection of measurements
on a condition that the one of the second collection of
measurements does not describe an object described by any existing
track in the first hypothesis.
Description
CROSS REFERENCE TO RELATED APPLICATION
[0001] This application is a continuation of U.S. patent
application Ser. No. 16/351,668 filed Mar. 13, 2019, which is a
continuation of U.S. patent application Ser. No. 14/835,094 filed
on Aug. 25, 2015, now U.S. Pat. No. 10,241,191 issued on Mar. 26,
2019, which claims priority from provisional application U.S.
Patent Application Ser. No. 62/041,454 filed on Aug. 25, 2014.
FIELD OF THE INVENTION
[0003] The present invention relates to the tracking of multiple
targets.
BACKGROUND OF THE INVENTION
[0004] Estimating the position of multiple targets with one or more
sensor requires assignment measurements to tracks
[0005] A track is defined as the unique path of a distinct object.
In situations where there are a multiple objects, such as for a
radar system in a car during rush hour traffic, it is necessary to
associate measurements with objects.
[0006] Track Oriented Multiple Hypothesis Testing is a powerful
technique for assigning measurements to tracks of objects when the
number of objects is unknown or changing. It is absolutely
essential for accurate tracking of multiple objects.
[0007] Tracking is based on the formulation of hypotheses.
Hypotheses are sets of tracks with consistent data, that is, where
no measurements are assigned to more than one track.
[0008] The track-oriented approach recomputes the hypotheses using
the newly updated tracks after each scan of data is received.
Rather than maintaining, and expanding, hypotheses from scan to
scan, the track-oriented approach discards the hypotheses formed on
scan $k-1$. The tracks that survive pruning are propagated to the
next scan k where new tracks are formed, using the new
observations, and reformed into hypotheses. Except for the
necessity to delete some tracks based upon low probability, no
information is lost because the track scores, that are maintained,
contain all the relevant statistical data.
[0009] Hypotheses are sets of tracks with consistent data, that is,
where no measurements are assigned to more than one track. The
track-oriented approach recomputes the hypotheses using the newly
updated tracks after each scan of data is received. Rather than
maintaining, and expanding, hypotheses from scan to scan, the
track-oriented approach discards the hypotheses formed on scan k-1.
The tracks that survive pruning are propagated to the next scan k
where new tracks are formed, using the new observations, and
reformed into hypotheses. Except for the necessity to delete some
tracks based upon low probability, no information is lost because
the track scores, that are maintained, contain all the relevant
statistical data.
[0010] Ding, et. al, U.S. Pat. No. 7,626,535, Dec. 1, 2009, "Track
Quality Based Multi-Target Tracker," discloses a track based
multi-target tracking system. Their method does not allow for
multiple sensors and provides not mechanism for multiple sensors.
This application improves on their patent by providing a rigorous
method for using mulitple sensors and a more robust tracking
algorithm.
[0011] Carson, et. al, U.S. Pat. No. 5,379,044, Jan. 3, 1995,
"Efficient Multi-Target Tracking System," cannot be adapted to
multiple-sensors. In addition, this application improves on its
processing method by using integer programming which is far more
efficient than the disclosed algorithm.
[0012] Miller, U.S. Pat. No. 4,072,943, Feb. 7, 1978, "Multi-Target
Tracking" provides no mechanism for associating measurements with
multi-targets nor can it be added to the disclosed system.
[0013] The cited patents for tracking targets do not solve the
fundamental problem of assigning multiple measurements to multiple
objects when the objects are observed by multiple sensors.
SUMMARY OF THE INVENTION
[0014] The present invention provides a multiple sensor target
tracking system.
[0015] The invention pertains to using track oriented multiple
hypothesis testing with multiple sensors.
BRIEF DESCRIPTION OF THE DRAWINGS
[0016] FIG. 1 shows a diagram of the sensing problem;
[0017] FIG. 2 shows a block diagram of the system;
[0018] FIG. 3 shows a decision tree.
[0019] FIG. 4 shows a flow chart;
DETAILED DESCRIPTION
[0020] In the following description, for purposes of explanation,
specific numbers, materials and configurations are set forth in
order to provide a thorough understanding of the invention. It will
be apparent, however, to one having ordinary skill in the art, that
the invention may be practiced without these specific details. In
some instances, well-known features may be omitted or simplified so
as not to obscure the present invention. Furthermore, reference in
the specification to "one embodiment" or "an embodiment" means that
a particular feature, structure or characteristic described in
connection with the embodiment is included in at least one
embodiment of the invention. The appearances of the phrase "in an
embodiment" in various places in the specification are not
necessarily all referring to the same embodiment.
[0021] During the course of this description like numbers will be
used to identify like elements according to the different views,
which illustrate the invention.
[0022] In this disclosure a Measurement is a single measurement
taken from a single sensor during a scan. A measurement by itself
does not yet imply any association with an object. A Scan is a
collection of measurements taken over one time interval. A scan can
include measurements from multiple sensors. A Track is a sequence
of measurements across multiple scans that are associated with the
same object. A Hypothesis is set of compatible tracks.
[0023] An embodiment of the tracking problem that is the subject of
this invention is shown in FIG. 1. This diagram shows various
elements of the tracking problem.
[0024] A block diagram of the tracking system is shown in FIG.
2.
[0025] Before discussing the method of hypothesis formation, it is
useful to first consider track formation and how tracks are
associated with unique objects.
[0026] New tracks may be formed in one of two ways:
[0027] The new track is based on some existing track, with the
addition of a new measurement
[0028] The new track is NOT based on any existing tracks; it is
based solely on a single new measurement
[0029] Each track is formed as a sequence of measurements across
multiple scans.
[0030] In addition to the raw measurement history, every track also
contains a history of state and covariance data that is computed
from a Kalman filter.
[0031] When a new measurement is appended to an existing track, we
are spawning a new track that includes all of the original track's
measurements, plus this new measurement.
[0032] Therefore, the new track is describing the same object as
the original track.
[0033] A new measurement can also be used to generate a completely
new track that is independent of past measurements.
[0034] When this is done, we are effectively saying that the
measurement does not describe any of the objects that are already
being tracked. It therefore must correspond to a new/different
object.
[0035] In this way, each track is given an object ID to distinguish
which object it describes. Within the context of track-tree
diagrams, all of the tracks inside the same track-tree have the
same object ID. For example, if at some point there are 10 separate
track-trees, this means that 10 separate objects are being tracked
in the MHT system. When a valid hypothesis is formed, it may turn
out that only a few of these objects have compatible tracks.
[0036] In MHT, a valid hypothesis is any compatible set of tracks.
In order for two or more tracks to be compatible, they cannot
describe the same object, and they cannot share the same
measurement at any of the scans. The task in hypothesis formation
is to find one or more combinations of tracks that: 1) are
compatible, and 2) maximize some performance function.
[0037] The hypothesis formation step is formulated as an integer
program (IP). Each track is given an aggregate score that reflects
the component scores attained from each measurement.
[0038] The IP formulation is constructed to select a set of
compatible tracks that add to give the highest score.
[0039] Let N.sub.T be the total number of tracks currently being
maintained, let N.sub.O be the number of different objects
represented by those tracks, and let N.sub.S be the number of
scans. Assuming measurements come from a single sensor, then for
any given scan, each track can have either a) no measurement or b)
a single measurement. Let the score of the measurement for track i
at scan j be denoted as s(i,j). The total score for each track is
then:
S ( i ) = j = 1 N S s ( i , j ) ##EQU00001##
[0040] Each track is a collection of measurements that, together,
represent the believed state evolution of one object. Therefore,
each track has an associated object ID, T(i). Multiple tracks may
share the same object ID. These are considered alternative tracks
on the same track-tree. Each of these tracks represents a different
sequence of measurements that could be used to describe the object.
For two tracks to be compatible within a hypothesis, they must meet
the following criteria:
[0041] Compatible tracks cannot have the same object ID.
[0042] Compatible tracks cannot share the same measurement.
[0043] At each scan j, a set of N.sub.M(j) measurements are taken.
Each of these measurements is assigned a unique index. Let m(i,j)
denote the measurement index for track i at scan j, where
m(i,j).di-elect cons.{1,2, . . . N.sub.M(j)}
[0044] Consider two tracks, i.noteq.i
[0045] For the two tracks to be compatible the object IDs must be
different and the measurements at each scan must be different.
[0046] This can be written as follows:
T ( i ) .noteq. T ( i ) m ( i , j ) .noteq. m ( i , j ) ,
.A-inverted. j = 1 , , N S ##EQU00002##
[0047] Now, let x(i) be a binary decision variable, representing
whether track i is selected or not. Also, let the set of tracks
whose object ID is p be represented as O(p) The object ID
constraints can be written as:
i .di-elect cons. ( p ) x ( i ) .ltoreq. 1 , .A-inverted. p = 1 , ,
N O ##EQU00003##
[0048] This says that there can be no more than one track with the
object ID of p. Applying this constraint for the full list of
object IDs ensures that each object is either a) not included in
the hypothesis, or b) represented by one track.
[0049] Let R(j,k) represent the set of tracks that have measurement
k at scan j.
[0050] The measurement index constraint can be rewritten as:
sum i .di-elect cons. ( j , k ) x ( i ) .ltoreq. 1 , .A-inverted. j
= 1 , , N S .A-inverted. k = 1 , , N M ( j ) ##EQU00004##
[0051] We can now put the constraints and the objective function
together to formulate the IP.
Maximize i = 1 N T S ( i ) x ( i ) Subject to i .di-elect cons. ( k
) x ( i ) .ltoreq. 1 , .A-inverted. k i .di-elect cons. ( j , k ) x
( i ) .ltoreq. 1 , .A-inverted. j = 1 , , N S .A-inverted. k = 1 ,
, N M ( j ) ##EQU00005##
[0052] This formulation can also be extended to solve for multiple
hypotheses, rather than just one.
[0053] In this case, the solution would provide the "M Best"
hypotheses, in descending order of score. This enables tracks to be
preserved from alternate hypotheses that may be very close in score
to the best.
[0054] To solve for a second hypothesis, we introduce a new set of
decision variables, y. This is similar to x but represents an
alternative set of tracks to form an alternative hypothesis. The
same objective function and constraints are applied to y. In
addition, we apply a new constraint which forces the score of $x$
to be higher than the score of $y$. This ensures that the two
solutions are different.
i = 1 N T S ( i ) x ( i ) .gtoreq. i = 1 N T S ( i ) y ( i ) +
.delta. ##EQU00006##
[0055] where .delta.>0 is small number. This constraint must be
applied to each pair of hypotheses.
[0056] The new IP formulation with two hypotheses becomes:
Maximize i = 1 N T S ( i ) x ( i ) + i = 1 N T S ( i ) y ( i )
Subject to i .di-elect cons. ( k ) x ( i ) .ltoreq. 1 ,
.A-inverted. k i .di-elect cons. ( j , k ) x ( i ) .ltoreq. 1 ,
.A-inverted. j = 1 , , N S .A-inverted. k = 1 , , N M ( j ) i
.di-elect cons. ( k ) y ( i ) .ltoreq. 1 , .A-inverted. k i
.di-elect cons. ( j , k ) y ( i ) .ltoreq. 1 , .A-inverted. j = 1 ,
, N S .A-inverted. k = 1 , , N M ( j ) i = 1 N T S ( i ) x ( i )
.gtoreq. i = 1 N T S ( i ) y ( i ) + .delta. ##EQU00007##
[0057] For M hypotheses, there are M sets of the original
constraints, plus M(M-1)/2 new constraints for solution uniqueness.
In general, there may not exist M valid hypotheses. Therefore, the
problem can be formulated and solved for M=1 first, then M=2, and
so on.
[0058] Previously we assumed that all measurements came from a
single sensor. We now consider the case where measurements are
obtained from multiple sensors.
[0059] There are some important differences in how track formation
is carried out when multiple sensors are present.
[0060] With a single sensor, we can add a NULL measurement or a
single real measurement at each scan. With multiple sensors, we can
add a NULL measurement, a single real measurement, or multiple real
measurements at each scan. If multiple measurements are added to a
track for a single scan, they must all come from different
sensors.
[0061] With a single sensor, if multiple measurements in one scan
were within the gate of a track, each spawned a new track with that
measurement appended. Now considering multiple sensors, we must
check the sensor ID of all the measurements in the gate of a track.
If these measurements are from different sensors, then they can all
be appended to the same track. However, if any sensor has multiple
measurements that are in the gate of a track, then new tracks must
be spawned for each of these different combinations of
measurements.
[0062] As before, let N.sub.T be the total number of tracks
currently being maintained, let N.sub.O be the number of different
objects represented by those tracks, and let N.sub.S be the number
of scans.
[0063] The track score is still defined according as shown
above.
[0064] Now let N.sub.Q.gtoreq.1 be the number of sensors. Then for
any given scan, each track can have either a) no measurement or b)
as many as N.sub.Q measurements.
[0065] The following criteria must be met for two tracks to be
compatible:
[0066] Compatible tracks cannot have the same object ID.
[0067] Compatible tracks cannot share the same measurement from the
same sensor.
[0068] The object ID constraints remain the same.
[0069] Previously, we defined R(j,k) as the set of tracks that have
measurement $k$ at scan j. We now introduce multiple sensors. Let
R*(j,k,q) be the set of tracks that have measurement k from sensor
q at scan j.
[0070] The measurement index constraint can now be written as:
i .di-elect cons. * ( j , k , q ) x ( i ) .ltoreq. 1 , .A-inverted.
j = 1 , , N S .A-inverted. k = 1 , , N M ( j ) .A-inverted. q = 1 ,
, N Q ##EQU00008##
[0071] The contents of the set R* for each (j,k,q) triple are a set
of incompatible tracks.
[0072] Track pruning is carried out on an intermittent basis.
Nominally, a set frequency can be used throughout the process--for
example, prune tracks once every 5 scans. Another option is to
conduct the pruning step when the number of tracks exceeds a set
threshold. In any case, the purpose of track pruning is to delete
the tracks that we think are wrong, and keep only those that we
think may be correct.
[0073] We employ a pruning method in which the following tracks are
preserved:
[0074] Tracks with the "N" highest scores
[0075] Tracks that are included in the "M best" hypotheses
[0076] Tracks that have both 1) the object ID and 2) the first "P"
measurements found in the "M best" hypotheses.
[0077] We use the results of hypothesis formation to guide track
pruning. The parameters N, M, P can be tuned to improve
performance. The objective with pruning is to reduce the number of
tracks as much as possible, while not removing any tracks that
should be part of the actual true hypothesis.
[0078] The second item listed above is to preserve all tracks
included in the "M best" hypotheses. Each of these is a full path
through a track-tree, which is clear. The third item listed above
is similar, but less constrained. Consider one of the tracks in the
"M best" hypotheses. We will preserve this full track. In addition,
we will preserve all tracks that stem from scan "P" of this
track.
[0079] FIG. 3 provides an example of which tracks in a track-tree
might be preserved. The diagram shows 17 different tracks over 5
scans. The green track represents one of the tracks found in the
set of "M best" hypotheses, from the hypothesis formation step.
This track would be preserved. The orange tracks all stem from the
node in this track at scan 2. These would be preserved if we set
P=2 from the description above.
[0080] A decision tree is shown in FIG. 3. The figure shows how the
tracks are preserved over multiple scans of measurements
[0081] A flow chart is shown in FIG. 4.
[0082] Multiple sensors generate measurements. Each sensor may
generate multiple measurements.
[0083] An object can only have one measurement per sensor. All
measurements create a new track. The covariances are used to
determine if a measurement should be associated with an existing
track or multiple existing tracks.
[0084] Hypotheses are formed.
[0085] Hypotheses are pruned. Ideally only one hypothesis remains.
Tracks that are not part of a retained hypothesis are removed.
[0086] Measurements are incorporated.
[0087] Tracks are propagated
[0088] Updated covariances are made available.
[0089] Estimated states are output.
[0090] Although the invention herein has been described with
reference to particular embodiments, it is to be understood that
these embodiments are merely illustrative of the principles and
applications of the present invention. It is therefore to be
understood that numerous modifications may be made to the
illustrative embodiments and that other arrangements may be devised
without departing from the spirit and scope of the present
invention as defined by the appended claims.
* * * * *