U.S. patent application number 14/482362 was filed with the patent office on 2014-12-25 for optimizations for live event, real-time, 3d object tracking.
The applicant listed for this patent is James A. Aman, William R. Haller. Invention is credited to James A. Aman, William R. Haller.
Application Number | 20140376874 14/482362 |
Document ID | / |
Family ID | 46206190 |
Filed Date | 2014-12-25 |
United States Patent
Application |
20140376874 |
Kind Code |
A1 |
Aman; James A. ; et
al. |
December 25, 2014 |
OPTIMIZATIONS FOR LIVE EVENT, REAL-TIME, 3D OBJECT TRACKING
Abstract
A system for automatically providing event non-video information
usable for indexing event video information, where the event takes
place at an event location for a duration of event time. The system
includes a non-video information system for receiving or
determining event non-video information and storing the non-video
information in a non-video dataset, a video information system for
receiving event video information captured by one or more filming
cameras and storing the video information in a video dataset, and a
video information retrieval system for randomly accessing the video
dataset, where the random access includes: (a) using event
non-video information to first determine an event time or duration
of event time, and (b) using the determined event time or duration
of event time to selectively access, retrieve, or provide event
video information including one or more video images of the
event.
Inventors: |
Aman; James A.; (Scottsdale,
AZ) ; Haller; William R.; (Bethlehem, PA) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Aman; James A.
Haller; William R. |
Scottsdale
Bethlehem |
AZ
PA |
US
US |
|
|
Family ID: |
46206190 |
Appl. No.: |
14/482362 |
Filed: |
September 10, 2014 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
12287339 |
Oct 8, 2008 |
|
|
|
14482362 |
|
|
|
|
10006444 |
Nov 20, 2001 |
7483049 |
|
|
12287339 |
|
|
|
|
09510922 |
Feb 22, 2000 |
6707487 |
|
|
10006444 |
|
|
|
|
09197219 |
Nov 20, 1998 |
6567116 |
|
|
09510922 |
|
|
|
|
Current U.S.
Class: |
386/223 |
Current CPC
Class: |
G01S 17/87 20130101;
A63B 2024/0025 20130101; A63B 2102/24 20151001; A63B 2220/806
20130101; G06K 9/00711 20130101; A63B 24/0021 20130101; A63B
2220/40 20130101; A63B 2208/12 20130101; A63B 2220/13 20130101;
G11B 27/22 20130101; G01S 17/66 20130101; A63B 71/0605 20130101;
Y10S 707/915 20130101; G06K 2009/3225 20130101; Y10S 707/913
20130101; Y10S 707/914 20130101; G06T 7/246 20170101; G06K 9/3216
20130101; A63B 2220/30 20130101; G11B 27/002 20130101 |
Class at
Publication: |
386/223 |
International
Class: |
G11B 27/22 20060101
G11B027/22; G06K 9/00 20060101 G06K009/00; G11B 27/00 20060101
G11B027/00 |
Claims
1. A system for automatically providing event non-video information
usable for indexing event video information, where the event takes
place at an event location for a duration of event time,
comprising: a non-video information system for receiving or
determining event non-video information, where the event non-video
information relates to event activities taking place at the event
location during the event time, where the event activities comprise
the movements of one or more event persons or event objects, and
where the event non-video information is relatable to the event
time and is stored in a non-video dataset; a video information
system for receiving event video information captured by one or
more filming cameras, where the event video information records at
least some of the event activities taking place at the event
location during the event time, and where the event video
information is relatable to event time and is stored in a video
dataset; and a video information retrieval system for randomly
accessing the video dataset, where the video information retrieval
system has access to both the non-video dataset and the video
dataset, where the random access includes: (a) using event
non-video information to first determine an event time or duration
of event time, and (b) using the determined event time or duration
of event time to selectively access, retrieve, or provide event
video information including one or more video images of the
event.
2. The system of claim 1 where the event non-video information
includes any one of, or any combination of event quantifications
relating to the event or one or more event persons or event
objects, where event quantifications include any one of, or any
combination of object tracking information or event statistics,
where the object tracking information includes any one of, or any
combination of the on-going location, orientation, trajectory,
acceleration or velocity of one or more event persons or event
objects, and where if the event is a sporting event, event persons
are athletes or game officials, event objects are game objects such
as a puck in the sport of ice hockey, and event statistics at least
include any one of, or any combination of: playing time, playing
time per event sub-locations, average speed, time in control of the
game object, number of scoring attempts, turnovers, steals or
passes.
3. The system of claim 2 where one or more identifying items are
affixed to one or more event persons or event objects prior to the
event time, where the identifying items are usable for determining
identifying item information that is non-video information, and
where the event quantifications are determined in part from the
identifying item information.
4. The system of claim 3 where the one or more identifying items
are marks detectable as video data, where the identifying item
marks are represented within tracking video information captured by
one or more tracking cameras, where the tracking video information
is distinct from the event video information, where the tracking
video information is usable for determining the identifying item
information by performing image analysis on the tracking video
information in order to locate and quantify the identifying item
information.
5. The system of claim 4 where the one or more identifying marks
are any one of, or any combination of non-visible or encoded, where
encoded marks include bar codes or similar formats.
6. The system of claim 3 where the object tracking information
determined from the identifying item information includes three
dimensional representations of one or more event persons or event
objects, where the three dimensional representations are used to
generate event animation information, and where the event video
information stored in the video dataset is any combination of the
event video information captured from cameras and the event
animation information generated from the object tracking
information.
7. The system of claim 6 where the video information retrieval
system is further enabled to retrieve any combination of the event
video information, event animation information or event non-video
information either local to the event or remote to the event, where
the time of access is either during the event time or after the
event time, and where the time of access during the event time is
either real-time or post real-time.
8. The system of claim 2 where the event non-video information
comprises in part object tracking information, where the object
tracking information is determined in part by performing image
analysis on tracking video information captured from one or more
tracking cameras, where the tracking video information is distinct
from the event video information and includes event activities
being performed during event time.
9. The system of claim 8 where the tracking video information is
usable as event video information.
10. The system of claim 8 where one or more identifying marks are
affixed to one or more event persons or event objects prior to the
event time, where the identifying marks are usable for determining
identifying mark information that is non-video information, where
the identifying marks are detectable as video data represented
within the tracking video information, where the tracking video
information is usable for determining the identifying mark
information by performing image analysis on the tracking video
information in order to locate and quantify the identifying mark
information, and where the event quantifications are determined in
part from the identifying mark information.
11. The system of claim 1 where the event non-video information
includes pre-event non-video information determined prior to the
duration of event time and live-event non-video information
determined during the duration of event time, where the pre-event
non-video information includes event person group information or
event location information, where the event location information
includes layout information describing the event area where the
event activities take place, where the event is a sporting event
and the person group information includes a list of players and
team associations if the sporting event is a team sport, and where
the live-event non-video information includes any one of, or any
combination of the event quantifications.
12. The system of claim 1 where the non-video information system,
the video information system and the video information retrieval
system operate in real-time such that the random access of the
event video information occurs either in real-time or post
real-time.
13. The system of claim 1 where individuals access and retrieve any
one of, or any combination of the event video information and the
event non-video information either local to the event or remote to
the event, where the time of access is either during the event time
or after the event time, and where the time of access during the
event time is either real-time or post real-time.
14. The system of claim 1 where any one of, or any combination of:
1) the non-video information system, 2) the video information
system, or 3) the video information retrieval system, comprise
either different systems or the same system.
15. The system of claim 2 where at least one of the one or more
filming cameras for capturing event video information are
adjustable filming cameras, where the adjustments include any one
of, or any combination of changes in the pan, tilt or zoom setting
of the adjustable filming camera, where each adjustable filming
camera is further adapted to receive electronic adjustment signals
and to make corresponding adjustments, and where the electronic
adjustment signals are generated in response to the event
quantifications.
16. The system of claim 15 where event quantifications including
object tracking data are used to assign or re-assign any adjustable
filming camera to film any event persons.
17. The system of claim 1 where the event video information system
has access to event location background image information
corresponding to the captured event video information, where the
event video information system compares the captured event video
information to the corresponding background image information in
order to create a minimal set of foreground pixels, and where the
event video information stored in the video dataset is limited
based at least in part upon the determined minimal set of
foreground pixels.
18. The system of claim 17 where the video information retrieval
system has access to the background image information corresponding
to the video dataset of foreground limited event video information,
where the video information retrieval system recreates non-limited
event video information comprising the minimal set of foreground
pixels combined with the corresponding background image
information.
19. The system of claim 17 where the event video information system
analyzes the minimal set of foreground pixels to create a stylized
graphical representation of the event foreground, and where the
event video information stored in the video dataset includes the
stylized graphical representation.
20. The system of claim 17 where the video information retrieval
system presents a graphical representation of the randomly accessed
event video information stored in the video dataset using at least
in part the stylized graphical representations of the event
foreground contained within the video dataset.
21. A system for allowing a user to interactively retrieve event
video information by first randomly accessing event non-video
information, comprising: a data synchronizer operative on a
computer for receiving or accessing event non-video information and
event video information, where the data synchronizer cross-links
the event non-video dataset with the event video dataset using
event time such that the event non-video information is relatable
to the event video information by event time, and a video
information retrieval system operative on a computer for randomly
accessing the event video information using the synchronized
non-video information, where the random access includes: (a) using
event non-video information to first determine an event time or
duration of event time, and (b) using the determined event time or
duration of event time to selectively access, retrieve, or provide
event video information including one or more video images of the
event.
22. The system of claim 21 where the event non-video information
includes any one of, or any combination of event quantifications
relating to the event or one or more event persons or event
objects, where event quantifications include any one of, or any
combination of object tracking information or event statistics,
where the object tracking information includes any one of, or any
combination of the on-going location, orientation, trajectory,
acceleration or velocity of one or more event persons or event
objects, and where if the event is a sporting event, event persons
are athletes or game officials, event objects are game objects such
as a puck in the sport of ice hockey, and event statistics at least
include any one of, or any combination of: playing time, playing
time per event sub-locations, average speed, time in control of the
game object, number of scoring attempts, turnovers, steals or
passes.
23. The system of claim 22 where one or more identifying items are
affixed to one or more event persons or event objects prior to the
event time, where the identifying items are usable for determining
identifying item information that is non-video information, where
the event quantifications are determined in part from the
identifying item information, and where the data synchronizer
additionally receives or accesses the identifying item information
as additional non-video information.
24. The system of claim 22 where the event non-video information
includes pre-event non-video information determined prior to the
duration of event time and live-event non-video information
determined during the duration of event time, where the pre-event
non-video information includes event person group information or
event location information, where the event location information
includes layout information describing the event area where the
event activities take place, where the event is a sporting event
and the person group information includes a list of players and
team associations if the sporting event is a team sport, and where
the live-event non-video information includes any one of, or any
combination of the event quantifications.
25. The system of claim 21 where the data synchronizer and the
video information retrieval system operate in real-time such that
the random access of the event video information occurs either in
real-time or post real-time.
26. A system for creating a non-video dataset in accordance with an
event including event non-video information relatable to event time
for use in providing random access to a video dataset captured in
accordance with the event including event video information
relatable to event time, comprising: a non-video information system
for receiving or determining event non-video information, where the
event non-video information relates to event activities taking
place at an event location during event time, where the event
activities comprise the movements of one or more event persons or
event objects, and a non-video dataset, where the non-video dataset
includes the event non-video information stored in relation to
event time, where the non-video dataset is relatable to a video
dataset stored in relation to event time, where the video dataset
includes event video information, where the event video information
is captured by one or more filming cameras, where the event video
information records at least some of the event activities taking
place at the event location during the event time and is relatable
to event time, where the non-video data is adapted to be used by a
video information retrieval system in order to randomly access the
video dataset, where the video information retrieval system has
access to both the non-video dataset and the video dataset, and
where the random access includes: (a) using event non-video
information to first determine an event time or duration of event
time, and (b) using the determined event time or duration of event
time to selectively access, retrieve, or provide event video
information including one or more video images of the event.
27. The system of claim 26 where the event non-video information
includes any one of, or any combination of event quantifications
relating to the event or one or more event persons or event
objects, where event quantifications include any one of, or any
combination of object tracking information or event statistics,
where the object tracking information includes any one of or any
combination of the on-going location, orientation, trajectory,
acceleration or velocity of one or more event persons or event
objects, and where if the event is a sporting event, event persons
are athletes or game officials, event objects are game objects such
as a puck in the sport of ice hockey, and event statistics at least
include any one of, or any combination of: playing time, playing
time per event sub-locations, average speed, time in control of the
game object, number of scoring attempts, turnovers, steals or
passes.
28. The system of claim 27 where one or more identifying items are
affixed to one or more event persons or event objects prior to the
event time, where the identifying items are usable for determining
identifying item information that is non-video information, where
the event quantifications are determined in part from the
identifying item information, and where the video information
retrieval system additionally receives or accesses the identifying
item information as additional non-video information.
29. The system of claim 26 where the event non-video information
includes pre-event non-video information determined prior to the
duration of event time and live-event non-video information
determined during the duration of event time, where the pre-event
non-video information includes event person group information or
event location information, where the event location information
includes layout information describing the event area where the
event activities take place, where the event is a sporting event
and the person group information includes a list of players and
team associations if the sporting event is a team sport, and where
the live-event non-video information includes any one of, or any
combination of the event quantifications.
30. The system of claim 26 where the non-video information system
and the video information retrieval system operate in real-time
such that the random access of the event video information occurs
either in real-time or post real-time.
Description
RELATED APPLICATIONS
[0001] The present invention is a continuation of U.S. patent
application Ser. No. 12/287,339 filed Oct. 8, 2008 (allowed); which
was a divisional of U.S. patent application Ser. No. 10/006,444
filed Nov. 20, 2001, and issued as U.S. Pat. No. 7,483,049 on Jan.
27, 2009; which was a continuation-in-part of U.S. patent
application Ser. No. 09/510,922 filed on Feb. 22, 2000, and issued
as U.S. Pat. No. 6,707,487 on Mar. 16, 2004; which was a
continuation-in-part of U.S. patent application Ser. No. 09/197,219
filed on Nov. 20, 1998, and issued as U.S. Pat. No. 6,567,116 on
May 20, 2003; with all of the foregoing applications and patents
incorporated in this application by reference.
TECHNICAL FIELD
[0002] The present invention relates to machine vision systems for
tracking the movement of multiple objects within a predefined area
or volume.
BACKGROUND OF THE INVENTION
[0003] Several systems currently exist in the commercial
marketplace for tracking the movement of one or more objects within
a limited area. Such systems are routinely used to follow the
movement of human subjects for a range of purposes including
medical analysis, input for lifelike animation, as well as sports
measurement. The following companies provide machine vision-based
motion analysis systems: [0004] Motion Analysis Corporation and
their HiRES 3D system; [0005] Vicon with their Vicon 250 and 512 3D
motion analysis systems; [0006] Ariel Dynamics, Inc. with their
APAS system; [0007] Charnwood Dynamics with their CODA motion
analysis system; [0008] Peak Performance Inc. with their Maus
system; [0009] Biogesta with their SAGA-3 RT System; [0010] Elite
with their ELITEPlus Motion Analyser System; [0011] Northern
Digital with their Optotrak and Polaris systems, and [0012]
Qualisys with their ProReflex system.
[0013] Each of these systems, which are capable of working in real
time and creating three-dimensional (3D) tracking information,
employ a system of markers to be placed upon the object(s) to be
tracked. The markers themselves are followed by an overlapping
configuration of tracking cameras. As image information is analyzed
from each camera's two-dimensional (2D) view, it is combined to
create the 3D coordinates of each marker as that marker moves about
in the designated tracking volume. Based upon the detected marker
3D locations as well as the pre-known relationship between the
markers and the objects, each system is then able to "re-assemble"
any given object's 3D movement. All of the systems share at least
portions of the following common attributes:
[0014] 1--All of the cameras that are used to view the marker and
therefore object movement are pre-placed in fixed strategic
locations designed to keep the entire tracking volume in view of
two or more cameras.
[0015] 2--Each camera is designed to capture a unique 2D view for a
fixed portion of the tracking volume. The entire set of captured 2D
information is combined by the system to create the 3D information
concerning all markers and therefore objects.
[0016] 3--For a marker to be located in the local (X, Y, Z)
coordinate system during any given instant, it must be visible to
at least two cameras within that instant.
[0017] 4--They use a single tracking energy that is either from the
visible spectrum (such as red light) or infrared (IR).
[0018] 5--They add additional tracking energy in the form of
LED-based ring lights attached to the tracking cameras.
[0019] 6--They place special retroreflective markers on the objects
to be tracked. These retroreflectors reflect a broad spectrum of
energy including visible and IR light.
[0020] 7--The markers do not comprise any special encoding and are
most often identical in size and shape. Typical shapes are a
rectangle, a circle, or a sphere.
[0021] 8--They use the unique positional combination (i.e., the
measured physical space relationship of the markers placed upon the
object) to encode that object's identity. Hence, no two objects can
have the same or a substantially similar positional combination of
markers placed upon them. This "constellation" of markers covers
the majority of the object's surface volume and as such requires
that the entire volume remain substantially in view at all
times.
[0022] 9--They determine, confirm, or both determine and confirm
the identity of each object simultaneously with the tracking of the
objects as they move throughout the entire tracking volume.
[0023] 10--After the cameras have been placed in their fixed
positions, they calibrate the system prior to tracking by moving a
special calibration tool throughout the combined views of all
cameras. The calibration tool consists of two or more markers that
are at a pre-known distance from each other. Once the calibration
has been completed, none of the cameras may be moved before or
during actual object tracking.
[0024] Each of these systems shares many drawbacks, especially when
considered for use in a live sporting environment such as a
sporting contest. Some of these drawbacks are as follows:
[0025] 1--All of the tracking cameras must be set into fixed
positions and then pre-calibrated prior to actual live tracking.
This requirement precludes the use of automatic pan, tilt, and zoom
cameras to collect additional information as directed by the system
in anticipation of marker inclusions.
[0026] 2--Each camera is positioned to have a unique and
substantially perspective view of a given portion of the tracking
volume. Each camera's perspective view contains a significant depth
of field. Any given object traveling throughout this depth of field
will be seen with substantially different resolutions depending
upon whether the object is at the nearest or farthest point with
respect to a given camera. Therefore, the system experiences a
non-uniform resolution per object throughout the entire tracking
volume. This non-uniform resolution affects at least the ease with
which the system may be scaled up to cover larger and larger
tracking volumes using a consistent camera arrangement.
[0027] 3--If the tracking energy is red light, any human observers
will also see the illuminated markers if they are within the narrow
retroreflected cone of light.
[0028] 4--If the tracking energy is red light, then the system is
susceptible to reflections of red light caused by pre-existing
lighting in the visible spectrum that may be reflected from red
colored portions of the tracked objects or the tracking volume
itself.
[0029] 5--In order to reduce unwanted reflections when working with
visible red light, the systems typically cover the objects in
darker material and place black matting on the movement surface to
help reduce unwanted reflections that become system noise. These
techniques are not appropriate for a "live" environment.
[0030] 6--If the tracking energy is IR, then the light sources
employed only emit IR without any additional visible light. The
additional visible light would normally act as an indicator to a
human observer that the light is on, naturally causing them not to
stare for prolonged periods. Continued exposure to any
high-intensity energy including IR light can damage the retina of
the eye.
[0031] 7--When working with IR, these systems do not employ IR
absorbent compounds to be placed upon the objects and tracking
volume background surfaces before any markers are attached as a
means of reducing unwanted reflections that become system
noise.
[0032] 8--Because the retroreflective markers work across a broad
spectrum, they will reflect any visible energy, not just the chosen
emitted tracking energy whether that be red light or IR. As such,
they will for instance retroreflect any pre-existing lighting or
portable lighting such as camera flashes that are typically used by
human observers in a live environment.
[0033] 9--Given that the preferred spherical markers do have an
appreciable size, they are limited in the number of places that
they can be placed upon the objects, especially in a live
environment. Due to their size, they are impractical for use in
live sporting contests especially contact sports such as ice hockey
where they may become dislodged during normal play.
[0034] 10--When the systems are used to track more and more
objects, each with many markers, more and more instances arise when
not all markers are in view of at least two cameras or in some
cases in view of any camera. This is referred to as "inclusions"
and also affects the ability of the system to accurately identify a
given object since its identity is encoded in the unique
"constellation" of the markers placed upon the object, the location
of one or more of which is now unknown.
[0035] 11--When the objects to be tracked are uniformed athletes
such as ice hockey players versus non-uniformed human subjects,
their body sizes and shapes become less distinguishable due to the
standard pad sizes of their equipment and their loose-fitting
jerseys. As body shapes become less distinguishable, then the
unique "constellation" of markers used to identify a given
uniformed athlete becomes less distinguishable and more markers
must be added in order to clearly identify individual players.
[0036] The present inventors have addressed many of these drawbacks
in their co-pending applications entitled: [0037] Multiple Object
Tracking System, application Ser. No. 09/197,219; Filed: Nov. 20,
1998 [0038] Method for Representing Real-Time Motion over the
Internet, application Ser. No. 09/510,922; Filed: Feb. 22, 2000
[0039] Employing Electromagnetic By-Product Radiation for Object
Tracking, application Ser. No. 09/881,430; Filed: Jun. 14, 2001
[0040] Visibly Transparent Wide Observation Angle Retroreflective
Materials, application Ser. No. 09/911,043; Filed: Jul. 23,
2001
[0041] Each of these patent applications is hereby incorporated by
reference into the present application.
[0042] In these patents applications, the present inventors
describe various aspects of a multiple object tracking system that
functions in general to track many types of objects but that is
especially constructed to track athletes during a live sporting
event such as an ice hockey game. These patent applications teach
at least the following novel components:
[0043] 1--The system employs a matrix of separate overhead tracking
cameras responsible for first locating any given object as a whole
in a local (X, Y) area rather than in a (X, Y, Z) volume coordinate
system. This technique yields a substantially uniform pixel
resolution per area tracked providing a simple and regular approach
to camera arrangement when the system is scaled to track larger
areas.
[0044] 2--The system employs separate sets of one or more pan,
tilt, and zoom cameras per player to be tracked. These moveable
cameras are automatically directed by the system based upon the (X,
Y) location information that was first determined using the
overhead tracking "area" cameras. Each of these volume cameras will
collect (X, Y, Z) information from a particular view of the player
to be combined with at least the (X, Y) information captured by the
"area" cameras concerning the same player. Due to the system's
ability to move and zoom each player-tracking camera, a
substantially uniform pixel resolution per player is achieved. This
technique provides a simple and regular approach to camera
arrangement when the system is scaled to track more and more
players.
[0045] 3--The system preferably employs a non-visible tracking
energy such as ultraviolet or infrared that is currently being
generated by pre-existing lighting within the tracking area.
[0046] 4--By using pre-existing lighting that is already in place
with a purpose of illuminating the playing area for human
observers, the system ensures that the observers will have a
visible light indicator that the lamp is on. This will naturally
keep the observer from staring at the lights and receiving an
overexposure of non-visible tracking energy.
[0047] 5--The system employs one or more reflective,
retroreflective, fluorescent, and fluorescent retroreflective
materials that are specifically designed to reflect only the chosen
non-visible ultraviolet or infrared tracking energy and to be
substantially transparent to visible light.
[0048] 6--The system preferably employs markers that are made of
ink which has minimal thickness and can be placed upon virtually
any surface such as in the case of hockey a player's helmet,
jersey, or gloves; the tape they use to wrap their stick; or the
puck.
[0049] 7--The system preferably encodes the player's unique
identity into the markings placed exclusively upon the "top
surface" of the player, such as the helmet or shoulders. In so
doing, the player's identity can be determined solely from the (X,
Y) area tracking cameras and is substantially unaffected by player
"bunching" and subsequent body marker inclusions that primarily
affect the view of the body below the helmet and shoulders.
[0050] 8--The system preferably takes advantage of the reduced
player movement and smaller area of the playing surface entrance
and exit as well as the team benches in order to perform player
identification. The unique characteristics of the entrance and exit
and benches provides the opportunity to focus the overhead (X, Y)
tracking cameras in a considerably smaller field-of-view such that
the players' helmets and attached markings are considerably
enlarged with respect to the entire captured image. This in turn
ensures that the space available for a marking on the helmet is
sufficient to completely encode and therefore identify a given
player through the use of more complex symbol patterns similar to
bar codes. As previously mentioned, since the unique player code is
therefore fully contained on the helmet, only the overhead (X, Y)
cameras are necessary to determine identity thereby eliminating the
effect of body marker inclusions caused when the (X, Y, Z) cameras'
fields-of-view are blocked.
[0051] 9--By separating entrance and exit and bench tracking and
identification from playing surface tracking, it is possible to
place a multi-frequency responsive marker at least on the player's
helmet. For instance, the complex symbol patterns used to encode
the player's identity can be created with an UV ink while the
helmet tracking mark can be created with an IR ink, or vice versa.
This switching of frequencies effectively doubles the available
marking area of at least the helmet and potentially any other "top
surface" such as the shoulders.
[0052] 10--The area cameras have mutually exclusive fields-of-view
with slight edge-to-edge overlap for calibration purposes. This
calibration process is performed prior to live tracking.
[0053] 11--The volume cameras are first calibrated with respect to
their pan, tilt, and zoom drive mechanisms also prior to actual
tracking. Their field-of-view will constantly overlap one or more
area cameras. The combination of this overlapping area and volume
information is then used by the system for dynamic re-calibration
and adjustment of the volume cameras. The system thereby permits
individual cameras to move and be recalibrated simultaneously with
actual tracking.
[0054] 12--The system employs absorber compounds that are to be
placed upon the objects and playing surface prior to placing the
markers in order to cut down or eliminate unwanted reflections that
are system noise.
[0055] 13--The system employs predictive techniques based upon the
object's last known position, acceleration, velocity, and direction
of travel to minimize the search time required to locate the object
in subsequent video frames.
[0056] Also currently existing in the commercial marketplace are
the following important components:
[0057] 1--Wide-angle retroreflectors are capable of reflecting
light in a wider cone than typical retroreflectors. These are
available in both cube cornered and microscopic bead optical body
formats. They provide the opportunity to move the lighting source
further away from the tracking cameras when using retroreflective
materials.
[0058] 2--Fluorescent laser dyes are capable of absorbing visible
light just below the 700 nm wavelengths that are still visible and
converting it into IR light just above the 700 nm region that is
non-visible. When using IR, these dyes provide the opportunity to
convert visible energy as emitted by pre-existing arena lighting
into IR tracking energy thus eliminating the need to add lighting
that specifically radiates IR into the tracking volume.
[0059] 3--Fluorescent laser dyes are capable of absorbing UV light
around 330 nm wavelengths and converting it into UV light around
390 nm. This conversion is important given that certain
commercially available low-cost digital imaging cameras are
designed to have a higher responsivity to UV light especially
around the 390 nm range. Furthermore, existing arena lighting such
as Metal Halide Lamps currently generate UV energy in the frequency
range of 315 to 400 nm. By absorbing the shorter wavelength UV
energy around 330 nm and then radiating additional UV energy around
390 nm, the fluorescent dyes will essentially "double up" on the
preferred narrow band of tracking frequencies.
[0060] 4--Notch filters may be used with the tracking cameras and
are capable of passing very narrow bands of specific frequencies of
energy. This provides the opportunity to place reflective,
fluorescent, or retroreflective materials that operate at different
frequency ranges onto different players to assist in the
identification process.
[0061] The present inventors have described in their co-pending
applications many useful component and system solutions to the
problems that are inherent within the existing systems. Additional
components as described above also exist. It is possible to create
several different and yet effective machine vision systems for
tracking multiple moving objects based upon the novel components
disclosed within the present application and four co-pending
applications. What is needed is an understanding of how all of
these teachings can be combined to form several different machine
vision systems, each with their own novel optimizations.
[0062] In addition to the aforementioned machine vision system
solutions to multi-object tracking it should be noted that at least
two other companies are attempting to provide systems for similar
purposes. Both Trackus, a Massachusetts-based company, and Orad, an
Israeli-based company, are attempting to develop real-time
"beacon"--based tracking technology for sporting events. Orad has
produced a working system to follow horse racing; Trakus is the
only company currently attempting to follow players in an ice
hockey game. While Orad's solution is essentially similar at the
highest levels, the technology will be explained based upon
information gathered concerning Trakus.
[0063] Trakus' solution includes a microwave based transmitter and
receiver that will track a single point within the helmet of each
player. There are many deficiencies with this proposed solution as
compared to machine vision systems in general and the novel
teachings of the present inventors in particular. One of the most
important distinctions is the "active" and potential harmful nature
of the microwave technology. If used for tracking youth sports, it
is anticipated that the average parent would balk at the idea of
placing even a low-power microwave device into the helmet of their
child. Furthermore, there are significant reflection problems due
to the hard interior surfaces of a hockey arena that must be
resolved before this technology can effectively track even a single
point (the helmet) on every player on both teams. As already
discussed, machine vision-based systems employ "passive" markers
that are capable of tracking 14 or more points (the head and every
major joint) on every player in real time. The present invention
furthermore uniquely teaches a system that can also track game
equipment and the puck, devices that have surfaces that cannot be
substantially altered by the normal size of traditional
markers.
SUMMARY OF THE INVENTION
[0064] For the purposes of disclosing the novel teachings of the
present invention, the exemplary application of following the
motion of hockey players, their equipment, and the game puck in a
live sporting event will be used to represent multi-object
tracking.
[0065] In order to create an optimal multi-object tracking system,
the present inventors have focused on four major factors as
follows:
[0066] A. The desired tracking information to be determined by the
system;
[0067] B. The characteristics of the objects to be tracked;
[0068] C. The characteristics of the tracking environment; as well
as
[0069] D. Traditional engineering goals.
[0070] A. With respect to the desired tracking information to be
determined by the system, the following characteristics were
considered:
[0071] 1. Is the desired representation to be visual for display
only or a mathematical model for measurement and rendering?;
[0072] 2. Is two or three-dimensional information preferred?;
[0073] 3. Is object orientation required in addition to location,
velocity, and acceleration?;
[0074] 4. Must this information be collected and available in real
time?
[0075] 5. What is the acceptable accuracy and precision with
respect to this information?;
[0076] 6. Will the system be required to identify the objects as
well as tracking them?; and
[0077] 7. Once an object has been identified, can this identity be
lost during tracking?
[0078] B. With respect to the objects to be tracked, the following
characteristics were considered:
[0079] 1. Is the object rigid or flexible?;
[0080] 2. In how many degrees of freedom will the object be
moving?;
[0081] 3. How fast will the objects be moving?;
[0082] 4. Are there multiple objects to be tracked and, if so, how
will this impact the tracking method?;
[0083] 5. Are there any restrictions or safety considerations
regarding the type of electromagnetic energy used to track the
objects?;
[0084] 6. What are the physical space limitations on the objects
for any marker- or beacon-based tracking system?; and
[0085] 7. What is the natural reflectivity of the various
background surfaces to the different potential tracking
energies?
[0086] C. With respect to the tracking environment, the following
characteristics were considered:
[0087] 1. Is the setting "live" or "controlled"?;
[0088] 2. What are the existing ambient electromagnetic
energies?;
[0089] 3. Are there any other pre-existing energy sources that may
have available by-product energy that could be used for
tracking?;
[0090] 4. What is the natural reflectivity of the various
background surfaces to the different potential tracking
energies?;
[0091] 5. What is the size of the tracking area relative to the
range of the potential tracking methods?; and
[0092] 6. Is the tracking environment physically enclosed within a
building or outside?
[0093] D. With respect to traditional engineering goals, the
following characteristics were considered:
[0094] 1. The system should be scalable and therefore comprise
uniform assemblies that can be combined into a matrix designed to
increase tracking coverage in terms of area, volume, and the number
of objects while still maintaining uniform performance.
[0095] 2. The system should be minimally intrusive upon the objects
to be tracked and upon the surrounding environment, especially if
that environment is a live setting;
[0096] 3. The tracking signal-to-noise ratio should be maximized;
and
[0097] 4. Manufacturing and installation costs should be minimized
and the resultant system should be simple for the user to maintain
and operate.
[0098] An optimized system design such as disclosed in the present
application must consider many of the above-stated questions and
criteria simultaneously. However, for the sake of consistency, each
of the pertinent questions and criteria listed above will be
considered in order.
[0099] Therefore, referring first to the characteristics of the
desired tracking information to be determined by the system, the
following is taught.
[0100] A.1. Is the desired representation to be visual for display
only or a mathematical model for measurement and rendering?
[0101] It is preferable that the tracking system creates a
mathematical model of the tracked players and equipment as opposed
to a visual representation that is essentially similar to a
traditional filming and broadcast system. A mathematical model
allows for the measurement of the athletic competition while also
providing the basis for a graphical rendering of the captured
movement for visual inspection from any desired viewpoint. Certain
systems exist in the marketplace that attempt to film sporting
contests for multiple viewpoints after which a computer system may
be used to rotate through the various overlapping views giving a
limited ability to see the contest from any perspective. All of the
aforementioned machine vision companies, such as Motion Analysis
and Vicon, generate a mathematical model.
[0102] This requirement of creating a mathematical model of human
movement necessarily dictates that at least one precise location on
the human body be identified and followed. For beacon-based systems
such as Trakus or Orad, following the beacon's signal provides both
identity and object location. To be implemented in a machine vision
system, this further implies that each of these locations remain
substantially in view of two or more tracking cameras at all times.
Furthermore, the use of markers strategically placed upon the
players can greatly simplify video frame analysis, as the markers
become consistent center points that reduce the need for detailed
edge detection techniques as well as identification and weighted
averaging of player surfaces. This marker technique is implemented
by all of the vision-based systems such as Motion Analysis and
Vicon and is preferred by the present inventors. As will be
discussed, the exact choice of the type, shape, size, and placement
of the preferred markers is significant to some of the novel
functions of the present invention and is different from existing
techniques.
[0103] A.2. Is two or three-dimensional information preferred?
[0104] Three-dimensional information provides the ability to
generate more realistic graphical renditions and to create more
detailed statistics and analyses concerning game play. All of the
aforementioned machine vision companies such as Motion Analysis and
Vicon attempt to generate three-dimensional data while the
beacon-based Trakus and Orad only generate two-dimensional
information.
[0105] This requirement of creating a three-dimensional
mathematical model of game play further dictates that at least the
major joints on a player are identified and tracked. For instance,
the player's helmet, shoulders, elbows, writs, torso, waist, knees,
and feet are all beneficial tracking points for a 3D model. This
informational goal in practice precludes beacon-based systems such
as Trakus and Orad since it would require a significant number of
beacons to be placed onto each player, often times in locations
that are not convenient. Furthermore, each beacon will create
additional signal processing and, given current state of the art in
microwave tracking, the system could not function quickly enough to
resolve all of the incoming signals.
[0106] For vision-based systems such as Motion Analysis and Vicon,
significant difficulties also begin to present themselves in
consideration of the requirement that each joint be in view of at
least two cameras at all times. As players move about and change
their body positions, individual joints can easily be lost from
view (inclusion) or take positions that from a given camera's "flat
2D" perspective appear to make them a part of a different player.
This is especially true in light of the small spherical markers
used by existing systems that are not in view from every possible
rotational angle of the joint. What is needed is a vision-based
tracking system that can identify and track players with one set of
cameras and then automatically direct a second set of cameras to
adjust their views so as to minimize these inclusions. What is
further needed is a tracking system that employs markers that can
cover a much larger area, for instance all the way around an elbow
rather than a single point or set of three points upon the elbow,
while at the same time remaining less obtrusive.
[0107] A.3. Is object orientation required in addition to location,
velocity, and acceleration?
[0108] Especially in the sport of ice hockey, it is important to
know the orientation of the player as a whole since, for instance,
they can be skating in a certain direction either facing forwards
or backwards. Furthermore, in any motion system designed to follow
the movement of a human joint that has multiple degrees of freedom,
it is necessary to determine the orientation of the joint, not
merely its position in order to create an accurate mathematical
model. This requirement exceeds the capacity of beacon-based
systems such as Trakus and Orad since the emitted signals are
uniformly omni-directional and therefore cannot be used to
determine rotation about the transmitting axis. For vision-based
systems such as Motion Analysis, Vicon and the present invention,
this requirement significantly favors the use of markers over edge
detection of player surfaces. This is because the markers have
clear center points and could be placed in a triangular format that
is a traditional method for orientation detection. The present
inventors prefer larger markings that form shapes such as an oval,
circle, or triangle over the placement of spherical markers as is
currently practiced.
[0109] A.4. Must this information be collected and available in
real time?
[0110] The ability to capture and analyze images, convert them into
a 3-D mathematical model, and then dynamically render a graphic
representation along with quantified statistics in real time offers
significant opportunities and challenges. For beacon-based systems
such as Trakus and Orad, there is a single set of receiving towers
throughout the arena that must process as many beacons as required
to follow all of the players to be tracked. Essentially, the entire
set of towers is necessary to cover the rink for even a single
player. Additionally, each player's beacon signal will be picked up
by each tower and then must be compared across all towers to
perform the location function. As more players are added, they will
create additional processing for each tower. Hence, the total
number of players is limited by the capacity of a single tower to
process its received signals in real time. While such beacon-based
systems may be scalable by playing area, therefore simply adding
additional towers will cover additional area, they are not scalable
by the number of players. Hence, for more players within a given
area, one cannot simply add more towers. These systems are
inherently "player bound" as opposed to "area bound."
[0111] For vision-based systems such as Motion Analysis and Vicon,
the goal of real-time information significantly challenges system
capacity especially when combined with the need to track multiple
players. This requirement limits the amount of image processing
time available to handle ambiguities created in the data set by the
inclusions that occur when players overlap, or bunch up, within a
given camera view. Similar to the beacon-based systems, these
camera systems can be said to be scalable by playing area since
each additional playing volume requires the uniform addition of
fixed perspective cameras. However, they too are "player bound"
since any number of players may at a single time end up in any
given playing volume creating a large, "included" data set that
cannot be sufficiently processed in real time if at all without
human assistance to clarify ambiguities.
[0112] The present inventors prefer to separate the function of
"area" tracking for 2D movements and identification from "player"
tracking for the full 3D data set. Hence, a matrix of overhead (X,
Y) cameras follows the movement of any number of players per single
camera across a single playing area. Since the number of markers
tracked from the overhead view is limited to primarily the helmet
and shoulders, the total number of marks, even for a large number
of players is still trackable. Furthermore, due to the overhead
view and the tendency for players to remain upright, it is expected
that there will be minimal instances of helmet or shoulder
"overhead inclusions." In addition to being tracked in 2D space by
the overhead cameras, each player will also be followed by at least
two and preferably four dedicated pan, tilt, and zoom perspective
cameras. These dedicated, movable, perspective cameras will be
automatically directed by the known (X, Y) location of each player
as determined by the overhead cameras. This combination of novel
techniques provides for scalability by both tracking area and
player. Hence, to cover more area simply add one or more overhead
"area" tracking cameras while to cover more players simply add
additional sets of one or more dedicated movable "player" tracking
cameras.
[0113] A.5. What is the acceptable accuracy and precision with
respect to this information?
[0114] The speed of a hockey puck can approach 100 mph while a
player may be skating upwards of 25 mph. As was previously
described in detail in the present inventors' co-pending
application for a Multiple Object Tracking System, minimum capture
rates of 40 frames per second for low-end commercial video cameras
will follow pucks at a maximum of 3.7 feet per frame and players at
0.9 feet per frame. While this is sufficient to create a realistic
mathematical model of game movement, faster cameras with between 2
to 4 times the capture rates are readily available. The
incorporation of faster cameras also requires the use of faster
computers and software algorithms to locate and identify markers
and therefore players and equipment within the allotted fraction of
a second. Currently, the beacon-based Trakus captures data at 30
locations per second.
[0115] A.6. Will the system be required to identify the objects as
well as tracking them?
[0116] The ability to identify the tracked objects is critical and
poses a more difficult problem for the video-based systems versus
the beacon systems. With a beacon system, the transmitted signal
can easily contain a uniquely encoded value that serves to
differentiate each player. For a video-based system, which uses a
single tracking energy and passive markers, the encoding must be
accomplished via some unique arrangement of markings. For vision
based systems such as Motion Analysis and Vicon, the current
practice is to consider the unique "constellation" of markers as
placed upon each players body to form an encoding for that player.
This thinking is predicated on the idea that no two individuals
have exactly the same body shape and hence the markers will always
be in at least slightly different configurations. If two players
did end up with "constellations" that were too similar in
configuration to be accurately differentiated, then one or more
additional markers would be added to at least one of the players to
sufficiently differentiate the one from the other. It should be
noted, that with this strategy, the majority of the markers play a
dual role of both joint and body segment tracking on an individual
basis and player identity on a collective basis.
[0117] A further practical limitation of this technique is its
real-life implementation with non-sophisticated system operators.
In other words, at the most difficult youth sport level, the ideal
tracking system must be operable by minimally trained lay people
such as coaches and parents. Placing these markers upon the
player's body in such a way as to guarantee a unique constellation
for all 16 to 20 players on a team will be overly restrictive. What
is needed is a simple approach to placing a minimum of a single
marker upon the player that is assured to provide a unique
identification during game tracking.
[0118] The present inventors prefer to isolate player
identification to those markings that can be viewed strictly from
the overhead cameras, i.e., the "top surface" of the player's body.
This minimizes the occasions when the markers that are relied upon
for identification are hidden (included) from camera view based
upon player bunching. In the preferred embodiment, the markings
placed on the player's helmet and possibly also on their shoulders
form a uniquely identifiable pattern, similar in form to bar
coding. The decision as to whether the shoulder markers need to be
included in the unique pattern depends mainly upon the total area
provided by the helmet alone and the number of pixels per inch
resolution of the viewing cameras. For effective machine vision,
any mark to be "seen" must be picked up by preferably two pixels.
Depending upon the pixel per inch resolution of the camera
configuration, these two pixels will represent a different sized
area. In practice, due to the expected movement and rotation of the
helmet that is inherently not a flat surface, it is preferred to
have many more pixels per inch of marking for accurate recognition.
Additionally, it is anticipated that there may be between thirty to
forty players and game officials on the ice over the course of a
given game. Each one of these players and officials must have a
unique tracking pattern on their helmet. Hence, if there is not
enough room on the helmet to include an easily recognizable, unique
encoding for each player, the shoulders can be used to augment the
coding scheme.
[0119] This novel approach provides the opportunity to optionally
separate the player identification process from the tracking
process. Since the players are all expected to enter and exit the
playing area via a limited passageway and furthermore to collect
just outside the playing area on team benches, it is anticipated
that these will present ideal areas to first ascertain and then
reconfirm unique player identities. Once the players are identified
as they first enter the playing area through the passageway, they
can then simply be tracked as they move throughout the playing
area. Should an ambiguity arise between two players during game
play, their identity can be sorted out once they return to the team
benches.
[0120] This separation of player identification from movement
tracking opens up the possibility of different camera and lens
configurations in the "identification areas" versus the "tracking
areas." Hence, the identification cameras could for instance be
focused on a much narrower field of view since the passageway and
team bench are considerably smaller in area than the playing
surface. By narrowing the field of view, the resultant pixels per
square inch of "id pattern" will be increased thereby reducing the
necessary size of this unique marking. Other possibilities are
anticipated such as the creation of a unique marking that is
detected in a first select frequency such as IR and an overlapping
tracking mark that is detected in a second select frequency such as
UR. (These frequencies could easily be reversed with no change to
the novel functionality.) In this embodiment, the "identification
areas" will be viewed with IR cameras while the "tracking areas"
will be viewed with UV cameras. It should be noted that both the IR
id mark and the UV tracking mark would be non-visible to the
players and viewing audience. Furthermore, these separate
"identification areas" will also facilitate tracking the limited
movement of the players until they enter the playing surface, i.e.
"tracking area."
[0121] A.7. Once an object has been identified, can this identity
be lost during tracking?
[0122] The answer to this question is entirely driven by the
selected tracking and identification technique. For instance, with
beacon-based identification such as Trakus and Orad, the identity
is only lost if the unique player signal is lost. For traditional
vision-based tracking such as Motion Analysis and Vicon, player
identity can be easily lost based upon a number of factors. These
factors include the number of cameras viewing a given playing
volume, the pixel per inch resolution for the allowed marker size
as well as the number of expected inclusions due to player
bunching. The final factor is randomly variable throughout a given
tracking session. The preferred embodiment of the present invention
has separated the identification of each player into a 2D overhead
viewing function that strictly focuses on the "top surface" of each
player. Since the number of inclusions of player's helmets is
expected to be minimal, especially from the overhead view is
expected to be minimal, the preferred embodiment will not
experience the same difficulties in maintaining player identity as
traditional systems.
[0123] B. Referring next to the characteristics of the objects to
be tracked, the preferred and alternate novel embodiments must
consider at least the following factors:
[0124] B.1. Is the object rigid or flexible?
[0125] The majority of objects (i.e., the players) are flexible
while some objects are rigid such as the sticks and puck. Flexible
objects such as players create difficulties for vision-based
tracking systems because their form is constantly changing. These
changes create greater opportunities for inclusions (blocked
markers) and for misinterpretation when multiple players wearing
multiple markers are each partially in view from any given camera.
For beacon-based systems such as Trakus and Orad, the only location
being tracked on the player is the helmet that is itself rigid. Due
to the nature of the beacon itself, it is immaterial whether the
object is flexible since the beacon's signal will transmit through
the player's body and most equipment. For machine vision systems
such as Motion Analysis and Vicon, this restriction significantly
adds to the difficulty of keeping any given marker in the view of
two or more cameras at a single instant, especially in light of the
multiple player requirements.
[0126] Due in part to the infinite variations of marker images that
can be created by any number of players holding their bodies in any
number of positions creating any number of inclusions, all within a
single camera field of view, the present inventors favor the novel
teachings of the present invention. More specifically, by isolating
player identification to markings placed upon the player's helmet
that is rigid and substantially in view of the overhead cameras at
all times, the flexibility of the player's body is removed as a
negative factor for the more complex purpose of identification.
Additionally, the use of larger ink markings provides for various
marker shapes such as circles around the elbows, torso, knees or
ankles or strips that can run down a player's arms, legs, torso or
stick, etc. This in turn allows the present invention to greatly
reduce the chances for total inclusion as the player's flex
individually and bunch together. And finally, the separation of
overhead (X, Y) "area" tracking that automatically directs the
dedicated pan, tilt and zoom "player" tracking cameras, which in
turn collect the 3D information, further minimizes the inclusions
created by multiple flexible objects.
[0127] B.2. In how many degrees of freedom will the object be
moving?
[0128] Similar to the difficulties created by the flexibility of a
player, the full six degrees of freedom within which a player and
or their limbs may travel creates increased challenges for
vision-based solutions such as Motion Analysis and Vicon. It should
first be noted that beacon-based systems such as Trakus and Orad
are not impacted by the freedom of the player to move their helmet
in any of the six possible directions from the current location. Of
course, the beacon solution itself has a much greater shortcoming
since it is in practice limited to a single beacon per player.
[0129] Returning to the vision systems such as Motion Analysis and
Vicon, the current practice is to place small spheres ranging in
size from approximately 1/8'' to 1'' in diameter upon the various
joints and locations to be tracked on any given player. Often,
these small spheres are themselves held out away from the players
body on a short stem of approximately 1/2''. This configuration is
uniformly adopted by the current 3D vision-based tracking systems
and represents a significant system limitation. The purpose of the
spherical shape of the marker is to ensure that a maximum image
size is created irrespective of the location of the marker (and its
associated body part) with respect to the viewing camera. Hence, no
matter what the angle of view, the marker will always show up as a
circle unless the view is in some way blocked. The purpose of the
stem is to hold the sphere out away from the body thereby reducing
the circumstances of partial inclusion created by the body surface
to which the marker has been attached. While this technique has
worked well for "controlled" situations, these markers are
unacceptable for "live" events.
[0130] What is needed is a method of marking the player's joints
that will be substantially visible from two or more cameras from
any view point despite the six degree freedom of player movement.
Given this requirement, the present inventors prefer the novel
technique of using a much larger "surface area" marking. This
essentially maximizes the marker view by both increasing the size
of the mark as well as continuously surrounding a joint or body
part such as the elbows, torso, knees or ankles. By both enlarging
the mark and surrounding the body part, the ability for that part
to been seen from any angle while it moves in six degrees of
freedom is significantly increased.
[0131] B.3. How fast will the objects be moving?
[0132] As was previously stated, the speed of a hockey puck can
approach 100 mph while a player may be skating upwards of 25 mph.
When players are spinning or turning at or near their full speed of
travel, the combined joint movement speed can present problems to
tracking systems with lower sampling rates. For beacon-based
systems such as Trakus and Orad, which take approximately 30
locations per second, these speeds would present a significant
challenge if the beacons could be placed upon a puck or even a
player's wrist. For vision-based systems such as Motion Analysis,
Vicon and the present invention, camera and video capture
technology can presently handle upwards of 240 frames per second
with acceptable 1 megapixel resolution.
[0133] B.4. Are there multiple objects to be tracked and, if so,
how will this impact the tracking method?
[0134] As has been previously suggested, it is necessary to track
the motion of at least 10 players and 3 game officials in the
playing area for the sport of ice hockey. Including the players'
benches, the total number of players and officials can exceed 40.
For other sports such as American football, the total players and
officials can exceed 60, over 20 at a time on the playing surface.
This requirement for ice hockey alone has pushed the limits of
beacon-based systems such as Trakus and Orad that are simply
attempting to track a single point, the helmet, for each player.
For vision-based systems such as Motion Analysis and Vicon, this
requirement also presents a significant challenge. All of the
existing vision tracking systems first place fixed field of view
cameras just outside the tracking area. These cameras are
strategically placed in an attempt to keep the maximum number of
markers within view of at least two cameras at all times even as
these markers travel about in six degrees of freedom. The prospect
of viewing multiple players within a given volume further
challenges the strategic placement of these fixed cameras. As more
and more players randomly congregate within any given volume, more
and more of their respective markers will be blocked, from any
given camera's field of view. Since the sum total of a player's
markers is used in and of itself for player identification, as
markers are lost from view this not only jeopardizes the accurate
tracking of an individual body part, but also increases the
incidence of improperly identified players.
[0135] The present inventors prefer the novel approach of
controllably directing one or more movable pan, tilt and zoom
cameras to follow each individual player based upon the first
determined (X, Y) location as obtained via the overhead cameras. In
this way the system's "total field-of-view" is dynamically
recreated and maximized for player and marker visibility. The
preferred embodiment includes movable cameras that are not
necessarily unconditionally dedicated to follow a single player. It
is anticipated that, given the known and projected locations of
each player in the tracking area, it may be beneficial to
dynamically switch one or more cameras away from one or more
players onto one or more other players.
[0136] B.5. Are there any restrictions or safety considerations
regarding the type of electromagnetic energy used to track the
objects?
[0137] For safety requirements the properties of both wavelength
and flux must be considered apart and in combination. The
beacon-based tracking implemented by Trakus employs microwaves
similar to cell phone technology. These frequencies of energy can
be harmful in larger amounts and, for this reason, Trakus pulses
their transmitter signals so as to reduce the average energy
exposed to the player through the helmet. However, the present
inventors see at least the potential for perceived harm since the
players, which in the case of recreational hockey are youths, will
be exposed to higher dosages of this energy as they tend to bunch
together during play or at least in the team bench area.
[0138] In the case of vision-based systems such as Motion Analysis
or Vicon, the primary tracking energy is visible, such as red
light. While these energies are not harmful to the players, they do
restrict the use of these systems to controlled settings where it
is acceptable for the athletes and viewers to see the markers and
their reflections. Motion Analysis, Vicon and other providers also
offer the choice of working in near IR frequencies. These IR
frequencies are safe for human exposure. The ring lights used to
illuminate the tracking area in these systems only emit IR energy,
however, and as such are not noticeably active on to the casual
viewer. There is some concern that an uninformed individual could
stare up at the camera with its IR ring light and receive an
overexposure of IR energy possibly damaging the retina of the eye.
For this reasons, these ring lights come with additional small
LED's that emit a visible light cue when the IR light is on,
essentially alerting people not to stare. This may not be a
sufficient mechanism for the uniformed casual viewer. When tracking
in IR energy, the present inventors prefer a light source that
emits both visible and IR light since the visible light will act as
a natural cue to dissuade the observer from staring.
[0139] Also taught in the preferred embodiment is the novel use of
UV energy for tracking purposes. UV energy is broken into three
types; UVC, UVB and UVA. UVC and UVB are generally thought to be
the most damaging while UVA is considered to be biologically safe
and much more prevalent in the atmosphere than UVB. UVA is
typically considered to be those wavelengths between 315 to 400 nm.
Visible light begins as 400 nm. The preferred embodiment employs
tracking frequencies centering around 390 nm, just shy of visible
light. This is an ideal wavelength since there are low-cost
industrial digital cameras whose responsivity curves peak at or
near this frequency. Furthermore, the metal halide lamps that are
often found in hockey rinks generate a significant amount of UVA
energy that could be used as an energy source.
[0140] B.6. What are the physical space limitations on the objects
for any marker or beacon-based tracking system?
[0141] To employ beacon-based tracking such as Trakus and Orad for
contact sports it is preferable to embed the beacon somewhere in
the player's equipment. Due to several other drawbacks, these
beacon systems have been limited to tracking a single point that
was centrally chosen to be the helmet. If other considerations were
permitting and additional beacons could be tracked per player, then
the requirement to embed beacons into the equipment in order to
avoid injury possible from player contact would itself becoming a
significant problem.
[0142] For vision-based systems such as Motion Analysis and Vicon,
the markers must reside outside of the player's exposed surface in
order to receive and reflect the tracking energy. Furthermore, as
was previously mentioned, these markers have been constructed as
spheres so as to maximize the reflected image independent of marker
angle to camera. These spheres strategically placed onto various
joints and body parts are impractical for live, uncontrolled
settings such as a full-contact sporting event. The preferred
embodiment of the present invention specifies the use of reflective
ink or paint that can be applied to the various substrates such as
plastic, wood, fabric, leather, rubber, etc. and will add minimal
thickness. Unlike to current practice of using visible spheres of
limited diameter that must protrude from the player to maximize
reflection, the preferred technique uses non-visible ink to place
large markings on the various joints and body parts. Depending upon
the transmissivity to the fabric being worn by the players, the ink
itself could be placed on the inside of this material and receive
and reflect the non-visible tracking energy directly through the
jersey. The end result of this type of preferred marking is to
overcome any physical space limitation considerations.
[0143] B.7. What is the natural reflectivity of the various
background surfaces to the different potential tracking
energies?
[0144] The reflectivity of the background surfaces, and for that
matter the "non-tracked" foreground surfaces, represents the most
significant source of system noise. For beacon-based systems such
as Trakus and Orad, the microwaves emitted by the players'
transmitters will reflect off hard surfaces such as metal and
concrete. Due especially to the enclosed environment of a hockey
arena, these reflections will continue to bounce between the
background surfaces within the tracking volume as they slowly
attenuate, thus contributing to significant noise problems.
[0145] For vision-based systems such as Motion Analysis and Vicon,
reflectivity issues are limited to the individual camera's field of
view. Hence, since the cameras are primarily focused on the playing
surface and surrounding boundaries, the reflection issues are
already diminished over beacon-based systems. However, for the
sport of ice hockey, the playing surface is ice that tends to be
highly reflective of visible, IR and UV energies. In the case of
visible red light as preferred by Motion Analysis and Vicon, these
reflections are typically handled via software processing. During
system calibration, the images captured by each camera are reviewed
for possible ice surface reflections especially from the ring
lights affixed to the tracking cameras. Any such undesirable
signals are "mapped" out via a software tool that essentially
informs the image analysis to ignore any and all data capture from
those coordinates. Of course, should valid marker data
coincidentally show up at that same location during a live filming,
it would be ignored as well.
[0146] In the preferred embodiment, absorbent compounds are applied
to the various background and "non-tracked" foreground surfaces.
For instance, if the tracking energy is UV, traditional UV
absorptive compounds as are well known in the commercial
marketplace can be used to absorb stray UV energy. The entire
player including their jersey, helmet, pads, gloves, stick, etc. as
well as the ice surface, boards and glass can all be first treated
with one or more UV absorptive compounds. This novel technique
significantly reduces system noise and essentially makes everything
but the subsequently applied markers invisible to the tracking
cameras. Similar techniques are possible if IR is used as the
tracking energy.
[0147] C. Referring next to the characteristics of the tracking
environment, the preferred and alternate novel embodiments must
consider the following questions:
[0148] C.1. Is the setting "live" or "controlled"?
[0149] Tracking players in a live sporting event such as ice hockey
adds a major restriction to the system requirements. Namely, the
system must be as unobtrusive as possible for both the players and
the viewing audience. In the case of the beacon systems from Trakus
and Orad, the transmitters are hidden and the tracking signal is
non-visible. The only possible drawback is that receiving towers
must be placed throughout the arena. Careful attention must be made
to keep these towers from blocking the view of nearby fans.
[0150] With respect to vision-based systems such as Motion Analysis
and Vicon, there are two major types of restrictions present in
their approach. First, the tracking energy itself is typically
within the visible spectrum (e.g. red light). This energy is
necessarily also visible to the players and audience. Second, as
previously discussed, the markers have substantial size. Their size
makes them noticeable to players and fans and potentially dangerous
to the players since they could apply additional pressure to their
bodies upon contact with other players, the ice surface or the
boards. Furthermore, these markers could easily be dislodged
creating at a minimum the stoppage of game play and or worse a
potential injury hazard. Motion Analysis and Vicon, as well as
other vision tracking providers, offer an option to work in IR
light. This change eliminates the direct interference of visible
light and its reflections off the ice surface with player and fan
vision.
[0151] However, the markers used by these types of systems have an
additional problem other than their size. Specifically, they are
constructed of a spherical material preferably a foam ball, that
has been covered with a retroreflective tape. The tape itself is
greyish in visible color. These tapes are typically made of a
material referred to as "cube cornered retroreflectors" but could
also be produced with microspheric devices. In either case, current
technology provides for optically transparent bodies that are
coated on their undersides with a reflective material such as
aluminum or silver, essentially forming a mirror. The additional
problem is the broad band frequency response of these microscopic
mirrors that includes visible light as well as UV and IR. Hence,
even if the tracking energy is switched to IR, other ambient
visible frequencies will themselves retroreflect off these markers
back towards the players and fans who are near the light source. As
players move about, these spheres will have a tendency to fluctuate
in brightness as they pass through various lighting channels. If a
fan or reporter were to use flash-based photography or illuminated
videotaping, these markers would retroreflect this energy back into
their camera causing noticeable bright spots on top of the greyish
circles.
[0152] To overcome these drawbacks, the present inventors teach the
use of ink-based markings that are applied directly to the
substrates worn or held by the players. Unlike the spherical
markers, these inks will not add any appreciable thickness to the
player's clothing or equipment. This effectively overcomes the
marker size problem. To overcome the broad band reflectivity
drawback, the preferred embodiment includes either reflective,
fluorescent or retroreflective inks and compounds that have been
engineered to only reflect the narrow band of non-visible tracking
energy and as such are substantially transparent to visible
light.
[0153] C.2. What are the existing ambient electromagnetic
energies?
[0154] Within the live setting of at least a hockey arena, there
will be large area high intensity discharge (HID) lighting such as
Metal Halide or Mercury Vapor lamps. These sources typically are
chosen because they generate a very broad range of energy
throughout the visible spectrum producing a natural-looking white
light. The placement of these lamps will be determined and
conducive to audience viewing considerations rather than object
tracking requirements. During the contest, these lights may be
altered in intensity or augmented with additional lighting for
visual effect. There is no assurance that the illumination levels
in the arena will remain consistent in intensity. Furthermore, it
must be anticipated that either fans or broadcasters will bring in
additional portable light sources for filming purposes that will
also add to the uncontrolled lighting levels.
[0155] For beacon-based systems such as Trakus and Orad that depend
upon microwave transmissions, the visible light frequencies do not
present any noise to the tracking system. Fans using portable cell
phones will be transmitting microwave energy but the beacon signals
can simply be set to emit at a different wavelength.
[0156] For vision-based systems such as Motion Analysis and Vicon,
these ambient frequencies are an important source of noise
especially in consideration of the various colored uniforms and
equipment worn by players during a typical live contest. Hence, if
the players are wearing red uniforms or they have red streaks on
their jerseys or equipment, then the ambient visible light sources
will tend to reflect this color into the tracking cameras. These
reflections are similar to those caused by the red LED ring lights
as their emitted energy strikes the ice surface. As previously
mentioned, the reflections from the ring lights are "mapped out"
via software during system calibration. This software technique
could not be used to "map out" the dynamically changing red light
reflections caused by the player's jersey and equipment colors. It
should be noted that these systems include red light filters on
their cameras effectively restricting noise to a narrow band
overlapping the emitted tracking energies, but not fully
eliminating the problem.
[0157] As previously mentioned, machine vision companies such as
Motion Analysis and Vicon do offer their systems using IR light.
This non-visible frequency is used when the customer requires that
the tracking be done in darkness. The IR tracking energy is not
being used to reduce system noise by cutting down on the visible
spectrum.
[0158] In order to avoid noise created by the existing visible
light sources, the present inventors prefer to work in a
non-visible tracking energy such as UV or IR and to employ
energy-absorptive techniques elsewhere described in the present and
co-pending applications.
[0159] C.3. Are there any other pre-existing energy sources that
may have available by-product energy that could be used for
tracking?
[0160] Several of the HID lamps used for large area illumination
also generate non-visible frequencies such as UV and IR. Typically,
the manufacturers of lights encase the inner bulb and filament with
an outer bulb that has been filled with a special vapor mixture and
coated with a special compound. The combination of the additional
glass bulb, the special vapor and the special coating act to absorb
the non-visible energies rather than release them into the
surrounding environment. The present inventors anticipate modifying
these various light sources to specifically emit some additional
portion of their generated non-visible energy that is currently
being absorbed.
[0161] C.4. What is the natural reflectivity of the various
background surfaces to the different potential tracking
energies?
[0162] Within a hockey arena, the hard mirror-like ice surface
presents a challenge to both beacon- and machine vision-based
tracking. For instance, other playing surfaces such as grass would
tend to absorb most of the ambient frequencies including microwave,
red light, UV and IR while only reflecting green light. Another
reason the present inventors prefer to track the players and
equipment using a non-visible energy such as UV or IR is that the
background can then be first treated with either non-visible
absorbers, reflectors, or both. This helps to create a clear
reflective distinction between the markers and the background
thereby facilitating image analysis. The present inventors also
prefer to place these non-visible absorbers or reflectors onto the
foreground objects such as the players and equipment prior to
marking them. In this way, the detectable intensity levels of any
unwanted reflections off the foreground objects can be controllably
differentiated from the marker's reflections.
[0163] C.5. What is the size of the tracking area relative to the
range of the potential tracking methods?
[0164] Some sports have relatively small playing areas and few
players, such as tennis, while other sports have very large areas
and many players, such as golf. Sports such as ice hockey,
football, soccer and baseball have mid-to-large sized areas with
many players to track simultaneously. Microwaves such as those used
in the beacon-based systems from Trakus and Orad work well in any
of these various-sized playing surfaces. Vision-based systems as
currently implemented by companies such as Motion Analysis and
Vicon are challenged to track areas significantly larger than a
tennis court in size.
[0165] The major reason for their difficulties is the "volume
tracking" approach they have taken to solving this problem. Simply
put, each contestant may move about the playing surface which
because of his or her own height and the expected flight of any
game objects, becomes a playing volume rather than an area. Every
portion of this volume must be in view of two or more cameras at
all times. The cameras must be fixed prior to the contest so that
they may be calibrated as a network. If players will have a
tendency to bunch up, additional cameras will be needed within any
given volume to create additional views thereby reducing
anticipated inclusions. The cameras must be limited in the field of
view so that they can maintain a sufficient resolution or pixels
per inch within their field in order to detect the reflected
markers. As the playing area widens, it becomes increasingly
difficult to place cameras close enough to the inner volumes so
that the ideal field of view is maintained per camera without
causing an obstruction to the players or viewing audience. This
obstruction would occur if a mounting structure were created to
hang the cameras directly above the inner volumes. Newer cameras
will continue to provide higher resolutions theoretically allowing
the cameras to move further back from any given volume and still
maintain the requisite pixels per inch resolution. As cameras pull
back, however, the distance between the energy tracking source, the
reflective marker and the cameras will continue to increase thereby
having a negative effect on signal strength.
[0166] The present inventors prefer to separate player (object)
tracking into two distinct sub-processes thereby eliminating the
aforementioned problems. In distinct contrast to the "volume
tracking" approach, the preferred embodiment of the present
invention relies upon a "player following" controlled by an "area
tracking" technique. In essence, the players are first tracked in
two dimensions, X and Y, throughout the playing area. The currently
determined location of each player is then used to automatically
direct that player's individual set of cameras that "follow" him or
her about the playing surface. This two-step approach has many
critical advantages when faced with tracking objects throughout a
larger volume. First, locating the "top surface" (i.e., helmet) of
each player in X-Y space for substantially the entire contest is
significantly simpler than trying to detect their entire form from
two or more cameras throughout the entire playing volume. Second,
by placing the player id on their "top surface" (i.e., their helmet
and if need be shoulder pads), the system is able to easily
identify each player while it also tracks their X-Y coordinates.
Third, by controllably directing one or more automatic pan, tilt
and zoom cameras to follow each player (and game object) the ideal
field of view and maximum resolution can be dynamically maintained
per player.
[0167] C.6. Is the tracking environment physically enclosed within
a building or outside?
[0168] The indoor enclosure is most prevalent in the sport of ice
hockey as opposed to other major sports such as football, soccer
and baseball. This particular requirement has a negative effect on
beacon-based systems such as Trakus and Orad since the hard
surfaces of the enclosed arena will cause the microwave tracking
energy to go through many reflections before finally dissipating.
For vision systems such as Motion Analysis and Vicon, the lower
ceilings that are typically found in local youth hockey arenas
present a different type of problem. Due to the "volume tracking"
approach just discussed, these systems ideally require the
placement of their tracking camera assemblies above the playing
surface or at least off to the side and very near the "tracking
volume." These assemblies include the camera, lens and filter, the
ring lights and power supply as well as a small computer processor
for initial data analysis and may typically cost around $15,000. In
total distance, each assembly may not be more than twenty to thirty
feet off the ice. This makes each assembly prone to damage when a
puck is either accidentally or intentionally shot at the camera
system. The present inventors prefer an enclosed assembly where the
camera, lens and related equipment are protected and yet still able
to view the ice surface below through a Plexiglas or similarly
transparent covering.
[0169] D. Referring next to traditional engineering goals, the
preferred and alternate novel embodiments must consider at least
the following factors:
[0170] D.1. The system should be scalable and therefore comprise
uniform assemblies that are combinable into a matrix designed to
increase tracking coverage in terms of area, volume or the number
of players (objects) while still maintaining uniform
performance.
[0171] In one respect, beacon-based systems such as Trakus and Orad
include a uniform assembly in the form of the player's transmitter
that is embedded within their helmet. This design is scalable by
the number of players since each additional player is simply given
an additional transmitter. In order to be scalable in terms of the
subsequent signal processing required to locate each transmitter in
X-Y space, however, the system will need to encode select
transmitters to select receiving towers. The present inventors do
not believe that companies such as Trakus and Orad are currently
practicing this technique. This concept requires that a capacity be
determined for a single set of receiving towers in terms of the
number of transmitters that can be tracked in real time. For
example, assume that four receiving towers are initially set up
within the arena and that they can only adequately track five
transmitters in real time. Once a sixth player is outfitted with a
transmitter, the entire system will lose its real time processing
capability. It is preferable to equip the initial four towers with
a first processor that eliminates any received signal that was not
generated by one of the initial five transmitters. In this way the
initial five players remain adequately tracked in real time. In
order to track an additional five players, an additional four
towers and processor should be added to the arena. These towers
would then be assigned to specifically process only those
transmitters with the unique codes corresponding to the second five
players. This approach will make the beacon-based system scalable.
Otherwise, the entire system will be "player bound" in that it will
have a maximum number of total transmitters that can be tracked
irrespective of the playing volume.
[0172] Similarly, instead of encoding each signal and digitally
filtering all incoming signals for selected codes, each maximum
group of transmitters could be assigned a different frequency. The
filtering process could then be moved to electronics. Another
possibility is to coordinate the timing of the transmitted signals
from the various groups of maximum transmitters. In essence, each
group of transmitters will be broadcasting their locating signal as
specified and non-overlapping intervals coordinated with the
receiving towers. The important point is to make the system
scalable by defining a minimum configuration of tracking apparatus
that can simply be repeated to either increase area, or in the case
of the beacon approach, increase players tracked.
[0173] For vision-based tracking systems such as Motion Analysis
and Vicon, the systems are not uniform in their "camera view per
volume." This lack of uniformity works against system scalability
and is due to several coincident design factors. First, the camera
assemblies are both relatively expensive in their own right and
when taken together represent a significant portion of the system
price. An attempt is being made to maximize the use of each
individual camera's field of view. Second, the field of view of
each individual camera is best thought of as a four-sided pyramid
where the apex emanates from the CCD array in the back of the
camera. In practice, the first several feet of the field of view
are not useable for tracking and even so are more than likely
outside of the tracking volume for pragmatic clearance
considerations. This can be thought of as taking the top off the
pyramid. The pixels per marker inch resolution at the nearest
versus farthest points in this pyramid can be substantially
different and must necessarily be limited by the minimum acceptable
resolution to identify a given size marker.
[0174] Third, to be scalable, each tracking volume serviced by two
or more assemblies should be either square or rectangular in its
cross-sectional shape parallel to the ice surface. In this fashion,
these volumes could simply be repeated with slight overlaps for
calibration purposes in order to create larger and larger tracking
volumes. However, this is difficult to accomplish in practice using
overlapping, four-sided pyramids. And finally, an individual player
with multiple markers attached could transverse any portion of the
tracking volume at any rotational angle and it is necessary that
each marker remain in the site of two cameras at all times. This
requirement calls for a minimum of four cameras surrounding any
given volume. In practice six cameras is a more acceptable minimum.
As additional players are added to the tracking volume, it is not
difficult to see how easy it will be for one player to block the
camera's view of another. Keeping in mind the player identification
technique practiced by companies such as Motion Analysis and Vicon
where the unique "full body" constellation of markers identifies a
skater, inclusions take on greater significance.
[0175] The task of designing a uniform configuration of acceptable
resolution throughout the entire tracking volume while also
accounting for the two camera minimum per marker and multiple
player bunching is formidable. Furthermore, the end result "wastes"
camera field of view since the actual cross-sectional shape formed
by the configuration will not be either a square or rectangle nor
will it even be uniform in size as it moves from the ice surface up
towards the cameras. Hence, if multiple of these "minimum tracking
volumes" were to simply be replicated and slightly overlapped, this
would result in a considerable loss of valuable tracking region
along the edges of the idealized square or rectangular volume. For
all of these reasons, companies such as Motion Analysis and Vicon
have not approached the larger "tracking volume" as a super-set of
smaller volumes in what could be called a scalable approach. For
all practical purposes, each unique tracking volume shape and size
in combination with the expected number of objects (or players)
requires a "custom" layout of tracking assemblies. As more and more
players continue to be added to the volume, more and more cameras
will be added based upon best judgments for placement. This entire
arrangement could be best described as "player bound" as well as
"quasi-volume bound."
[0176] As previously discussed, the present inventors prefer the
novel approach of separating the (X, Y) tracking of each player or
game object as a whole from the (X, Y, Z) determination of each
player's critical locations (e.g., joints and body parts).
Furthermore, the novel concept of locating all of the identity
markings on the "upper surface" of each player facilitates the
top-oriented field of view of the (X, Y) tracking assemblies which
naturally limits inclusions due to player bunching. Given this
separation of tasks, the present invention becomes highly scalable.
Each overhead (X, Y) area tracking assembly covers a fixed and
uniform square or rectangular tracking area. Furthermore, the pixel
resolution per this area remains substantially constant. To cover
more tracking area, simply add additional (X, Y) assemblies. Each
player that moves throughout this connected tracking area has their
helmets (located at a minimum) and ideally also their shoulders.
From this information, the player is identified along with
pertinent information including orientation, direction of movement,
velocity, and acceleration as well as current relative (X, Y)
location. Using the current (X, Y) location as well as the
direction of movement, velocity and acceleration, the preferred
embodiment controllably directs one or more (X, Y, Z) pan, tilt and
zoom cameras to automatically follow the given player. Furthermore,
by intelligent inspection of the various projected player paths,
the system optionally switches (X, Y, Z) cameras from one player to
another to best maximize overall tracking performance. By
constantly zooming each (X, Y, Z) camera to maximize player size
per field of view, a uniform and ideal pixel resolution per marker
square inch is maintained. To cover more players, simply add
additional (X, Y, Z) pan, tilt and zoom camera sets per added
player.
[0177] D.2. The system should be minimally intrusive upon the
objects to be tracked and upon the surrounding environment
especially if that environment is a live setting.
[0178] As was previously discussed, beacon-based systems such as
Trackus and Orad are environmentally transparent in terms of their
microwave tracking energy but are intrusive in terms of their
beacon technology. While these beacons are small and can
conceivably be reduced further in size, they will still always be
practically limited to locations where they can be embedded for
both the safety and comfort of the player as well as the protection
of the device itself.
[0179] As was also discussed, vision-based systems such as Motion
Analysis and Vicon are intrusive in three key areas. First, their
tracking energies are in the visible spectrum such as red light
that is visible to players and fans alike. Second, their markers
have substantial size and are both noticeable and prone to
accidents in a full contact sport such as ice hockey. And third,
these same markers are covered with retroreflective tape that is
broad spectrum reflective including visible light. As such,
existing rink illumination and portable lighting such as camera
flash bulbs will tend to create unwanted and distracting
reflections into the players' and fans' views.
[0180] And finally, the preferred embodiment of the present
invention overcomes each of these intrusions of the surrounding
environment through its novel teachings as follows. First, the
preferred tracking energy is in the non-visible frequencies such as
UVA or near IR. Second, the large spherical markers have been
replaced with an "invisible" ink or paint that can be adhered to
the many necessary substrates while adding minimal thickness. And
finally, this ink or paint, whether it be reflective, fluorescent
or retroreflective in nature, is made "invisible" by its
characteristic of only reflecting or emitting in the desired
non-visible frequencies.
[0181] D.3. The tracking signal-to-noise ratio should be
maximized.
[0182] The microwave-based beacon systems such as Trakus and Orad
will have minimal noise due to ambient electromagnetic energies.
However, due in part to the enclosed nature of a hockey arena and
more importantly to the hard surfaces within the arena such as the
ice surface itself, each signal transmitted from a player's helmet
will bounce off these surfaces creating significant signal noise
problems.
[0183] The vision-based systems such as Motion Analysis and Vicon
attempt to limit system noise by using a narrow band of tracking
energy such as red light with a corresponding camera filter.
Unfortunately, by working within the visible spectrum in their
preferred approach, they are susceptible to noise created by
existing rink lighting, additional portable lighting and the
reflections these sources will cause off player jerseys and
equipment. And finally, even if they work in the non-visible IR
region, both this IR energy and red light will reflect off the ice
surface creating false marker reflections that must be "mapped out"
via software techniques creating "dead spots" within the tracking
images.
[0184] The present invention is the only system to employ a
combination of tracking energy absorbent and reflective compounds
in order to "control" the tracking frequency noise. By using
absorbers, unwanted reflections are significantly reduced or
eliminated altogether. By using reflective compounds, unwanted
reflection can be increased to a designated intensity region that
is detectably different from the marker signal intensity. These
absorbent and reflective compounds can be applied to all of the
background and foreground substrates. By first addressing the noise
issues, the foreground objects are preferably set at a minimal or
zero reflection state while the background including the ice
surface, the boards and the glass is preferably set at a narrow
intensity range detectably different from the marker intensity.
[0185] D.4. Manufacturing and installation costs should be
minimized and the resultant system should be simple for the user to
maintain and operate.
[0186] To the extent that the present invention has been shown to
be scalable, its manufacturing and installation parameters are more
easily calculated and maximized thereby leading to reduced costs.
Furthermore, because of its uniform scalable design, the present
invention will minimize any requirements for operator intervention
to resolve inclusions, lost player identity, or similar confusion
due in general to signal-to-noise problems.
[0187] To the extent that the existing beacon- and machine
vision-based systems are not scalable, their manufacturing and
installation parameters are not as easily calculated and maximized
thereby leading to increased costs. Furthermore, because these
systems are not uniformly scalable in combination with the larger
tracking volume and number of tracking markers associated with 3D
multi-player movement, they are susceptible to errors from
inclusion and poor signal-to-noise ratios creating a need for
operator involvement.
[0188] Objects and Advantages
[0189] Accordingly, the objects and advantages of the present
invention are to:
[0190] 1--teach the fundamental component groups necessary for a
multi-object real-time 3D object tracking system;
[0191] 2--identify those individual components already in use
within currently available systems and to which component groups
they belong;
[0192] 3--teach those novel components suggested by the present
inventors in this and their other four co-pending applications and
to which component groups they belong;
[0193] 4--teach how the present inventors' novel components allow
currently available systems to better function in a live
application with multiple colliding objects, for instance a
sporting event such as ice hockey;
[0194] 5--identify the composition of currently available
multi-object real-time 3D object-tracking systems in terms of
actual components used from each group;
[0195] 6--teach a novel preferred embodiment for a multi-object
real-time 3D object-tracking system best suited for a live sporting
event such as ice hockey in terms of actual components used from
each group;
[0196] 7--teach several novel alternative embodiments using one or
more components of currently available systems mixed into the
preferred embodiment; and
[0197] 8--teach several novel variations using one or more of the
present inventors' novel components mixed into the currently
available systems.
[0198] Further objects and advantages are to provide:
[0199] 1--a system that is scalable and therefore comprises uniform
assemblies that are combinable into a matrix designed to increase
tracking coverage in terms of area, volume and the number of
objects while still maintaining uniform performance;
[0200] 2--a system that is minimally intrusive upon the objects to
be tracked and upon the surrounding environment especially if that
environment is a live setting;
[0201] 3--a maximized tracking signal to noise ratio; and
[0202] 4--a system with maximized manufacturing and installation
costs that is simple for the user to maintain and operate.
[0203] Still further objects and advantages of the present
invention will become apparent from a consideration of the drawings
and ensuing description.
DESCRIPTION OF THE DRAWINGS
[0204] FIG. 1 is a block diagram depicting all of the major
components necessary for the various multi-object tracking
machine-vision systems according to the present invention. These
components are broken into ten groups including Camera Assembly,
Tracking Frequency, Energy Source, Marker: Emission Method, Marker:
Physical Form, Marker: Reflective Shape, ID: Location, ID: Encoding
Method, ID: Obtained, and Calibration Method.
[0205] FIG. 2 is a block diagram highlighting the combination of
major components currently being used in several existing machine
vision systems with the main distinctive components of fixed (X, Y,
Z) volume tracking cameras, visible light, spherical (attached
ball) markers and a full body, unique constellation ID added by the
present invention.
[0206] FIG. 3 is a block diagram highlighting the combination of
major components in the preferred embodiment of the present
invention, including the use of fixed (X, Y) area tracking cameras
in combination with movable (X, Y, Z) volume tracking as well as
non-visible IR or UL, flat reflective markers, and a top surface of
body, encoded markings ID.
[0207] FIG. 4 is a block diagram highlighting the combination of
major components in an alternate embodiment of the present
invention. The main distinction between this alternate and the
preferred is the use of flat fluorescent markers instead of flat
reflective.
[0208] FIG. 5 is a block diagram highlighting the combination of
major components in another alternate embodiment of the present
invention. The main distinction between this alternate and the
preferred is the use of flat retroreflective markers instead of
flat reflective.
[0209] FIG. 6 is a block diagram highlighting the combination of
major components in still another alternate embodiment of the
present invention. The main distinction between this alternate and
the preferred is that the ID is obtained during rather than outside
game surface tracking.
[0210] FIG. 7 is a block diagram highlighting the combination of
major components in another alternate embodiment of the present
invention. The main distinction between this alternate and the
preferred is the use of fixed (X, Y, Z) volume tracking cameras
rather than the fixed (X, Y) area tracking and movable (X, Y, Z)
volume as well as flat retroreflective markers, and a full body,
unique constellation ID.
[0211] FIG. 8 is a block diagram highlighting the combination of
major components in an alternate embodiment of the present
invention. The main distinction between this alternate and the
alternate of FIG. 7 is the use of a top surface of body, encoded
markings ID.
[0212] FIG. 9 is a block diagram highlighting the combination of
major components in an alternate embodiment of the present
invention. The main distinction between this alternate and the
alternate of FIG. 8 is that the ID is obtained during rather than
outside game surface tracking.
[0213] FIG. 10 is a block diagram highlighting the combination of
major components in another alternate embodiment of the present
invention. The main distinction between this alternate and the
preferred is the use of fixed (X, Y, Z) volume tracking cameras in
place of the movable (X, Y, Z) volume.
[0214] FIG. 11 is a block diagram highlighting the combination of
major components in another alternate embodiment of the present
invention. The main distinction between this alternate and the
alternate of FIG. 10 is the use of flat fluorescent rather than
reflective markers.
[0215] FIG. 12 is a block diagram highlighting the combination of
major components in another alternate embodiment of the present
invention. The main distinction between this alternate and the
alternate of FIG. 11 is the additional use of movable (X, Y, Z)
volume tracking cameras as well as flat retroreflective rather than
reflective markers.
[0216] FIGS. 13a and 13b depict the theory and implementation of
fixed (X, Y, Z) volume tracking camera assemblies.
[0217] FIGS. 14a and 14b depict the theory and implementation of
fixed (X, Y) area tracking camera assemblies.
[0218] FIGS. 15a and 15b depict the theory and implementation of
movable (X, Y, Z) volume tracking camera assemblies.
[0219] FIG. 16a depicts the relationship between constant versus
dynamic field-of-view and its impact on the pixel resolution per
player (object) tracked.
[0220] FIG. 16b depicts how player bunching and therefore marker
inclusion is addressed by the use of movable (X, Y, Z) volume
tracking camera assemblies.
[0221] FIG. 17 is a side view drawing of a typical high intensity
discharge (HID) lamp of the type often used to illuminated large
open spaces such as a sporting arena or facility, further depicting
the spread of emitted electromagnetic frequencies ranging from UV
through visible light into IR.
[0222] FIG. 18a is a side view of the same HID lamp showing its
emitted energy being dispersed in multiple directions as it strikes
a typical reflective material.
[0223] FIG. 18b is a side view of the same HID lamp showing its
emitted energy being redirected back towards the lamp as it strikes
a typical retroreflective material.
[0224] FIG. 18c is a side view of the same HID lamp showing its
emitted energy being fluoresced and then dispersed in multiple
directions back towards the lamp as it strikes a typical
fluorescent material.
[0225] FIG. 19 is a side view of a typical HID lamp, further
depicting the spread of emitted electromagnetic frequencies ranging
from UV, through visible light into IR. Also shown are three
variably oriented retroreflective elements partially embedded in a
single binder that has been joined to a substrate. The elements and
binder have been depicted as transmissive to visible light while
the substrate is reflective. In response to the non-visible
frequencies of either UV or IR, at least some of the elements are
retroreflective while the substrate remains reflective.
[0226] FIG. 20a depicts a spherical retroreflective marker and its
circular reflection while FIG. 20b depicts a hockey player with
attached spherical markers.
[0227] FIG. 21a is a set of three perspective drawings depicting a
typical player's jersey, typical player's pads with tracking
patches in place, and then a combination of the jersey over the
pads with patches.
[0228] FIG. 21b is a set of two perspective drawings depicting a
hockey puck as well as a typical player's hockey stick, where each
has been augmented to include tracking ink on at least some portion
of its outer surfaces.
[0229] FIG. 21c is a set of two perspective drawings depicting a
typical hockey player's helmet which has been augmented to include
tracking stickers on at least some top portion of its outer
surface.
[0230] FIG. 22a depicts a hockey player set up with spherical
markers while FIG. 22b shows the resultant circular reflections
that will be seen with an appropriate vision system.
[0231] FIG. 22c depicts a hockey player set up with flat markers
while FIG. 22d shows the resultant multi-shape reflections that
will be seen with an appropriate vision system.
[0232] FIG. 23a depicts two different hockey players set up with
spherical markers. In the depicted view, the players are not
overlapping. FIG. 23b shows the resultant circular reflections that
will be detected by appropriate frame analysis.
[0233] FIG. 23c depicts the same two different hockey players that
were shown in FIG. 23a except that they are now overlapping. FIG.
23d shows the resultant circular reflections that will be detected
in this case.
[0234] FIG. 24 depicts the combination of encoded ID marks as well
as track marks that are preferably placed upon a helmet sticker.
Also shown by comparison is the difference between the "top
surface" helmet ID approach versus the full body "constellation" ID
approach.
[0235] FIG. 25 depicts the separation of player identification and
tracking on the non-playing surfaces of the entrance and exit
passageway and the team benches versus player tracking only (and
not identification) on the playing surface.
[0236] FIG. 26a depicts the pre-tracking calibration method most
typically used with current fixed (X, Y, Z) volume tracking
camera-based systems.
[0237] FIG. 26b depicts the dynamic calibration method taught for
use with the fixed (X, Y) area and movable (X, Y, Z) volume
tracking cameras of the preferred embodiment.
[0238] FIG. 27a depicts a typical hockey player's pads, helmet,
stick and puck being captured from an overhead X-Y filming camera
and displayed on a viewing screen.
[0239] FIG. 27b is similar to FIG. 27a except that now all of the
foreground objects have been first treated with an
energy-absorptive compound after which tracking marks have been
added to desired locations.
[0240] FIGS. 28a and 28b depicts the alternate embodiment that uses
fixed (X, Y) area cameras for player identification and tracking on
the non-playing surfaces while also using only movable (X, Y, Z)
volume tracking cameras to track the players on the playing
surface. Therefore the fixed (X, Y) area cameras are not first
being used to track the players on the playing surface in order to
direct the movable cameras but rather a predictive technique is
employed to guide the movable cameras as they follow their intended
targets.
[0241] FIG. 29a depicts a potential shoulder mark that along with
the jersey it is placed upon can potentially create up to three
different reflectivity intensity levels depending upon the use of
absorbent, reflective and retroreflective compounds.
[0242] FIG. 29b depicts the strategic regular placement of markings
upon the shaft of a hockey stick that allow the stick to serve as a
dynamic calibration tool similar to the preferred track mark on the
player's helmet.
[0243] FIG. 30 depicts the additional strategic regular placement
of markings upon the playing surface such as the boards and
channels that hold the glass enclosing the hockey rink. These
markings further serve as a dynamic calibration tool for use
especially by the movable (X, Y, Z) volume tracking cameras.
DETAILED DESCRIPTION OF THE INVENTION
[0244] Referring first to FIG. 1, there is shown a block diagram
depicting all of the major components 1000 anticipated to be
necessary for the various multi-object tracking machine vision
systems according to the present invention. These components are
broken into ten groups including Camera Assembly 500, Tracking
Frequency 510, Energy Source 520, Marker: Emission Method 530,
Marker: Physical Form 540, Marker: Reflective Shape 550, ID:
Location 560, ID: Encoding Method 570, ID: Obtained 580 and
Calibration Method 590. Camera Assembly 500 can be one or more of
Fixed (X, Y, Z) Volume Tracking assemblies 502, Fixed (X, Y) Area
Tracking assemblies 504 and Movable (X, Y, Z) Volume Tracking
assemblies 506. Tracking Frequency 510 can be one or more of
Visible Light 512, Infrared Light 514 or Ultraviolet Light 516.
Energy Source 520 can be one or more of Ring Lights Emitting
Visible or IR Frequencies 522, Existing Lights Emitting Visible
Frequencies 524 and Existing Lights Modified to Emit Non-Visible
Frequencies 526.
[0245] Within all major components 1000, there are three
characteristics of markers that are categorized as follows. The
possible Marker: Emission Method 530 is retroreflective 532,
reflective 534 or fluorescent 536. The possible Marker: Physical
Form 540 is spherical (attached ball) 542 or flat (embedded/applied
ink) 544. And finally, the Marker: Reflective Shape 550
corresponding to the Marker: Physical Form 540 is uniform circular
552 or non-uniform multi-shape 554, respectively.
[0246] Within components 1000, there are three characteristics of
ID (identification markers) categorized as follows. The possible
ID: Location 560 is on the full body 562 or top surface of body
564. The possible ID: Encoding Method 570 corresponding to the ID:
Location 560 is a unique constellation 572 or encoded markings 574.
And finally, the possible ID: Obtained 580 is during game surface
tracking 582 or outside of game surface tracking 584. The last
group of components 1000 is the Calibration Method 590 that can be
either pre-tracking 592 or simultaneously with tracking 594.
[0247] Referring now to FIG. 13a, there is shown an example of a
fixed (X, Y, Z) volume tracking camera 502 that comprises a camera
126, filter and connection to a local computer system for video
processing and analysis 160. Camera 126 can be one of any analog or
digital-imaging cameras as typically used for industrial vision
applications. One example is the Eagle digital camera used by
Motion Analysis Corporation that features a ceramic metal oxide
semiconductor (CMOS) image sensor with 1280.times.1024 pixel
resolution and a maximum capture rate of 600 million pixels per
second. It is important to note that, once in place, this volume
tracking camera 126 has a fixed field-of-view (FOV) similar to a
four-sided pyramid in shape within an image cone 121v. The actual
pixel resolution per inch of the FOV will vary throughout the
height 121h of the pyramid ranging from a higher value at the top
width 121tw to a lower value at the bottom width 121bw. These
cameras are typically secured from an overhead position to have a
perspective view arrangement 502m of the desired tracking volume as
shown in FIG. 13b.
[0248] Referring now to FIG. 13b, there is shown one particular
arrangement of fixed (X, Y, Z) cameras 502 that, when taken
together, form a uniquely shaped tracking volume through which a
player 17, wearing markers such as spherical markers 17sm, may
transverse. The resultant resolution per cross-sectional area of
this volume 121tv is non-uniform. For example, while skating
through any given point in the tracking volume, markers 17sm on one
body part of player 17 may be viewed by camera 126e with a much
lower resolution per inch than similar markers on a different body
part. Also, the second camera such as 126d may have a much
different pixel resolution of marker 17sm than camera 126e. Cameras
126a, 126b and 126c may each have obstructed views of marker
17sm.
[0249] Referring now to FIG. 14a, there is shown an example of
fixed (X, Y) area tracking camera 504, that comprises a tracking
camera 124 with a filter 124f that have been enclosed in a
protective housing 121 with a transparent underside 121a. Also
enclosed in housing 121 is an energy source 10 emitting tracking
energy 11 as well as unfiltered filming camera 125. Tracking camera
124 and filming camera 125 are connected to a local computer system
for video processing and analysis 160. The entire assembly included
within housing 121 is preferably secured in an overhead position
looking directly down at a subset of the tracking surface. From
this overhead position, camera 124 has a fixed FOV 120v that is
focused on the top surface of any players below and as such
maintains a substantially uniform pixel resolution per tracking
area FOV 120v.
[0250] Referring now to FIG. 14b, there is shown a scalable area
tracking matrix 504m comprising multiple fixed (X, Y) area tracking
cameras 120c aligned such that their FOVs 120v are substantially
side-by-side with a small overlap for calibration purposes.
Throughout this scalable matrix 504m, the top surface 110 of player
17 can be readily tracked.
[0251] Referring now to FIG. 15a, there is shown an example of
movable (X, Y, Z) volume tracking camera 506, that comprises a pan,
tilt and zoom camera 140 with a filter that is connected to local
computer system for video processing and analysis 160. Top surface
110 of player 17 is held in constant view by one or more of cameras
140 that are controllably panned, tilted and zoomed for maximum
desirable pixel resolution per player. The information for this
controlled movement is based either upon the current (X, Y)
coordinates of player 17 as previously determined from information
gather by scalable area tracking matrix 504m or by movement
tracking algorithms calculated by computer 160 to predict the next
possible location of player 17.
[0252] Referring now to FIG. 15b, there is shown a scalable volume
tracking matrix 506m comprising multiple movable volume tracking
cameras 506 where one or more cameras form an assembly and are
dynamically assigned to a player 17. As will be explained in more
detail using FIGS. 16a and 16b, this dynamic process of
automatically panning, tilting and zooming each movable camera to
maintain the maximum desirable pixel resolution per player provides
a substantial benefit over the arrangement of fixed volume tracking
cameras 502.
[0253] Referring now to FIG. 16a, there is shown a series of three
views depicting the top surface 110x of player 17 (i.e., player 17
with absorbers and markings applied) at close range, mid-range and
far range with respect to a fixed volume tracking camera 126. As
can be seen, the pixel resolution per the player's body surface
area is substantially different between close and far ranges.
However, FIG. 16a also shows a series of three views the same
player 17 and relative locations but now with respect to movable
volume tracking camera 140. As can be seen, the pixel resolution
per the player's body surface area is now substantially
uniform.
[0254] Referring now to FIG. 16b, there is shown an example matrix
of four FOV's 120' created by area tracking cameras 124. Within
this combined grid, several players having top surfaces such as
110x and 111x move freely about. In this particular example, four
movable cameras 140-a, 140-b, 140-c and 140-d are tracking the
player with top surface 110x. As depicted, the FOV's for cameras
140-b and 140-d are almost fully blocked by other players whereas
the FOV for camera 140-a is partially blocked but the FOV for
camera 104-c is clear. The preferred embodiment will automatically
reassign cameras such as 141-d that may already be tracking another
player, (e.g., the player with top surface 111x) to now follow a
different player with top surface 110x so as to ensure total
maximum player visibility. This reassignment decision can be based
upon the information gathered by the scalable area tracking matrix
504m, predictive calculations made by computer 160 concerning the
expected next positions of any and all players, or both.
[0255] Referring now to FIG. 17, there is shown an example of three
different Tracking Frequencies 510 being emitted by normal or
modified HID lamp 10. These include UV ray 11, visible ray 12 and
IR ray 13. As these rays 11, 12 and 13 strike reflective material
20 attached to substrate 30, they will cause reflected UV ray 11r,
visible ray 12r and IR ray 13r.
[0256] Referring now to FIGS. 18a, 18b and 18c, there is shown an
example of three different Marker: Emission Methods 530 caused by
reflective material 20a, retroreflective material 20b and
fluorescent material 20c. In FIG. 18a, lamp 10 emits rays 11, 12
and 13 which are then reflected off reflective material 20a in a
diffuse manner causing rays r1. In FIG. 18b, emitted rays 11, 12
and 13 are retroreflected off retroreflective material 20b in a
manner causing rays r2. In FIG. 18c, emitted rays 11, 12 and 13 are
first absorbed by fluorescent material 20c causing emitted rays r3.
Reflective material 20a and fluorescent material 20c have an
advantage over retroreflective material 20b in that their reflected
and fluoresced rays r1 and r3, respectively, will have a wider
viewing angle than retroreflected rays r2. Retroreflective material
20b has an advantage over materials 20a and 20c because its rays r2
will be of stronger combined energy for a longer distance.
Fluorescent material 20c has an advantage over materials 20a and
20b because it can absorb visible light readily available in larger
intensities within the ambient environment and convert this to a
non-visible tacking energy such as IR.
[0257] Referring now to FIG. 19, there is shown a novel
retroreflective material 100 that is similar to commercially
available cube-cornered or microbead retroreflectors except that it
has been modified to be transparent to any energies that are not
intended to be retroreflected. In the case where the tracking
frequency 510 is chosen to be UV light 516, HID lamp 10 is shown to
emit UV ray 11 that enters retroreflective element 20uv that is
coated with UV reflector 24uv. Reflector 24uv then reflects ray 11
back up through element 20uv becoming retroreflected ray 11r.
Visible ray 12 and IR ray 13 will pass through reflector 24uv. In
the case where the tracking frequency 510 is chosen to be IR light
514, HID lamp 10 is shown to emit IR ray 13 that enters
retroreflective element 20ir that is coated with IR reflector 24ir.
IR Reflector 24ir then reflects ray 13 back up through element 20ir
becoming retroreflected ray 13r. Visible ray 12 and UV ray 11 will
pass through IR reflector 24ir. Retroreflective elements 20uv and
20ir are embedded within binder 28 that is attached to substrate
30. Binder 28 is substantially transparent to UV ray 11a and IR ray
13a.
[0258] Referring now to FIG. 20a, there is shown an example of the
first of the two Marker: Physical Forms 540, namely spherical
(attached ball) 542, also referred to as 17sm. Spherical marker
17sm comprises a retroreflective sphere 17s that is attached to a
base 17b. Companies such as Motion Analysis and Vicon currently use
this type of marker. The typical retroreflective sphere 17s
retroreflects a broad spectrum of frequencies including UV ray 11,
visible ray 12 and IR ray 13 causing retroreflective rays 11a, 12a
and 13a respectively. These retroreflective rays 11a, 12a and 13a
then create resulting circular image 17c that is incident upon any
tracking cameras such as 124, 126 and 140. Image 17c is an example
of one of the two Marker: Reflective Shapes 550 for a marker,
namely uniform circular 552.
[0259] Referring now to FIG. 20b there is shown an example of the
first of two ID:Locations 562 for the player ID, namely full body
562. In FIG. 20b, spherical markers 17sm are placed at various key
locations over the entire body of player 17. For practical purposes
retroreflective tape 17t is used to cover the blade of stick
104.
[0260] Referring now to FIGS. 21a, 21b and 21c, shown are several
examples of the second of two Marker: Physical Forms 540, namely
flat (embedded/applied ink) 544. In FIG. 21a, right and left
tracking patches 107r and 107l are shown attached to player
shoulder pads 106 that are typically covered by jersey 105. Patches
107r and 107l have been pre-marked with special ink formulated to
reflect, retroreflect, or fluoresce only the desired tracking
energy. Such pre-markings include orientation marks 107r1 and 107l1
as well as bar code marks 107r2 and 107l2. In FIG. 21b, puck 103
has been coated with similar special ink 103a while the blade of
stick 104 has been wrapped with reflective tape 104a. And finally,
in FIG. 21c, sticker 109 has been applied to helmet 108 and
comprises a uniquely identifying mark created with similar special
ink. Each of FIGS. 21a, 21b and 21c illustrate retroreflected ray
11r.
[0261] Referring now to FIG. 22a, the information depicted in FIG.
20b is repeated to dramatize FIG. 22b that depicts the image formed
in computer 160 based upon uniform circular 552 reflections. This
distinct formation of marker reflections 17c can be used to
identify player 17 and is the first of two ID:Encoding Methods 570
called unique constellation 572. Companies such as Motion Analysis
and Vicon use the unique constellation 572 method for identifying
human objects such as player 17. Furthermore, these same systems
are designed to identify the human object while they are also
tracking their motion. This is the first way the player ID is
Obtained 580, namely during game surface tracking 582.
[0262] Referring now to FIG. 22c, there is shown information
similar to FIGS. 21a, 21b and 21c to dramatize FIG. 22d that
depicts the image formed in computer 160 based upon non-uniform
multi-shape 554 reflections. This collection of individual markings
17m that have been placed at various locations on player 17 are
only used to locate a particular body part and its orientation
rather than to identify the player 17. In the preferred embodiment
that employs these types of flat 544 markings, the identification
of player 17 is based upon a top surface of the body 564 Id
Location 560.
[0263] Referring now to FIGS. 23a, 23b, 23c and 23d there is
dramatized the problems inherent with full body 562 unique
constellation 572 player identification. In FIG. 23a, there is
shown two players 17 and 18 that are each pre-marked with a unique
constellation of spherical markers 17sm. The view of players 17 and
18 is not overlapping in FIG. 23a. The resultant image detected by
computer 160, namely of circular reflections 17c and 18c, is shown
in FIG. 23b. Reflections 17c and 18c are also not overlapping.
Referring now to FIG. 23c, players 17 and 18 are now overlapping
causing the resultant overlapping of reflections 17c and 18c as
shown in FIG. 23d. Note the considerably more difficult
identification problem presented to computer 160 as players such as
17 and 18 begin to block each other's view in one or more volume
tracking cameras such as 126 or 140.
[0264] Referring now to FIG. 24, there is shown a set of preferred
helmet stickers 109id64, 109id00, 109id14 and 109id13 implementing
the uniquely encoded markings 574 method of ID:Encoding Method 570.
The markings on stickers 109id64, 109id00, 109id14 and 109id13 are
created using the special ink formulated to reflect, retroreflect
or fluoresce preferably only the chosen Tracking Frequency 510. The
player id is preferably implemented as a traditional bar code and
could be embedded on the helmet stickers 109id64, 109id00, 109id14
and 109id13 in a non-visible IR or UV reflective, retroreflective
or fluorescent ink. Also depicted is helmet sticker 109tm that
includes a special tracking mark designed to help computer 160 both
locate helmet 108 as well as determine its orientation. This
special tracking mark may be created using either a non-visible IR
or UV reflective, retroreflective or fluorescent ink. Sticker
109id&tm combines both the id marks as well as the tracking
marks, and can either be created using the same non-visible
frequency, such as both IR or both UV, or different frequencies,
such as one IR and the other UV. Note the considerably simpler
identification problem presented to computer 160 as it analyzes
helmet stickers such as 109id&tm viewed by area tracking
cameras 124. Cameras such as 124 are looking down upon the top
surface of the bodies of players such as 17 and 18 and are
therefore not expected to experience information degradation due to
player overlapping. Stickers 109id64, 109id00, 109id14 and 109id13,
109tm and 109id&tm are represented generically as sticker 109
that is shown attached to helmet 108.
[0265] Referring now to FIG. 25, there is shown the second way in
which the player ID is Obtained 580, namely outside of game surface
tracking 584. Rink entrance and exit 12e as team bench 12f are in
constant view of one or more area ID & tracking cameras similar
to 124 except with a narrowed FOV 122v. Narrowing FOV 122v provides
an increased pixel resolution per inch when looking down upon the
players' helmets 108 and attached stickers such as 109tm&id.
The increased pixel resolution allows for more complex encoding,
i.e. patterns with smaller markings on the limited space of the
helmet sticker 109. The rink playing surface 102 is in constant
view of the scalable area tracking matrix 504m comprising multiple
cameras 124 with normal FOV's 120v.
[0266] Also shown are a single set of four movable volume tracking
cameras 140-a, 140-b, 140-c and 140-d that are for example
currently assigned to track the top surface 110 of a player
starting when he first enters the playing surface 102 from the
entranceway 12e. Tracking with cameras 140-a, 140-b, 140-c and
140-d continues as the player transverses surface 102 and ceases
when player exits surface 102 and enters team benches 12f. Once
within bench area 12f, area ID & tracking cameras similar to
124 track the player and also reconfirm the player's identity by
viewing helmet sticker 109tm&id. At any time, the player may
subsequently leave bench area 12f and reenter surface 102 where
again his motion is tracked by movable volume cameras 140-a, 140-b,
140-c and 140-d. Eventually, the player will either exit the
playing area through entrance and exit 12e or return again to bench
12f and be tracked and re-identified by the ID & tracking
cameras.
[0267] Referring now to FIG. 26a, there is shown the first type of
Calibration Method 590, namely pre-tracking 592. Companies such as
Motion Analysis and Vicon currently perform this method in order to
calibrate their fixed volume tracking cameras 126 after they have
been set into place. The calibration tool 130 comprises two or more
markers such as various-sized spherical balls 17sm whose dimensions
are pre-known and that are affixed on the tool 130 at pre-known
distances from each other. The calibration process begins when tool
130 is held up within the FOV of two cameras 126. Computer 160
receives images from each of these first two cameras and processes
individually the reflected circles from the calibration tool 130.
Using stereoscopic algorithms that are well known in the art, the
locations of each spherical marker 17sm on tool 130 are calculated
within a local coordinate system. The operator holding the tool 130
then moves it into the view of a third camera 126 while still being
in view of one of the two prior cameras 126. This technique is
continued until all of the fixed cameras 126 have been individually
added to the calibration of all previous cameras 126.
[0268] The present inventors anticipate that this same technique,
although it would not be ideal, could be used to pre-calibrate the
scalable area tracking matrix 504m. In consideration of area matrix
504m, the relative orientation of each camera 124 is primarily
side-by-side with its neighbors, allowing for a small overlap on
the edges of its FOV 120v. Furthermore, the preferred orientation
of FOV 120v is "top down," rather than the "perspective" view of
cameras 126. Given these arrangements, a preferable pre-tracking
calibration technique would be to use a traditional calibration
plate incorporating a fixed set of markings held at pre-known
distances from each other. This plate would then be held in a fixed
position facing up at the junctions between every two cameras 124
overlapping FOV's 120v. Again, using standard techniques well known
in the art, each of the two cameras could then be jointly
calibrated by computer 160. Proceeding throughout all camera
junctions in the same fashion would complete the calibration of the
network to itself. The only remaining task would be to calibrate
the entire matrix 504m to the playing surface 102, entrance and
exit 12e and team benches 12f. This could be accomplished by
placing a marking at a fixed pre-known location somewhere within
each of the areas of surface 102, entrance and exit 12e and team
benches 12f. Once captured by computer 160 through one or more
cameras within matrix 504m, these markings at pre-known locations
would serve to register the entire matrix.
[0269] Now referring to FIG. 26b, there is shown the second type of
Calibration Method 590, namely simultaneously with tracking 594.
This process begins after the scalable area tracking matrix 504m is
itself pre-calibrated as described in the previous paragraph. Once
each overhead camera 124, within assembly 120c has been calibrated,
it will be used as the basis for the dynamic re-calibration of
movable cameras 140 as they continually change their orientation
and FOV. After calibration, each camera 124 will have a fixed (X,
Y) coordinate system registered with the playing surface 102,
entrance and exit 12e and team bench 12f. Calibration simultaneous
with tracking 594 begins when a player 17 enters the view of at
least one area tracking camera 124 and is therefore detected by
computer 160. The markings that computer 160 will be viewing based
upon camera 124 will be those on the top surface of the body 574
including the helmet sticker 109tm&id. Stickers such as
109tm&id are similar to calibration tool 130 in that their
markings are pre-known in both size and orientation to each
other.
[0270] As depicted in FIG. 26b, at least one point on sticker
109tm&id that is in view of both fixed pre-calibrated camera
124 and movable camera 140 is first located in local rink (X, Y)
coordinates based upon information provided by camera 124. Once
located, the same point is analyzed by computer 160 from the images
captured by camera 140 along with other measurable information such
as the current rotations of the panning and tilting mechanisms
supporting camera 140 as well as the zooming mechanism associated
with its lens. During analysis, the determined (X, Y) location of
the captured point is used to center the (X, Y, Z) coordinate
system of camera 140. Once centered, the (Z) height scale can be
set and then used to apply to all other common points in view of
both the (X, Y) camera 124 and the (X, Y, Z) camera 140. These
points include not only those on helmet sticker 109tm&id but
also those throughout all the body of player 17.
[0271] Furthermore, it is expected that additional volume cameras
140 assigned to track the same player 17 will similarly be
simultaneously calibrated with camera 124. It should be noted that
player 17 may be straddling a boundary between area tracking
cameras 124 and as such two different volume cameras 140 may
actually be calibrated for the same player 17 by two different area
cameras 124. In practice, this is immaterial since the
pre-calibration by system 160 of the entire scalable area tracking
matrix 504m can be thought of as creating one large single area (X,
Y) tracking camera. Hence, it can be seen that each of the volume
cameras such as 140 in the present figure or 140-a, 140-b, 140-c
and 140-d of prior figures that are currently assigned to follow
player 17 are simultaneously calibrated frame-by-frame to the
overhead matrix 540m. Furthermore, once calibrated the multiple
cameras such as 140-a, 140-b, 140-c and 140-d may be used to
stereoscopically locate markings on player 17 that are not in view
of the overhead matrix 540m.
[0272] Referring now to FIG. 27a, there is shown an alternate
embodiment 120b to area (X, Y) tracking camera assembly 120c that
does not include additional overlapping filming camera 125. In this
alternate embodiment, enclosure 121 houses lamp 10 and tracking
camera 124 with visible light filter 124f and is enclosed on the
bottom surface by transparent cover 121a through which tracking
energy 11 may transmit. Alternate embodiment of area tracking
camera assembly 120b is also connected to computer 160 (not
depicted) that is in turn connect to video terminal 127 via cable
121c. Further shown is player 17 to which tracking patches 107r and
107l and helmet sticker 109 have been attached. Also shown are puck
103 with reflective ink 103a and stick 104 with ink 104a. Shown on
terminal 127 is camera image 128 that includes player 17. The body
of player 17 is portrayed as dimmed due to some reflectance of the
non-visible tracking energy while the patches, stickers and ink are
portrayed as white due to their higher engineered reflectance.
[0273] Referring now to FIG. 27b, there is shown an identical
arrangement to FIG. 27a except that player 17 has been first
treated with one or more tracking energy absorbent compounds after
which tracking patches, stickers and inks were applied. Similarly,
stick 104 has also been first treated. As such, player 17 has now
become 17 while stick 104 has become 104t. Due to this novel
application of energy absorbers, treated player 17 is no longer
visible on terminal 127. Terminal 127 displays camera image 128
provided by computer 160 in response to the images captured by the
various tracking cameras 124, 126 and 140.
[0274] The present inventors have listed many absorbers and
treatments that may be used especially to absorb UV frequencies in
the prior co-pending application entitled Employing Electromagnetic
By-Product Radiation in Object Tracking. Such treatments and
absorbers are also well known for the IR frequencies to someone
skilled in the art. Further, the present inventors have shown that
it may also be similarly beneficial to apply energy reflectors
rather than absorbers, especially with respect to the background
such as player surface 102. What is important is that the intensity
of the reflected signal off the tracking marks be clearly
distinguishable from any reflections off the background or
foreground (player's body and equipment). To gain this clarity of
signal differentiation, it may be desirable to either reduce
reflections through absorption or to increase reflection through
reflective materials. This concept of absorbers and reflectors for
the control of the signal-to-noise ratio was not listed in system
1000 as a separate component since the present inventors see it as
a beneficial optimization to every possible combination of
components listed in 1000.
[0275] Referring now to FIGS. 28a and 28b, there is shown an
advantageous novel modification to the preferred embodiment that
employs a scalable area-tracking matrix 504m along with a scalable
movable volume-tracking matrix 506m. Specifically, that portion of
the scalable area-tracking matrix 504m that was in place to track
players such as 17 while they moved about the playing surface 102
has been eliminated. The reduction represents a saving in system,
installation and maintenance costs.
[0276] Referring specifically to FIG. 28a, this is made possible by
the understanding that as player 17 with top surface 110 passes
through entrance 12e he will first still be viewed by the tracking
cameras 124 left in place to cover this area, through their FOV's
122v. These cameras will first identify player 17 and then follow
his movements up until he enters player surface 102. As player 17
enters surface 102, the computer 160 will automatically direct
cameras 140-a, 140-b, 140-c and 140-d to pick up player 17.
[0277] Referring specifically to FIG. 28b, as player 17 with top
surface 110 is first viewed, computer 160 will be constantly
calculating and revising its prediction of the player's next
movements and therefore whereabouts. Cameras 140-a, 140-b, 140-c
and 140-d will continuously pan, tilt and zoom to follow the travel
of player 17. Eventually, player 17 will leave surface 102 and
enter team bench 12f where he will again be in the view of tracking
cameras 124 left in place to cover this area, through their FOV's
122v. Computer 160 will constantly monitor player 17 even while he
remains on team bench 12f. At some point throughout the
competition, player 17 is expected to re-enter the playing surface
102. At this time computer 160 will automatically direct cameras
140-a, 140-b, 140-c and 140-d to follow the player's travel until
he either returns to team bench 12f or leaves through entrance and
exit 12e.
[0278] Referring now to FIG. 29a, there is shown another
advantageous modification to the preferred embodiment for marking
foreground objects such as jersey 105. Specifically, there are
three distinct areas of the foreground object for which it is
desirable to have clearly distinguishable reflected intensity
levels of the tracking energy UV or IR. The first area is the
jersey 105 itself. Second, there is the base 107 of tracking patch
107r and third, there is the tracking mark 107c of patch 107r.
Correspondingly, there are three appliques described in the present
invention that can create the desired distinguishable reflected
intensity levels: (1) UV and IR absorbent compounds, (2) UV and IR
reflective compounds, and (3) UV and IR retroreflective compounds.
One possible arrangement of these compounds is to first apply the
absorber to jersey 105, the reflective compound to base 107 and the
retroreflective compound to tracking mark 107c. Two other
combinations thought to be particularly useful are: (1) Using the
retroreflective compound on base 107 and using more absorber for
mark 107c; and (2) Using the reflective compound on jersey 105, the
absorber on base 107 and the retroreflective compound on mark 107c.
Other combinations are anticipated in combination with the
teachings of the present invention.
[0279] Referring now to FIG. 29b, there is shown another
advantageous modification to the preferred embodiment for creating
a dynamic calibration tool that can be used to help calibrate the
movable volume tracking matrix 506m simultaneously with tracking
594. Specifically, precisely measured and spaced track markings
104m have been placed onto stick 104. The exact size, shape and
spacing of these markings 104m are immaterial to the concept being
presently taught. Placing these markings upon a rigid object that
is used by each player during the game will provide computer 160
with a way to verify its calibration estimates of both fixed area
tracking camera 124 and especially movable volume cameras 140.
Another possibility is to place similar markings onto the pipes of
the goals on either end of the playing surface.
[0280] Referring now to FIG. 30, there is shown a further
advantageous modification to the preferred embodiment for assisting
in the dynamic calibration of movable volume tracking matrix 506m.
Rink playing surface 102 is shown in perspective view in between
near boards 103nb and far boards 103fb. Attached to near boards
103nb are glass support columns such as 105nc that are holding in
place glass panes such as 105ng. Attached to far boards 1031b are
glass support columns such as 105fc that are holding in place glass
panes such as 105fg. Placed on both the outer (shown) and inner
(not shown) surface of near support columns 105nc are reflective
markings 105nm. Similarly, placed on both the inner (shown) and
outer (not shown) surface of far support columns 105fc are
reflective markings 105fm. Also shown are markings 103fm on the
inner side of far boards 103fb. Similar markings on the inner side
of near boards 103nb are also anticipated. Moving about on surface
102 are players 16 and 17 that are being constantly tracked by
movable cameras 140-b and 140-d as well as 141-b and 141-d,
respectively. During player tracking as cameras such as 140-b,
140-d, 141-b and 141-d pan, tilt and zoom to change their FOV's,
they will be constantly picking up one or more reflective marks
such as 105nm, 105fm and 103fm. The exact size, shape and spacing
of these markings 105nm, 105fm and 103fm are immaterial to the
novel concept being presently taught. Placing these markings upon
at least the locations specified will provide computer 160 with a
means of verifying its calibration estimates of movable volume
cameras 140.
[0281] Summary of Optimized Systems
[0282] A careful study of the for prior co-pending applications
filed by applicants and identified above along with the previously
described components 1000 will suffice to teach those skilled in
the art how each component may operate as a functional part of a
complete system. Therefore, the remainder of this application will
focus on distinguishing at a higher level the various possible
novel optimized systems according to the present invention along
with discussions as to their tradeoffs.
[0283] Referring now to FIG. 2, there is shown a block diagram
depicting all of the major components 1000 of which a subset has
been identified as representative of real-time 3D-object tracking
system 1002. System 1002 comprises fixed (X, Y, Z) volume tracking
assemblies 502 that employ either visible light 512 or IR light 514
that is emitted from ring lights 522. This tracking energy is then
reflected by spherical (attached ball) 542 retroreflective 532
markers that are placed at various locations on the subject to be
tracked and that create a uniform circular 552 image in camera
assemblies 502. The full body 562 set of spherical markers 542 form
a unique constellation 572 used by system 1002 to identify each
subject during game surface tracking 582. The system 1002 is
calibrated prior to tracking 592.
[0284] While system 1002 accomplishes the goals of real-time 3D
tracking, it has several drawbacks as follows:
[0285] 1--Due to its non-uniform approach to camera placement, it
is difficult to scale up to track larger areas such as a hockey
rink;
[0286] 2--Due to its strategy of fixed volume tracking accomplished
with a complex overlapping network of camera field-of-views, the
pixels resolution per player is inconsistent and the system is
prone to lose markers when multiple players bunch up;
[0287] 3--When using visible light as the tracking energy, the
additional red light is added by the system because of the need to
use ring lights in combination with the retroreflective markers
creating a lighting system that is intrusive to both players and
audience;
[0288] 4--When using in IR light as the tracking energy, the ring
lights do not emit any additional visible light in combination with
the IR to act as a cue to both players and audience not to stare at
the lighting;
[0289] 5--The system employs retroreflective markers in order to
obtain the highest possible signal reflection but these materials
are broad-band reflectors that respond to the entire visible
spectrum causing unwanted reflections of ambient light sources that
is intrusive to both players and audience;
[0290] 6--The retroreflective markers are constructed to be
spherical balls that protrude away from the body helping to ensure
maximum visibility to the tracking cameras by consistently creating
a circular reflection from any angle, however, this very nature of
their protruding physical form makes them vulnerable to dislodge
during player contact;
[0291] 7--The uniform circular nature of the retroreflection caused
by the spherical marker is useful for centroid calculations and
therefore determining exact body points, however, it necessarily
forces more cameras since less player surface area can be marked
for viewing using a spherical shape;
[0292] 8--By attempting to combine player identification with body
joint tracking the system creates a difficult requirement that
forces camera views away from the top down view that rarely
experiences inclusions due to player bunching to a perspective view
that is very susceptible to inclusions;
[0293] 9--By attempting to combine player identification with body
joint tracking the system creates a difficult requirement that
substantially all markers placed on the full body must be in view
in order to identify a given player;
[0294] 10--By attempting to combine player identification with body
joint tracking the system loses an opportunity to perform player
identification off the game surface in either the player entrance
and exit or on the team benches. (These areas are short on space
for adequate perspective camera placement and are also very crowded
with players who in the case of team benches are expected to be
sitting therefore additionally hiding markers);
[0295] 11--By attempting to combine player identification with body
joint tracking the system (a) creates a difficult requirement that
each player have either a substantially different body shape and
therefore configuration of markers or that additional markers be
added to create a unique pattern, and (b) makes using the system
with up to twenty some players per team cumbersome as a
pre-tracking procedure must be instituted to ensure adequate
"constellation" differentiation per player; and
[0296] 12--The system has no method of simultaneously calibrating
the cameras during tracking that precludes the possibility of using
movable volume tracking cameras that could dynamically reconfigure
the tracking volume to better create a uniform pixel resolution per
player and reduce the number of marker inclusions due to player
bunching.
[0297] Referring now to FIG. 3, there is shown a block diagram
depicting all of the major components 1000 of which a subset has
been identified as the preferred embodiment 1004 of the present
invention. Preferred embodiment 1004 comprises fixed (X, Y) area
tracking assemblies 504 in combination with movable (X, Y, Z)
volume tracking assemblies 506, that employ either IR light 514 or
UV light 516 that is provided by existing lights modified to emit
non-visible frequencies 526. This tracking energy is then reflected
by flat (embedded/applied ink) 544 reflective 534 markers that are
placed at various locations on the subject to be tracked and that
create non-uniform multi-shape 552 images in camera assemblies 504
and 506. Specially encoded 574 flat 544 markers placed on the top
surface of the body 564 are used by system 1004, to identify each
subject outside of game surface tracking 584. In system 1004, while
fixed (X, Y) area tracking assemblies 504 are calibrated prior to
tracking, movable (X, Y, Z) volume tracking assemblies 506 are
calibrated simultaneously with tracking 592.
[0298] The preferred embodiment 1004 accomplishes the goals of
real-time 3D tracking without the limitations of currently
available system 1002 providing the following advantages:
[0299] 1--By limiting the fixed (X, Y) area tracking matrix to a
top view only, the system creates a scalable approach to camera
placement that provides a substantially uniform pixel resolution
per area;
[0300] 2--By implementing a separate matrix of movable (X, Y, Z)
volume tracking cameras to pick up the remaining side views of the
players, the system creates a scalable approach to camera placement
that provides a substantially uniform pixel resolution per
player;
[0301] 3--By using either or both UV and IR light as the tracking
energy that is emitted as a by-product from lamp sources that are
also providing visible lighting for general purposes, the system is
both non-intrusive and eye safe;
[0302] 4--By using reflective as opposed to retroreflective markers
the cone of reflection is opened up such that separate ring lights
are not required whose added energy and visible light would be
intrusive to both players and audience;
[0303] 5--By using markers that reflect only the narrow band of
tracking energy and specifically do not reflect visible light, the
marker's reflections are hidden from player and audience view;
[0304] 6--By using flat markers that are embedded into the
substrate the marker is no longer vulnerable to dislodge during
player contact;
[0305] 7--By using flat markers of non-uniform sizes and shapes the
markers are made visible for more camera angles thereby reducing
the incidence of inclusions;
[0306] 8--By using flat markers of non-uniform sizes and shapes the
markers can be made to cover significantly larger surface area
thereby reflecting more of the tracking energy;
[0307] 9--By using visibly transparent flat markers embedded into
the substrate adding minimal additional thickness, the system's
markers will now be completely undetected by both the players and
the audience;
[0308] 10--By first strategically applying a combination of
tracking energy absorptive or reflective compounds to the
background as well as foreground objects the system provides the
ability for more clearly distinguishing between background
reflections, foreground object reflections and marker
reflections;
[0309] 11--By separating player identification from the player
joint tracking and isolating the identification marker to the top
surface of the player, the system ensures a higher rate of player
identification due to fewer inclusions of identification
markers;
[0310] 12--By separating player identification from the player
joint tracking and isolating the identification marker to the top
surface of the player, the system eliminates the importance of
having substantially all body joint markers in view at all
times;
[0311] 13--By separating player identification from the player
joint tracking and isolating the identification marker to the top
surface of the player, the system provides the possibility of
performing player identification off the playing surface in the
limited area of the entrance and exit and team benches where player
movement is expected to be significantly reduced thereby
facilitating the identification process;
[0312] 14--By separating player identification from the player
joint tracking and isolating the identification marker to the top
surface of the player that can be zoomed in on while in the
restricted movement areas of the entrance and exit and team
benches, encoded markings similar to bar codes become feasible,
which encode markings can easily handle forty or more players and
avoid any cumbersome pre-tracking procedure to ensure adequate
marker "constellation" differentiation per player;
[0313] 15--By establishing a separate fixed (X, Y) area tracking
matrix that may be pre-calibrated for the X and Y dimensions and by
implementing fixed size and shape relationship markers on rigid
surfaces such as the player helmet and stick that can be used as
calibration tools, the system provides dynamic calibration of
movable cameras simultaneously with player tracking;
[0314] 16--By providing dynamic calibration of movable cameras
simultaneous with player tracking, the system further provides the
ability to dynamically recreate the optimal tracking volume for
minimal marker inclusions;
[0315] 17--By establishing a separate fixed (X, Y) area tracking
matrix that continually locates each player in (X, Y) space the
system provides the ability to automatically direct the pan, tilt
and zoom aspects of one or more movable cameras to follow each
player; and
[0316] 18--By establishing a system that dynamically follows each
player with greater accuracy and fewer marker inclusions, the
system provides the ability to predict more accurately the limited
range of movement that could be expected from any player in the
next instant, which ability provides a second method for
automatically directing the pan, tilt and zoom aspects of one or
more movable cameras to follow each player.
[0317] Referring now to FIG. 4, there is shown a block diagram
substantially similar to the preferred embodiment 1004 except that
it employs components taught by the present inventors to provide a
fluorescent alternative embodiment 1006. Embodiment 1006
specifically employs existing lights emitting visible frequencies
524 that are absorbed by fluorescent 526 markers that in turn emit
IR light 514 for the tracking frequency 510. All other aspects and
benefits of alternate embodiment 1006 are identical to preferred
embodiment 1004.
[0318] Embodiment 1006 accomplishes the goals of real-time 3D
tracking, without the limitations of system 1002, and provides the
following additional advantages:
[0319] 1--By using fluorescent markers that absorb in the visible
region the system can rely fully upon existing rink lighting
without modifications to its emissions spectrum to supply the
tracking energy, and
[0320] 2--By using fluorescent markers that emit in the IR region
the system can remain visually transparent to players and
audience.
[0321] Referring now to FIG. 5, there is shown a block diagram
substantially similar to the preferred embodiment 1004 except that
it employs components to provide a visibly transparent
retroreflective alternative embodiment 1008. Embodiment 1008
specifically employs visibly transparent retroreflective 532a
markers in combination with ring lights 522 emitting either IR
light 514 or UV light 516 tracking frequencies 510. All other
aspects and benefits of alternate embodiment 1008 are identical to
preferred embodiment 1004.
[0322] Embodiment 1008 accomplishes the goals of real-time 3D
tracking, without the limitations of system 1002, and provides the
following additional advantage: using retroreflective markers that
reflect only the narrow band of tracking frequencies of IR or UV,
the system provides for greater reflected signal strength while
still remaining visibly transparent to both players and
audience.
[0323] Referring now to FIG. 6, there is shown a block diagram
substantially similar to the preferred embodiment 1004 except that
it employs components to provide a game surface 1d tracking
alternative embodiment 1010. Embodiment 1010 specifically uses
fixed (X, Y) area tracking assemblies 504 to read the encoded
marking 574 player id located on the top surface of the body 564
during game surface tracking 582. All other aspects and benefits of
alternate embodiment 1010 are identical to preferred embodiment
1004.
[0324] Embodiment 1010 accomplishes the goals of real-time 3D
tracking, without the limitations of system 1002, and providing the
following additional advantage: identifying players simultaneously
with game surface tracking the system provides the option of
eliminating separate area tracking cameras in the non-playing
surfaces of the entrance and exit passageway and team benches.
[0325] Referring now to FIG. 7, there is shown a block diagram
substantially similar to the system 1002 except that it employs
components to provide a non-visible variation 1012. Variation 1012
specifically uses the visibly transparent retroreflective markers
532a first taught by the present inventors in their co-pending
application rather than traditional visibly retroreflective markers
as currently used. Variation 1012 further limits the tracking
frequencies 510 to either IR light 514 or UV light 516 and employs
flat (embedded/applied ink) 544 non-uniform multi-shape 554
markers.
[0326] Variation 1012 incrementally improves upon the real-time 3D
tracking implemented by system 1002 by providing the following
additional advantages:
[0327] 1--By using markers that reflect only the narrow band of
tracking energy and specifically do not reflect visible light, the
marker's reflections are hidden from player and audience view;
[0328] 2--By using flat markers that are embedded into the
substrate the marker is no longer vulnerable to dislodge during
player contact;
[0329] 3--By using flat markers of non-uniform sizes and shapes the
markers are made visible for more camera angles thereby reducing
the incidence of inclusions;
[0330] 4--By using flat markers of non-uniform sizes and shapes the
markers can be made to cover significantly larger surface area
thereby reflecting more of the tracking energy, and
[0331] 5--By using visibly transparent flat markers embedded into
the substrate adding minimal thickness, the system's markers will
now be completely undetected by both the players and the
audience.
[0332] Referring now to FIG. 8, there is shown a block diagram
substantially similar to the non-visible variation 1012 except that
it employs additional components to provide a top surface of body
encoded id variation 1014. Variation 1014 specifically uses flat
544 non-uniform 554 top surface of body 564 encoded markings 574 to
establish each player's identification during game surface tracking
582.
[0333] Variation 1014 incrementally improves upon the real-time 3D
tracking implemented by non-visible variation 1012 by providing the
following additional advantages:
[0334] 1--By separating player identification from the player joint
tracking and isolating the identification marker to the top surface
of the player, the system ensures a higher rate of player
identification due to fewer inclusions of identification
markers,
[0335] 2--By separating player identification from the player joint
tracking and isolating the identification marker to the top surface
of the player, the system eliminates the importance of having
substantially all body joint markers in view at all times, and
[0336] 3--By separating player identification from the player joint
tracking and isolating the identification marker to the top surface
of the player, the system provides the possibility of performing
player identification off the playing surface in the limited area
of the entrance and exit passageway and team benches where player
movement is expected to be significantly reduced thereby
facilitating the identification process.
[0337] Referring now to FIG. 9, there is shown a block diagram
substantially similar to the top surface of body encoded id
variation 1014 except that it employs additional components to
provide outside of game surface 1d variation 1016. Variation 1016
specifically establishes each player's identification outside of
game surface tracking 584 using fixed (X, Y) area tracking
assemblies 504 at least in these restricted areas.
[0338] Variation 1016 incrementally improves upon the real-time 3D
tracking implemented by top surface of body encoded id variation
1014 by providing the following additional advantages:
[0339] 1--By separating player identification into the outside of
game surface areas such as the entrance and exit passageway and
team benches, the identification marker on the top surface of the
player that can be zoomed in on without affecting body marker
tracking while on the game surface. The possibility of zoomed
fields of view for the fixed volume cameras in these special areas
makes encoded markings similar to bar codes feasible. These encoded
markings can easily handle forty or more players and avoid any
cumbersome pre-tracking procedure to ensure adequate marker
"constellation" differentiation per player.
[0340] 2--By using fixed (X, Y) area tracking cameras at least in
the identification areas of the entrance and exit passageway and
team benches, total camera use is made more efficient. The top-down
orientation of the (X, Y) camera is better suited than the
perspective orientation of the (X, Y, Z) camera for zoom-in viewing
of the top surface where the encoded markings are located.
[0341] Referring now to FIG. 10, there is shown a block diagram
substantially similar to outside of game surface 1d variation 1016
except that it employs additional components to provide existing
light source variation 1018. Variation 1018 specifically employs
existing lights modified to emit non-visible frequencies 526 whose
tracking energy is returned by the markers using the reflective 534
Marker: Emission Method 530.
[0342] Variation 1018 incrementally improves upon the real-time 3D
tracking implemented by outside of game surface 1d variation 1016
by providing the following additional advantages:
[0343] 1--By switching to reflective markers as opposed to
retroreflective, the cone of reflected energy is greatly expanded
and can now be thought of as omni-directional thus eliminating the
need to keep the camera's lens in close proximately to the emitting
light source, and
[0344] 2--By using existing lights as the tracking energy source,
no additional energy is required and therefore added to the ambient
lighting that would among other problems raise the temperature and
add additional production, installation and maintenance costs to
the system.
[0345] Referring now to FIG. 11, there is shown a block diagram
substantially similar to existing light source variation 1018
except that it employs additional components to provide fluorescent
variation 1020. Variation 1020 specifically employs existing lights
emitting visible frequencies 524 that are absorbed by fluorescent
526 markers that in turn emit IR light 514 for the tracking
frequency 510.
[0346] Variation 1020 incrementally improves upon the real-time 3D
tracking implemented by existing light source variation 1018 by
providing the following additional advantages:
[0347] 1--By using fluorescent markers that absorb in the visible
region the system can rely fully upon existing rink lighting
without modifications to its emissions spectrum to supply the
tracking energy, and
[0348] 2--By using fluorescent markers that emit in the IR region
the system can remain visually transparent to players and
audience.
[0349] Referring now to FIG. 12, there is shown a block diagram
substantially similar to outside of game surface 1d variation 1016
except that it employs additional components to provide movable
volume tracking variation 1022. Variation 1022 specifically employs
movable (X, Y, Z) volume tracking assemblies 506 along with a
calibration method 590 that is performed simultaneously with
tracking 594.
[0350] Variation 1022 incrementally improves upon the real-time 3D
tracking implemented by outside of game surface 1d variation 1016
by providing the following additional advantages:
[0351] 1--Since both fixed area and fixed volume tracking
assemblies are pre-calibrated prior to tracking, by adding
pre-known markers to rigid surfaces such as the player's helmet and
stick or the boards and their glass support columns, the system is
now able to calibrate movable cameras simultaneously with tracking,
and
[0352] 2--By adding movable (X, Y, Z) volume tracking cameras that
can remain calibrated as they pan, tilt and zoom, the system can
automatically augment the combined FOV created by existing fixed
volume tracking cameras whenever anticipated player bunching is
expected to create an unacceptable level of marker inclusions.
[0353] In summary, FIG. 1 represents those components either in use
within currently available systems such as Motion Analysis or Vicon
or the corresponding novel components taught by the present
inventors.
[0354] At least the following components are considered to be novel
and first taught by the present inventors for use within a
multi-object tracking system:
[0355] 1. fixed (X, Y) area tracking cameras 504 as a Camera
Assemblies 500,
[0356] 2. movable (X, Y, Z) volume tracking cameras 506 as a Camera
Assemblies 500,
[0357] 3. UV light 516 as a Tracking Frequency 510,
[0358] 4. existing lights emitting visible frequencies 524 as an
Energy Source 520,
[0359] 5. existing lights modified to emit non-visible frequencies
526 as an Energy Source 520,
[0360] 6. retroreflective (visibly transparent) 532b as an Emission
Method for a Marker 530,
[0361] 7. reflective (visibly transparent) 534 as an Emission
Method for a Marker 530,
[0362] 8. fluorescent (visibly transparent) 536 as an Emission
Method for a Marker 530,
[0363] 9. flat (embedded/applied ink) 544 as a Physical Form for a
Marker 540,
[0364] 10. non-uniform multi-shape 554 as a Reflective Shape for a
Marker 550,
[0365] 11. top surface of body 564 as a Location for the
Identification 560,
[0366] 12. encoded markings 574 as an Encoding Method for the
Identification 570,
[0367] 13. outside of game surface tracking 584 as a time to Obtain
the Identification 580, and
[0368] 14. simultaneously with tracking 594 as a time to perform
the Calibration Method 590.
[0369] FIG. 2 represents the system 1002 that comprises a
combination of components known and taught for use within a
multi-object tracking system. FIG. 7 through FIG. 12 represent some
of the possible and useful variations of the system 1002 including
various of the additional components either first taught by the
present inventors or first considered for use within such
multi-object tracking systems by the present inventors. FIG. 3
represents the preferred embodiment 1004 for a multi-object
tracking system. FIG. 4 through FIG. 6 represent some of the
possible and useful alternate compositions of the system 1002 in
consideration of known and taught components being applied in a
novel way.
[0370] There are other novel components first taught by the present
inventors that are not specifically identified in any of the FIGS.
1 through 12. Three such important components are:
[0371] 1--the use of absorbers or reflectors to control the
reflectivity of background and foreground objects creating a clear
distinction between their detected energy intensities and that of
the tracking markers,
[0372] 2--the use of strategically placed markings on one or more
rigid objects moving about with the players such as their helmet
and stick in order to assist in the dynamic calibration of
especially the movable volume tracking cameras, and
[0373] 3--the use of strategically placed markings on one or more
surfaces of the background such as the boards or their glass
support columns to assist in the dynamic calibration of especially
the movable volume tracking cameras.
[0374] The use of these components is considered to be important
and equally applicable to any such system 1002 through 1022.
CONCLUSION, RAMIFICATIONS, AND SCOPE OF INVENTION
[0375] Thus the reader will see that the present invention
successfully:
[0376] 1--teaches the fundamental component groups necessary for a
multi-object real-time 3D object tracking system;
[0377] 2--identifies those individual components already in use
within currently available systems and to which component groups
they belong;
[0378] 3--teaches those novel components suggested by the present
inventors in this and their other four co-pending applications and
to which component groups they belong;
[0379] 4--teaches how the novel components allow systems to better
function in a live application with multiple colliding objects, for
instance a sporting event such as ice hockey;
[0380] 5--identifies the composition of multi-object real-time 3D
object-tracking systems in terms of actual components used from
each group;
[0381] 6--teaches a novel preferred embodiment for a multi-object
real-time 3D object-tracking system best suited for a live sporting
event such as ice hockey in terms of actual components used from
each group;
[0382] 7--teaches several novel alternative embodiments using one
or more components of the systems mixed into the preferred
embodiment, and
[0383] 8--teaches several novel variations using one or more of the
novel components mixed into the system.
[0384] Furthermore, the reader will also see that, for at least the
preferred embodiment and to a great extant its alternates as well
as the variations of the systems, the present inventors have taught
how to construct a system that:
[0385] 1--is scalable and therefore comprises uniform assemblies
that are combinable into a matrix designed to increase tracking
coverage in terms of area, volume or the number of objects while
still maintaining uniform performance;
[0386] 2--is minimally intrusive upon the objects to be tracked and
upon the surrounding environment especially if that environment is
a live setting;
[0387] 3--maximizes tracking signal-to-noise ratio;
[0388] 4--minimizes manufacturing and installation costs, and
[0389] 5--simplifies maintenance and operation for the user.
[0390] While the above description contains many details, these
should not be construed as limitations on the scope of the
invention, but rather as an exemplification of preferred
embodiments thereof. Many aspects of the system's functionality are
beneficial by themselves without other aspects being present as
will be appreciated by those skilled in the art. Furthermore, all
of the novel combinations of components taught have anticipated
application beyond that of the tracking of live sporting events.
Examples of other applications include but are not limited to the
tracking of human actors for the creation of animated film
sequences, the tracking of human subjects for medical research, as
well as other object tracking functions currently performed by
existing systems.
[0391] From the foregoing detailed description of the present
invention, it will be apparent that the invention has a number of
advantages, some of which have been described above and others that
are inherent in the invention. Also, it will be apparent that
modifications can be made to the present invention without
departing from the teachings of the invention. Accordingly, the
scope of the invention is only to be limited as necessitated by the
accompanying claims.
* * * * *