U.S. patent application number 12/896045 was filed with the patent office on 2011-03-24 for method and apparatus for positional error correction in a robotic pool system using a cue-aligned local camera.
Invention is credited to Michael GREENSPAN.
Application Number | 20110070960 12/896045 |
Document ID | / |
Family ID | 36096969 |
Filed Date | 2011-03-24 |
United States Patent
Application |
20110070960 |
Kind Code |
A1 |
GREENSPAN; Michael |
March 24, 2011 |
METHOD AND APPARATUS FOR POSITIONAL ERROR CORRECTION IN A ROBOTIC
POOL SYSTEM USING A CUE-ALIGNED LOCAL CAMERA
Abstract
The present invention provides a method and apparatus for
accurately positioning a robotic pool-playing device. The system
comprises a computer controlled robotic positioning device, such as
a gantry robot, that can position a cue over the pool table and
place a shot. A global camera is mounted on the ceiling looking
down at the table, and the acquired images are transmitted to the
computer for analysis to determine the identity and locations of
the balls within the table coordinate reference frame. The computer
also automatically determines which ball to strike. An aspect of
the invention is the use of a local camera, mounted on or near the
robotic end-effector in a fixed relationship with the cue, to
improve the positioning error of the robotic device prior to
placing a shot. By comparing the ball locations perceived from the
vantage of the local camera with the known ball locations
determined from the global camera image, the invention can
calculate the acquired robotic positioning error, which can then be
corrected for prior to placing the shot.
Inventors: |
GREENSPAN; Michael;
(KINGSTON, CA) |
Family ID: |
36096969 |
Appl. No.: |
12/896045 |
Filed: |
October 1, 2010 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
11233375 |
Sep 23, 2005 |
7831337 |
|
|
12896045 |
|
|
|
|
60612247 |
Sep 23, 2004 |
|
|
|
Current U.S.
Class: |
473/2 ; 473/45;
901/47 |
Current CPC
Class: |
B25J 9/1697 20130101;
A63D 15/00 20130101 |
Class at
Publication: |
473/2 ; 473/45;
901/47 |
International
Class: |
A63D 15/00 20060101
A63D015/00; A63D 15/08 20060101 A63D015/08 |
Claims
1. A method of playing pool, comprising the steps of: a) acquiring
a first image of a pool table using a first camera placed above the
pool table and positioned that its image plane is substantially
parallel to both a playing surface of the pool table and a
longitudinal axis of a pool cue and transmitting this image to a
computer processing means for analysis, the result of which is a
determination of the position and identity of each pool ball on the
pool table; b) said computer processing means planning a shot of a
cue ball by the pool cue by calculating a desired position of the
pool cue with respect to the cue ball in a pool table frame of
reference including a tip position of said pool cue and orientation
of the longitudinal axis of the pool cue; c) said computer
processing means instructing a robot connected to said pool cue to
position the pool cue to a preferred location from which to place a
shot; d) acquiring a second image of a pool table using a second
camera mounted on an end-effector of the robot so that its image
plane is substantially perpendicular to both the pool table and
said longitudinal axis of the pool cue, the position of said second
camera being fixed with respect to said longitudinal axis of the
pool cue, transmitting the second image to the computer processing
means and determining locations of some of the pool balls within
this image within said second camera frame of reference, and
wherein if at least three pool balls are visible from both the
first and second cameras, then estimating and correcting for an
accumulated positional error of the robot and repositioning the
pool cue by moving at least one proximal robotic joint which also
repositions the second camera; and e) said computer processing
means instructing the robot to place a shot.
2. (canceled)
3. (canceled)
4. The method according to claim 1 wherein the location of the
longitudinal axis of the cue is determined within a coordinate
reference frame of the second camera.
5. The method according to claim 21 wherein the location of the
longitudinal axis of the cue is determined within a coordinate
reference frame of the second camera.
6. The method according to claim 4 wherein the position of the
second camera is fixed with respect to the longitudinal cue axis,
and the location of the cue axis within the coordinate reference
frame of the second camera is determined in a preprocessing step
including capturing an image using a third camera mounted so that
its image plane is substantially perpendicular to the longitudinal
axis of the cue, and including re-positioning the pool cue with one
degree-of-freedom by sliding along the cue axis repeatedly and the
center of the cue tip within the third camera image is determined
at each extreme cue position, and wherein by comparing the centers
of the tip at each extreme position, computing the angle between
the cue axis and the third camera whereupon the robotic joints are
repositioned so as to improve the perpendicularity of the cue axis
with the third camera image plane, and the process iterates, and
once the cue is thus positioned so that its axis is accurately
perpendicular to the third camera image plane, the relative
locations of the third camera and the local camera are then
determined by estimating a rigid transformation that relates them,
and the cue location is then directly propagated to the local
camera coordinate reference frame.
7. The method according to claim 5 wherein the position of the
second camera is fixed with respect to the longitudinal cue axis,
and the location of the longitudinal axis of the cue within the
coordinate reference frame of the second camera is determined in a
preprocessing step including capturing an image using a third
camera mounted so that its image plane is substantially
perpendicular to the longitudinal axis of the cue, and including
re-positioning the pool cue with one degree-of-freedom by sliding
along the cue axis repeatedly and the center of the cue tip within
the third camera image is determined at each extreme cue position,
and wherein by comparing the centers of the tip at each extreme
position, computing the angle between the cue axis and the third
camera whereupon the robotic joints are repositioned so as to
improve the perpendicularity of the cue axis with the third camera
image plane, and the process iterates, and once the cue is thus
positioned so that its axis is accurately perpendicular to the
third camera image plane, the relative locations of the third
camera and the local camera are then determined by estimating a
rigid transformation that relates them, and the cue location is
then directly propagated to the local camera coordinate reference
frame.
8. The method according to claim 6 wherein the location of the cue
ball and object ball within the coordinate reference frame of the
local camera is determined in a preprocessing step, and an
estimation and correction of some components of an accumulated
positional error of the robot is substantially achieved by matching
the current positions of the cue and object balls with those
positions that were respectively determined in the preprocessing
step.
9. The method according to claim 7 wherein the location of the cue
ball and object ball within the coordinate reference frame of the
local camera is determined in a preprocessing step, and an
estimation and correction of some components of an accumulated
positional error of the robot is substantially achieved by matching
the current positions of the cue and object balls with those
positions that were respectively determined in the preprocessing
step.
10. (canceled)
11. The method according to claim 22 wherein locations of the
landmarks have been predetermined with respect to the first camera
frame of reference.
12. A method of playing pool, comprising the steps of; a) acquiring
an image of a pool table using a camera mounted on an end-effector
of a robot so that its image plane is substantially perpendicular
to a longitudinal axis of the pool cue, the position of said camera
being fixed with respect to the longitudinal axis of the pool cue,
transmitting the image to a computer processing means for analysis,
the result of which is a determination of the position and identity
of each ball on the pool table within this image within a frame of
reference of the camera, and correcting for any errors based upon
the relative position of the cue axis and the cue ball; b) said
computer processing means planning a shot of a cue ball by the pool
cue by calculating a desired position of the cue with respect to
the cue ball; and c) said computer processing means instructing the
robot connected to said pool cue to place a shot.
13-20. (canceled)
21. A method of playing pool, comprising the steps of: a) acquiring
a first image of a pool table using a first camera placed above the
pool table and positioned that its image plane is substantially
parallel to both a playing surface of the pool table and a
longitudinal axis of a pool cue and transmitting this image to a
computer processing means for analysis, the result of which is a
determination of the position and identity of each pool ball on the
pool table; b) said computer processing means planning a shot of a
cue ball by the pool cue by calculating a desired position of the
pool cue with respect to the cue ball in a pool table frame of
reference including a tip position of said pool cue and orientation
of the longitudinal axis of the pool cue; c) said computer
processing means instructing a robot connected to said pool cue to
position the pool cue to a preferred location from which to place a
shot; d) acquiring a second image of a pool table using a second
camera mounted on an end-effector of the robot so that its image
plane is substantially perpendicular to both the pool table and
said longitudinal axis of the pool cue, the position of said second
camera being fixed with respect to said longitudinal axis of the
pool cue, transmitting the second image to the computer processing
means and determining locations of some of the pool balls within
this image within a second camera frame of reference, and wherein
if less than three pool balls are visible from either or both of
the first and second cameras, then estimating and correcting for
some but not necessarily all components of the accumulated
positional error of the robot, and repositioning the pool cue by
moving at least one proximal robotic joint which also repositions
the second camera; and e) said computer processing means
instructing the robot to place a shot.
22. A method of playing pool, comprising the steps of: a) acquiring
a first image of a pool table using a first camera placed above the
pool table and positioned that its image plane is substantially
parallel to both a playing surface of the pool table and a
longitudinal axis of a pool cue and transmitting this image to a
computer processing means for analysis, the result of which is a
determination of the position and identity of each pool ball on the
pool table; b) said computer processing means planning a shot of a
cue ball by the pool cue by calculating a desired position of the
pool cue with respect to the cue ball in a pool table frame of
reference including a tip position of said pool cue and orientation
of the longitudinal axis of the pool cue; c) said computer
processing means instructing a robot connected to said pool cue to
position the pool cue to a preferred location from which to place a
shot; and d) acquiring a second image of a pool table using a
second camera mounted on an end-effector of the robot so that its
image plane is substantially perpendicular to both the pool table
and said longitudinal axis of the pool cue, the position of said
second camera being fixed with respect to said longitudinal axis of
the pool cue, transmitting the second image to the computer
processing means and determining locations of some landmarks within
this image within said second camera frame of reference, and then
estimating and correcting for an accumulated positional error of
the robot and repositioning the pool cue by moving at least one
proximal robotic joint which also repositions the second camera;
and e) said computer processing means instructing the robot to
place a shot.
Description
CROSS REFERENCE TO RELATED APPLICATION
[0001] This patent application relates to, and claims the priority
benefit from, U.S. Provisional Patent Application Ser. No.
60/612,247 filed on Sep. 23, 2004 entitled METHOD AND APPARATUS FOR
POSITIONAL ERROR CORRECTION IN A ROBOTIC POOL SYSTEM USING A
CUE-ALIGNED LOCAL CAMERA, and which is incorporated herein in its
entirety.
FIELD OF THE INVENTION
[0002] This invention relates generally to the field of robotically
controlled games, and more particularly the present invention
relates to a method and apparatus for robotically controlled pool
games.
BACKGROUND OF THE INVENTION
[0003] The cue sports include pool, billiards, and snooker, and
have recently enjoyed a surge in popularity worldwide. There have
been a number of efforts at developing automation aids for these
games. One example is the Instant Pool Trainer [Lar01], in which a
camera is mounted on the ceiling aimed down at the table. Acquired
images are transmitted to a computer and automatically analysed
using image processing techniques. The system makes suggestions to
the human trainee about the next shot to place, the desired angle
of the cue, etc.
[0004] Other systems have attempted to fully automate the play by
adding a robotic component [Nak01, Qi99, Shu94, Chu02, Ali04,
Lon04, Che04]. In addition to the ceiling-mounted camera, these
systems involve some form of computer-controlled robotic actuation
device that can position a cue to the correct location and place a
shot. The most common example of such robotic devices are gantry
systems [Ali04, Lon04, Che04, Shu94], the first of which was
proposed by Shu et al. [Shu94]. Other proposed robotic devices
include a mobile robot that moves around the perimeter of the table
and extends a cue-like end-effector to place a shot [Qi99], and a
mobile robot that moves over the surface of the table [Lar02].
[0005] The cue sports demand a high degree of positional accuracy
when placing a shot, and one of the main challenges of a robotic
system is to position the cue to the desired location with
sufficient accuracy. The exact positional accuracy that is required
to play well has not been reported in the literature, and is
presumed to be unknown, although it is likely to be on the order of
0.1 mm or finer. Whereas mechanical devices can be positioned very
precisely, both sensor errors and robot calibration contribute to
limitations to positioning accuracy of such systems.
[0006] In the above cases where overhead cameras are the primary
sensor to resolve position, a limitation to accuracy is sensor
resolution. Standard CCD cameras that are suitable for machine
vision applications will often have 640.times.480 pixels. If the
entire length of a standard pool table extends the complete 640
pixels, then this resolves to .about.4 mm/pixel, which is at least
an order of magnitude too coarse. Using higher pixel-count sensors,
or multiple cameras, are possible remedies. In the case of multiple
sensors, each of which images a smaller region of the table at a
higher magnification, combining the partial images acquired by each
individual sensor into a global coordinate frame requires accurate
calibration of the extrinsic camera parameters. Radial distortions
in the optical systems also limit the accuracy of the cameras. A
further limitation is that, from the overhead vantage, the table
appears as a 2-D plane, and vertical displacements of the cue
(i.e., normal to the camera place) cannot be perceived. Controlling
these vertical displacement to allow the system to strike the cue
ball high or low forms an important part of the play.
[0007] The main limitation to positional accuracy is calibration of
the robotic device [Lon04]. The proper calibration of robotic
devices to ensure positional accuracy is a well-known and
challenging problem. The majority of robotic devices are equipped
with joint encoders that very precisely measure the location of
each revolute or translational joint. Despite their precision,
converting these joint values into an accurate position of the
robotic end-effector is not straightforward, as there are a number
of factors that cannot be directly measured which affect the
overall accuracy. The majority of industrial robotic devices do not
require absolute positioning accuracy, so long as they are precise
and repeatable, so this limitation on accuracy does not present a
barrier to use in many cases. An exception where absolute
positioning accuracy is required are Coordinate Measurement
Machines (CMMs). These devices are finely machined and calibrated
so that they can be used in metrological inspection applications.
The delicate mechanisms used in CMM construction would
unfortunately not be able to withstand a significant load or
impact, such as striking a ball.
[0008] In human play, it is an accepted practice to accurately
align the cue prior to a shot by locating the eye closely to the
axis of the cue [Kan99]. From this vantage, the locations of both
the cue ball (which is to be impacted by the cue) and the object
ball (which is to be impacted by the cue ball) can be seen. Small
positional variations of the cue axis as well as of the tip of the
cue can be perceived accurately, as they are parallel to the
human's retinal plane. Conversely, motions that are perpendicular
to the retinal plane, such as those parallel to the cue axis, are
less easily resolved, and are fortunately less important to
accurate shot placement.
[0009] Therefore, there is a need to provide a robotically
controlled pool game which overcomes the aforementioned
shortcomings.
SUMMARY OF INVENTION
[0010] The present invention provides a method and apparatus for
accurately positioning a robotic pool-playing device. The system
comprises a computer controlled robotic positioning device, such as
a gantry robot, that can position a cue over the pool table and
place a shot. A global camera is mounted on the ceiling looking
down at the table, and the acquired images are transmitted to the
computer for analysis to determine the identity and locations of
the balls within the table coordinate reference frame. The computer
also automatically determines which ball to strike.
[0011] The invention may use a local camera, mounted on or near the
robotic end-effector in a fixed relationship with the cue, to
improve the positioning error of the robotic device prior to
placing a shot. By comparing the ball locations perceived from the
vantage of the local camera with the known ball locations
determined from the global camera image, the invention can
calculate the acquired robotic positioning error, which can then be
corrected for prior to placing the shot.
[0012] Thus, in one aspect of the invention there is provided a
method of playing pool, comprising the steps of: [0013] a)
acquiring an image of a pool table using a first camera placed
above the pool table and positioned that its image plane is
substantially parallel to both a playing surface of the pool table
and a longitudinal axis of a pool cue and transmitting this image
to a computer processing means for analysis, the result of which is
a determination of the position and identity of each pool ball on
the pool table; [0014] b) said computer processing means planning a
shot of a cue ball by the pool cue by calculating a desired
position of the pool cue with respect to the cue ball in a pool
table frame of reference including a tip position of said pool cue
and orientation of the longitudinal axis of the pool cue; [0015] c)
said computer processing means instructing said robot connected to
said pool cue to position the pool cue to a preferred location from
which to place a shot; and [0016] d) said computer processing means
instructing the robot to place a shot.
[0017] In another aspect of the present invention there is provided
a method of playing pool, comprising the steps of; [0018] a)
acquiring an image of a pool table using a camera mounted on the
robotic end-effector so that its image plane is substantially
perpendicular to a longitudinal axis of the pool cue, the position
of said camera being fixed with respect to the longitudinal axis of
the pool cue, transmitting the image to a computer processing means
for analysis, the result of which is a determination of the
position and identity of each ball on the pool table within this
image within a frame of reference of the camera, and correcting for
any errors based upon the relative position of the cue axis and the
cue ball; [0019] b) said computer processing means planning a shot
of a cue ball by the pool cue by calculating a desired position of
the cue with respect to the cue ball; and [0020] c) said computer
processing means instructing the robot connected to said pool cue
to place a shot.
[0021] The present invention also provides an apparatus for playing
pool, comprising; [0022] a) a robot having a robotic arm with an
end effector, the robot mounted on a gantry for movement above a
pool table, a pool cue attached to the end-effector, the pool cue
having a longitudinal axis; [0023] b) image capturing means mounted
above the pool table for acquiring an image of the pool table and
positioned such that its image plane is substantially parallel to
both a playing surface of the pool table and said longitudinal axis
of said pool cue; [0024] c) a computer processing means connected
to said image capturing means for analysis of the images
transmitted thereto from the image capturing means, the computer
processing means including means for determining a position and
identity of each pool ball on the pool table within this image
within a frame of reference of the image capturing means, and
including means for correcting for any errors based upon the
relative position of the cue axis and the cue ball, said computer
processing means including means for planning a shot of a cue ball
by the pool cue by calculating a desired position of the cue with
respect to the cue ball; and [0025] d) a robotic controller
connected to said computer processing means for instructing the
robot to place pool shots with the cue.
[0026] In this aspect of the invention the apparatus may include a
second image capturing means mounted on the robotic end-effector so
that its image plane is substantially perpendicular to the
longitudinal axis of the pool cue for capturing an image of a pool
table using a camera, the position of said camera being fixed with
respect to the longitudinal axis of the pool cue.
[0027] The present invention also provides an apparatus for playing
pool, comprising; [0028] a) a robot having a robotic arm with an
end effector, the robot mounted on a gantry for movement above a
pool table, a pool cue attached to the end-effector, the cue having
a longitudinal axis; [0029] b) image capturing means mounted on the
robotic end-effector so that its image plane is substantially
perpendicular to a longitudinal axis of the pool cue for capturing
an image of a pool table using a camera, the position of said
camera being fixed with respect to the longitudinal axis of the
pool cue, [0030] c) a computer processing means for analysis of the
images transmitted thereto from the image capturing means, the
result of which is a determination of the position and identity of
each ball on the pool table within this image within a frame of
reference of the camera, and correcting for any errors based upon
the relative position of the cue axis and the cue ball, said
computer processing means planning a shot of a cue ball by the pool
cue by calculating a desired position of the cue with respect to
the cue ball; and [0031] d) a robotic controller connected to said
computer processing means for instructing the robot to place pool
shots with the cue.
BRIEF DESCRIPTION OF DRAWINGS
[0032] The following is a description, by way of example only, of
the method in accordance with the present invention, reference
being had to the accompanying drawings, in which:
[0033] FIG. 1 is a diagram of a computer controlled pool game
showing the system components with labeled coordinate frames;
[0034] FIG. 2 shows the system components of the computer
controlled pool game;
[0035] FIG. 3 shows a top global view of the pool table as taken
from the vantage point of a first overhead camera;
[0036] FIG. 4(a) shows a first local camera view from a second
local camera showing expected and actual pool ball positions;
and
[0037] FIG. 4(b) shows a second local camera view from the overhead
camera.
DETAILED DESCRIPTION OF THE INVENTION
[0038] The apparatus and method of the present invention utilizes
at least one, or two machine vision systems. Illustrations of the
system are shown generally at 10 in FIGS. 1 and 2.
[0039] In one embodiment of the invention a single, global overhead
camera is used. In this embodiment the system 10 uses global
overhead camera 12 for acquiring an image of pool table 14 with
camera 12 being placed above the pool table 14 and positioned such
that its image plane is substantially parallel to both the pool
table 14 and a longitudinal axis 23 of pool cue 22 and transmitting
this image to a computer 16 for analysis, the result of which is a
determination of the position and identity of each ball 18 on the
pool table 14.
[0040] Another computer program is used to plan a shot, which
indicates a desired position of the cue 22 with respect to the cue
ball in the table frame. The gantry robot 26 mounted above the pool
table 14 is controlled by a robotic controller 32 that communicates
with a computer 16 and the cue 22 is attached to the end-effector
28 whereby the cue is robotically controlled.
[0041] The shot planning computer program analyses the positions of
the balls 18 on the table, and uses geometric and physics
computations to determine the likelihood of sinking each target
ball 18. Depending upon the arrangement of balls 18 on the pool
table 14, some balls may be impossible to sink, whereas other balls
may have a number of possible shots. The program not only considers
the current arrangement of balls 18 on table 14, but also predicts
the resulting arrangement of the balls 18 subsequent to placing
each shot. The determination of which shot to place will therefore
also include the most favorable positioning of remaining balls 18
in future shots.
[0042] Once the most favorable shot has been determined, the
desired cue 22 position to place that shot is transmitted from the
computer 16 to the robotic controller 32 to reposition the cue 22.
The desired cue position includes not only the tip position 25 of
cue 22, but also the orientation of the cue axis 23. The robotic
system is then invoked to reposition the cue to the desired cue
position, and place a shot.
[0043] One of the main challenges is that the pool games require a
high degree of positioning accuracy over a large area. It is
possible to design and calibrate a robotic system that is highly
accurate (such as a CMM machine), but for a variety of reasons
(ruggedness, cost) it is more desirable for us to use a standard
robotic gantry that is inherently inaccurate. As an example, it is
not uncommon for gantry robots to have accuracies only on the order
of 0.01 m, which is clearly far too coarse to play a reasonable
game of pool.
[0044] Thus, in repositioning the cue 22, the robotic system may
accumulate error, as illustrated in FIG. 3. The actual position of
cue 22 may therefore not be perfectly accurate, but rather includes
some degree of positional error, which will degrade the performance
of the system. In order to compensate for the accumulated
positional error, another embodiment the system comprises two
cameras, the global camera 12 mounted on the ceiling aimed down at
the table 14 and another camera 30 mounted on the end-effector (or
cue 22) so that its image plane is substantially perpendicular to
the cue axis 23. It is difficult to position the camera 30 so that
it is exactly perpendicular to the cue axis 23. It is also
unnecessary, as nearly perpendicular is sufficient.
[0045] Generally in this embodiment the apparatus for playing
billiards can accurately position the inherently inaccurate gantry
mechanism by making use of information from both vision systems 12
and 30. First, the camera 12 determines the ball identities and
locations accurately, and the planning system identifies a shot.
The gantry robot 26 then moves the cue 22 into position to place
the shot, and in doing so, accumulates error, so that the position
of the cue 22 is inaccurate. To improve the cue accuracy, the
camera 30 next acquires an image, and extracts the locations of
landmarks (i.e., balls) from its own vantage. By comparing the
location of the landmarks in the frame of reference of camera 30
with those in the frame of reference of camera 12, and given prior
knowledge of the cue location in the frame of reference of camera
30, the error in the gantry robot 26 position can be estimated, and
its position refined. In this way, the use of two cameras can be
used to accurately position an inherently inaccurate positioning
device.
[0046] Another possibility is to use landmarks (or targets) other
than the pool ball positions to estimate and correct for the robot
positioning error. In a preprocessing step, a number of targets,
such as identifiable planar patterns, can be positioned at specific
accurate locations with respect to the pool table. Their positions
can therefore be identified within the frame of reference of the
overhead camera 12. Once the cue 22 has been positioned, then the
target locations can be identified within the frame of reference of
the end-effector mounted camera 30, and the positional error of the
cue 22 can then be estimated.
[0047] We assume that all ball 18 and "robot position(s)" are
expressed in some global coordinate frame, which we shall call the
table frame 38. It will be appreciated by those skilled in the art
that due to the characteristics of the robotic mechanism, this
"robotic position(s)" can be equivalently expressed in other terms
without any ambiguity, i.e., as robot joint values, or as the
position of certain other points on the robotic end effector. We
therefore often talk about robotic positions in a slightly more
general sense.
[0048] The position of this camera 30 is fixed with respect to the
longitudinal axis 23 of the cue, and the location of the cue axis
23 within the coordinate reference frame 40 of the local camera 30
is determined in a pre-processing step, which uses a third camera
42 (shown in FIG. 1). This third camera 42 is mounted on a tripod
36 such that its image plane is also substantially perpendicular to
the cue axis 23. The cue 22 itself may be repositioned with one
degree-of-freedom by sliding along the cue axis 23, and this is
done repeatedly and the center of the cue tip 25 within the third
camera 42 image is determined at each extreme cue position. By
comparing the centers of the cue tip 25 at each extreme position,
the angle between the cue axis 23 and the third camera 42 is
computed. The robotic joints are then repositioned so as to improve
the perpendicularity of the cue axis 23 with the image plane of the
third camera 42, and the process iterates.
[0049] Once the cue 22 is thus positioned so that its axis 23 is
accurately perpendicular to the image plane of the third camera 42,
the relative locations of the third camera 42 and the local camera
30 (which are substantially parallel) are then determined by
estimating the rigid transformation that relates them, and the cue
22 location is then directly propagated to the local camera 30
coordinate reference frame.
[0050] The rigid transformation that relates the relative locations
of the third camera 42 and the local camera 30 can be determined by
first calibrating the intrinsic parameters of each of these two
cameras, and then acquiring images of a common planar target of a
known dimension from both cameras. The locations of features on the
planar target can be used to determine the rigid transformation
between the cameras, as disclosed in [Zha98].
[0051] After the cue 22 is repositioned, and before placing a shot,
an image is acquired from the vantage of the local camera 30. This
image is transmitted to the computer 16 and the locations of the
balls 18 within this image are determined within the frame of
reference of local camera 30. If at least three balls 18 are
visible from both the global camera 12 and local camera 30, then
the accumulated positional error of the robotic system can be
estimated and corrected for.
[0052] For each ball 18, a ray can be inscribed between the optical
center of one of the cameras 12 or 30 and the center of the ball
18. With three balls 18 and two cameras 12 and 30, there are a
total of six such rays, i.e., three rays for each camera. By
establishing the correct correspondence between pairs of rays from
each respective camera, the relative positions of both cameras,
with respect to each other can be determined. Equivalently, if the
positions of the balls 18 and the global camera 12 are known within
the table 14 coordinate reference frame, then the position of the
local camera 30 can be determined with respect to the balls 18.
[0053] If there are more than three balls 18 in the scene, then the
solution is over-determined, such that more than one solution
exists. In this case, the position of the local camera 30 can be
determined from the set of possible solutions using least squares
estimation methods, such as singular value decomposition. If there
are less than three balls 18, then the solution is underdetermined,
such that not all position parameters (i.e. dimensions) can be
solved. Even in the underdetermined case, it is still possible to
estimate and correct some dimensions of the positional error using
this technique. For example, with two balls 18, the positional
error can be corrected up to a twist around the cue axis 23. With
only one ball 18, the translational error can be corrected,
although the direction of the shot may still contain errors. In
these last two cases, it is also possible to reposition the
end-effector so that the local camera 30 which is rigidly mounted
to the end-effector is also repositioned so that the local camera
30 can view a sufficient number of balls 18. As small motions of
the robotic device only accumulate small amounts of error, the cue
22 can then be repositioned accurately to place the desired
shot.
[0054] An example of the process of error correction is illustrated
in FIGS. 4(a) and 4(b). FIG. 4(a) shows the true and desired ball
locations from the vantage of the local vision system image. It can
be seen that positional error causes the true ball positions to
differ from the desired ball positions. FIG. 4 (b) shows the true
ball locations after the positional error has been corrected. After
correction, the true ball positions align more accurately with the
desired ball positions, leading to a more successful shot.
[0055] Another possible way to make use of the local vision system
to improve accuracy is to first position the robot so that the cue
22 is at an ideal location with respect to the cue ball 18' and
object balls. In this position, a cue ball 18' when struck by the
cue 22 will then proceed to strike an object ball 18'' so that the
trajectories traversed by the cue ball 18' (prior to striking the
object ball) and the object ball 18'' (after it has been struck by
the cue ball 18') are collinear. This represents a straight shot,
and any other shot can be achieved by first orienting the cue 18 in
this position, and then perturbing it slightly to a desired angle
and offset. The locations in the local camera 30 reference frame of
the cue ball 18' and object balls 18'' for an ideal straight shot
can be recorded. In subsequent shots, the robot joints can be
positioned so that the cue hall 18' and object ball 18'' fall in
these previously recorded positions within the local camera 30
reference frame, thereby ensuring that the cue 22 is in the ideal
position for a straight shot.
[0056] Utilizing an end-effector mounted camera is known in the
robotic literature as an eye-in-hand system. There have been a
number of such systems proposed to improve robotic accuracy for
grasping and other operations, including some work that includes
combinations of both end-effector mounted and global cameras
[Fla00]. Despite the great interest and prior work in robotic pool,
the use of a global camera 30 parallel to the table 14 plane in
combination with a local camera 30 perpendicular to the cue axis 23
has not been previously proposed, and is advantageous and novel to
this invention.
[0057] Another embodiment of the present invention utilizes only
the local camera 30 to determine the quality of the cue 22 position
solely with respect to the vantage of the local camera 30 image. In
this application, the cue 22 is controlled by either a human or a
robot, and the image would be transmitted, possibly wirelessly, to
a computer. The cue 22 position is determined with respect to the
imaged balls 18 as either good quality or poor quality, and this
information is communicated back to the positioning device for
refinement. While the vantage of the local camera 30 alone is more
limited than that of the combination of local camera 30 and global
camera 12, the flexibility of providing a self-contained cue 22
allows it to be used for human training as well as robotic
play.
[0058] As used herein, the terms "comprises", "comprising",
"including" and "includes" are to be construed as being inclusive
and open ended, and not exclusive. Specifically, when used in this
specification including claims, the terms "comprises",
"comprising", "including" and "includes" and variations thereof
mean the specified features, steps or components are included.
These terms are not to be interpreted to exclude the presence of
other features, steps or components.
[0059] The foregoing description of the preferred embodiments of
the invention has been presented to illustrate the principles of
the invention and not to limit the invention to the particular
embodiment illustrated. It is intended that the scope of the
invention be defined by all of the embodiments encompassed within
the following claims and their equivalents.
REFERENCES
[0060] [Lar01] Larsen, L. B., Jensen, P. M., Kammersgaard, K.,
Kromann, L., "The Automated Pool Trainer--A multi modal system for
learning the game of Pool", Intelligent Multi Media, Computing and
Communications: Technologies and Applications of the Future. John
Wiley and Sons, pp. 90-96, June 2001. [0061] [Kan99] Kanov, Gerry,
and Stauch, Shari, "Precision Pool", Human Kinetics, ISBN
0-88011-897-0, 1999. [0062] [Nak01] Nakama, H., Tokashiki, H., "The
Development of the Autonomous Billiard Robot", Bull. Fac. Eng.
Univ. Ryukus (Japan), no. 61, pp. 105-108, March 2001. [0063]
[Qi99] Bingchen Qi, and Okawa, Y., "Building an Intelligent Mobile
Platform for Billiards", Proc. 30.sup.th Intl. Sym. Rob., Tokyo,
pp. 605-612, 27-29 October 1999. [0064] [Shu94] Shu Sang, W. C.,
"Automating Skills Using a Robot Snooker Player", PhD Thesis,
Bristol University, U.K., April 1994. [0065] [Chu02] S. C. Chua, E.
K. Wong, Alan W. C. Tan, and V. C. Koo. "Decision algorithm for
pool using fuzzy system", iCAiET 2002: Intl. Conf. Al in Eng. &
Tech., pp 370-375, June 2002. [0066] [Ali04] Mohammad Ebne Alian,
Saeed Bagheri Shouraki, M. T. Manzuri Shalmani, Pooya Kaimian,
Payam Sabmeydani, "Roboshark: a gantry pool player robot", ISR
2004: 35.sup.th International Symposium of Robotics, Paris-Nord
Villepinte, France, 23-26 March, 2004. [0067] [Lon04] Fei Long,
Johan Herland, Marie-Christine Tessier, Darryl Naulls, Andrew Roth,
Gerhard Roth, Michael Greenspan, "Robotic Pool: An Experiment in
Automatic Potting", accepted in IROS'04, Sendai, Japan, Sep.
28-Oct. 2, 2004. [0068] [Lar02] James Larson, "Scratchy: An
Autonomous Pool-Playing Robot",
http://www.mil.ufl.edu/imdl/papers/1MDL Report Fall 02/larson
james/scratchy.pdf [0069] [Den04] H. Denman, N. Rea, A. Kokaram,
"Content-based analysis for video from snooker broadcasts",
Computer Vision and Image Understanding, no. 92, pp. 176-195, 2003.
[0070] [Fla00] Gregory Flandin, Francois Chaumette, Eric Marchand,
"Eye-in-hand/Eye-to-hand Cooperation for Visual Servoing",
ICRA2000: IEEE International Conference on Robotics and Automation,
San Francisco, April 2000 [0071] [Che04] B. Cheng, J. Li, and J.
Yang, "Design of the Neural-Fuzzy Compensator for a Billiard
Robot." IEEE Intl. Conf. Networking, Sensing & Control, pp.
909-913, 2004. [0072] [Zha98] Z. Zhang, "A Flexible New Technique
for Camera Calibration", Microsoft Research Technical Resport
MSR-TR-98-7, Dec. 2, 1998.
* * * * *
References