U.S. patent application number 15/866858 was filed with the patent office on 2018-07-05 for multi-user medical robotic system for collaboration or training in minimally invasive surgical procedures.
The applicant listed for this patent is Intuitive Surgical Operations, Inc.. Invention is credited to Stephen J. Blumenkranz, Modjtaba Ghodoussi, Brian D. Hoffman, Rajesh Kumar, David Q. Larkin, Amante A. Mangaser, Frederic H. Moll, Ranjan Mukherjee, Gunter D. Niemeyer, William C. Nowlin, Giuseppe Maria Prisco, J. Kenneth Salisbury, JR., Darrin R. Uecker, Yulun Wang, James W. Wright.
Application Number | 20180185110 15/866858 |
Document ID | / |
Family ID | 36780803 |
Filed Date | 2018-07-05 |
United States Patent
Application |
20180185110 |
Kind Code |
A1 |
Kumar; Rajesh ; et
al. |
July 5, 2018 |
Multi-User Medical Robotic System for Collaboration or Training in
Minimally Invasive Surgical Procedures
Abstract
A multi-user medical robotic system for collaboration or
training in minimally invasive surgical procedures includes first
and second master input devices, a first slave robotic mechanism,
and at least one processor configured to generate a first slave
command for the first slave robotic mechanism by switchably using
one or both of a first command indicative of manipulation of the
first master input device by a first user and a second command
indicative of manipulation of the second master input device by a
second user. To facilitate the collaboration or training, both
first and second users communicate with each other through an audio
system and see the minimally invasive surgery site on first and
second displays respectively viewable by the first and second
users.
Inventors: |
Kumar; Rajesh; (Sunnyvale,
CA) ; Hoffman; Brian D.; (Mountain View, CA) ;
Prisco; Giuseppe Maria; (Calci, IT) ; Larkin; David
Q.; (Menlo Park, CA) ; Nowlin; William C.;
(Los Altos Hills, CA) ; Moll; Frederic H.; (San
Francisco, CA) ; Blumenkranz; Stephen J.; (Los Altos
Hills, CA) ; Niemeyer; Gunter D.; (Pasadena, CA)
; Salisbury, JR.; J. Kenneth; (Mountain View, CA)
; Wang; Yulun; (Goleta, CA) ; Ghodoussi;
Modjtaba; (Santa Barbara, CA) ; Uecker; Darrin
R.; (San Mateo, CA) ; Wright; James W.; (Santa
Barbara, CA) ; Mangaser; Amante A.; (Goleta, CA)
; Mukherjee; Ranjan; (East Lansing, MI) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Intuitive Surgical Operations, Inc. |
Sunnyvale |
CA |
US |
|
|
Family ID: |
36780803 |
Appl. No.: |
15/866858 |
Filed: |
January 10, 2018 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
15607676 |
May 29, 2017 |
9867671 |
|
|
15866858 |
|
|
|
|
15006555 |
Jan 26, 2016 |
9666101 |
|
|
15607676 |
|
|
|
|
13965581 |
Aug 13, 2013 |
9271798 |
|
|
15006555 |
|
|
|
|
11319012 |
Dec 27, 2005 |
8527094 |
|
|
13965581 |
|
|
|
|
11025766 |
Dec 28, 2004 |
|
|
|
11319012 |
|
|
|
|
10214286 |
Aug 6, 2002 |
6858003 |
|
|
11025766 |
|
|
|
|
09436982 |
Nov 9, 1999 |
6468265 |
|
|
10214286 |
|
|
|
|
09433120 |
Nov 3, 1999 |
6659939 |
|
|
09436982 |
|
|
|
|
09399457 |
Sep 17, 1999 |
|
|
|
09433120 |
|
|
|
|
09374643 |
Aug 16, 1999 |
|
|
|
09399457 |
|
|
|
|
10948853 |
Sep 23, 2004 |
7413565 |
|
|
11319012 |
|
|
|
|
10246236 |
Sep 17, 2002 |
6951535 |
|
|
10948853 |
|
|
|
|
10051796 |
Jan 16, 2002 |
6852107 |
|
|
10246236 |
|
|
|
|
60725770 |
Oct 12, 2005 |
|
|
|
60109359 |
Nov 20, 1998 |
|
|
|
60109301 |
Nov 20, 1998 |
|
|
|
60109303 |
Nov 20, 1998 |
|
|
|
60150145 |
Aug 20, 1999 |
|
|
|
60116891 |
Jan 22, 1999 |
|
|
|
60116842 |
Jan 22, 1999 |
|
|
|
60109359 |
Nov 20, 1998 |
|
|
|
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
A61B 1/00193 20130101;
G09B 23/28 20130101; A61B 90/361 20160201; A61B 34/70 20160201;
A61B 34/30 20160201; A61B 90/37 20160201; A61B 34/37 20160201; A61B
2017/00017 20130101; Y10S 901/16 20130101 |
International
Class: |
A61B 34/37 20160101
A61B034/37; A61B 34/00 20160101 A61B034/00; A61B 90/00 20160101
A61B090/00 |
Claims
1. A robot system, comprising: a robot that has a camera and a
monitor; a first remote station that has a monitor and is
configured to access and control said robot, said first remote
station having a first input device operated by a first user to
cause movement of said robot; a second remote station that has a
monitor and is configured to access and control said robot, said
second remote station having a second input device operated by a
second user to cause movement of said robot; and, an arbitrator
that can operate in an exclusive mode to control access and control
movement of said robot exclusively by said first remote station or
second remote station, said arbitrator provides a mechanism that
allows said first remote station to exclusively access and control
movement of said robot, said mechanism denies exclusive access to
said robot by said second remote station.
2. A robot system, comprising: a first robot and a second robot
that each have a camera that can generate an image, a monitor, a
speaker and a microphone that can generate audio; a first remote
station that can access said first and second robots, said first
remote station including a camera, a monitor that can receive said
image from said first or second robots, a microphone and a speaker
that can produce audio provided by said first or second robots; a
second remote station that can access said first and second robots,
said second remote station including a camera, a monitor that can
receives said image from said first and second robots, a microphone
and a speaker that can produce audio provided by said first and
second robots; and a server coupled to said first and second robots
and said first and second remote stations, said server allows
exclusive access to said first robot by said first remote station
such that said image and audio from said first robot is provided to
said first remote station and said image and audio are not provided
to said second remote station and even though said second remote
station is prevented from accessing said first robot said server
allows said second remote station to access said second robot.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application is a continuation of U.S. patent
application Ser. No. 15/607,676, filed May 29, 2017, which is a
continuation of U.S. patent application Ser. No. 15/006,555, filed
Jan. 26, 2016, now U.S. Pat. No. 9,666,101, which is a divisional
of U.S. patent application Ser. No. 13/965,581, filed Aug. 13,
2013, now U.S. Pat. No. 9,271,798, which is a divisional of U.S.
patent application Ser. No. 11/319,012, filed Dec. 27, 2005, now
U.S. Pat. No. 8,527,094, which claims priority from U.S.
Provisional Application No. 60/725,770, filed Oct. 12, 2005, each
of which is incorporated herein by this reference.
[0002] U.S. patent application Ser. No. 11/319,012 is also a
continuation-in-part of U.S. patent application Ser. No.
11/025,766, filed Dec. 28, 2004, now abandoned, which is a
continuation of U.S. patent application Ser. No. 10/214,286, filed
Aug. 6, 2002, now U.S. Pat. No. 6,858,003, which is a divisional of
U.S. patent application Ser. No. 09/436,982, filed Nov. 9, 1999,
now U.S. Pat. No. 6,468,265, which claims priority from U.S.
Provisional Patent Application No. 60/109,359, filed Nov. 20, 1998,
U.S. Provisional Application No. 60/109,301, filed Nov. 20, 1998,
U.S. Provisional Application No. 60/109,303, filed Nov. 20, 1998,
and U.S. Provisional Application No. 60/150,145, filed Aug. 20,
1999, and which is a continuation-in-part of U.S. patent
application Ser. No. 09/433,120, filed Nov. 3, 1999, now U.S. Pat.
No. 6,659,939, which is a continuation-in-part of U.S. patent
application Ser. No. 09/399,457, filed Sep. 17, 1999, now
abandoned, which is a continuation-in-part of U.S. patent
application Ser. No. 09/374,643, filed Aug. 16, 1999, now
abandoned, which claims priority from U.S. Provisional Patent
Application No. 60/116,891, filed Jan. 22, 1999, U.S. Provisional
Patent Application No. 60/116,842, filed Jan. 22, 1999, and U.S.
Provisional Patent Application No. 60/109,359, filed Nov. 20, 1998,
each of which is incorporated herein by this reference.
[0003] U.S. patent application Ser. No. 11/319,012 is also a
continuation-in-part application of U.S. patent application Ser.
No. 10/948,853, filed Sep. 23, 2004, now U.S. Pat. No. 7,413,565,
which is a divisional of U.S. patent application Ser. No.
10/246,236, filed Sep. 17, 2002, now U.S. Pat. No. 6,951,535, which
is a continuation of U.S. patent application Ser. No. 10/051,796,
filed Jan. 16, 2002, now U.S. Pat. No. 6,852,107, each of which is
incorporated herein by this reference.
FIELD OF THE INVENTION
[0004] The present invention generally relates to minimally
invasive robotic surgery systems and in particular, to a multi-user
medical robotic system for collaboration or training in minimally
invasive surgical procedures.
BACKGROUND
[0005] While clinical growth of laparoscopic procedures has
stalled, tele-operated robotic surgical systems have been
successful in achieving greater procedure development and clinical
acceptance in several surgical fields. Two examples of such
surgical robotic systems include the da Vinci.RTM. Surgical System
of Intuitive Surgical, Inc., Sunnyvale, Calif., and the Aesop.RTM.
and Zeus.RTM. robot systems of Computer Motion, Inc., which has
been acquired by Intuitive Surgical, Inc.
[0006] For example, the da Vinci.RTM. surgical system can be used
for a wide variety of surgical procedures such as mitral valve
repair, Nissen Fundoplication for the treatment of GERD disease,
gastric bypass surgery for obesity, radical prostatectomy (da
Vinci.RTM. Prostatectomy) for the removal of the prostate,
esophageal surgery, thymectomy for myasthenia gravis, and
epicardial pacemaker leads for biVentricular resynchronization.
[0007] Minimally invasive surgery offers many benefits over
traditional open surgery techniques, including less pain, shorter
hospital stays, quicker return to normal activities, minimal
scarring, reduced recovery time, and less injury to tissue.
Consequently, demand for minimally invasive surgery is strong and
growing.
[0008] Since robotic minimally invasive surgery ("RMIS") is still a
nascent field, however, there are no commercially available
training systems that allow a trainee and mentor to experience the
same environment, and physically interact as they would in open or
even conventional laparoscopic surgery training. Instead, current
RMIS training consists of training courses explaining the robotic
device and surgical technique accompanied by laboratory practice in
animal and cadaver models, followed by watching already proficient
surgeons perform the procedure. A proficient surgeon then
assists/supervises the newly trained surgeon during his or her
initial procedures.
[0009] In a tele-robotic paradigm, this mentoring problem can be
generalized irrespective of the location of the two surgeons.
However, when they are collocated, the ability to view the surgical
scene together, combined with the ability to exchange or share
control of the instruments can enable physical interaction between
the trainee and the mentor, and provide a superior training
environment.
BRIEF SUMMARY
[0010] Thus, a multi-user medical robotic system which allows a
mentor surgeon to communicate with trainee surgeons, to see the
same surgical site as the trainee surgeons, to share control of
robotically controlled surgical instruments with the trainee
surgeons so that they may feel through their controls what the
mentor surgeon is doing with his/hers, and to switch control to
selected ones of the trainee surgeons and over-ride that control if
necessary during the performance of a minimally invasive surgical
procedure, would be highly beneficial for training purposes.
[0011] In addition, such a multi-user medical robotic system would
also be useful for collaborative surgery in which multiple surgeons
work together as a team (i.e., in collaboration) to perform a
minimally invasive surgical procedure.
[0012] Accordingly, one object of the present invention is to
provide a multi-user medical robotic system that facilitates
collaboration between surgeons while performing minimally invasive
surgical procedures.
[0013] Another object is to provide a multi-user medical robotic
system that facilitates training of surgeons to perform minimally
invasive surgical procedures.
[0014] These and additional objects are accomplished by the various
aspects of the present invention, wherein the embodiments of the
invention are summarized by the claims below.
[0015] Additional objects, features and advantages of the various
aspects of the present invention will become apparent from the
following description of its preferred embodiment, which
description should be taken in conjunction with the accompanying
drawings.
BRIEF DESCRIPTION OF THE DRAWINGS
[0016] FIG. 1 illustrates a top view of a multi-user medical
robotic system for collaboration or training in minimally invasive
surgical procedures, utilizing aspects of the present
invention.
[0017] FIGS. 2-3 illustrate simplified front views respectively of
mentor and trainee master control stations configured to utilize
aspects of the present invention.
[0018] FIG. 4 illustrates a block diagram of a master/slave control
system included in the multi-user medical robotic system, utilizing
aspects of the present invention.
[0019] FIGS. 5-9 illustrate block diagrams of selected master/slave
associations for a multi-user medical robotic system, utilizing
aspects of the present invention.
[0020] FIG. 10 illustrates a block diagram of components of the
multi-user medical robotic system for selective association of
masters and slaves, utilizing aspects of the present invention.
[0021] FIG. 11 illustrates an example of input/output ports for an
association module, utilizing aspects of the present invention.
[0022] FIGS. 12 and 13 illustrate routing tables corresponding to
the master/slave associations of FIGS. 9 and 8, respectively, of an
association module utilizing aspects of the present invention.
[0023] FIGS. 14 and 15 illustrate block diagrams for alternative
embodiments of a shared command filter of an association module,
utilizing aspects of the present invention.
[0024] FIGS. 16 and 17 schematically illustrate alternative robotic
telesurgical systems utilizing aspects of the present
invention.
DETAILED DESCRIPTION
[0025] FIG. 1 illustrates, as an example, a multi-user medical
robotic system 100 useful for collaboration or training in
minimally invasive surgical procedures. For example, in a
collaborative operation, a team of two or more proficient surgeons
may work together to perform a minimally invasive surgical
procedure, or an expert surgeon may advise a primary surgeon
performing a minimally invasive surgical procedure. In a hands-on
training environment, a mentor surgeon may act as a mentor or
teacher to train one or more trainee surgeons in minimally invasive
surgical procedures.
[0026] Although configured in this example for a local environment
with all participants locally present, the multi-user medical
robotic system 100 may also be configured through a network
connection for remote participation by one or more participants.
For example, a remote surgeon may provide guidance or support to a
primary surgeon at a local operating site. In such case, the
advising surgeon may share the immersive audio/video environment
with the primary surgeon, and may access the surgical instruments
as desired by the primary surgeon.
[0027] Although a training example is described herein, the
described components and features of the system 100 are also useful
in collaborative surgery. In particular, it is useful for a lead
surgeon in the case of a collaborative procedure to control the
selective association of certain surgical tools and/or an endoscope
with any one of the participating surgeons during a minimally
invasive surgical procedure, just as it is for a mentor surgeon in
the case of a training session to control the selective association
of certain surgical tools and/or an endoscope with any one of the
trainee surgeons during a minimally invasive surgical training
session. Also, it is useful in both the collaboration and training
environments for all participants to be able to view the surgical
site and to communicate with each other during the surgical
procedure or training session.
[0028] In reference to FIG. 1, a Mentor Surgeon (M) instructs or
mentors one or more Trainee Surgeons, such as (T1) and (TK), in
minimally invasive surgical procedures performed on a real-life or
dummy Patient (P). To assist in the surgical procedures, one or
more Assistant Surgeons (A) positioned at the Patient (P) site may
also participate.
[0029] The system 100 includes a mentor master control station 101
operative by the Mentor Surgeon (M), a slave cart 120 having a
plurality of slave robotic mechanisms (also referred to as "robotic
arm assemblies" and "slave manipulators") 121-123, and one or more
trainee master control stations, such as trainee master control
stations 131 and 161, operative by Trainee Surgeons, such as
Trainee Surgeons (T1) and (TK). The mentor master control station
101, in this example, communicates directly with the slave cart
120, and the trainee master control stations communicate indirectly
with the slave cart 120 through the mentor master control station
101.
[0030] The slave cart 120 is positioned alongside the Patient (P)
so that surgery-related devices (such as 157) included at distal
ends of the slave robotic mechanisms 121-123 may be inserted
through incisions (such as incision 156) in the Patient (P), and
manipulated by one or more of the participating surgeons at their
respective master control stations to perform a minimally invasive
surgical procedure on the Patient (P). Each of the slave robotic
mechanisms 121-123 preferably includes linkages that are coupled
together and manipulated through motor controlled joints in a
conventional manner.
[0031] Although only one slave cart 120 is shown being used in this
example, additional slave carts may be used as needed. Also,
although three slave robotic mechanisms 121-123 are shown on the
cart 120, more or less slave robotic mechanisms may be used per
slave cart as needed.
[0032] A stereoscopic endoscope is commonly one of the
surgery-related devices included at the distal end of one of the
slave robotic mechanisms. Others of the surgery-related devices may
be various tools with manipulatable end effectors for performing
the minimally invasive surgical procedures, such as clamps,
graspers, scissors, staplers, and needle holders.
[0033] Use of the stereoscopic endoscope allows the generation and
display of real-time, three-dimensional images of the surgical
site. Although the stereoscopic endoscope is preferred for this
reason, a monoscopic endoscope may alternatively be used where
either three-dimensional images are not needed or it is desirable
to reduce communication bandwidth requirements.
[0034] Alternatively, the system may include multiple endoscopes
providing each individual surgeon with a desired view of the
workspace. Advantageously, the multiple endoscopes may even be
packaged in a single instrument, but with separate steerable camera
tips. Optionally, these multiple endoscopes may provide different
fields of view such as using a very wide field of view (e.g. with a
fish-eye lens) that is appropriately rectified before being
displayed to the surgeon.
[0035] To facilitate collaboration between surgeons or training of
trainee surgeons in minimally invasive surgical procedures, each of
the participating surgeons has an associated display to view the
surgical site, and a communication means such as a microphone and
earphone set to communicate with other participating surgeons.
[0036] More particularly, a display 102 is provided with or
integrated into the mentor master control station 101, a display
132 is provided with or integrated into the trainee master control
station 131, and a display 142 is provided on a vision cart 141
which is in view of the one or more Assistant Surgeons (A), so that
the Mentor Surgeon (M), the Trainee Surgeon (T), and the Assistant
Surgeon(s) (A) may view the surgical site during minimally invasive
surgical procedures.
[0037] The vision cart 141, in this example, includes stereo camera
electronics which convert pairs of two-dimensional images received
from the stereoscopic endoscope into information for corresponding
three-dimensional images, displays one of the two-dimensional
images on the display 142 of the vision cart 141, and transmits the
information of the three-dimensional images over a stereo vision
channel 111 to the master control stations of participating
surgeons, such as the mentor master control station 101 and the
trainee master control stations, for display on their respective
displays. For displaying stereo information using properly
configured conventional displays, the vision cart 141 may contain
devices for frame synchronization, and in that case, conventional
video cables may be sufficient for sharing this information between
collocated surgeons.
[0038] The communication means provided to each of the participants
may include individual microphone and earphones (or speaker)
components, or alternatively, individual headphone sets, such as
headphone set 103 shown as being placed on the head of the Mentor
Surgeon (M), as part of a conventional audio system. Preferably a
duplex audio communication system (microphone and speaker pair) is
built into each surgeon's master control station. Alternatively,
headsets may be used, including those using wireless communications
to provide maximum comfort and freedom of movement to their users
or those that may be connected through wires to their respective
master control stations or slave cart, which are in turn, are
connected together through mentor/slave lines 110 and
mentor/trainee lines 112 for voice communications between the
Mentor, Trainee and Assistant Surgeons.
[0039] In addition to transmitting voice communications, the
mentor/slave and the mentor/trainee lines, 110 and 112, also
transmit data. For high bandwidth and low latency communication,
the lines 110 and 112, as well as the stereo vision channel lines
111, are preferably composed of fiber optic communication
cables/channels, which are especially useful when any of the mentor
master control station 101, the trainee master control stations
(such as 131 and 161), and the slave cart 120 are remotely situated
from the others. On the other hand, for co-located surgeons, normal
shielded video and audio cables may be sufficient, while fiber
optical communication channels may be used for the mentor/slave or
mentor/trainee data transfer lines.
[0040] FIGS. 2-3 illustrate simplified front views of the mentor
master control station 101 and the trainee master control station
131. The mentor master control station 101 includes right and left
master input devices, 203 and 204, whose manipulations by the
Mentor Surgeon (M) are sensed by sensors (not shown) and provided
to an associated processor 220 via an instrumentation bus 210.
Similarly, the trainee master control station 131 includes right
and left master input devices, 303 and 304, whose manipulations by
the Trainee Surgeon (T1) are sensed by sensors (not shown) and
provided to an associated processor 320 via an instrumentation bus
310. Each of the master input devices (also referred to herein as
"master manipulators") may include, for example, any one or more of
a variety of input devices such as joysticks, gloves, trigger-guns,
hand-operated controllers, and the like.
[0041] The mentor master control station 101 is preferably
configured with one or more switch mechanisms to allow the Mentor
Surgeon (M) to selectively associate individual of the slave
robotic mechanisms 121-123 with any of the master input devices of
the mentor master control station 101 and the trainee master
control stations. As one example, two switch mechanisms may be
activated by right or left buttons, 205 and 207, positioned on the
right and left master input devices, 203 and 204, so as to be
manipulatable by right and left thumbs of the Mentor Surgeon
(M).
[0042] As another example, two switch mechanisms may be activated
by right or left footpedals, 215 and 217, which are positioned so
as to be manipulatable by right and left feet of the Mentor Surgeon
(M). One switch mechanism may also be voice activated by the Mentor
Surgeon (M) using his headset 103 or another microphone (not
shown), which is coupled to the processor 220 so that it may
perform voice recognition and processing of the spoken instructions
of the Mentor Surgeon (M).
[0043] For complex associations of various aspects of system master
input devices and slave robotic mechanisms, a simple binary switch
(or combinations of switches) may not be suitable. In such cases, a
more flexible association selector may be required, such as a menu
of available options displayed on the display 102 of the mentor
master control station 101 that the Mentor Surgeon (M) may select
from, by using a conventional pointing device, touch screen, or
voice activation. The master input devices or input devices built
into the master input devices may also be used for this
purpose.
[0044] To perform a minimally invasive surgical procedure, the
operating surgeons perform the procedure by manipulating their
respective master input devices which in turn, causes associated
slave robotic mechanisms to manipulate their respective
surgery-related devices through minimally invasive incisions in the
body of the Patient (P) while the surgeons view the surgical site
through their respective displays.
[0045] The number of surgery-related devices used at one time and
consequently, the number of slave robotic mechanisms in the system
100 will generally depend on the diagnostic or surgical procedure
and the space constraints within the operating room among other
factors. If it is necessary to change one or more of the
surgery-related devices being used during a procedure, the
Assistant (A) may remove the surgery-related device that is no
longer needed from the distal end of its slave robotic mechanism,
and replace it with another surgery-related device from a tray of
such devices in the operating room. Alternatively, a robotic
mechanism may be provided for the surgeon to execute tool exchanges
using his/her master input device.
[0046] Preferably, the master input devices will be movable in the
same degrees of freedom as their associated surgery-related devices
to provide their respective surgeons with telepresence, or the
perception that the master input devices are integral with their
associated surgery-related devices, so that their respective
surgeons have a strong sense of directly controlling them. To this
end, position, force, and tactile feedback sensors are preferably
employed that transmit position, force, and tactile sensations from
the devices (or their respective slave robotic mechanisms) back to
their associated master input devices so that the operating
surgeons may feel such with their hands as they operate the master
input devices.
[0047] To further enhance the telepresence experience, the
three-dimensional images displayed on the displays of the master
control stations are oriented so that their respective surgeons
feel that they are actually looking directly down onto the
operating site. To that end, an image of the surgery-related device
that is being manipulated by each surgeon appears to be located
substantially where the surgeon's hands are located even though the
observation points (i.e., the endoscope or viewing camera) may not
be from the point of view of the image.
[0048] FIG. 4 illustrates, as an example, a block diagram of a
master/slave control system 400 for an associated master
manipulator and slave manipulator pair. An example of such a
master/slave manipulator pair is the master device input 203 of the
mentor master control station 101 and the slave robotic mechanism
121. Master manipulator inputs and corresponding slave manipulator
outputs are indicated by arrows AB, and slave manipulator inputs
and corresponding master manipulator outputs in the case of
feedback are indicated by arrows BA.
[0049] Although the master processing unit 420 and slave processing
unit 430 described herein may be implemented as analog circuitry,
preferably they are implemented digitally using conventional
Z-transform techniques for sampled data systems and provided in
program code executed by processors of master control stations
associated with the master and slave manipulators, 404 and 416, as
will be described in further detail in reference to FIG. 10.
[0050] In the following description, the master manipulator (i.e.,
master input device) 404 will be referred to as the master and the
slave manipulator (i.e., slave robotic mechanism) 416 will be
referred to as the slave, to simplify the description. Also,
positions sensed by joint encoders in the master manipulator as
well as those in the slave manipulator are referred to as "joint
space" positions. Furthermore, references to positions and
positioned signals may include orientation, location, and/or their
associated signals. Similarly, forces and force signals may
generally include both force and torque in their associated
signals.
[0051] For ease of explanation, the master/slave control system 400
will be described from an initial condition in which the master is
at an initial position and the slave is at a corresponding initial
position. However, in use, the slave tracks the master position in
a continuous manner.
[0052] Referring to the control system 400, the master is moved
from an initial position to a new position corresponding to a
desired position of the end effector (located on the distal end of
the slave) as viewed by the surgeon on his display. Master control
movements are input by the surgeon 402, as indicated by arrow AB1,
by applying a force to the master 404 to cause the master 404 to
move from its initial position to the new position.
[0053] As the master 404 is thus manipulated by the surgeon,
signals from the encoders on the master 404 are input to a master
controller 406 as indicated by arrow AB2. At the master controller
406, the signals are converted to a joint space position
corresponding to the new position of the master. The joint space
position is then input to a master kinematics converter 408 as
indicated by arrow AB3. The master kinematics converter 408 then
transforms the joint space position into an equivalent Cartesian
space position. This is optionally performed by a kinematics
algorithm including a Jacobian transformation matrix, inverse
Jacobian, or the like. The equivalent Cartesian space position is
then input to a bilateral controller 410 as indicated by arrow
AB4.
[0054] Position comparison and force calculation may, in general,
be performed using a forward kinematics algorithm which may include
a Jacobian matrix. The forward kinematics algorithm generally makes
use of a reference location, which is typically selected as the
location of the surgeon's eyes. Appropriate calibration or
appropriately placed sensors on the master control station can
provide this reference information. Additionally, the forward
kinematics algorithm will generally make use of information
concerning the lengths and angular offsets of the linkage of the
master. More specifically, the Cartesian position represents, for
example, the distance of the input handle from, and the orientation
of the input handle relative to, the location of the surgeon's
eyes. Hence, the equivalent Cartesian space position is input into
bilateral controller 410 as indicated by AB4.
[0055] In a process similar to the calculations described above,
the slave position is also generally observed using joint encoders
of the slave 416. In an exemplary embodiment, joint encoder signals
read from the slave 416 are provided to a slave controller 414, as
indicated by BA2, which converts the signals to a joint space
position corresponding to the initial position of the slave 416.
The joint space position is then input to a slave kinematics
converter 412 as indicated by arrow BA3. The slave kinematics
converter 412 then transforms the joint space position into an
equivalent Cartesian space position.
[0056] In this case, the forward kinematics algorithm used by the
slave kinematics converter 412 is preferably provided with the
referenced location of a tip of a stereoscopic endoscope capturing
images of the surgery site to be viewed on the surgeon display.
Additionally, through the use of sensors, design specifications,
and/or appropriate calibration, this kinematics algorithm
incorporates information regarding the lengths, offsets, angles,
etc., describing the linkage structure of the slave cart 120, and
set-up joints for the slave 416 (i.e., joints used to initially
position the slave that are subsequently locked during the
procedure) so that the slave Cartesian position transferred to the
bilateral controller 410 is measured and/or defined relative to the
tip of the stereoscopic endoscope.
[0057] At bilateral controller 410, the new position of the master
in Cartesian space relative to the surgeon's eyes is compared with
the initial position of the tip of the end effector connected at
the distal end of the slave 416 in Cartesian space relative to the
tip of the stereoscopic endoscope.
[0058] Advantageously, the comparison of these relative
relationships occurring in the bilateral controller 410 can account
for differences in scale between the master input device space in
which the master input device 404 is moved as compared with the
surgical workspace in which the end effectors on the distal end of
the slave robotic mechanism 416 move. Similarly, the comparison may
account for possible fixed offsets, should the initial master and
slave positions not correspond.
[0059] Since the master has moved to a new position, a comparison
by the bilateral controller 410 of its corresponding position in
Cartesian space with the Cartesian space position of the slave
corresponding to its initial position yields a deviation and a new
slave position in Cartesian space. This position is then input to
the slave kinematics converter 412 as indicated by arrow ABS, which
computes the equivalent joint space position commands.
[0060] These commands are then input to the slave controller 414 as
indicated by arrow AB6. Necessary joint torques are computed by the
slave controller 414 to move the slave to its new position. These
computations are typically performed using a proportional integral
derivative (P.I.D.) type controller. The slave controller 414 then
computes equivalent motor currents for these joint torque values,
and drives electrical motors on the slave 416 with these currents
as indicated by arrow AB7. The slave 416 is then caused to be
driven to the new slave position which corresponds to the new
master position.
[0061] The control steps involved in the master/slave control
system 400 as explained above are typically carried out at about
1300 cycles per second or faster. It will be appreciated that
although reference is made to an initial position and new position
of the master, these positions are typically incremental stages of
a master control movement. Thus, the slave is continually tracking
incremental new positions of the master.
[0062] The master/slave control system 400 also makes provision for
force feedback. Thus, should the slave 416 (i.e., its end effector)
be subjected to an environmental force at the surgical site, e.g.,
in the case where the end effector pushes against tissue, or the
like, such a force is fed back to the master 404 so that the
surgeon may feel it. Accordingly, when the slave 416 is tracking
movement of the master 404 as described above and the slave 416
pushes against an object at the surgical site resulting in an equal
pushing force against the slave 416, which urges the slave 416 to
move to another position, similar steps as described above in the
forward or control path take place in the feedback path.
[0063] The surgical environment is indicated at 418 in FIG. 4. In
the case where an environmental force is applied on the slave 416,
such a force causes displacement of the end effector. This
displacement is sensed by the encoders on the slave 416 which
generate signals that are input to the slave controller 414 as
indicated by arrow BA2. The slave controller 414 computes a
position in joint space corresponding to the encoder signals, and
provides the position to the slave kinematics converter 412, as
indicated by arrow BA3.
[0064] The slave kinematics converter 412 computes a Cartesian
space position corresponding to the joint space position, and
provides the Cartesian space position to the bilateral controller
410, as indicated by arrow BA4. The bilateral controller 410
compares the Cartesian space position of the slave with a Cartesian
space position of the master to generate a positional deviation in
Cartesian space, and computes a force value corresponding to that
positional deviation that would be required to move the master 404
into a position in Cartesian space which corresponds with the slave
position in Cartesian space. The force value is then provided to
the master kinematics converter 408, as indicated by arrow BA5.
[0065] The master kinematics converter 408 calculates from the
force value received from the bilateral controller 410,
corresponding torque values for the joint motors of the master 404.
This is typically performed by a Jacobian Transpose function in the
master kinematics converter 408. The torque values are then
provided to the master controller 406, as indicated by arrow BA6.
The master controller 406, then determines master electric motor
currents corresponding to the torque values, and drives the
electric motors on the master 404 with these currents, as indicated
by arrow BA7. The master 404 is thus caused to move to a position
corresponding to the slave position.
[0066] Although the feedback has been described with respect to a
new position to which the master 404 is being driven to track the
slave 416, it is to be appreciated that the surgeon is gripping the
master 404 so that the master 404 does not necessarily move. The
surgeon however feels a force resulting from feedback torques on
the master 404 which he counters because he is holding onto the
master 404.
[0067] In performing collaborative minimally invasive surgical
procedures or training in such procedures, it is useful at times
for the lead or mentor surgeon to selectively associate certain
master input devices with certain slave robotic mechanisms so that
different surgeons may control different surgery-related devices in
a collaborative effort or so that selected trainees may practice or
experience a minimally invasive surgical procedure under the
guidance or control of the mentor surgeon. Some examples of such
selective master/slave associations are illustrated in FIGS. 5-9,
wherein each master depicted therein includes the master
manipulator 404 and master processing 420 of FIG. 4 and each slave
depicted therein includes the slave manipulator 416 and slave
processing 430 of FIG. 4.
[0068] In FIG. 5, an exclusive operation master/slave association
is shown in which master 501 has exclusive control over slave 502
(and its attached surgery-related device), and master 511 has
exclusive control over slave 512 (and its attached surgery-related
device). In this configuration, the masters, 501 and 511, may be
controlled by the right and left hands of a surgeon while
performing a minimally invasive surgical procedure, or they may be
controlled by different surgeons in a collaborative minimally
invasive surgical procedure. The master/slave control system 400
may be used for each associated master/slave pair so that lines 503
and 513 (master to slave direction) correspond to its forward path
AB4 line and lines 504 and 514 (slave to master direction)
correspond to its feedback path BA5 line.
[0069] In FIG. 6, a unilateral control master/slave association is
shown in which master 601 has exclusive control over slave 602 (and
its attached surgery-related device), but input and reflected force
(or position) values are provided to the master 611 as well as the
master 601. In this configuration, although the master 611 cannot
control the slave 602, it tracks the master 601 so that a surgeon
holding the master input device of master 611 can feel and
experience movement of the master input device of master 601 as it
is being manipulated by another surgeon. Thus, this sort of
configuration may be useful in training surgeons by allowing them
to experience the movement of the master input device of the master
601 as it is being manipulated by a mentor surgeon during a
minimally invasive surgical procedure, while viewing the surgical
site in their respective displays and communicating with the mentor
surgeon using their respective headsets.
[0070] In FIG. 7, a modified version of the unilateral control
master/slave association is shown. In this configuration, not only
does the surgeon holding the master input device of master 711
experience the movement of (and forces exerted against) the master
input device of the master 701 as it is being manipulated by
another surgeon during a minimally invasive surgical procedure, the
surgeon associated with master 711 can also "nudge" the master
input device of the master 701 by manipulating his/her master input
device since a force value corresponding to such nudging is
provided back to the master 701, as indicated by the arrow 722.
This "nudging" master/slave configuration is useful for training
surgeons, because it allows a trainee surgeon to practice by
performing the surgical procedure by manipulating the slave 702
(and its attached surgery-related device) using the master input
device of his/her master 701, while the mentor surgeon monitors
such manipulation by viewing the surgical site on his/her display
while feeling the movement of the trainee surgeon's master input
device through input and feedback forces, respectively indicated by
arrows 721 and 704. If the mentor surgeon thinks that the trainee
surgeon should modify his/her operation of his/her master input
device, the mentor surgeon can nudge the trainee surgeon's master
input device accordingly, while at the same time, communicating
such recommendation verbally to the trainee surgeon using a shared
audio system through their respective headsets.
[0071] In FIG. 8, a unilateral, shared master/slave association,
which is a variant of the nudging configuration of FIG. 7, is shown
in which either (or both) masters 801 and 811 may control slave
802. In this configuration, not only does the surgeon holding the
master input device of master 811 experience the movement of (and
forces exerted against) the master input device of the master 801
as it is being manipulated by another surgeon during a minimally
invasive surgical procedure, the surgeon associated with master 811
can also control the slave 802 if desired, as indicated by the
arrow 813. This "override" master/slave configuration is useful for
training surgeons, because it allows a trainee surgeon to practice
by performing the surgical procedure by manipulating the slave 802
(and its attached surgery-related device) using the master input
device of his/her master 801, while the mentor surgeon monitors
such manipulation by viewing the surgical site on his/her display
while feeling the movement of the trainee surgeon's master input
device through input and feedback forces, respectively indicated by
arrows 821 and 804. If the mentor surgeon finds it necessary to
assume control of the slave 802 to avoid injury to a patient, the
mentor surgeon can assert such control accordingly, while at the
same time, communicating that he/she is taking over control
verbally to the trainee surgeon through a shared audio system.
[0072] In FIG. 9, a bilateral master/slave association is shown in
which masters, 901 and 912, and slaves, 902 and 912, all move in
tandem, tracking each other's movements. In this configuration, the
slave 912 (and its attached surgery-related device) may be
controlled by a surgeon using the master 901, while another surgeon
experiences its movement by loosely holding the master input device
for the other master 911. The slave 902 in this case is generally
non-operative in the sense that it is not directly participating in
the minimally invasive surgical procedure. In particular, the slave
902 either may not have the distal end of its slave robotic
mechanism inserted in the patient so that its robotic arm moves,
but does not result in any action taking place in the surgical
site, or the slave 902 may only include a computer model of the
linkages, joints, and joint motors of its slave robotic mechanism,
rather than the actual slave robotic mechanism.
[0073] However, the slave 902 does move in tandem with the slave
912 (in actuality or through simulation) as the surgeon
manipulating the master input device of the master 901 causes the
slave 912 to move, because a force (or position) value
corresponding to such manipulation is provided to the master 911,
as indicated by arrow 921, and the master 911 controls the slave
902 to move accordingly, as indicated by arrow 913. Any forces
asserted against the surgery-related device attached to the distal
end of the slave robotic mechanism of the slave 912 are then fed
back to the master input device of the master 911, as indicated by
the arrow 914.
[0074] Note that the surgeon associated with the master 911 can
effectively "nudge" the master 901 by manipulating the master input
device of the master 911. Therefore, the bilateral master/slave
association shown in FIG. 9 can also be used in the training of
surgeons in a similar manner as the "nudging" and unilateral,
shared master/slave associations respectively shown in FIGS. 7 and
8.
[0075] FIG. 10 illustrates a block diagram of components of the
multi-user medical robotic system for selective association of
master manipulators (also referred to as "master input devices"),
404 and 1004, with slave manipulators (also referred to as "slave
robotic mechanisms"), 416 and 1016. Although only two master
manipulators and two slave manipulators are shown in this example,
it is to be appreciated that any number of master manipulators may
be associated with any number of slave manipulators in the system,
limited only by master control station port availability, memory
capacity, and processing capability/requirements.
[0076] The master processing unit 420 includes the master
controller 406 and the master kinematics converter 408 and
generally operates as described in reference to FIG. 4, and the
master processing unit 1020 is similarly configured and
functionally equivalent to the master processing unit 420. The
slave processing unit 430 includes the slave controller 414, slave
kinematics converter 412, and the bilateral controller 410 and
generally operates as described in reference to FIG. 4, and the
slave processing unit 1030 is similarly configured and functionally
equivalent to the slave processing unit 430.
[0077] An association module 1001 includes a shared command filter
1002 and a routing table 1003 for selectively associating master
manipulators, 404 and 1004, with slave manipulators, 416 and 1016.
In brief, the routing table 1003 indicates which inputs are routed
to which outputs of the association module 1001, and the shared
command filter 1002 determines how shared command of a slave
manipulator by two master manipulators is handled. One or more
switch commands 1005 are provided to the association module 1001 as
a means for a user to alter parameters of the shared command filter
1002 or values in the routing table 1003 so as to change or switch
the selected associations between master and slave manipulators.
The current parameters of the shared command filter 1002 and/or
values in the routing table 1003 may be indicated to the user using
a plurality of icons on a graphical user interface of an auxiliary
display or the user's master control station display, or they may
be indicated by a plurality of light-emitting-diodes or other such
indicators on or adjacent to the user's master control station, or
they may be indicated by any other display mechanism.
[0078] The switch command(s) 1005 may be generated by any one or
combination of: the user interacting with one or more buttons on
the master input devices, the user interacting with one or more
foot pedals associated with the user's master control station, the
user providing recognizable voice commands to a voice recognition
(i.e., word recognition) and processing system, the user
interacting with one or more menus displayed on the user's master
control station display, or the user interacting with any other
conventional input mechanism of such sort.
[0079] In a preferred embodiment compatible with the multi-user
medical robotic system of FIG. 1, master processing 420 is
performed as executable program code on a processor associated with
the master control station of the master manipulator 404, and
master processing 1020 is also performed as executable program code
on a processor associated with the master control station of the
master manipulator 1004. Both master control stations in this case
may be Trainee master control stations, such as master control
stations 131 and 161 of FIG. 1, or one of the master control
stations may be the Mentor master control station 101 and the
other, a Trainee master control station.
[0080] The slave processing 430, the slave processing 1030, and the
association module 1001 are preferably included as executable
program or table code on the processor 220 associated with the
Mentor master control station 101. The switch command(s) 1005 in
this case originate from action taken by the Mentor Surgeon (M)
operating the Mentor master control station 101.
[0081] The Mentor master control station 101 preferably performs
the slave processing for all slave robotic mechanisms 121-123,
because it communicates directly with the slave robotic mechanisms
121-123, whereas the Trainee master control stations only
communicate indirectly with the slave robotic mechanisms 121-123
through the Mentor master control station 101. On the other hand,
the Trainee master control stations preferably perform the master
processing for their respective master input devices, so that such
processing may be performed in parallel with the slave processing
(while maintaining time synchronization) while off-loading these
processing requirements from the processor of the Mentor master
control station 101. Thus, this distribution of processing makes
efficient use of processor resources and minimizes processing
delay.
[0082] One feature of the present invention is the capability to
selectively associate on-the-fly both command and feedback paths
between the master and slave manipulators. For example, the
exclusive operation master/slave association shown in FIG. 5 may be
altered on-the-fly (i.e., during a minimally invasive surgical
procedure rather than at set-up) to the bilateral master/slave
association shown in FIG. 9 by re-associating the command path of
the master 501 from the slave 502 to the slave 512 while
maintaining the feedback path of the slave 502 to the master 501,
re-associating the command path of the master 511 from the slave
512 to the slave 502 while maintaining the feedback path of the
slave 512 to the master 511, providing a value indicating the input
force applied against the master 501 to the master 511, and
providing a value indicating the input force applied against the
master 511 to the master 501.
[0083] FIG. 11 illustrates an example of input/output ports for the
association module 1001, in which input ports A-F are shown on the
left side of the association module 1001 for convenience, and
output ports U-Z are shown on the right side of the association
module 1001 for convenience.
[0084] Input port A is assigned to the output of the master
processing 420 which is provided on line 1014 of FIG. 10, input
port B is assigned to the surgeon force input to the master
manipulator 404 which is provided on line 1042 of FIG. 10, input
port C is assigned to surgeon force input to the master manipulator
1004 which is provided on line 1052 of FIG. 10, input port D is
assigned to the output of the master processing 1020 which is
provided on line 1054 of FIG. 10, input port E is assigned to the
output of the slave processing 430 which is provided on line 1035
of FIG. 10, and input port F is assigned to output of the slave
processing 1030 which is provided on line 1075 of FIG. 10.
[0085] Output port U is assigned to the input to the slave
processing 430 which is provided on line 1024 of FIG. 10, output
port V is assigned to the input force to the master manipulator
1004 which is provided on line 1053 of FIG. 10, output port W is
assigned to the input force to the master manipulator 404 which is
provided on line 1042 of FIG. 10, output port X is assigned to the
input to the slave processing 1030 which is provided on line 1064
of FIG. 10, output port Y is assigned to the feedback to the master
processing 420 which is provided on line 1045 of FIG. 10, and
output port Z is assigned to the feedback to the master processing
1020 which is provided on line 1085 of FIG. 10.
[0086] FIG. 12 illustrates a routing table corresponding to the
master/slave association shown in FIG. 9, and FIG. 13 illustrates a
routing table corresponding to the master/slave association shown
in FIG. 8. Referring to FIG. 12, input port A is connected to
output port X (i.e., line 1014 is coupled to line 1064 of FIG. 10,
which corresponds to line 903 of FIG. 9), input port B is coupled
to output port V (i.e., line 1042 is coupled to line 1053 of FIG.
10, which corresponds to line 921 of FIG. 9), input port C is
connected to output port W (i.e., line 1052 is coupled to line 1043
of FIG. 10, which corresponds to line 922 in FIG. 9), input port D
is connected to output port U (i.e., line 1054 is coupled to line
1024 of FIG. 10, which corresponds to line 913 in FIG. 9), input
port E is connected to output port Y (i.e., line 1035 is coupled to
line 1045 of FIG. 10, which corresponds to line 904 in FIG. 9), and
input port F is connected to output port Z (i.e., line 1075 is
coupled to line 1083 of FIG. 10, which corresponds to line 914 in
FIG. 9).
[0087] If the Mentor Surgeon (M) is operating the master 901 and
desires at this point to change the master/slave association from
that of FIG. 9 to that of FIG. 8, he/she provides appropriate
switch command(s) 1005 by, for example, depressing a button on
his/her right-hand master input device corresponding to the master
901 so that the command output of the master 901 is provided to the
slave 902 instead of the slave 912, and selecting menu entries on
his/her display to stop providing commands to or receiving force
feedback from the slave 912, to provide the force feedback from the
slave 902 to the master 911 (as well as continuing to do so to the
master 901), and stop providing the input force exerted on the
master input device of the master 911 to the master 901.
Alternatively, as previously described, these switches may be done
using foot pedals, voice actuation, or any combination of buttons,
foot pedals, voice, display menu, or other actuation devices
controllable by the Mentor Surgeon (M).
[0088] FIG. 13 illustrates the routing table resulting from the
above described switch command(s) 1005 that places the master/slave
association into the configuration shown in FIG. 8. In this case,
input port A is connected to output port U (i.e., line 1014 is
coupled to line 1024 of FIG. 10, which corresponds to line 803 of
FIG. 8), input port B is coupled to output port V (i.e., line 1042
is coupled to line 1053 of FIG. 10, which corresponds to line 821
of FIG. 8), input port C is not connected to any output port, input
port D is connected to output port U (i.e., line 1054 is coupled to
line 1024 of FIG. 10, which corresponds to line 813 in FIG. 8),
input port E is connected to output ports Y and Z (i.e., line 1035
is coupled to line 1045 and 1085 of FIG. 10, which corresponds to
line 804 in FIG. 8), and input port F is not connected to any
output port.
[0089] Referring back to FIG. 8 now, it is noted that the slave 802
has two command inputs, one from the master 801 and another from
the master 811. This causes a control contention issue which may be
resolved by the shared command filter 1002 of the association
module 1001 of FIG. 10.
[0090] FIGS. 14 and 15 illustrate block diagrams for alternative
embodiments of the shared command filter 1002. As shown in FIG. 14,
the shared command filter 1002 takes the form of a simple arbiter,
selecting either a first command input CMD1 or a second command
input CMD2, depending upon a priority input which is provided as a
switch command 1005 to the association module 1001 by the Mentor
Surgeon (M) or programmed into or provided as a parameter value for
its process code. As shown in FIG. 15, the shared command filter
1002 may also take the form of a weighter or weighting function
that weights command inputs CMD1 and CMD2, and combines the
weighted values to determine a shared command value to be provided
to the slave. In this case, the respective weights of the first and
second command inputs, CMD1 and CMD2, depend on a weight input
which is provided as a switch command 1005 to the association
module 1001 by the Mentor Surgeon (M), or programmed into or
provided as parameter values for its process code.
[0091] In the foregoing description of the switching process from
one master/slave association to another, it has been assumed that
such switching occurs instantaneously. However, to avoid
undesirable transient movement of the slave robotic mechanisms, it
may be desirable in certain circumstances to phase-in the switching
process (i.e., gradually reducing the strength of the signal being
switched out while gradually increasing the strength of the signal
being switched in), or using a clutch mechanism that disengages
both signals and only engages the new signal, for example, after
making sure that the position of the slave robotic mechanism being
commanded by the new signal matches that of the old signal so that
a sudden movement will not occur as a result of the change.
[0092] Alternative telesurgical networks are schematically
illustrated in FIGS. 16 and 17. An operator O and an Assistant 43
may cooperate to perform an operation by passing control of
instruments between input devices, and/or by each manipulating
their own instrument or instruments during at least a portion of
the surgical procedure. Referring, now to FIG. 16, during at least
a portion of a surgical procedure, for example, cart 305 is
controlled by Operator O and supports an endoscope and two surgical
instruments. Simultaneously, for example, cart 308 might have a
stabilizer and two other surgical instruments, or an instrument and
another endoscope. The surgeon or operator O and assistant 43
cooperate to perform a stabilized heating heart coronary artery
bypass grafting (CABG) procedure by, for example, passing a needle
or other object back and forth between the surgical instruments of
carts 305, 308 during suturing, or by having the instruments of
cart 308 holding the tissue of the two vessels being anastomosed
while the two instruments of cart 305 are used to perform the
actual suturing. Such cooperation heretofore has been difficult
because of the volumetric space required for human hands to
operate. Since robotic surgical end effectors require much less
space in which to operate, such intimate cooperation during a
delicate surgical procedure in a confined surgical space is now
possible. Optionally, control of the tools may be transferred or
shared during an alternative portion of the procedure.
[0093] Referring now to both FIGS. 16 and 17, cooperation between
systems is also possible. The choice of how many masters and how
many corresponding slaves to enable on a cooperating surgical
system is somewhat arbitrary. Within the scope of the present
invention, one may construct a single telesurgical system's
architecture to handle five or six manipulators (e.g., two masters
and three or four slaves) or ten or twelve manipulators (e.g., four
masters and six or eight manipulators), although any number is
possible. For a system having multiple master controls, the system
may be arranged so that two operators can operate the same surgical
system at the same time by controlling different slave manipulators
and swapping manipulators as previously described.
[0094] Alternatively, it may be desirable to have a somewhat
modular telesurgical system that is capable both of conducting one
particular surgical operation with only one operator and, for
example, five or six manipulators, and which is also capable of
coupling to another modular system having five or six manipulators
to perform a second surgical procedure in cooperation with a second
operator driving the second system. For such modular systems, five
or six manipulator arms are preferably supported by the
architecture, although any number may be incorporated into each
system. One advantage of the modular system over a single, larger
system is that when decoupled, the modular systems may be used for
two separate simultaneous operations at two different locations,
such as in adjacent operating rooms, whereas such might be quite
difficult with a single complex telesurgical system.
[0095] As can be understood with reference to FIG. 16, a simple
manner of having two surgical systems, each having an operator, to
cooperate during a surgical procedure is to have a single image
capture device, such as an endoscope, produce the image for both
operators. The image can be shared with both displays by using a
simple image splitter. If immersive display is desired, the two
systems might additionally share a common point of reference, such
as the distal tip of the endoscope, from which to calculate all
positional movements of the slave manipulators. With the exception
of the imaging system, each control station might be independent of
the other, and might be operatively coupled independently to its
associated tissue manipulation tools. Under such a simple
cooperative arrangement, no swapping of slave manipulators from one
system to another would be provided, and each operator would have
control over only the particular slave manipulators attached
directly to his system. However, the two operators would be able to
pass certain objects back and forth between manipulators, such as a
needle during an anastomosis procedure. Such cooperation may
increase the speed of such procedures once the operators establish
a rhythm of cooperation. Such an arrangement scenario may, for
example, be used to conduct a typical CABG procedure, such that one
operator would control the endoscope and two tissue manipulators,
and the other operator would control two or three manipulators to
aid in harvesting the internal mammary artery (IMA) and suturing
the arterial blood source to the blocked artery downstream of the
particular blocked artery in question. Another example where this
might be useful would be during beating heart surgery, such that
the second operator could control a stabilizer tool in addition to
two other manipulators and could control the stabilizer while the
first operate performed an anastomosis.
[0096] One complication of simple cooperative arrangements is that
if the first operator desired to move the image capture device, the
movement might alter the image of the surgical field sufficiently
that the second operator would no longer be able to view his slave
manipulators. Thus, some cooperation between the operators, such as
audible communications, might be employed before such a
maneuver.
[0097] A slightly more complicated arrangement of surgical
manipulators on two systems within the scope of the present
invention, occurs when operators are provided with the ability to
"swap" control of manipulator arms. For example, the first operator
is able to procure control over a manipulator arm that is directly
connected to the second operator's system. Such an arrangement is
depicted in FIG. 16.
[0098] With the ability to operatively hook multiple telesurgical
systems together, an arrangement akin to a surgical production line
can be envisioned. For example, a preferred embodiment of the
present invention is shown in FIG. 17. Therein, a single master
surgeon O occupies a central master control operating room.
Satellite operating rooms (ORs) 952, 954 and 956 are each
operatively connected to the central master console via switching
assembly 958, which is selectively controlled by Operator O. While
operating on a first patient P1 in OR 956, the patients in ORs 954
and 952 are being prepared by assistants A2 and A3, respectively.
During the procedure on patient P1, patient P3 becomes fully
prepared for surgery, and A3 begins the surgery on the master
control console dedicated to OR 952 by controlling manipulator
assembly 964. After concluding the operation in OR 956, Operator O
checks with Assistant 43 by inquiring over an audio communications
network between the ORs whether Assistant 43 requires assistance.
OR 950 might additionally have a bank of video monitors showing the
level of activity in each of the ORs, thereby permitting the master
surgeon to determine when it would be best to begin to participate
in the various ongoing surgeries, or to hand control off to others
to continue or complete some of the surgeries.
[0099] Returning to the example, if Assistant 43 requests
assistance, O selects OR 952 via switching assembly 958, selects a
cooperative surgery set-up on an OR-dedicated switching assembly
960, and begins to control manipulator assembly 962. After
completion of the most difficult part of the surgery in OR 952, O
switches over to OR 954, where patient P2 is now ready for
surgery.
[0100] The preceding description is a mere example of the
possibilities offered by the cooperative coupling of masters and
slaves and various telesurgical systems and networks. Other
arrangements will be apparent to one of skill in the art reading
this disclosure. For example, multiple master control rooms can be
imagined in which several master surgeons pass various patients
back and forth depending on the particular part of a procedure
being performed. The advantages of performing surgery in this
manner are myriad. For example, the master surgeon O does not have
to scrub in and out of every procedure. Further, the master surgeon
may become extremely specialized in performing part of a surgical
procedure, e.g., harvesting an IMA, by performing just that part of
a procedure over and over on many more patients than he otherwise
would be able to treat. Thus, particular surgical procedures having
distinct portions might be performed much more quickly by having
multiple surgeons, with each surgeon each performing one part of
the procedure and then moving onto another procedure, without
scrubbing between procedures. Moreover, if one or more patients
(for whatever reason) would benefit by having a surgeon actually be
present, an alternative surgeon (different from the master surgeon)
may be on call to one or more operating rooms, ready to jump in and
address the patient's needs in person, while the master surgeon
moves on treat another patient. Due to increased specialization,
further advances in the quality of medical care may be
achieved.
[0101] In addition to enabling cooperative surgery between two or
more surgeons, operatively hooking two or more operator control
stations together in a telesurgical networking system also may be
useful for surgical training. A first useful feature for training
students or surgeons how to perform surgical procedures would take
advantage of a "playback" system for the student to learn from a
previous operation. For example, while performing a surgical
procedure of interest, a surgeon would record all of the video
information and all of the data concerning manipulation of the
master controls on a tangible machine readable media. Appropriate
recording media are known in the art, and include videocassette or
Digital Video Disk (DVD) for the video images and/or control data,
and Compact Disk (CD), e.g., for the servo data representing the
various movements of the master controls.
[0102] if two separate media are used to record the images and the
servo data, then some method of synchronizing the two would be
desirable during feedback, to ensure that the master control
movements substantially mirror the movements of the slave
manipulators in the video image. A crude but workable method of
synchronization might include a simple time stamp and a watch.
Preferably, both video images and servo data would be recorded
simultaneously on the same recording medium, so that playback would
be automatically synchronized.
[0103] During playback of the operation, a student could place his
hands on the master controls and "experience" the surgery, without
actually performing any surgical manipulations, by having his hands
guided by the master controls through the motions of the slave
manipulators shown on the video display. Such playback might be
useful, for example, in teaching a student repetitive motions, such
as during suturing. In such a situation, the student would
experience over and over how the masters might be moved to move the
slaves in such a way as to tie sutures, and thus hopefully would
learn how better to drive the telesurgical system before having to
perform an operation.
[0104] The principles behind this playback feature can be built
upon by using a live hand of a second operator instead of simple
data playback. For example, two master control consoles may be
connected together in such a way that both masters are assigned to
a single set of surgical instruments. The master controls at the
subordinate console would follow or map the movements of the
masters at the primary console, but would preferably have no
ability to control any of the instruments or to influence he
masters at the primary console. Thus, the student seated at the
subordinate console again could "experience" a live surgery by
viewing the same image as the surgeon and experiencing how the
master controls are moved to achieve desired manipulation of the
slaves.
[0105] An advanced version of this training configuration includes
operatively coupling two master consoles into the same set of
surgical instruments. Whereas in the simpler version, one console
was subordinate to the other at all times, this advanced version
permits both master controls to control motion of the manipulators,
although only one could control movement at any one time. For
example, if the student were learning to drive the system during a
real surgical procedure, the instructor at the second console could
view the surgery and follow the master movements in a subordinate
role. However, if the instructor desired to wrest control from the
student, e.g., when the instructor detected that the student was
about to make a mistake, the instructor would be able to override
the student operator by taking control over the surgical
manipulators being controlled by the student operator. The ability
to so interact would be useful for a surgeon supervising a student
or second surgeon learning a particular operation. Since the
masters on the instructor's console were following the surgery as
if he were performing it, wresting control is a simple matter of
clutching into the surgery and overriding the control information
from the student console. Once the instructor surgeon had addressed
the issue, either by showing the student how to perform a certain
part of the surgical procedure or by performing it himself, the
instructor could clutch out of the operation and permit the student
to continue.
[0106] An alternative to this "on-off" clutching whereby the
instructor surgeon is either subordinate to the student or in
command would be a variable clutch arrangement. For example, again
the instructor is subordinate to the student's performance of a
procedure, and has his masters follow the movement of the student's
master controls. When the instructor desires to participate in the
procedure, but does not desire to wrest all control from the
student, the instructor could begin to exert some control over the
procedure by partially clutching and guiding the student through a
certain step. If the partial control was insufficient to achieve
the instructor's desired result, the instructor could then
completely clutch in and demonstrate the desired move, as above.
Variable clutching could be achieved by adjusting an input device,
such as a dial or a foot pedal having a number of discrete settings
corresponding to the percentage of control desired by the
instructor. When the instructor desires some control, he or she
could operate the input device to achieve a setting of, for
example, 50 percent control, in order to begin to guide the
student's movements. Software could be used to calculate the
movements of the end effectors based on the desired proportionate
influence of the instructor's movements over the student's. In the
case of 50% control, for example, the software would average the
movements of the two sets of master controls and then move the end
effectors accordingly, producing resistance to the student's
desired movement, thereby causing the student to realize his error.
As the surgeon desires more control, he or she could ratchet the
input device to a higher percentage of control, finally taking
complete control as desired.
[0107] Other examples of hooking multiple telesurgical control
stations together for training purposes will be apparent to one of
skill in the art upon reading this disclosure. Although these
training scenarios are described by referring to real surgery,
either recorded or live, the same scenarios could be performed in a
virtual surgical environment, in which, instead of manipulating the
tissue of a patient (human or animal) cadaver, or model, the slave
manipulators could be immersed, in a virtual sense, in simulation
software. The software would then create a simulated virtual
surgical operation in which the instructor and/or student could
practice without the need for a live patient or an expensive model
or cadaver.
[0108] Although the various aspects of the present invention have
been described with respect to a preferred embodiment, it will be
understood that the invention is entitled to full protection within
the full scope of the appended claims.
* * * * *