U.S. patent application number 10/775754 was filed with the patent office on 2005-02-03 for video-based surgical targeting system.
Invention is credited to Chen, David T., McKenna, Michael A., Pieper, Steven D..
Application Number | 20050027186 10/775754 |
Document ID | / |
Family ID | 26958631 |
Filed Date | 2005-02-03 |
United States Patent
Application |
20050027186 |
Kind Code |
A1 |
Chen, David T. ; et
al. |
February 3, 2005 |
Video-based surgical targeting system
Abstract
A video-based surgical targeting system for targeting anatomical
structures within a patient.
Inventors: |
Chen, David T.; (Somerville,
MA) ; Pieper, Steven D.; (Thetford Center, VT)
; McKenna, Michael A.; (Cambridge, MA) |
Correspondence
Address: |
Mark J. Pandiscio
Pandiscio & Pandiscio
470 Totten Pond Road
Waltham
MA
02154
US
|
Family ID: |
26958631 |
Appl. No.: |
10/775754 |
Filed: |
February 10, 2004 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
10775754 |
Feb 10, 2004 |
|
|
|
10104256 |
Mar 21, 2002 |
|
|
|
6690960 |
|
|
|
|
10104256 |
Mar 21, 2002 |
|
|
|
09746992 |
Dec 21, 2000 |
|
|
|
6675032 |
|
|
|
|
60277664 |
Mar 21, 2001 |
|
|
|
Current U.S.
Class: |
600/407 |
Current CPC
Class: |
A61B 34/20 20160201;
A61B 2090/367 20160201; A61B 2034/2055 20160201; A61B 90/36
20160201; A61B 2034/105 20160201; A61B 2090/364 20160201; A61B
90/361 20160201; A61B 2034/102 20160201; A61B 34/25 20160201; A61B
2090/365 20160201 |
Class at
Publication: |
600/407 |
International
Class: |
A61B 005/05 |
Claims
What is claimed is:
1. A video-based surgical targeting system comprising: a
patient-specific database comprising a plurality of 2-D images of
the anatomical structure of a patient; a patient-specific 3-D
computer model of said anatomical structure of said patient, said
patient-specific 3-D computer model being assembled from said
plurality of 2-D images contained in said patient-specific
database; graft placement means for (i) inserting virtual grafts
into said 2-D images contained in said patient-specific database,
and/or (ii) or inserting virtual grafts into said 3-D computer
model; an image generator for generating a virtual image of the
anatomical structure modeled by said patient-specific 3-D computer
model; real-time image generating means for generating a real image
of the anatomical structure of a patient; video mixing means for
mixing said virtual image and said real image into an output image,
whereby said output image may comprise either one of the two images
exclusive of the other, or a composite of both images; and display
means for displaying said output image.
Description
REFERENCE TO PENDING PRIOR APPLICATIONS
[0001] This application:
[0002] (1) is a continuation in part-of-pending prior U.S. patent
application Ser. No. 09/746,992, filed Dec. 21, 2000 by David T.
Chen et al. for VIDEO-BASED SURGICAL TARGETING SYSTEM (Attorney's
Docket No. MMS-4 CON 2), which patent application is hereby
incorporated herein by reference; and
[0003] (2) claims benefit of prior pending U.S. Provisional Patent
Application Ser. No. 60/277,664, filed Mar. 21, 2001 by David T.
Chen et al. for VIDEO-BASED SURGICAL TARGETING SYSTEM (Attorney's
Docket No. MMS-B PROV), which patent application is hereby
incorporated herein by reference.
FIELD OF THE INVENTION
[0004] This invention relates to surgical systems in general, and
more particularly to surgical targeting systems.
BACKGROUND OF THE INVENTION
[0005] Many medical procedures must be carried out at an interior
site which is normally hidden from the view of the physician. In
these situations, the physician typically uses some sort of
scanning device to examine the patient's anatomy at the interior
site prior to, and in preparation for, conducting the actual
procedure itself. These scanning devices typically include MRI
devices, CT scanners, X-ray machines, ultrasound devices and the
like, and serve to provide the physician with a preliminary
knowledge of the patient's internal anatomical structure prior to
commencing the procedure. The physician can then use this
information to plan the procedure in advance, taking into account
patient-specific anatomical structure. In addition, the physician
can also use the information obtained from these scans to more
precisely identify the location of selected structures (e.g. tumors
and the like) which may themselves be located within internal
organs or other internal body structures. As a result, the
physician can more easily "zero in" on such selected structures
during the subsequent procedure, with less trauma to host organs or
other internal body structures. Furthermore, in many cases the
structures of interest may be quite small and difficult to identify
with the naked eye. In these cases, preliminary scanning of the
patient's internal anatomy using high resolution scanning devices
can help the physician locate such structures during the subsequent
procedure.
[0006] In general, scanning devices of the sort described above
tend to generate a series of two-dimensional (i.e., "2-D") images
of the patient's anatomical structure. For example, CT scanners
generate 2-D images which are viewed directly by the physician. By
viewing a plurality of these 2-D images, the physician can mentally
generate a three-dimensional (i.e., "3-D") sense of the patient's
anatomical structure.
[0007] Some scanning devices create large numbers of 2-D images
during the scanning process, with each 2-D image representing a
plane or slice taken through the scanned structure. Furthermore,
some scanning devices also have associated computer hardware and
software for building a 3-D computer model of the patient's scanned
structure using a plurality of these 2-D images. For example, some
MRI devices and some CT scanners leave such associated computer
hardware and software. In these cases, an operator using this
scanning equipment and associated computer hardware and software
can create a number of 2-D images, assemble them into a 3-D
computer model of the scanned structure, and then generate various
images of that 3-D computer model as seen from various angles so as
to enhance understanding of the patient's anatomical structure.
[0008] While the information generated by the aforementioned
scanning devices is generally of great benefit to physicians,
certain significant limitations still exist. For one thing, it can
be very difficult for a physician to physically locate a particular
anatomical structure during a medical procedure, even when that
structure is readily identifiable in a scanned image. This may be
because the structure is very small and difficult to see with the
naked eye, or because it is not readily visible against a
particular background, or because it is itself located within an
internal organ or other internal body structure, etc. For another
thing, even when the structure of interest is successfully located
by the physician, it can sometimes still be extremely difficult for
the physician to reliably direct a medical instrument to that
structure. This may be because the structure is quite small and
difficult to target accurately, or because intervening body
structure mandates a complex approach, etc.
OBJECTS OF THE INVENTION
[0009] Accordingly, one object of the present invention is to
provide a surgical targeting system to facilitate locating a
particular anatomical structure during a medical procedure.
[0010] Another object of the present invention is to provide a
video-based surgical targeting system to facilitate locating a
particular anatomical structure during a medical procedure.
[0011] And another object of the present invention is to provide a
video-based surgical targeting system which permits a series of
patient-specific 2-D images (obtained by scanning patient anatomy
using one or more scanning devices of the type described above) to
be assembled into a 3-D computer model of the patient's scanned
structure.
[0012] Still another object of the present invention is to provide
a video-based surgical targeting system which allows a physician to
view the aforementioned patient-specific 2-D images on a display in
any desired access sequence.
[0013] And another object of the present invention is to provide a
video-based surgical targeting system which allows a physician to
assemble a series of patient-specific 2-D images into a
patient-specific database, and then to generate virtual images from
the aforementioned patient-specific database, as seen from any
desired virtual camera position, for viewing on a display.
[0014] Yet another object of the present invention is to provide a
video-based surgical targeting system which allows a physician to
generate virtual images from the aforementioned 3-D computer model,
as seen from any desired virtual camera position, for viewing on a
display.
[0015] And another object of the present invention is to provide a
video-based surgical targeting system which permits a physician to
place virtual planning markers about any sites of interest while
viewing one or more of the aforementioned patient-specific 2-D
images, with those virtual planning markers then being incorporated
into the 3-D computer model, whereby those virtual planning markers
can be displayed in their appropriate 3-D positions when generating
virtual images of the 3-D computer model.
[0016] Still another object of the present invention is to provide
a video-based surgical targeting system which permits a physician
to place virtual planning markers about any sites of interest while
viewing virtual images of the 3-D computer model, with those
virtual planning markers then being incorporated into the 3-D
computer model, whereby those virtual planning markers can be
displayed in their appropriate positions when generating subsequent
virtual images of the 3-D computer model.
[0017] Still another object of the present invention is to provide
a video-based surgical targeting system which permits a physician
to place virtual planning markers about any sites of interest while
viewing virtual images of the 3-D computer model, with those
virtual planning markers then being incorporated into the 3-D
computer model and into the database of 2-D images, whereby those
virtual planning markers can be displayed in their appropriate
positions when subsequently generating virtual images of the 3-D
computer model or when subsequently displaying 2-D images from the
patient-specific database.
[0018] Yet another object of the present invention is to provide a
video-based surgical targeting system which permits a real image
obtained by a real-time imaging device (e.g. a video camera) to be
displayed to a physician, and which permits a virtual image
generated from the 3-D computer model to be displayed to a
physician, according to the directive of the physician.
[0019] Another object of the present invention is to provide a
video-based surgical targeting system which permits a real image
obtained by a real-time imaging device (e.g. a video camera) to be
appropriately merged with a corresponding virtual image generated
from the 3-D computer model.
[0020] And another object of the present invention is to provide a
video-based surgical targeting system which permits a real image
obtained by a real-time imaging device (e.g. a video camera) to be
appropriately merged with a corresponding virtual image generated
from the 3-D computer model, whereby the two images will be in
registration with one another.
[0021] Still another object of the present invention is to provide
a video-based surgical targeting system which permits a real image
obtained by a real-time imaging device (e.g. a video camera) to be
merged with a corresponding virtual image generated from the 3-D
computer model, whereby the two images will be in registration with
one another, and whereby the physician can choose to display either
one of the two images exclusive of the other, or a composite of
both images simultaneously.
[0022] Yet another object of the present invention is to provide a
video-based surgical targeting system which permits a real image
obtained by a real-time imaging device (e.g. a video camera) to be
merged with a corresponding virtual image generated from the 3-D
computer model, whereby the two images will be simultaneously
displayed in registration with one another, and whereby the
physician can modify the virtual image generated from the 3-D
computer model as needed, by clipping or fading, so as to expose
the virtual planning markers to view, with the virtual planning
markers being superimposed on the real image generated by the
real-time viewing device.
[0023] And another object of the present invention is to provide a
video-based surgical targeting system which permits a real image
generated by a real-time imaging device (e.g. a video camera) to be
merged with a corresponding virtual image generated from the 3-D
computer model, whereby the two images will be simultaneously
displayed in registration with one another, and whereby the
physician can modify the virtual image generated from the 3-D
computer model so as to expose only the virtual planning markers to
view, with the virtual planning markers being superimposed on the
real image generated by the real-time imaging device.
[0024] Still another object of the present invention is to provide
a video-based surgical targeting system which permits a real image
obtained by a real-time imaging device (e.g. a video camera) to be
merged with a corresponding virtual image generated from the 3-D
computer model, whereby the two images will be simultaneously
displayed in registration with one another, and whereby this
registration will be automatically maintained even as the real-time
imaging device is moved about relative to the anatomical site, with
the virtual image being automatically generated so as to follow the
real image.
[0025] Still another object of the present invention is to provide
a video-based surgical targeting system which permits a real image
obtained by a real-time imaging device (e.g. a video camera) to be
merged with a corresponding virtual image generated from the 3-D
computer model, whereby the two images will be simultaneously
displayed in registration with one another, and whereby this
registration will be automatically maintained through the use of a
computerized position and orientation tracker connected to the
imaging device even as the real-tine imaging device is moved about
relative to the anatomical site, with the virtual image being
automatically generated so as to follow the real image.
[0026] Still another object of the present invention is to provide
a video-based surgical targeting system which permits a real image
obtained by a real-time imaging device (e.g. a video camera) to be
merged with a corresponding virtual image generated from the 3-D
computer model, whereby the two images will be simultaneously
displayed in registration with one another, and whereby this
registration will be automatically maintained through the use of a
computer search algorithm based on the real image and the virtual
image even as the real-time imaging device is moved about relative
to the anatomical site, with the virtual image being automatically
generated so as to follow the real image.
[0027] Yet another object of the present invention is to provide a
video-based surgical targeting system which permits a real image
obtained by a real-tine imaging device (e.g. a video camera) to be
merged with a corresponding virtual image generated from the 3-D
computer model, whereby the two images will be simultaneously
displayed in registration with one another, and whereby a surgical
instrument can be tracked along with the real-time imaging device
in order that the surgical instrument can be guided about the
anatomical site using the virtual image generated from the 3-D
computer model.
[0028] And another object of the present invention is to provide a
video-based surgical targeting system which permits a virtual image
to be generated from the 3-D computer model, and further wherein
this virtual image can correspond to an "over the shoulder" view of
the working tip of a surgical instrument located at the anatomical
site.
[0029] And another object of the present invention is to provide an
improved method for locating anatomical structures during a medical
procedure.
SUMMARY OF THE INVENTION
[0030] These and other objects of the present invention are
achieved through the provision and use of a novel video-based
surgical targeting system.
[0031] In one form of the invention, the video-based surgical
targeting system comprises a patient-specific database comprising a
plurality of 2-D images of the anatomical structure of a patient; a
patient-specific 3-D computer model of the anatomical structure of
the patient, the patient-specific 3-D computer model being
assembled from the plurality of 2-D images contained in the
patient-specific database; marker placement means for (i) inserting
virtual planning markers into the 2-D images contained in the
patient-specific database, and/or (ii) adjusting the positions of
virtual planning markers inserted into the 2-D images contained in
the patient-specific database and thereafter incorporated into the
patient-specific 3-D computer model, or inserting virtual planning
markers into the 3-D computer model; an image generator for
generating a virtual image of the anatomical structure modeled by
the patient-specific 3-D computer model; real-time image generating
means for generating a real image of the anatomical structure of a
patient; video mixing means for mixing the virtual image and the
real image into an output image, whereby the output image may
comprise either one of the two images exclusive of the other, or a
composite of both images; and display means for displaying the
output image.
[0032] Preferably the video-based surgical targeting system also
includes registration means for placing the virtual image in
registration with the real image. These registration means may
include means for manually aligning the virtual image with the real
image, or means for automatically aligning the virtual image with
the real image. The registration means may also include tracking
means for tracking the position of the real-time image generating
means.
[0033] Preferably, the video-based surgical targeting system
further comprises instrument tracking means for tracking the
position of a surgical instrument.
[0034] In an alternative form of the invention, the video-based
surgical targeting system comprises a patient-specific database
comprising a plurality of 2-D images of the anatomical structure of
a patient; marker placement means for inserting virtual planning
markers into the 2-D images contained in the patient-specific
database; an image generator for generating a virtual image of the
anatomical structure represented by the plurality of 2-D images
contained in the patient-specific database; real-time image
generating means for generating a real image of the anatomical
structure of a patient; video mixing means for mixing the virtual
image and the real image into an output image, whereby the output
image may comprise either one of the two images exclusive of the
other, or a composite of both images; and display means for
displaying the output image.
[0035] In another alternative form of the invention, the
video-based surgical targeting system comprises a patient-specific
database comprising a plurality of images of the anatomical
structure of a patient; a patient-specific 3-D computer model of
the anatomical structure of the patient, the patient-specific 3-D
computer model being assembled from the plurality of images
contained in the patient-specific database; marker placement means
for (i) inserting virtual planning markers into the images
contained in the patient-specific database, and/or (ii) adjusting
the positions of virtual planning markers inserted into the images
contained in the patient-specific database and thereafter
incorporated into the patient-specific 3-D computer model, or
inserting virtual planning markers into the 3-D computer model; an
image generator for generating a virtual image of the anatomical
structure modeled by the patient-specific 3-D computer model;
real-time image generating means for generating a real image of the
anatomical structure of a patient; video mixing means for mixing
the virtual image and the real image into an output image, whereby
the output image may comprise either one of the two images
exclusive of the other, or a composite of both images; and display
means for displaying said output image.
[0036] In yet another alternative form of the invention, the
video-based surgical targeting system comprises a patient-specific
database comprising a plurality of 2-D images of the anatomical
structure of a patient; marker placement means for inserting
virtual planning markers into the 2-D images contained in the
patient-specific database; an image generator for generating a
virtual image of the anatomical structure defined by the
patient-specific database; real-time image generating means for
generating a real image of the anatomical structure of a patient,
the real-time image generating means being adapted so as to
dynamically update the patient-specific database via the real
image; video mixing means for mixing the virtual image and the real
image into an output image, whereby the output image may comprise
either one of the two images exclusive of the other, or a composite
of both images; and display means for displaying said output
image.
[0037] The present invention also comprises a method for targeting
an anatomical structure using the video-based surgical targeting
system.
BRIEF DESCRIPTION OF THE DRAWINGS
[0038] These and other objects and features of the present
invention will be more fully disclosed or rendered obvious by the
following detailed description of the preferred embodiments of the
invention, which are to be considered together with the
accompanying drawings wherein like numbers refer to like parts and
further wherein:
[0039] FIG. 1 is a schematic view of the major components of a
video-based surgical targeting system formed in accordance with the
present invention;
[0040] FIG. 2 is a view of an exemplary patient-specific 2-D image
of the sort contained in the patient-specific database;
[0041] FIG. 3 is a view of an exemplary patient-specific 2-D image
of the sort contained in the patient-specific database, wherein the
image has had several virtual planning markers placed into the
image;
[0042] FIG. 4 is a composite view of (i) a real image obtained by a
real-time imaging device (e.g. a video camera), and (ii) a virtual
image generated from the 3-D computer model, wherein the two images
are not yet in registration with one another;
[0043] FIG. 5 is a composite view of (i) a real image obtained by a
real-time imaging device (e.g. a video camera), and (ii) a virtual
image generated from the 3-D computer model, wherein the two images
have been placed in registration with one another;
[0044] FIG. 6 is a view like that of FIG. 5, except that the
virtual image generated from the 3-D computer model has been sliced
away to expose the virtual planning markers;
[0045] FIG. 7 is a view like that of FIG. 5, except that the
virtual image generated from the 3-D computer model has been faded
away to expose the virtual planning markers;
[0046] FIG. 8 is a schematic view showing the positional
relationships between various elements of the video-based surgical
targeting system and the patient's anatomical structure, as well as
the positional relationship between an exemplary surgical
instrument and the patient's anatomical structure;
[0047] FIG. 9 is a virtual image generated from the 3-D computer
model, showing a virtual "over the shoulder" view from the tip of a
tracked surgical instrument; and
[0048] FIG. 10 is a flowchart illustrating one way of operating a
video-based surgical targeting system formed in accordance with the
present invention.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0049] Looking first at FIG. 1, the starting point for the present
invention involves a data acquisition device 5 which is adapted to
generate patient-specific 2-D images. For example, data acquisition
device 5 might comprise an MRI device, a CT scanner or any other
scanning device of the sort adapted to provide a series of 2-D
images of the patient's anatomical structure, where each 2-D image
corresponds to a plane or slice taken through the scanned
structure. Alternatively, data acquisition device 5 might comprise
an X-ray machine or other imaging machine adapted to provide a set
of 2-D images of the patient's anatomical structure, where each 2-D
image corresponds to a data set relating to the scanned
structure.
[0050] The patient-specific 2-D images generated by data
acquisition device 5 are stored in a patient-specific database 10.
Patient-specific database 10 is constructed so that the
patient-specific 2-D images can be accessed individually or in any
particular sequence desired. Preferably patient-specific database
10 comprises an appropriate set of data contained in a computer
storage system.
[0051] The patient-specific 2-D images stored in patient-specific
database 10 are used in building a patient-specific 3-D computer
model 15. Patient-specific 3-D computer model 15 may be built from
the information stored in patient-specific database 10 using any
one of the many algorithms well known in the art, so long as the
patient-specific 3-D computer model is capable of generating
virtual images of the patient's anatomical structure from
substantially any location where a real camera may be positioned
relative to that anatomical structure during a given procedure.
Alternatively, patient-specific 3-D computer model 15 may be built
from the information stored in patient-specific database 10 using
any one of the many algorithms well known in the art, where the
patient-specific 3-D computer model is capable of generating
virtual images of the patient's anatomical structure from a
predetermined set of possible virtual camera positions. Preferably,
patient-specific 3-D computer model 15 is constructed so that it is
capable of generating virtual images of the patient's anatomical
structure from any location whatsoever, with any desired field of
view, and including any other camera-specific criteria desired
(e.g. particular camera characteristics such as focus, optical
characteristics, optical distortions, etc.). By way of example, one
possible algorithm for building the patient-specific 3-D computer
model out of the information stored in the patient-specific
database 10 might be the Marching Cubes algorithm.
[0052] In one preferred embodiment of the invention,
patient-specific 3-D computer model 15 comprises an appropriate
computer software model resident on an appropriate digital
computer. Patient-specific 3-D computer model 15 is preferably
constructed using a plurality of polygonal models to model the
patient's real anatomical structure. As is well known in the art,
such polygonal models generally comprise a collection of points
defining the surface of the 3-D computer model, along with some
connectivity information relating to how these surface points are
connected to one another. Preferably there is one polygonal model
per anatomical structure being modeled. Of course, the choice of
which particular computer software model is used to model a
particular anatomical structure ultimately depends on a variety of
factors, such as the particular anatomical structure being modeled,
the computer hardware available, the volume of data to be handled,
the particular medical procedure which is to be conducted, etc.
[0053] Of course, it is also anticipated that the patient-specific
3-D computer model could be dynamic as well as static. For example,
dynamic changes could occur in the model as a result of computer
simulation (e.g. through the use of a Finite Element Model) or
because more current real-time data (e.g. from video camera 45) is
used to update the model, or both.
[0054] It is also anticipated that patient-specific 3-D computer
model 15 might include data from sources other than the 2-D images
contained in patient-specific database 10. For example,
patient-specific 3-D computer model 15 might include information
obtained from a 3-D surface digitizer such as that used in certain
types of cranial surgery, or patient-specific 3-D computer model 15
might include information obtained from a temperature probe or a
chemical probe. In fact, in addition to the 2-D image data obtained
from patient-specific database 10, patient-specific 3-D computer
model 15 might include substantially any type of information
gathered by almost any type of device or sensor or process.
[0055] An integral part of the present invention involves the
placement of virtual planning markers into the patient-specific
data, using apparatus 20. This may be done by placing such virtual
planning markers into one or more of the patient-specific 2-D
images contained in patient-specific database 10 before
patient-specific 3-D computer model 15 is created; or by placing
such virtual planning markers into patient-specific 3-D computer
model 15 after that model has been created; or by some combination
of the two.
[0056] More particularly, apparatus 20 preferably comprises
computer hardware and software adapted to allow a physician to
access one or more of the patient-specific 2-D images contained in
patient-specific database 10 and present them for viewing, in the
manner shown in FIG. 2. Then, using a mouse or other data entry
device, the physician can place one or more virtual planning
markers 25 into an accessed patient-specific 2-D image, in the
manner shown in FIG. 3. These virtual planning markers 25 can
consist of substantially any geometric form such as a point, a
line, a circle, a plane, a path (either straight or curved) etc.,
and are positioned about anatomical structures of particular
interest to the physician. For example, virtual planning markers 25
might be placed about a suspected tumor, or they might be placed
about particularly sensitive anatomical structures such as vascular
tissue or nerve bundles, etc. Naturally, the particular geometric
form chosen for a particular virtual planning marker may depend on
a variety of factors, such as the particular anatomical structure
involved, the computer hardware available, the volume of data to be
handled, the particular medical procedure to be conducted, etc.
Regardless of the particular form of virtual planning marker
chosen, each virtual planning marker 25 which is placed on the 2-D
image using apparatus 20 is then stored in patient-specific
database 10 along with its associated 2-D image. As a result, when
patient-specific 3-D computer model 15 is subsequently created from
the data contained in patient-specific database 10,
patient-specific 3-D computer model 15 will incorporate virtual
planning markers 25 into the model.
[0057] Alternatively, a physician may use apparatus 20 to access
patient-specific 3-D computer model 15 and then, using a mouse or
other data entry device, adjust the position of one or more of the
virtual planning markers 25 which were previously placed into
patient-specific database 10 (and thus incorporated into
patient-specific 3-D computer model 15 when that 3-D computer model
was created). The adjusted positions of these virtual planning
markers 25 are then stored in patient-specific 3-D computer model
15. Furthermore, an updated 2-D image of these virtual planning
markers can then be incorporated into the patient-specific database
10.
[0058] Furthermore, where patient-specific 3-D computer model 15
has not already had a particular virtual planning marker 25
incorporated therein, a physician may also use apparatus 20 to
access patient-specific 3-D computer model 15, and then use a mouse
or other data entry device to insert one or more virtual planning
markers 25 directly into patient-specific 3-D computer model
15.
[0059] Once virtual planning markers 25 have been properly
positioned in patient-specific 3-D computer model 15, virtual
images incorporating these virtual planning markers can be created
by an image generator 30, fed through a video mixing device 35, and
then presented to the physician on display 40.
[0060] At the same time, real images obtained from a video camera
45 can be fed through video mixing device 35 and then presented to
the physician on display 40.
[0061] More particularly, image generator 30 comprises computer
hardware and software of the sort well known in the art for
generating virtual images from patient-specific 3-D computer model
15.
[0062] Video mixing device 35 comprises a video mixing device of
the sort well known in the art, whereby the surgeon can selectively
display (i) a virtual image created by image generator 30 from
patient-specific 3-D computer model 15, (ii) a real image obtained
from video camera 45, or (iii) a composite of the virtual image and
the real image, where the virtual image is superimposed against the
real image.
[0063] In order for the foregoing composite image to be more useful
to the physician, the virtual image generated from patient-specific
3-D computer model 15 should be placed into registration with the
real image obtained from video camera 45. This is done in several
steps.
[0064] First, and looking now at FIG. 4, a virtual image 50 is
created by image generator 30, fed through video mixing device 35,
and presented on display 40. Simultaneously, a real image 55 is
captured by video camera 45, fed through video mixing device 35,
and presented on display 40. Video mixing device 35 is arranged so
as to present virtual image 50 and real image 55 simultaneously on
display 40, with the virtual image being superimposed on the real
image (i.e., so that the virtual image is in the foreground and the
real image is in the background, in the manner shown in FIG.
4).
[0065] Second, and looking now at FIG. 5, the two images are placed
into proper registration with one another. This image registration
can be accomplished in either one of two ways.
[0066] A first, and generally more preferable, technique involves
holding the position of video camera 45 (and hence real image 55)
constant and moving the position of the "virtual object" or the
"virtual camera" by means of apparatus 57 until the virtual image
50 is brought into registration with real image 55. This can be
done automatically by having image generator 30 use a search
algorithm to match the virtual image to the real image, in which
case apparatus 57 includes computer hardware and software of the
sort well known in the art to cause image generator 30 to work
through a search algorithm to match the virtual image to the real
image; or it can be done manually, in which case apparatus 57
includes computer hardware and software of the sort well known in
the art to allow the physician to drag the virtual image 50 into
registration with real image 55, using a mouse or other data entry
device attached to image generator 30. One convenient way that
mouse motion can be used to control the 3-D movements or the
patient-specific model is to map such motion into vectors defined
by the view direction of the virtual camera, as will be well known
to persons skilled in the art.
[0067] A second, and generally less preferable, technique involves
holding the virtual image 50 constant and moving video camera 45
(or the patient) until real image 55 matches virtual image 50.
[0068] Regardless of which technique is used, once virtual image 50
has been matched to real image 55, the position of the "virtual
camera" (i.e., the location from which the virtual image appears to
be seen) will be matched to the actual position of video camera
45.
[0069] Once virtual image 50 has been placed into proper
registration with real image 55, image generator 30, video camera
45 and video mixing device 35 can be used to present the virtual
and real images on display 40 in various presentation formats so as
to facilitate a particular medical procedure. In particular, one
can use image generator 30, video camera 45 and video mixing device
35 to superimpose a virtual image (generated from patient-specific
3-D computer model 15) against a real image (generated by video
camera 45), with image generator 30 being directed to modify the
virtual image so as to expose one or more of the virtual planning
markers 25 present in patient-specific 3-D computer model 15,
whereby the anatomy highlighted by virtual planning markers 25 will
be brought quickly to the attention of the physician.
[0070] In this respect it is to be appreciated that since virtual
planning markers 25 will frequently be located within interior
portions of the structure modeled by patient-specific 3-D computer
model 15, exposing the otherwise-hidden virtual planning markers to
view will involve rendering some or all of the virtual anatomical
structure transparent or semi-transparent. This can be done either
by (i) slicing away any portions of the virtual image required in
order to expose virtual planning markers 25, whereby those virtual
planning markers will be rendered visible against the background
real image, in the manner shown in FIG. 6; or (ii) fading away some
or all of the virtual image so as to expose such virtual planning
markers against the real image, in the manner shown in FIG. 7. As a
result, a physician can then conduct a medical procedure with the
confidence of using real images generated by video camera 45, while
having virtual planning markers 25 superimposed against the real
image of the anatomical structure so as to help guide the
procedure.
[0071] While the foregoing system constitutes a major improvement
over the prior art, it has also been recognized that it can itself
be improved upon significantly by adding camera tracking means to
video camera 45.
[0072] More particularly, in many medical procedures, the position
of video camera 45 may change on a fairly frequent basis. Since it
is important for virtual image 50 (generated from patient-specific
3-D computer model 15) to remain in proper registration with real
image 55 (obtained from video camera 45), it is necessary for the
video-based surgical targeting system to reestablish proper
correspondence between the two images each time the video camera
moves. As noted above, this proper correspondence can be
reestablished each time video camera 45 moves, by either (i) having
the physician manually drag virtual image 50 into registration with
real image 55, using a mouse or other data entry device attached to
image generator 30, or (ii) having the system use a search
algorithm to match the virtual image to the real image. While both
of these procedures are generally capable (assuming that the
anatomical structure remains stationary) of accurately yielding the
desired image correspondence, each one suffers from a significant
disadvantage. In the first case, requiring the physician to
physically drag the virtual image into registration with the real
image each time the video camera moves can be inconvenient where
the video camera is moved about frequently. In the second case, it
can take the system a substantial amount of time to solve for the
correct image correspondence when using a search algorithm, even
where high speed hardware is utilized. Even where speed is not a
problem, automated search strategies of the type generally used are
prone to errors through false matches.
[0073] Accordingly, it has been recognized that by putting a
tracker on video camera 45, the position of the real camera can be
continuously monitored, whereby each time the real camera moves,
the position of the virtual camera can be correspondingly adjusted
by image generator 30. In this way, the virtual image generated by
image generator 30 can quickly and easily be maintained in
registration with the real image, regardless of how often video
camera 45 is moved.
[0074] More particularly, and looking now at FIG. 8, there is shown
a video camera 45, an anatomical structure 60, and a tracker system
65. Tracker system 65 comprises a tracker 70 which is attached to
video camera 45, and a tracker base 75 which defines the coordinate
system of the tracker system.
[0075] In this setting, M.sub.PC can be considered to represent the
matrix transformation from the patient's anatomical structure 60 to
camera 45; M.sub.CT can be considered to represent the matrix
transformation from camera 45 to tracker base 75; and M.sub.PT can
be considered to represent the matrix transformation from
anatomical structure 60 to tracker base 75.
[0076] M.sub.CT is known from the tracker system. Furthermore, once
the virtual image generated by image generator 30 has been placed
in registration with the real image generated by camera 45, the
virtual camera position will be known relative to the virtual
anatomical structure, and hence the real camera position will be
known relative to the real anatomical structure. Thus, real matrix
M.sub.PC will also be known. In addition, since M.sub.CT and
M.sub.PC are then both known, it is possible to solve for M.sub.PT.
Accordingly, the position of anatomical structure 60 will then also
be known within the relative coordinate system defined by the
tracker system.
[0077] In view of the foregoing, the virtual image generated by
image generator 30 can be quickly and easily maintained in
registration with the real image, regardless of how often camera 45
is moved.
[0078] Furthermore, another tracker 80 can be positioned on a
surgical instrument 85. In this case, tracker 80 will provide
M.sub.T1, where M.sub.T1 represents the matrix transformation from
tracker base 75 to instrument 85. Accordingly, once M.sub.PT and
M.sub.T1 are known, it is possible to solve for M.sub.P1, where
M.sub.P1 represents the matrix transformation from the patient's
anatomical structure 60 to instrument 85. With M.sub.P1 known, it
is then possible to track the position of surgical instrument 85
relative to anatomical structure 60. This is a very powerful tool,
since it allows a virtual representation of surgical instrument 85
to be accurately added to the properly registered virtual and real
images presented on display 40, i.e., it allows a virtual
representation of instrument 85 to be shown in proper registration
with a real image of anatomical structure 60 on display 40.
[0079] Furthermore, as the position of surgical instrument 85 is
tracked relative to anatomical structure 60, it is also possible to
provide a virtual image of surgical instrument 85 as that surgical
instrument 85 moves through the anatomical structure, even when
some or all of that instrument might be hidden from the view of
video camera 45. For example, and looking now at FIG. 9, it is
possible to generate a virtual "over the shoulder" view of the
distal tip of surgical instrument 85 moving through anatomical
structure 60, where that tip will be shown in proper correspondence
to various structures, e.g. a tumor 95, vascular structures 100,
etc. It is also possible to generate an additional virtual object
90 showing a line extended along the axis of the surgical
instrument 85 so that the physician can see where the surgical
instrument would go if moved further along its current trajectory.
Of course, since the virtual image generated by the video-based
surgical targeting system will then differ significantly from the
real image generated by video camera 45, video mixing device 35
should be directed to totally suppress the real image generated by
video camera 45 so that it will no longer be shown on display 40,
in order to avoid confusing the physician.
MODIFICATIONS OF THE PREFERRED EMBODIMENTS
[0080] It is, of course, possible to modify the preferred
embodiments disclosed above without departing from the scope of the
present invention.
[0081] Thus, for example, while only one camera 45 has been
disclosed above, it should be appreciated that two or more cameras
45 may be used. Furthermore, it should also be appreciated that
camera 45 may comprise a video camera, or it may comprise an
endoscope, or it may comprise some other type of real-time image
capturing means, e.g. it may comprise an ultrasound device.
[0082] Additionally, it should be recognized that the system might
be adapted so that patient-specific database 10 comprises non-2-D
images, e.g. database 10 might comprise one or more images
generated by a Cyberware 3-D scanner.
[0083] Alternatively, patient-specific 3-D computer model 15 might
be omitted from the apparatus entirely. In this case, the virtual
images could be generated from patient-specific database 10 by
image generator 30 through the use of a volume rendering procedure
of the sort well known in the art.
[0084] It is also anticipated that one could use a tracked surgical
instrument 85 to determine the location of anatomical structure 60.
This can be accomplished by using the tracked surgical instrument
85 to engage known fiducial points on the anatomical structure.
Alternatively, a tracked surgical instrument 85 can be used to
sample multiple surface points located on the anatomical structure
and then use a data matching procedure to correlate the sampled
points with either patient-specific database 10 or patient-specific
3-D computer model 15. By way of example, a least squares fit might
be used to correlate the sampled points with the 3-D computer
model.
[0085] Also, it should be appreciated that the video-based
targeting system could be used to target objects in non-medical
applications, e.g. it could be used to target objects concealed
within the interior of complex machines, or objects (e.g. a
conduit) concealed beneath surface of a structure (e.g. the floor
of a building).
[0086] Furthermore, it is possible to place tracking means directly
on the patient so as to track the position of the patient's
anatomical structure.
[0087] Still other changes will be obvious to a person skilled in
the art.
ADVANTAGES OF THE PRESENT INVENTION
[0088] Numerous advantages are obtained through the use of the
present invention.
[0089] For one thing, a surgical targeting system is provided to
facilitate locating a particular anatomical structure during a
medical procedure.
[0090] For another thing, a video-based surgical targeting system
is provided to facilitate locating a particular anatomical
structure during a medical procedure.
[0091] And a video-based surgical targeting system is provided
which permits a series of patient-specific 2-D images (obtained by
scanning patient anatomy using one or more scanning devices of (the
type described above) to be assembled into a 3-D computer model of
the patient's scanned structure.
[0092] Also, a video-based surgical targeting system is provided
which allows a physician to view the aforementioned
patient-specific 2-D images on a display in any desired access
sequence.
[0093] Furthermore, a video-based surgical targeting system is
provided which allows a physician to assemble a series of
patient-specific 2-D images into a patient-specific database, and
then to generate virtual images from the aforementioned
patient-specific database, as seen from any desired virtual camera
position, for viewing on a display.
[0094] And a video-based surgical targeting system is provided
which allows a physician to generate virtual images from the
aforementioned 3-D computer model, as seen from any desired virtual
camera position, for viewing on a display.
[0095] In addition, a video-based surgical targeting system is
provided which permits a physician to place virtual planning
markers about any sites of interest while viewing one or more of
the aforementioned patient-specific 2-D images, with those virtual
planning markers then being incorporated into the 3-D computer
model, whereby those virtual planning markers can be displayed in
their appropriate 3-D positions when generating virtual images of
the 3-D computer model.
[0096] And a video-based surgical targeting system is provided
which permits a physician to place virtual planning markers about
any sites of interest while viewing virtual images of the 3-D
computer model, with those virtual planning markers then being
incorporated into the 3-D computer model, whereby those virtual
planning markers can be displayed in their appropriate positions
when subsequently generating virtual images of the 3-D computer
model.
[0097] And a video-based surgical targeting system is provided
which permits a physician to place virtual planning markers about
any sites of interest while viewing virtual images of the 3-D
computer model, with those virtual planning markers then being
incorporated into the 3-D computer model and into the database of
2-D images, whereby those virtual planning markers can be displayed
in their appropriate positions when subsequently generating virtual
images of the 3-D computer model or when subsequently displaying
2-D images from the patient-specific database.
[0098] For another thing, a video-based surgical targeting system
is provided which permits a real image obtained by a real-time
imaging device (e.g. a video camera) to be displayed to a
physician, and which permits a virtual image generated from the 3-D
computer model to be displayed to a physician, according to the
directive of the physician.
[0099] And a video-based surgical targeting system is provided
which permits a real image obtained by a real-time imaging device
(e.g. a video camera) to be appropriately merged with a
corresponding virtual image generated from the 3-D computer
model.
[0100] Also, a video-based surgical targeting system is provided
which permits a real image obtained by a real-time imaging device
(e.g. a video camera) to be appropriately merged with a
corresponding virtual image generated from the 3-D computer model,
whereby the two images will be in registration with one
another.
[0101] Furthermore, a video-based surgical targeting system is
provided which permits a real image obtained by a real-time imaging
device (e.g. a video camera) to be merged with a corresponding
virtual image generated from the 3-D computer model, whereby the
two images will be in registration with one another, and whereby
the physician can choose to display either one of the two images
exclusive of the other, or a composite of both images
simultaneously.
[0102] And a video-based surgical targeting system is provided
which permits a real image obtained by a real-time imaging device
(e.g. a video camera) to be merged with a corresponding virtual
image generated from the 3-D computer model, whereby the two images
will be simultaneously displayed in registration with one another,
and whereby the physician can modify the virtual image generated
from the 3-D computer model as needed, by clipping or fading, so as
to expose the virtual planning markers to view, with the virtual
planning markers being superimposed on the real image generated by
the real-time imaging device.
[0103] And a video-based surgical targeting system is provided
which permits a real image generated by a real-time imaging device
(e.g. a video camera) to be merged with a corresponding virtual
image generated from the 3-D computer model, whereby the two images
will be simultaneously displayed in registration with one another,
and whereby the physician can modify the virtual image generated
from the 3-D computer model so as to expose only the virtual
planning markers to view, with the virtual planning markers being
superimposed on the real image generated by the real-time imaging
device.
[0104] Moreover, a video-based surgical targeting system is
provided which permits a real image obtained by a real-time imaging
device (e.g. a video camera) to be merged with a corresponding
virtual image generated from the 3-D computer model, whereby the
two images will be simultaneously displayed in registration with
one another, and whereby this registration will be automatically
maintained even as the real-time imaging device is moved about
relative to the anatomical site, with the virtual image being
automatically generated so as to follow the real image.
[0105] And a video-based surgical targeting system is provided
which permits a real image obtained by a real-time imaging device
(e.g. a video camera) to be merged with a corresponding virtual
image generated from the 3-D computer model, whereby the two images
will be simultaneously displayed in registration with one another,
and whereby this registration will be automatically maintained
through the use of a computerized position and orientation tracker
connected to the imaging device even as the real-time imaging
device is moved about relative to the anatomical site, with the
virtual image being automatically generated so as to follow the
real image.
[0106] Also, a video-based surgical targeting system is provided
which permits a real image obtained by a real-time imaging device
(e.g. a video camera) to be merged with a corresponding virtual
image generated from the 3-D computer model, whereby the two images
will be simultaneously displayed in registration with one another,
and whereby this registration will be automatically maintained
through the use of a computer search algorithm based on the real
image and the virtual image even as the real-time imaging device is
moved about relative to the anatomical site, with the virtual image
being automatically generated so as to follow the real image.
[0107] And a video-based surgical targeting system is provided
which permits a real image obtained by a real-time imaging device
(e.g. a video camera) to be merged with a corresponding virtual
image generated from the 3-D computer model, whereby the two images
will be simultaneously displayed in registration with one another,
and whereby a surgical instrument can be tracked along with the
real-time imaging device in order that the surgical instrument can
be guided about the anatomical site using the virtual image
generated from the 3-D computer model.
[0108] In addition, a video-based surgical targeting system is
provided which permits a virtual image to be generated from the 3-D
computer model, and further wherein this virtual image can
correspond to an "over the shoulder" view of the working tip of a
surgical instrument located at the anatomical site.
[0109] And an improved method for locating anatomical structures
during a medical procedure is provided.
[0110] In the preceding description, data acquisition device 5 was
used to generate patient-specific 2-D images, and these images were
stored in patient-specific database 10; the patient-specific 2-D
images stored in patient-specific database 10 were used to build
patient-specific 3-D computer model 15 viewable with image
generator 30; and the output of image generator 30 and video source
45 were connected through video mixing device 35, whereby the
surgeon can selectively display (i) a virtual image created by
image generator 30 from patient-specific 3-D computer model 15, or
(ii) a real image obtained from video source 45, or (iii) a
composite of the virtual image and the real image, where the
virtual image is superimposed against the real image.
[0111] In addition, means 20 are provided for the placement of
virtual planning markers 25 into the patient-specific data; this
may be done by placing such virtual planning markers into one or
more of the patient-specific 2-D images contained in
patient-specific database 10, or by placing such virtual planning
markers into patient-specific 3-D computer model 15, or by some
combination of the two. In accordance with one aspect of the
invention, virtual planning markers 25 placed in the
patient-specific 2-D images stored in patient-specific database 10
will be automatically incorporated into patient-specific 3-D
computer model 15, and virtual planning markers 25 placed into
patient-specific 3-D computer model 15 will be automatically
incorporated into the patient-specific 2-D images stored in
patient-specific database 10.
[0112] In this respect it should be appreciated that video source
45 may comprise a video camera, an endoscope, an ultrasound device,
a fluoroscope, or any other type of appropriate image capturing
means.
[0113] It should also be appreciated that it is possible to create
virtual planning markers 25 having a form other than the
sphere-like markers shown in FIG. 9.
[0114] By way of example, and looking now at FIG. 11, there is
shown the aorta 105 of a patient as generated by a video source 45,
e.g., a fluoroscope. Also shown is a series of sphere-like markers
25A placed into the system (e.g., by a human operator using a
mouse) and a series of line segments 25B extending between selected
ones of the sphere-like markers 25A. These sphere-like markers 25A
and line segments 25B may be used to plan a surgical procedure, to
determine anatomical lengths or angles, etc.
[0115] Also shown is a straight tube 25C which may also be used for
planning and measurement purposes, etc., a curved tube 25D which
may be used for planning and measurement purposes, and a box 25E
which may be used for planning and measurement purposes, e.g., for
volume calculations.
[0116] In addition, other geometric elements such as curved lines,
intersecting lines, etc. may also be provided for planning and
measurement purposes.
[0117] Significantly, it is also possible to insert into the system
virtual planning markers representing virtual grafts, virtual
implants, etc. By way of example, in FIG. 12 there is shown a
virtual graft 25F which represents an arterial stent which may be
deployed in the aorta, e.g., to treat an aortic aneurysm.
[0118] FIG. 12 is a view showing a virtual graft 25F positioned on
aorta 105.
* * * * *