U.S. patent application number 12/526886 was filed with the patent office on 2010-04-29 for an imaging device and method.
This patent application is currently assigned to NATIONAL UNIVERSITY OF SINGAPORE. Invention is credited to Beng Hai Lim, Timothy Poston, James Kolenchery Rappel.
Application Number | 20100103247 12/526886 |
Document ID | / |
Family ID | 39690357 |
Filed Date | 2010-04-29 |
United States Patent
Application |
20100103247 |
Kind Code |
A1 |
Lim; Beng Hai ; et
al. |
April 29, 2010 |
AN IMAGING DEVICE AND METHOD
Abstract
An imaging system comprising an image capture apparatus,
arranged to capture a stereoscopic image of an operator work site,
in communication with a display system; the display system arranged
to receive and display said stereoscopic image on a display screen
to said operator; wherein said display system is arranged such that
the display screen is placed intermediate the operator's eyes and
the work site.
Inventors: |
Lim; Beng Hai; (Singapore,
SG) ; Poston; Timothy; (Bangalore, IN) ;
Rappel; James Kolenchery; (singapore, SG) |
Correspondence
Address: |
ALSTON & BIRD LLP
BANK OF AMERICA PLAZA, 101 SOUTH TRYON STREET, SUITE 4000
CHARLOTTE
NC
28280-4000
US
|
Assignee: |
NATIONAL UNIVERSITY OF
SINGAPORE
Crescent
SG
|
Family ID: |
39690357 |
Appl. No.: |
12/526886 |
Filed: |
February 13, 2008 |
PCT Filed: |
February 13, 2008 |
PCT NO: |
PCT/SG08/00053 |
371 Date: |
December 22, 2009 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
60889674 |
Feb 13, 2007 |
|
|
|
Current U.S.
Class: |
348/47 ;
348/E13.074 |
Current CPC
Class: |
A61B 2090/502 20160201;
A61B 2090/371 20160201; A61B 90/36 20160201; A61B 90/50 20160201;
H04N 13/332 20180501; A61B 2090/367 20160201; G02B 30/24 20200101;
A61B 2090/372 20160201; H04N 13/239 20180501; A61B 90/361 20160201;
A61B 90/37 20160201; A61B 90/20 20160201 |
Class at
Publication: |
348/47 ;
348/E13.074 |
International
Class: |
H04N 13/02 20060101
H04N013/02 |
Claims
1. An imaging system comprising an image capture apparatus,
arranged to capture a stereoscopic image of an operator work site,
in communication with a display system; the display system arranged
to receive and display said stereoscopic image on a display screen
to said operator; wherein said display system is arranged such that
the display screen is placed intermediate the operator's eyes and
the work site.
2. The system according to claim 1, wherein the operator's eyes,
the display screen and work space are aligned.
3. The system according to claim 1, wherein the display screen is
supported by a selectively movable frame so as to be capable of
selectively moving into and out of alignment.
4. The system according to claim 3, wherein said movable frame
comprises a resiliently deformable arm mounted between said frame
and an anchor point, said arm capable of deforming to a range of
positions subject to the requirements of the operator.
5. The system according to claim 1, wherein the image capture
system includes a pair of cameras, the difference in images
captured by said cameras consistent with the difference in image
captured by the unassisted eye.
6. The system according to claim 5, wherein said image capture
system further includes mirrors corresponding to each camera, for
directing the respective images to said corresponding cameras.
7. The system according to claim 5, further including shutter
glasses, worn by the operator and arranged to provide alternating
images shown on the display screen to the operator.
8. The system according to claim 5, wherein the display screen
displays the images from each camera directed in differing
directions so as permit each image to correspond with the relevant
eye.
9. The system according to claim 1, wherein the image capture
system includes a camera and two mirrors, said mirrors arranged to
receive different images of said work site, and direct these to
said camera, said camera arranged to output said images to said
display device in an alternating sequence.
10. The system according to claim 6, wherein said mirrors are
curved.
11. The system according to claim 1, wherein the display system
includes two display screens so as to permit two operators working
simultaneously within the work site.
12. The system according to claim 1, wherein the display screen
alternates four images corresponding to two pairs of images so as
to permit two operators working simultaneously within the work
site.
13. A method for displaying an image, comprising the steps of:
capturing a stereoscopic image of an operator work site
communicating said image to a display system; displaying said
stereoscopic image on a display screen placed intermediate the
operator's eyes and the work site.
14. The method according to claim 13, wherein the capturing step
comprises capturing two images using a pair of cameras, the
difference in images captured by said cameras consistent with the
difference in image captured by the unassisted eye.
Description
FIELD OF THE INVENTION
[0001] The present invention relates to the fields of stereo
microscopy and digital stereo display.
BACKGROUND OF THE ART
[0002] Modern reconstructive surgery was made possible by the
stereo microscope. Early work showed that skin cannot simply be
grafted--cut from one place and reattached in another--without
attention to its blood supply. With no artery feeding it, it dies.
Maintaining the old blood vessels while new ones grow from the
attachment site places narrow constraints, on what donor and target
sites can be combined.
[0003] This problem is solved by the transplant of more than skin:
a surgical flap includes underlying tissue, and blood vessels which
the surgeon joins to vessels in the target site. Since the vessels
involved may be less than 1 mm in diameter, the accurate placement
of six small sutures for a join where the blood flow without
leaking requires magnification. This means more than merely an
enlarged view. Dexterity in suturing requires depth perception, so
that the needle can penetrate at a correct angle, including angles
away from the viewer. The depth cue of parallax is unavailable
through a microscope with a fixed viewpoint, perspective is
unhelpful in a view with no straight lines and limited focal depth,
and occlusion cannot show how far above the needle is above the
tissue. It is essential to have the depth cue of stereopsis, with a
lens system for each eye delivering views from different angles, to
the two eyepieces. We refer to a pair of views that permit
stereopsis as a stereo view or stereoscopic view, or as one having
stereo. If the difference is correctly structured, the user's brain
integrates the two views into a single scene with perceived depth,
just as for direct vision with two eyes. Much surgery depends
critically on this, as does dexterous work in other domains, such
as industrial micro-assembly.
[0004] However, the surgical microscope requires that the user's
head remain perfectly fixed, keeping the eyes to the eyepieces,
throughout a long series of delicate procedures with substantial
risk. This is an important source of stress on the surgeon, causing
pain in the neck and back, and requiring several rest pauses per
hour.
[0005] An alternative to enlarged display though optical lenses is
to show on an electronic display the output of a real-time camera,
digital or analogue. This technology is available, but in forms
that fail to support stereo, that require the user to look away
from the hands at a rotated view, or have both these problems. (A
view rotated from the natural direction requires the user to handle
the fact that "to turn the instrument in the image this way, I must
turn my hands that way," adding to the cognitive difficulty, strain
and learning curve of the task.
STATEMENT OF INVENTION
[0006] Therefore, in a first aspect, the invention provides an
imaging system comprising an image capture apparatus, arranged to
capture a stereoscopic image of an operator work site, in
communication with a display system; the display system arranged to
receive and display said stereoscopic image on a display screen to
said operator; wherein said display system is arranged such that
the display screen is placed intermediate the operator's eyes and
the work site.
[0007] In a second aspect, the invention provides a method for
displaying an image, comprising the steps of: capturing a
stereoscopic image of an operator work site; communicating said
image to a display system; displaying said stereoscopic image on a
display screen placed intermediate the operator's eyes and the work
site.
[0008] A panel is placed over the work site, at a height sufficient
to allow the insertion of instruments or tools. On this panel may
be placed a fast display (LCD or OLED, or other such technologies
as they arise, and which will be clear to the skilled person)
alternating between enlarged views of left and right camera images
from below the panel. The two distinct images may come from two
cameras, or alternatively by suitable mirror arrangements from a
single camera. Depending on the optical layout, the camera or
cameras may be entirely under the panel, or partially protrude from
under it.
[0009] The operators or operators (as in the case of two surgeons
cooperating in a single procedure) may wear shutter glasses which
block out alternate views, leaving visible the view appropriate to
each eye. This creates an enlarged view of the work site, appearing
to the visual system of the user in substantially the same location
and orientation as they appear to the user's motor cortex, via the
neuromotor system of the user's arm and hands. Motions of an
operator's head may make small differences in the visually apparent
placement in 3-dimensional space of the objects in the work site,
natural or inserted, but may still permit effective coordination of
manual control of tools and instruments with what is apparent to
the visual system.
BRIEF DESCRIPTION OF DRAWINGS
[0010] It will be convenient to further describe the present
invention with respect to the accompanying drawings that illustrate
possible arrangements of the invention. Other arrangements of the
invention are possible, and consequently the particularity of the
accompanying drawings is not to be understood as superseding the
generality of the preceding description of the invention.
DESCRIPTION OF PREFERRED EMBODIMENT
[0011] FIG. 1 shows a display panel 100 over a work site 110, using
a rigid support 160. The display panel 100 may be an LCD panel, and
in this case is support by a flexible arm 160. The arm is
sufficiently strong to support the panel 100, and also flexible
enough to be moved so as to position the panel for the convenience
of the operator.
[0012] In this case, The work site 110 involves micro0-surgery,
whereby an artery 140 is undergoing re-attachment. The artery 140
and scalpel 150 are illustrative of the items present on a work
site 110, appearing enlarged as arteries 141 and scalpel tip 151 on
the display panel 100, by an enlargement factor adjustable from 2
to a number on the order of 20. (Human fine motor control limits
the useful degree of enlargement. For most practitioners,
magnification above 15-fold may display tremor.) The height of the
panel 100 above the work site 110 may be adjustable or fixed in a
particular implementation, but may vary between 2 cm for high
enlargement to 20 cm for smaller magnification factors. Particular
implementations may vary the panel size for applications that use
particular ranges of magnification, but our initial preferred
embodiment uses a panel approximately 15 cm.times.15 cm square at a
height of approximately 8 cm above the work site.
[0013] In the preferred embodiment shown in FIG. 1, the
stereoscopic effect is achieved by alternating left and right views
with synchronized shutter glasses 130, worn by the operator/surgeon
to control which eye sees which view, but a glasses-free
`autostereo` solution may also be used if it has sufficient
resolution and supports stereo over a wide enough range of head
motion for user comfort. Similarly, if real-time holographic
cameras and enlarged holographic views become practical, they may
be used within the ambit of the present invention.
[0014] The views shown on the display 100 are taken by one or two
cameras 120. An exemplary two-camera configuration is shown in FIG.
2. A left-eye camera 220 and a right-eye camera 221 lie under the
display panel 200, pointing at mirrors 250. Light from the work
site 210 travels, via the mirrors 250, to the cameras 220 and 221
which create images to be shown on the display panel 200. The
dashed lines 260 show paths along which light is reflected to
cameras 220 and 221. The placement of the mirrors creates the
disparity of angle between the two views, geometrically analogous
to the difference between the viewing directions between two
unassisted human eyes, which creates the stereoptic perception of
depth when the views are shown on the display 200 and channeled to
the appropriate eyes by the shutter glasses 130, or other stereo
display mechanism used in the chosen embodiment. Typically an angle
of approximately 6.degree. between the lines 260, matching the
angular disparity of views of an object a half-meter distant from
eyes at a representative 6 cm separation in the human face, will
give a satisfactory experience of depth perception to the user,
with an apparent visual position for the center of the work site
210 that is substantially in agreement with the position at which
the user experiences it via the hands. (There cannot be exact
agreement in position between the physical objects and their stereo
images, matching them point by point, since the images are arranged
to be larger.) Optionally, the positions and angles of the mirrors
250 may be made user-adjustable, to customize the viewing
experience to the preferred head distance and to human variation in
eye separation.
[0015] The apparent depth may also be modified in software, by
moving the left and right images in opposite sideways senses across
the display, in ways familiar to those skilled in the art.
[0016] In our initial preferred embodiment the mirrors 250 are
planar, serving only the function of redirecting the view of
cameras 220 and 221, but optionally they may be curved,
contributing to the focusing geometry by which the images are
created. In the case where the mirrors 250 are planar, the images
collected by the cameras, as shown in FIG. 3, are precisely those
that would be collected by cameras in the locations 370 and 371,
unobstructed by the mirrors 350 or the display panel 300. The
cameras 320 and 321 thus correspond to `virtual cameras` in these
positions 370 and 371. It should be noted that if the real cameras
are horizontal (for most efficient use of space) and the light
paths 260 correspond to the central point of each image (centering
the same work site point in the image), the virtual cameras are not
parallel in their viewing axes. Such parallelism is required for
optimal stereoscopic viewing: the eyeballs rotate to non-parallel
angles in viewing a stereoscopic display, but unless each eye has a
separate screen which moves to remain at right angles to the eye's
central line of sight, this does not mean that the camera angles
should rotate. Where there is a display screen shared between the
left and right views, each camera axis should be at right angles to
that screen. This is possible for the virtual cameras 470 and 471,
by the use of 45.degree. mirror angles 450 as shown in FIG. 4. (The
light paths 460 to the center of the work site 410 are not in this
case central to the field of view of the cameras 420 and 421.)
Alternatively, a projective transformation in software can adjust
the camera views to those that would be acquired by parallel-axis
cameras. In every embodiment that uses mirrors, software must
transform the camera image to compensate for the optical view
reversal by the mirror, so that the direction of any motion in the
displayed image agrees substantially with the direction of the
real-world motion to which it corresponds.
[0017] As an alternative to the two-camera system shown in FIGS. 2,
3 and 4, a single camera may be used. FIG. 5 shows a configuration
to be placed under the display panel 100, with a horizontal camera
500 directed at a mirror directly over the work site 550. The
mirror has two planar sections 510 and 511, which are close to the
45.degree. angle previously shown but angled slightly inward. The
exact choice of mirror angle depends on the desired placement of
the virtual cameras 520 and 521, which `see` along the virtual rays
561 what the real camera sees along the reflected real rays 560:
the angle may be found by computations (involving also the chosen
distances from camera 500 to the mirror sections 510 and 511, and
the distance above the work site to be imaged by both views) which
are straightforward to one skilled in the art. This configuration
necessarily requires a software adjustment to create the images
that would be acquired by parallel cameras, as well as software
reversal of the mirror reversal.
It is not essential that the display panel 100 be horizontal. While
this is most convenient where (as in certain surgical procedures) a
single stereoscopic view is to be shared between two collaborating
users, a more comfortable view may in some circumstances be
obtained by tilting the display panel toward the viewer, who can
then look orthogonally at it from a less forward posture. The
precise tilt appropriate is a choice that depends on ergonomic
factors such as preferred posture for sustained micro-dexterous
work, and requires careful study for each application. It will in
some cases be preferable to make the angle adjustable, so that
users can adapt it to their own comfort and convenience according
to habit and body type.
[0018] Specifically for collaboration and training, an alternative
model would include two display panels tilted like the sides of a
roof and meeting along the ridge line, each showing the same
stereoscopic view from the same camera or pair of cameras. This is
preferred to giving each panel a distinct stereoscopic view, both
for reasons of economy and to ensure that each user sees exactly
the same view (though slightly differently distorted by individual
departures from the reference eye positions for which the system is
optimized). Such an identity of stereoscopic view minimizes the
risk of miscommunication between surgeons, or between instructor
and trainee, as to what a particular utterance refers to: for
example, if a scalpel tip is used as a pointer, the same physical
point will appear exactly behind it in both views.
[0019] As described so far, the views correspond to magnified
versions of those seen by a pair of eyes set in a face looking
vertically downward. As FIG. 1 illustrates, however, the surgeon's
eyes are typically not directly above the surgical site. Exactly
vertical views would thus create a certain degree of mismatch: for
example, an object moved vertically in the physical scene would
seem in the virtual scene to rise straight toward the viewer, in
disagreement with the user's sense of direction in the control of
his or her hands. For a single viewer this may be corrected by
simply tilting the display screen, replacing the vertical direction
by the straight line from the work site to bridge of the surgeon's
eyes.
[0020] Alternatively, however, we may slightly tilt the mirror pair
510 and 511 (or similarly for a two-camera configuration). If we do
this so that their common line is closer than 45o to the vertical,
the plane of the virtual cameras 620 and 621 and the work site 650
tilts through twice this change, and we produce a view appropriate
for a surgeon facing the apparatus from the side opposite to the
camera. Tilting in the opposite direction gives a view appropriate
to a surgeon on the side near the camera. (There is an also a
necessary adjustment in the height of the image shown on the
display 100, allowing for the foreshortening in the surgeon's view
of the said display, easily accomplished by software manipulation
of the image.)
[0021] Many microsurgical procedures involve two cooperating
surgeons, on opposite sides of the surgical site. The tilt just
described cannot be adjusted correctly for both of them in the same
image. The most economical solution is to accept the vertical view
as a compromise that they both see, as in the two-user surgical
microscope currently in use for this purpose. Alternatively, they
can each have their own tilted display 701 and 702 for viewing the
work site 710 in their natural directions, or they can share a
single display 801 in which four images alternate: the views for
Surgeon 1 left eye, Surgeon 2 left eye, Surgeon 1 right eye and
Surgeon 2 right eye, showing appropriate views of the work site
810, acquired by one, two or four cameras according to the mirror
arrangement used: appropriate mirror configurations generalizing
those in Drawings 2 to 6 will be evident to one skilled in the art.
This requires a display that is capable of cycling between four
images in the 1/60 sec. that is a comfortable standard for moving
video, and shutter glasses synchronised with it that can prevent
each eye from seeing the three images that are not intended for it,
out of each sequence of four. The required frame rate of 240 per
second seems unlikely for LCD technology at this time, but is well
within the potential capability of OLED displays.
* * * * *