U.S. patent application number 15/647956 was filed with the patent office on 2017-10-26 for method of determining the 3d positions of points on the surface of an object.
The applicant listed for this patent is VISION RT LIMITED. Invention is credited to Gideon HALE, Peter HANSON.
Application Number | 20170310953 15/647956 |
Document ID | / |
Family ID | 46261888 |
Filed Date | 2017-10-26 |
United States Patent
Application |
20170310953 |
Kind Code |
A1 |
HANSON; Peter ; et
al. |
October 26, 2017 |
METHOD OF DETERMINING THE 3D POSITIONS OF POINTS ON THE SURFACE OF
AN OBJECT
Abstract
A method of determining the 3D positions of points on the
surface of an object, the method comprising obtaining an images of
an object from a first view point and from a second view point,
fixed relative to the viewpoint of the first view point wherein the
images comprise a matrix of pixels, each of the pixels having an
associated pixel value, processing the images by determining for
pixels in a generated image weighted averages of the pixel values
of a pixel and adjacent pixels, identifying corresponding portions
of an object from a first view point and from a second view point
on the basis of the weighted average values of pixels, and
determining 3D positions of points corresponding to the
corresponding portions of images.
Inventors: |
HANSON; Peter; (London,
GB) ; HALE; Gideon; (London, GB) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
VISION RT LIMITED |
London |
|
GB |
|
|
Family ID: |
46261888 |
Appl. No.: |
15/647956 |
Filed: |
July 12, 2017 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
14386708 |
Sep 19, 2014 |
9736465 |
|
|
PCT/GB2013/051047 |
Apr 25, 2013 |
|
|
|
15647956 |
|
|
|
|
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
H04N 13/239 20180501;
H04N 13/246 20180501; H04N 5/2252 20130101; H04N 13/296 20180501;
A61N 5/1049 20130101; A61N 2005/1059 20130101 |
International
Class: |
H04N 13/02 20060101
H04N013/02; A61N 5/10 20060101 A61N005/10; H04N 5/225 20060101
H04N005/225; H04N 13/02 20060101 H04N013/02; H04N 13/02 20060101
H04N013/02 |
Foreign Application Data
Date |
Code |
Application Number |
Apr 26, 2012 |
GB |
1207299.7 |
Claims
1. A method of determining the 3D positions of points on the
surface of an object, the method comprising: obtaining an image of
an object from a first view point and from a second view point,
fixed relative to the viewpoint of the first view point wherein the
images comprise a matrix of pixels, each of the pixels having an
associated pixel value; processing the images by determining for
pixels in a generated image weighted averages of the pixel values
of a pixel and adjacent pixels; identifying corresponding portions
of an object from a first view point and from a second view point
on the basis of the weighted average values of pixels; and
determining 3D positions of points corresponding to the
corresponding portions of images.
2. The method of claim 1, wherein processing the images by
determining for pixels in a generated image weighted averages of
the pixel values of a pixel and adjacent pixels comprises
processing image data by applying an approximation of a Gaussian
blur to blocks of adjacent pixels.
3. The method of claim 2, wherein the weighted average comprises
scaling a pixel value by a quarter, pixel values for immediately
adjacent pixels by and eight and pixel values for diagonally
adjacent pixels by a sixteenth.
4. The method of claim 1, further comprising scaling pixel values
below a first threshold so as to increase the differences in pixel
values corresponding to different light levels in darker areas of
obtained images.
5. The method of claim 4, further comprising scaling pixel values
above a second threshold so as to reduce the differences in pixel
values corresponding to different light levels in lighter areas of
an obtained image.
6. The method of claim 5, further comprising mapping pixel values
between the first and second threshold by mapping the pixel values
to values between the scaled values corresponding to the first and
second thresholds.
7. The method of claim 1, wherein the pixel values comprise n bit
data and the pixel values are truncated by eliminating the least
significant bits prior to being utilised to identify corresponding
portions of an object from a first view point and from a second
view point.
8. The method of claim 1, wherein the pixel values comprise n bit
data and the pixel values are reduced the bit length prior to being
utilised to identify corresponding portions of an object from a
first view point and from a second view point on the basis of a
rounding of the more significant bits of the pixel values.
9. The method of claim 1, wherein identifying corresponding
portions of an object from a first view point and from a second
view point on the basis of the weighted average values of pixels
comprises identifying matching portions of images the size of said
portions being greater than the areas of image utilised to
determine weighted averages of averages of the pixel values of a
pixel and adjacent pixels.
10. A computer readable medium storing computer interpretable
instructions which when interpreted by a programmable computer
cause the computer to perform a method in accordance with claim 1.
Description
CROSS REFERENCE TO RELATED APPLICATION
[0001] This application is a divisional application of U.S.
application Ser. No. 14/386,708, filed Apr. 25, 2013, which is a
National Phase filing under 35 C.F.R. .sctn.371 of and claims
priority to PCT Application No.: PCT/GB2013/051047, filed on Apr.
25, 2013, and claims priority to British Application No.:
1207299.7, filed on Apr. 26, 2012, the contents of each of which
are hereby incorporated in their entireties by reference.
FIELD OF THE INVENTION
[0002] The present application concerns a method of determining the
3D positions of points on the surface of an object. More
specifically the present application concerns a method of
determining the 3D positions of points on the surface of an object
for monitoring the positioning and movement of patients. The
invention is particularly suitable for use with radio therapy
devices and computed tomography (CT) scanners and the like where
detection of patient movement or irregular breathing is important
for successful treatment.
BACKGROUND TO THE INVENTION
[0003] Radiotherapy consists of projecting onto a predetermined
region of a patient's body, a radiation beam so as to destroy or
eliminate tumours existing therein. Such treatment is usually
carried out periodically and repeatedly. At each medical
intervention, the radiation source must be positioned with respect
to the patient in order to irradiate the selected region with the
highest possible accuracy to avoid radiating adjacent tissue on
which radiation beams would be harmful.
[0004] When a tumour is located in the thoracic or abdominal chest
cavities, the position of a tumour can vary considerably (e.g. by
as much as a few centimetres) throughout the breathing cycle. In
order to obtain reliable CT scanning data it is therefore important
to obtain data consistently at the same point within a breathing
cycle.
[0005] Certain modern CT scanners are able to process CT scanning
data acquired at different times within a breathing cycle to
generate a representation of the 3D movement of internal organs
during breathing. Such "4D" representations of organ movement are
invaluable in determining a course of treatment for irradiating a
cancer. Again in order for quality planning data to be generated,
the timing within the breathing cycle when individual CT scans are
acquired must be known so that the 4D representation accurately
represents the movement of the internal organs.
[0006] When applying radiation to a patient, the gating of
treatment apparatus should be matched with the breathing cycle so
that radiation is focused on the location of a tumour and
collateral damage to other tissues is minimised. If movement of a
patient is detected the treatment should be halted to avoid
irradiating areas of a patient other than a tumour location. Also
if irregular breathing such as coughing is detected, treatment
should be halted as such irregular breathing may cause a tumour to
change location relative to the focus of a radiation beam.
[0007] For this reason a number of monitoring systems for
monitoring a patient's movement and breathing during radiotherapy
have therefore been proposed. These include Vision RT's patient
monitoring system which has previously been described in U.S. Pat.
No. 7,348,974, U.S. Pat. No. 7,889,906 and US2009-018711 all of
which are hereby incorporated by reference.
[0008] In the systems described in Vision RT's patent applications,
stereoscopic images of a patient are obtained and processed to
generate data identifying 3D positions of a large number of points
corresponding to points on the surface of an imaged patient. Such
surface data can be compared with data generated on a previous
occasion and used to position a patient in a consistent manner or
provide a warning when a patient moves out of position.
[0009] When a patient is monitored, relatively small motions on the
surface of a patient can indicate much larger motions of a tumour
within a patient's thoracic or abdominal chest cavity. Improvements
in the accuracy and consistency with which models of the surface of
a patient can be generated are therefore desirable.
SUMMARY OF THE INVENTION
[0010] In accordance with one aspect of the present invention there
is provided a method of determining the 3D positions of points on
the surface of an object, the method comprising obtaining an images
of an object from a first view point and from a second view point,
fixed relative to the viewpoint of the first view point wherein the
images comprise a matrix of pixels, each of the pixels having an
associated pixel value, processing the images by determining for
pixels in a generated image weighted averages of the pixel values
of a pixel and adjacent pixels, identifying corresponding portions
of an object from a first view point and from a second view point
on the basis of the weighted average values of pixels, and
determining 3D positions of points corresponding to the
corresponding portions of images.
[0011] Preferably, the processor may be operable to processes image
data received from the image detectors by determining a weighted
average of pixel values for a pixel and adjacent pixels. More
specifically the processor is operable to process image data by
applying an approximation of a Gaussian blur to blocks of adjacent
pixels. By processing image data in such a way random thermal
errors in image data may be substantially eliminated from the image
data as the action of blurring image data causes the random errors
to cancel out thereby facilitating matching between images obtained
by the two image detectors.
[0012] Preferably, the Gaussian blur applied to image data may
comprise determining a weighted average by scaling a pixel value
for a central pixel by a quarter, pixel values for immediately
adjacent pixels by an eighth and pixel values for diagonally
adjacent pixels by a sixteenth. In such an embodiment all the
scaling of pixel values could be determined by performing logical
shifts rather than by performing division and hence could be
performed rapidly using limited processing resources.
[0013] Preferably, the processor may be arranged to process image
data received from the image detectors by scaling pixel values
below a first threshold so as to increase the differences in pixel
values corresponding to different light levels in darker areas of
an image imaged by the image detectors. The processor may also
scale pixel values above a second threshold so as to reduce the
differences in pixel values corresponding to different light levels
in lighter areas of an image imaged by the image detectors and map
pixel values between the first and second threshold by mapping the
pixel values to values between the scaled values corresponding to
the first and second thresholds.
[0014] When obtaining image data of an object illuminated by a
speckle pattern projected by a speckle projector, light areas
generally correspond to areas well lit by the projector. However,
darker areas may correspond to either darker areas in the projected
pattern or areas which are substantially in shade. By scaling the
image data in the way described the contrast in darker areas in an
image is increased which facilitates the identification of
corresponding portions of images obtained by the image
detectors.
[0015] In some embodiments, the scaling of image values in the
manner described above can be achieved by doubling pixel values for
values below a first threshold; adding a constant value to pixel
values between a first and a second threshold and halving and
adding a second constant value to pixel values above the second
threshold. Such an embodiment would have the advantage that all the
processing could be achieved through a combination of binary shifts
and binary adding operations and hence could be rapidly using
limited processing resources.
[0016] In some embodiments, the image detectors may be arranged to
generate n bit data and the processor may be arranged to reduce the
bit length of the data prior to transmission from the camera
system. Thus for example the image detectors may generate 10 bit
data and 8 bit data may be output by the camera system or 12 bit
data may be generated and 10 bit data output thereby reducing the
time required to transmit image data to a computer where
corresponding portions of images obtained by the image detectors
can be identified so as to determine the 3D positions of surfaces
corresponding to such points.
[0017] In some embodiments, a reduction in data for transmission
may be achieved by truncating the n bit data by not transmitting
some of the least significant bits. Preferably however, rather than
truncating the data, a rounding operation is performed as this
improves the extent to which different patterns in image data can
be identified and helps with matching corresponding points in pairs
of captured images.
[0018] The stereoscopic camera system may be included as part of a
patient monitoring system. In such a system a 3D position
determination module may be provided which is operable to process
image data output by the stereoscopic camera system to identify
corresponding portions of images captured by different image
detectors from different viewpoints and determine the 3D positions
of points corresponding to the corresponding portions of
images.
[0019] A patient monitoring system may additionally comprise a
model generation module operable to generate a 3D computer wire
mesh model of a surface on the basis of the 3D positions of points
determined by the 3D determination module; and a matching module
operable to compare a generated model with a stored model
surface.
BRIEF DESCRIPTION OF THE DRAWINGS
[0020] An embodiment of the present invention will now be described
with reference to the accompanying drawings in which:
[0021] FIG. 1 is a schematic diagram of a patient monitor
incorporating a stereoscopic camera system in accordance with the
present invention;
[0022] FIG. 2 is a schematic perspective view of the exterior of
the camera system of FIG. 1;
[0023] FIG. 3 is a schematic block diagram of the interior of the
stereoscopic camera system of FIG. 1;
[0024] FIG. 4 is a flow diagram of the processing undertaken by the
main processor of the stereoscopic camera system of FIG. 1;
[0025] FIG. 5 is an illustration of an averaging mask for
processing an image to remove the effects of thermal noise;
[0026] FIG. 6 is a schematic block diagram of a filter for
implementing the averaging mask of FIG. 5; and
[0027] FIG. 7 is a graph illustrating relationships between scaled
input and output values increasing contrast in under illuminated
areas.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0028] FIG. 1 is a schematic diagram of a patient monitor
incorporating a stereoscopic camera system 10 in accordance with an
embodiment of the present invention. In this embodiment, the
stereoscopic camera system 10 is connected by wiring 12 to a
computer 14. The computer 14 is also connected to treatment
apparatus 16 such as a linear accelerator for applying radiotherapy
or an x-ray simulator for planning radiotherapy. A mechanical couch
18 is provided as part of the treatment apparatus upon which a
patient 20 lies during treatment. The treatment apparatus 16 and
the mechanical couch 18 are arranged such that under the control of
the computer 14 the relative positions of the mechanical couch 18
and the treatment apparatus 16 may be varied, laterally,
vertically, longitudinally and rotationally.
[0029] In use, the stereoscopic camera system 10 obtains video
images of a patient 20 lying on the mechanical couch 18. These
video images are passed via the wiring 12 to the computer 14. The
computer 14 then processes the images of the patient 20 to generate
a model of the surface of the patient 20. This model is compared
with a model of the patient 20 generated during earlier treatment
sessions. When positioning a patient 20 the difference between a
current model surface and a target model surface obtained from an
earlier session is identified and the positioning instructions
necessary to align the surfaces determined and sent to the
mechanical couch 18. Subsequently during treatment any deviation
from an initial set up can be identified and if the deviation is
greater than a threshold, the computer 14 sends instructions to the
treatment apparatus 16 to cause treatment to be halted until a
patient 20 can be repositioned.
[0030] In order for the computer 14 to process images received from
the stereoscopic camera system 10, the computer 14 is configured by
software either provided on a disk 22 or by receiving an electrical
signal 24 via a communications network into a number of functional
modules 26-34. It will be appreciated that the functional modules
26-34 illustrated in FIG. 1 are purely notional in order to assist
with the understanding of the working of the claimed invention and
may not in certain embodiments directly correspond with blocks of
code in the source code for the software. In other embodiments the
functions performed by the illustrated functional modules 26-34 may
be divided between different modules or may be performed by the
re-use of the same modules for different functions.
[0031] In this embodiment, the functional modules 26-34 comprise: a
3D position determination module 26 for processing images received
from the stereoscopic cameras 10, a model generation module 28 for
processing data generated by the 3D position determination module
26 and converting the data into a 3D wire mesh model of an imaged
computer surface; a generated model store 30 for storing a 3D wire
mesh model of an imaged surface; a target model store 32 for
storing a previously generated 3D wire mesh model; and a matching
module 34 for determining rotations and translations required to
match a generated model with a target model.
[0032] In use, as images are obtained by the stereoscopic camera
system 10, these images are processed by the 3D position
determination module 26 to identify 3D positions of corresponding
points in pairs of images. This is achieved by the 3D position
determination module 26 identifying of corresponding points in
pairs of images obtained by the stereoscopic camera system 10 and
then determining 3D positions for those points based on the
relative positions of corresponding points in obtained pairs of
images and stored data identifying the relative positions of
cameras obtaining the images.
[0033] Typically the identification of corresponding points is
based on analysis of image patches of around 16.times.16 pixels. In
order to assist with identifying and matching corresponding patches
as will be described the stereoscopic camera system 10 is arranged
to project a random or quasi random speckle pattern onto the
patient 20 being imaged so that different portions of the surface
of the patient 20 can be more easily distinguished. The size of the
speckle pattern is selected so that different patterns will be
apparent in different image patches.
[0034] The position data generated by the 3D position determination
module 26 is then passed to the model generation module 28 which
processes the position data to generate a 3D wire mesh model of the
surface of a patient 20 imaged by the stereoscopic cameras 10. In
this embodiment the 3D model comprises a triangulated wire mesh
model where the vertices of the model correspond to the 3D
positions determined by the 3D position determination module 26.
When such a model has been determined it is stored in the generated
model store 30.
[0035] When a wire mesh model of the surface of a patient 20 has
been stored, the matching module 34 is then invoked to determine a
matching translation and rotation between the generated model based
on the current images being obtained by the stereoscopic cameras 10
and a previously generated model surface of the patient stored in
the target model store 32.
[0036] The determined translation and rotation can then be sent as
instructions to the mechanical couch 18 to cause the couch to
position the patient 20 in the same position relative to the
treatment apparatus 16 as they were when they were previously
treated. Subsequently, the stereoscopic cameras 10 can continue to
monitor the patient 20 and any variation in position can be
identified by generating further model surfaces and comparing those
generated surfaces with the target model stored in the target model
store 32. If it is determined that a patient has moved out of
position, the treatment apparatus 16 can be halted and the patient
20 repositioned, thereby avoiding irradiating the wrong parts of
the patient 20.
[0037] As irradiation of adjacent tissues can be harmful, it is
very important that image data is captured in a manner which
maximises the consistency of image data obtained at different
times. The applicants have proposed a number of developments which
increase data consistency and hence improve patient monitoring.
[0038] FIG. 2 is a schematic perspective view of the exterior of
the camera system 10 and FIG. 3 is a schematic block diagram of the
interior of the stereoscopic camera system 10 in accordance with an
embodiment of the present invention.
[0039] In this embodiment the camera system 10 comprises a housing
40 which is connected to a bracket 42 via a hinge 44. The bracket
42 enables the camera system 10 to be attached in a fixed location
to the ceiling of a treatment room whilst the hinge 44 permits the
orientation of the camera system 10 to be orientated relative to
the bracket 42 so that the camera system 10 is arranged to view a
patient 20 on a mechanical couch 18.
[0040] A pair of lenses 46 are mounted at either end of the front
surface 48 of the housing 40. These lenses 46 are positioned in
front of image detectors 50 contained within the housing 40. In
this embodiment the image detectors 50 comprise CMOS active pixel
sensors. In other embodiments charge coupled devices could be used.
The image detectors 50 are arranged behind the lenses 46 so as to
capture images of a patient 20 via the lenses 46.
[0041] A speckle projector 52 is provided in the middle of the
front surface 48 of the housing 40 between the two lenses 46. The
speckle projector 52 includes a light source 54 which in this
embodiment comprises a 12 W red LED light. In other embodiments
other light sources such as a halogen lamp could be used.
[0042] The speckle projector 52 is arranged to illuminate a patient
20 with a non repeating speckled pattern of infrared light so that
when images of a patient 20 are captured by the two image detectors
corresponding portions of captured images can be distinguished. To
that end light from the light source 54 is directed via a film 56
with a random speckle pattern printed on the film. As a result a
pattern consisting of light and dark areas is projected onto the
surface of a patient 20.
[0043] A series of vents 58 are provided in the side walls 60 of
the housing 40. Further vents (not shown) are provided in the rear
62 of the housing 40. A fan (not shown) connected to a temperature
sensor (also not shown) is contained within the housing 40. The
temperature sensor is arranged to monitor the ambient temperature
within the interior of the housing 40 and if this varies to
activate the fan to draw air in via the vents 58 in the side walls
60 of the housing 40 and expel the air via the vents at the rear 62
of the housing 40. In this way an air flow of air at room
temperature is caused to circulate within the housing 40 and
maintain the interior of the housing 40 at a substantially constant
temperature.
[0044] To minimise temperature variations within the housing 40,
immediately adjacent the image detectors, the vents 58 are provided
in the side walls of the housing 40 slightly removed from the
sensors 50 so that when air is drawn into the interior of the
housing 40, it is not drawn past the sensors 50. Rather the fan and
temperature sensor are utilised to maintain the temperature of the
main body of air within the interior of the housing 40 at a
substantially constant level with the air adjacent the sensors 50
being kept at a constant temperature through passive communication
and convection with this main body of air. By separating the vents
58 from the sensors 50, the activation and deactivation of the fan
does not cause sudden changes in temperature adjacent the sensors
50.
[0045] The applicants have appreciated that sensitivity to
temperature is unexpectedly a significant cause of loss of accuracy
in a stereoscopic camera based patient monitoring system.
[0046] Variations in temperature cause small variations in the
position of the image detectors 50. Due to the sensitivity of the
image detectors such changes do not have to be high in order for
one part of an image corresponding to a pixel to be registered at a
different pixel. Thus for example it has been determined that
movements as small as 2.5 micrometers may cause one part of an
image to be registered in an adjacent pixel. When this occurs the
accuracy with which 3D positions of points imaged by the image
detectors 50 declines.
[0047] To substantially eliminate this source of error, in this
embodiment a heater 64 is provided attached to the rear of the
circuit board 66 on which each of the image detectors 50 is
mounted. Additionally on the circuit board a series of copper
conducting pads 68 are provided surrounding the image detector 50
except on one side enabling wiring 70 to connect the image
detectors 50 to a main processor 72. When the heaters 64 are
arranged to heat to a temperature above ambient room temperature
the effect of the heaters 64 and the conducting pads 68 is to
substantially isolate the image detectors 50 from external
variations in temperature. Thus effectively the image detectors 50
are enclosed in their own constant temperature micro-climates.
[0048] This is particularly the case where as far as possible the
image detectors 50 are removed from other external heat sources. In
the described embodiment this is achieved by for example using a
red LED as a light source 54 for the speckle protector 52 thus
reducing internal heating which occurs when for example an
iridescent light blub is used as less power is required to generate
the required light levels and further as the generated light is
only generated in a relatively narrow wave band it is not necessary
to include a coloured film to remove light at other wave bands
which acts to reflect light and heat back within the body of the
housing 40.
[0049] In addition the arrangement of the vents 58 adjacent the
image detectors 50 and the direction of airflow away from the
detectors 50 and out through the rear 62 of the housing 40 also
adds to the extent to which the image detectors 50 can be
maintained at a constant temperature as a constant stream of air at
ambient temperature passes by the detectors 50 and heating arising
from the workings of the camera system 10 is ejected from the rear
62 of the housing 40. The detectors 50 are thereby largely shielded
from outside heat sources and remain at a constant temperature as
determined by the heaters 64.
[0050] The processing of image data by the image detectors 50 to
the main processor 72 will now be described with reference to the
flow diagram of FIG. 4.
[0051] Image data obtained by an image detector such as a CMOS
active pixel sensor will comprise a matrix of pixel values
indicating the relative brightness or darkness of a portion of an
image. Typically such pixel values will be 10 bit pixel data with
values ranging from 0 to 1023 where 0 indicates a pixel in an image
being completely dark and 1023 indicates a pixel being bright
white.
[0052] When such image data is received by the main processor 72,
as an initial step (s4-1) the processor modifies the image data
associated with alternate columns in the matrix by applying a
correction factor to those lines.
[0053] Where a CMOS active pixel sensor is used as an image
detector, frequently alternate columns in an image will be updated
separately so as to enhance readout performance. Where such data is
captured and updated separately this can lead to artefacts
appearing in an image where the average brightness of alternate
columns differs. Such artefacts can be removed by determining the
average brightness of the sets of columns corresponding to columns
updated independently. Correction factors for causing the average
brightness of each set to be equal can then be determined and the
pixel values for columns in an image updated by adding the
determined correction values to the values output by the image
detector 50.
[0054] Having corrected the image data to account for artefacts
arising from the independent updating of interleaved columns in an
image, the main processor then (s4-2) processes the image data to
remove thermal noise from the image.
[0055] More specifically, errors in image values arise randomly due
to thermal effects, slightly increasing or decreasing the values
obtained from an image detector 50. This can lead to difficulties
in identifying corresponding portions of images captured by the two
image detectors 50.
[0056] To remove such errors, in this embodiment the main processor
72 takes advantage of the fact that such errors arise randomly and
therefore the numbers of errors which result in increases in pixel
values should be equal to the numbers of errors which result in
lower pixel values.
[0057] This is achieved by determining a scaled average of groups
of pixels using a mask which approximates a Gaussian blur. A
suitable mask is illustrated in FIG. 5.
[0058] In the case of FIG. 5 a pixel value for a pixel at position
x,y would be determined by calculating the sum of 1/4 the pixel
value of the pixel at x,y plus 1/8 of the sum of the pixel values
for the pixels at positions (x-1,y), (x+1,y), (x,y-1) and (x,y+1)
plus 1/16 of the sum of the pixel values for the pixels at
positions (x-1, y-1), (x-1,y+1), (x+1, y-1) and (x+1, y+1).
[0059] Where each of the pixels is associated with a random error
E, the effect of the summing the errors across the 9 adjacent
pixels will be to cancel out the error in the original pixel at the
expense of a slight loss in the acuity of the image data. However,
where similar processing is undertaken on image data received from
both image detectors 50, this loss of acuity will be the same
across the images and hence unlike the original random thermal
errors will not affect the ease with which corresponding portions
of images can be identified.
[0060] The small loss in image acuity does not adversely affect the
processing of a matching algorithm where any patch matching
algorithm matches larger groups of pixels than are processed by the
smoothing algorithm. Thus for example in the present embodiment the
smoothing algorithm is arranged to process 3.times.3 blocks of
pixels. Such processing would not adversely affect a patch matching
algorithm which for example undertook matching of 16.times.16 pixel
image patches.
[0061] It will be appreciated that when determining the scaled sum
described above, pixel values for the images with random thermal
errors removed can be determined solely by performing suitable
binary adding and shifting operations and hence an image can be
processed to remove such errors very rapidly. More specifically
division by 4, 8 or 16 can be determined by performing a binary
shift 2, 3, or 4 times. The described scaled averaging can
therefore be performed rapidly using very limited processing
resources and hence would not impose significant delays before the
processed image data can be transmitted to the computer 14 for
processing to match corresponding points in the captured
images.
[0062] A suitable circuit for implementing the averaging filter of
FIG. 5 is shown in FIG. 6. In this example the circuit comprises
two latches 74,76, two line stores 78,80 and a set of four binary
adders 82-88. Latches 74,76 are arranged to store pixel value data
until they are informed that the next pixel in an image is to be
read out. In contrast the pair of line stores 78,80 are arranged to
store data for lines of image data and then pass the data onto the
next latch as well as output individual items of data to the binary
adders 86, 88 one by one. Each of the binary adders 82-88 is
arranged to receive a pair of inputs and output a sum of the inputs
with a shifted result so that the output value is one half of the
sum of the pair of inputs.
[0063] When data is received initially it is read into a first
latch 74. Subsequently when the next item of pixel data is
received, data previously in the first latch 74 released and passed
to the second latch 76 and the new data is written into the first
latch 74. The effect of the first two latches 74,76 is therefore to
provide values corresponding to the pixel values for three adjacent
pixels to the first two binary adders 82,84. When this data is made
available, one of the binary adders 82 sums the first and third
pixel values and output a value corresponding to half that sum to
the second binary adder 84. The second binary adder 84 adds this
value to the value of the second pixel in the set of pixels being
processed and outputs half that sum.
[0064] Thus for example if the pixel values A.sub.1, A.sub.2 and
A.sub.3 where to be received, the output from the first binary
adder 82 would be (A.sub.1+A.sub.3)/2 and the output from the
second binary adder 84 would be (A.sub.1+A.sub.3)/4+A.sub.2/2.
[0065] This value is passed to a first one of the line stores 78
and also to the third binary adder 86. As noted above the two line
stores 78,80 are arranged to store data for a line of image data.
The first line store 78 is arranged to pass data for a line of an
image being processed to the second line store 80 in response to a
signal indicating that the next line of image data is being read
out. Additionally the line stores 78,80 are arranged to output
individual items of data one by one to the third and fourth adders
86,88 each time a new item of data is received into the latch. The
effect of this is that the line stores 78,80 output items of data
and binary adders 86,88 process the data calculated for a line of
three adjacent pixels in a similar manner to the processing of
individual items of pixel data by the first and second latches
74,76 and adders 82,84 except that the data is delayed by one or
two lines.
[0066] Thus in the case of processing a portion of an image with
pixel values A.sub.1, A.sub.2, A.sub.3, in a first line, B.sub.1,
B.sub.2, B.sub.3, in a second line, and C.sub.1, C.sub.2 and
C.sub.3, in the third line, the output from the third 86 binary
adder will be
(A.sub.1+A.sub.3)/8+A.sub.2/4+(C.sub.1+C.sub.3)/8+C.sub.2/4 which
is passed on to the fourth binary adder 88. When the fourth binary
adder receives this data this will then be combined with the result
of processing the second line of the image output by the second
binary adder 84 and stored in the first line store 78 which will be
(B.sub.1+B.sub.3)/4+B.sub.2/2 with the result that the final value
output will be
A.sub.1+A.sub.3)/16+A.sub.2/8+(B.sub.1+B.sub.3)/8+B.sub.2/4+C.sub-
.1+C.sub.3)/16+C.sub.2/8 i.e the required scaled sum of the 3 by 3
block of pixels.
[0067] Having processed the captured image data to reduce the
effects of thermal noise, the main processor then proceeds to
re-scale (s4-3) image data for pixels in the image to increase
contrast in under illuminated areas.
[0068] As noted above, in order so that the 3D position
determination module 26 can identify corresponding points in images
obtained by the two image detectors 50, a speckle projector 52
projects a non-repeating (e.g. random or quasi random) speckle
pattern onto the surface of a patient 20. Where a patient 20 is
directly illuminated by the speckle projector 52, this will cause
the surface of the patient to appear as either bright
areas--corresponding to areas of illumination or dark areas
corresponding to dark areas in the projected speckle pattern and it
is easy to distinguish between the two. However, in parts of the
surface of a patient which are not so well illuminated (e.g. areas
in shadow), the differences in appearance between illuminated areas
and areas corresponding to darker points in the projected speckle
pattern will not be as great.
[0069] Thus in order to increase the extent to which corresponding
points in obtained images can be matched, in this embodiment the
pixel values obtained after removing thermal noise are scaled in
the following way:
[0070] 1) All pixel values less than 128 are doubled by performing
a logical shift left.
[0071] Thus for example binary pixel value 12: 0000001100 would be
processed to become binary pixel value 24: 000011000
[0072] 2) All pixel values between 128 and 768 are increased by
adding 128 to the original pixel value;
[0073] Thus for example binary pixel value 296: 0100101000 would be
processed to become binary pixel value 424: 0110101000
[0074] 3) All pixel values between 768 and 1024 are halved by
performing a logical shift right and 512 is added to the
result.
[0075] Thus for example binary pixel value 998: 1111100110 would be
processed to become binary pixel value 1011: 111111001
[0076] FIG. 7 is a graph illustrating the relationships between
scaled input and output values processed in the manner described
above.
[0077] As will be apparent the effect of the processing is to map
the original input values so that differences between pixels having
values less than 128 are increased and the differences between
pixel having values over 768 are reduced.
[0078] It is desirable that as many points corresponding to points
on the surface of a patient can be identified. The applicants have
appreciated that where a portion of a patient 20 is well
illuminated it is still possible to distinguish between light and
dark areas of projected speckle pattern even when the contrast in
such areas is reduced. This contrasts with less well illuminated
areas where the differences between the light and dark areas of
projected speckle may be small. Thus the above processing has the
effect of increasing the numbers of corresponding points in
obtained pairs of images that the 3D position determination module
26 may match.
[0079] Having scaled the pixel values (s4-3) the main processor 72
then proceeds to reduce the volume of data to be transmitted to the
computer 14 by rounding (s4-4) the pixel values thereby converting
the 10 bit data to 8 bit data for faster data transmission.
[0080] One option to achieve such a reduction in data size would be
to truncate the 10 bit data by ignoring the two least significant
bits. Although this would achieve the desired data reduction, this
would involve abandoning any information contained in the least
significant bits. Thus for this reason rather than performing a
truncation operation, a rounding operation is preferable as this
enables any information retained in the least significant bits to
be utilised and thereby improve the accuracy of the point
matching.
[0081] Having performed the rounding operation, the main processor
72 then (s4-5) proceeds to transmit the processed data to the
computer 14 via the wiring 12 where the 3D position determination
module 26 processes the received image data to identify
corresponding portions of the images captured by the two image
detectors 50 and utilises the correspondences and information about
the relative positioning of the image detectors 50 to determine the
3D position of points on the surface of an imaged patient 20. When
processed image data has been transmitted to the computer 14, the
main processor 72 then proceeds to obtain the next frame of image
data from the two image detectors 50 for processing. Thus in this
way the computer 14 is able to monitor the movement and location of
the patient 20 in real time.
FURTHER EMBODIMENTS AND MODIFICATIONS
[0082] Although in the above described embodiment a camera system
including speckle projector 52 including an LED light source has
been described, it will be appreciated that an alternative light
sources such as a halogen lamp might be used. It will also be
appreciated that although in the above embodiment a system has been
described where a temperature sensor and a fan are included within
the body of the system in some embodiments the sensor and fan could
be omitted.
[0083] It will also be appreciated that as an alternative or in
addition to using the circulation of air to reduce the build up of
heat within the interior of a housing 40, in some embodiments heat
sources such as an LED light source 54 or the processor 72 could be
associated with one or more heat sinks provided on the exterior of
the housing 40 so that the heating within the interior of the
housing 40 caused by the light source 54 or processor 72 is
substantially eliminated.
[0084] Although in the above embodiment image data has been
described as being processed to reduce the effects of thermal
noise, in some embodiments it may be preferable to process only
parts of an image in the manner described. In particular it may be
preferable to process only those parts of an image corresponding to
areas of low illumination. This is because where an image is well
illuminated relative size of the thermal errors will be much less
significant.
[0085] In such an embodiment a modified filter system such as that
illustrated in FIG. 6 could be utilised. More specifically, the
illustrated system could be modified to cause unprocessed data from
the first latch 74 to be stored in a further line store and
released so as to provide an unfiltered version of the image data
which is synchronised with the output of the fourth binary adder
88. The system could then choose between the processed and
unprocessed data depending upon the perceived level of illumination
in an area of an image.
[0086] Although in the above embodiment, image processing has been
described as taking place within the stereoscopic camera system 10
and the computer 14, it will be appreciated that the described
processing could take place entirely in one location or the other
or could be divided between the stereoscopic camera system 10 and
the computer 14.
[0087] It will also be appreciated that although processing within
the stereoscopic camera system has been described as taking place
by means of being processed by a main processor 72, the processing
could be undertaken either by processing undertaken by a
programmable array or alternatively could be undertaken by suitable
software.
[0088] Although the embodiments of the invention described with
reference to the drawings comprise computer apparatus and processes
performed in computer apparatus, the invention also extends to
computer programs, particularly computer programs on or in a
carrier, adapted for putting the invention into practice. The
program may be in the form of source or object code or in any other
form suitable for use in the implementation of the processes
according to the invention. The carrier could be any entity or
device capable of carrying the program.
[0089] For example, the carrier may comprise a storage medium, such
as a ROM, for example a CD ROM or a semiconductor ROM, or a
magnetic recording medium, for example a floppy disc or hard disk.
Further, the carrier may be a transmissible carrier such as an
electrical or optical signal which may be conveyed via electrical
or optical cable or by radio or other means. When a program is
embodied in a signal which may be conveyed directly by a cable or
other device or means, the carrier may be constituted by such cable
or other device or means.
[0090] Alternatively, the carrier may be an integrated circuit in
which the program is embedded, the integrated circuit being adapted
for performing, or for use in the performance of, the relevant
processes.
* * * * *