U.S. patent application number 11/442702 was filed with the patent office on 2007-03-15 for multifocal imaging systems and method.
Invention is credited to Karsten Bahlman, Ki-Hean Kim, Timothy Ragan, Peter T.C. So.
Application Number | 20070057211 11/442702 |
Document ID | / |
Family ID | 37075066 |
Filed Date | 2007-03-15 |
United States Patent
Application |
20070057211 |
Kind Code |
A1 |
Bahlman; Karsten ; et
al. |
March 15, 2007 |
Multifocal imaging systems and method
Abstract
In the systems and methods of the present invention a multifocal
multiphoton imaging system has a signal to noise ratio (SNR) that
is reduced by over an order of magnitude at imaging depth equal to
twice the mean free path scattering length of the specimen. An MMM
system based on an area detector such as a multianode
photomultiplier tube (MAPMT) that is optimized for high-speed
tissue imaging. The specimen is raster-scanned with an array of
excitation light beams. The emission photons from the array of
excitation foci are collected simultaneously by a MAPMT and the
signals from each anode are detected using high sensitivity, low
noise single photon counting circuits. An image is formed by the
temporal encoding of the integrated signal with a raster scanning
pattern. A deconvolution procedure taking account of the spatial
distribution and the raster temporal encoding of collected photons
can be used to improve decay coefficient. We demonstrate
MAPMT-based MMM can provide significantly better contrast than
CCD-based existing systems.
Inventors: |
Bahlman; Karsten;
(Cambridge, MA) ; Kim; Ki-Hean; (Watertown,
MA) ; Ragan; Timothy; (Somerville, MA) ; So;
Peter T.C.; (Wayland, MA) |
Correspondence
Address: |
WEINGARTEN, SCHURGIN, GAGNEBIN & LEBOVICI LLP
TEN POST OFFICE SQUARE
BOSTON
MA
02109
US
|
Family ID: |
37075066 |
Appl. No.: |
11/442702 |
Filed: |
May 25, 2006 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
60684608 |
May 25, 2005 |
|
|
|
Current U.S.
Class: |
250/584 ;
348/79 |
Current CPC
Class: |
G01N 21/6486 20130101;
G02B 21/0032 20130101; G02B 21/002 20130101; G01N 21/6458 20130101;
G01N 21/6452 20130101; G02B 21/16 20130101; G02B 21/0076
20130101 |
Class at
Publication: |
250/584 ;
348/079 |
International
Class: |
G03B 42/08 20060101
G03B042/08 |
Claims
1. A multifocal imaging system comprising: a multifocal optical
device that provides a plurality of optical pathways; a scanner
that provides relative movement between the plurality of optical
pathways and a material to be imaged; an optical system that
couples light from the optical device onto a region of interest of
the material; a detector array that detects light from a plurality
of focal locations in the region of interest to generate image
data, the detector array having a plurality of detector elements
correlated with the focal locations; and an image processor
connected to the detector array.
2. The system of claim 1 wherein the scanner comprises a rotating
mirror or a resonant mirror.
3. The system of claim 1 wherein each detector element has a
collection area corresponding to a scattering distribution for each
of the plurality of focal locations.
4. The system of claim 1 wherein the detector array detects a
fluorescence signal from each focal location.
5. The system of claim 1 wherein the detector array comprises a
multi-anode photomultiplier tube imaging detector having at least
64 detector elements.
6. The system of claim 1 further comprising a focusing lens system
that adjusts a depth of focus within a sample in the range of 0
.mu.m to 2000 .mu.m.
7. The system of claim 1 wherein the detector comprises an array of
photomultiplier elements.
8. The system of claim 1 further comprising a computer program that
forms images including a deconvolution of pixel values with a
scattering correction function.
9. The system of claim 1 further comprising a computer program that
processes the image data.
10. The system of claim 9 wherein the program comprises a linear
deconvolution process.
11. The system of claim 9 wherein the program comprises a
non-linear deconvolution process.
12. The system of claim 9 wherein the program comprises a
deconvolution process including scattering correction function.
13. The system of claim 12 wherein the image data comprises a three
dimensional representation of a scanned region of interest, the
representation having a plurality of pixel values, the scattering
correction function including a plurality of adjacent pixel values
for each pixel value of the representation.
14. The system of claim 13 wherein the plurality of adjacent pixel
values comprises a weighting matrix that corrects for light
scattering from tissue along adjacent optical pathways.
15. The system of claim 13 further comprising a holder for a
material to be scanned.
16. The system of claim 1 wherein each of the plurality of optical
pathways defines a plurality of focal locations in an imaging
plane.
17. The system of claim 9 wherein the program comprises a
normalization process.
18. The system of claim 1 wherein the multifocal optical device
comprises a micro lens array.
19. The system of claim 16 wherein a distance between adjacent
focal locations in the imaging plane is between 40 and 200
microns.
20. The system of claim 18 wherein the micro lens array has at
least 64 lens elements.
21. The system of claim 1 wherein the detector array comprises a
first detector array and a second detector array.
22. The system of claim 1 wherein the optical system comprises a
moveable objective lens.
23. The system of claim 22 wherein the objective lens moves along
an axis through the region of interest.
24. The system of claim 1 wherein the optical system comprises a
first lens and a tube lens.
25. The system of claim 18 further comprising an iris defining an
exit aperture of the micro lens array.
26. The system of claim 1 further comprising a confocal pinhole
array adjacent to the detector array.
27. The system of claim 1 further comprising a bandpass filter
adjacent to the detector array.
28. The system of claim 1 further comprising a dichroic mirror that
reflects light returning from the region of interest onto the
detector array.
29. The system of claim 1 further comprising a first reflector
positioned along an optical path between a light source and a micro
lens array, the first reflector coupling light to a first
detector.
30. The system of claim 1 further comprising a beam expander
positioned between a light source and a micro lens array, a second
reflector and a second detector.
31. The system of claim 21 further comprising a reflector that
separates light returning from the region of interest onto a first
optical path towards the first detector array and onto a second
optical path towards a second detector array.
32. The system of claim 21 wherein the first detector array is a
first photomultiplier array and the second detector is a second
photomultiplier array.
33. The system of claim 15 wherein the holder is moveable in three
orthogonal directions.
34. The system of claim 1 further comprising a light source.
35. The system of claim 1 wherein the light source comprises a
laser.
36. The system of claim 34 further comprising a pulse compressor
optically coupled to the light source.
37. The system of claim 34 further comprising an attenuator that
adjusts light intensity.
38. The system of claim 18 further comprising a moveable micro lens
array holder.
39. The system of claim 38 wherein the moveable micro lens array
holder scans in three orthogonal directions.
40. The system of claim 1 further comprising a detector lens that
focuses light returning from each focal location onto a
corresponding detector element.
41. The system of claim 1 further comprising a controller connected
to the scanner that controls scanning speed and resolution.
42. The system of claim 41 wherein the focal locations are
separated from each other by at least 10 microns.
43. The system of claim 41 wherein the controller receives feedback
control signals from a detector that monitors a light
characteristics.
44. The system of claim 43 wherein the detector detects a reference
beam and generates reference signals.
45. The system of claim 1 further comprising a reflector that
reflects a portion of scanning light and a third detector that
measures the scanning light.
46. The system of claim 1 wherein the detector array comprises a
plurality of detector elements that detect light from focal
locations that are separated from each other by more than 25
microns.
47. The system of claim 1 wherein the optical pathways each have a
focal location within the region of interest, adjacent focal
locations being separated by distance in a range between 0.2 and 20
times a mean free path of light illuminating in a tissue or
material to be imaged.
48. The system of claim 47 wherein the distance between focal
locations is correlated with a material to be scanned.
49. The system of claim 1 wherein the detector array comprises
detector elements positioned at different focal distances to image
at different depths within the region of interest.
50. The system of claim 1 wherein the multifocal optical device
provides a plurality of optical pathways having different focal
depths within the region of interest.
51. The system of claim 34 wherein the light source emits light at
a wavelength such that at least two photons of the light that are
incident at a focal location of a material within the region of
interest are necessary induce a fluorescence emission from the
material.
52. The system of claim 51 wherein the light source emits at a
wavelength such that at least three photons of the light are
incident at a focal location are necessary to induce fluorescence
of the material.
53. The system of claim 1 wherein the multifocal optical device
comprises a diffractive optical element.
54. The system of claim 1 wherein the multifocal optical element
comprises a plurality of optical fibers.
55. The system of claim 1 further comprising a fiber optic device
that couples a light source to the multifocal optical element.
56. The system of claim 55 wherein the fiber optic device comprises
a coherent fiber optic bundle.
57. The system of claim 1 further comprising a fiber optic device
that transmits light along an optical path between the region of
interest and the detector array.
58. The system of claim 57 wherein the fiber optic device comprises
a multichannel plate.
59. The system of claim 1 further comprising a spectral dispersing
element that separates light returning from the region of interest
into a plurality of wavelengths that are detected by the detector
array.
60. The system of claim 59 wherein the spectral dispersing element
comprises a transmission grating.
61. The system of claim 1 wherein the system comprises a light
source connected to a probe with a fiber optic cable.
62. The system of claim 61 wherein the probe comprises a handle and
a distal probe.
63. The system of claim 62 wherein the handle houses the multifocal
optical element and the scanner and the distal probe houses the
optical system.
64. The system of claim 63 wherein the distal probe is rigidly
attached to the handle and further comprises a rigid center
endoscope body.
65. The system of claim 63 wherein the optical system includes a
distal lens.
66. The system of claim 63 wherein the optical system comprises a
first lens, a second lens and a distal objective lens.
67. The system of claim 1 further comprising a second light
source.
68. The system of claim 67 wherein the second light source provides
a stationary light beam that is optically coupled to an output lens
with a reflector.
69. The system of claim 63 wherein the handle further comprises the
detector array.
70. The system of claim 69 wherein the detector array comprises
array of photomultiplier tubes remotely connected to the image
processor.
71. The system of claim 1 wherein the detector comprises a CMOS
imaging device.
72. The system of claim 1 wherein the detector further comprising a
binning charge coupled device (CCD) camera such that each binned
pixel region has a light collection area corresponding to a
scattering distribution from each focal location.
73. The system of claim 1 wherein the detector array comprises a
plurality of avalanche photodiodes.
74. The system of claim 1 further comprising a laser light source
including a picosecond laser or a femtosecond laser.
75. The system of claim 1 wherein the system has a resolution in
the region of interest of about 0.1 microns to about 2.0
microns.
76. The system of claim 1 wherein the system images at least 5
frames per second, each frame having at least 256 by 256
pixels.
77. The system of claim 41 wherein the controller actuates
illumination of different focal regions and controls detector
readout in a time multiplexed process.
78. The system of claim 1 wherein the multifocal optical element is
moveable by the controller.
79. The system of claim 1 further comprising a confocal light
collection system
80. The system of claim 79 further comprising multiphoton light
excitation.
81. A method for multifocal imaging comprising: illuminating a
region of interest with light using a plurality of optical
pathways; providing relative movement between the plurality of
optical pathways and the region of interest; and detecting light
from a plurality of focal locations in the region of interest to
generate image data.
82. The method of claim 81 further comprising providing relative
movement by scanning with a rotating mirror or a resonant
mirror.
83. The method of claim 81 further comprising detecting with a
detector array having a plurality of detector elements, each
detector element having a collection area corresponding to a
scattering distribution for each of a plurality of focal
locations.
84. The method of claim 83 further comprising detecting a
fluorescence signal from each focal location, the detector being
connected to an image processor.
85. The method of claim 81 further comprising detecting with a
multi-anode photomultiplier tube imaging detector having at least
64 detector elements.
86. The method of claim 81 further comprising providing a focusing
lens system that adjusts a depth of focus within a sample in the
range of 0 .mu.m to 2000 .mu.m.
87. The method of claim 81 wherein the detector comprises an array
of photomultiplier elements.
88. The method of claim 81 further comprising forming images by a
deconvoluting pixel values with a scattering correction
function.
89. The method of claim 81 further comprising processing the image
data with a computer program on an image processor.
90. The method of claim 89 further comprising processing with the
program including a linear deconvolution process.
91. The method of claim 89 further comprising processing with the
program including a non-linear deconvolution process.
92. The method of claim 89 further comprising processing with a
deconvolution process including a scattering correction
function.
93. The method of claim 92 further comprising processing image data
including a three dimensional representation of a scanned region of
interest, the representation having a plurality of pixel values,
the scattering correction function including a plurality of
adjacent pixel values for each pixel value of the
representation.
94. The method of claim 93 further comprising using the plurality
of adjacent pixel values as a weighting matrix that corrects for
light scattering from tissue along adjacent optical pathways.
95. The method of claim 81 further comprising providing a holder
for a material to be scanned.
96. The method of claim 81 further comprising using each of the
plurality of optical pathways to illuminate a plurality of focal
locations in an imaging plane.
97. The method of claim 89 further comprising processing the image
data with a normalization process.
98. The method of claim 81 further comprising forming the plurality
of optical pathways with a micro lens array, a diffractive optical
element or a plurality of optical fibers.
99. The method of claim 96 further comprising providing a distance
between adjacent focal locations in the imaging plane between 40
and 200 microns.
100. The system of claim 98 further comprising providing a micro
lens array having at least 64 lens elements.
101. The method of claim 81 further comprising providing a detector
array having a first detector array and a second detector
array.
102. The method of claim 81 further comprising providing an optical
system having a moveable objective lens.
103. The method of claim 102 further comprising moving the
objective lens along an axis through the region of interest.
104. The method of claim 81 further comprising providing an optical
system having a first lens and a tube lens.
105. The method of claim 98 further comprising using an iris
defining an exit aperture of the micro lens array.
106. The method of claim 81 further comprising obtaining a confocal
image of a material with the detector array.
107. The method of claim 81 further comprising providing a bandpass
filter adjacent to the detector array.
108. The method of claim 81 further comprising providing a dichroic
mirror that reflects light returning from the region of interest
onto the detector array.
109. The method of claim 81 further comprising providing a first
reflector positioned along an optical path between a light source
and the multifocal optical device, the first reflector coupling
light to a first detector.
110. The method of claim 81 further comprising a beam expander
positioned between a light source and the multifocal optical
device, a second reflector and a second detector.
111. The method of claim 101 further comprising a reflector that
separates light returning from the region of interest onto a first
optical path towards the first detector array and onto a second
optical path towards a second detector array.
112. The method of claim 101 further comprising providing the first
detector array including a first photomultiplier array and
providing the second detector including a second photomultiplier
array.
113. The system of claim 81 further comprising providing optical
pathways that have a focal location within the region of interest,
adjacent focal locations being separated by distance in a range
between 0.2 and 20 times a mean free path of the illuminating light
of tissue or material being imaged.
114. The method of claim 81 further comprising providing a distance
between adjacent focal locations that is correlated with a mean
free path of light within a material to be scanned.
115. The method of claim 81 further comprising providing a detector
elements positioned at different focal distances to image at
different depths within the region of interest.
116. The method of claim 81 further comprising providing multifocal
optical device having a plurality of optical pathways having
different focal depths within the region of interest.
117. The method of claim 81 further comprising providing a light
source that emits light at a wavelength such that at least two
photons of the light that are incident at a focal location of a
material within the region of interest are necessary induce a
fluorescence emission from the material.
118. The method of claim 117 further comprising illuminating with
light at a wavelength such that at least three photons of the light
are incident at a focal location are necessary to induce
fluorescence of the material.
119. The method of claim 81 further comprising providing a fiber
optic device that couples a light source to the multifocal optical
element.
120. The method of claim 81 further comprising providing a fiber
optic device that transmits light along an optical path between the
region of interest and the detector array.
121. The method of claim 81 further comprising applying a dye to a
material to be imaged.
122. The method of claim 81 further comprising detecting a
fluorescent protein in tissue.
123. The method of claim 81 further comprising detecting a
genetically introduced fluorescent material.
124. The method of claim 81 further comprising detecting
autofluorescence of a material.
125. The met-hod of claim 81 further comprising collecting time
resolved spectroscopic data from the region of interest.
126. The method of claim 125 wherein the step of collecting time
resolved data comprises collecting fluorescence lifetime data.
127. The method of claim 81 further comprising performing harmonic
generation microscopy.
128. The method of claim 81 further comprising detecting Raman
scattered data from each of the focal locations.
129. The method of claim 128 further comprising performing a
coherent anti-Stokes Raman scattering measurement of a
material.
130. The method of claim 81 further comprising collecting a
multiphoton quantum data image from the region of interest.
131. The method of claim 81 further comprising collecting a surface
plasmon image from the region of interest.
132. The method of claim 81 further comprising performing
stimulated emission depletion microscopy of a material.
133. The method of claim 81 further comprising providing a probe
having a handle and a probe element connected to the handle and
illuminating a tissue region of a subject with the probe to collect
data.
134. The method of claim 133 further comprising inserting the probe
element within the body of a mammalian subject to collect image
data of tissue within the subject.
135. The method of claim 133 further comprising inserting the probe
element within a body cavity or lumen of a subject.
136. The method of claim 133 further comprising providing a control
circuit, a detector array, a multifocal optical element and an
optical scanner within the handle.
137. The method of claim 133 further comprising coupling a light
source to the handle with a fiber optic cable.
138. The method of claim 136 further comprising connecting the
control circuit to an external image processor.
139. The method of claim 133 wherein the probe element comprises an
endoscope body.
140. The method of claim 139 wherein the endoscope body has a
length of at least 5 cm.
141. The method of claim 81 further comprising forming a plurality
of beams that simultaneously provide focal locations at a plurality
of depths within a material to be scanning, and scanning the
material at the plurality of depths simultaneously to provide a
three dimensional image data set.
142. The method of claim 81 further comprising performing time
multiplexed illumination of focal locations.
143. The method of claim 142 further comprising using a controller
to actuate a light source to provide the time multiplexed
illumination.
144. The method of claim 142 further comprising selecting a pulse
separation and pulse width parameters.
145. The method of claim 142 further comprising detecting the focal
locations with a single detections channels.
146. The method of claim 81 further comprising forming an image of
a mammalian organ.
147. The method of claim 81 further comprising determining whether
tissue cells are cancerous.
148. The method of claim 81 further comprising forming an image of
vascular tissue.
149. The method of claim 81 further comprising sectioning a portion
of tissue such as brain tissue.
150. The method of claim 81 further comprising measuring a response
to a therapeutic agent in tissue.
151. A multifocal light detecting system comprising: a multifocal
optical device that provides a plurality of light beams; an optical
system that couples light from the optical device onto a region of
interest of the material; a detector device that detects light from
a plurality of focal locations in the region of interest to
generate data; and a processor connected to the detector.
152. The system of claim 151 further comprising a scanner such as a
rotating mirror or a resonant mirror.
153. The system of claim 151 wherein the detector has a collection
area corresponding to a scattering distribution for each of the
plurality of focal locations.
154. The system of claim 151 wherein the detector detects time
resolved data for deconvoultions.
155. The system of claim 151 further comprising a computer program
that process time resolved data in combination with spectroscopic
data to distinguish components of tissue.
156. A method for multifocal light detection comprising:
illuminating a region of interest with light using a plurality of
optical pathways; and detecting light from a plurality of focal
locations in the region of interest to generate data.
157. The method of claim 156 further comprising providing relative
movement between the pathways and a material by scanning with a
rotating mirror or a resonant mirror.
158. The method of claim 156 further comprising detecting with a
detector array having a plurality of detector elements, each
detector element having a collection area corresponding to a
scattering distribution for each of a plurality of focal locations
and collecting time realized data and fluorescence data.
159. The method of claim 156 further comprising a detection array
of photomultiplier elements.
160. The method of claim 156 further comprising providing a light
source that emits light at a wavelength such that at least two
photons of the light that are incident at a focal location of a
material within the region of interest are necessary induce a
fluorescence emission from the material.
Description
CROSS REFERENCE TO RELATED APPLICATIONS
[0001] This application claims the priority of U.S. Provisional
Application No. 60/684,608 filed May 25, 2005 entitled, MULTI FOCAL
MULTIPHOTON IMAGING SYSTEMS AND METHODS, the whole of which is
hereby incorporated by reference herein.
STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT
[0002] N/A
BACKGROUND OF THE INVENTION
[0003] Systems and methods for microscopic analysis of biological
material have been used for characterization and diagnosis in many
applications. Fluorescence microscopy, for example, has been used
for optical analysis including the histological analysis of excised
tissue specimens. Optical coherence tomography has been used for
three dimensional imaging of tissue structures, however, the
limited resolution of existing systems has constrained its use for
definitive pathological analysis. Confocal microscopy has been used
for high resolution imaging and has controllable depth of field but
limited imaging speed.
[0004] Multiphoton microscopy is based on the nonlinear excitation
of fluorophores in which fluorescence generation is localized at
the focus of excitation light. Multiphoton microscopy is used for
deep tissue imaging because of its subcellular three dimensional
(3D) resolution, minimal phototoxicity, and tissue penetration
depth of over a few hundred micrometers. It has become useful in
biomedical studies such as neuronal plasticity, angiogenesis in
solid tumors, transdermal drug delivery, and non-invasive optical
biopsy, for example.
[0005] A practical limitation of multiphoton microscopy is its
imaging speed which typically lies in a range of less than two
frames per second. While this speed is sufficient in many cases,
there remain applications in which can be enhanced by improvements
in imaging speed. There is a continuing need for further
improvements in microscopic analysis of biological materials for
numerous applications.
SUMMARY OF THE INVENTION
[0006] The present invention relates to systems and methods for the
multifocal imaging of biological materials. An optical system is
provided in which a plurality of optical pathways are used in
combination with focusing optics to provide a plurality of focal
locations within a region of interest of a material being optically
measured or imaged. The detector can comprise a plurality of
detector elements which are correlated with the plurality of focal
locations to provide for the efficient collection of light from the
material being imaged. A preferred embodiment of the invention
utilizes a scanning system that provides relative movement between
the material and the focal locations to provide for fast imaging of
the material.
[0007] In a preferred embodiment a light source, such as a laser,
is used with a multifocal optical element to provide an array of
spatially separated optical pathways. The multifocal optical
element can comprise a micro lens array, a diffractive optical
element, or a beam splitter device, for example, such that a
plurality of beams are provided that can be focused onto a
plurality of focal locations within a biological material to be
imaged.
[0008] An important issue in the collection of light from discrete
focal spots or locations within a turbid medium such as tissue is
the cross talk that can occur due to the scattering of light. This
cross talk can substantially limit the usefulness of the images of
the tissue that are produced. By increasing the distance between
adjacent focal spots such cross talk can be reduced or eliminated,
however, this reduces the resolution of the resulting image or
increases the time needed to scan the tissue. Thus it is desirable
to employ focal spacing of at least 10 microns and preferably more
than 25 microns.
[0009] In a preferred embodiment of the invention, high speed
multiphoton microscopy can measure biological systems such as, for
example, kinetic processes in the cytosol of a single cell, for
example, or imaging a volume of tissue. For example, high speed 3D
imaging can map 3D propagation of a calcium wave and the associated
physical contraction wave through a myocyte, or the rolling of
luckocytes within the blood vessel of a solid tumor. High speed 3D
microscopy provides for sampling a statistically significant volume
of biological specimens. Since the field of view of most
microscopes is limited to about 100 microns on a side with an
imaging depth of 100 microns, the measurement volume is limited to
only 1.times.10.sup.-3 mm.sup.3. While this volume is sufficient
for cellular imaging, many tissues have physiologically relevant
structures ranging from the cellular level up to several
millimeters in size. For example, a neuron with its extensive
dendritic tree can span a volume over 1 mm.sup.3 and many dermal
structures such as hair follicles and sabestious glands can not be
seen with images confined to an area of 100-200 micrometers. It is
desirable, for example, to image a hierarchy of cardiac structures
ranging from a single nucleus in a cardiac myocyte, to the
distribution of muscle fibers and blood vessels, to the structure
of chambers and heart valves with perfect registration across five
orders of magnitude by imaging a whole mouse heart. Equally
importantly, traditional 3D microscopes sample only tens to
hundreds of cells and can never achieve comparable statistical
accuracy and precision in many biomedical assays as techniques such
as flow cytometry and image cytometry. High speed imaging can
circumvent this difficulty by improving the number of cells or
tissue volume to be sampled. By performing high speed multiphoton
imaging, better quantitative measurements of transport pathways
across the stratum corneum in transdermal drug delivery
applications can be made, for example.
[0010] Systems and methods have been developed to enhance
multiphoton imaging speed. A first method increases the scanning
speed by using a high-speed scanner such as a polygonal mirror
scanner or a resonant mirror scanner instead of a
galvanometer-driven mirror scanner. This achieves an increase of
scanning speed of more than 10 frames per second in the imaging of
typical tissue specimens. In general, the system can operate at
frequencies in a range of 1 to 500 Hz. This method can be used for
turbid tissue imaging since it is not sensitive to the scattering
of emission photons. A second method increases the imaging speed by
parallelizing the multiphoton imaging process. It scans a sample
with a multiple of excitation foci instead of forming only a single
focus. These foci are raster scanned across the specimen in
parallel where each focus needs to cover a smaller area. The
emission photons from these foci are collected simultaneously with
a spatially resolved detector. One advantage of this method is that
the imaging speed is increased by the number of excitation foci
generated, without increasing the power of excitation light per
each focus. High speed scanning systems needs higher power to
compensate for the signal reduction per pixel due to the decrease
of pixel dwell time. Images can be obtained by selecting the depth
of focus to be positioned in a plane within the tissue or sample at
a depth in a range of 10 microns to 500 microns.
[0011] In another embodiment, fiber optics can be used to couple
the light source to the microlens array or other beam splitting
element. The system can be implemented as a handheld optical probe
for the diagnosis of dermal, cervical or colorectal cancer, for
example.
[0012] The brain is an inherently three dimensional organ composed
of many subregions. Accurate segmentation of brain morphology of
small mammals is currently challenged by the lack of techniques
which can sample the brain at high resolution over a large volume.
The current method of choice, serial section reconstruction, is
laborious, time consuming, and error prone. The device and methods
described herein can quickly image brains or thick tissue sections
of brains in 3D at sufficient resolution and over a large enough
volume to provide 3D images suitable for classification of brain
morphology and biochemical composition. The brain can be further
stained by dyes, such as nuclear dyes DAPI or Hoescht, either
through intravital injection, transgenic expression, or ex vivo
methods, to facilitate classification of regions. Automatic
segmentation routines can also be used to improve the
classification and automate portions of the process.
[0013] Accurate measurement of vasculature is important to
characterize many biomedical for vasculature related diseases. For
instance, proangiogenesis therapies are useful in such areas as
tissue engineering, wound healing, bone fractures and coronary
heart disease. Anti-angiongenesis treatments are important in
processes as cancer, blindness, and rheumatoid arthritis.
Unfortunately traditional histopathological analysis of tissue
sections is wholly inadequate to characterize the vasculature of a
tissue or organ as blood vessels form complex, multiscale 3D
networks, with feature spanning from the submicron to centimeter
scale. The device and methods described in the patent are capable
of acquiring high quality 3D datasets over 3D tissue and organ
samples suitable for characterization of the vasculature of the
tissue. To aid visualization of the vasculature, the tissue can be
stained by contrast agents which bind to the epithelial wall of the
blood vessels, or fill the interior of vessels. Automatic
segmentation routines can also be used to improve the
classification and automate portions of the process.
[0014] A large percentage of deaths are due to metastasis.
Unfortunately, the migration of cancer cells from the primary tumor
to secondary sites is a multi-step process which is not well
understood. Standard histopathological analysis is ill-suited to
study metastasis and suffers from a number of limitations. First,
it is extremely difficult to find rare metastatic cancer within a
3D bulk tissue using traditional 2D histopathology. In many
instances traditional 2D histopathology is unable to find evidence
of the presence of metastatic cancer cells in an organ of animal.
However, it is known that many subjects eventually develop tumors
at a later time. It is clear that traditional histopathology cannot
effectively detect rare cells. Another limitation is that the
present histopathology methods provide limited information about
the 3D spatial arrangement of cancer cells with the 3D vasculature
of the organ. It is known that one of the critical steps in
metastasis is extravasation into the surrounding stroma from the
vasculature so it is essential to be able to visualize this spatial
relationship between cancer cell and the endothelial blood vessel
wall. Preferred embodiments of the present invention are capable of
acquiring high quality 3D datasets over 3D tissue and organ samples
suitable for characterization of the metastases. To aid
visualization of the metastases, the cancer cell can be stained by
dyes or labeled with proteins such as OFP. Automatic segmentation
routines can also be used to improve the classification and
automate the localization of the cancer cells and tumors.
[0015] In order to understand the effects of a drug on an organism,
analysis at the tissue, whole organ, and whole organism level is
vitally important. ADME, efficacy and toxicology effects are known
to have strong spatial variations on the morphological, cellular
and biochemical state of a tissue. Even within a specific tissue
type, the response can be nonuniform due to variations in the
transport and distribution of a drug throughout tissue, epigenetic
expression, and cellular activity. The devices and methods
described herein can be used to provide morphological, biochemical
and spectroscopic information about the state of a tissue across
multiple length scales, from subcellular, whole tissue, whole organ
and even entire organism, in response to the treatment of a
molecular agent. Efficacy, ADME, and toxicology information can be
derived which provides a fuller and more accurate description to
predict the actual effect of drug candidate at the organism
level.
DESCRIPTION OF THE DRAWINGS
[0016] FIG. 1 is a schematic diagram of an imaging system in
accordance with a preferred embodiment of the invention.
[0017] FIGS. 2a-2c are images of human skin acquired with the
present invention including the stratum corneum layer, the stratum
granular and the basal layer, respectively.
[0018] FIG. 3 graphically illustrates the signal decay with
increasing imaging depth of conventional systems and those
incorporating the present invention.
[0019] FIGS. 4a-4d include images before and after deconvolution as
well as graphical illustration of scattering and crosstalk.
[0020] FIGS. 5a-5i are images based on CCD, MAPMT and deconvolution
thereof at the surface and difference depth of brain tissue.
[0021] FIG. 6 illustrates a method and apparatus for multi-focal,
multi-photon microscopy (MMM) according to a preferred embodiment
of the invention, showing parallelized illumination and detection
device with a common focusing device.
[0022] FIG. 7 illustrates a method and apparatus for multi-focal,
multi-photon microscopy (MMM) employing scanning and multi anode
PMT's, according to a preferred embodiment of the invention.
[0023] FIG. 8 illustrates another method and apparatus for
multi-focal, multi-photon microscopy (MMM) employing scanning and
multi anode PMT's, according to a preferred embodiment of the
invention.
[0024] FIG. 9 illustrates generating and detecting a 3D foci
pattern in a focal region.
[0025] FIGS. 10(a)-(i) illustrates close up views of the 3D focal
region generated by the setup in fig. W-10 and views of the 3D
scanning: (a) the focal region; (b) an array of excitation light
beams; (c) x/y view at first depth; (d) x/y view at second depth;
(e) x/z view; (f) multiple rows of excitation foci lie in different
focal planes and are all shown in this xy view; (g) xz view of the
rows shown in (f); (h) x/y view of the x/y scanning configuration
covering the x-y-z image (as in (f) all foci are shown, even though
they lie in different planes and in the lower part of the images
even behind each other; and (i) a view in the yz plane illustrating
scan progression.
[0026] FIG. 11a illustrates a further method and apparatus for a 3D
cytometer, based on multi-focal, multi-photon microscopy (MMM)
employing scanning and multi anode PMT's, according to a preferred
embodiment of the invention.
[0027] FIG. 11d illustrates an image of the array of foci in the
focus of the objective lens; the foci are 45 micrometers apart
resulting in a scanning field of 240 mm when 6.times.6 foci are
utilized.
[0028] FIG. 11(c) illustrates the Z-profile and corresponding fit
function of a 200 nm bead.
[0029] FIGS. 12(a)-(e) illustrates a further method and apparatus
for multi color detection MMM employing scanning and multi anode
PMT's according to a preferred embodiment of the invention: (a) the
setup in the xz-plane; (b) the foci and their scanning in the focal
xy-plane;
[0030] (c) detection path in the yz-plane (d) detection path
projected in the xz-plane; (e) the anodes of the multi anode PMT in
the x/z and x/y plane in conjunction with the detected colors. In
this case only visible light is shown. Any other light spectra can
be separated, though.
[0031] FIG. 13 illustrates a beam splitter configuration used in
some embodiments according to the invention.
[0032] FIGS. 14(a)-(d) illustrate preferred embodiments for
providing illumination beam paths in accordance with the
invention.
[0033] FIGS. 15(a)-(d) illustrated further preferred embodiments
for detecting light from different focal locations in accordance
with preferred embodiments of the invention.
[0034] FIG. 16 illustrates determining the optimal number of foci
at a certain laser power for samples with different damage
thresholds.
[0035] FIGS. 17(a) and (b) illustrate a time multiplexing
method.
[0036] FIGS. 18(a)-(c) illustrate a pixellated detector collection
method.
[0037] FIG. 19 illustrates an endoscope apparatus according to an
embodiment of the invention.
[0038] FIGS. 20(a) and (b) illustrates scattered light detection
with one PMT and one excitation focus according to an embodiment of
the invention.
[0039] FIGS. 21(a) and (b) illustrates scattered light detection
with two PMTs and one excitation foci according to an embodiment of
the invention.
[0040] FIGS. 22(a) and (d) illustrates scattered light detection
with two PMT's and two excitation foci according to an embodiment
of the invention.
[0041] FIGS. 23(a) and (b) illustrates reducing optical cross talk
by increasing the distances between the excitation foci and
distances between the detection elements.
[0042] FIGS. 24(a) and (b) illustrates reducing optical cross talk
according to an embodiment of the invention by increasing the
distance between the excitation foci and increasing the area of the
detection elements.
[0043] FIGS. 25(a)-(e) illustrate in tabular form two alternative
embodiments A and B of the invention in terms of changing optical
setup.
[0044] FIGS. 26(a)-(b) illustrates the different conjugated areas
of detection from each channel of the multi anode PMT in the
conjugated image plane for configuration A and B from FIG.
22(a)-(e).
[0045] FIGS. 26(c)-(d) illustrate an objective lens with large
field of view enables large separation of foci and thus enables low
optical cross reduction
[0046] FIGS. 27(a) and (b) illustrate data post-processing
sequences.
[0047] FIGS. 28(a) and (b) illustrate a normalization method.
[0048] FIGS. 29(a)-(c) illustrate a linear deconvolution
process.
[0049] FIGS. 30(a)-(d) illustrate further details for a linear
deconvolution process I: Signal distribution in multi channel
detector
DETAILED DESCRIPTION OF THE INVENTION
[0050] As the input power of excitation light increases, the signal
is increased proportionally to the square of input power,
S(t).varies.[P(t)].sup.2. However, there is a limitation in input
power level due to finite lifetimes of fluorophores.
[0051] In the multiphoton excitation of fluorophores with a pulsed
laser, the fluorophores, which are excited with the last pulse,
stay in the excitation state for a few nano-seconds (depending on
the fluorophore). Some excited fluorophores may not be excited
again with the next pulse of excitation light (12 ns later in case
of the laser having 80 MHz pulse repetition rate). Therefore,
signal level becomes saturated with a higher input power than the
limited input power level. The limitation on the input power level
is related to the excitation probability of a single fluorophore
with a single pulse, P.sub.pulse. It is formulated in the following
expression with the condition that excitation light is focused with
an objective into a fluorophore of an absorption coefficient
(.delta..sub.a). P pulse = .delta. a .function. [ .lamda. K a
.times. hc ] 2 .times. NA 4 .tau. p .times. f p 2 .function. [ P a
.function. ( t ) ] 2 ( 1 ) ##EQU1## The nominal conditions are that
the excitation light has the wavelength (.lamda.=800 nm), the pulse
width (.tau..sub.p=200 fs), the pulse repetition rate (f.sub.p=80
MHz), and the average power, P.sub.a(t). The numerical aperture
(NA) of the lens objective is 1, (NA=1). The fluorophore has the
absorption coefficient, (.delta..sub.a=10 GM), where 1 GM is
10.sup.-50 cm.sup.4.times.s/photon. In order to avoid the
saturation, P.sub.pulse must be less than 0.1 in general
(P.sub.pulse<0.1). With these conditions, the input power
(P.sub.a(t)), with which P.sub.pulse becomes close to the
saturation limit, is approximately 6 mW,
(P.sub.a.sup.sat(t)=.apprxeq.6 mW) in this example. In case the
concentration of the fluorophores is 10 .mu.M, the number of
emission photons collected per second is approximately
3.times.10.sup.7 photons/s with the assumption that the collection
efficiency of emission photons is approximately 0.01
(.epsilon..sub.col=0.01). Assuming that each pixel needs 300
photons and each image comprises of 256.times.256 pixels, the frame
rate that can be achieved with the input power under the saturation
limit is 1.5 frames/s. Although the higher frame rate is achieved
with specimens of higher fluorophore concentration, it is clear
that there is a limitation in input power level due to fluorophore
saturation.
[0052] MMM increases the frame rate by scanning with multiple
excitation foci. Therefore, MMM can achieve the higher frame rate,
while the input power for each excitation focus is kept below the
saturation limit. For example, the MMM system, which scans with an
8.times.8 array of excitation foci, can achieve the frame rate of
96 frames/s (=1.5 frames/s.times.64 foci). In a preferred
embodiment it is desirable to collect at least 15 frames per second
and preferably 30 frames per second or more. One practical
limitation in MMM is that more input power is required to generate
multiple excitation foci. The power requirement to generate 64 foci
is 384 mW (=64 foci.times.6 mW per each focus). Since available
laser sources can output approximately 2 W of power, enough power
is available for MMM.
[0053] The limit of optical imaging depth in tissues is limited by
photon interaction with tissue constituents. Photon scattering is a
dominant factor in multiphoton microscopy whereas the effect of
photon absorption is relatively negligible. Scattering of
excitation photons reduces the amount of fluorescence generated at
its focus, because less excitation photons reach the focal volume.
The emission photons from the focus are also scattered so that they
may not be collected by the optics in the detection path or
spatially dispersed in the imaging plane where detectors are
positioned. Since the excitation light has a longer wavelength than
the emission light, the excitation light typically experiences less
scattering than emission light. The effect of photon scattering is
expressed by the mean free path length of scattering, l.sup.s which
is the depth constant in exponential decay of unscattered photons,
S(z).varies.exp(-z/l.sup.s).
[0054] Intralipid emulsion can be used as a tissue phantom with
similar optical properties as tissue. The optical properties of 2%
intralipid are mean free path length at excitation wavelength (780
nm) of 167 .mu.m, (l.sub.ex.sup.s.apprxeq.167 .mu.m) and at
emission wavelength (515 nm) of 62.5 .mu.m,
(l.sub.em.sup.s.apprxeq.65 .mu.m). Since it is known that only
ballistic excitation photons contribute multiphoton excitation in
the depth of a few times of scattering length, the amount of
multiphoton excitation decays with the mean free path length of 84
.mu.m (=167 .mu.m/2) with the consideration that two-photon
excitation is a quadratic process. Conventional multiphoton
microscopy is based on the scanning of a single excitation focus
and the signal is collected using a detector without spatial
resolution such as a conventional photomultiplier tube (PMT). The
PMT has a large detection area and can collect most of the signal
generated at the excitation focus including a large fraction of the
scattered photons. Therefore, conventional multiphoton microscopy
is relatively immune to the scattering of emission photons by the
tissue. However, for an MMM system that utilizes a CCD detector to
distinguish the signals originated from each of the foci, the
scattering of emission photons seriously degrades the SNR of the
instrument for deep tissue imaging. The CCD camera has relatively
slow readout speed and typically integrates all the emission
photons during the acquisition of each frame. Because a CCD camera
contains pixels in which each pixel covers a 0.1 .mu.m.sup.2 region
in the specimen plane, scattered emission photons deflected from
their original paths are not collected in the correct pixel but are
distributed broadly across the imaging plane. The distribution of
scattered emission photons is very broad with its FWHM of 40 .mu.m
in the depth of 2.times.l.sub.em.sup.s. These scattered photons
result in a degradation of image SNR by more than one order of
magnitude when imaging depth is over 2.times.l.sub.em.sup.s,
compared with conventional multiphoton microscopy.
[0055] The major limitation of CCD-based MMM system lies in its
small pixel area. For conventional wide field imaging, a large
number of CCD pixels are needed to maintain good resolution while
covering a good size field of view. A 100 .quadrature.m size image
will require about 10.sup.7 pixels to be imaged at full optical
resolution (300 nm). The situation is very different for MMM
imaging. Since a femtosecond light source can only provide at
maximum 2-4 watts of optical power and typically about 50-100 mW
are required at each focus to generate an efficient muliphoton
excitation process for deep tissue imaging. An MMM system can
realistically and effectively scan about 20-40 foci in parallel
with tissue specimens. Since these foci are raster scanned across
the specimen, the image resolution is determined by the excitation
point spread function (PSF) of the light and is not sensitive to
the detector pixelation. In particular, a preferred embodiment uses
an MMM system having photon detectors containing only as many
elements as the number of excitation foci. The need for fewer
elements allows the use of a detector with a significantly larger
pixel area while maintaining a reasonable device size. A
multi-anode PMT (MAPMT) is a preferred detector for this
purpose.
[0056] A preferred embodiment of the present invention uses an
MAPMT instead of the CCD camera for the signal collection from
multiple foci. The MAPMT is similar to conventional PMTs with a
good quantum efficiency (over 20% in the blue/green spectral
range), negligible read noise and minimal dark noise with cooling.
MAPMT has a cathode and dynode chain with a geometry that ensures
that the spatial distribution of photons on the cathode is
reproduced accurately as electrons distribution at the anode. The
anode of the multi-anode PMT is divided rectilinearly into its
elements providing spatial resolution for the simultaneous
collection of signals from multiple locations. In one example, a
MAPMT, which has an array of 8.times.8 pixels (H7546, Hamamatsu,
Bridgewater, N.J.) is used. Note that a flat panel detector having
a pixel area of sufficient size can also be used. For example, a
binnable CMOS or CCD imaging sensor can be operated to read out
binned images at comparable frame rates with an effective pixel
size corresponding to that of a MAPMT.
[0057] A preferred embodiment of the invention uses the imaging
systems as described herein in conjunction with a system for
sectioning a sample such as a tissue sample that is described in
greater detail in U.S. application Ser. No. 10/642,447, by So, et
al. filed Aug. 15, 2003, the entire contents of which is
incorporated herein by reference.
[0058] The schematic of a preferred embodiment of the imaging
system 10 in accordance with the invention is shown in FIG. 1. The
light source 12 used is a Ti-Sapphire (Ti-Sa) laser (Tsunami,
Spectra-Physics, Mountain View, Calif.) pumped by a continuous
wave, diode-pumped, frequency-doubled Nd:YVO.sub.4 laser (Millenia,
Spectra-Physics, Mountain View, Calif.). It generates approximately
2 W at 800 nm wavelength which is sufficient for most MMM
applications. The excitation beam from the laser is optically
coupled using optical fiber 14 or free space lens system to a beam
expander 16 and then illuminates a microlens array 20 (1000-17-S-A,
Adaptive Optics, Cambridge, Mass.) which, in this example, is an
array of 12.times.12 (or 8.times.8) square microlenses that are 1
mm.times.1 mm in size and 17 mm in focal length. The degree of beam
expansion can be selected such that an array of 8.times.8 beam-lets
is produced after the microlens array. The beam-lets are collimated
after lens L1 and reflected onto an x-y scanner mirror 30 (6220,
Cambridge Technology, Cambridge Mass.) which is positioned in the
focal plane of lens L1. In this configuration, the beam-lets
overlap each other on the scanner mirror surface and are reflected
similarly by the rotation of the scanner mirror. After the scanner,
the beam-lets enter a coupling lens system such as a microscope
(BX51, Olympus, Melville, N.Y.) via a modified side port. A
combination of lenses L2 and L3 expands the beam-lets to fill the
back aperture of the objective lens 36 in order to use the full NA
of the objective lens. The scanning mirror is in the telecentric
plane of the back aperture of an objective lens so that the
beamlets are stationary on its back aperture independent of the
motion of the scanner mirror. The objective lens generates the
8.times.8 focus array of excitation light in the sample plane in
the specimen 34. The scanner mirror moves the array of excitation
foci in the sample plane in a raster pattern to cover the whole
sample plane. Alternatively, a digital micromirror (MEMS) device
can be used to control beam scanning in the sample plane. A
beamsplitter can also be used to split an input beam before the
microlens array. Another alternative embodiment employs a
diffractive optical element in conjunction with a beam splitter.
The objective used in this system is a 20.times. water immersion
lens with 0.95 NA (XLUMPLFL20XW, Olympus, Melville, N.Y.). The
excitation foci are separated from each other by 45 .mu.m in this
example so that the scanning area of each focus is 45
.mu.m.times.45 .mu.m. The frame size is 360 .mu.m.times.360 .mu.m
by scanning with the array of 8.times.8 foci. The frame rate to
generate images of 320.times.320 pixels becomes approximately 19
frames per second with the pixel dwell time of 33 .mu.s.
[0059] Emission photons are generated at the array of excitation
foci in the specimen and are collected by the same objective lens
forming an array of emission beam-lets. In case of a CCD-based MMM,
the emission beam-lets are reflected on a long-pass dichroic mirror
38 (650dcxxr, Chroma Technology, Brattleboro, Vt.) and get focused
in optional CCD camera 28 (PentaMax, Princeton Instruments,
Trenton, N.J.) with a lens (L3). The CCD camera integrates emission
photons during the scanning time of each frame to generate images.
In case of a preferred embodiment using an (without the CCD) MAPMT,
the emission beam-lets travel back to the scanner mirror 30
retracing the excitation paths. The emission beam-lets are
reflected by the scanner mirror. The emission beam-lets are
de-scanned and their propagation directions remain stationary
irrespective of the movement of the scanner. The emission beam-lets
are reflected by a long-pass dichroic mirror 32 (650dcxxr, Chroma
Technology, Brattleboro, Vt.) and are focused after lens (L4). A
short-pass filter (E700SP, Chroma Technology, Brattleboro, Vt.)
blocks any strayed excitation light. The focused emission beam-lets
are collected at the center of corresponding channels of a MAPMT 22
(H7546, Hamamatsu, Bridgewater, N.J.). The emission photons coming
from the array of excitation foci are collected by the MAPMT. An
image is formed by the temporal encoding of the integrated signal
with the known raster scanning pattern using image processor or
computer 24 and is electronically stored in memory and/or displayed
using display 26.
[0060] The pair of L2 and L4 lenses magnifies the array of emission
foci so that individual emission beamlets are focused at the center
of corresponding elements of the MAPMT. Further, since the emission
beam-lets are descanned, they remain stationary. Since the emission
beam-lets have to go through more optical elements, loss of
emission photons occurs. The transmission efficiency is
approximately 0.7. The signals from the MAPMT are collected by a
stack of four multi-channel photon counter card (mCPhC) which has
64 channels for simultaneous signal collection. Each mCPhC has 18
channels of photon counter circuits and can be housed 25 with a
digital interface to the computer 24. The mCPhC expandable so that
64 channels are readily implemented by using 4 cards in parallel.
The mCPhC has a 32-bit parallel interface with a computer for
high-speed data transfer. Currently, the speed is limited by the
speed of the computer PCI bus. Transfer rate can be more than one
hundred frames (320.times.320 pixels, 16 bit images) per
second.
[0061] Since the scattered emission photons have the spatial
distribution of 40 .mu.m as its FWHM at the imaging depth of
2.times.l.sub.em.sup.s, the sensitivity of the microscope is partly
determined by the effective detector area, the area in the sample
plane from which a detector collects emission photons. Since
microscopes are telecentric systems, the effective detector area is
linearly related with the detector size in the image plane. With a
magnification, M, and a linear dimension of detector, L.sub.D, the
linear dimension of effective detector area (LE) is
L.sub.E=L.sub.D/M. In general, the larger the effective detector
area, the more effective the detector can collect scattered
emission photons. In the case of using a 20.times. magnification
objective, a 10 mm diameter standard PMT has an effective detector
area of 500 .mu.m diameter that is significantly larger than the
width of the scattered emission photon distribution. Therefore,
standard PMTs have good collection efficiency of scattered emission
photons and allow very effective deep tissue imaging. In the case
of a spatially resolved detector, each pixel can be treated as an
individual detection element. For a CCD camera with 20
.mu.m.times.20 .mu.m pixels, each pixel has an effective detector
area of 1 .mu.m.times.1 .mu.m for 20.times. magnification.
Therefore, the CCD-based MMM system cannot utilize these scattered
emission photons which are distributed uniformly throughout the
image contributing to the background noise. In this example of the
MAPMT-based MMM system, the effective detector area of each channel
is 45 .mu.m.times.45 .mu.m. Therefore, the MAPMT can collect
significantly more scattered emission photons into the correct
channels than the CCD camera, because its effective detector area,
or detector element collection area, is comparable with, or
corresponds to, the width of the scattered photon distribution from
each focal area (45 microns.times.45 microns).
[0062] MAPMT-based MMM system can be easily converted to the
conventional multiphoton microscope which is based on single-focus
scanning and signal collection with PMTs. In the set up of
conventional multiphoton microscope, the excitation beam is not
expanded and goes directly onto the scanner without the combination
of the microlens array and lens L1. The rest of excitation beam
path is the same as MAPMT-based MMM. Specimens are scanned with a
single excitation focus. The emission light collected by the
objective lens is reflected on a dichroic mirror. The reflected
emission beam shrinks with a pair of lenses and is collected by a
detector (PMT). An image is formed by the temporal encoding of the
integrated signal with the known raster scanning pattern.
[0063] CCD-based MMM have limitations for turbid tissue imaging by
measuring the effect of emission photon scattering on PSF
(scattering function). Scattered emission photons form additional
intensity distribution around the PSF constructed with ballistic
unscattered photons. Their intensity distribution is quite wide
with its FWHM of 40 .mu.m at the imaging depth of
2.times.l.sub.em.sup.s. FWHM of the total PSF (including the
intensity distribution due to scattered emission photons) is not
changed due to scattering up to such depth because the wide
distribution of scattered emission photons does not contribute to
FWHM. In terms of contrast, signal decay in CCD-based MMM with the
increase of imaging depth is higher than that of SMM by an order of
magnitude at 2.times.l.sub.em.sup.s. Also the wide distribution of
scattered emission photons contributes as noise and causes loss of
contrast by another order of magnitude at the depth.
[0064] Imaging dermal structure based on autofluorescence has been
performed using the system of the present invention. Endogenous
fluorophores have low quantum yield and low extinction coefficients
compared with typical exogenous fluorescent labels. The dermal
structure imaged using a preferred embodiment of the present
invention has a layered structure with significantly different
indices of refraction resulting in significant spherical
aberration. Multiphoton imaging of dermal structures without
photodamage has a pixel rate of 15 KHz with 15 mW input power. In
this example an input power of 7 mW per focus at the specimen with
the excitation wavelength set at 800 nm. The objective used is
20.times. water immersion with 0.95 NA (XLUMPLFL20XW, Olympus,
Melville, N.Y.). With a frame rate of 2.5 fps for a 320.times.320
pixel image (4 KHz pixel rate) this is 10 times faster than the
previous systems. The epidermis is imaged down to the basal cell
layer using this MAPMT-based MMM. Representive layers from the
stratum corneum, stratum granular, and the basal layer are shown in
FIGS. 2a-2c. The signal from these layers are mostly due to the
fluorescence of NAD(P)H inside the cell. Thus an MAPMT-based MMM
has equivalent or improved sensitivity as conventional multiphoton
microscopy but with significantly increased imaging speed. The
intensity of the image is not uniform: the intensity is high in the
center and becomes dim in the corner of the image. This is because
the intensity of the excitation beam has Gaussian spatial
distribution so that the beamlets made from center part of the
expanded beam have higher intensity than the peripheral portions of
the beam. A beam splitter, serial dichroic mirrors or a top hat
holographic filter can be used to provide a more uniform array of
beams delivered to the individual focal positions.
[0065] Using both CCD and MAPMT detectors in MMM geometry, the
signal decay can be measured as a function of scattering length. As
the imaging depth increases, the signal is decreased due to
scattering of both excitation photons and emission photons. The
signal decay is measured by imaging 4 .mu.m diameter fluorescent
latex microspheres (F8858, Molecular Probes, Eugene, Oreg.)
immobilized in 3D by 2% argarose gel (UltraPure Low Melting Point
Argarose, Invitrogen, Carlsbad, Calif.). Intralipid emulsion
(Liposyn III, Abbott Laboratories, North Chicago, Ill.) is added to
the sample as a scatterer in various concentrations of 0.5 to 2%.
Intralipid emulsion of 2% volume concentration is known to have
similar scattering properties to those of tissues: mean free path
length (l.sup.s) of scattering is 80 .mu.m, 168 .mu.m at the
wavelength of emission (605 nm), excitation (800 nm) respectively.
The scattering properties of these intralipid solutions are
verified by diffusive wave measurements. Peak intensity of the
sphere image is a signal in the measurement and the decay of peak
intensity as a function of the imaging depth is measured at each
concentration. The signal decay can also be measured with a
conventional multiphoton microscope as a reference. Signal decays
in the three systems are measured down to a depth of 180 .mu.m
which is equivalent to 2.25.times.l.sub.em.sup.s (FIG. 5). The
signal decay is expressed an exponential function,
S(z)=exp(-cz/l.sub.em.sup.s). The decay coefficient, c is 1.22,
1.87, 2.30 in case of the conventional multiphoton microscopy,
MAPMT-based MMM, and CCD-based MMM respectively. The decay rate
from the conventional multiphoton microscope is the lowest as
expected. The decay is the combinational effect of both excitation
and emission photon scattering. Since the effect of excitation
photon scattering is the same, the difference in decay coefficient
is due to the effect of emission photon scattering. The decay
coefficient, c from MAPMT-based MMM (1.87) is lower than the one
from CCD-based MMM (2.30). However, the one from MAPMT-based MMM is
still higher than the one from the conventional multiphoton
microscope. It indicates that the spatial distribution of scattered
emission photons is wider than the effective detector area of the
MAPMT (45 .mu.m.times.45 .mu.m) so that some portion of the
scattered emission photons are collected in the neighboring
channels. The ratio of intensity sum collected in the neighboring
pixels of the MAPMT to the intensity in the correct pixel was
approximately 2 at the depth of 2.times.l.sub.em.sup.s.
[0066] Although a significant portion of the scattered emission
photons are still distributed outside the correct pixels in
MAPMT-based MMM, these photons can be effectively restored to the
correct pixels based on post-acquisition image processing. Note
that the photons acquired at each pixel are temporally encoded and
are organized to form an image based on the known scanner position
as a function of time. This is exactly how images are formed in a
conventional multiphoton or confocal microscope. A primary image is
formed by photons acquired at the correct pixels corresponding to
the fluorophore distribution in that portion of specimen. Note that
the scattered photons in the neighboring pixels are also similarly
temporally encoded. Therefore, secondary "ghost" images are formed
in the areas of the image covered by the neighboring pixels. As an
example, FIG. 4a is an image of spheres at 150 .mu.m deep from the
surface in 2% intralipid emulsion. The fact that the primary image
at one pixel is "copied" into neighboring pixel, the spatial
distribution of the scattered photons provides information for the
reassignment of these scattered photons back to the correct pixel.
Note that this temporally encoded information is not available in a
CCD-based MMM system where the temporal information is lost during
the integration process of the CCD. The effect of emission photon
scattering on imaging can be described as follows. Generally, an
image is formed as a convolution of source pixels and an emission
point spread function (PSFem). In the set up of MAPMT-based MMM,
the array of 8.times.8 pixels is collected together each time so
that the vector of pixels are acquired together, {S.sub.acq}
(64.times.1) which is the product of convolution matrix, [C]
(64.times.64) and the source pixels, {S.sub.s} (64.times.1),
{S.sub.acq}=[C].times.{S.sub.s}. The convolution matrix, [C] is
constructed based on the simplified PSF.sub.em, EPSF.sub.em in
which PSF.sub.em is spatially integrated over the effective
detector area of the individual pixels of the MAPMT. Since
EPSF.sub.em has a very coarse spatial resolution of 45 .mu.m with
the spatial integration, deconvolution with EPSF.sub.em becomes
simple and is less sensitive to noise. The study of emission photon
scattering on PSF.sub.em shows that the scattered emission photons
form additional intensity distribution around the PSF.sub.em, which
is formed with ballistic unscattered emission photons. Its
distribution is broad with its FWHM of 40 .mu.m range in the
imaging depth of 2.times.l.sub.s.sup.em. The change of PSF.sub.em
due to scattering (FIG. 4c) affects EPSF.sub.em by increasing
intensity in the neighboring pixel areas (FIG. 4d). In the process
of deconvolution, EPSF.sub.em is roughly estimated by measuring the
intensity ratio of the real image to the ghost images as a function
of imaging depth. The convolution matrix, [C].sub.est is
constructed based on the estimated EPSF.sub.em. The source pixel
vector, {S.sub.s}.sub.est is acquired by the product of the inverse
transform of [C].sub.est and the acquired pixel vector, {Sa,,q
{S.sub.s}.sub.est=[C].sub.est.times.{S.sub.acq} (2)
[0067] The restored image is presented in FIG. 4(b). The signal
decay of a depth sequence of restored images is measured and the
decay coefficient, c is significantly reduced to 1.58 after the
deconvolution algorithm because the scattered emission photon can
now be corrected and reassigned. The ghost images are almost
completely eliminated as a result. Restoration algorithms can be
further refined such as by adding maximum likelihood estimation to
minimize image structural overlap between neighboring pixels. This
simple deconvolution approach improves very effectively the
performance of MAPMT-based MMM and allows this system to perform
within a factor of two compared with conventional multiphoton
microscope.
[0068] The performance comparison of the two MMM systems can also
be evaluated for the imaging of biological tissues. The specimen is
an ex-vivo brain tissue section with neurons expressing green
fluorescent protein (GFP). Thy1-GFP transgenic mice are deeply
anesthetized with 2.5% Avertin (0.025 ml/g i.p.) and transcardially
perfused with PBS, followed by 4% paraformaldehyde. Brains are
dissected and placed overnight in cold 4% paraformaldehyde. 1-mm
thick coronal sections are taken by vibrotome, mounted and
coverslipped on microscope slides using adhesive silicone isolators
(JTR20-A2-1.0, Grace Bio-Labs, Bend, Oreg.). The specimen is imaged
in 3D with both CCD-based MMM and a MAPMT-based MMM. The objective
used is 20.times. water immersion with NA 0.95 (XLUMPLFL20XW,
Olympus, Melville, N.Y.). The input laser power is 300 mW at 890 nm
wavelength. The frame rate is 0.3 frames per second with
320.times.320 pixels. The slow frame rate is set in order to
collect enough emission photons up to 120 .mu.m deep. The total
imaging depth is 120 .mu.m with 1.5 .mu.m depth increment.
Representative images are shown in FIGS. 5a-5i. The first column of
images are from CCD-based MMM at surface, 30 .mu.m, and 75 .mu.m
deep. The second column of images are the ones from MAPMT-based
MMM, raw images and the third column are after deconvolution
processing. On the surface, the dendritic structures of neurons are
visible in all images. However, the image from CCD-based MMM does
not provide as good contrast of neurons as MAPMT-based MMM. This is
because some of the emission photons that are initially forward
propagating into the tissue are eventually backscattered. These
backscattered photons are acquired in the incorrect pixels of the
CCD and degrades the image SNR. Starting at about 30 .quadrature.m,
background noise increases and thin dendrite structure becomes
invisible in CCD-based MMM images. On the other hand, in the images
from MAPMT-based MMM, dendrites are still visible due to lower
background noise and higher SNR. In the image of 75 .mu.m deep from
MAPMT-based MMM, ghost images of a bright cell body appear in the
neighboring pixels. The ghost images are restored to the original
image after the deconvolution process is applied. And also it is
noted that the intensity of the original image is increased.
[0069] However, additional improvements of this system can be made.
First, since the MAPMT is positioned in the image plane, the
location of each excitation focus corresponds to the center
position of the matching pixel of the MAPMT. The effective detector
area scales quadratically with the separation of the foci.
Therefore, with wider foci separation, the MAPMT has higher
collection efficiency for scattered emission photons. In the
current configuration, the excitation foci are separated from each
other by 45 .mu.m so that the effective detector area for each
channel of the MAPMT is 45 .mu.m.times.45 .mu.m. The size of
imaging field with 8.times.8 foci becomes 360 .mu.m.times.360
.mu.m. As the excitation foci are separated more, the system
becomes less sensitive to the scattering of emission photons. The
maximum separation of excitation foci is limited by either the
field of view of the objective or apertures of other collection
optics. The 20.times. water immersion objective used has the field
of view of 1000 .mu.m in diameter. This allows positioning the foci
as far apart as almost 100 microns in this example.
[0070] A limitation of the MAPMT-based MMM system compared with a
CCD-based MMM design is that the signals are de-scanned. In the
de-scanned configuration, emission photons are processed by more
optical elements including the scanner mirror before they are
collected at the MAPMT suffering more optical loss at each
reflection. Further, the de-scanned geometry also has a longer
optical path that contributed to the loss of some scattered photons
due to the finite aperture of the optics. The signal collection
efficiency is approximately 70% in this example due to additional
optical elements. An MAPMT-based MMM system in a non-de-scanned
geometry for example can recover this loss.
[0071] The MAPMT is manufactured with a current quantum efficiency
of about 20% compared to 80% quantum efficiency of the CCD camera.
However, MAPMT has very low noise. It has 20 dark counts per second
without cooling and can be several orders of magnitude lower with
cooling. Since the MAPMT has a readout rate of approximately 20
KHz, the typical dark count per pixel. is less than
1.times.10.sup.-3. In comparison, the CCD noise is dominated by
both read noise and dark noise which are a few counts per pixel.
Therefore, for very low photon count situation, i.e. dim sample or
high frame rate, the MAPMT system can have superior performance.
MAPMTs with higher sensitivity cathode materials such as GaAsP can
provide a system with a quantum efficiency up to about 40-50%.
[0072] The photon sensitivity of each channel is not equal and can
vary up to 50%. This effect is further compounded by the Gaussian
geometry of the excitation beam which results in higher excitation
efficiency at the center pixels verses the edge region. This
problem has been solved previously using multiple reflecting beam
splitter to generate equal intensity beam-lets. The MAPMT-based MMM
system can be further improved by utilizing this type of beam
splitter with an additional flat field correction algorithm to
remove inherent sensitivity non-uniformity of the MAPMT.
[0073] There is also cross talk between neighboring pixels of
MAPMT. The typical crosstalk is minimal at about 2% when the
photons are collected at the center of each pixel. However, this
cross talk can be removed by post-processing of the image similar
to ghost image removal discussed previously.
[0074] In MMM imaging, more power of excitation light is required.
Assuming that input power of 10 mW is needed for each excitation
focus, generation of 64 excitation foci requires 640 mW input
power. In the imaging of turbid tissue specimens, more input power
is required to compensate the signal loss due to excitation photon
scattering. In case of a tissue specimen whose mean free path
length is 160 .mu.m at excitation wavelength, the input power of
2200 mW is required to image at 100 .mu.m deep, assuming that
signal level is decreased only due to excitation photon scattering
and there is no change in collection efficiency of emission
photons. Therefore, the current power of Ti-Sapphire laser is
limited for MMM imaging and can further increase in imaging speed
by the use of even more foci.
[0075] Referring to FIG. 6, a preferred embodiment provides for
parallelized illumination and detection device which uses a common
focusing device, such as an objective lens. The device provides
simultaneous measurement of intensity, lifetime, spectroscopic or
other information from the focal spots (foci 151, 152, 153). Light
from a first illumination light path 141, a second illumination
light path 142 and a third illumination light path 143, which paths
present to each other at relative angles, enter a common focusing
device 110 (such as an objective lens). The focusing device 110
generates from each illumination light path 141, 142, 143 a
separate intensity cone. A first detector 121, a second detector
122 and a third detector 123 detect light generated by the
intensity cones associated with each first, second and third
illumination path, respectively. Light from a first illumination
path 141 illuminates the focus spot 151 in the sample 105, with the
detected light following a first illumination and detection light
path 161 and first detection light path 111 to reach the first
detector 121. Similarly, light from second and third illumination
paths 142, 143 illuminate the focal locations 152 and 153,
respectively, in the sample 120, with the detected light following
second and third illumination and detection light paths 162 and 163
and second and third detection light path 112 and 113 to reach the
second and third detectors 122 and 123, respectively.
[0076] In the multi-photon case, light from each path generates a
3D intensity distribution in its associated focus, according to the
multi photon excitation process. The detectors 111,112,113 detect
all the light in the `detection cone` associated with their active
area. This light includes light generated by the light path
associated with each detector (for example, light from the first
focus 151 is detected by first detector 121), as well as light that
is generated in the first focus 151 but scatters around the first
focus 151 on its way to the first detector 151, and light that is
generated in the second and/or third foci 152, 153 and is then
scattered into the detection cone of the first detection light
detection path 111.
[0077] In the confocal case, a confocal pinhole is placed in front
of the detectors, for instance, in FIG. 6 a confocal pinhole can be
placed between each detector 121, 122 and/or 123 and the associated
reflectors and collimation lens 126, 127, 128, 131, 132 and/or 133,
respectively. As a consequence of the pin-hole impeding much of the
scattered light, according to the confocal principle, only light
from the focal spot associated with that detector is collected in
each detector. For example, following the first light path, only
light from the 3D light distribution in the first focus 151 is
detected by detector 121. A setup could as well consists of a
mixture of detectors with and without a confocal pinhole.
[0078] In order to reduce cross talk between the light beams due to
scattering, the illumination light and the associated detection can
be time multiplexed.
[0079] Still referring to FIG. 6, a device according to the
invention can become an imaging device by the illumination beams
141,142,143 being angle-scanned with respect to the focusing device
110. The imaging of the x-y planes is enabled by rotating the
device through two perpendicular angles theta and phi around the x
and y axes, respectively. The intensity information is recorded
along with the angular position of the device and reconstructed by
a image processor. Imaging of zy planes can be achieved as well by
scanning the sample in respect to the imaging device in xy. In an
imaging mode, in which the beams are scanned, the device is capable
of simultaneously generating 2D images of sub-regions of samples.
By simultaneously imaging with each separate illumination and
detection pathway, the speed at which images are generated can be
increased by the number of illumination paths and detection
channels.
[0080] Imaging in the z plane occurs by moving the imaging device
with respect to the sample, or vice-versa. The intensity
information is recorded along with the z-position of the sample or
device and reconstructed by an image processor.
[0081] Another embodiment according to the invention provides for a
multifocal, multiphoton microscope based on MAPMT, as illustrated
in FIG. 7, in which an expanded excitation beam 104A comes from
bottom of FIG. 7 and illuminates a square microlens array 140A. A
plurality of optical pathways is generated by the micro lens array
140A in conjunction with lens L1; for instance, in the embodiment
illustrated here the microlens array 140A splits the excitation
beam 104A into 8.times.8 multiple beams (i.e., 64 beamlets). In
FIG. 7, only two beamlets 141A, 142A are ray-traced. A specimen
105A is scanned with an 8.times.8 array of excitation foci 150A,
which includes focus spots 151A and 152A illuminated by beamlets
141A and 142A respectively. The sample area that each excitation
focus covers can be relatively small the focus is in x and y
direction, the full width half maximum (FWHM) of the focus is
200-1000 micro meter. In z direction the FWHM is 200-5000 micro
meter. In an imaging configuration each foci scans an area of the
size of the distance of the foci, meaning 10-1000 microns (the
scanning is accomplished by an optical scanner 180A such as, a
galvo-scanner). The two lenses L2 and L3 guide the plurality of
optical pathways onto the rear aperture of the focusing device
110A. The detection light paths, 111A and 112A, respectively,
resemble the illumination light path until the light paths are
separated by a light reflector, which is in this case a dichroic
mirror 130A. The light is then focused by a common lens L4 onto the
multi anode PMT detectors 120A. The emission beam-lets are
collected at pixels 121A and 122A, respectively, of a multi anode
PMT (MAPMT) 120A. The MAPMT 120A, which has the same number of
pixels as excitation beamlets, detects the signal of 8.times.8
pixels synchronized with the scanning. The intensity information is
recorded along with the angular position of the scanner and
reconstructed by an image processor.
[0082] As shown in FIG. 8, a further embodiment according to the
invention provides for a multifocal, multiphoton microscope based
on MAPMT, in which an expanded excitation beam 104B comes from
laser 101B and illuminates a micro lens array 140B. A plurality of
optical pathways is generated by the micro lens array 140B in
conjunction with lens L1; for instance, in the embodiment
illustrated here two beamlets 141B, 142B are ray-traced. A specimen
105B is scanned with an array of at least one excitation foci 151B
and/or 152B which are illuminated by beamlets 141B and 142B
respectively. The scanning is accomplished by a scanner 180B. The
two lenses L2 and L3 guide the plurality of optical pathways onto
the rear aperture of the focusing device 110B. The detection paths,
111B and 112B, respectively, depart from illumination light path
when separated by dichroic mirror 130B. The light is then focused
by a common lens L4 and reflector 134B onto two multi anode PMT
detectors 120B, 124B. The MAPMT detectors 120B, 124B each detect
the same number of pixels as are emitted excitation beamlets,
integrating the signal of the at least one pixel synchronized with
the scanning.
[0083] A z-piezo actuator 109B (such as MIPOS 250 SG,
micro-objective positioning system, integrated strain gauge motion:
200 .mu.m (closed loop), Piezo System Jena controllable by
controller 170B is attached to the objective lens 110B in order to
move it in the z direction for 3D image generation. The sample 105B
is attached to a sample stage 115B, which can be moved in x, y and
z directions, also controllable by controller 170B and/or computer
176B. Light reflector 134B (such as, for example, a dichroic
mirror) is positioned in the detection pathways to enable multi
channel imaging by a first MAPMT detector 120B and a second MAPMT
detector 124B, for multi channel imaging.
[0084] An IR block filter 116B (such as e700sp Special,
Multi-Photon Blocking, Block 750-1000nm>OD 6, Chroma Technology
Corp is positioned in the detection pathway to separate the long
wavelength excitation light from the short wavelength detection
light. The filter 116B is exchangeable with a variety of filters or
can be removed completely for reflected light confocal imaging. The
filter 116B can be mounted on a motorized mount, which allows it to
exchanged via a controller 170B and/or computer interface 176B. A
band-pass filter 117B (such as 560DCXR for detecting DAPI and FITC,
Chroma Technology Corp) (560DCXR for the transmission of light
generated by the excitation of GFP and Rhodamin, HQ460/40 for the
transmission of light generated by the excitation of DAPI, HQ630/60
for the transmission of light generated by the excitation of Alexa
594 bandpass; Chroma Technology Corp) is positioned in front of
each of the multi anode PMTs 120B, 124B, in order to detect certain
spectra. The band-pass filters 117B, 117B are exchangeable with
other different filters and can be mounted on a motorized mount
enabling changing of filters via a controller 170B and/or computer
interface 176B. The same sample region can then be imaged with a
different set of band-pass filters for more than two-color
imaging.
[0085] A detection-part light-shield enclosure 118B is used to
shield the detection part of the apparatus from ambient light. A
variable iris 119B (such as for the case of a manual version
D20S--Standard Iris, 20.0 mm max. Aperature; Thorlabs. Motorized
versions of equivalent devices are available as well) is positioned
in the focal plane of the micro lens array 140B in order to enable
single spot illumination. For 8.times.8 foci imaging, the iris 119B
is relatively open and for fewer or single spot imaging the iris is
relatively closed, so that only a few or one micro lenses
illuminate the sample. The variable iris 119B does not have to be
in a round shape; and it will be square in shape when only a
certain array of micro lenses should be blocked to enable
illumination with a view of selected foci only. The variable iris
119B can be motorized and controlled via controller 170B (such as,
for example by connection 191B), and/or via the computer 176B.
[0086] A micro lens foci mask 125B (such as a thin (for example 0.3
mm) aluminum sheet in which small wholes (for example 0.5 mm
wholes) are drilled, at the points of where the micro lens focuses)
positioned proximate the micro lens array 140B is a pinhole mask
with a large pinhole size that enables the transmission of most of
the light focused by the micro lenses, but otherwise it blocks
ambient and stray light from the laser.
[0087] A first reflector 131B generates a first laser reference
beam 165B from the incident laser beam (for monitoring the laser
illumination power, wavelength and angular pointing stability). The
reference beam 165B projects upon the diode or detector 160B which
generates a signal that measures the laser illumination power,
wavelength and angular pointing stability.
[0088] A further embodiment of the invention provides for a scan
reference beam 166B from a scan reference beam illumination source
168B to be projected via reflector 172B and reflector 132B onto the
scan region, whereupon the returning scan reference beam returns
via reflector 132B to pass through dichroic 172B and lens 174B to
be received by detector 164B. The scan beam is provided for
monitoring the scanning accuracy. Detector 164B can be a diode or
CCD detector or another type detection device. As shown in FIG. 8,
an embodiment of the invention can provide for the detector 164B to
be a CCD camera, which can be used to compare images generated by
CCD camera detection methods and other detection methods according
to the invention that employ one or more multi anode PMTs as
described above.
[0089] A high voltage power supply 188B supplies power to the multi
anode PMTs. Multi channel photon counting cards 184B, 186B are
connected to each element of the MAPMTs, with one photon counting
device for every multi anode PMT element, such as, for example,
MAPMT elements 120B and 124B. A computer 176B (including input
devices, such as, for example, a keyboard and mouse) can be
provided in one embodiment, connected to computer display 178B. The
computer 176B can be connected to controller 170B.
[0090] The computer 176B controls numerous elements of the
invention either directly and/or indirectly through controller
170B, and one skilled in the art will appreciate that numerous
alternative configurations can be implemented within the scope of
the invention.
[0091] One embodiment provides for the computer 176B to be
programmed with a processing software and for the computer 176B to
control a number of optical elements through a variety of
electronic interfaces. For example, without limitation, the
computer 176B and/or the controller 170B can be electronically
interfaced with the scanner 180B and the multi channel photon
counting cards 184B, 186B to perform the steps of scanning and data
acquisition. Further the computer 176B can perform imaging
post-processing steps. The display 178B can be used to display the
acquired images in real-time after further processing.
[0092] A laser power attenuator 163B can be provided to control the
laser incident power. The attenuator 163B can be controlled by the
controller 170B and/or by the computer 176B in order to enable
power adjustments for different samples and different locations in
samples. During imaging at different depths in the sample, for
example, the laser power can be automatically adjusted, so that the
laser power can be increased at higher penetration depth. The
attenuator 163B is integrated in order to make laser power
adjustments, such as, for example, low power at the sample surface
and increased power at increased penetration depth.
[0093] A third reflector 133B generates a second laser reference
beam 167B from the incident laser beam (also for monitoring the
laser illumination power, wavelength and angular pointing
stability). This second laser reference beam 167B projects upon a
second diode or detector 161B to generate a signal that measures
the laser illumination power, wavelength and angular pointing
stability. A laser power attenuator 163B controls the laser
incident power and is integrated in order to make laser power
adjustments, such as, for example, low power at the sample surface
and increased power at increased penetration depth. Laser 101B is
an illumination light source, such as a titanium sapphire laser
(Mai Tai HP, Spectra Physics).
[0094] Multi-photon microscopy works most efficiently with short
laser pulses owing to dispersion, the optical elements in the
illumination pathway broaden the initially short laser pulse. The
pulse compressor 102B is built from a pair of standard high
reflectance mirrors and a pair of prisms (IB-21.7-59.2-LAFN28,
Material: LaFN28; CVI Laser Corp., Albuquerque, New. Mex. 87123)
mounted on translational and rotational stages pre-chirps the laser
pulse in order to attain a short laser pulse in the focus of the
objective lens.
[0095] A confocal pinhole array optionally can be placed between
either of the multi anode PMT arrays 120B, 124B and the band-pass
filters 117B, 117B, respectively. This option enables the system to
be used for confocal microscopy or for multi-photon microscopy with
confocal detection.
[0096] A telescope 103B expands the laser beam. With different
expansion ratios, a different number of micro lenses can be
illuminated. With a small beam expansion for example, a relatively
smaller array of 2.times.2 micro lenses can illuminated and, thus,
an array of only 2.times.2 foci is generated. As the beam expansion
is made larger, an array of 8.times.8 or more micro lenses can be
illuminated and, thus, an array of 8.times.8 or more foci is
created. A further preferred embodiment employs a set of at least
two mirrors 135B, 136B after the telescope 3B for precise beam
alignment.
[0097] A mechanical micro lens holder 145B enables the precise
positioning of the micro lens array 140B with respect to the multi
anode PMTs 120B, 124B in the x, y and z directions. The holder 145B
can be a motorized holder and can be controlled through a computer
interface 176B, or, alternatively, can be controlled via a
controller 170B, which controller in turn can be directed by
computer 176B.
[0098] Mechanical multianode PMT holder 125B, 126B enables the
precise positioning of the multi anode PMTs 124B, 120B,
respectively, with respect to the micro lens array 140B in the x, y
and z directions. The holders 125B, 126B can be motorized holders
and can be controlled through a computer interface 176B, or,
alternatively, can be controlled via a controller 170B, which
controller in turn can be directed by computer 176B.
[0099] The computer 176B, or the controller 170B, or the computer
and controller together can be configured to control automatically
or to control in supervised fashion, one or more of the following
elements, without limitation: the scan reference beam illumination
source 168B, the sample piezo stage 115B, the objective z-piezo
stage 109B, the scan reference beam detector 164B, the scanner 180B
(by connection 193B), the IR block filter 116B (by connection
194B), the band-pass filters 117B (for example, by connection
195B), the laser source 101B, the laser attenuator 163B, the first
laser reference beam detector 161B, the second laser reference beam
detector 160B, the pulse compressor 102B, the multi-photon channel
counting cards 184B, 186B, the mechanical multi-anode PMT holders
125B, 126B (for example, by connection 190B), the variable iris
119B (such as, for example, by connection 191B), and the mechanical
micro lens array holder 145B (for example, by connection 192B).
[0100] The focal region has a focal pattern variation in xy plane.
The foci can be distributed unevenly, e.g., the rows and columns do
not have to be spaced uniformly.
[0101] Also, a system can be built, in which there are additional
rows and/or columns of PMT's at the outer region of the array of
detection tubes. For example, there can be more than 8.times.8 rows
and columns in both the micro lens array and the detector. This is
particularly important for detecting scattered photons of the outer
foci and for using the information of the scattered photons from
the outer foci for deconvolution purposes.
[0102] Further, an embodiment of the invention provides for a
system in which there are more detector elements than there are
foci, so that a plurality of detector elements (or detection
pixels) collect the photons of one, optically conjugated foci. For
example, a 16.times.16 detector array can used as a detector
device, while an array of 8.times.8 foci can be illuminated by an
8.times.8 multi lens array. Smaller and larger PMT-to-foci ratios
can be utilized. In particular, a detector array in which one foci
is optically conjugated to an uneven number of detector elements
can be employed. This is important for detecting scattered photons
in the neighboring channels of the to the focus optically
conjugated detector and for using the information of the scattered
photons for deconvolution purposes.
[0103] The image of the sample is formed by scanning in the optical
plane (xy) when the intensity signal from the detectors is
correlated with the foci positions. The foci scan the specimen in
the x direction, then move an increment in the y direction, and
then raster in the x direction again until the sample is fully
covered at some desired resolution. During the rastering, intensity
light signals are recorded by the multi anode PMT. These signals
are then saved along with the foci positions in the computer and
can be concurrently or afterwards displayed by the computer display
or other graphics outputs. The foci positions are known by the
scanner position (beam scan) or the sample position (stage scan).
The smaller the step increments, the higher the resolution the
final image will be. The scanning can be performed in a raster
fashion, or in many other ways, such as with time multiplexed
methods, or scanning simultaneously at different depths.
[0104] Referring to FIG. 9, an embodiment of the invention provides
for generating and detecting a 3D foci pattern in focal region
154C. A source of light is directed onto a micro lens array 140C
and a plurality of optical pathways is generated by the micro lens
array 140C in conjunction with lens L1; for instance, in the
embodiment illustrated here, the microlens array 140C splits the
excitation beam 104C into 8.times.8 multiple beams (i.e., 64
beamlets). In FIG. 9, only one beamlet 141C is ray-traced. A
specimen 105C is scanned with an 8.times.8 array of excitation
foci, which includes focus spot 151C illuminated by beamlet 141C.
The scanning is accomplished by an optical scanner 180C. The two
lenses L2 and L3 guide the plurality of optical pathways onto the
rear aperture of the focusing device 110C. The detection light
path, 111C, resembles the illumination light path until the light
paths are separated by a light reflector, which in this case is a
dichroic mirror 130C. The light is then focused by a common lens L4
onto the multi anode PMT detector pixel element 121C. Changing the
focal length or the positions of the micro lenses of the microlens
array 140C with respect to each other generates collimated and
non-collimated beams at the back aperture of the objective lens
110C. These beams generate a 3D pattern of foci. The 3D pattern of
foci generates light which is collected by the detector array.
According to the positions along the optical axis of the micro lens
array, the positions of the PMT's are changed. In case of
one-photon illumination, such as is illustrated in the "Option I"
detection region 124C, no confocal pinholes are placed in front of
the detectors. In an alternative embodiment, such as is illustrated
in the "Option II" detection region 126C, a plurality of confocal
pinholes 128C are placed in front of the plurality of detectors
cells. Each of the detection options 124C, 126C can be used for
single photon and/or for multi photon imaging. The MAPMT, which has
the same number of pixels as excitation beamlets, integrates the
signal of 8.times.8 pixels synchronized with the scanning, although
other array dimensions can also be used.
[0105] FIGS. 10(a)-(i) illustrate in greater detail the arrangement
and progression of foci corresponding to the relative shifting in
position of micro array lenses and MAPMT pixels shown in FIG. 9.
FIG. 10(a) shows an expanded detail of the focal region with 3D
foci pattern. FIG. 10(b) illustrates an array of excitation light
beams (in this case, an array of 2.times.8 beams) illuminating a
focusing device 110D, such as, for example, an objective lens, as
viewed here in the x-z plane. In accordance with differing degrees
of collimation of at least two of the light beams, focal points
151C and 152C are created for the two beams at certain distances,
d.sub.1 and d.sub.2, respectively, along the optical axis (z-axis).
According to the relative angle of the illumination light beams
with respect to the optical axis, the array of foci are separated
in the optical plane. Controlling parameters of the beams provider
selection of a variety of 3D foci distribution(s). For collimated
light, the excitation foci 151C is at a distance from the focal
objective, fobj, designated here as distance d.sub.1. For a second
beam that is not collimated perfectly, an excitation foci 152C is
at a distance d.sub.2 that is not equal to the focal objective, as
depicted in FIG. 10(b). FIG. 10(c), depicting a "static" view of an
x-y plane "slice" at focal depth d.sub.1, illustrates a first row
of 8 foci (of the 2.times.8 array in this example) all lying at the
same focal depth d.sub.1, understanding that any one of these foci
may correspond with the 151D focus point in the x-z plane view of
FIG. 10(b). Similarly, FIG. 10(d), depicting a "static" view of a
second x-y plane at focal depth d.sub.2, illustrates a second row
of 8 foci (of the 2.times.8 array) all lying at the same focal
depth d.sub.2, any one of which foci might correspond with the 152C
focus point in the x-z plane view of FIG. 10(b). FIG. 10(e)
illustrates the separation of the two x-y planes as viewed in the
y-z plane, where it can be seen that the two rows of foci are
separated by a difference in focal depth D=d.sub.1-d.sub.2. FIG.
10(f) shows a 3D, 8.times.8 beam matrix (64 beams) of excitation
foci that have been generated. Each row of 8 foci lies in a
different z plane, as depicted in a view of the same set of 64 foci
as seen in the y-z plane (FIG. 10(g)); this is a graphic depiction
of a "still" configuration, i.e., without the array being moved in
a scanning mode. FIG. 10(h) illustrates an x/y view in a scanning
configuration, where each line of foci is scanned in the xy plane
to cover the whole yx image in its particular z-plane. A number of
xy planes are shown simultaneously, but actually each plane lies at
a different focal depth on the z-axis. FIG. 10(i) provides a view
of a section in the yz plane, illustrating the scanning
configuration while the z/y scan is performed, i.e., scanning along
the y axis and through multiple depth layers in z. As a result, a
3D volume can be imaged by only scanning the foci array in xy. Note
that the x/y, x/z and y/z coordinates illustrate the associated
planes; they can be displayed with an arrow in their positive or
negative direction.
[0106] 3D AM-PMT MMM can be used in multi photon endoscope device
in accordance with another preferred embodiment of the invention
(to be added).
[0107] Referring to FIG. 11(c), a further embodiment provides for a
3D cytometer, based on multi-focal, multi-photon microscope with a
multi anode PMT detector. A 10 W solid state pump laser 100D pumps
a titanium sapphire laser 101D (Millennia X & Tsunami, Spectra
Physics, Mountain View, Calif.), which generates maximum output
power of 2.5 W at 800 nm, and 120 fs pulses at a repetition rate of
76 MHz. The light is conducted through two reflectors 137D, 138D
and then passes through a first telescope 103D, two additional
reflectors 139D, 133D, an attenuator 163D, and a second telescope
203D. After passing through another reflector 136D, the light is
subsequently split into an array of beams by the micro lens array
140D and is transmitted by lenses L1, L2 and L3 onto the back
aperture of the objective lens 110D, thus creating multiple foci in
the focal plane. The micro beams are scanned by a xy-scanner 180D
(Cambridge Technologies, Cambridge, Mass.). The fluorescence is
collected by the same lenses and separated from the illumination
light by a dichroic filter 130D and a two-photon block filter 116D.
The fluorescence passes through lens L4 and is then separated into
two spectral channels by the dichroic filter 134D and directed onto
the multi-anode PMTs 124D, 120D. The degree of spectral separation
can be chosen depending upon the application. The embodiment
disclosed here uses a red/green and a green/blue filter to
accomplish the spectral separation. The variation of the
magnification of the telescope 203D enables the utilization of, for
example, 4.times.4, 6.times.6 or 8.times.8 arrays of micro lenses,
among other size arrays.
[0108] FIG. 11(b) illustrates an image of the array of foci in the
focus of the objective lens, such image as can be taken by a CCD
camera, where here the foci are not scanning. The foci are 45 .mu.m
apart in resulting in a potential scanning field of 240 .mu.m when
6.times.6 foci are utilized.
[0109] FIG. 11(c) shows a z-profile and a corresponding fit
function of a 200 nm bead. The system shows a resolution of 2.4
.mu.m, which is close to a theoretical value of 2.2 .mu.m,
considering the under-fulfillment of the back aperture of the
objective lens. Acquisition speed for this scanning profile was 10
frames per second. The profile is averaged over 5 consecutive
pixels, reducing the sampling from 30 nm per pixel to 150 nm per
pixel.
[0110] Referring to FIG. 12(a) a further embodiment of the
invention provides for multi color detection MMM in the xz-plane.
An array of 2.times.8 beams is generated by the micro lens array
140E. The setup here is illustrated with two 1.times.8 beam lines.
The distance between the foci in each line is determined by the
combination of the source beam configuration and the micro lens
array 140E. Two light beams are conducted through the micro lens
array 140E and intermediate optics onto the focal plane of the
microscope in which they create two lines of 1.times.8 foci. For
simplified visualization, in FIG. 12(a) only 3 of the 16 beam
traces in a 2.times.8 setup are illustrated. The full field is then
scanned by the mirror oscillation of the scanning mirror 180E, in
which the scanning amplitudes need to be adapted to the distances
of the foci. On the detection side a holographic diffraction
grating 192E is incorporated that diffracts the multiple
wavelengths emitted from the sample onto the photo-multiplier
arrays of two stacked multi-anode PMTs 120E, 124E. In the setup the
two multi-anode PMTs will be stacked on top of each other, each
serving as a spectral detection device for one line of 1.times.8
foci. The grating 192E properties (pitch/inch) and the focal length
of the focusing lens (L4), which determines the distance between
the grating and the multi-anode PMT, have to be chosen in
accordance to the anticipated fluorescent probes used for staining
the tissue sample. For this embodiment, a transmission grating 192E
is used. Nevertheless, comparable and/or better efficiency can be
achieved in embodiments that use a reflection grating or a prism.
FIG. 12(b) illustrates the illumination foci and their scanning in
the focal xy-plane. Scanning is indicated for two arrays of 8 foci
each. FIG. 12(c) shows the detection path of two beams projected in
the yz-plane through grating 192E and lens L4 onto the stack of two
AM-PMTs 120E, 124E. FIG. 12(d) shows the detection path projected
in the xz-plane, where the beams are depicted passing through
grating 192E and lens L4 with each of eight color bands being
collected by the two AM-PMTs 120E and 124E. FIG. 12(e) illustrates
the anodes of the multi-anode PMTs 120E, 124E in the x/y plane,
showing that the 8.times.8 anode arrays of the detectors each
detects one of the two 1.times.8 beam lines, where each 1.times.8
beam line has been diffracted by the grating 192E into eight color
bands.
[0111] Referring to FIG. 13, a beam splitter device 400 can be used
to create a homogenous intensity profile over a plurality of
beamlets. Depending upon its design, the beam splitter splits one
beam into 256, 128, 64, 36 or 16 approximately equally powered
beams by one or more fully reflective or semitransparent mirrors.
In FIG. 13 50% and 100% indicate the percent reflectance of the
mirrors used, where a series of fully reflective mirrors 420 with
one longer semi-transparent 410 mirror splits the beam in the
x-plane (BS-X). By combining two such cubes in series, it is
possible to generate a 2D array of beamlets. The internal optics of
a second beam-splitting cube for the y-plane (BS-Y), are the same
as for x-plane beam splitting cube. The beams are then focused by
micro lens 430 (or via other multifocal optics) through lens 432
and objective lens 434 onto the focal plane.
[0112] FIGS. 14(a)-14(d) illustrate additional preferred
embodiments for providing multifocal illumination including a
micro-lens array 140N from expanded beam 201N in FIG. 14a and FIG.
14b a diffractive optical element 205N separates beam 201N into a
separated plurality of beams which are coupled to focal locations
as previously described herein. In FIG. 14c a plurality of optical
fibers 220N can be used to provide a plurality of beams with lens
L1 for delivery to the focal locations or spots. As seen in FIG.
14d the fibers 220N can position beams in different directions for
smaller or greater focal separation.
[0113] In addition to the primary use of this instrument for two
photon microscopy, other multi-photon sensing and imaging methods
can also be used with the system described herein including:
[0114] 2, 3, or more photon excitation microscopy,
[0115] second, third ore more Harmonic Generation microscopy,
[0116] coherent anti Stokes Raman scattering (CARS) microscopy,
[0117] multi photon quantum dot imaging,
[0118] surface plasmon imaging, and
[0119] Stimulated Emission Depletion (STED) microscopy
With the implementation of a confocal pinhole array, shown in FIG.
8, confocal microscopy can be preformed with the same
instrument.
[0120] FIGS. 15(a)-15(d) illustrate further preferred embodiments
for use with detectors which can be a multi anode PMT or an array
of single detectors, connected via optical fiber. The detectors can
be PMT's or avalanche photo diodes, or the detector array can be a
combined device (like a multi anode PMT), connected via optical
fiber, an avalanche photon diode array, a CMOS imaging detector, or
a CCD camera in which each pixel or each area of binned pixels is
correlated to one focus, or a CCD camera in which more than one
pixel or more than one binned pixel area is correlated to one
focus. As seen in FIG. 15a, the detector 210P can be coupled
directly to optical fibers 220P which receive light from lens L1.
As shown in FIG. 15b, individual detectors 210P can collect at
different angles, or as seen in FIGS. 15c and 15d, a detector array
212P can detect at the same or different angles respectively.
[0121] Referring to FIG. 16, the optimal number of foci for a two
photon excitation process at a certain laser power for samples with
different damage thresholds can be determined. The optimal number
of foci will depend on (i) the damage threshold, (ii) the quadratic
dependency of the two-photon signal to the laser power, and (iii)
the limited amount of laser power. In the graph shown in FIG. 16,
the laser power is limited to 1.2 W at the sample, while the damage
threshold of the sample ranges can be 10 mW, 20 mW and 50 mW. As a
result, the optimal number of foci is 120, 60 and 24 respectively.
In general, the appropriate power level for two-photon imaging is
constrained by two basic boundary conditions: (a) the minimum
accepted signal-to-noise ratio determines the minimum power that
can be used, whereas (b) the damage threshold of the sample
determines the maximum power. In an MMM system, the limited laser
power is distributed over a large number of foci. The best signal
is obtained when the number of foci is chosen in a manner such that
each of the foci delivers a power level just below the damage
threshold for the sample. The relationship is illustrated in FIG.
16. It is possible to obtain less signal from the sample as more
foci are used, owing to the squared dependence of signal on laser
power. A judicious choice of power levels and of number of foci
must be made in order to obtain optimal results. Therefore, a
preferred method and system provides for a versatile system in
which the number of foci can be varied with respect to the sample
threshold. The threshold can be different for different penetration
depths into the sample and can therefore be adjusted by the
attenuator 163B.
[0122] Time multiplexed illumination and detection enable MMM
microscopy with one detector only, which is gated to the excitation
light pulse. In one variation of a multi-photon MMM, the
illumination light source is a pulsed laser. In this case, a Ti:Sa
Laser with a repetition rate of approx. 80MHz and a pulse width of
approx. 100-200 fs as an example. In the standard illumination
version of this MMM, all beams carry the same pulse distribution
along time. As a consequence, the array of excitation foci in the
focal region is formed simultaneously. For image formation, during
or after at least one pulse has illuminated the sample, the beam or
sample is scanned on both axis perpendicular to the optical plane;
here indicated by x.gtoreq.0.
[0123] In the example of multiplexed operation 500 shown in FIGS.
17a and 17b 36 detection elements are collecting light from 18
simultaneously illuminated spots. The delay between the
illumination pulses 502 is alternated between foci shown at 504.
This configuration can be imagined to be accomplished with more
detection channels per simultaneously illuminated foci. For image
formation, during or after at least one pulse has illuminated the
sample, the beam or sample is scanned on both axes perpendicular to
the optical plane; here indicated at 506 by x.gtoreq.0. Depending
on the particular configuration, there are several different
advantages. First, the light in non-corresponding detection
channels has an additional time delay relative to light from the
foci corresponding to the detector channel 508 that is receiving
light at a particular time. In fast processes, the resulting
signals may not or may minimally overlap and thus be registered to
the proper foci directly. In slower processes, where the overlap
may be significant, the temporal separation will aid numerical
registration and deconvolution algorithms. Furthermore, when the
number of detectors matches or exceeds the number of illumination
foci, the response in the neighboring non-corresponding detectors
can be used to generate additional information about the sample.
The temporal delays introduced into the illumination foci mean that
this supersampling condition exists even when the number of
detectors is the same as the number of foci.
[0124] In anther example, alternating excitation foci pattern can
be detected by a multichannel detector with smaller number of
elements than number of foci.
[0125] In cases in which the repetition rate of the laser is lower
that in the case of the Ti:Sa laser, a time multiplexed MMM
illumination and a detection in a single channel can be used. In
this particular case, the repetition rate of the laser is a hundred
times lower than in previous examples. In a time multiplexed
version, each beam carries a pulse which is temporally separated in
regards to pulses of the other beams. In one particular case, they
are separated evenly over the time period of one laser repetition,
so that at evenly distributed time points, one single foci is
illuminated at a time. If a fast detector is correlated with the
pulse distribution and capable of detecting each pulse separately
during this short time period, an MMM with only one detection
element can be used. This detection element has a corresponding
detection area to collect light which is generated by each
individual foci during its scan. For image formation, during or
after at least one pulse cycle has illuminated the sample, the beam
or sample is scanned on both axis perpendicular to the optical
plane. As a result, optical cross-talk is completely eliminated, as
the light from the different foci is excited and detected at
different time points. Applications for this case are excitation
processes which appear instantly, like scattering effects (such as
Second Harmonic Generation (SHG) or Coherent Ramen Anti Stokes
scattering (CARS)). This configuration can be used suited for a
non-de-scanning configuration. Other repetition and detection rates
are possible.
[0126] In the case of using a pixellated detector such as a CCD or
a CMOS imager, an array of 3.times.3 beams, FIG. 18a, illuminates
the focusing device, forming an array of 3.times.3 foci (FIG. 18b).
As seen in FIG. 18c, the image of the scattering distribution of
the foci is imaged by the CCD camera. The wide-field image is
accumulated per scanned illumination point. The scattering
distribution of each foci can be recorded on many CCD pixels. The
illumination or sample can be scanned and the wide-field data can
be further processed to form an image or statistical representation
from many object points. This configuration can be employed in a
non-de-scanning configuration as well.
[0127] The above described systems and methods can be used for
imaging of all semi-transparent and highly scattering materials; 2D
and 3D, and in particular for imaging of human and animal tissue,
cells in suspension, plants and other biological material.
[0128] The illumination can be achieved with visible light and
alternated with the MMM scanning measurement or out of band
illumination light can be used and the camera measurement can be
taken simultaneously with the MMM measurement. This configuration
can be used for large field imaging, sample guided MMM
measurements, conventional staining measurements, and online MMM
measurement process control, for example, bubble formation
monitoring, and laser spot diagnostics.
[0129] There is a large variety of fluorescent that can be used
with various embodiments of the invention dyes. In general they
fall into two families: Dyes that have to be applied to stain the
tissue "from the outside" and dyes, that are expressed from animals
as proteins. Most commonly used dyes by external staining
MitoTracker Red, DAPI, Hoechst 33342, Cy2-IgG, Alexa Fluor 546,
Rhodamine 123, Alexa Fluor 488, FITC-IgG, Acridine Orange, Cy3-IgG,
Issamine Rhodamine, TexasRed-Phalloidin, TexasRed-IgG, Alexa Fluor
594, Propodium Idonide. Dyes genetically expressed by genetically
modified animals: green fluorescent protein (GFP) and other dyes in
this family: Enhanced GFP EGFP, Yellow fluorescent protein (YFP),
Enhanced YFP. Auto fluorescent imaging does not use a particular
dye, but can be used as part of an imaging technique.
[0130] Besides confocal microscopy (fluorescent, as well as
reflected light confocal), these include all other multi-photon
microscopy techniques, such as, 2, 3, or more photon excitation
microscopy, Second (SHG), Third (THG) ore more Harmonic Generation
microscopy, Coherent Anti Stokes Raman Scattering (CARS)
microscopy, multi photon quantum dot imaging, surface plasmon
imaging and Stimulated Emission Depletion (STED) Microscopy. These
techniques can be used with or without staining methods. The
scattering techniques, such as SHG, THG, CARS are developed to be
able to image without any staining involved.
[0131] FIG. 19 illustrates a probe or endoscope apparatus according
to an embodiment of the invention, having a handle portion 272F and
an insertable probe portion 270F, wherein light delivered from a
light source 244F (which can be a laser or other light source) is
delivered through an optical wave guide 234F (such as, for example,
optical fiber, hollow fiber optics, photonic band gap fiber, or
other wave guide) to an optical connector 224F (such as, for
example, a pigtail), whereupon an expanded beam 104F passes through
a lens or optionally through lens pair telescope 103F and then
through a micro lens array, or other optical device that creates a
plurality of optical pathways, 140F. The illumination path can then
pass through lens L1, dichroic 130F and lenses L2 and L3 onto the
rear aperture of the objective 110F The beam is made to scan by
scanner 180F which can tilt in the x and/or y directions, and the
return fluorescent signal is directed by dichroic 130F and
reflector 136F, optionally an IR block filter 116F through lens L4
and optionally a band pass filter 117F onto a multi-anode PMT
detector 120F. In an alternative embodiment, a plurality of
confocal pinholes 119F are placed in front of the plurality of
detectors cells. The detector 120F can be connected to a controller
170F and to an image processing computer 176F. The scanner 180F can
also be controllably connected by electrical connector 193F to a
controller 170F and/or computer 176F. In FIG. 19, the proportions
of the endoscope have relationship to the focal distances of the
micro lens array, fm, the lenses L1-L4, being f.sub.L1, f.sub.L2,
f.sub.L3 and f.sub.L4, offset distances d.sub.1 and d.sub.2, and
the size will be related to the relative size of the various
elements.
[0132] Referring to FIGS. 20(a)-(b) through FIGS. 24(a)-(b), in
which common elements share the same numbering between figures, the
active area, relative proximate orientation of active detector
elements (such as, for example, the active area of multiple anode
photomultiplier tube detector elements), and the distance of the
foci and the intermediate optics have an important relationship to
the effectiveness of detecting scattered light from one or more
light spots in a sample specimen, as explained in the
following.
[0133] FIG. 20(a) depicts one PMT and one excitation focus, and the
direction of scattered light with respect to the detection light
cone of the active detection area will control whether or not the
photon will be detected. FIG. 20(a) provides an illustration of how
the size of the active detection area relates to scattered light
detection from a spot created by multi photon excitation, as
follows: An illumination light beam 204G coming from the left
(parallel solid lines), generates a multi photon excitation light
spot 251G (so-called excitation point spread function) in the
sample 105G, in which the structure causes a multi-photon
excitation process. Within this 3D sample region, light is
generated according to the multi-photon excitation principle and
scattered on its path (such as, for example, an auto-fluorescent
tissue sample). The potential detection path is illustrated by the
very thin bounding lines enclosing the stippled shaded region,
which are geometrically determined by the side boundaries of the
active PMT detection area 222G of the detector 120G. In this
configuration, all the photons that propagate within the detection
cone, indicated by the shaded region, and that travel in the
direction of the detector 120G, are collected in the active
detection area 222G. Photons that propagate in the opposite
direction, or that are scattered outside of the detection cone
defined by the optics and thus outside of the active detection area
222G of the detector 120G, are not detected. This is depicted in
FIG. 20(b) as well, where the detection area 224G (dashed box) in
the sample focal plane 210G corresponds to the active detection
area 222G, while a potential scattering region 254G (circle)
extends beyond the confines of the detection area 224G. Three
examples of photon path are shown in FIG. 20(a), as follows: (1) An
unscattered photon 261G, traveling in the opposite direction to the
incident light beam follows a path 262G (solid line), which path
262G lies within the detection light cone and travels towards the
active detector area 222G and is thus collected; (2) a first
scattered photon 271G that is scattered within the detection cone
follows a path 272G (short-dash line) and travels in the direction
of the active detector area 222G and is thus detected; and (3) a
second scattered photon 281G that follows a path 282G (long-dash
line) which travels generally in the direction of the detector but
does not fall into the detection light cone of the active PMT
detection area 222G, and thus it is not detected. Light generated
in the spot 251G is detected by the same objective lens 110G. It
can as well be detected by an opposing objective lens and collected
by a detector associated with the detection area to the light spot
at the opposing side. Then also photons in the detection cone of
the second, opposing lens, traveling towards the direction of the
incident light into the opposing detector, are collected.
[0134] Referring to FIG. 21(a), when scattered light from one
excitation foci 251G (single excitation point spread function
(PSF)) is detected in a setup with two PMT elements 120G, 124G, the
gap 232G (also marked as "g") between the active detection area
222G of detector element 120G and the active area of the second
element 124G will correspond to the gap 236G in FIG. 21(b) between
two detection areas 224G, 226G in the sample focal plane. If the
scattering region 254G around the excitation foci 251G extends into
the detection area 226G, then the photon 281G that is scattered
beyond the detection cone for active detector area 222G of detector
120G can follow photon path 282G into the adjacent detector
124G.
[0135] Scattered light detection from a spot 251G created by multi
photon excitation, detected by two large area detectors 120G, 124G,
positioned next to each other, are separated by a distance: In this
case, the unscattered photon 261G and the first scattered photon
271G are still collected by the active detection area 222G of the
first detector 120G. The second scattered photon 281G is not lost,
but is collected by the second detector 124G. This effect of light
being scattered into detectors other than the optically conjugated
detectors is termed "optical cross talk".
[0136] FIG. 22(a) illustrates scattered light detection with two
PMTs 120G, 124G and two excitation foci 251G, 252G, where again the
issue of "optical cross talk" is relevant. Here a second
illumination light path 206G at an angle .PSI..sub.1 with respect
to the first illumination light path 204G, creates a second focus
252G (excitation PSF) at a distance .delta. from the illumination
light spot 251G. In FIG. 22(a), only one, unscattered photon 291G
is illustrated in order to simplify the drawing. This photon 291G
originating from 252G follows optical path 292G into detector 124G
(although the illustration includes a collimating lens between the
reflector and the detector and a refraction in path 272G by said
lens is depicted, owing to constraints in the size of the drawing
and to illustrate better the features emphasized here as aspects of
an embodiment of the invention no refraction in the paths of 282G
and 292G is depicted in this illustration). Light originating from
the second light spot 252G will be collected by the second detector
124G, but also by the first detector 120G as well because photons
from the second light spot 252G are similarly scattered as photons
from the first light spot 251G. FIG. 22(b) illustrates this by
showing scattering area 256G overlapping both the detection areas
226G and 224G (again the gap 236G between the detection areas in
the focal plane will correspond with the gap 232G between the
boundaries of the active detector regions). As a result, scattered
light from light spot 252G is falling in the detection cone of
first detector 120G and will be detected by that detector 120G
(i.e., optical crosstalk). With increased scattering of photons in
the sample, the optical crosstalk increases. Samples with a low
mean free path length (MFP) for photons will induce photons to
scatter more (scattering more times at equal traveling lengths);
thus, samples with low photon MFP will induce higher optical cross
talk. In addition, increasing the imaging depth will raises the
probability that a photon will be scattered on its way through the
sample to the detector, because the traveling length in the high
scattering media is longer. Because light with longer wavelength is
scattered significantly less, applications with relatively longer
detection wavelengths have often been preferred, in order to reduce
the effect of optical cross talk. However, preferred embodiments of
the invention provide methods for reducing optical cross talk
without having to shift to longer wavelengths.
[0137] Referring to FIG. 23(a), for instance, the invention
provides for reducing optical cross talk by increasing the gap
distance between the excitation foci, this gap distance depicted as
.gamma., and simultaneously increasing the gap 232G between the
active detection areas in the detection elements. In this case, the
second illumination light path 206G is separated further by an
angle .OMEGA..sub.2>.PSI..sub.1 from the first light path 204G,
generating an illumination light spot 252G in a location that is
larger distance .DELTA.>.delta. from the illumination light spot
251G. The detectors 120G, 124G are also separated from each other
by increasing gap 232G between the active detector areas to a value
"G", where G>g, such that the second unscattered photon path
282G no longer falls into the second detector 124G. FIG. 23(b),
illustrates this by showing no overlap between scattering regions
254G, 256G and the neighboring detection areas 226G, 224G,
respectively. With this configuration, optical cross talk is
reduced because fewer photons end up in the "wrong" channel;
however, some scattered photons will not be detected because their
paths will pass between the active detection areas of more widely
separated detectors. Thus, detection light (signal) is lost.
[0138] Referring to FIG. 24(a), a preferred embodiment of the
invention provides for reducing optical cross talk without inducing
signal loss, by increasing the distance between the excitation foci
and simultaneously increasing the active detection area of the
detector elements. By separating the foci and the associated
detectors, the optical cross talk is reduced. By increasing the
active area of the detectors, most scattered photons are collected.
In FIG. 24(a) this is depicted by the second scattered photon path
282G being collected by its corresponding detector 120G. FIG. 24(b)
shows the expanded detection areas 226G can encompass the
scattering
[0139] In a further embodiment, changing the optical configuration
of the apertures and focal length of the lenses in the optical
system can create the same effect. Changing the aperture and the
focal length of the micro lens array, increasing the area of the
scan mirror, changing the aperture and the focal length of the
lenses L1, L2, L3 and L4 has a similar effect of reducing cross
talk without loss of signal. An example case is presented in
tabular format in FIG. W-22(a)-(e).
[0140] The optical configurations for two alternative embodiments
of the invention, Example A and Example B, are presented in FIGS.
25(a)-(e). FIG. 25(a) lists the objective lens specifications,
which are the same for both Examples A and B (i.e., Olympus, 180 mm
tube lens, XLUMPLFL 20.times. magnification objective; water
immersion; 0.95 numerical aperture; 17.1 mm back aperture; 2 mm
working distance; 9 mm focal length; 22 mm field number; and 1.1 mm
corrected field).
[0141] FIGS. 25(b)-(c) list the details of the illumination path. A
different micro lens array is described for each embodiment, but in
both the micro lenses are square shaped.
[0142] In Example A, the side aperture pitch of each micro lens is
1.06 mm and the diagonal is about 1.5 mm. For the entire array in
Example A the 8 lenses per side create a side aperture of 8.48 mm
and a diagonal aperture across the array of 11.99 mm. The focal
distance of each micro lens is 17 mm in Example A. In the
embodiment of Example B, the aperture or pitch of each micro lens
in the array is 1.9 mm and its focal distance is 25 mm.
[0143] The focal lengths of the lenses L1 and L4 are 50 mm and
103.77 mm, respectively, in Example A, while in Example B they are
40 mm and 46.32 mm, respectively. When standard optical components
are used, they can approximate components with a focal distance of
100 mm and 45 mm for L4 in the configurations A and B,
respectively. In both the embodiments of Example A and B, the focal
lengths of lenses L3 and L4 are 30 mm and 125 mm, respectively. The
diameter of illumination of the back aperture of the objective lens
for both Examples A and B remains approximately constant at 13.0 mm
and 12.7 mm, respectively. This results in an `under-illumination`
of the back-aperture of the objective lens which has a back
aperture of 17.1 mm in diameter. This is desirable, so an optimal
(maximal) employment of the illumination light power is
warranted.
[0144] As listed in FIG. 25(d), the two embodiments, Example A and
B, achieve different distances between excitation foci in the
optical plane: Example A has a foci distance of 46 microns, whereas
Example B has a foci distance of 103 microns. The total optical
field, listed below the foci distance, results from the fact that
in this particular case, an 8.times.8 configuration of foci is
chosen. It has a square side of 366 microns for configuration A and
821 microns for configuration B, when the foci are scanned. FIG.
25(e) lists the details of the detection path for each example.
These alternative examples, A and B, can be created according to
the layouts of either FIG. 7 or FIG. 8, according to embodiments of
the invention. In both cases the size of the MAPMT can remain
constant at about 2 mm.times.2 mm. Employing a MAPMT with larger
detection elements can increase the detection efficiency.
[0145] FIG. 26(a)and (b) illustrate the foci distribution in the
focal plane of the objective lens for the embodiment Examples A and
B, respectively. The conjugated detection area of each channel of
the multi anode PMT is by a factor of 5 larger in the Example B
than in Example A.
[0146] The optimal distance between the foci is influenced by three
factors: (1) the optimal number of foci that are needed to generate
as much light as possible (this number can be distinguished in
accordance with the graph in FIG. 16; (2) the corrected field of
the focusing device, such as an objective lens (the larger the
corrected field, the further the foci can be separated from each
other and the more the optical cross talk can be reduced); and (3)
the numerical aperture (NA) of the objective lens for high
resolution imaging (the larger the numerical aperture of the
objective lens, the more photons can be collected and the better
the images are). Nevertheless, there is a compromise between the NA
of the lens and its effective field of view. Therefore, the
objective lens used in a most preferred embodiment of the invention
has a large NA of around 1.0 or greater and is capable of imaging a
large effective field of view, preferably of approximately 1-6
mm.
[0147] At a fixed number of foci, a large field objective provides
an advantage for certain embodiments of the invention, because the
foci can be further separated. An objective lens with large field
of view enables large separation of foci and thus reduces optical
crosstalk. In FIG. 26(c) and (d) two objective lenses with
different fields of view are shown, 600 micron objective field
versus 6000 micron objective field, respectively. The conjugated
detection area of each channel of the multi anode PMT associated
with the focal plane within this field of view is a factor of 100
larger in the objective of FIG. 26(c) versus FIG. 26(d).
Commercially available objective lenses with a large numerical
aperture (NA) of around and above 1 usually have a field of view
for which they are corrected between (100.times. objectives) around
200 mm and (20.times. objectives) 1000 mm. With the Olympus
XLUMPLFL20.times. water immersion objective mentioned above and
used in embodiment Examples A and B, when an array of 8.times.8
foci is employed, the optimal distance between the foci is 111.11
microns and the total field imaged is approximately 1000
microns.
[0148] In the embodiments shown in FIGS. 7 and 8, the active
detection area of the different detector channels in the MAPMT is
approx. 2 mm and limited by the commercially available MAPMT
devices. If this area is increases, the optical cross talk and the
collection light efficiency is increased.
[0149] One embodiment of a method for data post-processing
according to the invention is illustrated in FIG. 27(a) and
provides for data post-processing starting at a step 310H. Step
310H can include initiating a computer program and/or software
application automatically as part of a data acquisition step in a
computer that is connected directly to the imaging apparatus and/or
can include a series of human-supervised data-processing steps. The
data processing can be automatically intiated by the computer and
proceed entirely automatically according to a data-processing
control software application and/or the program may proceed
semi-automatically with opportunities for human supervision and
intervention in one or more of the data processing steps. An
embodiment of the data-processing method follows the start step
310H with a next step to load the image data 312H. This can include
accessing raw data and metadata from storage devices, where
metadata (data about the data) includes, inter alia and without
limitation: foci number, pixel dimensions; pixel spacing; channels;
instrument parameters (including, without limitation, optics,
objective, illumination, wavelengths, beam-splitting, phasing,
polarity, light pumping, pulse compression, chirping, upconversion,
dispersion, diffraction, source-light properties, source light
stability, source attenuation, reference scanning, micro lens
configuration and properties, focal lengths, filter types and
positioning, detection configuration, detector type, detector
active area, detector sensitivity and stability, and other detector
specifications and properties, inter alia); sample properties and
sample information, such as, for example, for biological samples
(including biological and non-biological information, such as, for
example, tissue type, specimen type, size, weight, source, storage,
tracking, scattering properties, stain/dye type, specimen history,
and other physical properties of the specimen) or sample properties
and sample information for chemical and/or physical material
samples; and scanner data, including, without limitation, scanning
type, scanning mode, scan method, tracking, frequency, certainty,
precision, scan stability, resolution and other information about
the scanning. Data can be stored as XML, text, binary or in any
fashion, in electronic form and/or in retrievable and scannable
physical formats. In one preferred embodiment of the data
post-processing method, the next step 314H is deconvolution of the
image data, which deconvolution is described further below.
Following this, the data can be saved in an optional step 316H,
whereupon the post-processing can optionally be stopped 318H. An
embodiment also allows the processing to continue to a next step
320H that comprises performing an intensity normalization on the
data, which normalization steps are described in more detail below,
then optionally saving the data (step 322H), and stopping the data
processing sequence (step 324H).
[0150] Referring to FIG. 27(b), another embodiment of the invention
provides for additional and/or alternative method(s) for data
post-processing, In one embodiment, the processing Alternatively,
an embodiment of the method The post-processing steps can include a
number of substeps. Step 332H can include accessing metadata from a
storage device, including here by reference all the description of
possible metadata described above for the steps illustrated in FIG.
27(a). Step 334H can include normalizing, filtering (de-noising),
and blending (integrating) of multifoci subimages, and further can
include registering subimages into a single image. Step 336H can
include filtering and normalizing images produced from corrected
subimages. Step 338H can include registering, building mosaics, and
blending sets of corrected images into a larger whole. Also,
optionally, at this step 338H, an embodiment of the method of the
invention provides for creating lower resolution images of the
larger image to facilitate access, as well as images from different
perspectives (such as, image views taken of the xy-, xz-, and/or
yz-planes) and creating data-compressed versions of the data and/or
results (e.g., JPEG, wavelet compression, inter alia without
limitation). Step 340H can include segmenting images into objects,
which segmentation step can either be manual, automated or a
combination of both. Step 342H can include parameterizing the
objects, samples or specimens (such as, for example, size, shape,
spectral signature). Step 344H can include classifying objects into
higher order structures/features (e.g. material stress or cracks,
vasculature, nuclei, cell boundaries, extra-cellular matrix, and
location, inter alia, without limitation). Step 346H can include
statistically analyzing parameterized objects (such as, for
example, by correlation methods, principal component analysis,
hierarchical clustering, SVMs, neural net classification, and/or
other methods). Step 332H can include presenting results to one or
more persons on one or more local or distant display devices
(examples include: 3D/2D images, annotated images, histograms,
cluster plots, overlay images, and color coded images, inter
alia).
[0151] The post-processing steps can include a number of substeps,
including, among other steps those illustrated in FIG. 27(b),
without limitation:
[0152] i) after data access 332H normalizing, filtering
(de-noising), blending (integrating) of multifoci subimages
334H;
[0153] ii) Registering subimages into a single image;
[0154] iii) Filtering and normalizing images produced from
corrected subimages 336H;
[0155] iv) Registering, building mosaics, and blending sets of
corrected images into a larger whole 338H;
[0156] v) Optionally, at this stage, lower resolution images can be
created of the larger image to facilitate access, as well as images
from different perspectives (xy, xz, yz).
[0157] vi) Data-compressed versions (e.g., JPEG, wavelet
compression, inter alia without limitation) can be produced;
[0158] vii) Segmenting images into objects 340H This segmentation
can either be manual, automated or a combination of both.
[0159] viii) Parameterizing the objects 342H (for instance, size,
shape, spectral signature).
[0160] ix) Classifying objects into higher order
structures/features 344H (e.g. material stress or cracks,
vasculature, nuclei, cell boundaries, extra-cellular matrix, and
location, inter alia, without limitation)
[0161] x) Statistically analyzing parameterized objects 346H (e.g.,
by correlation or other methods).
[0162] xi) Presenting results to user on display device 348H
(examples include: 3D/2D images, annotated images, histograms,
cluster plots, overlay images, and color coded images).
[0163] FIGS. 28(a) and(b) relate to image normalization. The multi
foci power map of the MMM when a micro lens array is implemented.
The numbers resemble the power in each foci in the sample. Due to
the Gaussian beam profile in an example, 51.2 mW are
inhomogeneously spread over the 36 foci. 24.1 mW contribute to the
foci lying beyond the 6.times.6 foci matrix and are thus lost.
[0164] The normalized signal distribution resembles the normalized
power map squared and shows an intensity drop of 45% toward the
corner PMTs in respect to the center PMTs. The laser power was
attenuated to 75.3 mW in the sample and can reach a maximal value
of approx. 645 mW, resembling a power of approx. 15 mW for the
center foci in the sample. Measured intensity profile can be
generated by imaging a homogeneously distributed fluorescent dye
under a cover slip. The intensity measurement is not only mapped by
the power/intensity distribution of the foci, but also by the
sensitivity of the detector array. As a result it resembles the
"true" measured intensity distribution. The image consists of
192.times.192 pixels and was generated by an array of 6.times.6
foci which were scanned across a uniform fluorescent dye
sample.
2D xy Image Normalization is Carried Out in Different Ways:
[0165] Case 1: The normalized inverse of this intensity image (from
a uniform fluorescent dye) is multiplied with the yx images taken
of the sample. The resulting images are then displayed and saved as
a normalized image.
[0166] Case 2: A large number of images from a sample at various
positions (and thus with a random underlying intensity structure)
is averaged. This image is then inversed and normalized. This image
is multiplied with the original data is then displayed and saved as
a normalized image.
[0167] Case 3: A simplified image is generated which consists of 36
sub-images (generated by the 6.times.6 foci). Each of the
sub-images carries the average intensity generated by the specific
foci. For example, all 32.times.32 pixels in the top left sub image
carry the same number; 45. The image is then inversed and
normalized. This image multiplied with the original data is then
displayed and saved as a normalized image. An image can be
generated either from the intensity image generated by the process
of case 1 (fluorescent image) or case 2 (over many images
averaged). 3D xyz image normalization is carried out in a similar
fashion as in case 2 of the xy image normalization. A z-intensity
profile (an example is FIG. 28b) is generated by averaging the
intensity signal the xy planes form different positions in z. As
the penetration depth increases, the average intensity decreases
along the z-axis. In order to get a good average intensity for the
z-intensity profile, images from a sample at various positions (and
thus with a random underlying intensity structure) are averaged.
This z-intensity profile is then inversed and normalized. Each
image plane is then multiplied with the according normalization
number generated by this process.
[0168] A method for multifocal multiphoton imaging of a specimen in
accordance with a preferred embodiment of the present
invention:
[0169] (0) Start
[0170] (1) Sample pre processing (optional)
[0171] (2) place the sample in the region of focus of the focusing
device (objective lens)
[0172] (3) determine imaging parameters
[0173] (4) set imaging parameters
[0174] (5) image
[0175] (6) Process images for feedback purposes (optional)
[0176] (7) display the images (optional)
[0177] (8) save the data
[0178] (9) process the data (optional)
[0179] (10) save the processed data (optional)
[0180] (11) display the processed data (optional)
[0181] Concerning the order of the steps, (1) and (2) can be
switched: (4), (5) and (6) can be switched.
[0182] In more detail these are
(1) Sample Pre Processing
[0183] Apply tissue staining
[0184] Apply optical clearing agents
(2) Place the Sample in the Region of Focus of the Focusing Device
(Objective Lens):
[0185] Determine desired sample region
[0186] Place the sample in the region of focus of the focusing
device (objective lens): or
[0187] Place the region of focus of the focusing device (objective
lens) on to the sample
(3) Determine Imaging Parameters
[0188] Determine imaging parameters by which include variable or
multiple values for each of the parameters [0189] measurements
performed manually or in an automated fashion outside or within the
imaging procedure comprising [0190] the region of interest [0191]
Sample shape [0192] Sample margins [0193] Emission intensity [0194]
Emission wavelength [0195] Emission polarization [0196] . . . *
[0197] sample [0198] type [0199] photons mean free path [0200]
power threshold [0201] sample fixation [0202] sample labeling
[0203] . . . * [0204] illumination [0205] detector [0206] . . . *
[0207] The imaging parameters comprise [0208] illumination
wavelength [0209] illumination power [0210] illumination
polarization [0211] scanning speed [0212] maximal penetration depth
[0213] sampling [0214] . . . * (4) Set Imaging parameters
[0215] A computer program is fed with the imaging potentially
dynamically adjustable (including feedback from the measurement)
parameters and controls the imaging procedure.
[0216] (5) image
Point measurement.
1D Scan: Collect Data from a Region of Interest
2D Scan:
[0217] Image a 2D region of interest by scanning the foci in
parallel across the imaging plane (XY)
[0218] the 2D scanning starts for example at a corner of an area
and is then scanned in a raster until the area is covered according
to the imaging parameters (current implementation)
[0219] or it can be scanned in any other way (even random scan is
allowed), as long as the area is covered according to the imaging
parameters and the position of the foci is known by the signal sent
or received by the scanner.
[0220] A key consideration for the improvement in the measurement
is that the detector measures the sample for each scanning
position, without overlap.
3D Scan:
[0221] move the focusing device (objective lens) in reference to
the sample along the optical axis (Z) and repeat the 2D imaging
process. Either, the focusing device (objective lens) or the sample
can be moved. Right now, the focusing device (objective lens) is
moved stepwise in regards to the fixed sample.
[0222] 2D imaging along the optical axis (Z) can begin at any point
in the sample and end at any point of the sample within the region
of interest.
[0223] the current movement is though depending on the application:
[0224] (a) begin scanning a 2D scan of the top layer of the tissue
sample, or [0225] begin scanning a little outside the sample, to be
able to determine the top, or [0226] begin scanning within the
sample, to prevent beam-sample interactions (like burning or such
thing), which take place at the surface-immersion medium barrier.
[0227] (b) then move the focusing device (objective lens) by means
of a piezo in the direction of the sample in increments determined
by the imaging parameters [0228] start at (a) again [0229] Stop at
a point in the sample, which is determined by the imaging parameter
[0230] Move the piezo back to its original position [0231] Move the
sample to a different position and start with (a) again (area
imaging)
[0232] For some applications it is preferable, if the movement can
be reversed (starting inside of the sample and then move out), or
performed in a random, fashion, covering the whole area, as long as
the z-position is known.
[0233] The z position of the foci is known as the piezo position is
known
[0234] The 2D scanning is done while the z-scan from one position
to the next takes place or after the z-scan has completed its move
to the next position.
[0235] Images of 2D sections can be done alone, without any 3D
movement involved.
(6) Process Images for Feedback Purposes (Optional)
(7) Display the Images (Optional)
(8) Save the Data
[0236] Process data before saving (Optionally)
(9) Post Process the Data (Optional)
[0237] Image normalization
[0238] linear image deconvolution
[0239] nonlinear image deconvolution
(10) Save the Processed Data (Optional)
(11) Display the Processed Data (Optional)
[0240] In addition to mechanistic applications, time-resolved
measurements, either alone or in conjunction with spectral
measurements, can greatly aid in distinguishing signals from
different reporter probes and processes, such as simple scattering
and non-linear scattering. For cytometry applications, the
additional information from time-resolved measurements can
potentially increase the number of probes which can be used
simultaneously, provide images cell morphology by detection of
second harmonic generation, and aid in deconvolution of images from
highly scattering samples.
[0241] FIGS. 29(a) and 29(c) relate to a deconvolution process. In
FIG. 29(a) Illumination foci in the optical plane. (foci f11-f33
are illustrated in an enlarged) along with an object, illuminated
by foci f22. In FIG. 29(b) the detection signals are scattered
along with the detection areas a11-a33. FIG. 29(c) are example of
signal counts detected by the associated channels of the multi
channel detector (signal from area all is collected by the detector
channel c11; a12 by c12 and so on) at a certain time point, when
the focus f1 scans the center of the object (a). The relative
signal distribution between the channels is dependent on the mean
free path of the detection photons in the media and the penetration
depth. It is constant however, if a homogeneous scattering
distribution is assumed (For many samples this can de assumed in
the first approximation). Low mean free path means highly
scattering, means higher amount of signal in other than channel
c22. As the penetration depth into the sample increases, the
chance, that a photon is scattered on its way to the detector array
increases and thus the described "optical cross talk" increases as
well. More scattered light is found in the channels, neighboring
the outer channels c11, c12, c13, c21, c23, c31, c32 and c33. The
signals in these detection elements are smaller though and are not
illustrated for simplicity.
[0242] FIGS. 30(a)-30(d) display a 1 dimensional (1D) deconvolution
exemplifying the final 2D deconvolution executed in the linear
image deconvolution. For simplification only nearest neighbors are
shown. A linear convolution with a delta function with inversed
side lobes (FIG. 30(c) (Illustration only in along one channel
number direction) results in a linearly de-convolved image in which
only the channel c22 carries a signal. In practice, this function
can either be modeled or measured. If the deconvolution process is
shown for simplicity only in x direction. It will be carried out in
both x and y directions, and will result in an image in which only
channel c22 will carry a signal.
[0243] Assuming a homogeneously scattering material (which can be
assumed for samples in the first approximation), the relative and
absolute height of the peaks of the delta function is fixed for
every channel at its neighbors at a certain imaging depth into the
sample. As a result, xy images can be linearly de-convoluted.
[0244] The linear deconvolution of cross-talk is primarily a 2D
process. The values of the weighting matrix depend on several
factors. The optical contribution to the cross talk increases with
increasing penetration depth. Furthermore, the channels have
different sensitivities, there is electronic cross talk between
channels that varies from channel to channel and other factors
influence the amount of total cross talk between the channels.
[0245] The cross talk for each individual channel can be determined
experimentally. An example is where, one focus illuminates the
sample or a test object and the whole array of detectors detects
the signal. At different penetration depths a cross talk matrix is
measured for each channel. This matrix is then used to carry out
the deconvolution. Data of such a measurement at the sample surface
and at a penetration depth of 200 mm, can be used. The measurement
is repeated for every channel for example by moving the iris from
transmitting light from one single micro lens to the next (in this
case for channels c11 to c33). Similar alternative methods are also
possible, for example by illuminating with all of the foci but
using a sample with large object spacing. Furthermore, models can
also replace experimental determination.
[0246] An entire 2D image consists of collections of ensembles for
a non linear deconvolution, of pixels, from each detector. The key
point is that relationships between entire ensembles, and certain
regions of pixels between ensembles, can be established to
constrain the variation of the weighting matrix to aid convergence
without assumptions, or with minimal assumptions, of the sample or
the processes which cause the variation in the weighting
matrix.
[0247] For example, continuity of the values across the boundaries
of the ensemble can be generally required. In the case with the
minimal assumption that the objects under observation are smaller
than the region covered by an individual ensemble, the ensembles
can be considered largely independent, except due to the cross-talk
introduced by the weighting matrix. The ideal image can be
recovered by simultaneously solving for a weighting matrix which
minimizes the covariance between ensembles. In the other case where
the objects under observation are of similar size or larger than
the regions covered by the ensembles, minimal models of the object
(such as from image morphology or segmentation of the collected
image, etc . . . ) can be used to form constraints.
[0248] Additional model dependent and independent constraints can
also be applied by consideration of the planes above and below the
plane under evaluation. Further constraints can also be applied to
the weighting matrix from either general (such as continuation,
smoothness, sharpness, etc . . . ) or model based considerations
.
[0249] While this invention has been particularly shown and
described with reference to preferred embodiments thereof, it will
be understood by those skilled in the art that various changes in
form and details may be made therein without departing from the
scope of the invention encompassed by the appended claims.
* * * * *