U.S. patent application number 09/758037 was filed with the patent office on 2002-07-11 for system and method for finding regions of interest for microscopic digital montage imaging.
Invention is credited to Beckstead, Jeffrey A., Feineigle, Patricia A., Gilbertson, John R. II, Hauser, Christopher R., Palmieri, Frank A. JR., Wetzel, Arthur W..
Application Number | 20020090120 09/758037 |
Document ID | / |
Family ID | 25050222 |
Filed Date | 2002-07-11 |
United States Patent
Application |
20020090120 |
Kind Code |
A1 |
Wetzel, Arthur W. ; et
al. |
July 11, 2002 |
System and method for finding regions of interest for microscopic
digital montage imaging
Abstract
A system for processing a thumbnail image from a microscope
slide to determine tissue locations on the slide. The system
comprises an image cropping component, a tissue finding component,
and a scan control component. The image cropping component crops
the thumbnail image and removes portions of the image that fall
outside of determined slide boundaries. The cropped image from the
image cropping component is inputted into the tissue finding
component. The tissue finding component identifies tissue regions
by applying a sequence of filters that incorporate knowledge of
typical appearance and location of tissue and non-tissue slide
regions. The tissue finding component outputs a tiling matrix whose
values indicate which tiles should be imaged. The scan control
component interprets the tiling matrix and transposes positions of
the tiling matrix into actual stage coordinate for a microscopic
imaging.
Inventors: |
Wetzel, Arthur W.;
(Murrysville, PA) ; Gilbertson, John R. II;
(Pittsburgh, PA) ; Beckstead, Jeffrey A.;
(Valencia, PA) ; Feineigle, Patricia A.;
(Pittsburgh, PA) ; Hauser, Christopher R.;
(Pittsburgh, PA) ; Palmieri, Frank A. JR.;
(Gibsonia, PA) |
Correspondence
Address: |
MORGAN LEWIS & BOCKIUS LLP
1111 PENNSYLVANIA AVENUE NW
WASHINGTON
DC
20004
US
|
Family ID: |
25050222 |
Appl. No.: |
09/758037 |
Filed: |
January 11, 2001 |
Current U.S.
Class: |
382/128 |
Current CPC
Class: |
G06V 20/693 20220101;
G06V 20/69 20220101; G06V 10/25 20220101 |
Class at
Publication: |
382/128 |
International
Class: |
G06K 009/00 |
Claims
What is claimed:
1. A system for processing a thumbnail image from a microscope
slide to determine tissue locations on the slide, the system
comprises: a plurality of first components for identifying tissue
regions and determining locations of tissue on the slide; a
motorized stage where a frame image containing the entire slide is
taken with a camera; and means for using information about the
locations to generate control parameters for the motorized stage
and the camera to ensure that a scanning process captures high
quality images of only the tissue regions.
2. The system of claim 1, wherein the plurality of first components
include an image cropping component for identifying tissue regions
on the slide to be scanned, wherein the image cropping component:
determines an approximate location of a slide boundary by searching
upper and lower intervals corresponding to regions expected to
contain upper and lower edges of the slide; re-examines the
approximate location to determine a more accurate location; and
removes portions of the image falling outside of the determined
slide boundary.
3. The system of claim 2, wherein the image cropping component
converts a copy of the thumbnail image to a grayscale image.
4. The system of claim 2, wherein the image cropping component
crops a color thumbnail image at estimated edge locations, wherein
multiple values of each pixel in the color thumbnail image are used
to achieve better results by identifying spurious features on the
slide.
5. The system of claim 4, wherein the image cropping component
crops the color thumbnail image at estimated boundary locations and
uniformly reduces the color thumbnail image size to produce a small
thumbnail image of the slide for rapid visual slide
identification.
6. The system of claim 3, wherein the image cropping component
identifies pixel blocks that are likely to contain remaining
boundary edges and flags these blocks as edges that should not be
considered for high-resolution imaging.
7. The system of claim 1, wherein the plurality of first components
include a tissue finding component that locates regions in the
thumbnail image that contain tissue of interest to a
specialist.
8. The system of claim 7, wherein a cropped grayscale image is
inputted into the tissue finding component from a imaging cropping
component, wherein the tissue finding component identifies tissue
regions by applying a sequence of filters that incorporate
knowledge of typical appearance and location of tissue and
non-tissue slide regions and outputs a tiling matrix whose values
indicate which tiles should be imaged.
9. The system of claim 8, wherein a first filter analyzes mean and
standard deviation of local pixel intensities and combines the mean
and the standard deviation to generate a threshold value for making
an initial classification of tissue versus non-tissue regions.
10. The system of claim 9, wherein the intensities are used to
differentiate tissue-containing regions from blank regions and
other non-tissue containing regions.
11. The system of claim 9, wherein the standard deviation
represents the amount of variation in pixel values and is a good
indicator of the border between tissue and the blank slide.
12. The system of claim 8, wherein morphological filters are
applied to threshold standard deviation data to refine
classification based on size and position of neighboring groups of
potential tissue pixels, wherein the morphological filters process
pixels of the cropped grayscale image in groups that correspond to
slide regions that can be imaged individually during a
high-resolution scanning process.
13. The system of claim 12, wherein the morphological filters
ensure that tiles that partially filled with tissue are classified
as tissue-containing tiles.
14. The system of claim 8, wherein other image characteristics can
be used to identify tissue from non-items of interest.
15. The system of claim 1, wherein the frame image is a single
macroscopic image.
16. The system of claim 1, wherein the frame image is multiple
macroscopic images.
17. The system of claim 1, wherein the plurality of first
components includes a scan control component that interprets a tile
matrix, outputted by a find tissue component, and transposes
positions of the tile matrix into actual stage coordinates for a
microscopic imaging.
18. A method for processing a thumbnail image from a microscope
slide to determine tissue locations on the slide, the method
comprises the steps of: flat-field correcting the thumbnail image
using a blank slide and a similar image obtained from a camera that
captured the thumbnail image; cropping the thumbnail image by an
image cropping component; inputting a cropped grayscale image into
a tissue finding component, wherein the tissue finding component
identifies tissue regions by applying a sequence of filters that
incorporate knowledge of typical appearance and location of tissue
and non-tissue slide regions and outputs a tiling matrix whose
values indicate which tiles should be imaged; and interpreting the
tiling matrix, by a scan control component, and transposing
positions of the tiling matrix into actual stage coordinate for a
microscopic imaging.
19. The method of claim 18, wherein the step of cropping further
comprises the steps of: determining an approximate location of a
slide boundary by searching upper and lower intervals corresponding
to regions expected to contain upper and lower edges of the slide;
re-examining the approximate location to determine a more accurate
location; and removing portions of the image falling outside of the
determined slide boundary.
20. The method of claim 19, further comprising the step of
converting a copy of the thumbnail image to a grayscale image.
21. The method of claim 19, further comprising the step of cropping
a color thumbnail image at estimated edge locations, wherein
multiple values of each pixel in the color thumbnail image are used
to achieve better results by identifying spurious features on the
slide.
22. The method of claim 19, further comprising the step of cropping
the color thumbnail image at estimated boundary locations and
uniformly reducing the color thumbnail image size to produce a
small thumbnail image of the slide for rapid visual slide
identification.
23. The method of claim 20, further comprising the steps of
identifying pixel blocks that are likely to contain remaining
boundary edges and flagging these blocks as edges that should not
be considered for high resolution imaging.
24. The method of claim 18, further comprising the steps of
analyzing mean and standard deviation of local pixel intensities
and combining the mean and the standard deviation to generate a
threshold value.
25. The method of claim 24, further comprising the step of using
the intensities to differentiate tissue-containing regions from
blank regions and other nontissue containing regions.
26. The method of claim 18, further comprising the step of applying
morphological filters to threshold standard deviation data to
refine classification based on size and position of neighboring
groups of potential tissue pixels, whereby the morphological
filters process pixels of the cropped grayscale image in groups
that correspond to slide regions that can be imaged individually
during a high-resolution scanning process.
27. A system for processing a thumbnail image from a microscope
slide to determine tissue locations on the slide, the system
comprises: an image cropping component for cropping the thumbnail
image; a tissue finding component that identifies tissue regions by
applying a sequence of filters that incorporate knowledge of
typical appearance and location of tissue and non-tissue slide
regions and outputs a tiling matrix whose values indicate which
tiles should be imaged; a scan control component for interpreting
the tiling matrix; and means for inputting a cropped image from the
image cropping component into the tissue finding component and for
transposing positions of the tiling matrix into actual stage
coordinate, by the scan control component, for a microscopic
imaging.
Description
FIELD OF THE INVENTION
[0001] The present invention relates to microscopic digital imaging
of complete tissue sections for medical and research use. In
particular it describes a method to find regions of interest for
high throughput montage imaging of microscope slides using a
standard microscope and cameras.
BACKGROUND OF THE INVENTION
[0002] Laboratories in many biomedical specialties, such as
anatomic pathology, hematology, and microbiology, examine tissue
under a microscope for the presence and the nature of disease. In
recent years, these laboratories have shown a growing interest in
microscopic digital imaging as an adjunct to direct visual
examination. Digital imaging has a number of advantages including
the ability to document disease, share findings, collaborate (as in
telemedicine), and analyze morphologic findings by computer. Though
numerous studies have shown that digital image quality is
acceptable for most clinical and research use, some aspects of
microscopic digital imaging are limited in application.
[0003] Perhaps the most important limitation to microscopic digital
imaging is a "subsampling" problem encountered in all single frame
images. The sub-sampling problem has two components: a field of
view problem and a resolution-based problem. The field of view
problem occurs when an investigator looking at a single frame
cannot determine what lies outside the view of an image on a slide.
The resolution-based problem occurs when the investigator looking
at an image is limited to the resolution of the image. The
investigator cannot "zoom in" for a closer examination or "zoom
out" for a bird's eye view. Significantly, the field of view and
resolution-based problems are inversely related. Thus, as one
increases magnification to improve resolution, one decreases the
field of view. For example, as a general rule, increasing
magnification by a factor of two decreases the field of view by a
factor of four.
[0004] To get around the limitations of single frame imaging,
developers have looked at two general options. The first option
takes the general form of "dynamic-robotic" imaging, in which a
video camera on the microscope transmits close to real time images
to the investigator looking at a monitor, while the investigator
operates the microscope by remote control. Though such systems have
been used successfully for telepathology, they do not lend
themselves to documentation, collaboration, or computer based
analysis.
[0005] The second option being investigated to overcome the
limitations inherit in single frame imaging is a montage (or
"virtual slide") approach. In this method, a robotic microscope
systematically scans the entire slide, taking an image at every
field. The individual images are then "knitted" together in a
software application to form a very large data set with very
appealing properties. The robotic microscope can span the entire
slide area at a resolution limited only by the power of the optical
system and camera. Software exists to display this data set at any
resolution on a computer screen, allowing the user to zoom in, zoom
out, and pan around the data set as if using a physical microscope.
The data set can be stored for documentation, shared over the
Internet, or analyzed by computer programs.
[0006] The "virtual slide" option has some limitations, however.
One of the limitations is file size. For an average tissue section,
the data generated at 0.33 um/pixel can be between two and five
gigabytes uncompressed. In an extreme case, the data generated from
one slide can be up to thirty-six gigabytes.
[0007] A much more difficult limitation with the prior systems is
an image capture time problem. Given an optical primary
magnification of twenty and a two-third inch CCD, the system field
of view is approximately (8.8 mm.times.6.6 mm)/20=0.44.times.0.33
mm. A standard microscope slide typically has a specimen area of 25
mm.times.50 mm or 12.5 square centimeters. This requires over
eighty-six hundred fields to image this entire specimen region.
However, the average tissue section for anatomic pathology is
approximately 2.25 square centimeters. This only requires
approximately fifteen hundred fields to cover the tissue alone,
approximately 80 percent less fields.
[0008] Traditionally, field rate in montage systems is limited by
three factors-camera frame rate, image processing speed, and the
rate of slide motion between fields. Given today's technology, the
limiting factor can be reduced to only the camera frame rate. Using
a 10 frame per second camera for the example above, imaging the
entire slide would require 860 seconds or 14.33 minutes. If only
the region of interest was imaged, this average time could be
reduced to 150 seconds or 2.5 minutes; substantially increasing the
slide throughput of an imaging system.
[0009] Thus, a system is needed to automatically find the region of
interest on a microscope slide and image only this region.
SUMMARY OF THE INVENTION
[0010] The present invention relates to a method and system for
processing a thumbnail image from a microscope slide to determine
tissue locations on the slide. The system comprises an image
cropping component, a tissue finding component, and a scan control
component. The image cropping component crops the thumbnail image
and removes portions of the image that fall outside of determined
slide boundaries. The cropped image from the image cropping
component is inputted into the tissue finding component. The tissue
finding component identifies tissue regions by applying a sequence
of filters that incorporate knowledge of typical appearance and
location of tissue and non-tissue slide regions. The tissue finding
component outputs a tiling matrix whose values indicate which tiles
should be imaged. The scan control component interprets the tiling
matrix and transposes positions of the tiling matrix into actual
stage coordinate for a microscopic imaging.
[0011] Accordingly, it is an object of the invention to provide a
microscopic imaging system for whole slide montage in which
standard microscope optics, off the shelf cameras and a simple
motorized stage can be used to select the region of interest, image
only this section and produce perfectly aligned image tiles.
[0012] The present invention uses a pre-scan process applied to a
macroscopic image of the entire slide, to guide a high-resolution
slide scanning process and ensure high-quality images of the entire
specimen are acquired. The pre-scan process includes an image
cropping component, a tissue-finding component, and a scan control
component. The image cropping and tissue finding components
identify interesting regions on the slide to be scanned. The scan
control component generates the control parameters for a motorized
microscopic imaging system.
[0013] It is another object of the invention to use a
high-resolution slide scanning process to control the operation of
the motorized stage and camera. This process utilizes information
gathered by the pre-scan process, namely the imaging regions, to
control the positioning of the stage to image only the regions of
interest and to ensure the individual images are well aligned.
[0014] Additional features and advantages of the invention will be
set forth in the description that follows, and in part will be
apparent from the description, or may be learned by practice of the
invention. The objectives and advantages of the invention to be
realized and attained by the microscopic image capture system will
be pointed out in the written description and claims hereof as well
as the appended drawings.
BRIEF DESCRIPTION OF THE DRAWINGS
[0015] The accompanying drawings, which are included to provide a
further understanding of the invention and are incorporated in and
constitute a part of this specification, illustrate embodiments of
the invention that together with the description serve to explain
the principles of the invention.
[0016] FIG. 1 illustrates an isometric view of the system in a
preferred embodiment;
[0017] FIG. 2 represents the results of the macroscopic image after
the cropping component has been applied to remove non-slide
regions;
[0018] FIG. 3 represents the results of the find tissue component;
and
[0019] FIG. 4 is an overlay of FIG. 2 and 3 representing the
regions of the slide to be imaged.
DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0020] Reference will now be made in detail to the preferred
embodiments of the present invention, examples of which are
illustrated in the accompanying drawings. The following paragraphs
describe the functionality of the inventive system and method for
high throughput montage imaging of microscope slides using a
standard microscope and cameras.
[0021] FIG. 1 illustrates a preferred embodiment of the invention.
In this embodiment, a slide 112 to be imaged is placed on a
thumbnail imaging position in a slide holder on a motorized stage
102. A single frame image containing the entire slide is taken with
a macro camera 106. This low-resolution image is analyzed by
software components to determine the locations of tissue on slide
112. This information can then be used to generate control
parameters for stage 102 and microscopic camera 104 to ensure that
the scanning process captures high quality images of only the
tissue regions, substantially reducing the time to scan an average
slide.
[0022] As is obvious to one skilled in the art, although capturing
the single macroscopic image saves time, it is not necessary for
the operation of the invention. Multiple macroscopic images may be
required to generate control parameters to the accuracy required
based on the ratio of the macroscopic to microscopic magnifications
and the camera specifications of each camera, if separate cameras
are utilized.
[0023] Specifically in a preferred embodiment, a pre-scan
processing of the low-resolution or thumbnail image includes an
image cropping component, a tissue-finding component and a scan
control component. The image cropping component and tissue finding
component identify tissue regions on the slide to be scanned. The
scan control component generates the necessary control parameters
to scan only the regions of interest under the microscopic
optics.
[0024] The first step in processing the thumbnail image consists of
flat-field correcting the macroscopic thumbnail image using a
similar image obtained from the same camera and a blank slide. This
removes any spatial light anomalies from the thumbnail image, which
may reduce the efficiency of the tissue-finding component. Given
the format, or size, of the camera and the aspect ratio of the
slide, a portion of the image will contain non-slide objects such
as the slide carrier. To remove these features, the thumbnail image
is cropped to extract only the slide information.
[0025] The image cropping is accomplished via a two-pass process.
The first pass determines an approximate location of the slide
boundary, and the second pass fine-tunes this estimate. The search
for the boundary is conducted over upper and lower intervals
corresponding to the regions expected to contain the upper and
lower slide edges, respectively. For this discussion, the slide or
region of interest is assumed to be positioned near the center,
vertically, in the thumbnail image. To facilitate this and
subsequent processing steps, a copy of the thumbnail image is
converted to grayscale. The portion of the image falling outside of
the identified slide boundary is removed. It should be noted that
the original color image is also cropped at the estimated edge
locations, and then is uniformly reduced in size to produce a small
thumbnail image of the slide for rapid, visual slide
identification.
[0026] Since the slide may not be oriented perfectly horizontal in
the original thumbnail image, the identified slide edges are likely
to lie at an angle. Thus, even after cropping, there may be
remnants of the slide edges or cover slip in the cropped image.
Therefore, the image-cropping component attempts to identify pixel
blocks that likely contain these remaining edges and flags these
blocks as edges that will not be considered for high resolution
imaging by the tissue finding component.
[0027] The resulting cropped grayscale image generated by the
image-cropping component serves as input to the tissue finding
component. This component locates regions in the thumbnail image
that contain tissue of interest to a specialist. In order to
minimize the time and storage space required to accomplish
high-resolution slide imaging, the inventive system captures only
those regions of the slide that contain tissue. This approach
requires that regions containing tissue be identified in the
thumbnail image.
[0028] The tissue finding component identifies tissue regions via a
sequence of filters that incorporate knowledge of the typical
appearance and location of tissue and nontissue slide regions.
Initial filtering steps analyze the mean and standard deviation of
the local pixel intensities. Pixel mean intensities are used to
differentiate tissue-containing regions from blank and other
non-tissue regions, such as those containing the slide label or
other markings. The standard deviation data represents the amount
of variation in pixel values and thus is a good indicator of the
border between tissue and the blank slide. The mean and standard
deviation data is combined to generate a threshold value that is
used to make an initial classification of tissue versus non-tissue.
Subsequently, morphological filters are applied to refine the
classification based on the size and position of neighboring groups
of potential tissue pixels.
[0029] The filters which comprise the tissue finding component
process the pixels of the cropped grayscale thumbnail image in
groups that correspond to slide regions, or tiles, that can be
imaged individually during the high-resolution scanning process.
These filters ensure that tiles only partially filled with tissue
are classified as tissue-containing tiles. The final output of the
filter sequence is a tiling matrix whose values indicate which
tiles should be imaged; the tiling matrix subsequently guides the
high-resolution scanning process.
[0030] The above description was based on using the mean and
standard deviation of the local pixels as the basis for detecting
regions of interest. It is obvious to one skill in the art that
other image characteristics can be also used to identify the
specimen from nonitems of interest such as dust and scratches.
[0031] This description was also based on processing a gray scale
macroscopic image, the same processing tools can be applied to each
of the color components (traditionally, red, green and blue) of a
color image. Additional processing tools can also be applied
between the color components to refine the tissue finding accuracy
and to remove features such as labels and writing that are not
critical to the application.
[0032] An example of the image cropping and find tissue processing
are shown in FIG. 2 3 and 4. FIG. 2 illustrates the macroscopic
image after flat-field correction and image cropping. FIG. 3
illustrates the results of the find tissue component. The resulting
tile matrix shown in FIG. 3 has a one-to-one correspondence to the
field of view of the microscopic camera. White pixels (binary 1)
signify field to be capture and black pixels represent regions not
to image. FIG. 4 illustrates an overlay FIG. 2 and 3 representing
the sections of the slide to be imaged. For this application
(anatomical pathology), it is imperative to image all suspect
regions that may contain tissue so conservative criteria were used
in the find tissue component, resulting in cover slip edges and
writing etched into the slide to be identified as to be imaged. The
savings in the acquisition time is representative by the ratio of
the white to black areas of FIG. 3. For this image, only 53% of the
slide region is to be imaged, including the label and cover slip
edges, and etched writing on the slide.
[0033] At the completion of the find tissue component, the scan
control component interprets the find tissue tile matrix (FIG. 3)
and transposes the positions into actual stage coordinates for the
microscopic imaging. A program running on a host computer controls
the operation by communicating with a stage controller and
microscopic camera 104. Actual scanning can occur in any fashion
such as by rows or columns, or in a step fashion to image
neighboring areas.
[0034] The foregoing description has been directed to specific
embodiments of this invention. It will be apparent, however, that
other variations and modifications may be made to the described
embodiments, with the attainment of some or all of their
advantages. Therefore, it is the object of the appended claims to
cover all such variations and modifications as come within the true
spirit and scope of the invention.
* * * * *