U.S. patent application number 12/313015 was filed with the patent office on 2009-10-22 for pathological tissue mapping.
This patent application is currently assigned to Aureon Laboratories, Inc.. Invention is credited to Angeliki Kotsianti, Olivier Saidi, Mikhail Teverovskiy.
Application Number | 20090262993 12/313015 |
Document ID | / |
Family ID | 34619517 |
Filed Date | 2009-10-22 |
United States Patent
Application |
20090262993 |
Kind Code |
A1 |
Kotsianti; Angeliki ; et
al. |
October 22, 2009 |
Pathological tissue mapping
Abstract
Embodiments of the present invention are directed to
quantitative analysis of tissues enabling the measurement of
objects and parameters of objects found in images of tissues
including perimeter, area, and other metrics of such objects.
Measurement results may be input into a relational database where
they can be statistically analyzed and compared across studies. The
measurement results may be used to create a pathological tissue map
of a tissue image, to allow a pathologist to determine a
pathological condition of the imaged tissue more quickly.
Inventors: |
Kotsianti; Angeliki; (New
York, NY) ; Saidi; Olivier; (Greenwich, CT) ;
Teverovskiy; Mikhail; (Harrison, NY) |
Correspondence
Address: |
MINTZ LEVIN COHN FERRIS GLOVSKY & POPEO
ONE FINANCIAL CENTER
BOSTON
MA
02111
US
|
Assignee: |
Aureon Laboratories, Inc.
Yonkers
NY
|
Family ID: |
34619517 |
Appl. No.: |
12/313015 |
Filed: |
November 14, 2008 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
10991897 |
Nov 17, 2004 |
7483554 |
|
|
12313015 |
|
|
|
|
60520815 |
Nov 17, 2003 |
|
|
|
Current U.S.
Class: |
382/128 |
Current CPC
Class: |
G06T 2207/30024
20130101; G06T 7/62 20170101; G06T 7/0012 20130101; G06T 2207/10056
20130101; G06T 7/11 20170101; G06K 9/00127 20130101; G06T
2207/20036 20130101; G06T 7/187 20170101; G06T 2207/30061 20130101;
G06T 2207/30056 20130101; G06T 7/155 20170101 |
Class at
Publication: |
382/128 |
International
Class: |
G06K 9/00 20060101
G06K009/00 |
Claims
1. A method for tissue analysis comprising: acquisition of a tissue
image corresponding to a tissue sample; segmentation of the image
into a plurality of objects; classifying the plurality of objects
into one or more object types; and quantifying at least one
parameter corresponding to at least one first object type to
produce a quantification result for each corresponding object of
the first object type.
2. The method according to claim 1, wherein the parameter is
selected from. the group consisting of: a size, a shape, a color,
spacing, color intensity, luminescence, an area a relationship to a
second object type and combinations of the foregoing.
3. The method according to claim 1, wherein an object type is
selected from the group consisting of: a basic object, a composite
object, a cell, and a cell component.
4. The method according to claim 1, further comprising:
establishing a range of the quantification results; dividing the
range into a plurality of bins, wherein each bin represents a
visual indicator for producing a modified image of the tissue
image; assigning each object of the first object type to a bin
based on the quantification result for each object; and modifying
corresponding pixels for each object of the first object type in
the modified image with the visual indicator of the bin
corresponding to the respective object.
5. The method according to claim 1, wherein the at least one
parameter is selected from the group consisting of: a second
classified object, a color, a shape and a predetermined area.
6. The method according to claim 4, further comprising classifying
the tissue sample based on the modified image.
7. The method according to claim 6, wherein the tissue is
classified as normal or abnormal.
8. The method according to claim 4, wherein the visual indicator is
selected from the group consisting of: color, color intensity,
size, shape, symbol, letter and number.
9. The method according to claim 4, wherein the tissue image
comprises an image of liver tissue and the classified first object
type comprises a hepatocyte and the parameter comprises fat
content.
10. The method according to claim 4, wherein the method is used to
identify toxic effect or response.
11. The method according to claim 4, wherein the method is be used
to identify immunological reactions.
12. The method according to claim 4, wherein the method may be used
to identify morphological lesions caused by disease selected from
the group consisting of: acute, sub-acute or chronic hepatitis;
inflammatory or necrotic cholestasis; fibrosis; granulomatous
hepatitis; macro or microvesicular steatosis; vascular lesions; and
hepatic tumors.
13. The method according to claim 6, further comprising training a
neural network and/or a support vector machine using the modified
image.
14. The method according to claim 12, wherein quantification
comprises a ratio of the area of fat contained in each hepatocyte,
to the area of the respective hepatocyte.
15. A method for tissue analysis comprising: acquisition of a
tissue image corresponding to a tissue sample; segmentation of the
image into a plurality of objects; classifying the plurality of
objects into one or more object types; quantifying at least one
parameter corresponding to at least one first object type to
produce a quantification result for each corresponding object of
the first object type; establishing a range of the quantification
results; dividing the range into a plurality of bins, wherein each
bin represents a visual indicator for producing a modified image of
the tissue image; assigning each object of the first object type to
a bin based on the quantification result for each object; and
modifying corresponding pixels for each object of the first object
type in the modified image with the visual indicator of the bin
corresponding to the respective object.
16. A computer application program operable on a computer system
for enabling the computer system to perform a method for tissue
analysis, the method comprising: acquisition of a tissue image
corresponding to a tissue sample; segmentation of the image into a
plurality of objects; classifying the plurality of objects into one
or more object types; quantifying at least one parameter
corresponding to at least one first object type to produce a
quantification result for each corresponding object of the first
object type; establishing a range of the quantification results;
dividing the range into a plurality of bins, wherein each bin
represents a visual indicator for producing a modified image of the
tissue image; assigning each object of the first object type to a
bin based on the quantification result for each object; and
modifying corresponding pixels for each object of the first object
type in the modified image with the visual indicator of the bin
corresponding to the respective object.
17. The computer application program according to claim 16, where
the method further includes displaying and/or printing the modified
image.
18. A computer readable medium having one or more computer
application programs and/or computer instructions for enabling a
computer system to perform a method for tissue analysis, the method
comprising: acquisition of a tissue image corresponding to a tissue
sample; segmentation of the image into a plurality of objects;
classifying the plurality of objects into one or more object types;
quantifying at least one parameter corresponding to at least one
first object type to produce a quantification result for each
corresponding object of the first object type; establishing a range
of the quantification results; dividing the range into a plurality
of bins, wherein each bin represents a visual indicator for
producing a modified image of the tissue image; assigning each
object of the first object type to a bin based on the
quantification result for each object; and modifying corresponding
pixels for each object of the first object type in the modified
image with the visual indicator of the bin corresponding to the
respective object.
19. The computer readable media according to claim 18, where the
method further includes displaying and/or printing the modified
image.
20. A computer system for performing a method for tissue analysis,
the method comprising: acquisition means for acquiring a tissue
image corresponding to a tissue sample; segmentation means for
segmenting the image into a plurality of objects; classifying means
for classifying the plurality of objects into one or more object
types; quantifying means for quantifying at least one parameter
corresponding to at least one first object type to produce a
quantification result for each corresponding object of the first
object type; establishment means for establishing a range of the
quantification results; dividing means for dividing the range into
a plurality of bins, wherein each bin represents a visual indicator
for producing a modified image of the tissue image; assigning means
for assigning each object of the first object type to a bin based
on the quantification result for each object; and modifying means
for modifying corresponding pixels for each object of the first
object type in the modified image with the visual indicator of the
bin corresponding to the respective object.
21. A system for tissue analysis comprising: an input for inputting
a digital image; an output comprising at least one of a display and
a printer; a processor for processing computer instructions and
data, the processor operating to: segmenting the image into a
plurality of objects; classifying the plurality of objects into one
or more object types; quantifying at least one parameter
corresponding to at least one first object type to produce a
quantification result for each corresponding object of the first
object type; establishing a range of the quantification results;
dividing the range into a plurality of bins, wherein each bin
represents a visual indicator for producing a modified image of the
tissue image; assigning each object of the first object type to a
bin based on the quantification result for each object; and
modifying corresponding pixels for each object of the first object
type in the modified image with the visual indicator of the bin
corresponding to the respective object.
Description
CLAIM TO PRIORITY
[0001] This application claims priority under 35 U.S.C.
.sctn.119(e) of U.S. patent application Ser. No. 60/520,815, filed
Nov. 17, 2003, the entire disclosure of which is herein
incorporated by reference in its entirety.
FIELD OF THE INVENTION
[0002] This invention relates to molecular biology, histology, and
clinical diagnostics. Clinical, micro-anatomic and molecular
profiles of disease are integrated to create a system for tissue
analysis which, in a preferred embodiment, comprises a pathological
mapping of a tissue image to determine a pathological status or
condition of the tissue in the image. The file of this patent
contains at least one figure executed in color. Copies of this
patent with color figures will be provided by the Office upon
request and payment of the necessary fee.
BACKGROUND OF THE INVENTION
[0003] Pathology is the medical science and specialty practice that
deals with all aspects of disease, but with special reference to
the essential nature, causes, and development of abnormal
conditions. This generally includes analysis of the structural and
functional changes that result from diseases.
[0004] To determine the causes of a disease, a pathologist may
study: how various internal and external injuries affect cells and
tissues, how a disease progresses (pathogenesis), and how a disease
manifests in a tissue (i.e., its clinical expression and the
lesions produced). In other words, pathology provides a scientific
foundation for clinical medicine and serves as a bridge between the
basic sciences and patient care.
[0005] Accordingly, accurate and repeatable quantitative analysis
of tissue is important to characterize a disease and evaluate
effects that new therapies might have. To date, little if any
reliable structural information exists at the tissue level (e.g.,
1-1000 microns, in the range of microscopic to mesoscopic). It is
believed that if reliable, multi-dimensional structural tissue
information (including, for example, clinical, molecular and
genetic information) existed in readily accessible databases. Such
information would enhance and accelerate new advances in tissue
engineering, drug design, gene discovery, proteomics, and genomics
research.
[0006] In order to facilitate the study and diagnosis of disease,
investigators have developed a variety of systems and methods.
Generally, prior art methods and systems relating to the study of
disease are slow, difficult and prone to error. Accordingly, there
exists a need for a system and/or method to quickly, efficiently,
and/or automatically quantify tissue for determining a condition of
a tissue.
SUMMARY OF THE INVENTION
[0007] The present invention presents methods and systems for
processing and analyzing a tissue image(s), and moreover, with
regard to some embodiments of the invention, for automating
object/feature extraction from tissue and/or determining
quantitative definition of tissue features. Embodiments of the
present invention produce a pathological tissue map (PTM) of the
tissue, which comprises a modified version of an image of the
tissue. The PTM classifies objects of the tissue into visible
indicators which may be analyzed quickly by a user (e.g.,
pathologist) and/or an algorithm, to more quickly determine a
tissue condition (e.g., normal versus abnormal). For example, a PTM
may be generated by quantifying a variety of micro-anatomic and/or
molecular data and associating a color grade with a range for that
particular data. Accordingly, the data may be rendered in a format
where areas of abnormality are identified in a specific color (red
for example), which may be easily identifiable to a viewer (e.g.,
pathologists, scientists or physicians).
[0008] In one embodiment of the invention, an automated tissue
processing system is disclosed, for advanced tissue image
classification of (for example) hematoxylin and eosin
(H&E)-stained tissue sections. Using such a system, tissue
images may be segmented then analyzed. Furthermore, using neural
network or support vector regression ("SVR"), the segmented images
may be used to train a biostatistical model to determine tissue
condition (e.g., normal versus abnormal).
[0009] In particular, such a system may facilitate distinguishing
and visualizing an object in a tissue image using predetermined
criteria. When an object is found, boundaries of the object may be
constructed using (for example) modified object extraction
algorithms used in the art.
[0010] Criteria for locating tissue objects may include, for
example, object color, color intensity, object morphology
(including material composition), object size and shape (e.g.,
dimensions, round, oval, etc.), arrangement of objects, or any
combination thereof. For example, with regard to color, a tissue
may be stained to highlight certain objects. To detect tissue
objects in an image, existing mathematical feature detection
algorithms may be used, or modified versions thereof, such as those
available with the Cellenger software product marketed by Definiens
A.G. Such algorithms may include, for example, dilation (adding
pixels to the boundary of an object), erosion (removing pixels on
the object boundaries), and thresholding. In addition, the
detection of background intensity is useful for object
determination and is required in some feature extraction
algorithms.
[0011] One can also apply one or more morphological filters to
enhance certain objects and suppress others. Such enhancements may
change the shape of an object contained within an image.
Morphological filters are preferably used prior to applying
character/shape recognition algorithms since these filters can
highlight the contour of objects which aid the recognition. For
example, a morphological filter may be used to enhance certain
objects of a particular size and the dilation and/or erosion
algorithms may be used to bring out the enhanced objects.
[0012] Embodiments of the invention may further include
quantitative determination of object geometry. One or more found
objects may be quantified (e.g., measured), and a modified tissue
image established with visual indicators indicating the quantified
objects. The modified image represents the PTM for pathological
analysis.
[0013] Still other embodiments of the present invention are
directed to databases, which may be used in conjunction with other
embodiments of the invention. Specifically, such databases may
include characterization data and/or associated images ("tissue
information") representative of a tissue population, and/or an
automated method to create such database and use of the database
for classification and evaluation of tissue specimens. For example,
samples of normal tissue specimens obtained from a subset of a
population of subjects with shared characteristics may be profiled
(e.g., objects extracted and classified as normal) in order to
generate a plurality of structural indices that correspond to
statistically significant representations of tissue associated with
the population.
[0014] The database may also include information from profiled
tissue images from samples of specimens of a particular tissue
obtained from a subset of a population with respect to certain
structural or other indicia, that correspond to a particular
clinical condition associated with that tissue. Such information
may be used to provide a comparison with information obtained from
additional specimens of the tissue, including specimens which may
have been previously profiled by other means or for other purposes.
Indicia may include at least one of cell density, matrix density,
blood vessel density, layer thickness or geometry, and the
like.
[0015] Embodiments of the invention may be used to identify a toxic
effect or response, immunological reactions, morphological lesions
caused by, for example, hepatitis (acute, subacute and chronic),
cholestasis (with and without inflammation or necrosis), fibrosis,
granulomatous hepatitis, steatosis (macro and microvesicular),
vascular lesions, and hepatic tumors. Further yet, embodiments of
the invention may be used to characterize pathological objects, for
example, Kupffer cell hyperplasia, cholangitis, cholangiolitis,
necrotizing angitis, sinusoidal dilatation, hepatoportal sclerosis
and venous thromboses.
BRIEF DESCRIPTION OF THE DRAWINGS
[0016] FIG. 1A is a general block diagram representing a process
flow for pathological tissue mapping according to some of the
embodiments of the present invention.
[0017] FIG. 1B is a block diagram representing a specific process
flow for pathological tissue mapping according to one embodiment of
the present invention.
[0018] FIG. 1C is a representative system for carrying out method
embodiments for the present invention.
[0019] FIG. 2 is a block diagram representing a process flow for
image segmentation according to some embodiments of the present
invention.
[0020] FIG. 3A is an original image of normal liver tissue.
[0021] FIG. 3B is a segmented image of the normal liver tissue of
FIG. 3A, illustrating hepatic nuclei, kupffer nuclei, sinusoids and
fat content.
[0022] FIG. 3C is an original image of abnormal liver tissue.
[0023] FIG. 3D is a segmented image of the abnormal liver tissue of
FIG. 3C, illustrating hepatic nuclei, kupffer nuclei, sinusoids and
fat content.
[0024] FIG. 4A is an original tissue image of a bile duct.
[0025] FIG. 4B is a segmented image of the bile duct of FIG. 4A,
illustrating bile duct lurnen, epithelial nuclei, hepatic artery
lumen, and hepatic nuclei.
[0026] FIG. 5A is an original tissue image of a hepatic vein.
[0027] FIG. 5B is a segmented image of the hepatic vein of FIG. 5A,
illustrating hepatic vein lumen, hepatic vein wall and hepatic
nuclei.
[0028] FIG. 6A is an original tissue image of a hepatic artery.
[0029] FIG. 6B is a segmented image of the hepatic artery of FIG.
6A, illustrating hepatic artery, red blood cells and hepatic
nuclei.
[0030] FIG. 7A is an original image of a hepatocyte.
[0031] FIG. 7B is a segmented image of the hepatocyte of FIG.
7A.
[0032] FIG. 8A is an H&E stained tissue image of normal liver
tissue.
[0033] FIG. 8B is a segmented image of the stained image of FIG.
8A.
[0034] FIG. 8C is a pathological tissue map of the original image
of FIG. 8A and segmented image of FIG. 8B.
[0035] FIG. 9A is an H&E stained tissue image of abnormal liver
tissue.
[0036] FIG. 9B is a segmented image of the stained image of FIG.
8A.
[0037] FIG. 9C is a pathological tissue map of the original image
of FIG. 8A and segmented image of FIG. 8B.
[0038] FIG. 10 illustrates nests of polygonal cells with pink
cytoplasm and distinct cell borders in squamous cell lung
carcinoma.
[0039] FIG. 11 is an image of columnar cells with reference to
bronchioloalveolar lung carcinoma.
[0040] FIG. 12 is an image showing small dark blue cells with
minimal cytoplasm packed together in sheets of oat cell
disease.
[0041] FIG. 13 is an image of tubular structures of malignant
glandular neoplasia (colon cancer).
[0042] FIG. 14 is an image of goblet cells (colon cancer).
[0043] FIG. 15 illustrates a pathological staging of bladder cancer
based on invasiveness.
[0044] FIG. 16 is an image of papillary projections for determining
transitional cell carcinoma of the urothelium.
[0045] FIG. 17 is an image of neoplastic cells having uniform oval
nuclei, abundant cytoplasm, and are arranged in ribbons of tissue
supported by delicate vascular cores or "stalks".
[0046] FIG. 18, a photomicrograph of carcinoma in situ in the
bladder.
DETAILED DESCRIPTION OF THE EMBODIMENTS
[0047] Unless otherwise defined, all technical and scientific terms
used herein have the same meaning as commonly understood by one of
ordinary skill in the art to which this invention belongs. Although
methods, systems and materials similar or equivalent to those
described herein can be used in the practice or testing of the
invention, suitable methods, systems and materials are described
below. In the case of conflict, the present specification,
including definitions, will control. In addition, the materials,
methods, and examples are illustrative only and not intended to be
limiting.
[0048] Moreover, although most of the embodiments of the present
invention will be described with reference to a liver tissue
analysis example, it is meant as an example only and not intended
to be limiting.
[0049] Some embodiments of the present invention relates to an
automated measurement and analysis system to quantitatively
evaluate one or more tissue features/objects. The tissue specimens
that can be analyzed by the present invention may include any
tissue of any organ system, including, for example, liver, kidney,
bile duct, gastrointestinal tract, lymphatic vessel, bronchia,
blood vessels, cardiac, and nerve tissues.
[0050] The images are processed to produce a modified image of a
tissue image with visual markers for indicating the pathology of
the tissue (the PTM), that can more easily be analyzed by a
diagnosis algorithm or pathologist. Once tissue specimens have been
prepared, generally, the process for producing a PTM includes:
acquiring an image of the tissue specimen; segmenting the image,
classifying one or more objects, quantifying one or more objects,
creating a modified image with visual indicators for the quantified
objects; and pathologically classifying the tissue. A general
overview of these steps is shown in FIG. 1A, with a more specific
flow illustrated in FIG. 1B.
[0051] FIG. 1C is a block diagram of a system for carrying out one
or another of the method embodiments according to the present
invention. As shown, a computer having an input module which may
comprise a keyboard, ports (e.g., USB, parallel, SCSI, serial, and
the like), a computing module (i.e., a computer workstation; a
processor), a display and a printer. The ports may be used to
connect image acquisition equipment (e.g., microscope having
digital camera/CCD/CMOS device), as well as connecting external
data storage devices (e.g., CD-ROM/RW; hard-drives, DVD, etc.). The
system may be part of a larger network, and may communicate with
such network either via wireless or wired (e.g., Ethernet)
connection.
[0052] Tissue images may be obtained in any number of ways familiar
to those of skill in the art. For example, X-ray images (including
CAT scan images) and MRI images may be used, digitized to be input
into a computer system. Particular preferred embodiments of the
invention may obtain images by taking a photograph (preferably
digital, but may be a traditional photo which is later digitized)
of a magnified section of a tissue slide (e.g., a cross-sectional
slice of tissue) on a microscope.
[0053] Segmenting tissue images may include one or more of:
preprocessing images to correct color variations; location of
tissue histopatholic objects; and classifying the found objects. A
general overview of the segmentation process is illustrated in FIG.
2.
[0054] Initially, images may be pre-processed to standardize color
variations from image to image (e.g., when using H&E stained
tissue) of a tissue, using, for example, color (histogram)
normalization. Images of tissues stained under different conditions
and time may have color variations from image to image which may
impair object classification in the image. Accordingly, histogram
equalization may be used to bring image colors into close
ranges.
[0055] To standardize the color variations in a set of images of a
particular tissue, one tissue image may be selected as the
representative image, and then the histogram for each of the tissue
images remaining in the set may be adjusted so that each matches
the histogram of the representative image. Alternatively, the
histograms of several images may be used to derive an average (for
example) histogram for the image set.
[0056] After pre-processing, tissue histopatholic objects are
located. Each object may be a basic object or a composite object.
Basic objects may include, for example, fundamental objects of
tissue, including cell components (e.g., nuclei, sinusoids, fat and
fat vacuoles, cytoplasm). Composite objects may be more complex
than basic objects and are typically constructed from basic
objects. Examples of composite objects include: cells (e.g.,
hepatocytes) and vascular tissues (e.g., bile duct, veins,
arteries).
[0057] For example, a composite object may represent an entire
cell, made up of basic objects including nuclei and cytoplasm (for
example). Each cell may be "grown" using a cell growing algorithm,
where a specific object ("seed") for cell formation (e.g., hepatic
nuclei for the hepatocytes) is used as the basis for forming the
cell, and then other objects are added to it.
[0058] Image segmentation may be based on object oriented image
analysis, where an image (preferably non-equalized) is partitioned
into homogenous groups with respect to color and shape of adjacent
areas (i.e., image objects). The image information can be
represented in different scale depending on the average size of the
objects. Accordingly, using spectral and shape characteristics,
image objects may then be referred to as instances of the tissue
histopathological objects.
[0059] Besides using spectral and shape criteria to find objects,
spatial relations between objects may also be taken into
consideration to find objects. For example, sinusoids may be
identified as elongated image objects containing red blood cells
located within a range of known distances from Kupffer cells.
Hepatocytes, tissue structure composed of cytoplasm, fat, fat
vacuoles and hepatic nuclei bordering along sinusoids, may be found
using hepatic nuclei objects as "seeds", and "growing" the
hepatocyte sequentially by adding surrounding image objects until
it reaches a sinusoid object. A region growing algorithm may be
used for such cell formation.
[0060] To further enhance and automate the analysis process, tools
commonly used with computer-aided-design (CAD) software may be used
with the image-processing embodiments of the invention to aid in
extracting objects from tissue images. The CAD tools offer the
ability to pick points and group them, fit polynomial curves or
splines to groups of points, and the ability to merge curve
segments in an ordered fashion so they bound regions of interest.
Such tools may be used to correct objects which have been
incorrectly extracted.
[0061] After objects (basic and/or composite) have been found, the
found objects may then be classified. For example, with nuclei
classification, image objects may be classified as "nuclei" versus
"non-nuclei" class objects using, for example, spectral and shape
characteristics. The nuclei objects may be further sub-divided in
two categories: "epithelial nuclei" and "inflammatory cells", for
example. Moreover, with regard to liver tissue analysis, color
intensity, shape and/or size thresholds may be used to classify the
"epithelial nuclei" objects as "hepatic nuclei" and "Kupffer cells"
nuclei objects. It is worth noting that sometimes a single
nucleolus object is actually a plurality of real nuclei merged
together. In such a situation, specialized morphological operations
may defuse the nuclei objects into respective nuclei.
[0062] After nuclei objects have been classified, white spaces of
the image may also be classified. White spaces are objects which
are non-nuclei objects, and may be determined based on an intensity
threshold (for example) of the non-nuclei objects. Objects such as
red-blood cells, fat, fat vacuoles and sinusoids objects may then
be derived from the white space.
[0063] Once objects have been classified, one or more objects, as
well as one or more parameters of objects (a basic object may, in
some embodiments, represent a parameter of a composite object, for
example) may be quantified to analyze the tissue to determine a
pathological condition of the tissue (e.g., normal versus abnormal)
via a PTM. In some embodiments, quantification relates to the
determination of a value for a specific object/parameter relative
to a granularity unit of the image. A granularity unit may comprise
another object, basic or composite (preferably composite), the
tissue image itself, or a specific area of the image, color, color
intensity, size, shape, and the like. The value of the specific
object/parameter may be a quantity, a color, color intensity, a
size, an area, or a shape. The value may also be a ratio; for
example, the ratio may be the area of the specific object relative
to the area of the granularity unit. For example, in liver cells, a
cell object (e.g., nucleus, fat) can be quantified by establishing
a ratio of the area of the cell object to that of the area of the
cell. Specifically, for each cell in the image, the cell area is
measured (A.sub.i), the object area (O.sub.i) is measured, and the
ratio of O.sub.i/A.sub.i is determined. A ratio interval may then
be set based on the range of ratios found in image.
[0064] The result of quantification may be organized into a number
of "bins", where each bin is associated with a particular visual
indicator (e.g., color). Representative pixels of the quantified
objects in a modified image of the original tissue image are then
marked with indicators (e.g., colorized) with the corresponding bin
indicator to produce the PTM. Accordingly, a pathologist can view
the PTM to easily determine the state of the tissue for a
particular object quantification. The visual indicator may comprise
a symbol, a color, a letter, a number and the like. Any visual
detail to display attention to the quantified object in the
modified image.
Liver Toxicology
[0065] For liver toxicology (for example) analysis, such
quantification may be the analysis of hepatocytes (granularity
unit) based on the fat content (fat molecule: quantified object) of
the cell (a fat PTM) or hepatic nuclei (nuclei PTM). Fat
accumulating in the liver is mainly in the form of triglycerides
and fatty acids, and is also present in small amounts in the form
of cholesterol, cholesterol ester and phospholipids. Fat
accumulation in the liver may be designated pathologically as
"fatty degeneration of the liver", and is also referred to as
"fatty change", "fat infiltration", "fat metamorphosis" and
"steatosis of the liver". Fatty liver is observed in a multitude of
conditions such as: obesity, hyperalimentation (hypemutrition),
alcoholic liver disease, diabetes mellitus, congestive heart
failure, drug intoxication, pregnancy, Rey's syndrome, malnutrition
(Kwashiorkor), chronic hepatitis and cirrhosis of different
etiology.
[0066] FIGS. 3A-6B represent example segmented images of original
tissue images.
[0067] For hepatic fat, the fat content generally ranges from 0 (a
cell free of fat) to 1 (a cell replaced by fat), with varying
degrees of fat therebetween (e.g., 0.1, 0.2, etc.). The range of
fat content may be divided into the ratio interval--into a number
of bins, each of which corresponds to a color (or color
intensity/shade), in a graded range. Each hepatocyte cell object is
then assigned to a particular bin based on its quantified fat
content. The pixels in a modified image of the original tissue
image corresponding to each hepatocyte cell object is then
colorized with the corresponding bin color to establish the PTM of
the tissue. The completed PTM may then be output on a LCD/CRT
display or output to a printer (and/or database) for review.
[0068] In general, in many quantification, the ratio interval may
be set up to vary from 0 to 1, but sometimes the bins derived from
the interval [0, 1] do not have enough resolution; almost all
ratios can fall into one or several bins. In order to set an
informative bin system, it is recommended to experimentally find a
meaningful ratio upper level (for example 0.5). The chosen upper
level should work over all cells presented in a studied image or
image set. It is worth noting that decreasing or increasing the
number of bins may result in under or over representation of cell
classes respectively.
[0069] In the liver toxicology example, hepatocytes having a low
fat content may be assigned to a blue bin, cells having a moderate
fat content may be assigned to a yellow bin, and cells having a
high fat content may be colored red. However, to achieve a smooth
color transformation between the three representative colors, for
example, multiple bins (representing shades between the colors
blue-to-yellow, and yellow-to-red) for cells having a particular
fat content may be used. For example, using 10 bins: bin 1=0 fat
content; bin 2=12.5% fat content; bin 3=25% fat content; bin
4=37.5% fat content; bin 5=50% fat content; bin 6=62.5% fat
content; bin 7=75% fat content; bin 8=87.5% fat content; bin 9=95%
fat content; and bin 10=100% fat content. Bin 1 may represent the
blue color, bin 5 yellow and bin 9 red. Thus, bins 2-4 may be
varying shades between blue and yellow and bins 6-8 may be varying
shades between yellow and red. Alternatively, bins 1-3 may be blue,
bins 4-7 may be yellow, and bins 8-10 red. FIGS. 8A-8C represents a
tissue image, a segmented image, and a PTM for a specimen of normal
liver tissue (bin legend also included), and FIGS. 9A-9C represent
the corresponding figures for abnormal liver tissue. As shown, the
PTM for the normal tissue includes a low fat content (generally
between 0.2 and 0.5), while the fat content is quickly determined
to be higher than that of the normal tissue because of the increase
in the number of hepatocytes colored yellow.
[0070] After the PTM is created, the PTM statistics (e.g.,
hepatocyte fat content) may be loaded into a database. For example,
the relative areas occupied by each cell class--percentage of cells
with low object content, with moderate content etc. Other
characteristics may be assigned to created cell classes.
Prostate Cancer Analysis
[0071] A PTM may be generated for other histopathological tissue
types or quantifications for prostate cancer. In prostate cancer,
the granularity unit may comprise a tissue core (tile) gland unit,
or to an entire tissue section. A prostate tissue core (tile) gland
unit is a key structure for accessing the distortion of the normal
prostate architecture (i.e., the degree of malignancy). A gland
unit includes lumen, epithelial cells and cytoplasm objects. The
relative lumen area with respect to tissue core area may serve as
the quantification object for a PTM. This ratio characterizes
cancer development in the tissue core: the more aggressive a
cancer, the more gland units with small relative area values
exist.
[0072] A PTM may also be created to determine Gleason grade on an
entire tissue section. The tissue section is partitioned on uniform
gland units, and assigned a Gleason grade. The Gleason grade is an
integer number from 1 to 5, characterizing cancer aggressiveness.
For example, five (5) bins may be established, each corresponding
to a particular Gleason grade. Thereafter, each gland unit is
matched with a bin, and the pixels in the tissue image
corresponding to a respective gland unit are then colorized
according to the color of the respective bin. The PTM is then
generated and output to a user.
Other Applications of PDMs
[0073] The following is a list of cancers in which embodiments of
the present invention may aid in determining the pathology
thereof.
[0074] Squamous cell Lung Carcinoma. Cytoplasm, distinct cell
borders and/or interceller bridges may be quantified and used to
generate a PTM to diagnosing or determining an extent of squamous
cell carcinoma. Poorly differentiated carcinomas have a worse
prognosis and they are more aggressive than the well
differentiated. A well-differentiated carcinoma resemble a normal
lung architecture. FIG. 10 illustrates this cancer, showing nests
of polygonal cells with pink cytoplasm and distinct cell borders.
The nuclei are hyperchromatic and angular.
[0075] Bronchioloalveolar Lung carcinoma. Columnar cells may be
quantified to determine diagnosis and/or extent of
bronchioloalveolar carcinoma. Cancerous columnar cells are
well-differentiated and can be seen in FIG. 11.
[0076] Small cell Anaplastic (oat cell). Cells having minimal
cytoplasm may be quantified to produce a PTM to determine a
diagnosis and extent of small cell anaplastic (oat cell). As shown
in FIG. 12, small cell anaplastic is evident from the small dark
blue cells with minimal cytoplasm are packed together in sheets,
which typify oat cell disease.
[0077] Colon Cancer. Malignant glandular neoplasia, which are
tubular structures (FIG. 13), with necrosis and hyperchromasia, may
be quantified to produce a PTM to determine colon cancer. In
addition, the cancer may be diagnosed by reviewing cancerous goblet
cells (FIG. 14) may also be quantified to produce a PTM for colon
cancer.
[0078] Bladder cancer. Muscle invasiveness of transitional cell
carcinomas may be quantified and used to produce a PTM, to
determine bladder cancer. FIG. 15 illustrates a pathological
staging of bladder cancer based on invasiveness. Quantification of
papillary projections (FIG. 16 illustrating cancerous projections)
for determining transitional cell carcinoma of the urothelium may
also be used to produce a PTM. As shown in FIG. 17, neoplastic
cells have uniform oval nuclei, abundant cytoplasm, and are
arranged in ribbons of tissue supported by delicate vascular cores
or "stalks". FIG. 18, a photomicrograph of "carcinoma in situ" in
the bladder. The epithelial cells on the left have malignant
cytologic objects including very large, irregularly shaped and
darkly staining nuclei, which contrasts with the normal appearance
of the urothelial cells on the right. Accordingly, the foregoing
may be quantified to produce a PTM.
Pathology Models
[0079] A PTM and/or basic object measurements may form a feature
vector for biostatistical modeling, where advanced statistical
models are used in order to classify the tissue image as being
normal, abnormal, diseased and the like. Specifically, a neural
network or SVR machine may be trained to make a comparison of a PTM
to a PTM (or statistics thereof) from profiled data. To that end,
one embodiment of the invention provides a method of automated
H&E image analysis for liver toxicology and other medical
areas.
Database
[0080] The present invention is also directed to a robust database
that is based upon input parameters that may be uniformly
investigated and extracted from different studies. Specifically,
embodiments of the invention include a database that allows input
and retrieval of data and images needed to compare studies taking
place at different times, with different protocols, and with
measurements made by different systems. Accordingly, the database
may preserve the utility of the stored information through
continued lossless combination and comparability with subsequently
acquired information and the accessibility of the stored images for
automated re-analysis.
[0081] Images and data may be stored together or separately
(preferred). The data may be kept in spreadsheets, or through
fields of a relation database. If the images and data are
separately stored, the images and data can be merged using
hyperlinks (for example). From a practical standpoint, a more
robust database that manages the input and retrieval of data and
images may be used to compare studies taking place at different
times, with different protocols, and with measurements made by
different systems. The database may include sufficient and accurate
information to enable the user to normalize the results to make
meaningful comparison between studies.
EXAMPLES
Example 1
Liver Tissue Image Segmentation--Portal Tract
Bile Duct
[0082] Analysis of Bile Duct demonstrates that it is a tissue
structure consisting of lumen (white area on the original image
fragment FIG. 4A; colored yellow on the segmented image FIG. 4B)
lined by simple cuboidal or columnar epithelium (epithelial nuclei
painted by blue color on the segmented image).
Vessels
Hepatic Vein
[0083] Analysis of the Hepatic Vein (see FIG. 5A original image;
FIG. 5B segmented image) which is the largest diameter vessel,
reveals it to be another tissue structure consisting of lumen
(large white area on the original image fragment colored light grey
on the segmented image) which has the typical, thin-walled
structure relative to the diameter of the lumen and irregular
outline of all veins (colored aquamarine on the segmented
image).
Hepatic Artery and Arterioles
[0084] The smaller diameter, thick-walled vessels with the typical
structure of arterioles and arteries are branches of the Hepatic
Artery which supplies oxygenated blood to the liver. The Hepatic
Artery is composed of a large white area (lumen) surrounded by a
smooth muscle fibers wall that his thickness approaches the
diameter of the lumen. Occasionally red blood cells can be found
within the lumen area. See FIG. 6A original image and FIG. 6B
segmented image of Hepatic artery.
Lymphatics
[0085] Another type of vessel, lymphatics, are also present in the
portal tracts, but since their walls are delicate and often
collapsed they are not readily seen.
Hepatocytes
[0086] Hepatocytes are large, polyhedral cells which have a
variable cytoplasmic appearance depending on the nutritional and
health status of the body. In well-nourished individuals,
hepatocytes store significant quantities of glycogen and process
large quantities of lipid. Both of these metabolites are partially
removed during routine histological preparation thereby leaving
irregular, unstained areas within the cytoplasm. (vacuoles). The
remaining cytoplasm is strongly eosinophilic due to a high content
of organelles.
[0087] The nuclei of hepatocytes are relatively large with
peripherally dispersed chromatin and prominent nucleoli. The
nuclei, however, vary greatly in size. Occasional binucleate cells
are seen in section although up to 25% of all hepatocytes are
bionucleate. The arrangement of hepatocytes within the liver
parenchyma is distinct. The hepatocytes form flat, anastomosing
plates usually one cell thick between which sinusoid course.
[0088] Analysis of hepatocytes (FIGS. 7A-7B; 8A-8B) reveals cells
formed by hepatic nuclei (dark ring on the pink background) and
surrounding cytoplasm. The cell boundaries often go along
sinusoids. A healthy cell may have an insignificant amount of fat.
The more fat present in the cell, the more abnormal the cell is,
and the liver is diagnosed as fatty liver. A hepatic nuclei can be
completely replaced by excess fat deposit within the liver cell.
FIGS. 8A-8C depict images of normal hepatocytes and FIGS. 9A-9C are
images of abnormal hepatocytes containing excess fat.
[0089] The resulting PTM for the present example is presented in
FIGS. 8C (normal fat content) and 9C (abnormal fat content). The
color changes from blue (low fat content) through yellow (moderate
fat content) to red (high fat content). As is clear, there is a
significant amount of fat (light round different size objects)
around the hepatic nuclei in the abnormal hepatocytes.
Example 2
[0090] This study was undertaken to demonstrate neural network and
linear discriminant analysis (LDA) modeling capabilities of the
present invention. Specifically, the study involved the acquisition
and analysis of sections of rat liver with the overall objective
being to classify the sections as normal or abnormal. Being able to
automate this process while simultaneously achieving a high-level
of classification accuracy allows for the creation of a
high-throughput platform used to objectively screen for toxicities
in pre-clinical studies.
[0091] The study was divided into two phases. The initial phase
used a set of 100 rat liver sections as a training set; 80 normal
liver sections and 20 abnormal. The image analysis process was then
applied to an unlabeled set of 100 rat liver sections in the second
phase of the study in which the statistical models designed in the
training phase were tested.
Pathology
[0092] Both the training and test set of rat liver sections were
H&E-stained slides. Each set consisted of 100 slides. The
training set of slides contained 80 normal liver sections and 20
abnormal liver sections. The testing set contained no information
as to whether the sections were considered normal or abnormal.
[0093] Images were taken by a pathologist, using the Spot Insight
QE digital camera mounted on the Nikon Eclipse E400 microscope with
the use of the Advance Spot software. The working objective was a
20.times.Plan Apo, and 24 bits/pixel color images were taken and
stored in TIF uncompressed file format with size 1200.times.1600
pixels. The resolution was 2744 pixels/mm.
Tissue Image Processing
[0094] The tissue image processing system provides necessary
information for the objective classification of an H&E stained
liver section as being normal or abnormal, where basic and
composite histopathological objects in the tissue image were found
and quantified.
[0095] The image segmentation was conducted by partitioning a
tissue image into non-overlapping, constituent connected regions
and assigning a meaningful label to each region. The labels
correspond to histopathological objects of the liver tissue. The
image analysis method defines quantitative characteristics
(measurements) for all objects detected on the segmented tissue
image. The implemented image processing system consists of three
main components: preprocessing, image segmentation and object
measurements.
Basic and Composite Morphological Objects
[0096] The following basic pathological objects were selected:
nuclei, sinusoids, fat, fat vacuoles, blood vessels: hepatic veins
and arteries, cytoplasm, red-blood cells. The nuclei were further
classified as hepatic, kupffer, epithelial and inflammatory cells.
The considered morphological structures which were composed of the
basic objects were: hepatocytes with fat and fat vacuoles,
hepatocytes with hepatic nuclei, hepatocytes. The hepatocytes are
morphological tissue elements formed by hepatic nuclei and attached
cytoplasm.
Preprocessing
[0097] Color variations from image to image are the most common
drawback of the H&E staining procedure. The spectral properties
of the same objects also vary from image to image which affects the
accuracy and robustness of segmentation. A color normalization
technique based on histogram matching was used in order to address
the color variation problem.
[0098] An image having good staining quality and representative
color was chosen as a reference. The color histograms of the
remaining images were transformed to match the RGB histograms of
the reference image.
Image Segmentation
[0099] The tissue images are 24 bits/pixel color images stored in
TIF file format with size 1200.times.1600 pixels. In the image
processing system, each image is represented by six layers: three
original RGB layers and three normalized layers. Basic pathological
objects form classes; segmented objects (e.g. nuclei) are the
instances of a class. Besides the basic classes special auxiliary
classes were created.
[0100] Conceptually the image processing system was designed as a
multilevel system. Each level is a virtual image plane with class
instances corresponding to a certain processing stage.
[0101] Level 1 is the starting level where the whole image is
partitioned into non-overlapping, unclassified regions (image
objects). The image objects may be merged by some criteria on the
upper levels forming the super-objects with respect to objects on
the lower level (sub-objects). The image objects may be networked
so that each object is associated to its neighborhood, its
super-object and its sub-objects. All sequential processing is
about proper management (classification and merging) of the
obtained image objects. At the beginning, all image objects were
classified into three auxiliary classes "nuclei" (dark), "white
space" (light) objects and "unclassified" objects respectively.
Nuclei Segmentation
[0102] The nuclei segmentation started from the second level. Three
color normalized layers were used to classify image objects as
"nuclei" and "unclassified" objects. The pathologically valid
nuclei (instances of the class "Nuclei") were formed from the
"nuclei" objects with the use of growing (adding the neighboring
"unclassified" objects to a nucleus) and fusion (merging of same
class objects into one object), and morphological opening/closing
algorithms were applied in order to improve nuclei
segmentation.
White Space Segmentation
[0103] The segmentation results from the second level were carried
over to the third level using a level copy operation. On that
level, all "unclassified" objects were classified to "white space"
and "unclassified" (remaining) objects, respectively. Image object
brightness was used as the primary object for classification. A set
of complications prevented the system from using brightness based
threshold as ultimate classifier of "white space" objects: "white
space" objects are not always "light", low contrast images produces
false "white space" objects, and "white space" area on the tissue
often filled with blood and other fluids.
[0104] In order to overcome the above outlined problems, actual
"white space" objects were composed with the use of the mentioned
growing, fusion and morphological opening algorithms. The obtained
"white space" objects were classified to (Levels 3 and 4): red
blood cells, sinusoids: elongated, contain red blood cells and
within certain distance from kupffer cells, fat vacuoles: round,
small and relatively dark "white space" objects, vessels:
relatively big "white space" objects with smooth shape, and fat:
remaining "white space" objects. All the remaining "unclassified"
image objects in Level 4 are classified as instances of
"cytoplasm". After fusion they form the cytoplasm area.
Morphological Object Segmentation
[0105] The morphological object segmentation is an example of a
high stage of the tissue image processing. The detected
histopathological basic objects such as hepatic nuclei and
cytoplasm were used to form the hepatocytes. The hepatocytes
formation algorithm may be outlined as follows. The hepatic nuclei
were used as seeds. A region growing algorithm was applied in order
to grow hepatocytes from the cytoplasm, fat and fat vacuoles class
objects. The cell continued growing until the following conditions
were met: a) two growing hepatic cells touch each other; and b) the
hepatic cell achieved a predefined size (measured as the distance
from the seed). In the case when two or more hepatic nuclei were
located close together, a modified growing algorithm kept the
hepatocytes isolated.
Object Measurements
[0106] The object measurement is the final stage of tissue image
processing. The measurements are quantifications of all segmented
histopathological basic objects and structures.
Histopathological Object Quantification
[0107] For each segmented class of the basic histopathological
objects of the tissue image, the following data were output: number
of the objects (n/a for cytoplasm area), class area relative to the
total tissue area (%), individual object statistics: min/max area,
average and standard deviation of the area values over the
image.
Morphological Object quantification
[0108] The unique phase of the tissue image processing is the
quantification of morphological objects. The analysis of
hepatocytes based on the fat and hepatic nuclei contents are
examples of such quantification.
[0109] The ratio of the fat (hepatic nuclei) area of a single
hepatocyte to the hepatocyte total area is determined. The ratio
constitutes the fat (hepatic nuclei) content. It serves as a
measurement of cell health: normal vs. abnormal. The theoretical
fat and hepatic nuclei contents range from 0 (a cell free of fat or
hepatic nuclei) to 1 (a cell replaced by fat or hepatic nuclei).
This range is divided into a number of bins. Coloring each cell
based on a color associated with its bin range produces the
steatotic PTM. The color changes from blue (a low content) through
yellow (a moderate content) to red (a high content).
[0110] The fat PTM was processed on the Level 5 objects and nuclei
density PTM on the Level 6 objects. All hepatocytes were classified
into ten classes: "Fat Ratio" 1-10 and "Hepatic Nuclei Ratio"
1-10.
[0111] The PDMs, hepatocytes area and basic object measurements
form a feature vector for biostatistical modeling.
Segmentation Accuracy
[0112] The global segmentation accuracy for all objects, as
measured by a pathologist's assessment, was 80%-90%.
[0113] The preferred embodiments described herein are provided to
enable any person skilled in the art to make and use the present
invention. The various modifications to these embodiments will be
readily apparent to those skilled in the art, and the generic
principles defined herein may be applied to other embodiments
without the use of the inventive faculty. Thus, the present
invention is not intended to be limited to the embodiments shown
herein but is to be accorded the widest scope consistent with the
principles and novel features disclosed herein. Accordingly, this
invention includes all modifications encompassed within the spirit
and scope of the invention as defined by the claims.
[0114] The automation of object extraction in embodiments of the
present invention create a high throughput capability that enables
analysis of serial sections for more accurate measurements.
Measurement results may be input into a relational database where
they can be statistically analyzed and compared across studies. As
part of the integrated process, results may also be imprinted on
the images themselves to facilitate auditing of the results. The
analysis may be fast, repeatable and accurate while allowing the
pathologist to control the measurement process.
* * * * *