U.S. patent application number 09/970610 was filed with the patent office on 2002-09-05 for neural network assisted multi-spectral segmentation system.
Invention is credited to Raz, Ryan.
Application Number | 20020123977 09/970610 |
Document ID | / |
Family ID | 21708431 |
Filed Date | 2002-09-05 |
United States Patent
Application |
20020123977 |
Kind Code |
A1 |
Raz, Ryan |
September 5, 2002 |
Neural network assisted multi-spectral segmentation system
Abstract
In a segmentation method and system, a plurality of digitized
images having different optical bands are acquired for the same
micrographic scene of a biological sample. Each digitized image has
a plurality of pixels having values from each digitized image. The
pixel values are processed to identify nuclear or cytoplasmic
material utilizing previously developed classification information
developed from at least one cell having known regions of nuclear or
cytoplasmic material. In a preferred embodiment, the neural network
comprises a hardware-encoded algorithm in the form of a look-up
table.
Inventors: |
Raz, Ryan; (Windsor,
CA) |
Correspondence
Address: |
LERNER, DAVID, LITTENBERG,
KRUMHOLZ & MENTLIK
600 SOUTH AVENUE WEST
WESTFIELD
NJ
07090
US
|
Family ID: |
21708431 |
Appl. No.: |
09/970610 |
Filed: |
October 4, 2001 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
09970610 |
Oct 4, 2001 |
|
|
|
09040378 |
Mar 18, 1998 |
|
|
|
09040378 |
Mar 18, 1998 |
|
|
|
PCT/CA96/00619 |
Sep 18, 1996 |
|
|
|
60003964 |
Sep 19, 1995 |
|
|
|
Current U.S.
Class: |
706/15 ;
706/924 |
Current CPC
Class: |
G06T 7/136 20170101;
G06T 7/11 20170101; G06V 20/69 20220101; G06T 7/155 20170101; G06T
2207/30024 20130101; G01N 15/1475 20130101; G06T 2207/10056
20130101; G06T 2207/20036 20130101; G06T 2207/20084 20130101 |
Class at
Publication: |
706/15 ;
706/924 |
International
Class: |
G06F 015/18 |
Claims
1. A method of analyzing cells, comprising: a. providing a
plurality of digitized images of at least one cell of regions of
unknown nuclear or cytoplasmic material, each digitized image being
formed from an optical image having a plurality of pixels
associated therewith, each digitized image being formed in a narrow
band of optical wavelength different from the other digitized
images, each of the plurality of pixels having values from each of
the digitized images; and b. analyzing the values for each pixel to
identify nuclear and cytoplasmic material of the at least one cell
of unknown regions of nuclear or cytoplasmic material, said
analyzing step utilizing previously developed classification
information for discriminating nuclear or cytoplasmic material, the
previously developed classification information being developed
from at least one cell of known regions of nuclear or cytoplasmic
material.
2. The method of claim 1, wherein the step of analyzing the values
for each pixel includes utilizing a classifier trained on a
training set of data developed from images having known regions of
nuclear and cytoplasmic material.
3. The method of claim 1, wherein the previously developed
classification information includes values for pixels stored in a
look-up table in a memory storage device.
4. The method of claim 3, wherein the previously developed
classification information has memory addresses in the look-up
table, the memory addresses comprising a concatenation of the
values from each of the digitized images representing the same
region of the at least one cell, each of the digitized images being
drawn from the band of optical wavelength.
5. The method of claim 1 wherein the previously developed
classification information includes a predetermined discriminant
between values for pixels representing regions of nuclear and
cytoplasmic material.
6. The method of claim 1, further comprising the step of assigning
a classification to each pixel as representing nuclear or
cytoplasmic material.
7. The method of claim 6, further comprising: a. forming an
absorption map from each of the digitized images to represent the
light absorption characteristics associated with each value for
each pixel before the step of analyzing; and b. assigning a
classification to each pixel based upon absorption
characteristics.
8. The method of claim 7, wherein the step of forming an absorption
map comprises applying a formula to each of the digitized
images.
9. The method of claim 5, wherein the step of analyzing includes
applying a linear discriminant analysis to define a linear boundary
between values for pixels representing regions of nuclear and
cytoplasmic material.
10. The method of claim 9, wherein the linear discriminant analysis
discriminates between pixels of nuclear material and at least two
types of cytoplasmic material.
11. The method of claim 1, wherein the at least one cell of unknown
nuclear or cytoplasmic material comprises a cellular sample
prepared according to the Papanicolaou staining procedure.
12. The method of claim 1, further comprising developing the
previously developed classification information by: a. providing a
plurality of digitized images of at least one cell of regions of
known nuclear or cytoplasmic material, each digitized image being
formed from an optical image having a plurality of pixels
associated therewith, each digitized image being formed in a narrow
band of optical wavelength different from the other digitized
images, each of the plurality of pixels having values from each of
the digitized images; and b. assigning a classification to each
pixel as representing regions of nuclear or cytoplasmic material;
and c. storing the classifications in a look-up table in a memory
storage device.
13. The method of claim 12, wherein the step of storing includes
storing the classifications in locations of the look-up table, the
locations having addresses corresponding to the pixels.
14. The method of claim 1, wherein the step of analyzing comprises
neural network processing of the values for pixels of the digitized
images.
15. The method of claim 14, wherein the neural network processing
comprises accepting one or more inputs into at least one processing
element, multiplying the inputs by weighing factors, and applying a
formula to the weighed inputs to provide an output to a plurality
of other processing elements.
16. The method of claim 15, wherein each value for a pixel is
accepted and processed by a different one of the at least one
processing elements.
17. A method of analyzing cells, comprising: a. providing a
plurality of digitized images of at least one cell, each digitized
image being formed in a narrow band of optical wavelength different
from the other digitized images and including at least one first
digitized image in a wavelength of between 525 to 575 nanometers,
at least one second digitized image in a wavelength of between 565
to 582 nanometers, and at least one third digitized image in a
wavelength of between 625 to 635 nanometers; and b. analyzing the
digitized images to identify nuclear and cytoplasmic material of
the at least one cell.
18. The method of claim 17, wherein the at least one first
digitized image has a wavelength of between 525 to 535 nanometers,
the at least one second digitized image has a wavelength of between
572 to 582 nanometers, and the at least one third digitized image
has a wavelength of between 625 to 635 nanometers.
19. The method of claim 17, wherein the at least one first
digitized image has a wavelength of between 535 to 545 nanometers,
the at least one second digitized image has a wavelength of between
572 to 582 nanometers, and the at least one third digitized image
has a wavelength of between 625 to 635 nanometers.
20. The method of claim 17, wherein the at least one first
digitized image has a wavelength of between 565 to 575 nanometers,
the at least one second digitized image has a wavelength of between
565 to 575 nanometers, and the at least one third digitized image
has a wavelength of between 625 to 635 nanometers.
21. The method of claim 17, wherein the step of analyzing the
pixels includes utilizing a classifier trained on a set of data
developed from images having known regions of nuclear and
cytoplasmic material.
22. The method of claim 17, wherein the step of analyzing includes
analyzing the digitized images based upon previously developed
classification information.
23. The method of claim 22, wherein the step of analyzing comprises
analyzing values for each pixel, each pixel having values from each
of the digitized images.
24. The method of claim 17, further comprising the step of
assigning a classification to each pixel as representing regions of
nuclear or cytoplasmic material.
25. The method of claim 17, wherein the at least one cell comprises
a cellular sample prepared according to the Papanicolaou staining
procedure.
26. The method of claim 22, wherein the previously developed
classification information is developed from at least one cell of
known regions of nuclear or cytoplasmic material for analyzing
cells of unknown regions of nuclear or cytoplasmic material.
27. The method of claim 26, further comprising the step of storing
the previously developed classification information in a look-up
table in an electronic memory device.
28. The method of claim 17, wherein the step of analyzing comprises
neural network processing of the digitized images.
Description
CROSS-REFERENCE TO RELATED APPLICATION
[0001] This application is a continuation of U.S. application Ser.
No. 09/040,378, filed Mar. 18, 1998, the disclosure of which is
hereby incorporated by reference herein. The '378 application is
the national phase of International Application No. PCT/CA
96/00619, filed Sep. 18, 1996, the disclosure of which is hereby
incorporated by reference herein. This application also claims
benefit of U.S. Provisional Application No. 60/003,964, filed Sep.
19, 1995, the disclosure of which is hereby incorporated by
reference herein.
FIELD OF THE INVENTION
[0002] The present invention relates to automated diagnostic
techniques in medicine and biology, and more particularly to
multi-spectral segmentation of nuclear and cytoplasmic objects.
BACKGROUND OF THE INVENTION
[0003] Automated diagnostic systems in medicine and biology often
rely on the visual inspection of microscopic images. Known systems
attempt to mimic or imitate the procedures employed by humans. An
appropriate example of this type of system is an automated
instrument designed to assist a cytotechnologist in the review or
diagnosis of Pap smears. In its usual operation such a system will
rapidly acquire microscopic images of the cellular content of the
Pap smears and then subject them to a battery of image analysis
procedures. The goal of these procedures is the identification of
images that are likely to contain unusual or potentially abnormal
cervical cells.
[0004] The image analysis techniques utilized by these automated
instruments are similar to the procedures consciously, and often
unconsciously, performed by the human cytotechnologist. There are
three distinct operations that must follow each other for this type
of evaluation: (1) segmentation; (2) feature extraction; and (3)
classification.
[0005] The segmentation is the delineation of the objects of
interest within the micrographic image. In addition to the cervical
cells required for an analysis there is a wide range of
"background" material, debris and contamination that interferes
with the identification of the cervical cells and therefore must be
delineated. Also for each cervical cell, it is necessary to
delineate the nucleus with the cytoplasm.
[0006] The Feature Extraction operation is performed after the
completion of the segmentation operation. Feature extraction
comprises characterizing the segmented regions as a series of
descriptors based on the morphological, textural, densitometric and
calorimetric attributes of these regions.
[0007] The Classification step is the final step in the image
analysis. The features extracted in the previous stage are used in
some type of discriminant-based classification procedure. The
results of this classification are then translated into a
"diagnosis" of the cells in the image.
[0008] of the three stages outlined above, segmentation is the most
crucial and the most difficult. This is particularly true for the
types of images typically encountered in medical or biological
specimens.
[0009] In the case of a Pap smear, the goal of segmentation is to
accurately delineate the cervical cells and their nuclei. The
situation is complicated not only by the variety of cells found in
the smear, but also by the alterations in morphology produced by
the sample preparation technique and by the quantity of debris
associated with these specimens. Furthermore, during preparation it
is difficult to control the way cervical cells are deposited on the
surface of the slide which as a result leads to a large amount of
cell overlap and distortion.
[0010] Under these circumstances a segmentation operation is
difficult. One known way to improve the accuracy and speed of
segmentation for these types of images involves exploiting the
differential staining procedure associated with all Pap smears.
According to the Papanicolaou protocol the nuclei are stained dark
blue while the cytoplasm is stained anything from a blue-green to
an orange-pink. The Papanicolaou Stain is a combination of several
stains or dyes together with a specific protocol designed to
emphasize and delineate cellular structures of importance for
pathological analysis. The stains or dyes included in the
Papanicolaou Stain are Haematoxylin, Orange G and Eosin Azure (a
mixture of two acid dyes, Eosin Y and Light Green SF Yellowish,
together with Bismark Brown). Each stain component is sensitive to
or binds selectively to a particular cell structure or material.
Haematoxylin binds to the nuclear material coloring it dark blue.
Orange G is an indicator of keratin protein content. Eosin Y stains
nucleoli, red blood cells and mature squamous epithelial cells.
Light Green SF yellowish acid stains metabolically active
epithelial cells. Bismark Brown stains vegetable material and
cellulose.
[0011] The combination of these stains and their diagnostic
interpretation has evolved into a stable medical protocol which
predates the advent of computer-aided imaging instruments.
Consequently, the dyes present a complex pattern of spectral
properties to standard image analysis procedures. Specifically, a
simple spectral decomposition based on the optical behavior of the
dyes is not sufficient on its own to reliably distinguish the
cellular components within an image. The overlap of the spectral
response of the dyes is too large for this type of straight-forward
segmentation.
[0012] The use of differential staining characteristics is only the
means to the end in the solution to the problem of segmentation. Of
equal importance is the procedure for handling the information
provided by the spectral character of the cellular objects when
making a decision concerning identity.
[0013] In the art, attempts have been made to automate diagnostic
procedures, however, there remains a need for a system for
performing the segmentation process.
BRIEF SUMMARY OF THE INVENTION
[0014] The present invention provides a Neural-Network Assisted
Multi-Spectral Segmentation (also referred to as the NNA-MSS)
method and system.
[0015] The first stage according to the present invention comprises
the acquisition of three images of the same micrographic scene.
Each image is obtained using a different narrow band-pass optical
filter which has the effect of selecting a narrow band of optical
wavelengths associated with distinguishing absorption peaks in the
stain spectra. The choice of optical wavelength bands is guided by
the degree of separation afforded by these peaks when used to
distinguish the different types of cellular material on the slide
surface.
[0016] The second stage according to the invention comprises a
neural-network (trained on an extensive set of typical examples) to
make decisions on the identity of material already deemed to be
cellular in origin. The neural network decides whether or not a
picture element in the digitized image is nuclear or not nuclear in
character. With the completion of this step the system can continue
on applying a standard range of image processing techniques to
refine the segmentation. The relationship between the cellular
components and the transmission intensity of the light images in
each of the three spectral bands is a complex and non-linear one.
By using a neural network to combine the information from these
three images it is possible to achieve a high degree of success in
separating the cervical cell from the background and the nuclei
from the cytoplasm. A success that would not be possible with a set
of linear operations alone.
[0017] The diagnosis and evaluation of Pap smears is aided by the
introduction of a differential staining procedure called the
Papanicolaou Stain. The Papanicolaou Stain is a combination of
several stains or dyes together with a specific protocol designed
to emphasize and delineate cellular structures of importance to
pathological analysis. The stains or dyes included in the
Papanicolaou Stain are Haematoxylin, Orange G and Eosin Azure (a
mixture of two acid dyes, Eosin Y and Light Green SF Yellowish,
together with Bismarck Brown). Each stain component is sensitive to
or binds selectively to a particular cellular structure or
material. Haematoxylin binds to the nuclear material coloring it
dark blue; Orange G is an indicator of keratin protein content;
Eosin Y stains nucleoli, red blood cells and mature squamous
epithelial cells; Light Green SF yellowish stains metabolically
active epithelial cells; Bismarck Brown stains vegetable material
and cellulose.
[0018] According to another aspect of the invention, three optical
wavelength bands are used in a complex procedure to segment
Papanicolaou-stained epithelial cells in digitized images. The
procedure utilizes standard segmentation operations (erosion,
dilation, etc.) together with the neural-network to identify the
location of nuclear components in areas already determined to be
cellular material.
[0019] The purpose of the segmentation is to extract the cellular
objects, i.e. to distinguish the nucleus of the cell from the
cytoplasm. According to this segmentation the multi-spectral images
are divided into two classes: cytoplasm objects and nuclear
objects, which are separated by a multi-dimensional threshold t
which comprises a 3-dimensional space.
[0020] The neural network according to the invention comprises a
Probability Projection Neural Network (PPNN). The PPNN according to
the present invention features fast training for a large volume of
data, processing of multi-modal non-Gaussian data distribution,
good generalization simultaneously with high sensitivity to small
clusters of patterns representing the useful subclasses of cells.
In another aspect, the PPNN is implemented as a hardware-encoded
algorithm.
[0021] A method of analyzing cells comprises providing a plurality
of digitized images of at least one cell of regions of unknown
nuclear or cytoplasmic material. Each digitized image is formed
from an optical image having a plurality of pixels associated
therewith. Each digitized image is formed in a narrow band of
optical wavelength different from the other digitized images. Each
of the plurality of pixels has values from each of the digitized
images. The method further includes analyzing the values for each
pixel to identify nuclear and cytoplasmic material of the at least
one cell of unknown regions of nuclear or cytoplasmic material. The
analyzing step utilizes previously developed classification
information for discriminating nuclear or cytoplasmic material. The
previously developed classification information is developed from
at least one cell of known regions of nuclear or cytoplasmic
material.
[0022] The step of analyzing the values for each pixel preferably
includes utilizing a classifier trained on a training set of data
developed from images having known regions of nuclear and
cytoplasmic material.
[0023] In a preferred embodiment, the previously developed
classification information preferably includes values for pixels
stored in a look-up table in a memory storage device. The
previously developed classification information preferably has
memory addresses in the look-up table, the memory addresses
comprising a concatenation of the values from each of the digitized
images representing the same region of the at least one cell. Each
of the digitized images are drawn from a band of optical
wavelength.
[0024] In a preferred embodiment, the previously developed
classification information includes a predetermined discriminant
between values for pixels representing regions of nuclear and
cytoplasmic material.
[0025] The method preferably includes the step of assigning a
classification to each pixel as representing nuclear or cytoplasmic
material.
[0026] In a preferred embodiment, an absorption map is formed from
each of the digitized images to represent the light absorption
characteristics associated with each value for each pixel before
the step of analyzing the values for each pixel. A classification
is assigned to each pixel based upon absorption characteristics.
The step of forming an absorption map preferably comprises applying
a formula to each of the digitized images.
[0027] The step of analyzing may include applying a linear
discriminant analysis to define a linear boundary between values
for pixels representing regions of nuclear and cytoplasmic
material. The linear discriminant analysis preferably discriminates
between pixels of nuclear material and at least two types of
cytoplasmic material.
[0028] The at least one cell of unknown nuclear or cytoplasmic
material may comprise, for example, a cellular sample prepared
according to the Papanicolaou staining procedure.
[0029] The previously developed classification information may be
developed by: providing a plurality of digitized images of at least
one cell of regions of known nuclear or cytoplasmic material, each
digitized image being formed from an optical image having a
plurality of pixels associated therewith, each digitized image
being formed in a narrow band of optical wavelength different from
the other digitized images, each of the plurality of pixels having
values from each of the digitized images; assigning a
classification to each pixel as representing regions of nuclear or
cytoplasmic material; and storing the classifications in a look-up
table in a memory storage device.
[0030] The step of storing preferably includes storing the
classifications in locations of the look-up table, the locations
having addresses corresponding to the pixels. In a preferred
embodiment, the step of analyzing comprises neural network
processing of the values for pixels of the digitized images. The
neural network processing comprises accepting one or more inputs
into at least one processing element, multiplying the inputs by
weighing factors, and applying a formula to the weighed inputs to
provide an output to a plurality of other processing elements. Each
value for a pixel is preferably accepted and processed by a
different one of the at least one processing elements.
[0031] In another aspect of the present invention, a method of
analyzing cells comprises providing a plurality of digitized images
of at least one cell. Each digitized image is formed in a narrow
band of optical wavelength different from the other digitized
images and including: at least one first digitized image in a
wavelength of between 525 to 575 nanometers; at least one second
digitized image in a wavelength of between 565 to 582 nanometers;
and at least one third digitized image in a wavelength of between
625 to 635 nanometers. The digitized images are analyzed to
identify nuclear and cytoplasmic material of the at least one
cell.
[0032] For example, the at least one first digitized image may have
a wavelength of between 525 to 535 nanometers, the at least one
second digitized image may have a wavelength of between 572 to 582
nanometers, and the at least one third digitized image may have a
wavelength of between 625 to 635 nanometers.
[0033] For example, the at least one first digitized image may have
a wavelength of between 535 to 545 nanometers, the at least one
second digitized image may have a wavelength of between 572 to 582
nanometers, and the at least one third digitized image may have a
wavelength of between 625 to 635 nanometers.
[0034] For example, the at least one first digitized image may have
a wavelength of between 565 to 575 nanometers, the at least one
second digitized image may have a wavelength of between 565 to 575
nanometers, and the at least one third digitized image may have a
wavelength of between 625 to 635 nanometers.
[0035] The step of analyzing the pixels preferably includes
utilizing a classifier trained on a set of data developed from
images having known regions of nuclear and cytoplasmic material.
The step of analyzing preferably includes analyzing the digitized
images based upon previously developed classification
information.
[0036] The step of analyzing may comprise analyzing values for each
pixel, each pixel having values from each of the digitized
images.
[0037] A classification is preferably assigned to each pixel as
representing regions of nuclear or cytoplasmic material. The at
least one cell may comprise, for example, a cellular sample
prepared according to the Papanicolaou staining procedure.
[0038] The previously developed classification information is
preferably developed from at least one cell of known regions of
nuclear or cytoplasmic material for analyzing cells of unknown
regions of nuclear or cytoplasmic material.
[0039] The method, in certain preferred embodiments, includes the
step of storing the previously developed classification information
in a look-up table in an electronic memory device. In certain
preferred embodiments, the step of analyzing may comprise neural
network processing of the digitized images.
[0040] In a further aspect, the present invention provides a method
for identifying nuclear and cytoplasmic objects in a biological
specimen, said method comprising the steps of: (a) acquiring a
plurality of images of said biological specimen; (b) identifying
cellular material from said images and creating a cellular material
map; (c) applying a neural network to said cellular material map
and classifying nuclear and cytoplasmic objects from said
images.
[0041] In another aspect, the present invention provides a system
for identifying nuclear and cytoplasmic objects in biological
specimen, said system comprising: (a) image acquisition means for
acquiring a plurality of images of said a biological specimen; (b)
processing means for processing said images and generating a
cellular material map identifying cellular material; (c) neural
processor means for processing said cellular material map and
including means for classifying nuclear and cytoplasmic objects
from said images.
[0042] In another aspect, the present invention provides a
hardware-encoded neural processor for classifying input data, said
hardware-encoded processor comprising: (a) a memory having a
plurality of addressable storage locations; (b) said addressable
storage locations containing classification information associated
with the input data; (c) address generation means for generating an
address from said input data for accessing the classification
information stored in said memory for selected input data.
[0043] A preferred embodiment of the present invention will now be
described, by way of example, with reference to the following
specification, claims, and drawings.
BRIEF DESCRIPTION OF THE DRAWINGS
[0044] FIG. 1 shows in flow chart form a neural network assisted
multi-spectral segmentation method according to the present
invention;
[0045] FIG. 2 shows in diagrammatic form a processing element for
the neural network;
[0046] FIG. 3 shows in diagrammatic form a neural network
comprising the processing elements of FIG. 2;
[0047] FIG. 4 shows in diagrammatic form a training step for the
neural network;
[0048] FIG. 5 shows in flow chart form a clustering algorithm for
the neural network according to the present invention; and
[0049] FIG. 6 shows a hardware implementation for the neural
network according to the present invention.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT
[0050] The present invention provides a Neural Network Assisted
Multi-Spectral Segmentation (also referred to as NNA-MSS) system
and method. The multi-spectral segmentation method is related to
that described and claimed in co-pending International Patent
Application No. CA96/00477 filed Jul. 18, 1996 and in the name of
the applicant.
[0051] The NNA-MSS according to the present invention is
particularly suited to Papanicolaou-stained gynaecological smears
and will be described in this context. It is however to be
understood that the present invention has wider applicability to
applications outside of Papanicolaou-Stained smears.
[0052] Reference is first made to FIG. 1 which shows in flow chart
a Neural Network Assisted Multi-Spectral Segmentation (NNAMSS)
method 1 according to the present invention.
[0053] The first step 10 involves inputting three digitized images,
i.e. micrographic scenes, of a cellular specimen. The images are
taken in each of the three narrow optical bands: 540.+-.5 nm;
577.+-.5 nm and 630.+-.5 nm. (The images are generated by an
imaging system (not shown) as will be understood by one skilled in
the art, and thus need not be described in detail here.) The images
are next processed by the multi-segmentation method 1 and neural
network as will be described.
[0054] As shown in FIG. 1, the images are subjected to a leveling
operation (block 12). The leveling operation 12 involves removing
the spatial variations in the illumination intensity from the
images. The leveling operation is implemented as a simple
mathematical routine using known image processing techniques. The
result of the leveling operation is a set of 8-bit digitized images
with uniform illumination across their fields.
[0055] The 8-bit digitized images first undergo a series of
processing steps to identify cellular material in the digitized
images. The digitized images are then processed by the neural
network to segment the nuclear objects from the cytoplasm
objects.
[0056] Referring to FIG. 1, following the leveling operation 12 the
next operation comprises a threshold procedure block 14. The
threshold procedure involves analyzing the leveled images in a
search for material of cellular origin. The threshold procedure 14
is applied to the 530 nm and 630 nm optical wavelength bands and
comprises identifying material in the image of cellular origin as
regions of the digitized image that fall within a range of specific
digital values. The threshold procedure 14 produces a single binary
"map" of the image where the single binary bit identifies regions
that are, or are not, cellular material.
[0057] The threshold operation 14 is followed by a dilation
operation (block 16). The dilation operation 16 is a conventional
image processing operation which modifies The binary map of
cellular material generated in block 14. The dilation operation
allows the regions of cellular material to grow or dilate by one
pixel in order to fill small voids in large regions. Preferably,
the dilation operation 16 is modified with the condition that the
dilation does not allow two separate regions of cellular material
to join to make a single region, i.e. a "no-join" condition. This
condition allows the accuracy of the binary map to be preserved
through dilation operation 16. Preferably, the dilation operation
is applied twice to ensure a proper filling of voids. The result of
the dilation operations 16 is a modified binary map of cellular
material.
[0058] As shown in FIG. 1, the dilation operation 16 is followed by
an erosion operation (block 18). The erosion operation 18 brings
the modified binary map of cellular material (a result of the
dilation operation 16) back to its original boundaries. The erosion
operation 18 is implemented using conventional image processing
techniques. The erosion operation 18 allows the cellular boundaries
in the binary image to shrink or erode but will not affect the
filled voids. Advantageously, the erosion operation 18 has the
additional effect of eliminating small regions of cellular material
that are not important to the later diagnostic analysis. The result
of the erosion operation 18 is a final binary map of the regions in
the digitized image that are cytoplasm.
[0059] The next stage according to the invention, is the operation
of the neural network at block 20. The neural network 20 is applied
to the 8-bit digitized images, with attention restricted to those
regions that lie within the cytoplasm as determined by the final
binary cytoplasm map generated as a result of the previous
operations. The neural network 20 makes decisions concerning the
identity of individual picture elements (or "pixels") in the binary
image as either being part of a nucleus or not part of a nucleus.
The result of the operation of the neural network is a digital map
of the regions within the cytoplasm that are considered to be
nuclear material. The nuclear material map is then subjected to
further processing. The neural network 20 according to the present
invention is described in detail below.
[0060] Following the application of the neural network 20, the
resulting nuclear material map is subjected to an erosion operation
(block 22). The erosion operation 22 eliminates regions of the
nuclear material map that are too small to be of diagnostic
significance. The result is a modified binary map of nuclear
regions.
[0061] The modified binary map resulting from the erosion operation
22 is then subjected to a dilation operation (block 24). The
dilation operation 24 is subject to a no-join condition, such that,
the dilation operation does not allow two separate regions of
nuclear material to join to make a single region. In this way the
accuracy of the binary map is preserved notwithstanding the
dilation operation. The dilation operation 24 is preferably applied
twice to ensure a proper filling of voids. The result of these
dilation operations is a modified binary map of nuclear
material.
[0062] Following the dilation operation 24, an erosion operation is
applied (block 26). Double application of the erosion operation 26
eliminates regions of the nuclear material in the binary map that
are too small to be of diagnostic significance. The result is a
modified binary map of nuclear regions.
[0063] The remaining operations involve constructing a binary map
comprising high gradients, i.e. boundaries, of pixel intensity, in
order to sever nuclear regions that share high gradient boundaries.
The presence of these high gradient boundaries is evidence of two,
closely spaced but separate nuclei.
[0064] The first step in severing the high-gradient boundaries in
the nuclear map is to construct a binary map of these high gradient
boundaries using a threshold operation (block 28) applied to a
Sobel map.
[0065] The Sobel map is generated by applying the Sobel gradient
operator to the 577 nm 8-bit digitized image to determine regions
of that image that contain high gradients of pixel intensity (block
29). (The 8-bit digitized image for the 577 nm band was obtained
from the leveling operation in block 12.) The result of the Sobel
operation in block 29 is an 8-bit map of gradient intensity.
[0066] Following the threshold Sobel operation 28, a logical NOT
operation is performed (block 30). The logical NOT operation 30
determines the coincidence of the two states, high-gradients and
nuclei, and reverses the pixel value of the nuclear map at the
point of the coincidence in order to eliminate it from regions that
are presumed to be nuclear material. The result of this logical
operation is a modified nuclear map.
[0067] The modified nuclear map is next subjected to an erosion
operation (block 32). The erosion operation 32 eliminates regions
in the modified nuclear map that are too small to be of diagnostic
significance. The result is a modified binary map of nuclear
regions.
[0068] After the application of the gradient technique for severing
close nuclear boundaries (blocks 28 and 30) and the erosion
operation (block 32) for clearing the image of insignificant
regions, the binary map of nuclear regions is dramatically altered.
To restore the map to its original boundaries while preserving the
newly-formed separations, the process applies a dilation operation
at block 34. The dilation operation 34 includes the condition that
no two nuclear regions will become joined as they dilate and that
no nuclear region will be allowed to grow outside its old boundary
as defined by the binary map that existed before the Sobel
procedure was applied. The dilation operation 34 is preferably
applied four times. The result is a modified binary map of nuclear
material.
[0069] With the application of the dilation operation 34, the
nuclear segmentation procedure according to the multi-spectral
segmentation process 1 is complete and the resulting binary nuclear
map is labeled in block 36, and if required further image
processing is applied.
[0070] As described above, the operation at block 20 in FIG. 1
comprises neural network processing of the digitized images. In
general, the neural network 20 is a highly parallel, distributed,
information processing system that has the topology of a directed
graph. The network comprises a set of "nodes" and series of
"connections" between the nodes. The nodes comprise processing
elements and the connections between the nodes represent the
transfer of information from one node to another.
[0071] Reference is made to FIG. 2 which shows a node or processing
element 100a for a backpropagation neural network 20. Each of the
nodes 100a accepts one or more inputs 102 shown individually as
a.sub.1, a.sub.2, a.sub.3 . . . a.sub.n in FIG. 2. The inputs 102
are taken into the node 100a and each input 102 is multiplied by
its own mathematical weighting factor before being summed together
with the threshold factor for the processing element 100a. The
processing element 100a then generates a single output 104 (i.e.
b.sub.j) according to the "transfer function" being used in the
network 20. The output 104 is then available as an input to other
nodes or processing elements, for example processing elements 100b,
100c, 100d, 100e and 100f as depicted in FIG. 1.
[0072] The transfer function may be any suitable mathematical
function but it is usual to employ a "sigmoid" function. The
relationship between the inputs 102 into the node 100 and the
output 104 is given by expression (1) as follows:
b={.SIGMA.w.sub.jia.sub.1-.theta..sub.i}.sup.-1 (1)
[0073] where b.sub.j is the output 104 of the node 100, a.sub.i is
the value of the input 102 to the node labeled "I", w.sub.ji is the
weighting given to that input 102, and .theta..sub.j is the
threshold value for the node 100. In the present application, the
transfer function is modeled after a sigmoid function.
[0074] In its general form, the nodes or processing elements for
the neural network are arranged in a series of layers denoted by
106, 108 and 110 as shown in FIG. 3. The first layer 106 comprises
nodes or processing elements 112 shown individually as 112a, 112b,
112c, 112d and 112e. The first layer 106 is an input layer and
accepts the information required for a decision.
[0075] The second layer 108 in the neural network 20 is known as
the hidden layer and comprises processing elements 114 shown
individually as 114a, 114b, 114c, 114d and 114e. All of the nodes
112 in the input layer 106 are connected to all of the nodes 114 in
the hidden layer 108. It will be understood that there may be more
than one hidden layer, with each node in the successive layer
connected to each node of the previous layer. For convenience only
one hidden layer 108 is shown in FIG. 3.
[0076] The (last) hidden layer 108 leads to the output layer 110.
The output layer 110 comprises processing elements 116 shown
individually as 116a, 116b, 116c, 116d and 116e in FIG. 3. Each
node 114 of the (last) hidden layer 108 (FIG. 3) is connected to
each node 116 of the output layer 110. The output layer 110 renders
the decision to be interpreted by subsequent computing
machinery.
[0077] The strength of the neural network architecture is its
ability to generalize based on previous training of particular
examples. In order to take advantage of this, the neural network is
presented a series of examples of the type of objects that it is
destined to classify. The backpropagation neural network organizes
itself by altering the multiplicity of its connection weights and
thresholds according to its success in rendering a correct
decision. This is called supervised learning wherein the operator
provides the network with the information regarding its success in
classification. The network relies on a standard general rule for
modifying its connection weights and thresholds based on the
success of its performance, i.e. back-propagation.
[0078] In the context of the multi-spectral segmentation process,
the multi-spectral images are divided into two classes:
[0079] C.sub.o--cytoplasm and C.sub.1--nuclear, separated by the
multi-dimensional threshold t which comprises a 3-dimensional
space. The distribution of the pixels for the nuclear and cytoplasm
objects is complex and the 3-D space comprises numerous clusters
and non-overlapped regions. It has been found that the optimal
threshold has a complex non-linear surface in the 3-D space, and
the neural network according to the present invention provides the
means for finding the complex threshold surface in the 3-D space in
order to segment the nuclear and cytoplasmic objects.
[0080] According to this aspect of the invention, the neural
network 20 comprises an input layer 106, a single hidden layer 108,
and an output layer 110. The input layer 106 comprises three nodes
or processing elements 112 (FIG. 3) for each of the three 8-bit
digitized values for the particular pixel being examined. (The
three digitized values arise from the three leveled images
collected in each of the three optical bands, as described above
with reference to FIG. 1.) The output layer 110 comprises a single
processing element 116 (FIG. 3) which indicates whether the pixel
under examination is or is not part of the nucleus.
[0081] Before the neural network 20 can be successfully operated
for decision-making it must first be "trained" in order to
establish the proper combination of weights and thresholds. The
training is performed outside of the segmentation procedure on a
large set of examples. Errors made in the classification of pixels
in the examples are "back-propagated" as corrections to the
connection weights and the threshold values in each of the
processing units. Once the classification error is acceptable the
network is "frozen" at these weight and threshold values and it is
integrated as a simple algebraic operation into the segmentation
procedure as shown at block 20 in FIG. 1.
[0082] In a preferred embodiment, the neural network 20 according
to the invention comprises a probability Projection Neural Network
which will also be referred to as a PPNN. The PPNN according to the
present invention features fast training for a large volume of
data, processing of multi-modal non-Gaussian data distribution,
good generalization simultaneously with high sensitivity to small
clusters of patterns representing the useful subclasses of cells.
In another aspect, the PPNN is well-suited to a hardware-encoded
implementation.
[0083] The PPNN according to the invention utilizes a Probability
Density Function (PDF) estimator. As a result, the PPNN is suitable
for use as a Probability Density Function estimator or as a general
classifier in pattern recognition. The PPNN uses the training data
to create an N-dimensional PDF array which in turn is used to
estimate the likelihood of a feature vector being within the given
classes as will now be described.
[0084] To create and train the PPN network, the input space is
partitioned into m.times.m.times. . . . m discrete nodes (if the
discrete input space is known, then m is usually selected less than
the range). For example, for a 3-D PDF array creating a
2.sup.6.times.2.sup.6.times.2.sup.6 grid is sufficient.
[0085] As shown in FIG. 4, the next step involves mapping or
projecting the influence of the each training pattern to the
neighbor nodes. This is accomplished according to expression (2) as
shown below: 1 P j [ o , 1 , , n - 1 ] = P j - 1 [ o , 1 , , n - 1
] + d j [ o , 1 , , n - 1 ] . 1 , if r x - 0 0 , if r k r 0 d j [ o
, 1 , , n - 1 ] = { 1 - r x i = 0 2 n - 1 ( 1 - r 1 ) if r k < r
o ( 2 )
[0086] where P.sub.j [.chi..sub.o, .chi..sub.1, . . .
.chi..sub.n-1] is the current value of the [.chi..sub.o,
.chi..sub.1, . . . .chi..sub.n-1] node after the j'th iteration;
d.sub.j [.chi..sub.o, .chi..sub.1, . . . .chi..sub.n-1] represents
the influence of j'th input pattern to the [.chi..sub.o,
.chi..sub.1, . . . .chi..sub.n-1] node; r.sub.k is the distance
from the pattern to the k'th node; r.sub.0 is the minimum distance
between two neighbor nodes; and n is the dimension of the
space.
[0087] From expression (1), it will be appreciated that 2 j 2 n d k
( j ) - 1
[0088] represents the normalized values.
k=1
[0089] Once the accumulation of P.sub.N [.chi..sub.o, .chi..sub.1,
. . . .chi..sub.n-1] (where j=N--number of the training patterns)
is completed, a normalization operation is performed to obtain the
total energy value for PPNN E.sub.PPN-1. The normalized values
(i.e. P*) for PPNN are calculated according to expression (3) as
follows:
P*.sub.N [.chi..sub.o, .chi..sub.1, . . . .chi..sub.n-1]=P.sub.N
[.chi..sub.o, .chi..sub.1, . . . .chi..sub.n-1]/N (3)
[0090] For feed-forward calculations the trained and normalized
nodes P*.sub.N [.chi..sub.o, .chi..sub.1, . . . .chi..sub.n-1] and
the reverse mapping are utilized according to expression (4) given
below, 3 h j [ o , n - 1 ] - i = 0 2 n - 1 p ( i ) [ o , 1 N , , n
- 1 ] d j ( f ) [ o , 1 , , n - 1 ] , ( 4 )
[0091] where d.sub.j.sup.(i) [.chi..sub.o, .chi..sub.1, . . .
.chi..sub.n-1] are calculated according to expression (1)
above.
[0092] To solve a two class (i.e. C.sub.0--cytoplasm and
C.sub.1--nuclear) application using the PPNN according to the
present invention, two networks must be trained for each class
separately, that is, P.sub.CO [.chi..sub.o, .chi..sub.1, . . .
.chi..sub.n-1] and P.sub.C1 [.chi..sub.o, .chi..sub.1, . . .
.chi..sub.n-1]. Because both PPNN are normalized, they can be
joined together according to expression (5) below as follows:
P.sub.C0/C1 [.chi..sub.o, .chi..sub.1, . . . .chi..sub.n-1] and
P*.sub.C0 [.chi..sub.o, .chi..sub.1, . . . .chi..sub.n-1]-P*.sub.C1
[.chi..sub.o, .chi..sub.1, . . . .chi..sub.n-1] (5)
[0093] The final decision from expressions (4) and (5) is given
by
C.sub.0, if h.sub.j>0
Pattern.sub.j .epsilon.C.sub.1, if h.sub.j.ltoreq.0 (6)
[0094] While the PPNN according to the present invention is
particularly suited to handle multi-modal data distributions, in
many practical situations there will be an unbalanced data set.
This means that some clusters will contain less data samples than
other clusters and as a result some natural clusters which were
represented with a small number of patterns could be lost after
PPNN joining. To solve this problem there is provided an algorithm
which equalizes all natural clusters according to another aspect of
the invention.
[0095] Reference is next made to FIG. 5, which shows in flow chart
form an embodiment of a clustering algorithm 200 according to the
present invention. All training patterns, i.e. N samples, in block
202 and a given number (i.e. "K") of clusters in block 204 are
applied to a K-mean clustering operation block 206. The clustering
operation 206 clusters the input data and generates clusters 1
through K (block 208). Next, all the training data which belongs to
an i.sup.th-cluster is extracted into a separate subclass. For each
sub-class of training data, a normalized PPNN, i.e. E.sub.i=1, is
created (block 210). The final operation in the clustering
algorithm comprises joining all of the K PPNN's together and
normalizing the resulting PPNN by dividing all nodes by the number
of clusters (block 212). The operation performed in block 212 may
be expressed as follows:
E=(E.sub.1+ . . . +E.sub.k)/K-1
[0096] It will also be understood that the clustering algorithm 200
may be implemented to the each class separately before creating the
final classifier according the expression (6) above, as follows.
The optimal number of clusters for each of two classes may be found
from final PPNN performance analysis (expression (6) above). First,
the number of clusters for PPN.sub.2=1 are fixed and the optimal
number of clusters for PPN.sub.1 are found. Next, the reverse
variant is modeled as: PPN.sub.1=1, .LAMBDA. PPN.sub.2=opt. Lastly,
the two optimal networks PPN.sub.1.sup.opt .LAMBDA.
PPN.sub.2.sup.opt are combined together according to expression
(6).
[0097] While the neural network assisted multi-spectral
segmentation process is described with a Probability Projection
Neural Network according to the present invention, it will be
understood that other conventional neural networks are suitable,
including for example, Backpropagation (BP) networks, Elliptic
Basic Functions (EBF) networks, and Learning Vector Quantization
(LQV) networks. However, the PPNN is preferred. The performance
results of the Probability Projection Neural Net have been found to
exceed those achieved by conventional networks.
[0098] According to another aspect of the present invention, the
neural network assisted multi-spectral segmentation process is
implemented as a hardware-encoded procedure embedded in
conventional FPGA (Field Programmable Gate Array) logic as part of
a special-purpose computer.
[0099] The hardware implementation of this network is found in the
form of a look-up table contained in a portion of hardware memory
(FIG. 6). As described above, the neural network 20 comprises three
input nodes and a single, binary output node. The structure of the
neural network 20 according to the present invention also
simplifies the hardware implementation of the network.
[0100] As shown in FIG. 6, the three input nodes correspond to
three optical bands 301, 302, 303 used in gathering the images. The
images taken in the 530 nm and 630 nm bands have 7-bits of useful
resolution while the 577 nm band retains all 8-bits. (The 577 nm
band is centered on the nucleus.) The performance of the neural
network 20 is then determined for all possible combinations of
these three inputs. Since there are 22 bits in total, there are 222
or 4.2 million possible combinations. To create the look-up table,
all input pixels in the space (2.sup.7.times.2.sup.7.times.2.sup.8
variants for the three images in the present embodiment) are
scanned and the look-up table is filled with the PPNN decision,
i.e. 1--pixel belongs to nuclear; 0--pixel doesn't belong to
nuclear, for all each of these pixel combinations.
[0101] The coding of the results (i.e. outputs) of the neural
network comprises assigning each possible combination of inputs a
unique address 304 in a look-up table 305 stored in memory. The
address 304 in the table 305 is formed by joining together the
binary values of the three channel values indicated by 306, 307,
308, respectively in FIG. 6. For example, as shown in FIG. 6, the
pixel for the image from the first channel 301 (i.e. 530 nm) is
binary 0101011, the pixel for the image from the second channel 302
(i.e. 630 nm) is binary 0101011, and the pixel for the image from
the third channel 303 (i.e. 577 nm) is binary 00101011, and
concatenated together binary representations 306, 307, 308 form the
address 304 which is binary (01010110101100101011). The address 304
points to a location in the look-up table 305 (i.e. memory) which
stores a single binary value 309 that represents the response of
the neural network to this combination of inputs, e.g. the logic 0
at memory location 010101101101100101011 signifies that the pixel
in question does not belong to the nucleus.
[0102] The hardware-encoding of NNA-MSS advantageously allows the
process to execute at a high speed while making a complex decision.
Secondly, as experimental data is further tabulated and evaluated
more complex decision spaces can be utilized to improve
segmentation accuracy. Thus, an algorithm according to the present
invention can be optimized further by the adjustment of a table of
coefficients that describe the neural-network connection weights
without the necessity of altering the system architecture.
[0103] The present invention may be embodied in other specific
forms without departing from the spirit or essential
characteristics thereof. Therefore, the presently discussed
embodiments are considered to be illustrative and not restrictive,
the scope of the invention being indicated by the appended claims
rather than the foregoing description, and all changes which come
within the meaning and range of equivalency of the claims are
therefore intended to be embraced therein.
* * * * *