U.S. patent application number 13/852672 was filed with the patent office on 2014-02-06 for methods and software for screening and diagnosing skin lesions and plant diseases.
The applicant listed for this patent is George Zouridakis. Invention is credited to George Zouridakis.
Application Number | 20140036054 13/852672 |
Document ID | / |
Family ID | 49261255 |
Filed Date | 2014-02-06 |
United States Patent
Application |
20140036054 |
Kind Code |
A1 |
Zouridakis; George |
February 6, 2014 |
Methods and Software for Screening and Diagnosing Skin Lesions and
Plant Diseases
Abstract
Provided herein are portable imaging systems, for example, a
digital processor-implemented system for the identification and/or
classification of an object of interest on a body, such as a human
or plant body. The systems comprise a hand-held imaging device,
such as a smart device, and a library of algorithms or modules that
can be implemented thereon to process the imaged object, extract
representative features therefrom and classify the object based on
the representative features. Also provided are methods for the
identifying or classifying an object of interest on a body that
utilize the algorithms and an automated portable system configured
to implement the same.
Inventors: |
Zouridakis; George;
(Houston, TX) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Zouridakis; George |
Houston |
TX |
US |
|
|
Family ID: |
49261255 |
Appl. No.: |
13/852672 |
Filed: |
March 28, 2013 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
61616633 |
Mar 28, 2012 |
|
|
|
Current U.S.
Class: |
348/77 ;
382/110 |
Current CPC
Class: |
G06T 2207/20081
20130101; G06T 2207/10024 20130101; G06T 2207/30088 20130101; A61B
5/0077 20130101; G06K 2209/053 20130101; G06T 7/0012 20130101; A61B
5/444 20130101; G06K 2209/05 20130101 |
Class at
Publication: |
348/77 ;
382/110 |
International
Class: |
G06T 7/00 20060101
G06T007/00 |
Goverment Interests
FEDERAL FUNDING LEGEND
[0002] This invention was made with governmental support under
Grant Number IR21AR057921 awarded by the National Institutes of
Health. The government has certain rights in the invention.
Claims
1. A portable imaging system, comprising: a hand-held imaging
device having a digital camera, a display, a memory, a processor
and a network connection; and a library of algorithms tangibly
stored in the memory and executable by the processor, said
algorithms configured for identification of an object of interest
present on a body.
2. The portable imaging system of claim 1, further comprising
algorithms tangibly stored and processor executable algorithms
configured to display the object of interest and results of the
classification thereof.
3. The portable imaging system of claim 1, wherein the algorithms
comprise processor-executable instructions to: segment the imaged
object to detect a border of the object; extract features from the
segmented object image; and classify the object based on the
extracted features.
4. The portable imaging system of claim 3, wherein the
processor-executable instructions to segment the object function
to: determine an initial contour of the imaged object; classify
pixels as contained within the initial contour as foreground, as
contained without the initial contour as background or as remaining
pixels; and apply a classifier to the remaining pixels for
classification as foreground or background.
5. The portable imaging system of claim 4, wherein the
processor-executable instructions to extract features function to:
divide the segmented object image into regions based on saliency
values calculated for at least one patch within the segmented
object; divide the regions into two regions of higher or lower
intensity based on average intensity values thereof; and extract
feature representations from a sampling of patches within the
intensity regions based on sampling percentages determined for the
regions.
6. The portable imaging system of claim 5, wherein the
processor-executable instructions to classify the object function
to: input the extracted feature representations into a support
vector machine trained with manually segmented objects; and
classify the object based on a comparison of the inputted extracted
features with those in the trained support vector machine.
7. The hand-held imaging system of claim 1, wherein the hand-held
imaging device is a smart device.
8. The hand-held imaging system of claim 1, wherein the body is a
human body or a plant body.
9. The hand-held imaging system of claim 1, wherein the object of
interest is a lesion, an ulcer, or a wound.
10. A method for identifying an object of interest present on a
body, comprising: acquiring an image of the object of interest on
the body via the imaging device comprising the portable imaging
system of claim 1; processing the acquired object image via the
algorithms tangibly stored in the imaging device; and identifying
the object in the image based on patterns of features present in
the imaged object, thereby identifying the object of interest on
the body.
11. The method of claim 10, further comprising: displaying the
results of image processing as each result occurs.
12. The method of claim 10, wherein identifying the object occurs
in real time.
13. The method of claim 10, wherein the object of interest is a
melanoma or a Buruli ulcer.
14. A digital processor-implemented system for classifying an
object of interest on an animal or plant body in real time,
comprising: a portable smart device comprising the processor, a
memory and a network connection; and modules tangibly stored in the
memory comprising: a module for segmentation of an imaged object; a
module for feature extraction within the segmented object image;
and a module for classification of the object based on extracted
features.
15. The digital processor-implemented system of claim 14, further
comprising a module tangibly stored in the memory for display of
the object of interest and results of the classification
thereof.
16. The digital processor-implemented system of claim 14, wherein
the segmentation module comprises processor executable instructions
to: obtain luminance and color components of the imaged object;
classify pixels comprising the image as object pixels, if they
belong to a common luminance and color foreground, as background
pixels if they belong to a common luminance and color background or
as remaining pixels; and apply a classifier to the remaining pixels
to classify them as object or foreground.
17. The digital processor-implemented system of claim 16, wherein
the feature extraction module comprises processor executable
instructions to: calculate a saliency value for a plurality of
patches within the segmented object and separate the patches into
regions based on the saliency values; calculate an average
intensity for the regions to identify them as a higher or as a
lower intensity region; determine a sampling percentage for the
intensity regions; sample patches within the intensity regions by
corresponding sampling percentages; and extract one or more feature
representations for the object.
18. The digital processor-implemented system of claim 16, wherein
the feature extraction module comprises processor executable
instructions to: read input white light image as RGB and the
segmentation result of the region; read input multispectral images
in color channels and transform to gray scale; register
multispectral images via maximization of mutual information with
white light image as reference; extract feature representations
within the ROI of multispectral images and within white light
images; and select one or more relevant features from a pool of the
extracted features.
19. The digital processor-implemented system of claim 16, wherein
the feature extraction module comprises processor executable
instructions to: read input white light image as RGB and the
segmentation result of the region; read input multispectral images
in color channels and transform to gray scale; register
multispectral images via maximization of mutual information with
white light image as reference; determine V.sub.mel, V.sub.blood,
and V.sub.oxy for each ROI pixel to reconstruct maps of melanin,
blood and oxygenating percentage; extract feature representations
within the ROI from the reconstructed maps; and select one or more
relevant features from a pool of the extracted features.
20. The digital processor-implemented system of claim 17, wherein
the classification module comprises processor executable
instructions to: train a support vector machine (SVM) with known
manually segmented objects; and classify the object based on the
extracted feature representations inputted into the SVM.
21. The hand-held imaging system of claim 14, wherein the object of
interest is a lesion, an ulcer, a wound, or skin.
22. A digital processor-implemented method for classifying an
object of interest on an animal or plant body in real time,
comprising the processor executable steps of: digitally imaging the
object of interest with the smart device comprising the digital
processor-implemented system of claim 14; processing the digital
image through the system modules, said modules comprising
algorithms configured for: segmenting the image based on saliency
values to identify pixels thereof as comprising the imaged object
or the background of the image to obtain an object boundary;
extracting features from regions within the object boundary; and
comparing the extracted features to known object features in a
support vector machine trained on the known features to obtain a
classification of the object; and displaying the processed images
and classification results on a display comprising the smart
device.
23. The digital processor-implemented method of claim 20, wherein
the support vector machine is trained on features comprising a
melanoma or a Buruli ulcer.
24. A digital processor-readable medium tangibly storing
processor-executable instructions to perform the digital processor
implemented method of claim 20.
25. A computer-readable medium tangibly storing a library of
algorithms to classify an object of interest on a human or plant
body, said algorithms comprising processor-executable instructions
operable to: obtain luminance and color components of the imaged
object; classify pixels comprising the image as object pixels, if
they belong to a common luminance and color foreground, as
background pixels if they belong to a common luminance and color
background or as remaining pixels; apply a classifier to the
remaining pixels to classify them as object or foreground; extract
one or more feature representations for the object; train a support
vector machine (SVM.degree. with known manually segmented objects;
and classify the object based on the extracted feature
representations inputted into the SVM.
26. The computer-readable medium of claim 25, wherein the
instructions to extract one or more feature representations for the
object comprise: calculate a saliency value for a plurality of
patches within the segmented object and separate the patches into
regions based on the saliency values; calculate an average
intensity for the regions to identify them as a higher or as a
lower intensity region; determine a sampling percentage for the
intensity regions; sample patches within the intensity regions by
corresponding sampling percentages; and extract the one or more
feature representations for the object.
27. The computer-readable medium of claim 25, wherein the
instructions to extract one or more feature representations for the
object comprise: read input white light image as RGB and the
segmentation result of the region; read input multispectral images
in color channels and transform to gray scale; register
multispectral images via maximization of mutual information with
white light image as reference; extract feature representations
within the ROI of multispectral images and within white light
images; and select one or more relevant features from a pool of the
extracted features.
28. The computer-readable medium of claim 25, wherein the object is
the skin, said instructions to extract one or more feature
representations for the object comprising: read input white light
image as RGB and the segmentation result of the region; read input
multispectral images in color channels and transform to gray scale;
register multispectral images via maximization of mutual
information with white light image as reference; determine
V.sub.mel, V.sub.blood, and V.sub.oxy for each ROI pixel to
reconstruct maps of melanin, blood and oxygenating percentage;
extract feature representations within the ROI from the
reconstructed maps; and select one or more relevant features from a
pool of the extracted features.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This non-provisional application claims benefit of priority
under 35 U.S.C. .sctn.119(e) of provisional application U.S. Ser.
No. 61/616,633, filed Mar. 28, 2012, now abandoned, the entirety of
which is hereby incorporated by reference.
BACKGROUND OF THE INVENTION
[0003] 1. Field of the Invention
[0004] The present invention relates to the fields of dermoscopy
and software for the practice thereof on handheld biomedical
screening devices. Specifically, the present invention provides a
library of image processing and texture analysis algorithms that
run on embedded devices for screening lesions and for plant
diseases.
[0005] 2. Description of the Related Art
[0006] The American Cancer Society predicts that there will be
approximately 68,130 new cases of melanoma and 8,700 deaths because
of melanoma in US in 2010 (1). Thus, detection of an early melanoma
is of paramount importance for successful skin cancer screening.
The use of dermoscopy, an imaging technique to visualize structures
inside pigmented skin lesions beyond the naked eye, and
computerized systems for automated classification of dermoscopic
images (2) can drastically improve the diagnostic accuracy of early
melanoma. Image classification using interest points has shown
success in previous studies (3).
[0007] In recent years, cellular phones have made the transition
from simple dedicated telephony devices to being small, portable
computers with the capability to perform complex, memory- and
processor-intensive operations. These new smart devices, generally
referred to as smartphones, provide the user with a wide array of
communication and entertainment options that until recently
required many independent devices. Given advances in medical
imaging, smartphones provide an attractive vehicle for delivering
image-based diagnostic services at a low cost.
[0008] As such, the new generation of smart handheld devices with
sophisticated hardware and operating systems has provided a
portable platform for running medical diagnostic software, such as
the heart rate monitoring (4), diabetes monitoring (5), and
experience sampling (6) applications, which combine the usefulness
of medical diagnosis with the convenience of a handheld device.
Their light operating systems, such as the Apple.RTM. iOS.RTM. and
Google.RTM. Android.RTM., the support for user friendly touch
gestures, the availability of an SDK for fast application
development, the rapid and regular improvements in hardware, and
the availability of fast wireless networking over Wi-Fi and 3G make
these devices ideal for medical applications.
[0009] Thus, there is a recognized need in the art for algorithms
for improved detection, analysis and classification of skin lesions
that can run on devices with limited memory and computational
speed. More specifically, the prior art is deficient in image
sampling, processing and texture analysis methods, algorithms and
plug-in features, for detection and diagnosis of skin and ocular
diseases and plant diseases, that are configured to operate on
smart handheld devices. The present invention fulfills this
long-standing need and desire in the art.
SUMMARY OF THE INVENTION
[0010] The present invention is directed to a portable imaging
system. The portable imaging system comprises a hand-holdable
imaging device having a digital camera, a display, a memory, a
processor and a network connection and a library of algorithms
tangibly stored in the memory and executable by the processor,
where the algorithms are configured for identification of an object
of interest present on a body. The present invention is directed a
related portable imaging system further comprising algorithms
tangibly stored and processor executable algorithms configured to
display the object of interest and results of the classification
thereof.
[0011] The present invention is directed to a method for
identifying an object of interest present on a body. The method
comprises acquiring an image of the object of interest on the body
via the imaging device comprising the portable imaging system
described herein and processing the acquired object image via the
algorithms tangibly stored in the imaging device. The object in the
image is identified based on patterns of features present in the
imaged object, thereby identifying the object of interest on the
body. The present invention is directed to a related method further
comprising the step of displaying the results of image processing
as each result occurs.
[0012] The present invention is directed further to a digital
processor-implemented system for classifying an object of interest
on an animal or plant body in real time. The system comprises a
portable smart device comprising the processor, a memory and a
network connection and modules tangibly stored in the memory. The
modules comprise a module for segmentation of an imaged object, a
module for feature extraction within the segmented object image and
a module for classification of the object based on extracted
features. The present invention is directed to a related digital
processor-implemented system further comprising a module tangibly
stored in the memory for display of the object of interest and
results of the classification thereof.
[0013] The present invention is directed further still to a digital
processor-implemented method for classifying an object of interest
on an animal or plant body in real time. The method comprises the
processor executable steps of digitally imaging the object of
interest with the smart device comprising the digital
processor-implemented system described herein, processing the
digital image through the system modules and displaying the
processed images and classification results on a display comprising
the smart device. The modules comprise algorithms configured for
segmenting the image based on saliency values to identify pixels
thereof as comprising the imaged object or the background of the
image to obtain an object boundary, extracting features from
regions within the object boundary and comparing the extracted
features to known object features in a support vector machine
trained on the known features to obtain a classification of the
object.
[0014] The present invention is directed further still to a digital
processor-readable medium tangibly storing processor-executable
instructions to perform the digital processor implemented method
described herein.
[0015] The present invention is directed further still to a
computer-readable medium tangibly storing a library of algorithms
to classify an object of interest on a human or plant body. The
algorithms comprises processor-executable instructions operable to
obtain luminance and color components of the imaged object,
classify pixels comprising the image as object pixels, if they
belong to a common luminance and color foreground, as background
pixels if they belong to a common luminance and color background or
as remaining pixels, apply a classifier to the remaining pixels to
classify them as object or foreground, calculate a saliency value
for a plurality of patches within the segmented object and separate
the patches into regions based on the saliency values, calculate an
average intensity for the regions to identify them as a higher or
as a lower intensity region, determine a sampling percentage for
the two intensity regions, sample patches within the intensity
regions by corresponding sampling percentages, extract one or more
feature representations for the object, train a support vector
machine (SVM) with known manually segmented objects, and classify
the object based on the extracted feature representations inputted
into the SVM.
[0016] Other and further aspects, features and advantages of the
present invention will be apparent from the following description
of the presently preferred embodiments of the invention given for
the purpose of disclosure.
BRIEF DESCRIPTION OF THE DRAWINGS
[0017] So that the matter in which the above-recited features,
advantages and objects of the invention, as well as others which
will become clear are attained and can be understood in detail,
more particular descriptions and certain embodiments of the
invention briefly summarized above are illustrated in the appended
drawings. These drawings form a part of the specification. It is to
be noted, however, that the appended drawings illustrate preferred
embodiments of the invention and, therefore, are not to be
considered limiting in their scope.
[0018] FIGS. 1A-1C depict segmentation examples based on saliency
(FIG. 1A) and nonuniform (FIG. 1B) sampling and classification
performance for different ratios of sampling densities between the
informative and homogenous region (FIG. 1C). Blue circles represent
patch centers in the more informative (darker) region, while red
crosses correspond to patch centers in the less informative (more
homogeneous) region.
[0019] FIGS. 2A-2B depict an original prior art image (7) (FIG. 2A)
and interest points (FIG. 2B) detected by SIFT using a threshold
adjusted to retain 0.25% of points in the lesion.
[0020] FIGS. 3A-3B depict an original prior art image (7) (FIG. 3A)
and Frangi filtered image (FIG. 3B). Higher responses, i.e.,
brighter spots, are seen at curvilinear structures in the periphery
of the lesion.
[0021] FIG. 4 depicts the segmentation inside the lesion in FIG.
2B.
[0022] FIGS. 5A-5B illustrate the effect of the total points
sampled on classification accuracy for balanced accuracy (BAC)
(FIG. 5A) and area under the receiver operating characteristic
curve (AUC) (FIG. 5B) for a pigmented skin lesion (PSL).
[0023] FIGS. 6A-6B are comparisons of various sampling schemes for
24.times.24 patches for balanced accuracy (BAC) (FIG. 6A) and area
under the receiver operating characteristic curve (AUC) (FIG.
6B).
[0024] FIGS. 7A-7B are comparisons of single scale and multi-scale
samplings for various sampling schemes for balanced accuracy (BAC)
(FIG. 7A) and area under the receiver operating characteristic
curve (AUC) (FIG. 7B).
[0025] FIG. 8 depicts the lesion classification process with steps
selecting a patch from the lesion, applying a 3-level Haar wavelet
transform, extracting texture features, building a histogram using
the cluster centers obtained during training, and inputting the
histogram to the trained SVM classifier to classify the lesion.
[0026] FIGS. 9A-9D depicts the scanning application on an
Apple.RTM. iPhone.RTM. device.
[0027] FIGS. 10A-10C illustrate the error ratio distribution of
segmentation methods Fuzzy-C Means (FIG. 10A), ISODATA (FIG. 10B)
and Active Contour (FIG. 10C) on the dataset of 1300 skin lesion
images. The dotted line marks the threshold for correct
segmentation.
[0028] FIG. 11 depicts the ROC curve for lesion classification.
[0029] FIG. 12 depicts the results of blue-whitish veil detection
on skin lesion images.
[0030] FIGS. 13A-13B depict varying patch size (FIG. 13A) and
varying local bin size (FIG. 13B). Sen1 and Spec1 represent
sensitivity and specificity of first global approach. Sen2 and
Spec2 represent sensitivity and specificity of second global
approach.
[0031] FIG. 14 depicts varying global bin size. Blue bar represents
sensitivity and red bar represents specificity.
[0032] FIGS. 15A-15B depict scaling light intensity (FIG. 15A) and
shifting light intensity (FIG. 15B). Sen1 and Spec1 represent
sensitivity and specificity of first global approach. Sen2 and
Spec2 represent sensitivity and specificity of second global
approach.
[0033] FIGS. 16A-16B depict the smartphone screen showing the image
with menu choices (FIG. 16A) and the results and diagnosis after
comparison with the 7-Points Criteria for detecting melanoma (FIG.
16B).
[0034] FIGS. 17A-17B depict a flowchart of the classification
process in each of the algorithmic segmentation (FIG. 17A) and
feature extraction and classification (FIG. 17B) modules.
[0035] 18 FIGS. 18A-18F are examples of Buruli ulcer (BU)
segmentation showing the original BU image with manual segmentation
(FIG. 18A), the initial mask by fusion (FIG. 18B), level set
segmentation in color (FIG. 18C) and luminance (FIG. 18D) channels,
segmentation after pixel classification (FIG. 18E), and the final
segmentation result (FIG. 18F). The line(s) around the lesions show
the result for automatic segmentation (FIGS. 18B-18E), the ground
true from an expert dermatologist (FIG. 18A) or both (FIG.
18F).
[0036] 19 FIGS. 19A-19D depict different segmentation methods by AT
(FIG. 19A), GVF (FIG. 19B), LS (FIG. 19C), and the instant
segmentation method (FIG. 19D) where results from automatic
segmentation and ground true from an expert dermatologist are
included.
[0037] FIGS. 20A-20B are examples of image patterns closest to the
two cluster centroids for Buruli lesions (FIG. 20A) and non-Buruli
lesions (FIG. 20B).
[0038] FIGS. 21A-21B illustrate the effect of sampling strategy on
classification performance. FIG. 21A shows the effect of patch
number and FIG. 21B shows the effect of patch size.
[0039] FIGS. 22A-22B illustrates ulcer variation of a Buruli ulcer
from early to late stage (FIG. 22A) and demonstrates that the
algorithms can distinguish between early and late stage ulcers
(FIG. 22B).
[0040] FIG. 23 depicts the architecture for the multispectral
imaging process.
[0041] FIG. 24 depicts a flowchart of the classification process in
algorithmic feature extraction and classification modules for
multispectral images.
[0042] FIG. 25 depicts a flowchart of the classification process in
algorithmic feature extraction and classification modules for an
optical skin model.
DETAILED DESCRIPTION OF THE INVENTION
[0043] As used herein, the term "a" or "an", when used in
conjunction with the term "comprising" in the claims and/or the
specification, may refer to "one," but it is also consistent with
the meaning of "one or more," "at least one," and "one or more than
one." Some embodiments of the invention may consist of or consist
essentially of one or more elements, method steps, and/or methods
of the invention. It is contemplated that any method or composition
described herein can be implemented with respect to any other
method or composition described herein.
[0044] As used herein, the term "or" in the claims refers to
"and/or" unless explicitly indicated to refer to alternatives only
or the alternatives are mutually exclusive, although the disclosure
supports a definition that refers to only alternatives and
"and/or."
[0045] As used herein, the term "about" refers to a numeric value,
including, for example, whole numbers, fractions, and percentages,
whether or not explicitly indicated. The term "about" generally
refers to a range of numerical values (e.g., +/-5-10% of the
recited value) that one of ordinary skill in the art would consider
equivalent to the recited value (e.g., having the same function or
result). In some instances, the term "about" may include numerical
values that are rounded to the nearest significant figure.
[0046] As used herein, the terms "body" and "subject" refer to a
mammal, preferably human, or to a plant.
[0047] As used herein, the term "object" in reference to a body or
a subject, refers to a lesion, wound, ulcer or other condition, or
the skin or a region comprising the same that is located on the
subject or body or refers to an area on the subject or body
suspected of being or is malignant or associated with a disease or
other pathophysiological condition.
[0048] In one embodiment of the present invention there is provided
a portable imaging system, comprising a hand-holdable imaging
device having a digital camera, a display, a memory, a processor
and a network connection; and a library of algorithms tangibly
stored in the memory and executable by the processor, said
algorithms configured for identification of an object of interest
present on a body. Further to this embodiment the portable imaging
system comprises algorithms tangibly stored and processor
executable algorithms configured to display the object of interest
and results of the classification thereof.
[0049] In both embodiments the algorithms may comprise
processor-executable instructions to segment the imaged object to
detect a border of the object; extract features from the segmented
object image; and classify the object based on the extracted
features.
[0050] In an aspect of both embodiments the processor-executable
instructions to segment the object may function to determine an
initial contour of the imaged object; classify pixels as contained
within the initial contour as foreground, as contained without the
initial contour as background or as remaining pixels; and apply a
classifier to the remaining pixels for classification as foreground
or background.
[0051] In another aspect of both embodiments the
processor-executable instructions to extract features may function
to divide the segmented object image into regions based on saliency
values calculated for at least one patch within the segmented
object; divide the regions into two regions of higher or lower
intensity based on average intensity values thereof; and extract
feature representations from a sampling of patches within the
intensity regions based on sampling percentages determined for the
regions.
[0052] In yet another aspect of both embodiments the
processor-executable instructions to classify the object may
function to input the extracted feature representations into a
support vector machine trained with manually segmented objects; and
classify the object based on a comparison of the inputted extracted
features with those in the trained support vector machine.
[0053] In all embodiments and aspects thereof the hand-held imaging
device may be a smart device. Also, the body may be a human body or
a plant body. In addition representative examples of the object of
interest are a lesion, an ulcer, or a wound.
[0054] In another embodiment of the present invention there is
provided a method for identifying an object of interest present on
a body, comprising acquiring an image of the object of interest on
the body via the imaging device comprising the portable imaging
system described supra; processing the acquired object image via
the algorithms tangibly stored in the imaging device; and
identifying the object in the image based on patterns of features
present in the imaged object, thereby identifying the object of
interest on the body.
[0055] Further to this embodiment the method comprises displaying
the results of image processing as each result occurs. In both
embodiments identifying the object of interest occurs in real time.
Also, in both embodiments the object of interest may be a melanoma
or a Buruli ulcer.
[0056] In yet another embodiment of the present invention there is
provided a digital processor-implemented system for classifying an
object of interest on an animal or plant body in real time,
comprising a portable smart device comprising the processor, a
memory and a network connection; and modules tangibly stored in the
memory comprising a module for segmentation of an imaged object; a
module for feature extraction within the segmented object image;
and a module for classification of the object based on extracted
features. Further to this embodiment the digital
processor-implemented system comprises a module tangibly stored in
the memory for display of the object of interest and results of the
classification thereof. Representative examples of the object may
be a lesion, an ulcer or a wound.
[0057] In both embodiments the segmentation module comprises
processor executable instructions to obtain luminance and color
components of the imaged object; classify pixels comprising the
image as object pixels, if they belong to a common luminance and
color foreground, as background pixels if they belong to a common
luminance and color background or as remaining pixels; and apply a
classifier to the remaining pixels to classify them as object or
foreground.
[0058] In an aspect of both embodiments the feature extraction
module may comprise processor executable instructions to calculate
a saliency value for a plurality of patches within the segmented
object and separate the patches into regions based on the saliency
values; calculate an average intensity for the regions to identify
them as a higher or as a lower intensity region; determine a
sampling percentage for the two intensity regions; sample patches
within the intensity regions by corresponding sampling percentages;
and extract one or more feature representations for the object.
Also in both embodiments the classification module comprises
processor executable instructions to train a support vector machine
(SVM) with known manually segmented objects; and classify the
object based on the extracted feature representations inputted into
the SVM.
[0059] In another aspect the feature extraction module may comprise
processor executable instructions to read input white light image
as RGB and the segmentation result of the region; read input
multispectral images in color channels and transform to gray scale;
register multispectral images via maximization of mutual
information with white light image as reference; extract feature
representations within the ROI of multispectral images and within
white light images; and select one or more relevant features from a
pool of the extracted features.
[0060] In yet another aspect the feature extraction module may
comprise processor executable instructions to read input white
light image as RGB and the segmentation result of the region; read
input multispectral images in color channels and transform to gray
scale; register multispectral images via maximization of mutual
information with white light image as reference; determine
V.sub.mel, V.sub.blood, and V.sub.oxy for each ROI pixel to
reconstruct maps of melanin, blood and oxygenating percentage;
extract feature representations within the ROI from the
reconstructed maps; and select one or more relevant features from a
pool of the extracted features.
[0061] In yet another embodiment of the present invention there is
provided a digital processor-implemented method for classifying an
object of interest on an animal or plant body in real time,
comprising the processor executable steps of digitally imaging the
object of interest with the smart device comprising the digital
processor-implemented system described supra; processing the
digital image through the system modules, the modules comprising
algorithms configured for segmenting the image based on saliency
values to identify pixels thereof as comprising the imaged object
or the background of the image to obtain an object boundary;
extracting features from regions within the object boundary; and
comparing the extracted features to known object features in a
support vector machine trained on the known features to obtain a
classification of the object; and displaying the processed images
and classification results on a display comprising the smart
device. In this embodiment the support vector machine may be
trained on features comprising a melanoma or a Buruli ulcer.
[0062] In a related embodiment there is provided a digital
processor-readable medium tangibly storing processor-executable
instructions to perform the digital processor implemented method
described herein.
[0063] In yet another embodiment of the present invention there is
provided a computer-readable medium tangibly storing a library of
algorithms to classify an object of interest on a human or plant
body, said algorithms comprising processor-executable instructions
operable to obtain luminance and color components of the imaged
object; classify pixels comprising the image as object pixels, if
they belong to a common luminance and color foreground, as
background pixels if they belong to a common luminance and color
background or as remaining pixels; apply a classifier to the
remaining pixels to classify them as object or foreground;
calculate a saliency value for a plurality of patches within the
segmented object and separate the patches into regions based on the
saliency values; calculate an average intensity for the regions to
identify them as a higher or as a lower intensity region; determine
a sampling percentage for the two intensity regions; sample patches
within the intensity regions by corresponding sampling percentages;
extract one or more feature representations for the object; train a
support vector machine (SVM) with known manually segmented objects;
and classify the object based on the extracted feature
representations inputted into the SVM.
[0064] In one aspect the instructions to extract one or more
feature representations for the object may calculate a saliency
value for a plurality of patches within the segmented object and
separate the patches into regions based on the saliency values;
calculate an average intensity for the regions to identify them as
a higher or as a lower intensity region; determine a sampling
percentage for the intensity regions; sample patches within the
intensity regions by corresponding sampling percentages; and
extract the one or more feature representations for the object.
[0065] In another aspect the instructions to extract one or more
feature representations for the object may read input white light
image as RGB and the segmentation result of the region; read input
multispectral images in color channels and transform to gray scale;
register multispectral images via maximization of mutual
information with white light image as reference; extract feature
representations within the ROI of multispectral images and within
white light images; and select one or more relevant features from a
pool of the extracted features.
[0066] In yet another aspect the instructions to extract one or
more feature representations for the object may read input white
light image as RGB and the segmentation result of the region; read
input multispectral images in color channels and transform to gray
scale; register multispectral images via maximization of mutual
information with white light image as reference; determine
V.sub.mel, V.sub.blood, and V.sub.oxy for each ROI pixel to
reconstruct maps of melanin, blood and oxygenating percentage;
extract feature representations within the ROI from the
reconstructed maps; and select one or more relevant features from a
pool of the extracted features.
[0067] The present invention provides a library of algorithms,
algorithm modules, plug-ins, and methods utilizable on a handheld
portable imaging device, such as a smart device, for the
identification and/or classification of an object of interest on a
body. The library may be a C/C++ based library and may be
downloaded and accessed as a plug-in or stored in the memory on a
handheld smart device. Representative examples of such smart
devices are, but not limited to, Apple.RTM. iOS.RTM.-based devices,
such as iPhone.RTM., iPad.RTM. and iPod Touch.RTM., which are
trademarks of Apple Inc., registered in the U.S. and other
countries, and Android.RTM. based devices, which is a registered
trademark of Google Inc. The library comprises algorithms useful
for the processing, texture analysis and classification of an image
of an object of interest on a human or plant body skin lesion as
malignant or benign. The algorithms are generally applicable to
segmentation, feature extraction and classification of the object.
For example, the object of interest may be identified or classified
as a benign or malignant lesion, an ulcer, for example, but not
limited to, a Buruli ulcer, or a wound.
[0068] Thus, provided herein are methods, particularly, digital
processor-implemented methods for identifying or classifying the
object of interest via the algorithms or modules comprising the
same. The method utilizes the algorithmic library implemented on a
handheld smart device, for example, a smartphone, as described
herein. An image is acquired with the smartphone. Image acquisition
optionally may utilize an external attachment that can provide
illumination and magnification. The library of algorithms or
modules is implemented on the smart device to operate on the image
and a classification score is obtained, as described. As such
cancers and other diseases of a human or plant body may be
diagnosed. Particularly, the method enables diagnosis of a skin
cancer, such as melanoma, or a skin disease, such as Buruli ulcer.
A sufficient resolution enables the detection of and distinction
among other types of lesions, ulcers or wounds. Moreover, the
algorithms are configured to enable to run a Buruli analysis and a
skin lesion analysis at the same time. The algorithms provided
herein also are configured to process multispectrally-acquired
images or optical images obtained with lights of different
frequencies utilizing feature extraction and classification
algorithms.
[0069] The present invention also includes a digital
processor-readable or computer-readable medium that can tangibly
store the algorithms and/or the processor-executable instructions
or methods contained therein. Such readable media are well-known
and standard in the art and can comprise a memory, such as on the
smart device or other networked computer or a diskette, memory
stick or flash drive from which the algorithms can be downloaded to
the smart device, for example as a plug-in.
[0070] Moreover, the present invention provides an automated
portable system for identification or classification of the object
of interest. The system comprises a smart handheld device having an
operating system, memory, processor, display screen, and lens and
means for image acquisition, such as a digital camera, and is
networkable, as is well-known and standard in the art. The system
is configured to acquire and display an image of the object and to
operate the library/modules of algorithms on the acquired image to
process, identify patterns of object features, and classify the
object. While the system is configured to operate on the smart
handheld device, the device can wirelessly port the information or
results to another smart device or desktop computer by methods
well-known in the art.
[0071] It is contemplated that the library of algorithms provided
herein may be utilized to detect ocular diseases, such as glaucoma.
With an imaging technology suitable to allow visualization of the
retinal tissue at an adequate resolution, the library of algorithms
provided herein may be utilized for automatic analysis of the
acquired image noninvasively in real time. Moreover, a handheld
smart device comprising the library may be useful in obtaining and
analyzing in real time infrared images of leaves to detect plant
diseases or insect infestations or fungal infections in good time
to save the crop. This results in successful agricultural
management and improved crop productivity.
[0072] The following examples are given for the purpose of
illustrating various embodiments of the invention and are not meant
to limit the present invention in any fashion.
Example 1
Methods
[0073] An image, such as skin lesion, can be acquired using a
smartphone camera, with or without an external attachment that can
provide illumination and magnification, or can be loaded from the
photo library to provide the diagnosis in real time. The
application can process images taken either with the iPhone.RTM.
camera or taken with an external camera and uploaded to the image
library on the phone. To test the application, images from a large
commercial library of skin cancer images that were annotated by
dermatologists (7) are uploaded to the phone. Intra-observer and
inter-observer agreement could be low for certain criteria (8). All
images were segmented manually to provide an evaluative level
against which the automated techniques of the applications
presented herein were compared. Prior to processing, if necessary,
the images are converted from color to 256 level greyscale.
[0074] The iPhone 4.RTM. is a smartphone developed by Apple Inc.,
and features an Apple.RTM. A4 ARM.RTM. processor running at 1 GHz,
512 MB DDR SDRAM (Advanced RISC Machines, Ltd), up to 32 GB of
solid state memory, a 5 MPixel built-in camera with LED flash, and
3G/Wi-Fi/Bluetooth.RTM. (Bluetooth SIG) communication networks.
Thus, it has all the features, computational power, and memory
needed to run the complete image acquisition and analysis procedure
in quasi real time. There is lot of potential for such a device in
medical applications, because of its low cost, portability, ease of
use, and ubiquitous connectivity.
7-Point Checklist Criteria for Diagnosis of Melanoma
[0075] The 7-point checklist includes seven dermoscopic features
that can be detected with high sensitivity and decent specificity
by even less experienced clinicians. The seven points of the list
are subdivided into three major and four minor criteria, reflecting
their importance in defining a melanoma. To score a lesion, the
presence of a major criterion is given two points and that of a
minor one point. If the total score is greater than or equal to 3,
the lesion is classified as melanoma.
[0076] The major criteria are: 1) Atypical pigment network: Black,
brown, or gray network with irregular meshes and thick lines; 2)
Blue-whitish veil: Confluent gray-blue to whitish-blue diffuse
pigmentation associated with pigment network alterations,
dots/globules and/or streaks; and 3) Atypical vascular pattern:
Linear irregular or dotted vessels not clearly combined with
regression structures and associated with pigment network
alterations, dots/globules and/or streaks.
[0077] The minor criteria are 1) Irregular streaks: Irregular more
or less confluent, linear structures not clearly combined with
pigment network lines; 2) Irregular pigmentation: Black, brown,
and/or gray pigment areas with irregular shape and/or distribution;
3) Irregular dots/globules: Black, brown, and/or gray round to
oval, variously sized structures irregularly distributed; 4)
Regression structures: Associated white scar-like and gray blue,
peppering, multiple blue gray dots.
Example 2
Non-Uniform Sampling for Bag-of-Features Classification
Dataset
[0078] A dataset with 645 epiluminescence microscopy (ELM) images,
in which 491 lesions are benign and 154 lesions are melanoma, was
used. The images are mainly collected from a commercial database
(7). The total image size ranges from 712.times.454 to
1,024.times.768 pixels, while the lesion size ranges from 7,662 to
804,527 pixels.
Procedure
[0079] For bag-of-features-based classification, first, patches are
extracted from every lesion, and then, for each patch, patch
descriptors are generated using color moments and Haar wavelet
coefficients, which capture the color and texture information.
Then, each patch is assigned to a codeword from a pre-learned
codebook, using hard assignment, as described herein. After that, a
final feature vector is generated by pooling the assignments of all
patches extracted from the lesion. For the classifier, support
vector machines (SVMs) with a .chi..sup.2 kernel are used, which
currently represent state-of-art settings for a bag-of-features
model.
[0080] Ten times ten-fold stratified cross-validation was performed
to evaluate the performance of the method. Performance criteria
include sensitivity, specificity, balanced accuracy (BAC, i.e.,
average of sensitivity and specificity), and area under the
receiver operating characteristic curve (AUC). For sensitivity and
specificity, both the mean and 95% confidence interval estimated
from a binomial distribution are reported, and the average for BAC.
Similarly, for AUC both the mean value and the standard deviation
obtained are shown.
Occurrence-Based Contextual Saliency
[0081] The saliency measure S.sub.i.sup.o uses co-occurrence
information between patches and codewords in images. Given a patch
x.sub.i, saliency is defined as the average likelihood of the image
patches,
S i o = 1 n ? .PHI. ( x j | x i ) . ? indicates text missing or
illegible when filed ( Eq . 1 ) ##EQU00001##
Here, .phi.(x.sub.j|x.sub.i) can be interpreted as a compatibility
score between x.sub.i and x.sub.j. That is, if patch x.sub.i has
been seen in an image, a higher score for patch x.sub.j means that
it will appear in the same image with a higher probability. The
function .phi.(x.sub.j|x.sub.i) can be computed in the following
steps.
[0082] Let x.sub.i and y.sub.i denote a patch and the corresponding
patch descriptor, respectively. Then each image patch x.sub.i is
assigned to a codeword w.sub.a in the codebook. These assignments
may be soft or hard, and the probability of patch being assigned to
codeword is given by
.alpha..sub.ia=P(w.sub.ay.sub.i) (Eq. 2).
The value w.sub.a is discrete, while the values of .alpha..sub.ia
for all codewords sum to one, i.e.,
.SIGMA..sup.mp(w.sub.a|y.sub.i)=1.
The value of .phi.(x.sub.j|x.sub.i) is computed by marginalizing
over all possible codeword assignment for x.sub.j.
.PHI. ( x j | x i ) = ? p ( w k | ? ) .PHI. ( w k | ? ) , ?
indicates text missing or illegible when filed ( Eq . 3 )
##EQU00002##
where .phi.(w.sub.b|x.sub.j) is the compatibility score between
w.sub.b and x.sub.i. In the same way, .phi.(w.sub.b|x.sub.i) can be
obtained by marginalizing over all possible codeword assignment for
x.sub.i
.PHI. ( w k | x i ) = ? p ( ? ) p ( ? ) . ? indicates text missing
or illegible when filed ( Eq . 4 ) ##EQU00003##
By rearranging Equation 1, Equation 3, and Equation 4, Eq. 5 was
obtained
? = 1 n ? p ( ? ) p ( ? ) p ( ? ) . ? indicates text missing or
illegible when filed ( Eq . 5 ) ##EQU00004##
where n is the number of pixels, m is the number of clusters,
p ( w a | y i ) = { 1 if y i w a 0 otherwise ( Eq . 6 ) p ( w b | y
j ) = { 1 if y j w b 0 otherwise ( Eq . 7 ) ##EQU00005##
where p(w.sub.b|w.sub.a) is the empirical conditional probability
of observing codeword w.sub.b given that codeword w.sub.a has been
observed somewhere in the image. These can be learned through
Maximum Likelihood Estimator (MLE) counts from the training
images.
[0083] Since saliency measures how well an individual patch can
predict the occurrence of other patches in the same image, pixels
with higher saliency values always belong to the relatively more
homogenous background region. Therefore, some informative patches
with lower saliency values are missed when using saliency-based
image sampling.
Nonuniform Sampling Strategy
[0084] The basic idea of nonuniform sampling is to sample more
patches from dermoscopically interesting regions, which are
obtained by segmentation according to patch saliency and pixel
intensity. It consists in four main steps:
[0085] 1. Calculate saliency values for all patches;
[0086] 2. Separate the lesion into two regions based on
saliency;
[0087] 3. Choose informative and homogeneous regions according to
pixel intensities;
[0088] 4. Decide sampling densities for the two separate
regions.
[0089] In Step 1, saliency values are calculated using Equation 3
for each patch, and then k-means clustering is applied to separate
the lesion into two regions. Subsequently, the region with lower
pixel intensities was chosen as the informative region, and the
other region as the homogenous one. That is because with pigmented
skin lesions, dermatologists always pay more attention to the dark
areas of a lesion to diagnose a melanoma. Then the sampling density
for each region was chosen. When random sampling is applied to
these two distinct regions, more patches are extracted from the
informative region and fewer from homogeneous one. The sampling
densities are controlled by the following equation,
P.sub.i=(.alpha.A.sub.i/.alpha.A.sub.i+A.sub.h).times.100% (Eq.
8)
where P.sub.i represents the percentage of patches sampled from the
informative region, and A.sub.i and A.sub.h are the areas of the
informative and homogeneous regions, respectively. The ratio of
sampling densities of informative to homogeneous region is
.alpha..
[0090] The coefficient .alpha. can be fixed or be allowed to vary
dynamically. A dynamic .alpha. can be obtained by considering the
saliency values of the two regions, which means that a big
difference in saliency will result in a big difference in sampling
density between two regions. FIGS. 1A-1B shows examples of
saliency-based sampling and nonuniform sampling. It is obvious that
saliency-based sampling incorrectly captures more information from
the homogeneous background and misses informative patches that it
considers as nonsalient. In contrast, the proposed nonuniform
method correctly emphasizes the region that contains more DIPs.
Comparison of Sampling Strategies
[0091] The instant method is compared against plain saliency-based
sampling (8), whereby patches with higher saliency values are
considered of higher importance. To avoid the influence of the
number of patches extracted and of the patch size, for both
sampling strategies the patch size was fixed to 24.times.24 pixels
and the patch percentage to approximately 4%. Table 1 shows the
resulting classification performance for the two approaches. It can
be seen that the proposed method achieves better accuracy than
saliency based sampling. That is because patches with higher
saliency values always belong to the relatively more homogenous
region of the lesion, while the most informative patches from the
skin lesion that are less frequent, and thus have lower saliency
values, are missed. On the other hand, the method uses saliency as
a measure to separate the image into more informative and less
informative regions, while pixel intensities are used to identify
the informative one. In this way, when analyzing pigmented skin
lesions, more patches are sampled from the informative region that
contains more DIPs.
TABLE-US-00001 TABLE 1 Classification performance using different
sampling strategies Sensitivity Specificity BAC AUC Methods (95%
CI) (95% CI) (95% CI) (std) Saliency 89.53 84.45 88.99 96.78 .+-.
[86.91, 91.79] [85.73, 90.81] [86.32, 91.30] 2.12 Nonuniform 93.67
92.00 92.83 98.69 .+-. [91.50, 95.42] [89.63, 93.97] [90.57, 94.70]
1.12
Effects of Saliency
[0092] To demonstrate the benefits of using saliency for lesion
sampling, as a control another sampling method was added, which
both segments an image and chooses the informative region based on
pixel intensity. Again, patches of size 24.times.24 pixels covering
4% of the total lesion area are sampled with this method, and
random sampling is applied in the separate regions. Table 2 shows
the resulting classification performance for the two approaches. It
can be seen that lesion separation according to patch saliency can
achieve a better classification performance than separation based
only on pixel intensity. Thus, saliency provides an effective way
to separate a lesion into informative and homogeneous regions.
TABLE-US-00002 TABLE 2 Classification performance using different
sampling strategies Sensitivity Specificity BAC AUC Methods (95%
CI) (95% CI) (95% CI) (std) Intensity 88.67 86.49 87.58 95.89 .+-.
[85.96, 91.01] [83.61, 89.03] [84.79, 90.02] 2.45 Nonuniform 93.67
92.00 92.83 98.69 .+-. [91.50, 95.42] [89.63, 93.97] [90.57, 94.70]
1.12
Effects of Sampling Density
[0093] The ratio of sampling densities between informative and
homogeneous regions can also affect classification accuracy. For
the nonuniform sampling method, different values for the
coefficient .alpha. in Equation 4 were tested, which represent
different ratios of sampling densities whose influence is shown in
FIG. 1C. When a equals one, nonuniform sampling is equivalent to
uniform sampling. As a increases above one, and more patches are
sampled from the informative region, the classification accuracy
also increases. However, when a becomes too large, the overall
performance decreases as well. This suggests that there is a
minimum amount of complementary information provided by the
homogeneous region that is essential for accurate classification.
In summary, patches from informative regions that contain more
dermoscopically interesting features should be sampled more
densely, but patches from homogeneous regions which provide
complementary information should not be ignored. The best
performance can be achieved when .alpha. lies within (1, 2).
Example 3
Evaluation of Sampling Strategies of Dermoscopic Interest Points
(DIPs) in Melanomas Dataset
[0094] A dataset with 1,505 epiluminescence microscopy (ELM)
images, in which 1,098 lesions are benign and 407 lesions are
melanoma were used. The image size ranges from 712.times.454 to
1,024.times.768 pixels, and lesion size ranges from 7,662 to
804,527 pixels. Manual segmentation of all lesions is used to
ensure that evaluation of the various sampling strategies is not
affected by possible differences in automated identification of the
lesion boundary.
Procedure
[0095] The lesion classification procedure (9) consists of five
main steps: image sampling, feature extraction, coding, pooling,
and final classification (10). For a given image, identify DIPs
inside the lesion are identified first and then a patch is
extracted around each DIP. On each patch, several low level texture
and color features were computed using Haar wavelets and color
moments, which are important for melanoma detection. In the coding
stage, a patch is assigned to a codeword from a pre-learned
codebook using hard or soft assignment. Herein, each patch is
assigned to its nearest neighbor in the codebook with hard
assignment. The assignments of all patches extracted from a lesion
are pooled into one feature vector. The last step is to classify
the lesion based on the feature vector obtained from pooling.
[0096] Ten times ten-fold stratified cross-validation is performed
using sensitivity, specificity, balanced accuracy (BAC, i.e.,
average of sensitivity and specificity), and area under the
receiver operating characteristic curve (AUC) as performance
criteria. For sensitivity and specificity, the mean and 95%
confidence interval (CI) estimated from a binomial distribution are
reported, and their average for BAC. For AUC both the mean value
and standard deviation (std) of the values obtained are shown.
Image Sampling Strategies
[0097] The sampling operator selects N pixels inside a lesion and
then it centers a p.times.p pixel patch at each pixel location. For
DIP detection, four sampling strategies are investigated. The first
two are specifically designed for blobs and curvilinear components,
respectively, which are the typical structures seen inside a lesion
(11). The other two, however, are not targeting any particular
lesion structure; yet, they result in excellent image
classification performance.
[0098] Detector for blobs and corners: Blobs, dots, and globular
structures are frequently observed in a lesion. The scale-invariant
feature transform (SIFT) (12) is used to detect these structures, a
procedure also used in (11) (FIGS. 2A-2B).
[0099] Detector for curvilinear structures: the SIFT operator is
not stable for ridge detection (12) and it may fail to localize
curvilinear structures in the lesion, as it was noted also by Zhou
et al (11). Instead, for curvilinear structures, a Frangi filter
(Frangi) is applied at three scales CT=1, 2, and 3 (10). Points
with higher filter responses have higher probabilities of being
curvilinear structures. A Frangi filter is similar to the Steger
filter used in Zhou et al. (FIGS. 3A-3B)
[0100] Grid sampling: Sampling on a regular grid of size of g
(Grid-g) placed on a lesion. When g is small, this is also called
dense sampling.
[0101] Radial sampling: Sampling using polar coordinates on axes
placed on the lesion with origin at the center of the lesion
(Radial). The rationale behind this scheme is that a lesion
generally follows a radially growing pattern (14).
Feature Pooling Schemes
[0102] The popular average pooling and spatial pooling schemes are
investigated. Average pooling uses averaging of the class
assignments across all patches. This is equivalent to building a
normalized histogram, whereby each bin corresponds to a codeword in
a codebook and the bin's value is proportional to the number of
patches assigned to that codeword. Spatial pooling detects
homogeneous regions inside a lesion and then uses average pooling
in each homogeneous region. A lesion is segmented into 3 to 8
regions using the normalized cut method (FIG. 4). Tiny regions are
grouped with nearby larger ones. Thus, after spatial pooling, a
single vector (histogram) is produced for each segmented region. In
the proposed method, a whole lesion is represented as a fully
connected weighted graph, whose nodes correspond to homogeneous
regions. The weight of an edge is the Euclidean distance between
the vectors of the two connected nodes (regions). Then a lesion is
represented using six features implemented in the graph measure
toolbox (15), namely clustering coefficient, maximized modularity,
characteristic path length, eccentricity for each vertex, radius,
and diameter of the graph (graph eccentricity, radius, and diameter
are not the same lesion measures defined in (2)). Tree and graph
schemes have been proposed before (16-17), however, not for
malignant classification. This proposed weighted graph model
extends recent work in which a non-weighted graph lesion
representation was employed for melanoma detection (18).
Codebook and Classifier Implementation Details
[0103] Codebooks are built using K-mean clustering on a set of
patches obtained by randomly sampling 1,000 patches from every
lesion so that every lesion contributes equally to the codebook
construction. Thus, the evaluation uses transductive inference
(19), i.e., in this classifier learning method labeled training
data and unlabeled testing data were employed, while for testing
labels were predicted for the latter. The number of cluster is 200
for wavelet features and 100 for color features. The overall
performance is not sensitive to these choices. Separate codebooks
are built for wavelet and color, and different codebooks for the
three patch sizes: 16, 24, and 32. By default, average pooling is
used, if not specified otherwise. This classifier uses support
vector machines (SVMs) with a .chi..sup.2 kernel, which is the
state-of-the-art setting for BoF model. For graph theory features,
a Gaussian kernel which is the common choice for SVMs is used. The
threshold for the classifier's output was chosen by maximizing the
average of sensitivity and specificity on the labeled training
data. For classifier combination, simple ensemble averaging is used
(weighted combination (20) yielded very similar results on this
dataset).
The Effect of Number of Patches Sampled
[0104] Choosing the same number of patches for all lesions is not
reasonable, since lesions differ in size. Instead, a number of
patches proportional to that lesion's area were chosen. Simple grid
sampling was used and grid size was chosen from the set {1, 5, 10,
20, 40, 100}. Using a grid size g is equivalent to sampling
approximately (100/g.sup.2) % points from a lesion. A square patch
of 24 pixels in size is used. FIGS. 5A-5B show that this percentage
value affects significantly both performance measures. BAC starts
to converge when the number of points approaches 4% of the lesion's
area, while AUC converges earlier at about 1%. Thus, only 4% of
points, i.e., Grid-5, from a lesion need to be sampled without
decreasing performance significantly for both BAC and AUC.
The Effect of Sampling Strategy
[0105] Now four lesion sampling methods, Grid-5, Radial, SIFT, and
Frangi, are considered and the parameters and thresholds of the
latter three methods are adjusted to retain 4% of all possible
samples. In addition, the classifiers from Radial, SIFT, and Frangi
are combined with Grid-1 (denoted as Com) to test whether
classification accuracy improves when combining classifier training
with interest points located at dermoscopic structures instead of
simply using all possible points alone, i.e., Grid-1. FIGS. 6A-6B
show that regular grid sampling Grid-5 provides results comparable
to Radial, SIFT, and Frangi. A comparison between Com and Grid-1
reveals only a marginal improvement in BAC, but no improvement in
AUC, when incorporating the more complicated interest point
detectors instead of using simple dense sampling alone.
The Effect of Sampling at Multiple Scales
[0106] For each sampling strategy, square patches of size 16, 24,
and 32 pixels are extracted, and the classifiers obtained from
these three scales are grouped. For the multi-scale model of Com,
12 classifiers are ensembled from four sampling methods and three
scales. FIGS. 7A-7B shows that multi-scale sampling can improve the
performance of some methods compared to sampling at a single scale
with patches of size 24. However, none of the multi-scale models in
FIGS. 8A-8B is significantly better than Grid-1 using a single
scale sampling.
The Effect of Spatial Pooling
[0107] Spatial pooling for patches of size 16.times.16 centered on
every pixel are used, since it was observed empirically that a
patch of size 16 performs better than size 24 or size 32 for graph
theory features. The classifiers built from spatial pooling and
Grid-1 are ensembled, and the combined model is denoted as DenSpa.
DenSpa is compared with Grid-1, Com, and the multiscale models of
Grid-1 and Com, denoted as GridMuI and ComMuI, respectively, in
Table 3. DenSpa performs the best among the five schemes in all
measures. The mean sensitivity of the other four methods without
spatial pooling is below the 95% CI of DenSpa. The improvement for
specificity is not so significant, but the AUC of DenSpa is
significantly different from the other four methods as revealed by
an unpaired t-test at 95% confidence level.
TABLE-US-00003 TABLE 3 Classification Performance Methods
Sensitivity (95% CI) Fuzzy c-Means AUC (std) Grid-1 82.49 [79.16,
85.40] 83.32 [81.39, 85.07] 90.93 .+-. 2.67 GridMu1 82.31 [78.92,
85.18] 84.15 [82.28, 85.88] 91.16 .+-. 2.55 Com 81.96 [78.67,
84.97] 84.68 [82.82, 86.37] 90.87 .+-. 2.60 ComMul 82.40 [79.16,
85.40] 84.19 [82.37, 85.97] 90.99 .+-. 2.53 DenSpa 86.17 [83.11,
88.80] 84.68 [82.82, 86.37] 92.71 .+-. 2.25
Example 4
[0108] Portable Library for Melanoma Detection: Comparison of
Smartphone Implementation with Desktop Application
[0109] The automated procedure for lesion classification is based
on the bag-of-features framework (3,9) and comprises the main steps
of lesion segmentation, feature extraction, and classification.
Dataset
[0110] A total of 1300 artifact free images are selected: 388 were
classified by histological examination as melanoma and the
remaining 912 were classified as benign.
Image Segmentation
[0111] In addition to the lesion, images typically include
relatively large areas of healthy skin, so it is important to
segment the image and extract the lesion to be considered for
subsequent analysis. To reduce noise and suppress physical
characteristics, such as hair in and around the lesion that affect
segmentation adversely, a fast two-dimensional median filtering
(21) is applied to the grey scale image. The image is then
segmented using three different segmentation algorithms, namely
ISODATA (iterative self-organizing data analysis technique
algorithm) (22), fuzzy c-means (23-24), and active contour without
edges (25). The resulting binary image is further processed using
morphological operations, such as opening, closing, and connected
component labeling (26). When more than one contiguous region is
found, additional processing removes all regions except for the
largest one. The end result is a binary mask that is used to
separate the lesion from the background.
Feature Extraction
[0112] Among the criteria employed by dermatologists to detect
melanoma, as described by Menzies rules (27) and the 7-point list
(28), texture analysis is of primary importance, since, among other
things, malignant lesions exhibit substantially different texture
patterns from benign lesions. Elbaum et al. (29) used wavelet
coefficients as texture descriptors in their skin cancer screening
system MelaFindR and other previous work (9) has demonstrated the
effectiveness of wavelet coefficients for melanoma detection.
Therefore, this library includes a large module dedicated to
texture analysis.
[0113] Feature extraction works as follows: for a given image, the
binary mask created during segmentation is used to restrict feature
extraction to the lesion area only. After placing an orthogonal
grid on the lesion, patches of size K.times.K pixels were sampled
repeated from the lesion, where K is user defined. Large values of
K lead to longer algorithm execution time, while very small values
result in noisy features. Each extracted patch is decomposed using
a 3-level Haar wavelet transform (30) to get 10 sub-band images.
Texture features are extracted by computing statistical measures,
like mean and standard deviation, on each sub-band image, which are
then are put together to form a vector that describes each
patch.
Image Classification
[0114] A support vector machine (SVM) is trained using a subset
(training set) of the total images available, and the resulting
classifier is used to determine whether the rest of the images,
i.e., test set, are malignant or benign (3).
Training:
[0115] 1. For each image in the training set,
[0116] (a) Segment the input image to extract the lesion.
[0117] (b) Select a set of points on the lesion using a rectangular
grid of size M pixels.
[0118] (c) Select patches of size K.times.K pixels centered on the
selected points.
[0119] (d) Apply a 3-level Haar wavelet transform on the
patches.
[0120] (e) For each sub-band image compute statistical measures,
namely mean and standard deviation, to form a feature vector
F.sub.i={m.sub.1, sd.sub.1, m.sub.2, sd.sub.2, . . . }.
[0121] 2. For all feature vectors F.sub.i extracted, normalize each
dimension to zero mean and unit standard deviation.
[0122] 3. Apply the K-means clustering (31) to all feature vectors
F.sub.i from all training images to obtain L clusters with centers
C={C.sub.1, C.sub.2, . . . , C.sub.L}.
[0123] 4. For each training image build a L-bin histogram. For
feature vector F.sub.i, increment the jth bin of the histogram such
that min.sub.j C.sub.i-F.sub.i.
[0124] 5. Use the histograms obtained from all the training images
as the input to a SVM classifier to obtain a maximum margin
hyperplane that separates the histograms of benign and malignant
lesions.
[0125] The value of parameter M is a trade-off between accuracy and
computation speed. Small values of M lead to more accurate
classification results, but computation time increases accordingly.
When the algorithm runs on the smartphone device, to reduce
computation time, M=10 was chosen for grid size, K=24 for patch
size, and L=200 as the number of clusters in the feature space. By
exhaustive parameter exploration (9), it was determined that these
parameters are reasonable settings for the dataset. FIG. 8
summarizes in graphical form the feature extraction and
classification steps of the proposed procedure.
Testing: Each test image is classified using the following
steps:
[0126] 1. Read the test image and perform Steps 1(a)-(e) and 2 in
the training algorithm, to obtain a feature vector F.sub.i that
describes the lesion.
[0127] 2. Build an L-bin histogram for the test image. For all
feature vectors F.sub.i extracted from the image, increment the jth
bin of the histogram, such that minj C.sub.i-F.sub.i, where this
time the cluster centers C.sub.i are the centers identified in Step
3 of the training procedure.
[0128] 3. Submit the resulting histogram to the trained SVM
classifier to classify the lesion.
[0129] For test images, likelihood of malignancy can be computed
using the distance from the SVM hyperplane. Training of the SVM
classifier was performed off-line on a desktop computer, while
testing is performed entirely on the smartphone device.
iPhone 4.RTM. Implementation
[0130] A menu based application is developed that implements the
automated procedure outlined in the previous sections (FIGS.
9A-9D). The user can take a picture of a lesion or load an existing
image from the phone photo library. The image is then analyzed on
the phone in quasi real time and the results of classification are
displayed on the screen.
Comparison of the Segmentation Methods
[0131] To assess the performance of the proposed application the
automatic segmentation is compared with manual segmentation. For
each image an error, defined (32) as the ratio of the
nonoverlapping area between automatic and manual segmentation
divided by the sum of the automatic and manually segmented images,
was calculated. The error ratio is zero when the results from
automatic and manual segmentation match exactly, and 100 percent
when the two segmentations do not overlap. Thus the error is always
between zero and 100 percent, regardless of the size of the lesion.
An earlier study found that when the same set of images was
manually segmented by more than one expert, the average variability
was about 8.5 percent (32). This same figure was used and included
an additional tolerance of 10 percent error to account for the
large number of images in the dataset. Therefore, the cutoff for
error ratio was set to 18.5 percent, considering that a lesion is
correctly segmented by the automated procedure if the error ratio
is less than 18.5 percent.
[0132] The dataset of 1300 skin lesion images was segmented using
the three segmentation techniques mentioned previously. Table 4
shows the number of images correctly segmented, and the mean and
standard deviation of error for all images. The active contour
method was found to be the most accurate, as it had the highest
number of images segmented correctly and least mean error ratio.
ISODATA and Fuzzy c-Means, in that order, followed the active
contour method in accuracy. FIGS. 10A-10C show the error
distribution of three segmentation methods, where the number of
images is plotted against the error ratio. The threshold of 18.5%
is marked as the vertical dotted line. Of the 1300 images examined,
754 images had error ratio below 8.5% (variability of error among
domain experts), 1147 images had error ratio below 18.5% (threshold
for correct segmentation), and 153 images with error ratio above
18.5%. This seemingly high error is due to the fact that manual
lesion segmentation yields a smooth boundary, while automatic
segmentation detects fine edges on the border.
TABLE-US-00004 TABLE 4 Performance of Segmentation Techniques Fuzzy
Active ISODATA c-Means Contour Images correctly segmented 883 777
1147 Images incorrectly segmented 417 523 153 Mean Error 19.46%
20.40% 9.69% Standard Deviation Error 22.41 19.80 6.99
Classification Accuracy
[0133] 10 trials of 10-fold cross validation were performed on the
set of 1300 images. The dataset was divided into 10 folds, nine
folds with 39 melanoma and 92 benign lesions and the remaining fold
with 37 melanoma and 87 benign lesions. Of the 10 folds, nine were
used for training and one was used for testing. 10 rounds of
validation were performed where each fold was chosen for testing,
to get 10.times.10=100 experiments. An average over these 100
experiments demonstrated 80.76% sensitivity and 85.57%
specificity.
[0134] FIG. 11 shows the receiver operating characteristic (ROC)
curve of classification computed from testing data (33), whereby
the area under the curve is 91.1%. The threshold to maximize the
mean of sensitivity and specificity on the training set was chosen.
The 95% confidence interval on testing data was estimated using a
binomial distribution for sensitivity to be [77.1%, 83.9%] and for
specificity was estimated to be [83.5%, 87.4%]. The classification
accuracy is same on the desktop computer and iPhone 4.RTM.
smartphone.
Execution Time
[0135] The time taken for active contour segmentation and
classification on the Apple.RTM. iPhone 4.RTM. smartphone is
compared with a typical desktop personal computer (2.26 GHz
Intel.RTM. Core.TM. 2 Duo with 2 GB RAM). The classification time
includes time taken for feature extraction. The average image size
in the dataset is 552.times.825 pixels. Table 5 shows computation
time in seconds for both platforms. For the largest image in the
dataset which has dimensions 1879.times.1261 pixels, segmentation
takes 9.71 sec and classification takes 2.57 sec on the
iPhone.RTM.. Thus, the whole analysis procedure takes under 15 sec
to complete. This proves that the library is light enough to run on
a smartphone which has limited computation power.
TABLE-US-00005 TABLE 5 Computation Time Mean time (sec) Apple .RTM.
IPhone 4 .RTM. Desktop computer Segmentation 883 777 Classification
417 523
Example 5
Detection of Blue-Whitish Veil in Melanoma Using Color
Descriptors
Dataset
[0136] There were 1,009 ELM skin lesion images collected from a
widely available commercial database (7), with full annotations for
the ABCD rule and 7-point checklist. In this dataset, 252 images
are benign, 757 images are melanoma. Presence of 163 blue-whitish
veil skin lesions in this dataset were labeled by expert
dermatologists.
Local Classification
[0137] For each local neighborhood of pixels, color histograms were
computed, i.e. distribution of pixel intensities in various color
models (RGB, HSV, YUV, O1O2O3, Nrgb). These color models are used
because of their invariance to change in lighting conditions (34).
These local features are used for detection of color in that local
neighborhood.
Training: For all images belonging to the training set perform the
following steps:
[0138] 1) Segment the input image to extract the skin lesion
(ROI).
[0139] 2) Perform transformation of the input RGB image to
different color spaces (O.sub.1O.sub.2O.sub.3, HSV, Nrgb, YUV).
[0140] 3) From ROI select non-overlapping patches of size K.times.K
pixels.
[0141] 4) Extract low-level color features from these K.times.K
patches. For each channel in all color spaces build a separate P
bin histogram H. [0142] a) For all pixels belonging to the
extracted patch, increment the j.sup.th bin of histogram where
j=I.sub.c/P.times.M.sub.c. I.sub.c is pixel intensity and M.sub.c
is maximum intensity in the specified color space. [0143] b)
Concatenate all the extracted histograms to form a feature vector
F.sub.i={H.sub.1,H.sub.2, . . . }, for a given patch in the image.
[0144] c) Based on prior knowledge of the patch color, mark the
feature vector as blue-whitish veil or not blue-whitish veil.
[0145] 5) Perform step 4 for all the patches in the ROI to obtain
F.sub.i's.
[0146] 6) Input all F.sub.i extracted from step 5 to linear SVM to
obtain maximum margin hyperplane.
Testing: For all images belonging to the testing set perform the
following steps:
[0147] 1) Perform steps 1-4 from training.
[0148] 2) For each feature vector F.sub.i belonging to a patch.
in the image use SVM to classify as blue-whitish veil or not
blue-whitish veil.
[0149] 3) Classify all extracted patches in the ROI.
Global-Level Classification: Approach 1
[0150] In the second step local classification features are used to
perform a global-level classification. Experiments were performed
with two choices of global level classifiers. The first global
classifier builds a probability distribution of the local
classification result. The second global classifier uses a trivial
approach to mark positive presence of color, when one or more local
neighborhoods have been marked with presence of color.
Training: After applying patch level classification on all images
in the training set, the following steps were performed:
[0151] 1) For each patch in the training image perform patch-level
classification to obtain probability estimate of the membership of
the patch to blue-whitish veil/not blue-whitish veil.
[0152] 2) Build a B bin, global-level histogram G.sub.i for each
training image. For all patches in the training image:
[0153] a) Increment the j.sup.th bin of the histogram G.sub.i,
where j=round(P.sub.es/B), P.sub.es is the probability estimate of
a patch membership to blue-whitish veil/not blue-whitish veil.
[0154] 3) Perform steps 1 and 2 for all images in the training set
to obtain histogram G.sub.i.
[0155] 4) Input all G.sub.i obtained from step 3 to linear SVM to
obtain maximum margin hyperplane.
Testing:
[0156] 1) Perform steps 1 and 2 for a test image to obtain
histogram G.sub.i.
[0157] 2) Use SVM from the training to mark presence of
blue-whitish veil in the given test image.
Global-Level Classification: Approach 2
[0158] In this classifier a trivial approach is used, i.e., an
image with one or more blue-whitish patch is marked for positive
presence of blue-whitish veil. If none of the patches in the image
have blue-whitish veil, then blue-whitish veil is absent from the
image.
Classification Results
[0159] A set of 326 non blue-whitish veil lesions were selected
randomly and were combined with 163 blue-whitish veil lesion to
form a subset. The reason for subset selection is to have a
proportionate representation of the both classes. For each trial of
cross-validation a new subset of non blue-whitish veil images was
selected randomly from the whole dataset.
Ten trials of 10-fold cross validation were performed on a set of
489 images. The dataset was divided into 10 folds, nine folds with
16 blue-whitish veil lesions and 32 lesions where it is absent and
the remaining fold with 19 blue-whitish veil lesions and 38 non
blue-whitish veil lesions. Out of the 10 folds, nine are used for
training and one for testing. 10 rounds of validation were
performed, such that each fold was chosen for testing at least
once. Therefore there were 10.times.10=100 experiments. An averaged
over 100 experiments was obtained which demonstrated sensitivity
and specificity of the algorithm. FIG. 12 depicts the results of
blue-whitish veil detection on th skin lesion images. Table 6 shows
the classification accuracy of both global level approaches. It was
observed that the trivial approach performs better.
TABLE-US-00006 TABLE 6 Classification Accuracy of blue-whitish veil
detection Sensitivity Specificity Global-level Approach 1 90.59%
65.50% Global-level Approach 2 95.27% 63.90%
[0160] The low specificity of the blue-whitish veil detection is
because of large false positives due to regression structures.
Regression structures are one of the minor criterion of the 7-point
checklist. It is defined as associated white and blue areas which
are virtually indistinguishable from blue-whitish veil (7).
Experiments also were performed for detection of both blue-whitish
veil and regression structures. Table 7 shows the classification
accuracy of both global level approaches. It was observed that the
specificity has increased substantially because of lower false
positives.
TABLE-US-00007 TABLE 7 Classification Accuracy of blue-whitish veil
and regression structures Sensitivity Specificity Global-level
Approach 1 95.64% 72.30% Global-level Approach 2 96.66% 68.28%
[0161] Parameter exploration was performed to find the most
suitable choice of the non-overlapping square patch size used for
extraction of local color features. FIG. 13A shows classification
accuracy by varying the patch size. It was observed that small
patch size introduce noise and for large patch sizes the
performance degrades because good discriminative local features can
no longer be detected. It also was observed that first global
approach is more stable to the choice of patch size.
[0162] Parameter exploration was performed to find the most
suitable choice for histogram quantization of color models to
represent local features. FIG. 13B shows the classification
accuracy with varying bin size of locallevel color histograms. The
first global approach depends upon the quantization of the
global-level feature histogram. FIG. 14 illustrates that smaller
bin size of histogram has better specificity.
Simulation of Variance in Lighting Condition
[0163] Variance in lighting conditions was simulated by scaling and
shifting the pixels intensity values in the dataset to show
stability of the algorithm. The pixels intensity was multiplied by
a scaling factor that varies from 0.25 to 2.0. FIG. 15A shows that
the classification accuracy is invariant to light intensity
scaling. Illumination change also was simulated by shifting pixels
intensity values in the dataset images. The pixels intensity was
shifted by adding and subtracting a value that varies from -50 to
50. FIG. 15B shows that the classification accuracy is invariant to
light intensity shifting.
Example 6
Instantiation of 7-Point Checklist on Smart Handheld Devices
Dataset
[0164] Only images considered as low difficulty by the experts (7)
were chosen. There were 385 low difficulty images in the database
(7) and the segmentation methods described herein could provide a
satisfactory boundary for 347 (90.13%) of them. In the selected set
of 347 images: 110 were classified by the 7-point list as melanoma
and the remaining 237 were classified as benign.
Feature Extraction
[0165] To identify a region of interest (ROI), an image is first
converted to greyscale, and then fast median filtering (21) for
noise removal is performed, and followed by ISODATA segmentation
(23), and several morphological operations. From the ROI, color and
texture features relating to each criterion on the 7-point
checklist were extracted, as follows.
I. Texture Features: They provide information on the various
structural patterns (7) of 7-point checklist, such as pigmentation
networks, vascular structures, and dots and globules present in a
skin lesion. Haar wavelet coefficients (9) and local binary
patterns (34) can be utilized for melanoma detection.
[0166] Haar Wavelet: From the ROI non-overlapping K.times.K blocks
of pixels were selected, where K is a user defined variable.
Computation time for feature extraction is directly proportional to
the block size K. The block of pixels is decomposed using a
three-level Haar wavelet transform (30) to get 10 sub-band images.
Texture features were extracted by computing statistical measures,
like the mean and standard deviation, on each sub-band image, which
are then combined to form a vector W.sub.i={m.sub.1, sd.sub.1,
m.sub.2, sd.sub.2, . . . }. Haar wavelet extraction for texture
feature is as follows:
[0167] 1) convert the color image to greyscale; select a set of
points in the ROI using a rectangular grid of size M pixels.
[0168] 2) Select patches of size KxK pixels centered on the
selected points.
[0169] 3) Apply a 3-level Haar wavelet transform on the
patches.
[0170] 4) For each sub-band image compute statistical measures,
namely mean and standard deviation, to form a feature vector
W.sub.i={m.sub.1, sd.sub.1, m.sub.2, sd.sub.2, . . . }.
[0171] 5) For all feature vectors W.sub.i extracted, normalize each
dimension to zero-mean and unit-variance.
[0172] 6) Apply K-means clustering (31) to all feature vectors
W.sub.i from all training images to obtain L clusters with centers
C={C.sub.1, C.sub.2, . . . , CL}.
[0173] 7) For each image build an L-bin histogram H.sub.i. For
feature vector W.sub.i, increment the jth bin of the histogram such
that min.sub.j.parallel.C.sub.j-W.sub.i.parallel..
[0174] The value of parameter M is a trade-off between accuracy and
computation speed. When the algorithm runs on a handheld device to
reduce computation time, M=10 for the grid size, K=24 for patch
size, and L=200 as the number of clusters in the feature space were
chosen. As previously demonstrated, these parameters are reasonable
settings (9).
[0175] Local Binary Pattern (LBP): LBP is a robust texture operator
(35) defined on a greyscale input image. It is invariant to
monotonic transformation of intensity and invariant to rotation. It
is derived using a circularly symmetric neighbor set of P members
on a circle of radius R denoted by LBP.sub.PR.sup.riu (35). The
parameter P represents the quantization of angular space in the
circular neighborhood, and R represents the spatial resolution. A
limited number of transitions or discontinuities (0/1 changes in
LBP) are allowed to reduce the noise and for better discrimination
of features. The number of transitions in LBP were restricted to P,
and transitions greater than that are considered equal. An
occurrence histogram of LBP with useful statistical and structural
information is computed as follows:
[0176] 1) Convert the color image to greyscale.
[0177] 2) Select pixels belonging to ROI and compute local binary
pattern LBP.sub.PR.sup.riu (35).
[0178] 3) Build an occurrence histogram, where the j.sup.th bin of
the histogram is incremented, if the number of transitions in LBP
is j.
[0179] 4) Repeat steps 2 and 3 for all pixels in ROI.
The occurrence histograms for LBP.sub.16,2 and LBP.sub.24,3 were
built and concatenate them to form a feature vector L.sub.i. II.
Color Features: Detection of the 7-point checklist criteria, such
as blue-whitish veil and regression, which consist of mixtures of
certain colors, can be achieved by analyzing the color intensity of
pixels in the lesion (36). To reduce the variance due to the
lighting conditions in which dermoscopic images were taken, the HSV
and LAB color spaces were considered also, which are invariant to
illumination changes (34).
[0180] Color Histograms: To extract the color information of a
lesion, a color histogram was computed from the intensity values of
pixels belonging to the ROI. Additional images in the HSV and LAB
color spaces are obtained from the original RGB image. The
intensity range of each channel is divided into P fixed-length
intervals. For each channel a histogram was built to keep count of
the number of pixels belonging to each interval, resulting in a
total of nine histograms from three color spaces. Statistical
features, such as standard deviation and entropy (Eq. 9), of the
nine histograms are also extracted as features for classification.
More specifically, entropy is defined as
entropy = i = 1 P f ( histogram [ i ] ) ( Eq . 9 ) ##EQU00006##
where histogram[i] is the normalized pixel count of i.sup.th bin
and
f ( n ) = { n .times. log 2 ( n ) if n > 0 0 if n = 0.
##EQU00007##
The color histogram feature extraction steps are as follows:
[0181] 1) Obtain skin lesion image in HSV and LAB color space from
input RGB image
[0182] 2) For each channel in all three color spaces build a
separate P bin histogram
[0183] 3) For all pixels belonging to ROI, increment the j.sup.th
bin of histogram where j=I.sub.c/P.times.M.sub.c. I.sub.c and
M.sub.c are pixel intensity and maximum intensity in the specified
color space.
[0184] 4) Compute the standard deviation and entropy of the
histogram.
[0185] 5) Repeat steps 3 and 4 for all the channels in RGB, HSV,
and LAB color space. Color histogram and statistical features are
combined to form a feature vector C.sub.i.
Classification
[0186] The features from color histogram C.sub.i, Haar wavelet
H.sub.i, and LBP L.sub.i are combined to form F.sub.i={C.sub.i,
H.sub.i, L.sub.i}. For each criterion in the 7-point checklist a
filtered feature selection was performed to obtain a subset of
F.sub.i with the highest classification accuracy. Correlation
coefficient values (between F.sub.i and each criterion) are used as
the ranking criterion in the filters. The size of the subset and
the parameters of the linear support vector machine (SVM) are
obtained by grid search. Each criterion requires both training and
testing.
Training: The training algorithm is as follows:
[0187] 1) Segment the input image to obtain region of interest.
[0188] 2) Extract Color histogram, Haar wavelet, and Local binary
pattern. Concatenate them to form feature vector F.sub.i.
[0189] 3) Repeat steps 1 and 2 for all training images.
[0190] 4) Perform filter feature selection to choose a subset of
features S.sub.i from F.sub.i.
[0191] 5) Input S.sub.i to linear SVM to obtain maximum margin
hyperplane.
Testing: Image classification is performed as follows:
[0192] 1) Read the input image and perform steps 1 and 2 from the
training algorithm.
[0193] 2) For each criterion use the SVM coefficients obtained from
training to make a prediction.
[0194] 3) Scores from all major and minor criteria are summed up.
If the score is greater than or equal to 3 then lesion is
classified as melanoma, otherwise as benign.
Classification Results
[0195] Generally, the end user can take an image of the skin lesion
using the 5 megapixel built in camera with LED flash, or load the
image from media library. The image is analyzed in quasi real time
and the final result displayed on the screen. FIGS. 16A-16B depict
the menu based application in use on an Apple.RTM. iPhone.RTM.
device showing an image of a skin lesion acquired by the device
with the options of choose image, take photo or 7-Point Rule. The
next screen displays the scores for each of the 7 criteria, a total
score and a diagnosis based on the same.
[0196] A 10-fold cross validation was performed on the set of 347
images to test the menu based application by comparing the
classification accuracy of each criterion separately against the
overall final classification by expert physicians (7). The dataset
was divided into 10 folds, nine folds with 11 melanoma and 23
benign lesions and the remaining fold with 11 melanoma and 30
benign lesions. Of the 10 folds, nine were used for training and
one was used for testing. 10 rounds of validation were performed
where each fold was chosen for testing and the rest were for
training, to get 10 experiments. The classification accuracy of
each criterion was compared and the overall decision of the 7-point
checklist with dermatology and histology.
[0197] Table 8 presents the sensitivity and specificity of the
algorithm in classification of each of the 7-point checklist
criterion. There was lower accuracy for the regression structures,
because they are usually indistinguishable from the blue-whitish
veil via dermoscopy (7). However, this is not an issue, as is it
only necessary to obtain a minimum score of 3 to correctly detect a
melanoma.
TABLE-US-00008 TABLE 8 Classification for all Criteria Sensitivity
Specificity Atypical Pigment Network 72.86% 70.40% Blue-Whitish
Veil 79.49% 79.18% Aypical Vascular Pattern 75.00% 69.66% Irregular
Streaks 76.74% 79.31% Irregular Pigmentation 69.47% 74.21%
Irregular Dots and Globules 74.05% 74.54% Regressive Structures
64.18% 67.86%
[0198] In Table 9 the sensitivity and specificity of the algorithms
is compared with the decision made by expert clinicians via
dermoscopy. Table 10 presents the confusion matrix computed using
the sum of ten confusion matrices from the ten test sets, given
from the 10-fold cross validation. Another classification
experiment using SVM also was performed, where the 7-point
checklist was ignored and each skin lesion was directly classified
as melanoma or benign. The feature vectors, feature selection
scheme, and final ground truth (melanoma/benign) are the same as
the classification using the automated 7-point checklist. Table 5
shows that classification accuracy is much lower when the 7-point
checklist is ignored.
TABLE-US-00009 TABLE 9 Classification Accuracy Sensitivity
Specificity 7-Point Checklist 87.27% 71.31% Ignoring 7-Point
Checklist 74.78% 70.69%
TABLE-US-00010 TABLE 10 Confusion Matrix of the Automated Decision
Predicted Confusion Matrix Melanoma Benign Dermoscopy Melanoma 96
14 Benign 68 169
Execution Time
[0199] The time needed for classification using the ISODATA
segmentation algorithm (23) on the Apple.RTM. iPhone 3G.RTM. is
compared with a typical desktop computer (2.26 GHz Intel.RTM.
Core.TM. 2 Duo with 2 GB RAM, Intel Corporation). The average image
size in the dataset is 552.times.825 pixels. The classification
time includes time taken for feature extraction. Table 11 shows
computation time in seconds for both platforms. It can be seen that
the whole procedure takes under 10 sec to complete thereby
demonstrating that the application is light enough to run on a
smartphone which has limited computation power.
TABLE-US-00011 TABLE 11 Mean Computation Time Mean time (sec) Apple
.RTM. IPhone 3G .RTM. Desktop Computer Segmentation 2.3910 0.1028
Classification 7.4710 0.2415
Example 7
Implementation for Buruli Ulcers
[0200] FIGS. 17A-17B are flowcharts depicting the algorithm steps
in the segmentation, feature extraction and classification modules
for identification/classification of an object of interest, for
example, a Buruli ulcer, as in the below example, or a
melanoma.
Segmentation
[0201] Both color and luminance are important characteristics for
Buruli lesion segmentation. The key idea of the segmentation method
simply starts by considering the common foreground and background
obtained by the luminance and color components as lesion and skin,
respectively, and then applying a supervised classifier to the
remaining pixels is key. Segmentation 100 comprises the following
steps.
[0202] First, contour initialization 110 comprises the following
steps:
[0203] 1) Read input RGB image at 111;
[0204] 2) Transform RGB image to four other color spaces: La*b*,
HSV, YCbCr, Lu*v* at 112;
[0205] 3) Apply Otsu's thresholding method (1) to eight color
channels: a*, b*, H, S, Cb, Cr, u*, v*, to obtain eight
segmentation masks at 113.
[0206] 4) Fuse these eight masks by a voting system to form a new
mask at 114. For each pixel, if more than three masks agree to be
foreground, then it is classified as a lesion pixel;
[0207] 5) Draw a convex hull at 115 which covers the fused mask to
be the initialized contour for the following steps.
[0208] Secondly, contour evolution 120 comprises the following
steps:
[0209] 1) For each segmentation mask obtained from the eight color
channels, calculate the correlation coefficient with the fused mask
at 121;
[0210] 2) Apply the Chan-Vese Level Set segmentation method (2) for
the color channel which has the largest correlation coefficient, to
obtain a mask based on color information M.sub.c at 122.
[0211] Basically, in Chan-Vese, given an image I.OR right..OMEGA.,
the region-based active contour model (37) assumes that image I is
formed by two regions of approximately piecewise constant intensity
c1 and c2 separated by a curve C, which minimizes the energy-based
objective function:
? ( c 1 , c 2 , C ) = .mu. length ( C ) + .lamda. 1 ? 1 N i = 1 N I
i ( x ) - c 1 , i 2 x + .lamda. 2 ? 1 N i = 1 N I i ( x ) - c 2 , i
2 x ? indicates text missing or illegible when filed ( Eq . 10 )
##EQU00008##
where the parameters .mu.>0 and .lamda..sub.1,.lamda..sub.2>0
are positive weights for the regularizing term and the fitting
term, respectively. When applying the level set approach (38), the
curve C can be represented as the zero level set C(t)={(x)|.phi.(t,
x)=0} of a higher dimensional level set function .PHI.(t, x). Then
the energy function can be rewritten as
E ( .PHI. , c 1 , c 2 ) = .mu. .intg. .OMEGA. ? ( .PHI. )
.gradient. .PHI. x + .intg. .OMEGA. 1 N i = 1 N .lamda. 1 I i ( x )
- ? 2 H ( .PHI. ) x + .intg. .OMEGA. 1 N i = 1 N .lamda. 2 I i ( x
) - ? 2 ( 1 - H ( .PHI. ) ) x , ? indicates text missing or
illegible when filed ( Eq . 11 ) ##EQU00009##
where H is the Heaviside function. The evolution of is governed by
the following motion partial differential equation (PDE):
.differential. .PHI. .differential. t = ? ( .PHI. ) [ .mu. ? - 1 N
i = 1 N .lamda. 1 I i ( x ) - ? 2 + 1 N i = 1 N .lamda. 2 I i ( x )
- ? 2 ] ? indicates text missing or illegible when filed ( Eq . 12
) ##EQU00010##
where .delta.(.PHI.) is a regularized version of the Dirac delta
function. The evolution can be solved using finite differences, by
updating each c.sub.1,i and c.sub.2,i by the average of channel I,
calculated inside (C) and outside (C).
[0212] 3) Transform RGB image to gray scale image at 124; and
[0213] 4) Apply the Chan-Vese Level Set segmentation method for the
gray scale image, to obtain a mask based on luminance M.sub.i at
125.
[0214] Thirdly, pixel classification 130 comprises the following
steps:
[0215] 1) For each pixel at 131, if it belongs to the common
foreground of M.sub.c and M.sub.i, then it is classified as a
foreground pixel or if it belongs to the common background of
M.sub.c and M.sub.i, then it is classified as a background pixel or
it remains to be determined;
[0216] 2) From the common background and foreground of M.sub.c and
M.sub.i, randomly sample 5000 pixels respectively, train a linear
SVM using RGB and Lu*v* value at 132;
[0217] 3) For each remaining pixel at 133, its RGB and Lu*v* value
are used as input for the classifier to obtain a decision on this
pixel being background or foreground.
Feature Extraction and Classification
[0218] As the first step, image sampling is a critical component
when bag-of-features methods are used for image classification. The
algorithms provided herein enable more patches from dermoscopic
interest regions to be sampled based on saliency values. Given a
patch, saliency (39) is defined as shown in Eqs. 5-7.
[0219] Feature extraction 200 comprises the following steps:
[0220] 1) Read input for the RGB image and the segmentation result
in the region of interest (ROI) at 201;
[0221] 2) Extract color moment and wavelet coefficients for each
patch inside ROI and assign a corresponding cluster number for each
patch at 202;
[0222] 3) Calculate saliency value according to Eqs. (5-7) for each
patch inside ROI at 203;
[0223] 4) Use k-means clustering to separate the lesion into two
regions based on saliency values at 204;
[0224] 5) Calculate the average intensity for separate regions
respectively at 205. Region with higher intensity is denoted as
R.sub.h, and R.sub.l is for region with lower intensity;
[0225] 6) Decide, at 206, sampling percentage for two separate
regions by
P.sub.i=(.alpha.A.sub.i/.alpha.A.sub.i+A.sub.h).times.100% (Eq.
13),
where P.sub.h is the percentage of patches sampled from R.sub.h,
A.sub.h and A.sub.l are area of R.sub.h, and R.sub.l respectively;
.alpha. is a coefficient to control the percentage, here .alpha. is
set to be 1.5; and
[0226] 7) Randomly sample patches from R.sub.h, and R.sub.l by
corresponding sampling percentages and extract the bag-of-feature
representation for each lesion at 207.
[0227] Classification 300 comprises the steps of:
[0228] 1) Training for SVM using manually segmented images with
known labels for Buruli and non-Buruli ulcers at 301; and
[0229] 2) Extracted features are used as input at 302 for the
classifier to obtain a decision whether or not the lesion is a
Buruli lesion.
Example 8
Obtention of Dermoscopic Images for Detection of Buruli Ulcers
[0230] Images were 24 bit full color with typical resolution of
4320.times.3240 pixels. Data were collected in endemic BU
communities of Cote d'Ivoire and Ghana with the help of local
collaborators to the project that included medical doctors,
District Surveillance Officers, and community health workers, using
a DermLite II Multi-Spectral device (www.dermlite.com) for image
acquisition. The device could provide white light for
crosspolarization epiluminescence imaging, blue light for surface
coloration, yellow light for superficial vascularity, and red light
for deeper coloration and vascularity, using 32 bright LEDs, eight
per color. This device was attached to a Sony Cybershot DSC-W300
high-resolution camera, which provided a resolution of 13.5 MP. The
study has received IRB approval from the Human Subjects Protection
Committee at the University of Houston, as well as in Ghana and
Ivory Coast, and all subjects and their parents gave written
informed consent to the study in their native language.
Example 9
Application of Segmentation Scheme to Suspected Buruli Lesions
[0231] A set of dermoscopic images of 26 suspected BU lesions were
obtained as described herein. In the preprocessing step, images
were first downsampled to 1080.times.810 pixels, and then processed
with a 5.times.5 median filter and a Gaussian lowpass filter of the
same size to remove extraneous artifacts and reduce the noise
level. For postprocessing, morphological filtering was applied, and
a distance transform (40) was used to make the borders smoother. As
ground truth for the evaluation of the border detection error, for
each image, manual segmentation was performed by a field expert in
Africa just after acquisition. Three different metrics were used to
quantify the boundary differences, namely XOR error rate (XER)
(41), true detection rate (TDR), and false positive rate (FPR)
(11), defined as follows,
XER(A;M)=(A M)=M 100%
TDR(A;M)=(A\M)=M 100%
FPR(A;M)=(A\M)=M 100%; (Eq. 14)
where A denotes the area of automatic segmentation and M denotes
the manual segmentation area obtained by the expert.
[0232] Images illustrating segmentation schemes are shown in FIGS.
18A-18D. Particularly, FIG. 18A shows the original lesion with the
manual segmentation from the expert. The lesion consists of two
main parts: central areas with variegated distinctive colors, and
the surrounding erythematous areas which exhibit a smooth
transition to normal skins. Also, the complex texture of normal
skin caused by the infected skin makes the segmentation task more
challenging. FIG. 18B shows the initial contour obtained by the
fusion of thresholding segmentations from different color channels.
The initial mask covers the most significant lesion colors. FIGS.
18C-18D present the segmentation results after contour evolution in
the color and luminance components, respectively. It is obvious
that the segmentation in the color channel is good at detecting the
central area of a lesion with significant colors and misses the
surrounding areas, while segmentation in the luminance channel is
able to find the surrounding area, but always includes part of
normal skin because of the smooth transition. The combination of
color and luminance information by pixel classification is shown in
FIG. 18E, while FIG. 18F presents the final segmentation result
after morphological postprocessing. The latter is close to the
expert's segmentation and detects both parts of the lesion
successfully.
Comparison of Segmentation with Other Methods
[0233] The proposed segmentation method (based on Fusion and
Classification, FC) was compared with three popular methods applied
to skin lesion segmentation, namely adaptive thresholding (AT) (7),
gradient vector flow (GVF) (8), and level set (LS) (11)
segmentation. The initialization of contour for GVF and LS were
both completed by the first step of the segmentation scheme. For
GVF snake, the elasticity, rigidity, viscosity, and regularization
parameters were .alpha.=0:05, .beta.=0:01, .gamma.=1, and k=0:6,
respectively. The maximum iteration number was 75. The LS method
was processed in the L*a*b* color space, using parameters
.lamda..sub.1=1, .lamda..sub.2=1; and .mu.=0:1. The maximum number
of iterations was 150. For this segmentation scheme, the same
parameters as in the LS method were used for the contour evolution
step, where 5000 foreground and 5000 background points were
randomly sampled to train the classifier. The segmentation results
obtained are shown in FIGS. 19A-19D. Among these approaches, the AT
and LS methods were disturbed by the illumination of the
surrounding normal skins, the GVF method converged to some noisy or
spurious edge points, while the method described herein
successfully detected both the central and surrounding areas of the
lesion, resulting in an accurate border.
[0234] To quantify the performance of different segmentation
methods, three different metrics, namely XER (41), TDR, and FPR
(42) were used to measure the segmentation accuracy, as described
(43). XER is computed as the number of pixels for which the
automatic and manual borders disagree divided by the number of
pixels in the manual border. It takes into account two types of
errors: pixels classified as lesion by the expert that were not
classified as such by the automatic segmentation and pixels
classified as lesion by the automatic segmentation that were not
classified as such by the expert, while the TDR method focuses on
the former and the FPR focuses on the latter, respectively. Table
12 shows the segmentation performance of the different methods.
TABLE-US-00012 TABLE 12 Segmentation performance of different
methods Methods XER (std) TDR (std) FPR (std) AT 39.46 .+-. 26.14
84.84 .+-. 17.22 24.30 .+-. 00 GVF 24.01 .+-. 12.02 79.10 .+-.
12.97 4.17 .+-. 4.08 LS 26.54 .+-. 19.78 90.06 .+-. 8.44 16.60 .+-.
21.42 FC 19.25 .+-. 9.28 85.70 .+-. 9.86 5.15 .+-. 5.3
[0235] The LS method can achieve the highest TDR at the cost of a
higher FPR, because it always includes lesions and part of normal
skin. On the contrary, the GVF method performs the best in FPR at
the cost of missing some actual lesion areas. Overall, the
segmentation method provided herein can achieve the best XER while
keeping a relatively high TDR and low FPR, and outperform other
state-of-art segmentation methods in Buruli lesion images.
Example 10
A Classifier for Automatic Detection of Buruli Lesions
[0236] A set of dermoscopic images of 58 lesions, in which 16
lesions were confirmed BU and 42 lesions were non-BU lesions were
obtained, as described herein. Images were first downsampled to
1080 .ANG..about.810 pixels, then manual segmentation of all
lesions was applied to ensure that evaluation of classification
performance was not affected by possible discrepancies in the
automated identification of the lesion boundaries. The default
setup for bag-of-features was as follows: patches were sampled on a
regular grid of 5 .ANG..about.5, with patch size 24 .ANG..about.24
pixels; color moments and wavelet coefficients were the patch
descriptors; the codebook was generated by k-means clustering with
a size of 50 codewords; an SVM classifier with RBF kernel was used
for the final classification step. Leave-One-Out cross-validation
was implemented to evaluate the performance of the method.
Performance criteria included sensitivity, specificity, and
balanced accuracy (BAC, i.e., average of sensitivity and
specificity).
Codeword Representation
[0237] The idea of bag-of-features is that the large set of
collected samples can be automatically arranged into sub-clusters
sharing similar color and texture patterns. If some of these image
patterns, i.e., cluster centroids, are distinctive, then the
distributions of image patterns which represent the skin lesions
can have strong discriminative power. In other words, if the common
image patterns of BU images are distinguishable enough from those
patterns of non-BU images, then the bag-of-features method can be a
good way to classify BU and non-BU images.
[0238] FIGS. 18A-18B show the shared image patterns of BU and
non-BU lesions, respectively. The collection of samples from BU and
non-BU images is clustered into 15 subclasses, and the patches that
are closest to the cluster centroids are displayed. Most of the BU
image patterns are light in color and homogeneous in texture,
corresponding to discolored and necrotic skin, while non-BU
patterns are darker and have more complex textures.
Effect of Sampling Strategies
[0239] One of the main parameters governing classification accuracy
and processing time is the number of patches sampled. Since lesions
differ in size, the number of patches proportional to that lesion's
area was chosen. Regular grid sampling and random sampling were
applied with a patch size of 24.times.24 pixels, respectively. Grid
sampling extracted patches on a regular grid with size chosen from
the set {1, 2, 5, 10, 20, 50, 100}. Using a grid size g is
equivalent to sampling approximately (100/g.sup.2) % points from a
lesion. Random sampling sampled patches randomly with the
corresponding percentage. FIG. 21A shows the classification
accuracy for different patch numbers. For both grid and random
sampling, accuracy increases significantly as more patches are
sampled, but it starts to converge when more than 4% of patches are
sampled. Thus, only 4% of points need to be sampled from the lesion
to achieve a maximum accuracy, but in substantially shorter
time.
[0240] Patch size is another factor that can affect time and
accuracy. Square patches of a size chosen from the set {8, 16, 24,
32, 40} on a grid of size 5.times.5 were extracted. FIG. 21B
illustrates the impact of patch size on classification performance.
A medium patch size of 24.times.24 pixels achieved the best
performance in our experiments. Small patches can be processed very
fast, but they capture less information, while large patches
provide very good sensitivity. However, patches of very large size
ignore some details of local characteristics and result in much
higher computational complexity.
Effect of Patch Descriptors
[0241] In the bag-of-features method, patch descriptors are used to
characterize image patches and to discover similar patterns across
images. Different patch descriptors were tested, i.e., color
moments and wavelet coefficients individually, as well as the
combination of these two. Color moments captured color and shape
information, and wavelet coefficients captured texture-related
features. Table 13 shows that both single descriptors can achieve
an accuracy around 80%, but that the combination can make a
significant improvement to 95%, indicating that both color and
texture are important to discriminate BU from non-BU images.
TABLE-US-00013 TABLE 13 Classification Performance of Different
Patch Descriptors Descriptors Sensitivity (%) Specificity (%)
Accuracy (%) Color 87.50 69.05 78.27 Texture 87.50 83.33 85.41
Combined 100 90.48 95.24
Effect of Codebook Size
[0242] The number of codebook centers is another factor that
affects the performance of bag-of-feature methods. Five codebook
sizes were chosen from the set {10, 25, 50, 100, 200}. When the
codebook size is small, patches are assembled into fewer groups,
therefore the discriminative power is not strong. As patches are
grouped into more clusters, the accuracy also increases; however,
when the codebook size becomes too large, the dimension of the
feature vector is also very large, so the overall performance
decreases because of over-fitting.
Effect of SVM Kernels
[0243] The performance of different types of SVM kernels was
investigated. Table 14 shows that the performance of the linear
kernel is the worst, while the nonlinear RBF and the chi square
kernels, which map the feature vector to a higher dimension feature
space, can achieve better performance.
TABLE-US-00014 TABLE 14 Classification Performance of Different SVM
Kernels Kernels Sensitivity (%) Specificity (%) Accuracy (%) Linear
81.25 83.33 82.29 RBF 100 90.48 95.24 Chi-square 100 88.10
94.05
Depiction of Buruli Ulcer on a Smart Device
[0244] The algorithms described herein can detect and diagnose a
Buruli ulcer in early or late stage (FIG. 22A). FIG. 22B
illustrates the grouping of early and late lesions obtained from
the bag-of-features and feature histograms created from wavelet and
color moment features.
Example 11
Implementation for Multispectral Imaging
[0245] Lights of different frequencies can penetrate different skin
depths. For instance, blue light with a shorter wavelength of about
470 nm forms images of surface coloration, yellow light of about
580 nm forms images for superficial vascularity and red light with
a longer wavelength of about 660 nm penetrates deeper and
visualizes deeper vascularity. In this example, the algorithms are
applied to the classification of a Buruli ulcer. However, this
algorithmic process is applicable to the identification and
classification of other objects of interest, as described
herein.
Architecture Overview
[0246] The architecture 400 (FIG. 23) for the extension and control
of the processing chain for multispectral images comprises a
framework 410 having the primary tiers Script Manager 412 and
Processing Engine 415 on an application programmer interface (API)
platform 420 with associated hardware 430. The Script Manager tier
handles the configuration and execution of the DSL scripts 414 that
represent process chains. A process chain encapsulates the
particular execution steps required for analysis of a category of
skin lesion, as described herein. Processing chains, comprising one
or more process stages, are described using a DSL designed for
skin-lesion image processing. Each process stage may consist of one
or more image processing modules (IMPs), which are typically
implemented in the C or C++ programming language. The DSL exposes
these processes in a manner that allows an end user to chain IMPs,
either in serial or parallel, without having intimate knowledge of
the IMP implementation or the programming language that was used to
develop it.
[0247] The following is an example of a DSL implementation of the
process chain:
TABLE-US-00015 Define_chain "classifier_rule_chain", image do
artificat_removal hair_removal segmentations = in_parallel do
fuzzy_c_means active_contours end segmentation = score_and_return
segmentations with segmentation do extract_features classify_lesion
end end
[0248] Process chains are completely configurable with only changes
to the DSL scripts, allowing users to quickly try several analysis
approaches. IMPs can be added to the system by developers who have
minimal knowledge of the overall framework. Process chains can
include other chains, so that it is possible for example to run a
skin cancer and a Buruli analysis on the same lesion at the same
time
[0249] The processing engine 415 comprises a script processor 416
and stage components 418. The processing engine executes the preset
scripts in the script processor, returning the results to the
script manager. The processing engine is responsible for reading
and interpreting the script, managing the script as it runs and
instantiating process steps, as required. Also, the processing
engine interfaces with the underlying operating system's API 420 to
facilitate the use of native processing capabilities, including
process and memory management and user interaction.
Segmentation
[0250] Segmentation is performed as per process 100 described
herein.
Feature Extraction
[0251] In feature extraction color moments for white light images,
histogram of intensities for multispectral images, and texture
properties for both from an object of interest, such as, but not
limited to, a Buruli ulcer are extracted. Extracted features are
used as input to support vector machine (SVM), which outputs the
classification, such as whether the lesion is Buruli or not or
whether the lesion is malignant or not.
[0252] Feature extraction 500 (FIG. 24) comprises the steps of:
[0253] 1) Read at 501 the input white light image (RGB) and
segmentation result (region of interest (ROI));
[0254] 2) Read at 502 the input multispectral images of blue,
yellow, and red channel, and transform to gray scale images;
[0255] 3) Use white light image as a reference image, do image
registration for multispectral images by maximizing mutual
information at 503;
[0256] 4) Extract bag-of-feature representation within ROI from the
white light image with wavelet coefficients and color moment,
respectively at 504;
[0257] 5) Extract bag-of-feature representation within ROI from
multispectral images with wavelet coefficients and histograms,
respectively, at 505; and
[0258] 6) Pool features from the white light image and
multispectral images together, and perform feature selection to
choose relevant features at 506.
Classification
[0259] Classification (FIG. 24) is performed as per process 300
described herein.
Example 11
Optical Skin Model
[0260] Skin tissues have different absorption and scattering
properties when lights of different frequencies passing through
different skin layers. The attenuation coefficients of epidermis
and dermis are related to three parameters, e.g. the volume
fractions of melanin, blood, and oxygenation respectively. Provided
herein is an implementation of an algorithm for multispectral image
classification based on the following optical skin models.
I.sub.det(.lamda.)*I.sub.calibration(.lamda.).sup.=S*A.sub.epi(.lamda.).-
sup.2*A.sub.dermis(.lamda.) (eq. 15)
where .lamda. is the wavelength, I.sub.det(.lamda.) is the detected
intensity at each pixel for each wavelength,
I.sub.calibration(.lamda.) is the calibration factor,
A.sub.epi(.lamda.) is the attenuation of the light intensity after
passing through the epidermis, and A.sub.dermis(.lamda.) is the
attenuation of light intensity passing through dermis.
[0261] Here, A.sub.epi(.lamda.) is related to the volume fraction
of melanin. It can be determined by,
.lamda..sub.epi(.lamda.)=exp[-.mu..sub.a(epi)(.lamda.)t.right
brkt-bot. (eq. 16)
where t is the thickness of epidermis, which can be considered to
be constant as 0.6 mm, and
.mu..sub.a(epi)(.lamda.)=V.sub.mel.mu..sub.a(mel)(.lamda.)+(1-V.sub.mel)-
.mu..sub.a(skin)(.lamda.): (eq. 17)
where .mu..sub.a(mel) is the melanin absorption coefficient, and
.mu.a(skin) (.lamda.) is the absorption coefficient of normal skin.
These two coefficients are known parameters. The remaining variable
is the volume fraction of melanin in the epidermis V.sub.me.
[0262] In addition, A.sub.dermis(.lamda.) is related to volume
fraction of blood in the tissue and the percent of that blood that
is oxygenated. It can be determined by,
A.sub.dermis(.lamda.)=1.06-1.45[.mu..sub.a(dermis)(.lamda.)/.mu.'.sub.s(-
.lamda.)].sup.0.35 (eq. 18)
where .mu.'.sub.s(.lamda.) is the reduced scattering coefficient of
the dermis, which is determined by the wavelength and
.mu..sub.a(dermis)(.lamda.)=V.sub.blood.mu..sub.a(blood)(.lamda.)+(1-V.s-
ub.blood).mu..sub.a(skin)(.lamda.) (eq. 19)
where V.sub.blood is the volume fraction of blood in the dermis
layer and,
.mu..sub.a(blood).lamda.)=V.sub.oxy.mu..sub.a(oxy)(.lamda.)+(1-V.sub.oxy-
).mu..sub.a(deoxy)(.lamda.) (eq. 20)
where V.sub.oxy is the fraction of blood that is oxygenated,
.mu..sub.a(oxy)(.lamda.) and .mu..sub.a(deoxy)(.lamda.) are the
absorption coefficients of HbO.sub.2 and Hb respectively. So the
three remaining variables are: V.sub.mel, V.sub.blood, and
V.sub.oxy. By inserting Eq. 16-20 into Eq. 15, and using
intensities obtained from three different channels, three unknown
physiological parameters V.sub.mel, V.sub.blood, and V.sub.oxy can
be solved.
Segmentation
[0263] Segmentation is performed as per process 100 described
herein.
Feature Extraction
[0264] Feature extraction 700 (FIG. 25) comprises the steps of:
[0265] 1) Read at 601 input white light image (RGB) and
segmentation result (region of interest (ROI));
[0266] 2) Read at 602 input multispectral images of blue, yellow,
and red channel, and transform to gray scale images;
[0267] 3) Use white light image as a reference image, do image
registration for multispectral images by maximizing mutual
information at 603;
[0268] 4) For each pixel within ROI, solve V.sub.mel, V.sub.blood,
and V.sub.oxy by Eqs. 5-7 to reconstruct maps of melanin, blood,
and oxygenating percentage at 604;
[0269] 5) Extract bag-of-feature representation within ROI from the
reconstructed maps with wavelet coefficients and histograms,
respectively, at 605; and
[0270] 6) Pool features from reconstructed images and perform
feature selection to choose relevant features at 606.
Classification
[0271] Classification (FIG. 25) is performed as per process 300
described herein.
[0272] The following references are cited herein. [0273] 1. Jemal
et al. CA: A cancer Journal for Clinicians, 60(5):277, 2010. [0274]
2. Ganster et al. IEEE Transactions on Medical Imaging,
20(3):233-239, 2001. [0275] 3. Csurka et al. Workshop on
Statistical Learning in Computer Vision, ECCV, 22(1), 2004. [0276]
4. Morris and Guilak, Pervasive Computing, IEEE, 8(2):57-61,
April-June 2009. [0277] 5. Logan et al. American Journal of
Hypertension, 20(9):942-8, 2007. [0278] 6. Hicks et al. Wireless
Health 2010, WH'10:34-43, New York, N.Y., USA 2010. [0279] 7.
Argenziano et al. Dermoscopy: A Tutorial, Vol. 12, February 2000.
[0280] 8. Argenziano et al. J Am Acad Dermatol, 48(5):679-93, 2003.
[0281] 9. Situ et al. Annual International Conference of the IEEE
Engineering in Medicine and Biology Society, 1:3110, 2008. [0282]
10. Yang et al. IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), IEEE, 2009. [0283] 11. Zhou et al. Biomedical
Imaging: From Nano to Macro, 2009, IEEE:1318-1321, 2009. [0284] 12.
Lowe, D. G., International Journal of Computer Vision,
60(2):91-110, 2004. [0285] 13. Frangi et al. Medical Image
Computing and Computer-Assisted Interventation-MICCAI 98:130, 1998.
[0286] 14. Zhou et al. Biomedical Imaging: From Nano to Macro,
2008, 5.sup.th IEEE:1318-800-803, 2008. [0287] 15. Rubinov, M. and
Sporns, O., Neuroimage, 2009. [0288] 16. Cucchiara, R. and Grana,
C., Knowledge-Based Intelligent Information Engineering Systems and
Allied Technologies: Kes 2002: 166, 2002. [0289] 17. Sadeghi et al.
Proceedings of SPIE, 7623:762312, 2010. [0290] 18. Situ et al.
Annual International Conference of the IEEE Engineering in Medicine
and Biology Society, 2010. [0291] 19. Vapnik, V., Statistical
Learning Theory, Vol. 2, Wiley New York, 1998. [0292] 20. Lanckriet
et al. Journal of Machine Learning Research, 5:27-72, 2004. [0293]
21. Huang et al. Acoustics, Speech and Signal Processing, IEEE
Transactions on, 27:13-18, February 1979. [0294] 22. Ridler, T. W.
and Calvard, S., IEEE Transactions on Systems, Man and Cybernetics,
SMC-8:630-632, 1978. [0295] 23. Dunn, J. C., Journal of
Cybernetics, 3:32-57, 1973. [0296] 24. Bezdek, J. C., Pattern
Recognition with Fuzzy Objective Function Algorithms, Plenum Press,
1981. [0297] 25. Chan and Vese, IEEE Transaction on Image
Processing, 10:266-277, 2001. [0298] 26. Shapiro, L. G. and
Stockman, G. C., Computer Vision, Prentice Hall, 2001. [0299] 27.
Menzies et al. Arch Dermatol, 132(10):1178-1182, 1996. [0300] 28.
Argenziano et al. Arch Dermatol, 134:1563-70, 1998. [0301] 29.
Elbaum et al. Jour of American Academy of Dermatology,
44(2):207-218, 2001. [0302] 30. StolInitz et al. IEEE Computer
Graphics and Applications, 15:76-84, 1995. [0303] 31. Hartigan, J.
A. and Wong, M. A., Applied Statistics, 28:100-108, 1979. [0304]
32. Xu et al. Image and Vision Computing, 17:65-74, 1999. [0305]
33. Fawcett, T., Pattern Recognition Letters, 27(8):861-874, 2006.
[0306] 36. van de Sande et al. IEEE Transactions on Pattern
Analysis and Machine Intelligence, 32(9):1582-96, 2010. [0307] 35.
Ojala et al. IEEE Transactions on Pattern Analysis and Machine
Intelligence, 24:971-987, 2002. [0308] 36. Stanley et al. Skin
Research and Technology, 13(1):62-72, 2007. [0309] 37. Chan et al.
Journal of Visual Communication and Image Representation,
11(2):130-141, 2000. [0310] 38. Osher, S. and Sethian, J. A.
Journal of Computational Physics, 79(1):12-49, 1988. [0311] 39.
Parikh et al. Computer Vision ECCV 2008, pp. 446-459, 2008. [0312]
40. Sanniti di Baja, G. and Svensson, S. In Pattern Recognition,
2000. Proceedings. 15.sup.th International Conference on, Vol
2:1030-1033, 2000. [0313] 41. Celebi et al. Computerized Medical
Imaging and Graphics, 33(2):148-153, 2009. [0314] 42. Silveira et
al. IEEE Journal of Selected Topics in Signal Processing, pages
35-45, 2009. [0315] 43. C. Drummond, C. and Butler, J. R. Emerg
Infect Dis, 10. [0316] 44. Fan et al. Visual categorization with
bags of keypoints. In workshop on Statistical Learning in Computer
Vision, ECCV, Vol. 22, 2004. [0317] 45. Leung, T. and Malik, J.
International Journal of Computer Vision, 43:29-44, 2001. [0318]
46. June et al. Proc. European Conference on Computer Vision, pgs.
490-503, 2006. [0319] 47. Gool et al. Computer Vision and Image
Understanding, 94:3-27, 2004. [0320] 48. Situ et al. Biomedical
Imaging: From Nano to Macro, 2011 IEEE International Symposium,
pgs. 109-112, Mar. 30-Apr. 2, 2011.
[0321] The present invention is well adapted to attain the ends and
advantages mentioned as well as those that are inherent therein.
The particular embodiments disclosed above are illustrative only,
as the present invention may be modified and practiced in different
but equivalent manners apparent to those skilled in the art having
the benefit of the teachings herein. Furthermore, no limitations
are intended to the details of construction or design herein shown,
other than as described in the claims below. It is therefore
evident that the particular illustrative embodiments disclosed
above may be altered or modified and all such variations are
considered within the scope and spirit of the present invention.
Also, the terms in the claims have their plain, ordinary meaning
unless otherwise explicitly and clearly defined by the
patentee.
* * * * *