U.S. patent application number 14/833182 was filed with the patent office on 2015-12-17 for method and system of classifying medical images.
This patent application is currently assigned to RAMOT AT TEL-AVIV UNIVERSITY LTD.. The applicant listed for this patent is Bar-IIan Research And Development Company Ltd., RAMOT AT TEL-AVIV UNIVERSITY LTD., Tel HaShomer Medical Research Infrastructure and Services Ltd.. Invention is credited to Uri AVNI, Jacob GOLDBERGER, Hayit GREENSPAN, Eli KONEN, Michal SHARON.
Application Number | 20150363672 14/833182 |
Document ID | / |
Family ID | 45352602 |
Filed Date | 2015-12-17 |
United States Patent
Application |
20150363672 |
Kind Code |
A1 |
GREENSPAN; Hayit ; et
al. |
December 17, 2015 |
METHOD AND SYSTEM OF CLASSIFYING MEDICAL IMAGES
Abstract
A method of generating a category model for classifying medical
images. The method comprises providing a plurality of medical
images each categorized as one of a plurality of categorized
groups, generating an index of a plurality of visual words
according to a distribution of a plurality of local descriptors in
each the image, modeling a category model mapping a relation
between each visual word and at least one of the categorized groups
according to the index, and outputting the category model for
facilitating the categorization of an image based on local
descriptors thereof.
Inventors: |
GREENSPAN; Hayit; (Tel-Aviv,
IL) ; GOLDBERGER; Jacob; (Tel-Aviv, IL) ;
AVNI; Uri; (Tel-Aviv, IL) ; KONEN; Eli;
(Tel-Aviv, IL) ; SHARON; Michal; (Moshav
Ganei-Yochanan, IL) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
RAMOT AT TEL-AVIV UNIVERSITY LTD.
Tel HaShomer Medical Research Infrastructure and Services Ltd.
Bar-IIan Research And Development Company Ltd. |
Tel-Aviv
Ramat-Gan
Ramat-Gan |
|
IL
IL
IL |
|
|
Assignee: |
RAMOT AT TEL-AVIV UNIVERSITY
LTD.
Tel-Aviv
IL
Tel HaShomer Medical Research Infrastructure and Services
Ltd.
Ramat-Gan
IL
Bar-IIan Research And Development Company Ltd.
Ramat-Gan
IL
|
Family ID: |
45352602 |
Appl. No.: |
14/833182 |
Filed: |
August 24, 2015 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
13170200 |
Jun 28, 2011 |
9122955 |
|
|
14833182 |
|
|
|
|
61358979 |
Jun 28, 2010 |
|
|
|
Current U.S.
Class: |
382/128 |
Current CPC
Class: |
G06K 9/6218 20130101;
G06K 9/6256 20130101; G06K 2009/4666 20130101; G06K 9/4661
20130101; G06K 9/6267 20130101; G06T 3/0056 20130101; G06T 7/0012
20130101; G06F 16/5866 20190101; G06K 9/4676 20130101; A61B 6/503
20130101; A61B 6/5217 20130101; G06F 16/51 20190101; G06T
2207/30061 20130101; G06T 2207/30048 20130101; G06K 9/46
20130101 |
International
Class: |
G06K 9/62 20060101
G06K009/62; G06T 3/00 20060101 G06T003/00; G06K 9/46 20060101
G06K009/46; A61B 6/00 20060101 A61B006/00; G06T 7/00 20060101
G06T007/00 |
Claims
1. A computerized method of classifying a medical image using a
category model, comprising: using at least one processing unit for
executing code instructions for: receiving an examined medical
image; identifying image coordinates of a plurality of image
patches in said examined medical image, each image patch is
represented by a plurality of repeatable multidimensional features
in a pixel area of said examined medical image; providing a
category model which maps a plurality of visual-words in a space,
each said visual-word is represented by a plurality of reference
repeatable multidimensional features in a reference pixel area and
image coordinates indicative of a location of said reference pixel
area in said space and is associated with at least one of a
plurality of image categories; using said plurality of image
patches and said image coordinates of a plurality of image patches
to identify a group of said plurality of visual-words in said
examined medical image; and categorizing a pathology in said
examined medical image according to said group.
2. The computerized method of claim 1, presenting said pathology in
a client terminal used to provide said examined medical image.
3. The computerized method of claim 1, wherein said group is
identified without segmenting said examined medical image.
4. The computerized method of claim 1, wherein said group is
identified is performed without registering said examined medical
image.
5. The computerized method of claim 1, further comprising updating
said category model according to said pathology.
6. The computerized method of claim 1, wherein said category model
comprises less than 700 visual words.
7. The computerized method of claim 1, wherein said category model
is generated by an analysis of a training set having more than
10,000 medical images.
8. The computerized method of claim 1, wherein said category model
is generated by clustering a plurality of image patches from a
plurality of medical images in a plurality of clusters, said
plurality of visual words being defined according to said plurality
of clusters.
9. The computerized method of claim 8, wherein said clustering is
performed according to a principal component analysis (PCA).
10. The computerized method of claim 8, wherein said plurality of
medical images are provided from a picture archiving communication
system (PACS).
11. The computerized method of claim 1, wherein said category model
is modeled using a support vector machine (SVM) training
procedure.
12. The computerized method of claim 11, wherein said SVM training
procedure is a multi-class SVM with a radial basis function (RBF)
kernel.
13. The computerized method of claim 1, wherein the category model
is updated upon each usage of the category model.
14. The computerized method of claim 1, further comprising
normalizing each image patch, wherein each normalized image patch
is formed from a transformation of intensity values from a
corresponding image patch, to render the image patch less variant
to brightness, and to provide local contrast enhancement.
15. The computerized method of claim 14, wherein said intensity
values from the image patch are obtained from pixels of the image
patch.
16. The computerized method of claim 1, wherein said repeatable
multidimensional features in each said image are from three
dimensional images.
17. The computerized method of claim 1, further comprising
outputting said category model for facilitating the categorization
of an image based on local descriptors thereof including said image
from three dimensional images.
18. The computerized method of claim 1, wherein the pathology is
selected from the group consisting of enlarged heart, lung
infiltrates, right pleural effusion and left pleural effusion.
19. A system of classifying a medical image using a category model,
comprising: an interface adapted for receiving an examined medical
image; a memory adapted to store a category model which maps a
plurality of visual-words in a space, each said visual-word is
represented by a plurality of reference repeatable multidimensional
features in a reference pixel area and image coordinates indicative
of a location of said reference pixel area in said space and is
associated with at least one of a plurality of image categories; a
code store adapted for a code; at least one processing unit for
executing said code; wherein said code comprising: code
instructions for identifying image coordinates of a plurality of
image patches in said examined medical image, each image patch is
represented by a plurality of repeatable multidimensional features
in a pixel area of said examined medical image; code instructions
for using said plurality of image patches and said image
coordinates of a plurality of image patches to identify a group of
said plurality of visual-words in said examined medical image; and
code instructions for categorizing a pathology in said examined
medical image according to said group.
Description
RELATED APPLICATIONS
[0001] This application is a continuation of U.S. patent
application Ser. No. 13/170,200 filed Jun. 28, 2011, which claims
the benefit of priority under 35 USC 119(e) of U.S. Provisional
Patent Application No. 61/358,979 filed Jun. 28, 2010. The contents
of the above applications are all incorporated by reference as if
fully set forth herein in their entirety.
FIELD AND BACKGROUND OF THE INVENTION
[0002] The present invention relates to analysis of medical images
and, more particularly, but not exclusively to automatic analysis
and classification of medical images depicting an organ or a human
body system.
[0003] Systems and devices for visualizing the inside of living
organisms are among the most important medical developments in the
last thirty years. Systems like X-ray scanners, computerized
tomography (CT) scanners and magnetic resonance imaging (MRI)
scanners allow physicians to examine internal organs or areas of
the body that require a thorough examination. In use, the
visualizing scanner outputs a medical image, such as a
cross-sectional image, or a sequence of computerized
cross-sectional images of a certain body organ, which is then
diagnosed by radiologists and/or other physicians.
[0004] In most hospitals and radiology centers, the medical images
are transferred to a picture archiving communication system (PACS)
before being accessed by the radiologists. The PACS is installed on
one or more of computers, which are dedicated for storing,
retrieving, distributing and presenting the stored 3D medical
images. The 3D medical images are stored in an independent format.
The most common format for image storage is digital imaging and
communications in medicine (DICOM).
[0005] The rapid growth of computerized medical imagery using PACS
in hospitals throughout the world led to the development of systems
for classifying visual medical data. For example, International
Patent Application Publication No. WO/2007/099525, filed in Feb.
18, 2007 describes a system for analyzing a source medical image of
a body organ. The system comprises an input unit for obtaining the
source medical image having three dimensions or more, a feature
extraction unit that is designed for obtaining a number of features
of the body organ from the source medical image, and a
classification unit that is designed for estimating a priority
level according to the features.
[0006] Another example is described in U.S. Pat. No. 6,754,675
filed on Jul. 16, 2001 which describes image retrieval system
contains a database with a large number of images. The system
retrieves images from the database that are similar to a query
image entered by the user. The images in the database are grouped
in clusters according to a similarity criterion so that mutually
similar images reside in the same cluster. Each cluster has a
cluster center which is representative for the images in it. A
first step of the search to similar images selects the clusters
that may contain images similar with the query image, by comparing
the query image with the cluster centers of all clusters. A second
step of the search compares the images in the selected clusters
with the query image in order to determine their similarity with
the query image.
SUMMARY OF THE INVENTION
[0007] According to some embodiments of the present invention there
is provided a method of generating a category model for classifying
medical images. The method comprises providing a plurality of
medical images each categorized as one of a plurality of
categorized groups, generating an index of a plurality of visual
words according to a distribution of a plurality of local
descriptors in each the image, modeling a category model mapping a
relation between each the visual word and at least one of the
plurality of categorized groups according to the index, and
outputting the category model for facilitating the categorization
of an image based on local descriptors thereof.
[0008] Optionally, the method further comprises dividing the
plurality of medical images among the plurality of categorized
groups.
[0009] Optionally, the index comprises less than 700 visual
words.
[0010] Optionally, the plurality of medical images are part of a
training set having more than 10,000 medical images.
[0011] Optionally, the generating comprises clustering the
plurality of local descriptors in a plurality of clusters, the
plurality of visual words being defined according to the plurality
of clusters.
[0012] More optionally, the clustering is performed according to a
principal component analysis (PCA).
[0013] Optionally, the modeling is performed using a support vector
machine (SVM) training procedure.
[0014] Optionally, the SVM training procedure is a multi-class SVM
with a radial basis function (RBF) kernel.
[0015] Optionally, the plurality of medical images are provided
from a picture archiving communication system (PACS).
[0016] Optionally, the plurality of categorized groups define a
plurality of pathologies.
[0017] Optionally, the method further comprises automatically
categorizing the plurality of medical images.
[0018] According to some embodiments of the present invention there
is provided a method of classifying a medical image using a
category model. The method comprises providing a category model
which maps a plurality of visual-words in a space, each the
visual-word being associated with at least one of a plurality of
image categories, receiving an examined medical image, identifying
a group of the plurality of visual-words in the examined medical
image, using the category model to match the group with an image
category of the plurality of image categories, and outputting the
image category.
[0019] Optionally, the outputting comprises presenting the image
category in a client terminal used to provide the examined medical
image.
[0020] Optionally, identifying is performed without segmenting the
examined medical image.
[0021] Optionally, identifying is performed without registering the
examined medical image.
[0022] More optionally, the method further comprises updating the
category model according to the matching.
[0023] According to some embodiments of the present invention there
is provided a medical image analysis system of classifying a
medical image using a category model. The system comprises a
repository which stores a category model mapping a plurality of
visual-words in a space, each the visual-word being associated with
at least one of a plurality of image categories, an input unit
which receives an examined medical image, a categorization module
which identifies a group of the plurality of visual-words in the
examined medical image and uses the category model to match the
group with an image category of the plurality of image categories,
and a presentation unit which present the image category in
response to the receiving of the examined medical image.
[0024] Unless otherwise defined, all technical and/or scientific
terms used herein have the same meaning as commonly understood by
one of ordinary skill in the art to which the invention pertains.
Although methods and materials similar or equivalent to those
described herein can be used in the practice or testing of
embodiments of the invention, exemplary methods and/or materials
are described below. In case of conflict, the patent specification,
including definitions, will control. In addition, the materials,
methods, and examples are illustrative only and are not intended to
be necessarily limiting.
[0025] Implementation of the method and/or system of embodiments of
the invention can involve performing or completing selected tasks
manually, automatically, or a combination thereof. Moreover,
according to actual instrumentation and equipment of embodiments of
the method and/or system of the invention, several selected tasks
could be implemented by hardware, by software or by firmware or by
a combination thereof using an operating system.
[0026] For example, hardware for performing selected tasks
according to embodiments of the invention could be implemented as a
chip or a circuit. As software, selected tasks according to
embodiments of the invention could be implemented as a plurality of
software instructions being executed by a computer using any
suitable operating system. In an exemplary embodiment of the
invention, one or more tasks according to exemplary embodiments of
method and/or system as described herein are performed by a data
processor, such as a computing platform for executing a plurality
of instructions. Optionally, the data processor includes a volatile
memory for storing instructions and/or data and/or a non-volatile
storage, for example, a magnetic hard-disk and/or removable media,
for storing instructions and/or data. Optionally, a network
connection is provided as well. A display and/or a user input
device such as a keyboard or mouse are optionally provided as
well.
BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING(S)
[0027] The patent or application file contains at least one drawing
executed in color. Copies of this patent or patent application
publication with color drawing(s) will be provided by the Office
upon request and payment of the necessary fee.
[0028] Some embodiments of the invention are herein described, by
way of example only, with reference to the accompanying drawings.
With specific reference now to the drawings in detail, it is
stressed that the particulars shown are by way of example and for
purposes of illustrative discussion of embodiments of the
invention. In this regard, the description taken with the drawings
makes apparent to those skilled in the art how embodiments of the
invention may be practiced.
[0029] In the drawings:
[0030] FIG. 1 is a flowchart of a method of generating a category
model for classifying medical images, according to some embodiments
of the present invention;
[0031] FIG. 2 is a method of classifying a medical image using a
category model, for example as generated according to FIG. 1,
according to some embodiments of the present invention;
[0032] FIG. 3 is a schematic illustration of a medical image
analysis system of classifying a medical image using a category
model, for example as generated according to FIG. 1, according to
some embodiments of the present invention;
[0033] FIG. 4A is a distribution images across categories;
[0034] FIG. 4B depicts a graph which illustrates the effect of
dictionary size on the accuracy of categorization using a category
model generated as depicted in FIG. 2, according to some
embodiments of the present invention;
[0035] FIG. 4C depicts a graph which illustrates the effect of
dictionary size on the accuracy of categorization when the image
patches have between 5 and 8 feature components, according to some
embodiments of the present invention;
[0036] FIG. 5 is a graph mapping the relation between the weight of
spatial features in x-axis and the classification accuracy in
y-axis where the bars show mean and standard deviation of 20
experiments;
[0037] FIG. 6 is a set of images where the first two images are the
query images and the following images (left to right, top to
bottom) are the retrieval results; and
[0038] FIG. 7 is a graph depicting the relation between the
precision shown for first 5, 10, 15, 20 and 30 returned images and
the number of images.
DESCRIPTION OF EMBODIMENTS OF THE INVENTION
[0039] The present invention relates to analysis of medical images
and, more particularly, but not exclusively to automatic analysis
and classification of medical images depicting an organ or a human
body system.
[0040] According to some embodiments of the present invention there
are provided systems and methods of modeling a category model which
is used for classifying medical images. The method is based on an
analysis of a plurality of medical images, such as X-ray scans and
volumetric scan images. Each medical image is categorized, manually
and/or automatically, as one of a plurality of categorized groups,
for example according to visual characteristic of one or more
pathologies. This allows generating an index, a dictionary, of
visual words, which are patterns of salient local image patches.
The dictionary is generated according to a distribution of a
plurality of local descriptors in each image. Now, a category model
mapping a relation between each visual word and one or more of the
plurality of categorized groups is modeled according to the index.
In such a manner, the category model may be provided, for example
sent, for facilitating the categorization of an image based on
local descriptors thereof.
[0041] According to some embodiments of the present invention there
are provided systems and methods of classifying a medical image
using a category model, such as the category model which is
outlined above and described below. This method is based on a
category model which maps a plurality of visual-words in a space
where each visual-word is associated with one or more image
categories. The category model may be locally stored in a computing
unit that implements the method or in a remote and/or external
database. Now, an examined medical image is received and a group of
visual-words which are documented in the category model are
extracted from the examined medical image, optionally using an
index of visual words, such as the aforementioned dictionary. This
allows using the category model to match the group with an image
category of the plurality of image categories and outputting the
image category.
[0042] Before explaining at least one embodiment of the invention
in detail, it is to be understood that the invention is not
necessarily limited in its application to the details of
construction and the arrangement of the components and/or methods
set forth in the following description and/or illustrated in the
drawings and/or the Examples. The invention is capable of other
embodiments or of being practiced or carried out in various
ways.
[0043] Reference is now made to FIG. 1, which is a flowchart of a
method of generating a category model for classifying medical
images, according to some embodiments of the present invention.
[0044] First, as shown at 101 a training set having a plurality of
medical images is received. As used herein, a medical image means
an X-Ray scan image, a computerized tomography (CT) scan image, a
magnetic resonance imager (MRI) scan image, and a positron emission
tomography (PET)-CT scan image. For example, the images are taken
from a medical database, such as PACS or radiology information
system (RIS). Optionally, the number of medical images in the
training set is between few hundreds to few hundreds of thousands
or even more. For example, the training set includes about 1200
medical images or about 65,000 medical images as exemplified below.
Optionally, the number images changes according to the number of
possible pathologies which are categorized in the category model.
Optionally a ratio of about 2000 images per category is
maintained.
[0045] Now, as shown at 102, local descriptors, which may be
referred to herein as image patches, are identified in each one of
the provided medical images. The local descriptors are repeatable
multidimensional features so that if there is a transformation
between two instances of an object, corresponding points are
detected and substantially identical descriptor values are obtained
around each. Optionally each image patch is represented by a
multidimensional record.
[0046] Optionally, the descriptors are resistant to geometric and
illumination variations, for example as described in any of the
following T. Lindenberg, Scale-space theory in computer vision,
Kluwer Academic Publishers, 1994, D. G. Lowe, Object Recognition
from local scale-invariant features, ICCV (International Conference
on Computer Vision), 1999; J. Matas, J. Burianek, and J. Kittler.
Object recognition using the invariant pixel-set signature, BMVC
(British Machine Vision Conference), 2000; and F. Schaffalitzky and
A. Zisserman. Viewpoint invariant texture matching and wide
baseline stereo, ICCV, 2001, which are incorporated herein by
reference.
[0047] Optionally, the image patches are acquired using one or more
patch sampling strategies such as random sampling and/or grid
sampling, optionally with spacings. Optionally, the size of a patch
is of 9.times.9 pixels. Optionally, image patches along the border
of the image are considered ignored. Optionally, the intensity
values within an image patch are normalized to have zero mean and
unit variance. This provides local contrast enhancement and
augments the information within the image patches. Optionally,
image patches that have a single intensity value of black are
ignored.
[0048] According to some embodiments of the present invention, the
data dimensionality and optionally the computational complexity of
reducing the level of noise, may be diminished using a procedure
such as a principal component analysis (PCA), principal component
regression (PCR) and/or partial least squares (PLS) regression. For
example data dimensionality of each 9.times.9 image patch is
reduced in size from 81 to 7.
[0049] For example, when PCA is used, a resultant PCA component
does not contain information regarding the average intensity of the
respective image patch. This average value contains information
that discriminates between the dark background and the bright
tissue and may be used to distinguish between tissue types. In such
embodiments, the mean gray level of the image patch may be taken as
an additional multidimensional features feature.
[0050] Optionally, the center of each image patch, coordinates (x,
y) is added to a respective image patch multidimensional record as
two additional features, for example as an overall ten-dimensional
image patch representation. The addition of the spatial coordinates
to the image patch multidimensional record introduces spatial
information into the image representation. Optionally, the relative
feature weights in the proposed system are tuned experimentally on
a test/cross-validation set, for example as described in the
example below.
[0051] Optionally a dataset which documents the image patches is
generated for each image in the training set. The dataset is
optionally a multidimensional record.
[0052] Now, as shown at 103, a dictionary is generated according to
the image patches. First, some or all of the images are selected.
Now, the image patches of the selected images are clustered in a
plurality of clusters distributed in a feature space, which may be
referred to herein as an image patch space. Each cluster is defined
in a different subspace which may be referred to herein as visual
word, for example using iterative square error partitioning and/or
hierarchical technique. The visual words form an index or a
codebook, referred to herein as a dictionary, of the image patches
in a feature space. Optionally, the number of visual words is
limited to a predefined amount. Optionally, the predefined amount
is 700 or less, for example as shown in FIGS. 4B and 4C and
described below. Optionally, each visual word includes 7 PCA
coefficients, for example as described above.
[0053] Optionally, a k-means algorithm is used to cluster the image
patches. This algorithm proceeds by iterated assignments of image
patches to their closest cluster centers (visual word) and
re-computation of the cluster centers (other visual words), see O.
Duda, P. E. Hart, D. G. Stork, Pattern classification, John Wiley
& Sons, 2000, which is incorporated herein by reference. Note
that this dictionary development step is done in an unsupervised
mode without any reference to the image categories, such as
pathologies.
[0054] As shown at 104, each image is represented as a bag of
visual words, namely a dataset of visual words which appears in the
image, such as a visual word vector. The visual words are selected
according to the image patches which have been identified in each
image. The bag of visual words, which may be referred to herein as
a visual-word vector, contains the presence and/or absence
information of each visual word from the dictionary in the image,
the count of each visual word (i.e., the number of image patches in
the corresponding visual word cluster), or the count weighted by
other factors. Optionally, the visual-word vector is represented as
a histogram wherein each bin in the histogram is a visual word
index number selected out of the dictionary and generated
automatically from the data.
[0055] As shown at 105, the plurality of medical images, are
categorized according to one or more pathologies which have been
identified as depicted therein. The categorization is optionally
performed manually, for example by a diagnosis of one or more, such
as physicians, for example orthopedic physician and radiologists.
Alternatively, the categorization may be performed automatically,
for example using known image classification methods, and/or by an
analysis of a diagnosis and/or a textual description that is
attached to the image. Alternatively, the categorization may be
semi automatic, for example by a combination of an automatic
textual and/or image classification methods and a manual
verification of one or more practitioners. Each visual-word vectors
is categorized according to the image which is related thereto.
[0056] Now, as shown at 106, the categorized visual-word vectors of
the categorized images are combined to create a category model.
[0057] Optionally, giving the categorized visual-word vectors,
which may be divided to categories, a support vector machine (SVM)
training algorithm builds a category model that allows estimating
to which one of the categories, if any, a certain medical image
which is not from the training set is related. Optionally, the
category model is an SVM model in which the visual-word vectors are
represented as points in space, mapped so that the categorized
visual-word vectors of the separate categories are divided by a
clear gap that is as wide as possible. Optionally, the SVM training
algorithm is a multi-class SVM that is optionally implemented as a
series of one-vs-one binary SVMs with a radial basis function (RBF)
kernel, for example based on the LIBSVM library, found in
http://www.csie.ntu.edu.tw/.about.cjlin/libsvm/, which is
incorporated herein by reference. Optionally, SIFT image features
are extracted from each image and used to reduce the visual word
extraction time.
[0058] Now, the category model is outputted, as shown at 107,
facilitating the categorization of new medical image which is
mapped into the space of the category model and predicted to belong
to a category based on which side of the gap they fall on.
[0059] Reference is now made to FIG. 2, which is a method 200 of
classifying a medical image using a category model, for example as
generated according to FIG. 1, according to some embodiments of the
present invention.
[0060] First, as shown at 201, a category model which maps a
plurality of categorized visual-words and/or visual-word vectors in
space is received. The category model is optionally generated based
on a training set of a plurality of exemplary medical images, for
example as depicted in FIG. 1.
[0061] As shown at 202, an examined medical image is received.
Optionally, the examined medical image is uploaded from a PACS
and/or a non transitory storage medium, such as a CD, a DVD, and/or
a memory card, to a client terminal which implements the method 200
and/or a client terminal connected to a computing unit which
implements the method 200. The client terminal may be a laptop, a
Smartphone, a cellular phone, a tablet, a personal computer a
personal digital assistance (PDA) and the like.
[0062] Now, as shown at 203, a visual word vector and/or a
histogram are generated according to an analysis of the image. The
visual word vector represents image patches of the image which
correspond with visual words at the space of the category model.
The conversion is optionally similar to the described in relation
to blocks 102, 103, and 105 where image patches are identified and
matched with visual words in the dictionary to generate the
respective bag of visual words.
[0063] Now, as shown at 204, the visual words of the examined image
are matched with the category model. The match maps the visual
words of the vector in the space of the category model. The mapping
is to a subspace, or to the proximity of a subspace, which is
associated with a certain category mapped in the category model.
This allow, as shown at 205, the categorization of the examined
image. As shown at 206, the categorization is outputted, for
example presented to the operator of the client terminal, forwarded
to a database which hosts the examined image for an association
therewith, and/or sent, for example via an email service, to a
practitioner which is related to the examined image and/or to the
imaged patient.
[0064] Optionally, each shown at 207, each examined image and/or
the related visual word vector and the categorization thereof is
used to update the category model. In such a manner, the category
model is improved each time it is being used for categorizing a
medical image. The update may be performed by rerunning the
dictionary generation process and respectively the category model
generation process depicted in blocks 103, 104, and 106 of FIG.
1.
[0065] It should be noted that the method depicted in FIG. 2 allows
categorizing medical images such as 2 dimensional (2D) X-Ray images
and 3D CT or MRI images without segmentation and/or registration.
In such a manner, the computational complexity involved in
categorizing each examined image is minimal. Such a method maybe
implemented on thin end client with limited computational
power.
[0066] Reference is now made to FIG. 3, which is a schematic
illustration of a medical image analysis system of classifying a
medical image using a category model, for example as generated
according to FIG. 1, according to some embodiments of the present
invention. The medical image analysis system 301 comprises an input
module 302 for obtaining or receiving a medical image, a repository
303 for storing the category model and a categorization module 304
for categorizing the received medical image according to the
category model. The input module 302 is designed to receive the
medical image either directly from a medical imaging system or
indirectly via a content source such as a PACS server, a PACS
workstation, a computer network, or a portable memory device such
as a DVD, a CD, a memory card, etc. Each received medical image is
preferably associated with medical information. Such medical
information may comprise the patient age, gender, medical
condition, ID, and the like. Optionally, the medical image found in
a digital imaging and communications in medicine (DICOM)
object.
[0067] Optionally, the input module 302 is to forward the received
medical image to the categorization module 304. The categorization
module 304 optionally implements the method depicted in FIG. 2 so
as to categorize the received image. The system 301 further
includes a presentation unit 305, such as a display for presenting
the categorization performed by the categorization module 304. The
categorization may be displayed in a window or any other graphical
user interface (GUI). When such an embodiment is used, the medical
image analysis system 301 can alert the user on real time whenever
a critical pathological categorization has been identified in one
of the received medical images. Such an embodiment increases the
effectiveness of a therapy given to patients as it alarms the
system user regarding a pathological indication immediately after
the medical image has been acquired. Optionally, the medical image
analysis system 301 includes a model generation model which is set
to generate and optionally to update the category model, for
example as described above in relation to FIG. 1 and block 207 of
FIG. 2.
[0068] It is expected that during the life of a patent maturing
from this application many relevant systems and methods will be
developed and the scope of the term client terminal, computing
unit, and image processing is intended to include all such new
technologies a priori.
[0069] As used herein the term "about" refers to .+-.10%.
[0070] The terms "comprises", "comprising", "includes",
"including", "having" and their conjugates mean "including but not
limited to". This term encompasses the terms "consisting of" and
"consisting essentially of".
[0071] The phrase "consisting essentially of" means that the
composition or method may include additional ingredients and/or
steps, but only if the additional ingredients and/or steps do not
materially alter the basic and novel characteristics of the claimed
composition or method.
[0072] As used herein, the singular form "a", "an" and "the"
include plural references unless the context clearly dictates
otherwise. For example, the term "a compound" or "at least one
compound" may include a plurality of compounds, including mixtures
thereof.
[0073] The word "exemplary" is used herein to mean "serving as an
example, instance or illustration". Any embodiment described as
"exemplary" is not necessarily to be construed as preferred or
advantageous over other embodiments and/or to exclude the
incorporation of features from other embodiments.
[0074] The word "optionally" is used herein to mean "is provided in
some embodiments and not provided in other embodiments". Any
particular embodiment of the invention may include a plurality of
"optional" features unless such features conflict.
[0075] Throughout this application, various embodiments of this
invention may be presented in a range format. It should be
understood that the description in range format is merely for
convenience and brevity and should not be construed as an
inflexible limitation on the scope of the invention. Accordingly,
the description of a range should be considered to have
specifically disclosed all the possible subranges as well as
individual numerical values within that range. For example,
description of a range such as from 1 to 6 should be considered to
have specifically disclosed subranges such as from 1 to 3, from 1
to 4, from 1 to 5, from 2 to 4, from 2 to 6, from 3 to 6 etc., as
well as individual numbers within that range, for example, 1, 2, 3,
4, 5, and 6. This applies regardless of the breadth of the
range.
[0076] Whenever a numerical range is indicated herein, it is meant
to include any cited numeral (fractional or integral) within the
indicated range. The phrases "ranging/ranges between" a first
indicate number and a second indicate number and "ranging/ranges
from" a first indicate number "to" a second indicate number are
used herein interchangeably and are meant to include the first and
second indicated numbers and all the fractional and integral
numerals therebetween.
[0077] It is appreciated that certain features of the invention,
which are, for clarity, described in the context of separate
embodiments, may also be provided in combination in a single
embodiment. Conversely, various features of the invention, which
are, for brevity, described in the context of a single embodiment,
may also be provided separately or in any suitable subcombination
or as suitable in any other described embodiment of the invention.
Certain features described in the context of various embodiments
are not to be considered essential features of those embodiments,
unless the embodiment is inoperative without those elements.
[0078] Various embodiments and aspects of the present invention as
delineated hereinabove and as claimed in the claims section below
find experimental support in the following examples.
[0079] Reference is now made to the following example, which
together with the above descriptions, illustrates some embodiments
of the invention in a non limiting fashion.
[0080] In the example system and method validation was conducted
using a database of 12,000 categorized medical images, radiographs.
This dataset is the basis for the ImageClef 2007 medical image
classification competition; see T. Deselaers et al. Overview of the
imageclef 2007 object retrieval task, in workshop of the cross
language evaluation forum 2007, volume 5152, 2008, which is
incorporated herein by reference. A set of 11,000 medical images
have been used for training, and 1000 serve for testing. There are
116 different categories within the archive, differing in either
the examined region, the image orientation with respect to the body
or the biological system under evaluation. Several of these images
are presented in FIG. 4A. The distribution of the images across the
categories is non-uniform; the most frequent category contains over
19% of the images in the database, while many categories are
represented by less than 0.1% of the images. The system parameters
have been optimized using the training portion of this set, by
running 20 cross-validation experiments trained on 10,000 images
and verified on 1000 randomly drawn test images. Each parameter was
optimized independently. As FIG. 4B shows, increasing the number of
dictionary words proved useful up to 700 words. Beyond this value
the running time increased significantly, with no evident
improvement in the classification rate. FIG. 4B also demonstrates
that using an SVM classifying algorithm provides results that are
more than 3% higher than the best K-NN classifier (k=3). The effect
of the number of PCA components was examined next. FIG. 4C shows
similar classification results in the range of 5 to 8 components,
with an average classification rate of approximately 90% using the
SVM classifying algorithm. Based on the above experiments, a
dictionary size of 700 visual words was selected, where each word
contains 7 PCA coefficients.
[0081] Incorporating spatial coordinates of the patch as additional
features improves the classification performance noticeably, as
seen in FIG. 5, which is a graph mapping the relation between the
weight of spatial features in x-axis and the classification
accuracy in y-axis where the bars show mean and standard deviation
of 20 experiments.
[0082] The optimal range for the (x, y) coordinates is [-3; 3]. The
patch variance normalization step improves the classification rate
as well: with no normalization, the average classification rate is
88:19, while with normalization it climbs to 90:9. Using SIFT
features with the SVM classification increased significantly the
feature extraction time, and achieved an average of 85.4%
classification accuracy, well below the classification rate of a
raw patch based classification.
[0083] Using the parameter set defined above, classification of
previously unseen 1000 test images was conducted. The overall
classification rate achieved is 89:1%. The total running time for
the whole system, training and classification, was approximately 40
minutes on the full resolution images, and 3 minutes on the 1/4
scaled down images, as measured on dual quad-core Intel Xeon 2.33
GHz.
[0084] Reference is now also made to another example in which a
system and a method validation were conducted using a database of
66,000 categorized medical images, radiographs. This dataset is
optionally, the ImageClef 2008 database; see
http://www.imageclef.org/ImageCLEF2008, which is incorporated
herein by reference. In ImageClef 2008 a large-scale medical image
retrieval competition was conducted. A database of over 66,000
images was used with 30 query topics. Each topic is composed of one
or more example images and a short textual description in several
languages. The objective is to return a ranked set of 1000 images
from the complete database, sorted by their relevance to the
presented queries. Sample queries from this challenge and the first
few returned images are seen in FIG. 6 which depicts a set of
images where the first two images are the query images and the
following images (left to right, top to bottom) are the retrieval
results. The retrieved results were manually judged for relevance
by medical experts. FIG. 7 is a graph depicting the relation
between the precision shown for first 5, 10, 15, 20 and 30 returned
images and the number of images. The precision achieved using the
method described above is marked with (*). The other outcomes are
achieved using visual retrieval algorithms described in the Muller
et al. Overview of the imageclefmed 2008 medical image retrieval
task. In CLEF working notes
(http://www.clef-campaign.org/2008/working_notes/CLEF2008WN-Contents.html-
.), which is incorporated herein by reference.
[0085] In this Figure, the line labeled `Proposed System` depicts
the outcomes achieved when using image patch normalization and the
line labeled `Not Normalized` depicts the outcomes achieved when
using the patch original gray levels. The normalized patch approach
in the proposed system is shown to rank first among the automatic
purely visual retrieval systems.
[0086] The retrieval system is computationally efficient, with an
average retrieval time of less than 400 ms per query.
[0087] Categorization on the Pathology Level
[0088] Image similarity-based categorization and retrieval becomes
of clinical value once the task involves a diagnostic-level
categorization, such as healthy vs. pathology. Optionally, the
category models generated as described in the examples above were
examined on chest x-rays obtained for various clinical indications
in the emergency room of Sheba medical center. 102 frontal chest
images have been used; from which 26 diagnosed as normal and 76 as
having have one or more pathologies, such as lung infiltrates, left
or right pleural effusion or an enlarged heart shadow. X-ray
interpretations, made by two radiologists, served as the referral
gold standard. Inconclusive results were not included in this set.
Four sample images from this data are presented in FIG. 7. A
patch-based classifying was implemented using an SVM classifying
algorithm with two classes, the classification was conducted for
each pathology type, and for healthy vs. any pathology. In order to
preserve the generalization ability of the classifiers, system
parameters were tuned using the general ImageClef 2007 database and
were not specifically tuned to the lung pathology task. A leave one
out classification was performed (results averaged over 102
trials). Table 1 summarizes the classification results:
TABLE-US-00001 Normal Abnormal images images Sensitivity
Specificity Any Pathology 22/26 74/76 94.8 91.7 Enlarged heart
20/23 43/44 95.3 93.5 Lung Infiltrates 23/33 27/34 76.7 73.0 Right
pleural effusion 12/23 42/51 57.1 79.2 Left pleural effustion 15/27
38/47 62.5 76.0
[0089] The software identified correctly 74 out of 76 abnormal and
22 out of 26 normal x-rays with 4 false positives and 2 false
negatives cases, resulting in a sensitivity of 94.87% and
specificity of 91.67%. In the task of between-pathology
discrimination, the performance depends on the pathology type: it
is highly accurate in detecting enlarged hearts, with a sensitivity
of 95.24% and specificity of 93.48%. It is less accurate in
detecting lung infiltrates and effusions. Briefly stated, in this
study a patch-based classification system was applied to a variety
of medical image archives, in categorization and retrieval tasks.
The exemplary system was tuned to achieve high accuracy, with an
average of over 90% correct classification on a publicly available
database of 12,000 medical radiographs. In the ImageClef 2008
medical annotation challenge it ranked second. It is a highly
efficient, with less than 200 milliseconds training and
classification time per image. Using the same methods, an image
retrieval utility, which was ranked first in ImageClef 2008 among
the visual retrieval systems was developed. Extending the system to
pathology-level discrimination showed initial results for lung
disease categorization.
[0090] Although the invention has been described in conjunction
with specific embodiments thereof, it is evident that many
alternatives, modifications and variations will be apparent to
those skilled in the art. Accordingly, it is intended to embrace
all such alternatives, modifications and variations that fall
within the spirit and broad scope of the appended claims.
[0091] All publications, patents and patent applications mentioned
in this specification are herein incorporated in their entirety by
reference into the specification, to the same extent as if each
individual publication, patent or patent application was
specifically and individually indicated to be incorporated herein
by reference. In addition, citation or identification of any
reference in this application shall not be construed as an
admission that such reference is available as prior art to the
present invention. To the extent that section headings are used,
they should not be construed as necessarily limiting.
* * * * *
References