U.S. patent application number 16/990752 was filed with the patent office on 2020-11-26 for image processing device and method, recording medium and program.
The applicant listed for this patent is SONY CORPORATION. Invention is credited to KAZUKI AISAKA, SEIJI KOBAYASHI, TOMOYUKI OOTSUKI.
Application Number | 20200372286 16/990752 |
Document ID | / |
Family ID | 1000005008447 |
Filed Date | 2020-11-26 |
![](/patent/app/20200372286/US20200372286A1-20201126-D00000.png)
![](/patent/app/20200372286/US20200372286A1-20201126-D00001.png)
![](/patent/app/20200372286/US20200372286A1-20201126-D00002.png)
![](/patent/app/20200372286/US20200372286A1-20201126-D00003.png)
![](/patent/app/20200372286/US20200372286A1-20201126-D00004.png)
![](/patent/app/20200372286/US20200372286A1-20201126-D00005.png)
![](/patent/app/20200372286/US20200372286A1-20201126-D00006.png)
![](/patent/app/20200372286/US20200372286A1-20201126-D00007.png)
![](/patent/app/20200372286/US20200372286A1-20201126-D00008.png)
![](/patent/app/20200372286/US20200372286A1-20201126-D00009.png)
![](/patent/app/20200372286/US20200372286A1-20201126-D00010.png)
View All Diagrams
United States Patent
Application |
20200372286 |
Kind Code |
A1 |
AISAKA; KAZUKI ; et
al. |
November 26, 2020 |
IMAGE PROCESSING DEVICE AND METHOD, RECORDING MEDIUM AND
PROGRAM
Abstract
There is provided an image processing device including a motion
vector detection portion that performs comparison of a
substantially spherical photographic subject such that, among a
plurality of captured images including the photographic subject, an
image as a processing target and another image as a comparison
target are compared using each of the plurality of captured images
as the processing target, and which detects a motion vector of a
whole three-dimensional spherical model with respect to the
processing target, a motion compensation portion that performs
motion compensation on the processing target, based on the motion
vector of each of the plurality of captured images that is detected
by the motion vector detection portion, and a synthesis portion
that synthesizes each of the captured images that are obtained as a
result of the motion compensation performed by the motion
compensation portion.
Inventors: |
AISAKA; KAZUKI; (KANAGAWA,
JP) ; OOTSUKI; TOMOYUKI; (KANAGAWA, JP) ;
KOBAYASHI; SEIJI; (TOKYO, JP) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
SONY CORPORATION |
Tokyo |
|
JP |
|
|
Family ID: |
1000005008447 |
Appl. No.: |
16/990752 |
Filed: |
August 11, 2020 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
15988561 |
May 24, 2018 |
10762379 |
|
|
16990752 |
|
|
|
|
14977599 |
Dec 21, 2015 |
9990562 |
|
|
15988561 |
|
|
|
|
13594382 |
Aug 24, 2012 |
9224209 |
|
|
14977599 |
|
|
|
|
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
G06T 15/00 20130101;
G06T 2207/30041 20130101; G06T 7/285 20170101; G06T 7/20 20130101;
G06T 7/207 20170101; A61B 3/0025 20130101; A61B 3/12 20130101; G06K
9/48 20130101; G06K 9/00208 20130101; G06T 7/223 20170101; A61B
3/14 20130101 |
International
Class: |
G06K 9/48 20060101
G06K009/48; G06T 7/20 20060101 G06T007/20; G06T 15/00 20060101
G06T015/00; G06K 9/00 20060101 G06K009/00; G06T 7/223 20060101
G06T007/223; G06T 7/207 20060101 G06T007/207; G06T 7/285 20060101
G06T007/285; A61B 3/00 20060101 A61B003/00; A61B 3/12 20060101
A61B003/12; A61B 3/14 20060101 A61B003/14 |
Foreign Application Data
Date |
Code |
Application Number |
Aug 31, 2011 |
JP |
2011-188277 |
Claims
1. A data processing apparatus comprising circuitry configured to
obtain a plurality of eyeball image data captured over a period of
time, detect a plurality of motion vectors of the eyeball image
data, and perform motion compensation based on the motion vectors
and a model with respect to an eyeball.
2-8. (canceled)
Description
BACKGROUND
[0001] The present technology relates to an image processing device
and method, a recording medium and a program, and particularly
relates to an image processing device and method, a recording
medium and a program that are capable of easily improving image
quality of an image obtained by capturing a substantially spherical
body, using a simple configuration.
[0002] A fundus examination is known in which a fundus oculi, such
as a retina in an eyeball, an optic papilla or the like, is
observed through a pupil. The fundus examination is performed using
a special device, such as a funduscope or a fundus camera, for
example. The fundus examination is performed such that, for
example, when an image of the fundus oculi in the eyeball of a test
subject is captured by a fundus camera and a resultant captured
image (hereinafter referred to as a fundus oculi image) is
displayed on a monitor or the like, an observer observes the fundus
oculi image. In order for the observer to perform accurate
observation, the image quality of the fundus oculi image is
improved.
[0003] As a known technique to improve the image quality of the
fundus oculi image, there is a technique in which, for example,
data of a plurality of fundus oculi images that are sequentially
captured are synthesized while taking into account the fact that
the eyeball is a substantial sphere. With this technique, the
plurality of fundus oculi images that are captured over a certain
period of time are synthesized. Therefore, if the fundus oculi
moves in the certain period of time, the image quality improvement
of the fundus oculi image is hindered. To address this, Japanese
Patent Application Publication No. JP-A-2011-087672 discloses a
technique that performs alignment of rotation directions of a
plurality of three-dimensional images of a fundus oculi. The
three-dimensional images are formed using tomographic images of the
fundus oculi that are obtained by optical coherence tomography
(OCT). Further, for example, Japanese Patent Application
Publication No. JP-A-2010-269016 discloses a technique that uses an
affine transformation to perform alignment including rotation of a
plurality of fundus oculi images.
SUMMARY
[0004] However, with the technique described in Japanese Patent
Application Publication No. JP-A-2011-087672, the tomographic
images by OCT am required in order to form the three-dimensional
images, resulting in an increase in the device size. Further, with
the technique described in Japanese Patent Application Publication
No. JP-A-2010-269016, the affine transformation, which is used for
rotation of two-dimensional images, is used for the fundus oculi
images. As a result, it is difficult to accurately perform
alignment of the images of the eyeball that is a three-dimensional
sphere.
[0005] In summary, in recent years, there is a demand to easily
obtain a fundus oculi image with a high image quality using a
simple configuration. However, this demand is not sufficiently
satisfied by known technologies including those described in
Japanese Patent Application Publication No. JP-A-2011-087672 and
Japanese Patent Application Publication No. JP-A-2010-269016. The
above circumstances apply not only to the fundus oculi image, but
also apply to an image obtained by capturing a substantially
spherical body.
[0006] The present technology has been devised in light of the
above circumstances, and makes it possible to easily improve image
quality of an image obtained by capturing a substantially spherical
body, using a simple configuration.
[0007] According to an embodiment of the present technology, there
is provided an image processing device including a motion vector
detection portion that performs comparison of a substantially
spherical photographic subject such that, among a plurality of
captured images including the photographic subject, an image as a
processing target and another image as a comparison target are
compared using each of the plurality of captured images as the
processing target, and which detects a motion vector of a whole
three-dimensional spherical model with respect to the processing
target, a motion compensation portion that performs motion
compensation on the processing target, based on the motion vector
of each of the plurality of captured images that is detected by the
motion vector detection portion, and a synthesis portion that
synthesizes each of the captured images that are obtained as a
result of the motion compensation performed by the motion
compensation portion.
[0008] With respect to each of a plurality of blocks that are
divided up from the processing target, the motion vector detection
portion may detect a local motion vector by performing block
matching with the comparison target. The motion vector detection
portion may detect the motion vector of the whole three-dimensional
spherical model with respect to the processing target, using the
local motion vector of each of the plurality of blocks.
[0009] The motion vector detection portion may convert the local
motion vector with respect to each of the plurality of blocks in
the processing target into a local spherical motion vector in the
three-dimensional spherical model. The motion vector detection
portion may detect the motion vector of the whole three-dimensional
spherical model with respect to the processing target, using the
local spherical motion vector of each of the plurality of
blocks.
[0010] The motion vector detection portion may convert each of the
plurality of blocks in the processing target into a plurality of
spherical blocks in the three-dimensional spherical model. With
respect to each of the plurality of spherical blocks, the motion
vector detection portion may detect, as the local motion vector, a
local spherical motion vector by performing block matching with the
comparison target. The motion vector detection portion may detect
the motion vector of the whole three-dimensional spherical model
with respect to the processing target, using the local spherical
motion vector of each of the plurality of spherical blocks.
[0011] The motion vector detection portion may convert each of the
processing target and the comparison target into a spherical image
in the three-dimensional spherical model. The motion vector
detection portion may perform matching between the spherical image
of the processing target and the spherical image of the comparison
target, and thereby may detect the motion vector of the whole
three-dimensional spherical model with respect to the processing
target.
[0012] The photographic subject may be a findus oculi.
[0013] The three-dimensional spherical model may be switched and
used in accordance with conditions of the photographic subject.
[0014] An image processing method, a recording medium and a program
according to the embodiment of the present technology are the image
processing method, the recording medium and the program
corresponding to the image processing device according to the
embodiment of the present technology described above.
[0015] In the image processing device and method, the recording
medium and the program according to the embodiment of the present
technology, comparison of a substantially spherical photographic
subject is performed such that, among a plurality of captured
images including the photographic subject, an image as a processing
target and another image as a comparison target are compared using
each of the plurality of captured images as the processing target,
a motion vector of a whole three-dimensional spherical model with
respect to the processing target is detected, motion compensation
on the processing target is performed, based on the motion vector
of each of the plurality of captured images that is detected, and
each of the captured images that are obtained as a result of the
motion compensation is synthesized.
[0016] According to another embodiment of the present technology,
there is provided an image processing device including a conversion
portion that, among a plurality of captured images that include a
substantially spherical photographic subject, converts an image as
a processing target and another image as a comparison target into
spherical images on a three-dimensional spherical model, using each
of the plurality of captured images as the processing target, an
extraction portion that extracts features of each of the spherical
image of the processing target and the spherical image of the
comparison target, an alignment portion that aligns positions of
the features such that the features match each other, and a
synthesis portion that synthesizes each of the captured images that
are obtained as a result of the alignment performed by the
alignment portion.
[0017] A blood vessel shape may be used as the feature.
[0018] The photographic subject may be a fundus oculi.
[0019] The three-dimensional spherical model may be switched and
used in accordance with conditions of the photographic subject.
[0020] An image processing method, a recording medium and a program
according to another embodiment of the present technology are the
image processing method, the recording medium and the program
corresponding to the image processing device according to the
embodiment of the present technology described above.
[0021] In the image processing device and method, the recording
medium and the program according to the another embodiment of the
present technology, among a plurality of captured images that
include a substantially spherical photographic subject, an image as
a processing target and another image as a comparison target are
converted into spherical images on a three-dimensional spherical
model, using each of the plurality of captured images as the
processing target, features of each of the spherical image of the
processing target and the spherical image of the comparison target
are extracted, positions of the features are aligned such that the
features match each other, and each of the captured images that are
obtained as a result of the alignment are synthesized.
[0022] According to the present technology described above, it is
possible to easily improve image quality of an image obtained by
capturing a substantially spherical body, using a simple
configuration.
BRIEF DESCRIPTION OF THE DRAWINGS
[0023] FIG. 1 is a block diagram showing a configuration example of
a fundus oculi image processing device to which the present
technology is applied;
[0024] FIG. 2 is a block diagram showing a configuration example of
a motion vector detection portion;
[0025] FIG. 3 is a diagram illustrating specific processing of the
motion vector detection portion;
[0026] FIG. 4 is a flowchart illustrating a flow of fundus oculi
image generation processing;
[0027] FIG. 5 is a flowchart illustrating a flow of motion vector
detection processing;
[0028] FIG. 6 is a block diagram showing a configuration example of
a motion vector detection portion;
[0029] FIG. 7 is a diagram illustrating specific processing of the
motion vector detection portion;
[0030] FIG. 8 is a flowchart illustrating a flow of motion vector
detection processing;
[0031] FIG. 9 is a block diagram showing a configuration example of
a motion vector detection portion;
[0032] FIG. 10 is a diagram illustrating specific processing of the
motion vector detection portion;
[0033] FIG. 11 is a flowchart illustrating a flow of motion vector
detection processing;
[0034] FIG. 12 is a block diagram showing a configuration example
of a fundus oculi image processing device;
[0035] FIG. 13 is a diagram illustrating specific processing of a
feature extraction portion and an alignment portion;
[0036] FIG. 14 is a diagram showing a configuration example of a
blood vessel alignment processing portion;
[0037] FIG. 15 is a diagram illustrating specific processing of the
blood vessel alignment processing portion;
[0038] FIG. 16 is a flowchart illustrating a flow of fundus oculi
image generation processing;
[0039] FIG. 17 is a flowchart illustrating a flow of blood vessel
alignment processing; and
[0040] FIG. 18 is a block diagram showing a hardware configuration
example of an image processing device to which the present
technology is applied.
DETAILED DESCRIPTION OF THE EMBODIMENT(S)
[0041] Hereinafter, four embodiments (hereinafter, respectively
referred to as first to fourth embodiments) of the present
technology will be explained in the following order.
[0042] 1. First embodiment (an example in which a motion vector
detected from each block is applied to a spherical model)
[0043] 2. Second embodiment (an example in which a motion vector of
each block applied to the spherical model is detected)
[0044] 3. Third embodiment (an example in which a motion vector is
detected from a fundus oculi image applied to the spherical
model)
[0045] 4. Fourth embodiment (an example in which fundus oculi
images applied to the spherical model are aligned)
First Embodiment
[0046] Configuration Example of Fundus Oculi Image Processing
Device
[0047] FIG. 1 is a block diagram showing a configuration example of
a fundus oculi image processing device to which the present
technology is applied.
[0048] A fundus oculi image processing device 10 shown in FIG. 1
captures an image of a fundus oculi, such as a retina in an
eyeball, an optic papilla or the like, of a test subject. The
fundus oculi image processing device 10 performs image processing
on data of the obtained fundus oculi image to improve image
quality, and causes the fundus oculi image after the image
processing to be displayed.
[0049] In order to reduce a burden on the test subject, the fundus
oculi image processing device 10 captures an image of a
photographic subject (namely, the fundus oculi of the test subject)
while suppressing an amount of light irradiated onto the
photographic subject. As a result, data of one sheet of a fundus
oculi image can be obtained. However, in order to obtain a higher
quality fundus oculi image, the fundus oculi image processing
device 10 performs image capture of the photographic subject a
plurality of times, and performs various types of image processing,
which will be described later, on data of a plurality of captured
images obtained each time. As a result of performing such various
types of image processing, a fundus oculi image with higher image
quality is displayed by the fundus oculi image processing device
10.
[0050] Note that the single image capture described herein means a
series of operations of the fundus oculi image processing device 10
until light is accumulated on each of pixels of an imaging element
and an electrical signal (data of each pixel) is output from each
of the pixels. In this case, a number of times of image capture and
a time interval between each image capture are not particularly
limited. For example, if the time interval between each image
capture is reduced to be approximately 1/30 seconds and image
capture is performed 300 times consecutively at that time interval,
10 seconds of moving images are obtained. In summary, a plurality
of times of image capture described herein is a concept including
image capture to obtain a plurality of still images and image
capture to obtain moving images.
[0051] The fundus oculi image processing device 10 configured in
this manner includes an imaging portion 21, an image processing
portion 22, a storage portion 23 and an output portion 24.
[0052] The imaging portion 21 captures an image of the fundus oculi
of the test subject located at a predetermined position, as a
photographic subject, and outputs data of the obtained fundus oculi
image. For example, the imaging portion 21 can have a configuration
including a charge coupled device (CCD) imaging element, a
complementary metal oxide semiconductor (CMOS) imaging element and
the like. However, the imaging portion 21 is not limited to this
configuration, and it can have any configuration as long as it can
output data of the fundus oculi image. Note that, in the present
embodiment, the imaging portion 21 has a function of irradiating
light onto the photographic subject that is being captured, in
order to obtain a fundus oculi image with higher image quality.
[0053] As a technique to obtain a fundus oculi image with higher
image quality, a technique is known in which the amount of light
irradiated onto the fundus oculi of the test subject is increased
during image capture. However, if the amount of light irradiated
onto the fundus oculi of the test subject is increased during image
capture, a burden on the test subject is increased. As a result,
there is a possibility that an unnecessary influence will be
exerted on an observation target or a psychological burden on the
test subject will be increased. Further, in this case, since an
increased amount of light is irradiated onto the fundus oculi of
the test subject during image capture, the test subject feels that
it is too bright and closes his/her eye or moves and there is a
possibility that the fundus oculi image with high image quality
cannot be obtained. To address this, in the present embodiment, the
amount of light irradiated onto the fundus oculi is not increased
during image capture, and the imaging portion 21 repeats image
capture of the fundus oculi a plurality of times while maintaining
a state in which low-intensity light is irradiated onto the fundus
oculi. Note that image capture of the fundus oculi by the imaging
portion 21 may be performed a plurality of times to obtain still
images or may be performed once to obtain moving images.
[0054] Each of the fundus oculi images obtained by the single image
capture in this manner has low image quality because the light
irradiated during image capture is weak (dark). For this reason, in
the present embodiment, the image processing portion 22 performs
predetermined image processing on data of the fundus oculi images
obtained by each of the plurality of times of image capture by the
imaging portion 21, and thus generates and outputs data of one
sheet of a fundus oculi image with high image quality. Note that,
in this case, the improvement in image quality becomes easier when
the plurality of fundus oculi images, which are image processing
targets, are more similar to each other. Therefore, it is
preferable to reduce a time taken to perform a plurality of times
of imaging operations of the imaging portion 21.
[0055] The image processing portion 22 causes the data of the
fundus oculi image after the image processing, namely, the data of
the fundus oculi image with high image quality, to be stored in the
storage portion 23 or to be output from the output portion 24.
[0056] The storage portion 23 is formed by, for example, a hard
disk, a flash memory or a random access memory (RAM), and stores
the data of the fundus oculi image supplied from the image
processing portion 22. The data of the fundus oculi image stored in
the storage portion 23 is read by a playback portion or the like
(not shown in the drawings), and is output to the output portion 24
for display or transmitted to another device. Other image
processing is performed on the data of the fundus oculi image by an
image processing portion (not shown in the drawings) other than the
image processing portion 22.
[0057] The output portion 24 includes a monitor, such as a cathode
ray tube (CRT), a liquid crystal display (LCD) or the like, and an
output terminal etc. The output portion 24 outputs and displays on
the monitor the data of the fundus oculi image output from the
image processing portion 22, or outputs the data to an external
device from the output terminal of the output portion 24.
[0058] Further, hereinafter, a detailed configuration of the image
processing portion 22 included in the fundus oculi image processing
device 10 shown in FIG. 1 will be explained. The image processing
portion 22 includes an input image buffer 31, a motion vector
detection portion 32, a motion compensation portion 33, a synthesis
portion 34 and a super-resolution processing portion 35.
[0059] The input image buffer 31 is configured as one region of a
given storage medium, such as a hard disk, a flash memory or a RAM,
for example. The input image buffer 31 holds data of fundus oculi
images with low image quality that are sequentially supplied from
the imaging portion 21, as respective data of input images. The
data of each of the input images is read out from the input image
buffer 31 at a predetermined timing and is supplied to the motion
vector detection portion 32 or the motion compensation portion
33.
[0060] The data of the fundus oculi images obtained by the imaging
portion 21 are obtained by performing image capture a plurality of
times, and the fundus oculi images are not necessarily exactly the
same as each other. For example, it is conceivable that a
positional displacement occurs in some or all of the fundus oculi
images. Accordingly, if the plurality of fundus oculi images are
simply synthesized, there is a possibility that the resultant
fundus oculi image becomes a blurred image or a double image due to
the positional displacement or the like. Therefore, before the
synthesis portion 34 synthesizes the data of the plurality of
fundus oculi images, it is necessary that the motion vector
detection portion 32 detects a motion vector and the motion
compensation portion 33 performs motion compensation using the
motion vector to thereby reduce a difference (a positional
displacement etc.) between the images to be synthesized.
[0061] The motion vector detection portion 32 reads out data of a
processing target input image and data of an input image that is
captured at a different time from the processing target input
image, from the input image buffer 31. Next, the motion vector
detection portion 32 compares the read-out data of the two sheets
of input images and thereby detects a motion vector of the whole
fundus oculi in the processing target input image. Note that, as a
motion vector detection technique, the present embodiment adopts a
technique in which the eyeball is modeled as a three-dimensional
sphere and a motion vector of the whole sphere is detected. More
specifically, among the plurality of captured images including the
substantially spherical photographic subject, the motion vector
detection portion 32 performs comparison of the captured image as a
processing target and the captured image as a comparison target,
using a three-dimensional spherical model (hereinafter simply
referred to as a spherical model) of the photographic subject. The
comparison is performed using each of the plurality of captured
images as a processing target. This technique will be described
later with reference to FIG. 2, along with a detailed configuration
of the motion vector detection portion 32.
[0062] The motion compensation portion 33 reads out the data of the
processing target input image from the input image buffer 31. At
the same time, the motion compensation portion 33 acquires, from
the motion vector detection portion 32, the motion vector of the
whole fundus oculi in the processing target input image. Then, the
motion compensation portion 33 uses the motion vector of the whole
fundus oculi to perform motion compensation on the data of the
processing target input image. The motion compensation is
processing that moves the processing target input image on the
spherical model in accordance with the motion vector of the whole
fundus oculi in the processing target input image. By doing this, a
difference (a positional displacement etc.) between the plurality
of fundus oculi images is reduced. Data of fundus oculi images
after the motion compensation is supplied to the synthesis portion
34.
[0063] In this manner, the respective data of the plurality of
fundus oculi images obtained by the imaging portion 21 performing
image capture a plurality of times are sequentially used as a
processing target, and after the motion compensation portion 33
performs motion compensation on the respective data, they are
sequentially supplied to the synthesis portion 34.
[0064] When all the data of the plurality of fundus oculi images
are supplied from the motion compensation portion 33, the synthesis
portion 34 synthesizes the data of the plurality of fundus oculi
images and thereby generates data of one sheet of a fundus oculi
image. The synthesis portion 34 supplies the generated data to the
super-resolution processing portion 35.
[0065] The super-resolution processing portion 35 performs
super-resolution processing on the data of the fundus oculi image
synthesized by the synthesis portion 34, and thereby generates data
of the fundus oculi image with a higher resolution than that at the
time of synthesis. Note that any processing method can be used to
perform the super-resolution processing by the super-resolution
processing portion 35. For example, a method described in Japanese
Patent Application Publication No. JP-A-2010-102696, or a method
described in Japanese Patent Application Publication No.
JP-A-2010-103981 may be used to perform the super-resolution
processing. Note however that, in the super-resolution processing,
processing in accordance with features of a living organism is
performed so that a higher-resolution image with less noise can be
obtained. The data of the high-resolution fundus oculi image
generated in this manner by the super-resolution processing portion
35 is stored in the storage portion 23 or output from the output
portion 24.
[0066] Next, a detailed configuration of the motion vector
detection portion 32 will be explained.
[0067] Configuration example and processing of motion vector
detection portion
[0068] FIG. 2 is a block diagram showing a configuration example of
the motion vector detection portion 32. FIG. 3 is a diagram
illustrating specific processing of the motion vector detection
portion 32.
[0069] As shown in FIG. 2, the motion vector detection portion 32
includes a local motion vector detection portion 41, a spherical
motion vector conversion portion 42, a spherical model storage
portion 43 and a fundus oculi motion vector detection portion
44.
[0070] The local motion vector detection portion 41 reads out data
of a processing target fundus oculi image 51-i and data of a
comparison target fundus oculi image 51-j that is different from
the processing target, from among respective data of n sheets of
fundus oculi images 51-1 to 51-n that are respectively held as
input image data in the input image buffer 31 as shown in FIG. 3.
Here, n is an integer equal to or larger than 2 and indicates a
total number of the input images held in the input image buffer 31.
i is an integer equal to or larger than 1 and equal to or smaller
than n-1. j is an integer equal to or larger than 1 and equal to or
smaller than n, and is an integer different from the integer i. For
example, in the present embodiment, j=i+1 is set. In summary, an
image adjacent to a processing target image is used as a comparison
target. The local motion vector detection portion 41 divides up the
processing target fundus oculi image 51-i into a plurality of
blocks having a prescribed size, and sequentially sets each of the
plurality of blocks as a processing target block 61-i.
[0071] The local motion vector detection portion 41 also divides up
the comparison target fundus oculi image 51-j into a plurality of
blocks having a prescribed size, in a similar manner to the above.
Each time the local motion vector detection portion 41 sets each of
the plurality of blocks as a comparison target block 61-j in a
raster scan order, for example, the local motion vector detection
portion 41 repeatedly calculates a degree of similarity between the
processing target block 61-i and the comparison target block 61-j.
In summary, so-called block matching is performed.
[0072] The local motion vector detection portion 41 detects local
motion in the processing target block 61-i, as a motion vector my,
based on a positional relationship between the processing target
block 61-i and the comparison target block 61-j matched with
(namely, having a highest degree of similarity with) the processing
target block 61-i.
[0073] The spherical motion vector conversion portion 42 applies
the processing target block 61-i to a spherical model 63 that is
stored in the spherical model storage portion 43, as shown in FIG.
3. By doing this, the processing target block 61-i is converted
into a predetermined block 64-i on the spherical model 63. Note
that the converted block 64-i is hereinafter referred to as a
spherical processing target block 64-i. In this case, the motion
vector my in the processing target block 61-i is converted into a
motion vector mvr in the spherical processing target block 64-i.
Note that the motion vector mvr after the conversion is hereinafter
referred to as a spherical motion vector mvr.
[0074] This type of the spherical motion vector nvr is obtained by
repeatedly performing the above-described series of processing for
each of the plurality of blocks divided up from the processing
target fundus oculi image 51-i.
[0075] With respect to the processing target fundus oculi image
51-i, the fundus oculi motion vector detection portion 44 detects a
motion vector 65 of the whole fundus oculi, based on the spherical
motion vector mvr of each of the plurality of blocks. The fundus
oculi motion vector detection portion 44 supplies to the motion
compensation portion 33 the motion vector 65 of the whole fundus
oculi in the processing target fundus oculi image 51-i.
[0076] Thus, motion compensation using the motion vector 65 of the
whole fundus oculi is performed on the data of the processing
target fundus oculi image 51-i by the motion compensation portion
33, as described above with reference to FIG. 1.
[0077] The configuration of the fundus oculi image processing
device 10 is explained above with reference to FIG. 1 to FIG. 3.
Next, processing (hereinafter referred to as fundus oculi image
generation processing) that is performed by the fundus oculi image
processing device 10 configured as described above will be
explained with reference to FIG. 4.
[0078] Flow of fundus oculi image generation processing
[0079] FIG. 4 is a flowchart illustrating a flow of the fundus
oculi image generation processing.
[0080] At step S11, the imaging portion 21 reduces the amount of
light and captures an image of the fundus oculi of the test subject
a plurality of times. Note that, as described above, image capture
of the fundus oculi by the imaging portion 21 may be performed a
plurality of times to obtain still images or may be performed once
to obtain moving images.
[0081] At step S12, the image processing portion 22 causes the
input image buffer 31 to store the data of the plurality of fundus
oculi images with low image quality obtained by the processing at
step S11.
[0082] At step S3, the motion vector detection portion 32 performs
motion vector detection processing and detects the motion vector of
the whole fundus oculi. Note that the motion vector detection
processing will be described in more detail later with reference to
FIG. 5.
[0083] At step 14, the motion compensation portion 33 uses the
motion vector of the whole fundus oculi detected by the processing
at step S13 to perform motion compensation on the data of the
processing target input image read out from the input image buffer
31. Note that the motion compensation portion 33 performs similar
motion compensation also on the data of the input image on which
the processing has been performed.
[0084] At step S15, the image processing portion 22 determines
whether or not all the data of the fundus oculi images have been
set as the processing target. In the present embodiment, the data
of the processing target fundus oculi images are respective data of
the fundus oculi images 51-1 to 51-(n-1). Therefore, the data of
the comparison target fundus oculi images are respective data of
the fundus oculi images 51-2 to 51-n.
[0085] When all the data of the fundus oculi images have not yet
been set as the processing target, NO is determined at step S15 and
the processing is returned to step S13. Then, the processing from
step S13 onward is repeated. More specifically, loop processing
from step S13 to step S15 is repeated until all the data of the
fundus oculi images are set as the processing target. For example,
as described later with reference to FIG. 5, if a motion vector is
detected for the fundus oculi image 51-2, which is the comparison
target of the fundus oculi image 51-1 set as the processing target
by the motion vector detection processing at step S3, then the
fundus oculi image 51-2 is set as the processing target image, and
a motion vector for an adjacent comparison target fundus oculi
image 51-3 is detected. This type of processing is sequentially
repeated.
[0086] After that, when all the data of the fundus oculi images
have been set as the processing target, YES is determined at step
S15 and the processing proceeds to step S16.
[0087] At step S16, the synthesis portion 34 synthesizes the motion
compensated data of the plurality of the fundus oculi images. Thus,
data of one sheet of a fundus oculi image is generated from the
data of the n sheets of fundus oculi images.
[0088] At step S17, the super-resolution processing portion 35
performs super-resolution processing on the synthesized data of the
fundus oculi image. Thus, data of the fundus oculi image with a
higher resolution than that at the time of synthesis at step S16 is
generated.
[0089] At step S18, the image processing portion 22 causes the
storage portion 23 to store the data of the fundus oculi image on
which the super-resolution processing has been performed, or causes
the output portion 24 to output the data.
[0090] This completes the fundus oculi image generation processing.
Next, the motion vector detection processing at step S13 will be
explained.
[0091] Flow of motion vector detection processing
[0092] FIG. 5 is a flowchart illustrating a flow of the motion
vector detection processing performed at step S13 shown in FIG.
4.
[0093] At step S31, the local motion vector detection portion 41
reads out data of two sheets of fundus oculi images, which are a
processing target and a comparison target, from the input image
buffer 31. For example, data of two sheets of adjacent fundus oculi
images, the fundus oculi image 51-1 and the fundus oculi image
51-2, are read out.
[0094] At step 532, the local motion vector detection portion 41
sets a processing target block from a processing target image (for
example, the fundus oculi image 51-1) and performs block matching
with a comparison target fundus oculi image (for example, the
fundus oculi image 51-2). As a result, a motion vector in the
processing target block is detected.
[0095] At step S33, the spherical motion vector conversion portion
42 applies the motion vector detected at step S32 to the spherical
model stored in the spherical model storage portion 43, and thereby
converts the motion vector to a spherical motion vector.
[0096] At step 534, the motion vector detection portion 32
determines whether or not all the blocks have been processed. More
specifically, the motion vector detection portion 32 determines
whether or not spherical motion vectors corresponding to all the
blocks divided up from a sheet of the processing target fundus
oculi image (for example, the sheet of the fundus oculi image 51-1
read out by the processing at step S31) have been detected.
[0097] When all the blocks have not yet been processed, NO is
determined at step S34 and the processing is returned to step S32.
Then, the processing from step S32 onward is repeated. More
specifically, loop processing from step S32 to step S34 is repeated
until all the blocks are processed.
[0098] After that, when all the blocks have been processed, YES is
determined at step S34 and the processing proceeds to step S35.
[0099] At step S35, with respect to the processing target fundus
oculi image (for example, the sheet of the fundus oculi image 51-1
read out by the processing at step S31), the fundus oculi motion
vector detection portion 44 detects a motion vector of the whole
fundus oculi based on the respective spherical motion vectors of
the plurality of blocks.
[0100] This completes the motion vector detection processing.
[0101] In this manner, the image processing using a spherical model
is performed on the plurality of fundus oculi images that are
obtained while suppressing the amount of irradiated light in order
to reduce a burden on the test subject. Thus, data of one sheet of
a fundus oculi image with high image quality is generated.
[0102] In the present embodiment, general block matching is used as
degree of similarity calculation between the processing target
block divided up from the processing target fundus oculi image and
the comparison target block divided up from the comparison target
fundus oculi image. Since this type of general block matching is
applied, high-speed processing can be easily achieved. Further,
since the block matching is performed for each of the processing
target blocks, it is possible to reduce a memory amount to be
used.
Second Embodiment
[0103] In the motion vector detection portion 32 of the first
embodiment, a spherical model is applied to the motion vector in
the processing target block detected by block matching. However,
the target to which the spherical model is applied is not limited
to this example. For example, the spherical model may be applied to
the processing target block before the block matching is performed
and the block matching may be performed thereafter.
[0104] Note that a fundus oculi image processing device according
to a second embodiment has basically the same function and
configuration as those of the fundus oculi image processing device
10 shown in FIG. 1. Therefore, hereinafter, an explanation of same
portions as those of the fundus oculi image processing device 10
shown in FIG. 1 is omitted, and a different portion only, namely, a
motion vector detection portion 70 that differs from the motion
vector detection portion 32 of the fundus oculi image processing
device 10 shown in FIG. 1, will be explained.
[0105] Configuration example and processing of motion vector
detection portion 70
[0106] FIG. 6 is a block diagram showing a configuration example of
the motion vector detection portion 70. FIG. 7 is a diagram
illustrating specific processing of the motion vector detection
portion 70.
[0107] As shown in FIG. 6, the motion vector detection portion 70
includes a local spherical motion vector detection portion 71, a
spherical model storage portion 72 and a fundus oculi motion vector
detection portion 73.
[0108] The local spherical motion vector detection portion 71 reads
out data of a processing target fundus oculi image 81-i and data of
a comparison target fundus oculi image 81-j that is different from
the processing target, from among respective data of n sheets of
fundus oculi images 81-1 to 81-n that are respectively held as
input image data in the input image buffer 31, as shown in FIG. 7.
Here, n is an integer equal to or larger than 2 and indicates a
total number of the input images held in the input image buffer 31.
i is an integer equal to or larger than 1 and equal to or smaller
than n. j is an integer equal to or larger than 1 and equal to or
smaller than n, and is an integer different from the integer i.
[0109] The local spherical motion vector detection portion 71
divides up the processing target fundus oculi image 81-i into a
plurality of blocks having a prescribed size, and sequentially sets
each of the plurality of blocks as a processing target block
91-i.
[0110] The local spherical motion vector detection portion 71
applies the processing target block 91-i to a spherical model 92
that is stored in the spherical model storage portion 72, as shown
in FIG. 7. By doing this, the processing target block 91-i.
converted into a predetermined block 93-i on the spherical model
92. Note that the converted block 93-i is hereinafter referred to
as the processing target spherical block 93-i.
[0111] The local spherical motion vector detection portion 71 also
divides up the comparison target fundus oculi image 81-j into a
plurality of blocks having a prescribed size, in a similar manner
to the above. Each time the local spherical motion vector detection
portion 71 sets each of the plurality of blocks as a comparison
target block 91-j in a raster order, for example, the local
spherical motion vector detection portion 71 also applies the
comparison target block 91-j to the spherical model 92. By doing
this, the comparison target block 91-j is converted into a
predetermined block 93-j on the spherical model 92. Note that the
converted block 93-j is hereinafter referred to as the comparison
target spherical block 93-j.
[0112] The local spherical motion vector detection portion 71
repeatedly calculates a degree of similarity between the processing
target spherical block 93-i and the comparison target spherical
block 93-j. In summary, so-called block matching is performed.
[0113] The local spherical motion vector detection portion 71
detects local motion of the sphere in the processing target
spherical block 93-i, as the spherical motion vector mvr, based on
a positional relationship between the processing target spherical
block 93-i and the comparison target spherical block 93-j matched
(namely, having a highest degree of similarity) with the processing
target spherical block 93-i.
[0114] This type of the spherical motion vector mvr is obtained by
repeatedly performing the above-described series of processing for
each of the plurality of blocks divided up from the processing
target fundus oculi image 81-i.
[0115] With respect to the processing target fundus oculi image
81-i, the fundus oculi motion vector detection portion 73 detects a
motion vector 95 of the whole fundus oculi, based on the respective
spherical motion vectors mvr of the plurality of blocks. The fundus
oculi motion vector detection portion 73 supplies the motion vector
95 of the whole fundus oculi in the processing target fundus oculi
image 81-i to the motion compensation portion 33.
[0116] Thus, motion compensation using the motion vector 95 of the
whole fundus oculi is performed on the data of the processing
target fundus oculi image 81-i by the motion compensation portion
33, as described above with reference to FIG. 1.
[0117] Next, an explanation will be given of fundus oculi image
generation processing of the fundus oculi image processing device
10 according to the second embodiment that has the motion vector
detection portion 70 configured in this manner. The fundus oculi
image generation processing according to the second embodiment is
performed in accordance with the flowchart shown in FIG. 4, in a
similar manner to the first embodiment. However, content of motion
vector detection processing at step S13 in the second embodiment is
different from that in the first embodiment. Therefore,
hereinafter, the motion vector detection processing at step S13 in
the second embodiment will be explained with reference to FIG.
8.
[0118] Flow of motion vector detection processing
[0119] FIG. 8 is a flowchart illustrating a flow of the motion
vector detection processing.
[0120] At step S51, the local spherical motion vector detection
portion 71 divides up the respective data of the two sheets of
fundus oculi images (which are the processing target and the
comparison target) read out from the input image buffer 31 into a
plurality of blocks. The local spherical motion vector detection
portion 71 applies each of the plurality of blocks to a spherical
model and converts each of the plurality of blocks to a spherical
block.
[0121] At step 552, the local spherical motion vector detection
portion 71 sets a processing target spherical block from the
processing target fundus oculi image.
[0122] At step S53, the local spherical motion vector detection
portion 71 performs block matching between the processing target
spherical block and a comparison target spherical block. Thus, a
spherical motion vector in the processing target spherical block is
detected.
[0123] At step 554, the motion vector detection portion 70
determines whether or not all the blocks have been processed. More
specifically, the motion vector detection portion 70 determines
whether or not spherical motion vectors corresponding to all the
blocks divided up from one sheet of the processing target fundus
oculi image have been detected.
[0124] When all the blocks have not yet been processed, NO is
determined at step S54 and the processing is returned to step S52.
Then, the processing from step S52 onward is repeated. More
specifically, loop processing from step S52 to step S54 is repeated
until all the blocks are processed.
[0125] After that, when all the blocks have been processed, YES is
determined at step S54 and the processing proceeds to step S55.
[0126] At step S55, with respect to the processing target findus
oculi image, the fundus oculi motion vector detection portion 73
detects a motion vector of the whole fundus oculi based on the
respective spherical motion vectors of the plurality of blocks.
[0127] This completes the motion vector detection processing.
[0128] In the present embodiment, block matching is performed after
the processing target block divided up from the processing target
fundus oculi image and the comparison target block divided up from
the comparison target fundus oculi image have each been applied to
the spherical model. Therefore, the blocks on the spherical model
are used in the degree of similarity calculation between the
processing target fundus oculi image and the comparison target
fundus oculi image. Thus, the motion vector of the whole fundus
oculi, which is a substantially spherical body, can be detected
accurately. Since the image processing using this motion vector is
performed on the captured fundus oculi images, a fundus oculi image
with higher image quality is generated in comparison with the first
embodiment.
Third Embodiment
[0129] In the motion vector detection portions 32 and 70 according
to the first and second embodiments, the spherical model is applied
to the processing target block divided up from the processing
target image. However, the target to which the spherical model is
applied is not limited to this example. For example, the spherical
model may be applied to the whole fundus oculi image and thereafter
matching may be performed.
[0130] Note that a fundus oculi image processing device according
to a third embodiment has basically the same function and
configuration as those of the fundus oculi image processing device
10 shown in FIG. 1. Therefore, hereinafter, an explanation of same
portions as those of the fundus oculi image processing device 10
shown in FIG. 1 is omitted, and only a different portion, namely, a
motion vector detection portion 100 that differs from the motion
vector detection portion 32 of the fundus oculi image processing
device 10 shown in FIG. 1, will be explained.
[0131] Configuration example and processing of motion vector
detection portion 100
[0132] FIG. 9 is a block diagram showing a configuration example of
the motion vector detection portion 100. FIG. 10 is a diagram
illustrating specific processing of the motion vector detection
portion 100.
[0133] As shown in FIG. 9, the motion vector detection portion 100
includes a fundus oculi sphere conversion portion 101, a spherical
model storage portion 102 and a fundus oculi motion vector
detection portion 103.
[0134] The fundus oculi sphere conversion portion 101 reads out
data of a processing target fundus oculi image 111-i and data of a
comparison target fundus oculi image 11-j that is different from
the processing target, from among respective data of n sheets of
fundus oculi images 111-1 to 111-n that are respectively held as
input image data in the input image buffer 31, as shown in FIG. 10.
Here, n is an integer equal to or larger than 2 and indicates a
total number of the input images held in the input image buffer 31.
i is an integer equal to or larger than 1 and equal to or smaller
than n-1. j is an integer equal to or larger than 1 and equal to or
smaller than n, and is an integer different from the integer i. For
example, in the present embodiment, j=i+1 is set.
[0135] As shown in FIG. 10, the fundus oculi sphere conversion
portion 101 applies the whole of the processing target fundus oculi
image 111-i to a spherical model that is stored in the spherical
model storage portion 102. Thus, the processing target fundus oculi
image 111-i is converted into a fundus oculi image 112i on the
spherical model. Note that the converted fundus oculi image 112i is
hereinafter referred to as the processing target spherical fundus
oculi image 112i.
[0136] The fundus oculi sphere conversion portion 101 also applies
a comparison target fundus oculi image 111-j to the spherical model
stored in the spherical model storage portion 102, in a similar
manner to the above. By doing this, the comparison target fundus
oculi image 111-j is converted into a fundus oculi image 113j on
the spherical model. Note that the converted fundus oculi image
113j is hereinafter referred to as the comparison target spherical
fundus oculi image 113j.
[0137] The fundus oculi motion vector detection portion 103
repeatedly calculates a degree of similarity while rotating the
processing target spherical fundus oculi image 112i and the
comparison target spherical fundus oculi image 113j. In summary,
matching is performed on the whole sphere. Here, a technique for
matching between the processing target spherical fundus oculi image
112i and the comparison target spherical fundus oculi image 113j is
not particularly limited.
[0138] The fundus oculi motion vector detection portion 103 detects
a motion vector 114 of the whole fundus oculi, based on a
positional relationship between the processing target spherical
fundus oculi image 112i and the comparison target spherical fundus
oculi image 113j matched (namely, having a highest degree of
similarity) with the processing target spherical fundus oculi image
112i. The fundus oculi motion vector detection portion 103
supplies, to the motion compensation portion 33, the motion vector
114 of the whole fundus oculi in the processing target spherical
fundus oculi image 112i.
[0139] Thus, motion compensation using the motion vector 114 of the
whole fundus oculi is performed on the data of the processing
target fundus oculi image 111-i by the motion compensation portion
33, as described above with reference to FIG. 1.
[0140] Next, an explanation will be given of fundus oculi image
generation processing of the fundus oculi image processing device
10 according to the third embodiment that has the motion vector
detection portion 100 configured in this manner. The fundus oculi
image generation processing according to the third embodiment is
performed in accordance with the flowchart shown in FIG. 4, in a
similar manner to the first embodiment. However, content of motion
vector detection processing at step S13 in the third embodiment is
different from that in the first embodiment. Therefore,
hereinafter, the motion vector detection processing at step S13 in
the third embodiment will be explained with reference to FIG.
11.
[0141] Flow of Motion Vector Detection Processing
[0142] FIG. 11 is a flowchart illustrating a flow of the motion
vector detection processing.
[0143] At step S71, the fundus oculi sphere conversion portion 101
reads out data of two sheets of fundus oculi images, which are the
processing target and the comparison target, from the input image
buffer 31.
[0144] At step S72, the fundus oculi sphere conversion portion 101
applies the processing target whole fundus oculi image and the
comparison target whole fundus oculi image read out at step S71 to
the spherical model stored in the spherical model storage portion
102. The fundus oculi sphere conversion portion 101 converts them
to spherical fundus oculi images, respectively.
[0145] At step S73, the fundus oculi motion vector detection
portion 103 performs matching between the converted processing
target spherical fundus oculi image and the converted comparison
target spherical fundus oculi image, and detects a motion vector of
the whole fundus oculi.
[0146] This completes the motion vector detection processing.
[0147] In the present embodiment, matching is performed after the
processing target whole fundus oculi image and the comparison
target whole fundus oculi image have each been applied to the
spherical model. Therefore, the whole fundus oculi images on the
spherical model are used in the degree of similarity calculation
between the processing target fundus oculi image and the comparison
target fundus oculi image. Thus, the motion vector of the whole
fundus oculi, which is a substantially spherical body, can be
detected accurately. Since the image processing using this motion
vector is performed on the captured fundus oculi images, a fundus
oculi image with higher image quality is generated in comparison
with the second embodiment.
Fourth Embodiment
[0148] In the fundus oculi image processing devices 10 according to
the first to third embodiments, the motion vector of the whole
fundus oculi is detected by performing matching between the
processing target fundus oculi image and the comparison target
fundus oculi image. However, in many cases, the fundus oculi image
basically has a substantially uniform color in the whole image.
Further, since the amount of irradiated light is reduced, the
fundus oculi image tends to be a relatively dark image. In
addition, a plurality of times of image capture by the imaging
portion 21 is performed in a relatively short time and under
conditions that are as close to each other as possible. Therefore,
an amount of motion between the plurality of fundus oculi images
tends to be relatively small. Furthermore, even when there is a
motion, it is rare that some regions of the fundus oculi image show
a significantly large movement compared to the other regions, and
substantially the whole image tends to move almost uniformly.
Accordingly, detection of the motion vector may become difficult.
To address this, instead of detecting the motion vector, alignment
of the fundus oculi image may be performed using biological
information of the photographic subject, with respect to the whole
fundus oculi image.
[0149] In the present embodiment, for example, information of a
blood vessel shape is used as the biological information of the
photographic subject that is used for alignment of the fundus oculi
image. Note that the biological information of the photographic
subject is not limited to this example, and it may be any
information. For example, information of the shape of a nerve, a
nerve papilla or the like may be adopted as the biological
information of the photographic subject. Further, when an organ or
a cell is used as the photographic subject, information of the
shape etc. of the cell or a nucleus of the cell may be adopted as
the biological information of the photographic subject.
Furthermore, a plurality of types of biological information (for
example, a blood vessel and an optic papilla etc.) may be combined
and adopted.
[0150] Configuration example and processing of fundus oculi image
processing device
[0151] FIG. 12 is a block diagram showing a configuration example
of a fundus oculi image processing device 200.
[0152] The fundus oculi image processing device 200 shown in FIG.
12 has basically the same function and configuration as those of
the fundus oculi image processing device shown in FIG. 1.
Therefore, hereinafter, an explanation of same portions as those of
the fundus oculi image processing device 10 shown in FIG. 1 is
omitted, and only a different portion, namely, an image processing
portion 212 that differs from the image processing portion 22 of
the fundus oculi image processing device 10 shown in FIG. 1, will
be explained.
[0153] The image processing portion 212 includes an input image
buffer 221, a fundus oculi sphere conversion portion 222, a
spherical model storage portion 223, a feature extraction portion
224, an alignment portion 225, a synthesis portion 226 and a
super-resolution processing portion 227.
[0154] The input image buffer 221 has basically the same function
and configuration as those of the input image buffer 31 shown in
FIG. 1. The input image buffer 221 holds data of fundus oculi
images with low image quality that are sequentially supplied from
the imaging portion 21, as respective data of the input images. The
data of each of the input images is read out from the input image
buffer 221 at a predetermined timing and is supplied to the fundus
oculi sphere conversion portion 222.
[0155] The fundus oculi sphere conversion portion 222 has basically
the same function and configuration as those of the fundus oculi
sphere conversion portion 101 shown in FIG. 9. The fundus oculi
sphere conversion portion 222 reads out data of a processing target
fundus oculi image and data of a comparison target fundus oculi
image that is different from the processing target, from among
respective data of a plurality of fundus oculi images that are
respectively held as input image data in the input image buffer
221. Then, the fundus oculi sphere conversion portion 222 applies
the processing target fundus oculi image and the comparison target
fundus oculi image to a spherical model that is stored in the
spherical model storage portion 223. Thus, the processing target
fundus oculi image and the comparison target fundus oculi image are
converted into fundus oculi images on the spherical model. Note
that the converted processing target fundus oculi image is
hereinafter referred to as a processing target spherical fundus
oculi image. Further, the converted comparison target fundus oculi
image is hereinafter referred to as a comparison target spherical
fundus oculi image. The fundus oculi sphere conversion portion 222
supplies the processing target spherical fundus oculi image and the
comparison target spherical fundus oculi image to the feature
extraction portion 224.
[0156] The feature extraction portion 224 includes a blood vessel
extraction portion 231 and an intersection extraction portion 233.
The alignment portion 225 includes a blood vessel alignment
processing portion 232 and an intersection alignment processing
portion 234. Specific processing of the feature extraction portion
224 and the alignment portion 225 will be explained with reference
to FIG. 13.
[0157] As shown in FIG. 13, the blood vessel extraction portion 231
extracts features (processing 261, 262), such as the shape and
position etc., of a blood vessel, from each of a processing target
spherical fundus oculi image 251 and a comparison target spherical
fundus oculi image 252 supplied from the fundus oculi sphere
conversion portion 222. At this time, the blood vessel extraction
portion 231 uses an R component of RGB components to extract a
blood vessel from each of the processing target spherical fundus
oculi image 251 and the comparison target spherical fundus oculi
image 252, as in a method described in "Fundus oculi image
synthesis method using blood vessel features", Katsuyoshi Tanabe,
Tetsuro Tsubouchi, Ilidenori Okuda, Masahiro Oku, 2007. The blood
vessel extraction portion 231 supplies the features, such as the
shape and position etc., of the blood vessel extracted from each of
the processing target spherical fundus oculi image 251 and the
comparison target spherical fundus oculi image 252 to the blood
vessel alignment processing portion 232, as a blood vessel
extraction result of each of the images.
[0158] As shown in FIG. 13, the blood vessel alignment processing
portion 232 performs blood vessel alignment processing (processing
266) between the processing target spherical fundus oculi image 251
and the comparison target spherical fundus oculi image 252, using
the blood vessel extraction result of the processing target
spherical fundus oculi image 251 and the blood vessel extraction
result of the comparison target spherical fundus oculi image 252
that are supplied from the blood vessel extraction portion 231.
[0159] The blood vessel alignment processing portion 232 supplies
the synthesis portion 226 with a processing target spherical fundus
oculi image 253 on which the blood vessel alignment processing has
been performed. Note that the configuration and processing of the
blood vessel alignment processing portion 232 will be explained in
more detail later with reference to FIG. 14 and FIG. 15.
[0160] Further, before the blood vessel alignment processing (the
processing 266) is performed using the blood vessel extraction
results, simple alignment may be performed using positions of blood
vessel intersections. Note that the blood vessel intersection is a
portion at which blood vessels intersect (including a case of a
torsion position in actuality) in the fundus oculi image, or a
portion at which the blood vessels diverge.
[0161] In this case, as shown in FIG. 13, the blood vessel
extraction portion 231 supplies a blood vessel extraction result of
the processing target spherical fundus oculi image 251 that is
obtained by the processing 261, to the intersection extraction
portion 233. Further, as shown in FIG. 13, the blood vessel
extraction portion 231 supplies a blood vessel extraction result of
the comparison target spherical fundus oculi image 252 that is
obtained by the processing 262, to the intersection extraction
portion 233.
[0162] As shown in FIG. 13, the intersection extraction portion 233
extracts an intersection (processing 263) using the blood vessel
extraction result of the processing target spherical fundus oculi
image 251 that is supplied from the blood vessel extraction portion
231. Further, the intersection extraction portion 233 extracts an
intersection (processing 264) using the blood vessel extraction
result of the comparison target spherical fundus oculi image 252
that is supplied from the blood vessel extraction portion 231. The
intersection extraction portion 233 supplies positions of the
intersections respectively extracted from the processing target
spherical fundus oculi image 251 and the comparison target
spherical fundus oculi image 252 to the intersection alignment
processing portion 234, as intersection extraction results.
[0163] As shown in FIG. 13, the intersection alignment processing
portion 234 performs intersection alignment processing (processing
265) between the processing target spherical fundus oculi image 251
and the comparison target spherical fundus oculi image 252, using
the respective intersection extraction results of the processing
target spherical fundus oculi image 251 and the comparison target
spherical fundus oculi image 252 that are supplied from the
intersection extraction portion 233. The intersection alignment
processing portion 234 supplies a result of the intersection
alignment processing to the blood vessel alignment processing
portion 232, as an intersection alignment result.
[0164] The blood vessel alignment processing portion 232 uses the
intersection alignment result supplied from the intersection
alignment processing portion 234 as a spherical fundus oculi image
in an initial state, and further performs blood vessel alignment
processing (processing 266) using the blood vessel extraction
results on the spherical fundus oculi image in the initial state.
More specifically, while performing alignment in a similar manner
to the intersection alignment in accordance with the intersection
alignment result, the blood vessel alignment processing portion 232
superimposes the respective blood vessel extraction results and
sets the superimposed image as the initial state.
[0165] In this manner, the blood vessel alignment processing
portion 232 can further perform alignment using the blood vessel
extraction results on the spherical fundus oculi image for which
alignment has been simply performed using the intersection.
Therefore, the blood vessel alignment processing portion 232 can
perform alignment more easily and at a higher speed.
[0166] Note that, further, alignment using other biological
information may be adopted at the same time. For example, firstly,
while performing alignment at a position of the optic papilla,
alignment using the intersection may be further performed on the
spherical fundus oculi image in the initial state, using the
spherical fundus oculi image obtained by superimposing the
processing target spherical fundus oculi image 251 and the
comparison target spherical fundus oculi image 252 as the spherical
fundus oculi image in the initial state.
[0167] When all the data of the plurality of spherical fundus oculi
images that have been aligned are supplied from the blood vessel
alignment processing portion 232, the synthesis portion 226
synthesizes the respective data of the plurality of spherical
fundus oculi images and thereby generates data of one sheet of a
fundus oculi image. The synthesis portion 226 supplies the
generated data to the super-resolution processing portion 227.
[0168] The super-resolution processing portion 227 performs
super-resolution processing on the data of the fundus oculi image
synthesized by the synthesis portion 226, and thereby generates
data of the fundus oculi image with an even higher resolution than
that at the time of synthesis. The data of the high-resolution
fundus oculi image generated in this manner by the super-resolution
processing portion 227 is stored in the storage portion 23 or
output from the output portion 24.
[0169] Next, the configuration and processing of the blood vessel
alignment processing portion 232 will be explained in detail.
[0170] Configuration example and processing of blood vessel
alignment processing portion
[0171] FIG. 14 is a diagram showing a configuration example of the
blood vessel alignment processing portion 232. FIG. 15 is a diagram
illustrating specific processing of the blood vessel alignment
processing portion 232.
[0172] As shown in FIG. 14, the blood vessel alignment processing
portion 232 includes a superimposition processing portion 271, a
shift processing portion 272, a stretch processing portion 273, a
rotation processing portion 274, a zoom-in/zoom-out processing
portion 275, a convergence determination portion 276 and an
adjustment portion 277.
[0173] The superimposition processing portion 271 superimposes the
respective blood vessel extraction results of the processing target
spherical fundus oculi image 251 and the comparison target
spherical fundus oculi image 252 that are supplied from the blood
vessel extraction portion 231. When alignment using the
intersection is performed, the superimposition processing portion
271 superimposes the respective blood vessel extraction results
while performing alignment in a similar manner to the intersection
alignment using the intersection alignment result supplied from the
intersection alignment processing portion 234. The superimposition
processing portion 271 supplies a superimposed result to the shift
processing portion 272. Note that the blood vessel alignment
processing portion 232 performs alignment so that a blood vessel
extraction result 292 approaches a blood vessel extraction result
291.
[0174] As shown in FIG. 15, the shift processing portion 272
performs a vertical/horizontal shift 281 that causes the whole of
the blood vessel extraction result 292 to move (shift) in a given
direction, such as a vertical direction or a horizontal direction.
The shift processing portion 272 supplies a superimposed result to
the stretch processing portion 273 in a state in which the blood
vessel extraction result 292 approaches the blood vessel extraction
result 291 to a maximum extent. Although any method can be used to
determine how close the blood vessel extraction result 291 and the
blood vessel extraction result 292 are to each other, a difference
between absolute values of the two images, for example, can be used
for determination. More specifically, the shift processing portion
272 causes the whole of the blood vessel extraction result 292 to
move (shift) and searches for a position at which the difference
between the absolute values of the blood vessel extraction result
291 and the blood vessel extraction result 292 is minimum. This
determination method also applies to the following processing
portions.
[0175] As shown in FIG. 15, the stretch processing portion 273
performs a vertical/horizontal stretch 282 that stretches (deforms)
the blood vessel extraction result 291 in a given direction, such
as the vertical direction or the horizontal direction. The stretch
processing portion 273 supplies a superimposed result to the
rotation processing portion 274 in a state in which the blood
vessel extraction result 292 approaches the blood vessel extraction
result 291 to a maximum extent. For example, the stretch processing
portion 273 stretches (deforms) the blood vessel extraction result
292 in a given direction, and searches for a shape that causes the
difference between the absolute values of the blood vessel
extraction result 291 and the blood vessel extraction result 292 to
be minimum.
[0176] As shown in FIG. 15, the rotation processing portion 274
performs a rotation 283 that rotates the blood vessel extraction
result 292 in left and right directions, and supplies a
superimposed result to the zoom-in/zoom-out processing portion 275
in a state in which the blood vessel extraction result 292
approaches the blood vessel extraction result 291 to a maximum
extent. For example, the rotation processing portion 274 rotates
the blood vessel extraction result 292 in the left and right
directions, and searches for a direction in which the difference
between the absolute values of the blood vessel extraction result
291 and the blood vessel extraction result 292 is minimum.
[0177] As shown in FIG. 15, the zoom-in/zoom-out processing portion
275 performs a zoom-in/zoom-out 284 that zooms in or zooms out the
blood vessel extraction result 292, and supplies a superimposed
result to the convergence determination portion 276 in a state in
which the blood vessel extraction result 292 approaches the blood
vessel extraction result 291 to a maximum extent. For example, the
zoom-in/zoom-out processing portion 275 zooms in or zooms out the
blood vessel extraction result 292, and searches for a size that
causes the difference between the absolute values of the blood
vessel extraction result 291 and the blood vessel extraction result
292 to be minimum.
[0178] The convergence determination portion 276 determines whether
or not the alignment has converged, based on the supplied
superimposed result. For example, the convergence determination
portion 276 causes each of the above-described processing to be
repeatedly performed, and compares an alignment result obtained
this time with an alignment result of the previous time. When the
blood vessel extraction result 292 is closer to the blood vessel
extraction result 291 than the previous time, the convergence
determination portion 276 determines that the alignment has not
converged. When the blood vessel extraction result 292 is not
closer to the blood vessel extraction result 291 than the previous
time (for example, when the difference between the absolute values
of the blood vessel extraction result 291 and the blood vessel
extraction result 292 is not smaller than the previous time), the
convergence determination portion 276 determines that the alignment
has converged.
[0179] When it is determined that the alignment has not converged
(for example, when the difference between the absolute values of
the blood vessel extraction result 291 and the blood vessel
extraction result 292 is smaller than the previous time), the
convergence determination portion 276 returns the superimposed
result to the shift processing portion 272 and causes the alignment
to be performed again.
[0180] When it is determined that the alignment has converged, the
adjustment portion 277 performs adjustment of the alignment based
on cumulative convergence results obtained until the previous time.
For example, it is assumed that first to fifth fundus oculi images,
which are in a time ascending order, are obtained by performing
image capture five times consecutively. In this case, when the
fifth fundus oculi image is set as the processing target, the
fourth fundus oculi image is used as the comparison target.
[0181] Note that the convergence result is a result of performing
alignment such that the fifth fundus oculi image approaches the
fourth fundus oculi image. In summary, at a stage when the
convergence is complete, the fifth fundus oculi image has only
approached the fourth fundus oculi image. However, in the synthesis
portion 266 that will be described later, an image (an aligned
image) obtained by causing the fifth fundus oculi image to approach
the first fundus oculi image is used as a synthesis target.
Therefore, it is necessary to cause the image immediately after the
convergence, namely, the image (the aligned image) obtained by
causing the fifth fundus oculi image to approach the fourth fundus
oculi image, to further approach the first fundus oculi image
(namely, it is necessary to adjust the alignment).
[0182] In order to cause the fourth fundus oculi image to approach
the first fundus oculi image (namely, in order to perform
adjustment of the alignment), it is necessary to use the cumulative
convergence results obtained until the previous time. Specifically,
adjustment of the alignment to cause the fourth fundus oculi image
to approach the first fundus oculi image is performed by
cumulatively using a convergence result (a last convergence result)
obtained by causing the fifth fundus oculi image to approach the
fourth fundus oculi image, a convergence result (a second last
convergence result) obtained by causing the fourth fundus oculi
image to approach the third fundus oculi image, a convergence
result (a third last convergence result) obtained by causing the
third fundus oculi image to approach the second fundus oculi image,
and a convergence result (a fourth last convergence result)
obtained by causing the second fundus oculi image to approach the
first fundus oculi image. Note that the order of adjustment of the
alignment is not limited to this example. Conversely, adjustment of
the alignment may be performed to approach a plurality of fundus
oculi images to the fifth fundus oculi image. The adjustment
portion 277 supplies the synthesis portion 226 with a processing
target spherical fundus oculi image 293 that has been aligned.
[0183] Note that, in the above explanation, as a specific example
of the alignment, four processing steps, i.e., the
vertical/horizontal shift 281, the vertical/horizontal stretch 282,
the rotation 283 and the zoom-in/zoom-out 284, are performed in
this order. However, this is merely an example and a processing
step other than the above-described processing steps may be further
performed, or a part of the above-described processing steps may be
omitted. Further when a plurality of processing steps are performed
as described above, the processing order can be set as desired.
[0184] Further, the feature extraction portion 224 and the
alignment portion 225 may perform alignment using a histogram of an
edge portion, as described in, for example, "Shape Matching and
Object Recognition Using Shape Contexts", Serge Belongie, Jitendra
Malik, Jan Puzicha, 2002.
[0185] Further, any method can be used to determine whether or not
the alignment has converged, and a method other than that described
above may be used. For example, it may be determined that the
alignment has converged when the difference between the absolute
values of the blood vessel extraction result 291 and the blood
vessel extraction result 292 is equal to or smaller than a
predetermined threshold value.
[0186] Note that the alignment using the intersection of blood
vessels is also performed basically in the same manner as the
alignment using the whole blood vessels. In other words, the
intersection alignment processing portion 234 has basically the
same configuration as the blood vessel alignment processing portion
232, and basically performs the same processing, the only
difference being whether the biological information used to perform
alignment is the shape of the whole blood vessels or the
intersection of the blood vessels.
[0187] As described above, since the fundus oculi image is an image
of a living organism, the fundus oculi image processing device 200
makes use of features of the image and thereby performs alignment
in the whole image using the biological information included in the
fundus oculi image. By doing this, the fundus oculi image
processing device 200 can achieve more accurate alignment more
easily.
[0188] The configuration of the fundus oculi image processing
device 200 is explained above with reference to FIG. 12 to FIG. 15.
Next, fundus oculi image generation processing that is performed by
the fundus oculi image processing device 200 configured in this
manner will be explained with reference to FIG. 16.
[0189] Flow of findus oculi image generation processing
[0190] FIG. 16 is a flowchart illustrating a flow of the fundus
oculi image generation processing.
[0191] At step S91, the imaging portion 21 reduces the amount of
light and captures an image of the fundus oculi of the test subject
a plurality of times. Note that, as described above, image capture
of the fundus oculi by the imaging portion 21 may be performed a
plurality of times to obtain still images or may be performed once
to obtain moving images.
[0192] At step S92, the image processing portion 212 causes the
input image buffer 221 to store data of the plurality of fundus
oculi images with low image quality obtained by the processing at
step S91.
[0193] At step S93, the fundus oculi sphere conversion portion 222
reads out data of two sheets of fundus oculi images, which are a
processing target and a comparison target, from the input image
buffer 221.
[0194] At step S94, the fundus oculi sphere conversion portion 222
applies the whole processing target fundus oculi image and the
whole comparison target fundus oculi image that are read out at
step S93 to the spherical model stored in the spherical model
storage portion 223, and converts them into a processing target
spherical fundus oculi image and a comparison target spherical
fundus oculi image, respectively.
[0195] At step S95, the blood vessel extraction portion 231
extracts the shape and position of the blood vessels from the
processing target spherical fundus oculi image. The extracted shape
and position of the blood vessels are supplied to the blood vessel
alignment processing portion 232 as a blood vessel extraction
result.
[0196] At step S96, the blood vessel extraction portion 231
extracts the shape and position of the blood vessels from the
comparison target spherical fundus oculi image. The extracted shape
and position of the blood vessels are supplied to the blood vessel
alignment processing portion 232 as a blood vessel extraction
result.
[0197] At step S97, the feature extraction portion 224 determines
whether or not to perform intersection alignment.
[0198] When the intersection alignment is not to be performed, NO
is determined at step S97 and the processing proceeds to step S101.
Note that processing from step S101 onward will be described
later.
[0199] On the other hand, when the intersection alignment is to be
performed, YES is determined at step S97 and the processing
proceeds to step 598. In this case, the blood vessel extraction
portion 231 supplies the blood vessel extraction results extracted
at step S95 and step S96 to the intersection extraction portion
233.
[0200] At step S98, the intersection extraction portion 233
extracts an intersection using the blood vessel extraction result
of the processing target spherical fundus oculi image. The position
of the extracted intersection is supplied to the intersection
alignment processing portion 234 as an intersection extraction
result.
[0201] At step S99, the intersection extraction portion 233
extracts an intersection using the blood vessel extraction result
of the comparison target spherical fundus oculi image. The position
of the extracted intersection is supplied to the intersection
alignment processing portion 234 as an intersection extraction
result.
[0202] At step S100, the intersection alignment processing portion
234 performs intersection alignment processing between the
processing target spherical fundus oculi image 251 and the
comparison target spherical fundus oculi image 252, using the
intersection extraction results supplied at step S98 and step
S99.
[0203] Note that the intersection alignment processing at step S10
is performed in the same manner as blood vessel alignment
processing that will be described later with reference to FIG. 17,
except that the intersection of blood vessels is used for alignment
instead of the whole blood vessels. Therefore, an explanation of
the intersection alignment processing at step S100 is omitted here
to avoid repetition.
[0204] At step S101, the blood vessel alignment processing portion
232 performs the blood vessel alignment processing. More
specifically, with respect to the spherical fundus oculi image
which has been simply aligned using the intersection, the blood
vessel alignment processing portion 232 further performs blood
vessel alignment using the blood vessel extraction results. The
blood vessel alignment processing at step S101 will be described in
more detail later with reference to FIG. 17.
[0205] At step S102, the image processing portion 212 determines
whether or not all the data of the fundus oculi images have been
set as the processing target.
[0206] When all the data of the fundus oculi images have not yet
been set as the processing target, NO is determined at step 102 and
the processing is returned to step 593. Then, processing from step
S93 onward is repeated. More specifically, loop processing from
step S93 to step S102 is repeated until all the data of the fundus
oculi images are set as the processing target.
[0207] After that, when all the data of the fundus oculi images
have been set as the processing target, YES is determined at step
102 and the processing proceeds to step S103.
[0208] At step S103, the synthesis portion 226 synthesizes data of
the plurality of spherical fundus oculi images that have been
aligned. As a result, data of one sheet of a fundus oculi image is
generated.
[0209] At step S104, the super-resolution processing portion 227
performs super-resolution processing on the synthesized data of the
fundus oculi image. As a result, data of the fundus oculi image
with a higher resolution than that at the time of synthesis at step
S103 is generated.
[0210] At step S105, the image processing portion 212 causes the
storage portion 23 to store the data of the fundus oculi image on
which the super-resolution processing has been performed, or causes
the output portion 24 to output the data.
[0211] This completes the fundus oculi image generation processing.
Next, the blood vessel alignment processing at step S101 will be
explained.
[0212] Flow of blood vessel alignment processing FIG. 17 is a
flowchart illustrating a flow of the blood vessel alignment
processing at step S101 shown in FIG. 16.
[0213] At step S111, the superimposition processing portion 271
determines whether or not the intersection alignment has been
performed.
[0214] When the intersection alignment has been performed, YES is
determined at step S111 and the processing proceeds to step
S12.
[0215] At step S112, the superimposition processing portion 271
sets the intersection alignment result as a superimposed result. In
accordance with the superimposed result, the superimposition
processing portion 271 superimposes the respective blood vessel
extraction results of the processing target spherical fundus oculi
image and the comparison target spherical fundus oculi image that
are supplied from the blood vessel extraction portion 231.
[0216] On the other hand, when the intersection alignment has not
been performed, NO is determined at step S111 and the processing
proceeds to step S113.
[0217] At step S113, the superimposition processing portion 271
superimposes the respective blood vessel extraction results of the
processing target spherical fundus oculi image and the comparison
target spherical fundus oculi image.
[0218] At step S114, the shift processing portion 272 performs
shift alignment that shifts the blood vessel extraction result of
the processing target spherical fundus oculi image.
[0219] At step S115, the stretch processing portion 273 performs
stretch alignment that elongates and contracts the blood vessel
extraction result of the processing target spherical fundus oculi
image.
[0220] At step S116, the rotation processing portion 274 performs
rotation alignment that rotates the blood vessel extraction result
of the processing target spherical fundus oculi image.
[0221] At step S117, the zoom-in/zoom-out processing portion 275
performs zoom-in/zoom-out alignment that zooms in or zooms out the
blood vessel extraction result of the processing target spherical
fundus oculi image.
[0222] At step S118, the convergence determination portion 276
determines whether or not the alignment has converged.
[0223] When the alignment has not converged, NO is determined at
step S118 and the processing is returned to step S114. Then,
processing from step S114 onward is repeated. More specifically,
loop processing from step S114 to step S118 is repeated until the
alignment converges.
[0224] After that, when the alignment has converged, YES is
determined at step S118 and the processing proceeds to step
S119.
[0225] At step S119, the adjustment portion 277 adjusts the
alignment based on cumulative convergence results obtained until
the previous time. As a result, the aligned processing target
spherical fundus oculi image is output to the synthesis portion
226.
[0226] This completes the blood vessel alignment processing, and
the processing proceeds to step S102 shown in FIG. 16.
[0227] In this manner, the alignment of the fundus oculi images is
performed using biological information of the photographic subject.
Thus, even in a situation in which it is difficult to detect a
motion vector, a higher quality fundus oculi image can be obtained
while reducing a burden on the test subject.
[0228] Note that the spherical model used in the above-described
examples may be switched to an appropriate spherical model for each
test subject in accordance with conditions of each test subject,
such as an eye axis length, eyesight and the like.
[0229] Further, the fundus oculi is not a perfect sphere.
Therefore, mask processing can also be performed on a region that
cannot approximate a sphere using a spherical model, so that the
region is not used for matching processing or alignment
processing.
[0230] [Application to a Program of the Present Technology]
[0231] The series of processes described above can be executed by
hardware and can also be executed by software.
[0232] In this case, a personal computer shown in FIG. 18, for
example, may be used as at least a part of the above-described
image processing device.
[0233] In FIG. 18, a CPU 301 performs various types of processing
in accordance with a program stored in a ROM 302. Further, the CPU
301 performs various types of processing in accordance with a
program that is loaded from a storage portion 308 to a RAM 303.
Data etc. that is necessary for the CPU 301 to perform the various
types of processing is also stored in the RAM 303 as
appropriate.
[0234] The CPU 301, the ROM 302 and the RAM 303 are mutually
connected via a bus 304. An input output (I/O) interface 305 is
also connected to the bus 304.
[0235] An input portion 306 that is formed by a keyboard, a mouse
and the like, and an output portion 307 that is formed by a display
and the like are connected to the I/O interface 305. Further, the
storage portion 308 that is formed by a hard disk and the like, and
a communication portion 309 that is formed by a modem, a terminal
adaptor and the like are connected to the I/O interface 305. The
communication portion 309 controls communication that is performed
with another device (not shown in the drawings) via a network
including the Internet.
[0236] Further, a drive 310 is connected to the/O interface 305
according to need. A removable media 311 that is formed by a
magnetic disk, an optical disk, a magneto optical disk, a
semiconductor memory, or the like is attached as appropriate. Then,
a computer program that is read from the removable media 311 is
installed in the storage portion 308 according to need.
[0237] When the series of processing is performed by software, a
program that forms the software is installed from a network or a
recording medium to a computer that is incorporated in a dedicated
hardware, or to, for example, a general-purpose personal computer
that can perform various types of functions by installing various
types of programs.
[0238] The recording medium that includes this type of program is
not only formed by the removable media (package media) 311 that is
distributed separately from a main body of the device as shown in
FIG. 18 in order to provide the user with the program, but is also
formed by the ROM 302 in which the program is recorded and which is
provided to the user in a state in which it is incorporated in
advance in the main body of the device, the hard disk included in
the storage portion 308, or the like. The removable media 311 is
formed by a magnetic disk (including a floppy disk) in which the
program is recorded, an optical disk (including a compact disk-read
only memory (CD-ROM) and a digital versatile disk (DVD)), a magneto
optical disk (including a mini-disk (MD)), a semiconductor memory,
or the like.
[0239] Note that, in this specification, steps that write the
program to be recorded in the recording medium do not necessarily
have to be performed in time series in line with the order of the
steps, and instead may include processing that is performed in
parallel or individually.
[0240] It should be understood by those skilled in the art that
various modifications, combinations, sub-combinations and
alterations may occur depending on design requirements and other
factors insofar as they are within the scope of the appended claims
or the equivalents thereof.
[0241] Additionally, the present technology may also be configured
as below.
(1) An image processing device including:
[0242] a motion vector detection portion that performs comparison
of a substantially spherical photographic subject such that, among
a plurality of captured images including the photographic subject,
an image as a processing target and another image as a comparison
target are compared using each of the plurality of captured images
as the processing target, and which detects a motion vector of a
whole three-dimensional spherical model with respect to the
processing target;
[0243] a motion compensation portion that performs motion
compensation on the processing target, based on the motion vector
of each of the plurality of captured images that is detected by the
motion vector detection portion; and
[0244] a synthesis portion that synthesizes each of the captured
images that are obtained as a result of the motion compensation
performed by the motion compensation portion.
(2) The image processing device according to (1),
[0245] wherein with respect to each of a plurality of blocks that
are divided up from the processing target, the motion vector
detection portion detects a local motion vector by performing block
matching with the comparison target, and
[0246] wherein the motion vector detection portion detects the
motion vector of the whole three-dimensional spherical model with
respect to the processing target, using the local motion vector of
each of the plurality of blocks.
(3) The image processing device according to (1) or (2),
[0247] wherein the motion vector detection portion converts the
local motion vector with respect to each of the plurality of blocks
in the processing target into a local spherical motion vector in
the three-dimensional spherical model, and
[0248] wherein the motion vector detection portion detects the
motion vector of the whole three-dimensional spherical model with
respect to the processing target, using the local spherical motion
vector of each of the plurality of blocks.
(4) The image processing device according to (1), (2), or (3)
[0249] wherein the motion vector detection portion converts each of
the plurality of blocks in the processing target into a plurality
of spherical blocks in the three-dimensional spherical model,
[0250] wherein with respect to each of the plurality of spherical
blocks, the motion vector detection portion detects, as the local
motion vector, a local spherical motion vector by performing block
matching with the comparison target, and
[0251] wherein the motion vector detection portion detects the
motion vector of the whole three-dimensional spherical model with
respect to the processing target, using the local spherical motion
vector of each of the plurality of spherical blocks.
(5) The image processing device according to any one of (1) to
(4),
[0252] wherein the motion vector detection portion converts each of
the processing target and the comparison target into a spherical
image in the three-dimensional spherical model, and
[0253] wherein the motion vector detection portion performs
matching between the spherical image of the processing target and
the spherical image of the comparison target, and thereby detects
the motion vector of the whole three-dimensional spherical model
with respect to the processing target.
(6) The image processing device according to any one of (1) to
(5).
[0254] wherein the photographic subject is a fundus oculi.
(7) The image processing device according to any one of (1) to
(6),
[0255] wherein the three-dimensional spherical model is switched
and used in accordance with conditions of the photographic
subject.
(8) An image processing device including:
[0256] a conversion portion that, among a plurality of captured
images that include a substantially spherical photographic subject,
converts an image as a processing target and another image as a
comparison target into spherical images on a three-dimensional
spherical model, using each of the plurality of captured images as
the processing target;
[0257] an extraction portion that extracts features of each of the
spherical image of the processing target and the spherical image of
the comparison target;
[0258] an alignment portion that aligns positions of the features
such that the features match each other; and
[0259] a synthesis portion that synthesizes each of the captured
images that are obtained as a result of the alignment performed by
the alignment portion.
(9) The image processing device according to (8),
[0260] wherein a blood vessel shape is used as the feature.
(10) The image processing device according to (8) or (9),
[0261] wherein the photographic subject is a fundus oculi.
(11) The image processing device according to (8), (9), or
(10),
[0262] wherein the three-dimensional spherical model is switched
and used in accordance with conditions of the photographic
subject.
[0263] The present technology can be applied to an image processing
device.
[0264] The present application is a continuation of U.S. patent
application Ser. No. 13/594,382 filed on Aug. 24, 2012 which claims
priority of the Japanese Patent Application No. JP2011-188277 filed
on Aug. 31, 2011 in the Japan Patent Office, the entire contents of
which are incorporated herein by reference.
* * * * *