U.S. patent application number 10/404659 was filed with the patent office on 2003-10-09 for image processing device for stereo image processing.
This patent application is currently assigned to Exploitation of Next Generation Co., Ltd.. Invention is credited to Arima, Yutaka.
Application Number | 20030190073 10/404659 |
Document ID | / |
Family ID | 28672710 |
Filed Date | 2003-10-09 |
United States Patent
Application |
20030190073 |
Kind Code |
A1 |
Arima, Yutaka |
October 9, 2003 |
Image processing device for stereo image processing
Abstract
In a correlation process between image patterns seen by two
stereo-viewing cameras, the process can be made faster by using
only the information related to a coordinate in the direction
having a parallax, concerning the position of a characteristic
point in the image patterns. In addition, by introducing a process
for verifying the positional information of the characteristic
point obtained by a camera for verification, it is possible to
suppress a decrease in the characteristic information contained in
the image patterns and to improve the precision of the correlation
process. As a result, it is possible to achieve a stereo image
processing device which operates at high speed. Further, since the
stereo image processing device can be realized by a comparatively
simple circuit configuration, it is expected that the device can
contribute to the commercialization of in-vehicle safety monitor
devices and the like.
Inventors: |
Arima, Yutaka; (Iizuka-shi,
JP) |
Correspondence
Address: |
Finnegan, Henderson, Farabow,
Garrett & Dunner, L.L.P.
1300 I Street, N.W.
Washington
DC
20005-3315
US
|
Assignee: |
Exploitation of Next Generation
Co., Ltd.
|
Family ID: |
28672710 |
Appl. No.: |
10/404659 |
Filed: |
April 2, 2003 |
Current U.S.
Class: |
382/154 ;
348/E13.015; 348/E13.016 |
Current CPC
Class: |
G06T 2207/10012
20130101; H04N 13/243 20180501; G06T 7/593 20170101; H04N 13/246
20180501 |
Class at
Publication: |
382/154 |
International
Class: |
G06K 009/00 |
Foreign Application Data
Date |
Code |
Application Number |
Apr 3, 2002 |
JP |
2002-137801 PAT. |
Claims
What is claimed is:
1. An image processing device comprising: a plurality of image
capturing units, disposed in different positions from each other,
outputting image data based on two-dimensional positional
information, in regard to a position in the three-dimensional space
detectable within the field of vision; and a verification process
unit finding the correspondence relation between said positional
information in said plurality of image capturing units
corresponding to the identical position in said field of vision, in
response to said image data received from said plurality of image
capturing units, and outputting the obtained results.
2. The image processing device according to claim 1, wherein said
plurality of image capturing units are disposed in predetermined
positions, respectively, said verification process unit includes a
first memory previously storing as data the correspondence relation
between said positional information in said plurality of image
capturing units for each point indicated by said positional
information, the correspondence relation being predetermined
uniquely according to said predetermined positions, and an address
to select said data outputted from said first memory is set in
accordance with said image data received from said plurality of
image capturing units.
3. The image processing device according to claim 2 further
comprising a distance information storage unit for previously
storing information indicative of the distances from said plurality
of image capturing units, with respect to said each point indicated
by said positional information.
4. The image processing device according to claim 3, wherein said
distance information storage unit is constituted by a second memory
previously storing information indicative of the distances from
said plurality of image capturing units as data, and having the
same input address as said first memory.
5. The image processing device according to claim 1, wherein said
plurality of image capturing units are so disposed as to share one
of said two-dimensional coordinates, and said verification process
unit performs the process by using only a part related to the other
one of said two-dimensional coordinates out of said image data
obtained from each of said plurality of image capturing units.
6. An image processing device comprising: a plurality of first
image capturing units and at least one second image capturing unit,
disposed in different positions from each other, each outputting
two-dimensional image data, in regard to a position in the
three-dimensional space detectable in the field of vision; and
correlation process units determining the presence or absence of a
characteristic point for each point in said field of vision, in
response to said image data received from said plurality of first
image capturing units and said at least one second image capturing
unit, wherein said correlation process units determine whether said
characteristic point commonly detected by said plurality of first
image capturing units is detected at a corresponding
two-dimensional position of said second image capturing unit or
not, and output the obtained results.
7. The image processing device according to claim 6, wherein said
correlation process units include memory unit(s) provided
corresponding to said at least one second image capturing unit,
respectively; each of said memory unit(s) stores as data the
presence or absence of the detection of said characteristic point
in each point in said field of vision by the corresponding second
image capturing unit, and an address to select said data outputted
from said memory units is set in accordance with said image data
received from said plurality of first image capturing units.
8. The image processing device according to claim 6, wherein said
plurality of first image capturing units and said at least one
second image capturing unit are so disposed as to share one of said
two-dimensional coordinates, and said correlation process units
perform the process by using only a part related to the other one
of said two-dimensional coordinates out of said image data obtained
from said plurality of first image capturing units and said at
least one second image capturing unit.
9. The image processing device according to claim 6, wherein a
plurality of said second image capturing units are disposed in
different positions from each other.
10. An image processing device comprising: a plurality of image
capturing units provided in correspondence with a plurality of
different view points, respectively; and means for outputting the
correspondence relation of positional information seen by said
plurality of image capturing units, with respect to the
characteristic points on the respective positions, concerning all
positions in the three-dimensional space detectable in the field of
vision.
11. The image processing device according to claim 10, wherein
memory is used as said means for outputting the correspondence
relation of said positional information.
12. An image processing device comprising: a plurality of first
image capturing units provided in correspondence with at least two
different view points; at least one second image capturing unit
provided in correspondence with at least one different viewpoint;
and means for detecting the presence or absence of a characteristic
point in a position seen by said second image capturing unit, said
position corresponding to positional information of said
characteristic point seen by said plurality of first image
capturing units.
13. The image processing device according to claim 12, wherein
memory is used as said means for detecting the presence or absence
of the characteristic point.
14. The image processing device according to claim 12, wherein a
plurality of said different view points are provided.
Description
BACKGROUND OF THE INVENTION
[0001] 1. Field of the Invention
[0002] The present invention relates to an image processing device
for quickly extracting distance information from stereo images. The
present invention realizes a distance extracting process with a
simple circuit configuration at high speed, thereby providing an
inexpensive initial visual information processing device. The art
of the present invention is expected to be used for industrial
products such as robot visions and in-vehicle safety monitor
devices.
[0003] 2. Description of the Background Art
[0004] A human's initial visual information process involves a
function of extracting distance information obtained by two-eye
stereo viewing in addition to a function of photo-sensing with high
adaptability. In other words, a human's sense of sight has a
function of processing image information to estimate
three-dimensional information including the extent of the space
from two two-dimensional patterns reflected on the retinas of both
eyes. Above all, a function of instantly acquiring the distance to
the target in the field of vision is one of the considerably
important abilities for a human to act safely. The high-speed
estimation of the distance to the target by the stereo viewing is
also a considerably important image information processing
technique to realize industrial products such as robot visions and
in-vehicle safety monitor devices, and their commercialization has
been strongly desired.
[0005] Distance (depth) estimation by stereo images is generally
achieved by a correlation process for specifying the same target
between two images, and a distance calculation process for
calculating the distances from the two different view points on the
basis of parallax information obtained from these view points.
[0006] Conventional stereo image processing devices has required a
large number of time for a correlation process. This is because the
amount of calculation necessary for the correlation process is huge
and also because the device has a limited calculation capability.
The amount of calculation necessary for the conventional stereo
image correlation process is generally in proportion to the product
of the third power of the number of pixels in the direction having
a parallax and the number of pixels in the direction having no
parallax. For example, in the case of image data consisting of
about 100,000 pixels (352.times.288 pixels), the correlation
process requires over 100 billion number of calculations. This
involves the problem that even a processor with 100 GOPS (Giga
Operations Per Second) could offer a process speed which does not
satisfy 8 screens per second. The process speed of this level is
insufficient for cars and other comparatively fast-moving devices
to be put in the actual use. Therefore, it is an object of the
present invention to estimate distance information from stereo
images at higher speed as compared with conventional devices.
SUMMARY OF THE INVENTION
[0007] An object of the present invention is to provide an image
processing device which performs a stereo image process at high
speed.
[0008] An image processing device according to the present
invention includes: a plurality of image capturing units, disposed
in different positions from each other, outputting image data based
on two-dimensional positional information, in regard to a position
in the three-dimensional space detectable within the field of
vision; and a verification process unit finding the correspondence
relation between the positional information in the plurality of
image capturing units corresponding to the identical position in
the field of vision, in response to the image data from the
plurality of image capturing units, and outputting the obtained
results.
[0009] Preferably, the plurality of image capturing units are
disposed in predetermined positions, respectively, the verification
process unit includes a first memory previously storing as data the
correspondence relation between the positional information in the
plurality of image capturing units for each point indicated by the
positional information, the correspondence relation being
predetermined uniquely according to the predetermined positions,
and an address to select the data outputted from the first memory
is set in accordance with the image data received from the
plurality of image capturing units.
[0010] Further preferably, the image processing device further
includes a distance information storage unit for previously storing
information indicative of the distances from the plurality of image
capturing units, with respect to each point indicated by the
positional information.
[0011] The distance information storage unit is constituted by a
second memory previously storing information indicative of the
distances from the plurality of image capturing units as data, and
having the same input address as the first memory.
[0012] Preferably, the plurality of image capturing units are so
disposed as to share one of the two-dimensional coordinates, and
the verification process unit performs the process by using only a
part related to the other one of the two-dimensional coordinates
out of the image data obtained from each of the plurality of image
capturing units.
[0013] Another image processing device of the present invention
according to different configuration includes: a plurality of first
image capturing units and at least one second image capturing unit,
disposed in different positions from each other, each outputting
two-dimensional image data, in regard to a position in the
three-dimensional space detectable in the field of vision; and
correlation process units determining the presence or absence of a
characteristic point for each point in the field of vision, in
response to the image data received from the plurality of first
image capturing units and the at least one second image capturing
unit, the correlation process units determining whether the
characteristic point commonly detected by the plurality of first
image capturing units is detected at a corresponding
two-dimensional position of the second image capturing unit or not,
and outputting the obtained results.
[0014] Preferably, the correlation process units include memory
unit(s) provided corresponding to the at least one second image
capturing unit, respectively, each of the memory unit(s) stores as
data the presence or absence of the detection of the characteristic
point in each point in the field of vision by the corresponding
second image capturing unit, and an address to select the data
outputted from the memory units is set in accordance with the image
data received from the plurality of first image capturing
units.
[0015] Further preferably, the plurality of first image capturing
units and the at least one second image capturing unit are so
disposed as to share one of the two-dimensional coordinates, and
the correlation process units perform the process by using only a
part related to the other one of the two-dimensional coordinates
out of the image data obtained from the plurality of first image
capturing units and the at least one second image capturing
unit.
[0016] Preferably, a plurality of second image capturing units are
disposed in different positions from each other.
[0017] Still another image processing device of the present
invention according to still another configuration includes: a
plurality of image capturing units provided in correspondence with
a plurality of different view points, respectively; and a part for
outputting the correspondence relation of positional information
seen by the plurality of image capturing units, with respect to the
characteristic points on the respective positions, concerning all
positions in the three-dimensional space detectable in the field of
vision.
[0018] Preferably, memory is used as the part for outputting the
correspondence relation of the positional information.
[0019] Yet another image processing device of the present invention
according to yet another configuration includes: a plurality of
first image capturing units provided in correspondence with at
least two different view points, respectively; at least one second
image capturing unit provided in correspondence with at least one
different viewpoint; and a part for detecting the presence or
absence of the characteristic point in a position seen by the
second image capturing unit, the position corresponding to
positional information of the characteristic point seen by the
plurality of first image capturing units.
[0020] Preferably, memory is used as the part for detecting the
presence or absence of the characteristic point. It is also
preferable that a plurality of different view points is
provided.
[0021] Therefore, a main advantage of the present invention is to
be able to perform stereo image processing at high speed because a
correlation process for the specific points (characteristic points)
between image patterns obtained in a plurality of image capturing
units can be performed by using positional information in the
respective image capturing units.
[0022] The stereo image processing can be made faster by performing
a correlation process by using only the information related to the
coordinate in the direction having a parallax between the plurality
of image capturing units.
[0023] Recognizing the same characteristic points by using another
image capturing unit for verification can realize a correlation
process with high precision.
[0024] The foregoing and other objects, features, aspects and
advantages of the present invention will become more apparent from
the following detailed description of the present invention when
taken in conjunction with the accompanying drawings.
BRIEF DESCRIPTION OF THE DRAWINGS
[0025] FIG. 1 is a block diagram showing the configuration example
of a stereo image processing device of according to a first
embodiment of the present invention;
[0026] FIG. 2 is a conceptual diagram for describing the principle
of a correlation-verification process in the image processing
device according to the present invention; and
[0027] FIG. 3 is a block diagram showing the configuration example
of a stereo image processing device according to a second
embodiment of the present invention.
DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0028] The present invention proposes a circuit configuration for
estimating distance information from stereo images at higher speed
as compared with the conventional level. The present invention has
a feature of performing a process by using only positional (the
coordinate in the direction having a parallax) information of
characteristic points in image patterns. Consequently, the amount
of information to be processed in order to extract the correlation
between two images constituting stereo images can be generally
lessened as compared with the image information (the number of
pixels in the direction having a parallax). As a result, the
correlation process can be performed at high speed.
[0029] However, when image patterns are converted into the
positional information of characteristic points (or information as
to whether each pixel has a characteristic or not), characteristic
information contained in the patterns is considerably decreased.
This brings about the problem of making it impossible to specify a
correct correlation by the conventional stereo image correlation
processing method.
[0030] Therefore, the present invention solves this problem by
introducing a process of verifying the positional information of
image characteristic points by using a camera for verification. The
stereo image processing device according to the present invention
can achieve higher-speed processing as compared with the
conventional devices and also can be constituted by a simple
circuit, so that it is expected to contribute to the
commercialization of in-vehicle safety monitor device and the
like.
[0031] Hereinafter, embodiments of the present invention will be
described in detail with reference to the drawings.
[0032] First Embodiment
[0033] With reference to FIG. 1, a stereo image processing device
100 according to a first embodiment of the present invention
includes: a sequencer 107 for allowing an image process to perform
in the predetermined procedure; a memory 108 for storing a map for
verification (hereinafter, referred as "verification map memory");
a memory 109 for storing a map for distance (hereinafter, referred
as "distance map memory"); two cameras 117 for stereo viewing
(hereinafter, referred to as "stereo-viewing cameras"); and a
camera 118 for verification (hereinafter, referred to as a
"verifying camera"). Reference numerals 101 and 102 in FIG. 1
denote X-coordinate data and Y-coordinate data, respectively.
[0034] Each of stereo-viewing cameras 117 are provided with a
memory 110 for storing coordinate data (hereinafter, referred as
"coordinate data memory"), a coordinate conversion unit 113, and an
image processing unit 115. On the other hand, verifying camera 118
is provided with a memory 111 for storing a map indicative of the
presence or absence of characteristic points (hereinafter, referred
to as "characteristic point map memory"), and an image processing
unit 115 for extracting characteristic points.
[0035] Verification map memory 108, distance map memory 109,
coordinate data memories 110, and characteristic point map memory
111 are all designed to output data 106 corresponding to an
inputted address 105, respectively.
[0036] Two stereo-viewing cameras 117 and verifying camera 118
provided as image capturing units each generate image data 116
according to the image in the field of vision. Stereo-viewing
cameras 117 and verifying camera 118 can be cameras of the same
kind. Image processing units 115 each determine the presence or
absence of characteristic points for each pixel from the image data
outputted from the corresponding cameras 117 or 118 and generate
data 114 indicative of the presence or absence of characteristic
points showing the results of the determination. Thus, data 114
indicative of the presence or absence of characteristic points is
generated for each pixel. Image processing units 115 respectively
provided to stereo-viewing cameras 117 and verifying camera 118 can
have the same configuration.
[0037] Coordinate conversion units 113 each convert data 114
indicative of the presence or absence of characteristic points
obtained by the process for extracting characteristic points by
image processing units 115 into coordinate data in the direction
having a parallax, and output the converted data as coordinate data
112.
[0038] In the stereo image processing device according to the
present invention, there is no limitation on the configuration of
image processing units 115 which extract characteristic points from
image patterns and coordinate conversion units 113 which perform
coordinate conversion from data 114 indicative of the presence or
absence of characteristic points into coordinate data 112. For
example, it is possible to process edges or corners related to
luminance in the extraction of the presence or absence
characteristic points. Examples of the process for coordinate
conversion include a method of writing coordinate data and data
indicative of the presence or absence of characteristic points to
memory as write-enable signals, and the like.
[0039] As will be described below, in the stereo image processing
according to the present invention, a correlation process is
unnecessary for the coordinate in the direction having no parallax,
so that the correlation process for the coordinate having a
parallax is repeated for each coordinate having no parallax. For
example, in the case where three cameras 117 and 118 shown in FIG.
1 are arranged horizontally, that is, on the same Y coordinate, the
correlation process for X-coordinate (horizontal direction) is
repeated for each Y coordinate (vertical direction). Therefore,
description will be given of the procedure relating to the
correlation process in the direction of X coordinate in a single Y
coordinate. Actually, the procedure described below is performed
for all Y coordinates so as to complete a correlation process about
the whole image. Assume that in the description of the present
embodiment, stereo-viewing cameras 117 and verifying camera 118 are
arranged horizontally and have the same Y coordinate.
[0040] The X coordinate data of the characteristic points obtained
from two stereo-viewing cameras 117 is temporarily stored in the
corresponding coordinate data memories 110. Data 114 indicative of
the presence or absence of characteristic points obtained from
verifying camera 118 is temporarily stored in the characteristic
point map memory 111. Data 114 indicative of the presence or
absence of characteristic points is represented as "1" when it is a
characteristic point and as "0" when it is not a characteristic
point. The addresses in characteristic point map memory 111 are
made to correspond to the X coordinates.
[0041] After the data from cameras 117 and 118 are stored in the
corresponding coordinate data memories 110 and characteristic point
map memory 111, a process for specifying the correlation between
the characteristic points is performed by using the
characteristic-point coordinate stored in two coordinate data
memories 110. Thus, the same characteristic point is specified.
[0042] In the stereo image processing according to the present
invention, in order to specify correlation, all correlation
combinations of characteristic points are verified to check whether
they are correct or not. The data used for the verification
corresponds to the data for verification stored in characteristic
point map memory 111.
[0043] Thus, in coordinate data memories 110 and characteristic
point map memory 111, it is necessary to perform the storage of
data related to the characteristic points and a
correlation-verification process. By forming each of coordinate
data memories 110 and characteristic point map memory 111 into a
two-bank structure, it is possible to process these operations
concurrently by pipeline system, thereby increasing the speed of
the stereo image process as a whole.
[0044] Next, description will be given of the principle of the
correlation-verification process with reference to FIG. 2.
[0045] Assume herein that two stereo-viewing cameras 117 see
characteristic points 130 to 134 indicated by five black circles in
the field of vision. This means that each of stereo-viewing cameras
117 sees characteristic points 130 to 134 at the positions (X
coordinate) where straight lines connecting the respective
characteristic points 130 to 134 and the lens focus of each camera
reach.
[0046] Consequently, the intersections (25 points) of the straight
lines connecting five characteristic points 130 to 134 and the lens
focus points of two stereo-viewing cameras 117 are recognized as
the points where characteristic points can exist. However, it is
impossible to specify the sites where the characteristic points
actually exist (at least 5 sites) out of the 25 intersections
because characteristic points 130 to 134 cannot be distinguished
only by the images obtained from these two stereo-viewing cameras
117.
[0047] That is why the stereo image processing device according to
the present invention employs verifying camera 118. Verifying
camera 118 shown in FIG. 2 is supposed to see characteristic points
130 to 134 at the positions (X coordinate) where straight lines
connecting the respective characteristic points and the lens focus
of each camera reach. As a result, the presence or absence of the
characteristic points can be determined by checking whether the
characteristic points are visible in the X coordinates seen by
verifying camera 118 which correspond to the respective
intersections of the straight lines coming from the two
stereo-viewing cameras 117.
[0048] The verification process according to the present invention,
as shown in FIG. 1, makes use of verification map memory 108 in
which previously-prepared information related to the map for
verification correspondence is stored as data for verification. The
verification map contains information indicative of the
correspondence relation between the combinations of the points on X
coordinates seen by two stereo-viewing cameras 117 and X
coordinates seen by verifying camera 118 corresponding to the
points in the space as the target. The correspondence relation,
that is, the data for verification can be uniquely set in advance
by predetermining the location to install cameras 117 and 118.
[0049] In other words, verification map memory 108 is so prepared
in advance that if two X coordinates obtained when a certain
characteristic point is seen by two stereo-viewing cameras 117 are
inputted as address 105, then the X coordinates to be seen by
verifying camera 118 corresponding to the two X coordinates are
outputted as data 106.
[0050] In the process for verification and determination, two X
coordinates are accessed as address 105 to verification map memory
108, and data 106 outputted therefrom is used as address 105 of
characteristic point map memory 111. In response to this, data 106
outputted from characteristic point map memory 111 as an effective
signal 104 according to which the presence or absence of
characteristic points is determined.
[0051] For example, when characteristic point map memory 111
outputs data 106 having the value "1" while using the output data
of verification map memory 108 as an input address, it is
determined that there is a characteristic point. On the other hand,
when outputted data 106 is "0", it is determined that there is no
characteristic point. This verification process is repeated for all
the combinations of coordinate data stored in two coordinate data
memories 110.
[0052] As shown in FIG. 1, provided that the data for distance
between each of stereo-viewing cameras 117 and the target is
previously stored in distance map memory 109 in correspondence with
the respective combinations of X coordinates seen by two
stereo-viewing cameras 117, and also provided that verification map
memory 108 and distance map memory 109 have the same address 105,
when the correlation is determined to be correct, data 106
outputted from distance map memory 109 be treated as the data 103
for distance indicative of the estimated distance of the
characteristic point. This facilitates the estimation of distance
of characteristic points.
[0053] Sequencer 107 shown in FIG. 1 performs address control at
the time of data storage to coordinate data memories 110 and
characteristic point map memory 111, and also performs address
control of coordinate data memories 110 at the time of correlation
verification. Sequencer 107 can calculate the number of the
repeated processes related to X coordinates and output the value as
Y coordinate data 101.
[0054] Second Embodiment
[0055] The configuration according to the first embodiment shown in
FIG. 1 has the problem that a point 135 indicated by a white circle
in FIG. 2 is determined to have a characteristic point, regardless
of the actual presence or absence of the characteristic point
there. This is because verifying camera 118 sees point 135 and
characteristic point 134 as if they were overlapped and on the same
X coordinate. The problem of this kind is likely to happen when the
number of the characteristic points becomes large in the field of
vision.
[0056] In a second embodiment, description will be given of a
stereo image processing device which avoids such an inconvenience
by increasing the number of verifying cameras 118.
[0057] With reference to FIG. 3, a stereo image processing device
100# according to the second embodiment differs from stereo image
processing device 100 according to the first embodiment in that a
plurality of verifying cameras 118 are provided. Similar to the
case shown in FIG. 1, image processing unit 115 for extracting
characteristic points and characteristic point map memory 111 are
further provided corresponding to the newly provided verifying
cameras 118.
[0058] A logical circuit 120 is further provided to perform AND
logical operation between the signals outputted from characteristic
point map memories 111, which respectively correspond to the
verifying cameras 118. The output signal of logical circuit 120 is
outputted as effective signal 104.
[0059] This configuration makes it possible to secure the
determination between the presence and absence of a characteristic
point by the plurality of verifying cameras 118 even when the
number of characteristic points becomes large in the field of
vision.
[0060] As described above, the configuration according to the
present invention can achieve a device for estimating distance
information at high speed on the basis of the stereo images only by
using some memories and a simple logical circuit. For example, in
the case of image data containing about 100,000 pixels
(352.times.288 pixels), assuming that the characteristic points in
the image has an incidence of 10%, about 360,000 memory accesses
can be performed for the correlation process so as to estimate the
distances of all the characteristic points. As an example, when a
memory with access time of 20 ms is used, processing one stereo
image requires about 7 ms, so 140 screens can be processed per
second. If the incidence of characteristic points is 5%, then 560
screens can be processed per second. In that case, memories 110 for
storing coordinate data require a capacity of about 320 bytes;
memories 111 for storing a map indicative of the presence or
absence of characteristic points require a capacity of about 352
bytes; and each of verification map memory 108 and distance map
memory 109 requires about 1.2 mega bytes.
[0061] Although the present invention has been described and
illustrated in detail, it is clearly understood that the same is by
way of illustration and example only and is not to be taken by way
of limitation, the spirit and scope of the present invention being
limited only by the terms of the appended claims.
* * * * *