U.S. patent application number 11/409474 was filed with the patent office on 2006-09-14 for image processor and monitoring system.
This patent application is currently assigned to Matsushita Electric Industrial Co., Ltd.. Invention is credited to Hirofumi Ishii, Masamichi Nakagawa, Kunio Nobori, Shusaku Okamoto.
Application Number | 20060203092 11/409474 |
Document ID | / |
Family ID | 18639321 |
Filed Date | 2006-09-14 |
United States Patent
Application |
20060203092 |
Kind Code |
A1 |
Nobori; Kunio ; et
al. |
September 14, 2006 |
Image processor and monitoring system
Abstract
The image processor of the present invention generates a more
natural synthesized image from camera images taken with a plurality
of cameras capturing the surroundings of a vehicle. A parameter
storage section stores a plurality of image synthesis parameter
groups representing the correspondence between the camera images
and the synthesized image and having different spatial or temporal
resolution relations. A parameter selection section selects the
image synthesis parameter group according to the output of a
vehicle motion detection section for detecting the motion of the
vehicle such as the driving speed and direction. An image synthesis
section generates the synthesized image from the camera images
according to the selected image synthesis parameter group.
Inventors: |
Nobori; Kunio; (Osaka,
JP) ; Nakagawa; Masamichi; (Osaka, JP) ;
Ishii; Hirofumi; (Kanagawa, JP) ; Okamoto;
Shusaku; (Osaka, JP) |
Correspondence
Address: |
GREGORY A. STOBBS
5445 CORPORATE DRIVE
SUITE 400
TROY
MI
48098
US
|
Assignee: |
Matsushita Electric Industrial Co.,
Ltd.
Osaka
JP
|
Family ID: |
18639321 |
Appl. No.: |
11/409474 |
Filed: |
April 21, 2006 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
10800941 |
Mar 15, 2004 |
|
|
|
11409474 |
Apr 21, 2006 |
|
|
|
09843962 |
Apr 27, 2001 |
6734896 |
|
|
10800941 |
Mar 15, 2004 |
|
|
|
Current U.S.
Class: |
348/148 ;
348/E7.086; 382/104 |
Current CPC
Class: |
B60R 1/00 20130101; B60R
2300/607 20130101; B60R 2300/8093 20130101; G06T 7/97 20170101;
G06T 5/50 20130101; G06T 3/4038 20130101; B60R 2300/302 20130101;
B60R 2300/105 20130101; G06T 2207/20228 20130101; H04N 7/181
20130101; G06T 11/00 20130101 |
Class at
Publication: |
348/148 ;
382/104 |
International
Class: |
G06K 9/00 20060101
G06K009/00; H04N 7/18 20060101 H04N007/18 |
Foreign Application Data
Date |
Code |
Application Number |
Apr 28, 2000 |
JP |
2000-130171 |
Claims
1. An image processor for receiving camera images taken with a
plurality of cameras capturing the surroundings of a vehicle and
generating a synthesized image from the camera images, the image
processor comprising: a parameter generation section constructed to
be able to generate a plurality of image synthesis parameter groups
each representing the correspondence between the camera images and
the synthesized image and having different spatial or temporal
resolution relations, wherein the synthesized image is generated
from the camera images according to the image synthesis parameter
group generated by the parameter generation section, and the
parameter generation section switches the image synthesis parameter
group to be generated according to an output of a vehicle motion
detection section for detecting the motion of the vehicle.
2. The image processor of claim 1, wherein the parameter generation
section comprises: a parameter storage section for storing the
plurality of image synthesis parameter groups, and a parameter
selection section for selecting at least one among the plurality of
image synthesis parameter groups stored in the parameter storage
section according to the output of the vehicle motion detection
section.
3. The image processor of claim 1, wherein the camera images are
interlaced images, and the plurality of image synthesis parameter
groups include at least a frame-base image synthesis parameter
group and a field-base image synthesis parameter group.
4. The image processor of claim 3, wherein the parameter generation
section generates the field-base image synthesis parameter group
when the motion of the vehicle detected by the vehicle motion
detection section is relatively fast, and generates the frame-base
image synthesis parameter group when the motion of the vehicle is
relatively slow.
5. the image processor of claim 1, wherein the vehicle motion
detection section detects the motion of the vehicle from the camera
images.
6. The image processor of claim 1, wherein the plurality of cameras
are constructed to be able to switch a capture pattern according to
an input switch signal, and the parameter generation section sends
the switch signal to the cameras, together with generating the
image synthesis parameter group, according to the output of the
vehicle motion detection section, to switch the capture patterns of
the cameras.
7. The image processor of claim 6, wherein the parameter generation
section selects the image synthesis parameter group and switches
the capture patterns of the cameras according to an output of a
vehicle status detection section for detecting brightness of the
surroundings of the vehicle, in addition to the output of the
vehicle motion detection section.
8. The image processor of claim 1, wherein the parameter generation
section generates the image synthesis parameter group according to
an output of a vehicle status detection section for detecting the
vehicle status such as an operation by a driver of the vehicle or
whether or not an obstacle exists in the surroundings of the
vehicle, in addition to the output of the vehicle motion detection
section.
9. A monitoring system comprising: a plurality of cameras for
capturing the surroundings of a vehicle; an image processing
section for receiving camera images from the plurality of cameras
and generating a synthesized image from the camera images; and a
display section for displaying the synthesized image generated by
the image processing section, wherein the image processing section
comprises a parameter generation section constructed to be able to
generate a plurality of image synthesis parameter groups each
representing the correspondence between the camera images and the
synthesized image and having different spatial or temporal
resolution relations, the synthesized image is generated from the
camera images according to the image synthesis parameter group
generated by the parameter generation section, and the parameter
generation section generates at least one of the plurality of image
synthesis parameter groups according to an output of a vehicle
motion detection section for detecting the motion of the
vehicle.
10. An image processor for receiving camera images taken with a
plurality of cameras capturing the surroundings of a vehicle and
generating a synthesized image from the camera images, the image
processor comprising: a parameter storage section for storing a
plurality of sets of an image synthesis parameter group
representing the correspondence between the camera images and the
synthesized image and a filter parameter group corresponding to the
image synthesis parameter group; a parameter selection section for
selecting at least one among the plurality of sets of the image
synthesis parameter group and the filter parameter group stored by
the parameter storage section according to an output of a vehicle
motion detection section for detecting the motion of the vehicle
and an output of a vehicle status detection section for detecting
the status of the vehicle such as an operation by a driver of the
vehicle or whether or not an obstacle exists in the surroundings of
the vehicle; and a filtering section for performing frequency band
limitation filtering for the camera images according to the filter
parameter group of the set selected by the parameter selection
section, wherein the synthesized image is generated from the camera
images filtered by the filtering section according to the image
synthesis parameter group of the set selected by the parameter
selection section.
11. The image processor of claim 10, wherein the filter parameter
group includes filtering setting data for each pixel position of
the camera image.
12. A monitoring system comprising: a plurality of cameras for
capturing the surroundings of a vehicle; an image processing
section for receiving camera images from the plurality of cameras
and generating a synthesized image from the camera images; and a
display section for displaying the synthesized image generated by
the image processing section, wherein the image processing section
comprises: a parameter storage section for storing a plurality of
sets of an image synthesis parameter group representing the
correspondence between the camera images and the synthesized image
and a filter parameter group corresponding to the image synthesis
parameter group; a parameter selection section for selecting at
least one among the plurality of sets of the image synthesis
parameter group and the filter parameter group stored by the
parameter storage section according to an output of a vehicle
motion detection section for detecting the motion of the vehicle
and an output of a vehicle status detection section for detecting
the status of the vehicle such as an operation by a driver of the
vehicle or whether or not an obstacle exists in the surroundings of
the vehicle; and a filtering section for performing frequency band
limitation filtering for the camera images according to the filter
parameter group of the set selected by the parameter selection
section, wherein the synthesized image is generated from the camera
images filtered by the filtering section according to the image
synthesis parameter group of the set selected by the parameter
selection section.
13. An image processor for receiving camera images taken with a
plurality of cameras capturing the surroundings of a vehicle and
generating a synthesized image from the camera images, the image
processor comprising: a brightness correction parameter calculation
section for calculating brightness correction parameters for
correcting the brightness and tint of the camera images; and a
brightness correction section for correcting the brightness and
tint of the camera images using the brightness correction
parameters calculated by the brightness correction parameter
calculation section, wherein the synthesized image is generated
from the plurality of camera images subjected to brightness
correction by the brightness correction section according to an
image synthesis parameter group representing the correspondence
between the camera images and the synthesized image, the image
synthesis parameter group includes overlap area data for an overlap
area on the synthesized image in which coverages of the plurality
of cameras overlap, the overlap area data indicating coordinates of
pixels in camera images corresponding to the overlap area, and the
brightness correction parameter calculation section receives the
overlap area data and calculates the brightness correction
parameters using brightness and tint data for the pixels in the
camera images corresponding to the overlap area indicated by the
overlap area data.
14. The image processor of claim 13, wherein the brightness
correction parameter calculation section performs statistical
processing on the brightness in the overlap area for the camera
images corresponding to the overlap area, and calculates the
brightness correction parameters based on the processing
results.
15. The image processor of claim 13, wherein, when a plurality of
overlap areas exist, the brightness correction parameter
calculation section sets priorities to the overlap areas to be
considered during the calculation of the brightness correction
parameters according to an output of a vehicle motion detection
section for detecting the motion of the vehicle.
16. The image processor of claim 13, wherein the brightness
correction section is incorporated in the cameras.
17. A monitoring system comprising: a plurality of cameras for
capturing the surroundings of a vehicle; an image processing
section for receiving camera images from the plurality of cameras
and generating a synthesized image from the camera images; and a
display section for displaying the synthesized image generated by
the image processing section, wherein the image processing section
comprises: a brightness correction parameter calculation section
for calculating brightness correction parameters for correcting the
brightness and tint of the camera images; and a brightness
correction section for correcting the brightness and tint of the
camera images using the brightness correction parameters calculated
by the brightness correction parameter calculation section, wherein
the synthesized image is generated from the plurality of camera
images subjected to brightness correction by the brightness
correction section according to an image synthesis parameter group
representing the correspondence between the camera images and the
synthesized image, the image synthesis parameter group includes
overlap area data for an overlap area on the synthesized image in
which coverages of the plurality of cameras overlap, the overlap
area data indicating coordinates of pixels in camera images
corresponding to the overlap area, and the brightness correction
parameter calculation section receives the overlap area data and
calculates the brightness correction parameters using brightness
and tint data for the pixels in the camera images corresponding to
the overlap area indicated by the overlap area data.
18. An image processor for receiving camera images taken with a
plurality of cameras capturing the surroundings of a vehicle and
generating a synthesized image from the camera images, wherein, in
an overlap area in which coverages of a plurality of cameras
overlap on the synthesized image, a camera image used for
generation of the synthesized image is selected among camera images
from the plurality of cameras according to an output of a vehicle
motion detection section for detecting the motion of the vehicle or
an output of a vehicle status detection section for detecting the
status of the vehicle such as an operation by a driver of the
vehicle or whether or not an obstacle exists in the surroundings of
the vehicle.
19. An image processor for receiving camera images taken with a
plurality of cameras capturing the surroundings of a vehicle and
generating a synthesized image from the camera images, wherein, in
an overlap area in which coverages of a plurality of cameras
overlap on the synthesized image, weights to camera images from the
plurality of cameras are set according to an output of a vehicle
motion detection section for detecting the motion of the vehicle or
an output of a vehicle status detection section for detecting the
status of the vehicle such as an operation by a driver of the
vehicle or whether or not an obstacle exists in the surroundings of
the vehicle.
20. The image processor of claim 19, wherein the image processor
includes: an image synthesis parameter group associating pixels in
the synthesized image with pixels in the camera images; and a table
representing the correspondence between a weight reference number
and a combination of sets of weighting information, wherein a
portion of the image synthesis parameter group corresponding to the
overlap area holds any of the weight reference numbers shown in the
table.
21. A monitoring system comprising: a plurality of cameras for
capturing the surroundings of a vehicle; an image processing
section for receiving camera images from the plurality of cameras
and generating a synthesized image from the camera images; and a
display section for displaying the synthesized image generated by
the image processing section, wherein, in an overlap area in which
coverages of a plurality of cameras overlap on the synthesized
image, a camera image used for generation of the synthesized image
is selected among camera images from the plurality of cameras
according to an output of a vehicle motion detection section for
detecting the motion of the vehicle or an output of a vehicle
status detection section for detecting the status of the vehicle
such as an operation by a driver of the vehicle or whether or not
an obstacle exists in the surroundings of the vehicle.
22. A monitoring system comprising: a plurality of cameras for
capturing the surroundings of a vehicle; an image processing
section for receiving camera images from the plurality of cameras
and generating a synthesized image from the camera images; and a
display section for displaying the synthesized image generated by
the image processing section, wherein, in an overlap area in which
coverages of a plurality of cameras overlap on the synthesized
image, weights to camera images from the plurality of cameras are
set according to an output of a vehicle motion detection section
for detecting the motion of the vehicle or an output of a vehicle
status detection section for detecting the status of the vehicle
such as an operation by a driver of the vehicle or whether or not
an obstacle exists in the surroundings of the vehicle.
23. An image processor for receiving camera images taken with a
plurality of cameras capturing the surroundings of a vehicle and
generating a synthesized image from the camera images, wherein, the
image processor includes an image synthesis parameter group
associating pixels in the synthesized image with pixels in the
camera images, and in an overlap area in which coverages of the
plurality of cameras overlap on the synthesized image, the image
synthesis parameter group includes weights set to camera images
from the plurality of cameras represented by a dithering method.
Description
BACKGROUND OF THE INVENTION
[0001] The present invention relates to an image processing
technique for generating a synthesized image from images taken with
a plurality of cameras. In particular, the present invention
relates to a technique that can be effectively applied to
monitoring systems used as an assist in ensuring safety driving of
a vehicle and the like.
[0002] In recent years, with widespread use of car-mounted displays
and cost reduction in video equipment such as cameras, devices for
monitoring the surroundings of a vehicle with cameras to assist
safety driving have been commercialized and are now becoming
popular.
[0003] As an example of such conventional vehicle surroundings
monitoring devices, the following construction is known. That is, a
camera is placed at a position on a vehicle where an image of a
blind area for the driver, such as the rear of the vehicle, can be
taken with the camera, and the image taken (camera image) is shown
on a display for monitoring.
[0004] For example, Japanese Laid-Open Patent Publication No.
58-110334 discloses the following construction. A plurality of
cameras are placed at different positions on a vehicle. A plurality
of camera images taken with the cameras are modified and
synthesized, to generate a synthesized image as is viewed from the
above of the vehicle with the vehicle in the center and display the
synthesized image on an in-car display. Using this vehicle
surroundings monitoring device, the driver sitting on the driver's
seat can monitor on the display a blind area of which direct view
is not possible. This contributes to avoiding an accident that
would otherwise have occurred, and also facilitating the
driving.
PROBLEMS TO BE SOLVED
[0005] However, the inventors of the present invention have found
from experiments and examinations that the conventional
construction has the following problems.
[0006] FIGS. 22A to 22C illustrate examples of image synthesis
performed when a vehicle is moving. In the case of using cameras
for interlaced scanning, an input image as shown in FIG. 22A is
obtained when the vehicle is moving. That is, every other
horizontal line is displaced resulting in comb-shaped appearance.
This input image is not particularly strange as long as it is
displayed as it is on a display for interlaced scanning. However,
when this input image is modified for synthesis, the comb-shaped
displacement is also modified during the image modification as
shown in FIG. 22B, and this causes a lag between the image-taking
timing and the displaying timing. The resultant image makes the
observer feel strange. That is, so-called interlace noise appears
on the synthesized image more significantly.
[0007] The unnaturalness as described above does not appear when a
one-field synthesized image is generated from one-field input
images. However, the resolution of an input image is low in this
per-field synthesis, compared with per-frame synthesis. Therefore,
when an image of two fields is observed as a frame image as shown
in FIG. 22C, comb-shaped displacement occurs due to an error in
quantization of coordinate values even when the image is still.
This deteriorates the image quality, especially in the area of a
synthesized image where an input image has been enlarged, when the
vehicle is standing still.
[0008] That is, as the first problem, when a one-frame synthesized
image is generated from one-frame (two-field) input images, the
synthesized image is unnatural when the vehicle is moving and when
a moving image is taken. When a one-field synthesized image is
generated from one-field input images, the resolution of the
synthesized image is low.
[0009] As the second problem, in modification of a camera image,
the input image may be contracted depending on the composition of a
synthesized image. As a result, aliasing distortion may partially
be generated in the synthesized image.
[0010] As the third problem, the junctures of input images on a
synthesized image may be unnatural. For example, because white
balancing and the like are adjusted separately for the respective
cameras, the brightness and the tint may fail to match at the
junctures as shown in FIG. 23A. Also, in some image synthesis
methods, the junctures of images may be displaced due to
calculation error, erroneous detection of the position of an
object, and the like, as shown in FIG. 23B.
SUMMARY OF THE INVENTION
[0011] An object of the present invention is providing an image
processor for generating a synthesized image from a plurality of
camera images, which can generate a more natural synthesized image
compared with the conventional processors.
[0012] Specifically, the image processor of the present invention,
which receives camera images taken with a plurality of cameras
capturing the surroundings of a vehicle and generates a synthesized
image from the camera images, includes a parameter generation
section constructed to be able to generate a plurality of image
synthesis parameter groups each representing the correspondence
between the camera images and the synthesized image and having
different spatial or temporal resolution relations, wherein the
synthesized image is generated from the camera images according to
the image synthesis parameter group generated by the parameter
generation section, and the parameter generation section switches
the image synthesis parameter group to be generated according to an
output of a vehicle motion detection section for detecting the
motion of the vehicle.
[0013] According to the invention described above, it is possible
to switch the image synthesis parameter group used for generation
of a synthesized image according to the motion of the vehicle. This
makes it possible to reduce the unnaturalness of the synthesized
image observed when the vehicle is moving and when an image of a
moving object is taken, and also prevent the reduction in
resolution observed when the vehicle is standing still.
[0014] In the image processor of the present invention, the
parameter generation section preferably includes: a parameter
storage section for storing the plurality of image synthesis
parameter groups; and a parameter selection section for selecting
at least one among the plurality of image synthesis parameter
groups stored in the parameter storage section according to the
output of the vehicle motion detection section.
[0015] In the image processor of the present invention, preferably,
the camera images are interlaced images, and the plurality of image
synthesis parameter groups include at least a frame-base image
synthesis parameter group and a field-base image synthesis
parameter group. Preferably, the parameter generation section
generates the field-base image synthesis parameter group when the
motion of the vehicle detected by the vehicle motion detection
section is relatively fast, and generates the frame-base image
synthesis parameter group when the motion of the vehicle is
relatively slow.
[0016] In the image processor of the present invention, the vehicle
motion detection section preferably detects the motion of the
vehicle from the camera images.
[0017] In the image processor of the present invention, preferably,
the plurality of cameras are constructed to be able to switch a
capture pattern according to an input switch signal, and the
parameter generation section sends the switch signal to the
cameras, together with generating the image synthesis parameter
group, according to the output of the vehicle motion detection
section, to switch the capture patterns of the cameras. Preferably,
the parameter generation section selects the image synthesis
parameter group and switches the capture patterns of the cameras
according to an output of a vehicle status detection section for
detecting brightness of the surroundings of the vehicle, in
addition to the output of the vehicle motion detection section.
[0018] In the image processor of the present invention, preferably,
the parameter generation section generates the image synthesis
parameter group according to an output of a vehicle status
detection section for detecting the vehicle status such as an
operation by a driver of the vehicle or whether or not an obstacle
exists in the surroundings of the vehicle, in addition to the
output of the vehicle motion detection section.
[0019] The monitoring system of the present invention includes the
image processor described above as an image processing section.
[0020] Alternatively, the image processor of the present invention,
which receives camera images taken with a plurality of cameras
capturing the surroundings of a vehicle and generates a synthesized
image from the camera images, includes: a parameter storage section
for storing a plurality of sets of an image synthesis parameter
group representing the correspondence between the camera images and
the synthesized image and a filter parameter group corresponding to
the image synthesis parameter group; a parameter selection section
for selecting at least one among the plurality of sets of the image
synthesis parameter group and the filter parameter group stored by
the parameter storage section according to an output of a vehicle
motion detection section for detecting the motion of the vehicle
and an output of a vehicle status detection section for detecting
the status of the vehicle such as an operation by a driver of the
vehicle or whether or not an obstacle exists in the surroundings of
the vehicle; and a filtering section for performing frequency band
limitation filtering for the camera images according to the filter
parameter group of the set selected by the parameter selection
section, wherein the synthesized image is generated from the camera
images filtered by the filtering section according to the image
synthesis parameter group of the set selected by the parameter
selection section.
[0021] According to the invention described above, the filter
parameter group is selected according to the motion and status of
the vehicle, and the camera images are subjected to frequency band
limitation filtering according to the selected filter parameter
group. This effectively suppresses the aliasing distortion in the
synthesized image.
[0022] In the image processor of the present invention described
above, the filter parameter group preferably includes filtering
setting data for each pixel position of the camera image.
[0023] The monitoring system of the present invention includes the
image processor described above as an image processing section.
[0024] Alternatively, the image processor of the present invention,
which receives camera images taken with a plurality of cameras
capturing the surroundings of a vehicle and generates a synthesized
image from the camera images, includes: a brightness correction
parameter calculation section for calculating brightness correction
parameters for correcting the brightness and tint of the camera
images; and a brightness correction section for correcting the
brightness and tint of the camera images using the brightness
correction parameters calculated by the brightness correction
parameter calculation section, wherein the synthesized image is
generated from the plurality of camera images subjected to
brightness-correction by the brightness correction section
according to an image synthesis parameter group representing the
correspondence between the camera images and the synthesized image,
the image synthesis parameter group includes overlap area data for
an overlap area on the synthesized image in which coverages of the
plurality of cameras overlap, the overlap area data indicating
coordinates of pixels in camera images corresponding to the overlap
area, and the brightness correction parameter calculation section
receives the overlap area data and calculates the brightness
correction parameters using brightness and tint data for the pixels
in the camera images corresponding to the overlap area indicated by
the overlap area data.
[0025] According to the invention described above, brightness
correction parameters are calculated using information on the
brightness and tint of the camera images corresponding to the
overlap area on the synthesized image, and brightness correction is
performed using the brightness correction parameters. This reduces
the unnaturalness of the juncture on the synthesized image.
[0026] In the image processor of the present invention described
above, preferably, the brightness correction parameter calculation
section performs statistical processing on the brightness in the
overlap area for the camera images corresponding to the overlap
area, and calculates the brightness correction parameters based on
the processing results.
[0027] In the image processor of the present invention described
above, when a plurality of overlap areas exist, the brightness
correction parameter calculation section preferably sets priorities
to the overlap areas to be considered during the calculation of the
brightness correction parameters according to an output of a
vehicle motion detection section for detecting the motion of the
vehicle.
[0028] In the image processor of the present invention described
above, the brightness correction section is preferably incorporated
in the cameras.
[0029] The monitoring system of the present invention includes the
image processor described above as an image processing section.
[0030] Alternatively, the image processor of the present invention
receives camera images taken with a plurality of cameras capturing
the surroundings of a vehicle and generates a synthesized image
from the camera images, wherein, in an overlap area in which
coverages of a plurality of cameras overlap on the synthesized
image, a camera image used for generation of the synthesized image
is selected among camera images from the plurality of cameras
according to an output of a vehicle motion detection section for
detecting the motion of the vehicle or an output of a vehicle
status detection section for detecting the status of the vehicle
such as an operation by a driver of the vehicle or whether or not
an obstacle exists in the surroundings of the vehicle.
[0031] Alternatively, the image processor of the present invention
receives camera images taken with a plurality of cameras capturing
the surroundings of a vehicle and generates a synthesized image
from the camera images, wherein, in an overlap area in which
coverages of a plurality of cameras overlap on the synthesized
image, weights to camera images from the plurality of cameras are
set according to an output of a vehicle motion detection section
for detecting the motion of the vehicle or an output of a vehicle
status detection section for detecting the status of the vehicle
such as an operation by a driver of the vehicle or whether or not
an obstacle exists in the surroundings of the vehicle.
[0032] In the image processor of the present invention described
above, the processor includes: an image synthesis parameter group
associating pixels in the synthesized image with pixels in the
camera images; and a table representing the correspondence between
a weight reference number and a combination of sets of weighting
information, wherein a portion of the image synthesis parameter
group corresponding to the overlap area holds any of the weight
reference numbers shown in the table.
[0033] The monitoring system of the present invention includes the
image processor described above as an image processing section.
[0034] Alternatively, the image processor of the present invention
receives camera images taken with a plurality of cameras capturing
the surroundings of a vehicle and generating a synthesized image
from the camera images, wherein the image processor includes an
image synthesis parameter group associating pixels in the
synthesized image with pixels in the camera images, and in an
overlap area in which coverages of the plurality of cameras overlap
on the synthesized image, the image synthesis parameter group
includes weights set to camera images from the plurality of cameras
represented by a dithering method.
BRIEF DESCRIPTION OF THE DRAWINGS
[0035] FIG. 1 is a structural view of a vehicle surroundings
monitoring system of EMBODIMENT 1 of the present invention.
[0036] FIGS. 2A to 2C are views for description of image synthesis
using an image synthesis parameter group.
[0037] FIG. 3 shows an example of selection of image synthesis
parameter groups in EMBODIMENT 1 of the present invention.
[0038] FIG. 4 shows an example of selection of image synthesis
parameter groups and capture patterns in EMBODIMENT 1 of the
present invention.
[0039] FIG. 5 shows an example of selection of image synthesis
parameter groups and capture patterns in the case of using an
output of a brightness detection means in EMBODIMENT 1 of the
present invention.
[0040] FIG. 6 shows an example of selection of image synthesis
parameter groups in the case of using outputs of a switch detection
means and an obstacle detection means in EMBODIMENT 1 of the
present invention.
[0041] FIGS. 7A and 7B show exemplary compositions of a synthesized
image.
[0042] FIG. 8 is a structural view of a vehicle surroundings
monitoring system of EMBODIMENT 2 of the present invention.
[0043] FIG. 9 is a view showing a filter parameter group.
[0044] FIG. 10 shows an example of selection of image synthesis
parameter groups and filter parameter groups in EMBODIMENT 2 of the
present invention.
[0045] FIGS. 11A and 11B are views showing filter parameter
groups.
[0046] FIG. 12 is a structural view of a vehicle surroundings
monitoring system of EMBODIMENT 3 of the present invention.
[0047] FIGS. 13A to 13D are views showing an image synthesis
parameter group for generating a synthesized image from a plurality
of camera images.
[0048] FIGS. 14A and 14B are views showing a synthesized image
including an overlap area and an image synthesis parameter group
for generating the synthesized image.
[0049] FIG. 15 is a view showing the positions at which cameras and
obstacle sensors are placed in EMBODIMENT 4 of the present
invention.
[0050] FIGS. 16A to 16C are views showing a synthesized image and
image synthesis parameter groups for generating the synthesized
image.
[0051] FIGS. 17A to 17C are views showing examples of selection of
an image synthesis parameter group in EMBODIMENT 4 of the present
invention.
[0052] FIG. 18 is a view showing another image synthesis parameter
group in EMBODIMENT 4 of the present invention.
[0053] FIG. 19 is a view showing yet another exemplary image
synthesis parameter group in EMBODIMENT 4 of the present
invention.
[0054] FIGS. 20A and 20B are views showing other image synthesis
parameter groups in EMBODIMENT 4 of the present invention.
[0055] FIG. 21 is a view showing another construction for
implementing the present invention.
[0056] FIGS. 22A to 22C are views for description of conventional
problems.
[0057] FIGS. 23A and 23B are views for description of conventional
problems.
DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0058] Hereinafter, preferred embodiments of the present invention
will be described with reference to the accompanying drawings.
Embodiment 1
[0059] FIG. 1 illustrates a construction of a vehicle surroundings
monitoring system of EMBODIMENT 1 of the present invention.
Referring to FIG. 1, cameras 101 take images, A/D converters 102
digitalize analog image signals, and frame memories 103 temporarily
hold the digitalized input images. Each of the frame memories 103
have a so-called double-buffer structure, which temporarily holds
immediately preceding one-frame image data while continuing to
store the images output from the camera 1. In other words, the
frame memories 103 are constructed to permit read of arbitrary
pixel data in the immediately preceding one-frame image in response
to a read request from an image synthesis section 104.
[0060] A vehicle motion detection section 107 detects motion of a
vehicle. A parameter storage section 109 stores in advance a
plurality of image synthesis parameter groups representing the
correspondence between camera images and a synthesized image. A
parameter selection section 108 selects one of the plurality of
image synthesis parameter groups stored in the parameter storage
section 109 according to the motion of the vehicle detected by the
vehicle motion detection section 107. The image synthesis section
104 sequentially reads images from the frame memories 103 according
to the image synthesis parameter group selected by the parameter
selection section 108 to combine the images and output a
synthesized image. A D/A converter 105 converts the synthesized
image to analog signals, and a display 106 displays the
analog-converted synthesized image.
[0061] The A/D converters 102, the frame memories 103, the image
synthesis section 104, the D/A converter 105, the parameter
selection section 108, and the parameter storage section 109
constitute an image processor or an image processing section. The
parameter selection section 108 and the parameter storage section
109 constitute a parameter generation section.
[0062] In this embodiment, assume that images handled by the
cameras 101 and the display 106 are interlaced scanning images.
Also assume that the vehicle motion detection section 107 detects
the rotational speed of the wheels of the vehicle, the driving
direction, and the like as the motion of the vehicle, from signals
sent from sensors placed on an axle of a wheel, the shift lever,
and the like.
[0063] The system also includes a vehicle status detection section
200, which will be described later together with the usage
thereof.
[0064] Hereinafter, the operation of the vehicle surroundings
monitoring system with the above construction will be
described.
[0065] FIGS. 2A to 2C illustrate how to combine images using an
image synthesis parameter group. FIG. 2A illustrates an example of
a camera image that is a one-frame image from a camera 1 placed as
shown in FIG. 15. FIG. 2B illustrates an example of a one-frame
synthesized image obtained by modifying and combining four camera
images.
[0066] FIG. 2C is an example of an image synthesis parameter group,
in which respective sets of coordinates have one-to-one
correspondence with the coordinates of pixels in the synthesized
image. Each set of coordinates includes parameters representing the
camera number with which a camera image is taken and a
two-dimensional array indicating the coordinates of a pixel in the
camera image used to generate the pixel in the synthesized image
corresponding to the set of coordinates. In the example shown in
FIG. 2C, information of "camera No. 1, coordinates (xi, yi)" is
stored in the coordinates (xo, yo) in the image synthesis parameter
group. This indicates that the pixel at the coordinates (xo, yo) in
the synthesized image is generated by the pixel at the coordinates
(xi, yi) in the camera image from the camera 1. In this way, it is
possible to describe the correspondence between the pixels in a
plurality of camera images and those in a synthesized image by use
of an image synthesis parameter group constructed as shown in FIG.
2C.
[0067] In this embodiment, it is assumed that the parameter storage
section 109 stores a plurality of image synthesis parameter groups
having different spatial or temporal resolutions for at least
either the camera images or the synthesized image.
[0068] As such a plurality of image synthesis parameter groups
having different spatial or temporal resolutions, assume
specifically that the parameter storage section 109 stores a
frame-base image synthesis parameter group and a field-base image
synthesis parameter group having the same composition of the
synthesized image. In the frame-base image synthesis parameter
group, a one-frame camera image corresponds to a one-frame
synthesized image. In the field-base image synthesis parameter
group, the first field of a camera image corresponds to one field
of a synthesized image, while the second field of the camera image
corresponds to the other field of the synthesized image. The
field-to-field correspondence between the camera image and the
synthesized image can be easily described by imposing a restriction
that the part of a camera image having an even y coordinate value
corresponds to the part of a synthesized image having an even y
coordinate value and that the part of the camera image having an
odd y coordinate value corresponds to the part of the synthesized
image having an odd y coordinate value.
[0069] FIG. 3 shows an example of the operation of the parameter
selection section 108. Referring to FIG. 3, the parameter selection
section 108 selects the image synthesis parameter group according
to the vehicle speed detected by the vehicle motion detection
section 107 based on predetermined criteria. Specifically, the
parameter selection section 108 selects the frame-base image
synthesis parameter group when the motion of the vehicle is
relatively slow, for example, when the vehicle speed is less than
10 km/h, and selects the field-base image synthesis parameter group
when the motion of the vehicle is relatively fast, for example,
when the vehicle speed is 10 km/h or more. The image synthesis
section 104 reads pixel data of camera images from the frame
memories 103 according to the camera numbers and the coordinate
values in the image synthesis parameter group selected by the
parameter selection section 108, to generate a synthesized image.
The generated synthesized image is converted to analog signals by
the D/A converter 105 and displayed on the display 106.
[0070] By the operation as described above, images are combined
every field when the vehicle speed is high and the motion in the
camera image is large. This enables generation of a natural
synthesized image free from comb-shaped displacement. On the
contrary, images are composed every frame when the vehicle speed is
low. This enables generation of a clear synthesized image with high
resolution. That is, by changing the image synthesis method
according to the motion of the vehicle, it is possible to reduce
the unnaturalness of the synthesized image that is otherwise
observed when the vehicle is moving, and also prevent the reduction
in the resolution of the synthesized image that is otherwise
observed when the vehicle is substantially standing still.
[0071] In this embodiment, the vehicle motion detection section 107
detects the motion of the vehicle, represented by the rotational
speed of the wheels, the driving direction, and the like, from
signals sent from sensors placed on an axle, the shift lever, and
the like. Alternatively, the vehicle motion may be detected from
camera images.
[0072] In the case of detecting the vehicle motion from camera
images, images from one of the cameras 101 are sequentially read
from the frame memory 103, to obtain an image composed of a
plurality of temporally continuous frames or fields and calculate a
motion vector in the image. Based on the calculated motion vector
and camera parameters such as the pre-measured position, direction,
and focal distance of the camera 101, the direction and speed of
the motion of the road surface in the image are determined and
output as the vehicle speed.
[0073] By providing the vehicle motion detection section 107 that
detects the vehicle motion from camera images as described above,
it is no more required to supply inputs other than the images, such
as the signals from sensors placed on an axle and the shift lever.
Therefore, a simple system construction can be realized.
[0074] The cameras 101 may have a function of switching a capture
pattern according to an input switch signal. In such a case, the
parameter selection section 108 may be constructed to switch the
capture pattern of the cameras 101, in addition to selecting the
image synthesis parameter group as described above, according to
the vehicle motion detected by the vehicle motion detection section
107.
[0075] FIG. 4 is a view showing an example of the operation of the
parameter selection section 108 in the case described above. In
this example, it is assumed that the cameras 101 permit switching
between interlaced imaging (one field every 1/60 second) and
non-interlaced imaging (one frame every 1/30 second) and switching
of the shutter speed (exposure time: 1/30 second, 1/60 second, and
1/120 second).
[0076] As shown in FIG. 4, the parameter selection section 108
selects the image synthesis parameter group according to the
vehicle speed detected by the vehicle motion detection section 107
based on predetermined criteria, and also switches the capture
pattern of each camera 101. Specifically, the parameter selection
section 108 selects the frame-base image synthesis parameter group
when the vehicle speed is less than 10 km/h, or the field-base
image synthesis parameter group when it is 10 km/h or more. In
addition, the parameter selection section 108 switches the capture
pattern of each camera 101 so that non-interlaced imaging and
1/30-second exposure time are selected when the vehicle speed is
less than 10 km/h, interlaced imaging and 1/60-second exposure time
are selected when the vehicle speed is between 10 km/h or more and
less than 30 km/h, and interlaced imaging and 1/120-second exposure
time are selected when the vehicle speed is 30 km/h or more.
[0077] By the operation described above, when the vehicle speed is
less than 10 km/h, images are taken by non-interlaced imaging and
combined per frame. This increases the resolution of the
synthesized image. When the vehicle speed is 10 km/h or more,
images are taken by interlaced imaging and combined per field. This
decreases the resolution, but natural motion is obtained. In
addition, by switching the exposure time with the vehicle speed, a
natural synthesized image with reduction in blurring due to the
motion is obtained. In other words, a synthesized image with
optimum quality determined by the motion of the vehicle can be
generated by the combination of the selection of the image
synthesis parameter group and the switching of the capture
pattern.
[0078] The construction shown in FIG. 1 includes the vehicle status
detection section 200. The outputs of the vehicle status detection
section 200 can be used in the selection of the image synthesis
parameter group and the switching of the capture pattern, to
further improve the quality of the synthesized image. The vehicle
status detection section 200, which is constructed of an optical
sensor placed on the vehicle and the like, includes a means 210 for
detecting the brightness in the surroundings of the vehicle, a
means 220 for detecting the switching operation by the driver, and
a means 230 for detecting an obstacle in the surroundings of the
vehicle.
[0079] FIG. 5 is a view showing an example of the operation of the
parameter selection section 108 using the output of the brightness
detection means 210. In general, the exposure time required to take
one-frame image by interlaced imaging is half that required by
non-interlaced imaging. In the case of a moving image, a longer
exposure time causes blurring but provides a bright image. On the
contrary, a shorter exposure time reduces blurring but darkens the
image. Based on this fact, the parameter selection section 108
performs the selection of the image synthesis parameter group and
the switching of the capture pattern of each camera 101 in
consideration of the brightness in the surroundings of the vehicle
detected by the brightness detection means 210, in addition to the
vehicle speed detected by the vehicle detection section 107. This
enables generation of a synthesized image with optimum quality
determined by the brightness in the surroundings of the vehicle and
the vehicle speed.
[0080] FIG. 6 is a view showing an example of the operation of the
parameter selection section 108 using the outputs of the switch
detection means 220 and the obstacle detection means 230. Assume
that the parameter storage section 109 stores frame-base and
field-base image synthesis parameter groups for two types of
compositions shown in FIGS. 7A and 7B, and that the driver is
allowed to select one of three options, "composition A",
"composition B", and "automatic", by switch operation. The switch
detection means 220 detects the switch operation by the driver. The
obstacle detection means 230 measures the distance from an obstacle
using an ultrasonic obstacle sensor placed on the vehicle and
outputs the results.
[0081] The parameter selection section 108 selects a suitable image
synthesis parameter group according to the detected status of the
vehicle, that is, the selection by the driver and the distance from
an obstacle, and the vehicle speed detected by the vehicle motion
detection section 107.
[0082] Composition A of FIG. 7A includes a wider range of the
surroundings of the vehicle, while composition B of FIG. 7B shows a
narrower range of the surroundings of the vehicle in an enlarged
view. Therefore, supposing the vehicle speed is the same, greater
comb-shaped displacement is generated in composition B than in
composition A. In view of this fact, the vehicle speed with which
the image synthesis parameter group is switched from the frame base
to the field base may be changed according to the motion and status
of the vehicle, as shown in FIG. 6. This enables generation of a
synthesized image with further optimum quality.
[0083] The vehicle status detection section 200 may also detect the
operations of the shift lever, the winker, and the like by the
driver.
[0084] In this embodiment, the field-base and frame-base image
synthesis parameter groups were used as an example. It is also
possible to use other types of image synthesis parameter groups
that are different in spatial or temporal resolution relation from
each other. For example, when a one-frame camera image is
constructed of a plurality of images taken for different durations,
it is possible to use a plurality of image synthesis parameter
groups having different spatial or temporal resolution
combinations, to provide substantially the same effects as those
described above.
[0085] It should be noted that the criteria for selection of a
parameter group shown in FIGS. 3 to 6 and 10 are mere examples and
that a parameter group may be selected according to the motion and
status of the vehicle based on criteria other than the exemplified
ones.
[0086] In this embodiment, field-base and frame-base image
synthesis parameter groups were prepared. Alternatively, only a
frame-base image synthesis group may be prepared to realize the
operation described above. That is, a field-base image synthesis
parameter group can be prepared by rounding the y coordinate value
yi in a camera image to an odd or even number according to the
field to which the y coordinate value yo in the frame-base image
synthesis parameter group belongs. Therefore, only a frame-base
image synthesis parameter group may be stored in the parameter
storage section 109. When a frame-base image synthesis parameter
group is required, the parameter selection section 108 reads the
frame-base image synthesis parameter group as it is. When a
field-base image synthesis parameter group is required, the
parameter selection section 108 may convert the frame-base image
synthesis parameter group stored in the parameter storage section
109 to a field-base image synthesis parameter group in the manner
described above.
Embodiment 2
[0087] FIG. 8 illustrates a construction of a vehicle surroundings
monitoring system of EMBODIMENT 2 of the present invention. This
construction is different from the construction shown in FIG. 1 in
that a filtering section 310 is provided between the A/D converters
102 and the frame memories 103 to perform filtering of camera
images. A parameter storage section 330 stores a plurality of sets
of an image synthesis parameter group representing the
correspondence between camera images and a synthesized image and a
filter parameter group corresponding to the image synthesis
parameter group. A parameter selection section 320 selects a set of
the image synthesis parameter group and the filter parameter group
from the parameter storage section 330 according to the vehicle
motion detected by the vehicle motion detection section 107 and the
vehicle status detected by the vehicle status detection section
200. The filtering section 310 performs frequency-band limitation
filtering for respective camera images according to the filter
parameter group in the set selected by the parameter selection
section 320.
[0088] FIG. 9 shows an example of the filter parameter group, which
specifies the cut-off frequency for each camera. The filtering
section 310 performs low-pass filtering with the cut-off frequency
specified in the filter parameter group for digitized image data
output from the A/D converters 102.
[0089] In the illustrated example, the cut-off frequency is
provided for a pixel clock fs, and "1/2fs" is specified when the
input image data is to be output as it is and "1/4fs" is specified
when the input image data is to be filtered with a cut-off
frequency of 1/4fs. The cut-off frequency is calculated in advance
based on a sampling theorem according to the contraction rate of
the portion of the camera image that is most contracted during
image synthesis according to the image synthesis parameter group.
For example, when the contraction rate of the portion of a camera
image from the camera 1 that is most contracted during image
synthesis is 1/2 in terms of the length, the cut-off frequency for
the camera 1 is set at 1/4 fs.
[0090] The contraction rate of a camera image can be determined
using the coordinate values for the camera image in an image
synthesis parameter group. For example, assume that the parameter
element at coordinates (x0, y0) in an image synthesis parameter
group includes "camera No. 1, coordinates (x1, y1)", and the
parameter element at coordinates (x0+1, y0) in the image synthesis
parameter group includes "camera No. 1, coordinates (x2, y2)". In
this case, when the distance between the coordinates (x1, y1) and
the coordinates (x2, y2) is D, the portions at and around the
coordinates (x1, y1) and the coordinates (x2, y2) in the camera
image from the camera 1 have been contracted to 1/D in the
synthesized image. In this way, it is possible to obtain the
contraction rates at and around certain pixels in a camera image.
The minimum of the thus-obtained contraction rates is the
contraction rate of the most contracted portion of the camera
image.
[0091] The parameter selection section 320 selects the image
synthesis parameter group and the filter parameter group according
to the vehicle speed detected by the vehicle motion detection
section 107 and the vehicle status detected by the vehicle status
detection section 200 as shown in FIG. 10, for example. The other
operation is substantially the same as that described in EMBODIMENT
1.
[0092] As described in relation to the problems to be solved, when
an image is contracted in modification, aliasing distortion may be
generated in a synthesized image unless a high-range frequency
component is removed according to the contraction rate. In this
embodiment, such aliasing distortion can be eliminated by
performing the low-pass filtering according to the filter parameter
group. Moreover, in the case of switching the composition of a
synthesized image as shown in FIG. 7, the filter parameter group
may be switched together with the switching of the composition.
This enables generation of a synthesized image with optimum quality
with reduced aliasing distortion determined by the status and speed
of the vehicle.
[0093] FIGS. 11A and 11B are views for description of another
example of the filter parameter group. The filter parameter group
shown in FIG. 11B includes parameter elements having one-to-one
correspondence with the coordinates of pixels in a one-frame camera
image. Each parameter element specifies the cut-off frequency for
filtering of each camera image. In other words, the filter
parameter group includes filtering setting data for respective
pixel positions of each camera image.
[0094] The parameter selection section 320 selects the image
synthesis parameter group and the filter parameter group according
to the vehicle speed detected by the vehicle detection section 107
and the vehicle status detected by the vehicle status detection
section 200, and outputs the selected results while performing
sequential scanning. The filtering section 310 performs low-pass
filtering for the digitized image data output from the A/D
converters 102 according to the cut-off frequency information in
the filter parameter group output from the parameter selection
section 320.
[0095] In the filter parameter group shown in FIG. 9, only one
cut-off frequency is provided for one camera image. This is not
necessarily suitable when the contraction rate varies with the
positions on a camera image in the image synthesis. On the
contrary, in the filter parameter group shown in FIG. 11B, the
cut-off frequency can be changed with the positions on a camera
image, and thus the non-contracted portion of the input image is
prevented from blurring.
[0096] The filter parameter group shown in FIG. 11B includes
filtering setting data having one-to-one correspondence with the
coordinates of pixels in a one-frame camera image. In general,
filter parameters of adjacent pixels tend to have similar values.
Therefore, common setting data may be held for a plurality of
adjacent pixels. For example, common setting data may be held every
rectangular area of 8.times.8 pixels. By this setting, in addition
to preventing the non-contracted portion of the image from
blurring, the data amount of the filter parameter group can be
reduced compared with the case of holding filtering setting data
every pixel.
[0097] A more significant effect can be provided by combining the
constructions of EMBODIMENT 1 and EMBODIMENT 2. That is, the
interlace noise can be eliminated by the technical feature of
EMBODIMENT 1 and the aliasing distortion can be eliminated by the
technical feature of EMBODIMENT 1. This enables generation of a
synthesized image with good quality even when the vehicle is moving
at high speed.
Embodiment 3
[0098] FIG. 12 illustrates a construction of a vehicle surroundings
monitoring system of EMBODIMENT 3 of the present invention. This
construction is different from the construction shown in FIG. 1 in
that a brightness correction section 410 is provided between the
frame memories 103 and the image synthesis section 104 to correct
the brightness and tint of camera images. A brightness correction
parameter calculation section 420 is also provided to calculate
brightness correction parameters used by the brightness correction
section 410.
[0099] FIGS. 13A to 13D illustrate an example of an image synthesis
parameter group for generating a synthesized image from a plurality
of camera images. FIGS. 13A and 13B are examples of images taken
with two cameras placed on the rear of the vehicle. In the
illustrated example, an image of the area just behind the vehicle
is taken doubly by cameras 1 and 2. FIG. 13C shows an example of a
synthesized image, and FIG. 13D shows an example of an image
synthesis parameter group.
[0100] In the example shown in FIG. 13D, the parameter element at
the coordinates (xo, yo) in the image synthesis parameter group
indicates that a synthesized image should be generated by combining
the pixel at the coordinates (Xi.sub.1, Yi.sub.1) in the image from
the camera 1 and the pixel at the coordinates (Xi.sub.2, Yi.sub.2)
in the image from the camera 2 at a weighting ratio of 0.7:0.3. The
image synthesis section 104 assigns weights to the respective
camera images and then adds, to sequentially output the results as
a synthesized image. This enables generation of a smooth
synthesized image free from formation of a conspicuous juncture due
to difference in brightness and tint between the camera images,
compared with the case of generating each pixel of a synthesized
image from a single pixel of one camera image.
[0101] Moreover, in this embodiment, brightness correction
parameters are calculated using information on an overlap area on a
synthesized image.
[0102] Specifically, the image synthesis parameter group stored in
the parameter storage section 109 includes overlap area data
indicating the coordinates of a pixel in a camera image that
corresponds to an overlap area on a synthesized image. The
brightness correction parameter calculation section 420 receives
the overlap area data included in the image synthesis parameter
group and calculates a brightness correction parameter using
brightness and tint data for the pixel in the camera image
corresponding to the overlap area indicated by the overlap area
data.
[0103] FIGS. 14A and 14B illustrate an example of a synthesized
image including an overlap area and an example of an image
synthesis parameter group for this synthesized image. In the
synthesized image of FIG. 14A, the hatched portions represent the
coverages of cameras 1 and 2, which overlap at the rear of the
vehicle forming an overlap area. The image synthesis parameter
group of FIG. 14B includes overlap area data indicating the
coordinates of pixels in the camera images corresponding to the
overlap area, as partly shown in FIG. 14B. That is, as the overlap
area data it is shown that the pixel at the coordinates (Xi1, Yi1)
in the image from camera 1 and the pixel at the coordinates (Yi2,
Yi2) in the image from camera 2 correspond to the coordinates (xo,
yo) in the overlap area.
[0104] It is possible to determine whether a given position is
inside a non-overlap area or an overlap area depending on whether
the weighting ratio is "1:0" or otherwise. Alternatively, a given
position may be determined to be inside an overlap area if two or
more effective camera numbers are specified, and inside a
non-overlap area if only one effective camera is specified.
[0105] The parameter selection section 108 selects the image
synthesis parameter group according to the vehicle speed detected
by the vehicle motion detection section 107 and the vehicle status
detected by the vehicle status detection section 200 based on
predetermined criteria, sequentially reads the selected parameters
by interlaced scanning, and outputs the results to the image
synthesis section 104. The overlap area data in the image synthesis
parameter group output from the parameter selection section 108 is
also output to the brightness correction parameter calculation
section 420.
[0106] The brightness correction parameter calculation section 420
determines the distributions of brightness and tint of the
respective camera images in the overlap area, from the overlap area
data output from the parameter selection section 108 and the pixel
values read from the frame memories 103 by the image synthesis
section 104, to calculate brightness correction parameters with
which the distributions of brightness and tint of the respective
camera images can substantially match with each other in the
overlap area.
[0107] The brightness correction section 410 corrects the
brightness for the pixel values read from the frame memories 103 by
the image synthesis section 104 according to the brightness
correction parameters calculated by the brightness correction
parameter calculation section 420, and output the results to the
image synthesis section 104. The image synthesis section 104
generates a synthesized image from the camera images of which
brightness has been corrected by the brightness correction section
410, according to the image synthesis parameter group output from
the parameter selection section 108.
[0108] A procedure of calculating the brightness-correction
parameters will be described in a specific example.
[0109] Assume that the brightness correction parameter calculation
section 420 outputs as the brightness correction parameters a gain
coefficient and an offset coefficient with which the averages and
distributions of the brightness of respective camera images can
match with each other in the overlap area. Also assume that the
brightness correction section 410 corrects the pixel values of the
respective camera images with a linear equation using the gain
coefficient and the offset coefficient as the brightness correction
parameters.
[0110] Assume that a plurality of overlap areas-exist. For example,
assume that there exist four overlap areas, a first overlap area
formed by cameras 1 and 2, a second overlap area formed by cameras
1 and 3, a third overlap area formed by cameras 2 and 4, and a
fourth overlap area formed by cameras 3 and 4.
[0111] First focusing on the first overlap area, the brightness
correction parameter calculation section 420 accumulates images
taken with the cameras 1 and 2 read from the frame memories 103 by
the image synthesis section 104 by one frame each, and obtains the
averages and distributions of the brightness of the images taken
with the cameras 1 and 2 in the first overlap area. Likewise, for
the second to fourth overlap areas, the averages and distributions
of the brightness of camera images in each of the overlap areas are
obtained.
[0112] Next, as brightness correction parameters, a gain and an
offset are determined so that the averages and distributions of the
brightness of the images from the cameras 1 and 2 in the first
overlap area match with each other. For example, consider the case
that the average and distribution of the brightness of the image
from the camera 1 are 100 and 10, respectively, and the average and
distribution of the brightness of the image from the camera 2 are
121 and 11, respectively. Then, the averages and distributions of
the images from the cameras 1 and 2 will match with each other when
the gain and the offset are set to 1 and 0 respectively for the
image from the camera 1, and 10/11 and -10 respectively for the
image from the camera 2.
[0113] Using the brightness correction parameters determined for
the first overlap area, the average and distribution of the
brightness of the image from the camera 1 in the second overlap
area are corrected. Note that in this example where the gain is 1
and the offset is 0 for the image from the camera 1, no correction
is required for the average and distribution of the brightness of
the image from the camera 1. Brightness correction parameters for
an image from the camera 3 are then determined so that the averages
and distributions of the brightness of the images from the cameras
1 and 3 match with each other. Likewise, the average and
distribution of the brightness of the image from the camera 2 in
the third overlap area are corrected using the brightness
correction parameters determined in the first overlap area.
Brightness correction parameters for an image from the camera 4 are
then determined so that the averages and distributions of the
brightness of the images from the cameras 2 and 4 match with each
other.
[0114] In the calculation procedure described above, proper
brightness correction may not be obtained in the fourth overlap
area. In view of this, as another calculation procedure, brightness
correction parameters may be calculated temporarily in the
respective overlap areas. For a camera image existing in a
plurality of overlap areas, the average of the temporarily
calculated brightness correction parameters may be determined as
the final brightness correction parameters for the camera image.
This procedure fails to completely match the brightness and the
tint between camera images in each overlap area. However, as the
entire synthesized image, it is expected that all of the overlap
areas have no great displacement in brightness and tint.
[0115] When a plurality of overlap areas exist, the priorities
given to the overlap areas to be followed during the calculation of
brightness correction parameters may be determined according to the
motion of the vehicle detected by the vehicle motion detection
section 107.
[0116] For example, when the vehicle is moving backward, a high
priority may be given to an overlap area located at the rear of the
vehicle on the synthesized image. When the vehicle is moving
forward, a high priority may be given to an overlap area located at
the front of the vehicle on the synthesized image. In general, the
driver tends to pay attention to the rear of the vehicle when
driving backward and to the front of the vehicle when driving
forward. Therefore, by giving priorities, the brightness and the
tint can be made uniform in the overlap area at the position to
which the driver tends to pay attention, although they may not be
uniform in other overlap areas. The resultant synthesized image is
therefore more natural for the driver.
[0117] As described above, in this embodiment, brightness
correction parameters are calculated using data on the brightness
and tint of pixels in camera images corresponding to an overlap
area, and the brightness is corrected using the parameters. This
enables generation of a smooth synthesized image free from
formation of a conspicuous juncture due to difference in brightness
and tint between the camera images.
[0118] The construction of FIG. 12 includes the brightness
correction section 410. However, if the cameras 101 incorporate
therein a function corresponding to that of the brightness
correction section 410, the brightness correction section 410 may
be omitted and thus the system construction can be simplified.
[0119] The technique for brightness correction is not limited to
that described in this embodiment, but other techniques may also be
used as long as the brightness and tint of camera images in an
overlap area can be made close to each other, and substantially the
same effect as that described in this embodiment can be
attained.
[0120] In this embodiment, weighting information in the image
synthesis parameter group was used to identify an overlap area.
Alternatively, in place of the weighting information, some
exclusive information indicating whether or not a given position is
inside an overlap area may be included in the image synthesis
parameter group.
Embodiment 4
[0121] In EMBODIMENT 4 of the present invention, in an overlap area
of the coverages of a plurality of cameras, the selection of the
camera images used for image synthesis and the setting of weights
for the respective camera images are appropriately changed.
[0122] The construction of the vehicle surroundings monitoring
system of this embodiment is substantially the same as that shown
in FIG. 1, except that the parameter storage section 109, which is
the same in construction, includes a plurality of image synthesis
parameter groups different in the selection of camera images used
for image synthesis in an overlap area.
[0123] FIG. 15 is a view showing the positions of cameras and
obstacle sensors as the obstacle detection means 230 on a vehicle
in this embodiment. Referring to FIG. 15, cameras 1 and 2 are
placed on pillars at the rear of the vehicle, where the camera 1
takes images on the left side of the rear area of the vehicle,
while the camera 2 takes images on the right side thereof. The
coverages of the cameras 1 and 2 overlap in the rear of the
vehicle. Two obstacle sensors are placed on the rear of the vehicle
to enable detection of an obstacle in the rear area of the
vehicle.
[0124] FIGS. 16A to 16C are views illustrating examples of a
synthesized image and image synthesis parameter groups for
generation of the synthesized image. FIG. 16A illustrates a
synthesized image, and FIGS. 16B and 16C illustrate two image
synthesis parameter groups A and B for generation of the
synthesized image of FIG. 16A. The image synthesis parameter group
A (FIG. 16B) and the image synthesis parameter group B (FIG. 16C)
are used to generate the same synthesized image in composition
(FIG. 16A), but use a different camera image for image synthesis in
the overlap area in the rear of the vehicle. The image synthesis
parameter group A uses an image from the camera 1 while the image
synthesis parameter group B uses an image from the camera 2.
[0125] The operation of the vehicle surroundings monitoring system
of this embodiment will be described.
[0126] When the obstacle detection means 230 detects an obstacle on
the right or left side of the rear area of the vehicle, it outputs
the position of the detected obstacle to the parameter selection
section 108. The parameter selection section 108 selects an image
synthesis parameter group according to the position of the detected
obstacle and the switch operation by the user.
[0127] For example, the image synthesis parameter group A is
selected when the obstacle exists on the left side of the rear
area, and the image synthesis parameter group B is selected when
the obstacle exists on the right side of the rear area. The image
synthesis section 104 reads camera images from the frame memories
103 according to the selected image synthesis parameter group to
generate a synthesized image.
[0128] By the operation described above, in the case shown in FIG.
17A, for example, where an obstacle such as another vehicle exists
on the left side of the rear area, the image synthesis parameter
group A is selected. This makes the juncture formed between the
camera images less conspicuous on the resultant synthesized image
as shown in FIG. 17B. If the image synthesis parameter group B is
selected, the juncture between the camera images will run across
the obstacle vehicle on the synthesized image, making the vehicle
unnatural with the juncture as shown in FIG. 17C. In other words,
by selecting the image synthesis parameter group according to the
position of an obstacle, the existence of the juncture can be made
less conspicuous on the synthesized image, and thus a synthesized
image with natural appearance can be generated.
[0129] The parameter selection section 108 may otherwise switch the
image synthesis parameter group according to the motion of the
vehicle detected by the vehicle motion detection section 107, in
place of the output from the vehicle status detection section 200.
By switching in this manner, the camera image used for image
synthesis in the overlap area can be switched according to the
direction and speed of the driving of the vehicle. Therefore, by
selecting a camera image from a camera of which coverage is closer
to the forward area of the vehicle, for example, it is possible to
reduce formation of a juncture between camera images in and around
an area mostly viewed by the driver, and thus display a synthesized
image with which the driver can feel easy in driving. Naturally, it
is also possible to use both the outputs from the vehicle status
detection section 200 and the vehicle motion detection section
107.
[0130] FIG. 18 illustrates another example of the image synthesis
parameter group in this embodiment. The image synthesis parameter
group of FIG. 18 includes a plurality of sets of weighting
information for camera images in an overlap area on a synthesized
image. For example, the position at the coordinates (x0, y0) in the
image synthesis parameter group is a position at which a pixel at
the coordinates (Xi1, Yi1) in a camera image from the camera 1 and
a pixel at the coordinates (Xi2, Yi2) in a camera image from the
camera 2 overlap with each other. This position includes two sets
of weighting information, A (0.7:0.3) and B (0.2:0.8) as the
weighting ratio used during synthesis.
[0131] The parameter selection section 108 selects one of a
plurality of sets of weighting information included in the image
synthesis parameter group according to the output of the vehicle
status detection section 200. For example, the weighting
information A is selected when an obstacle exists on the left side
of the rear area of the vehicle, and the weighting information B is
selected when an obstacle exists on the right side.
[0132] Thus, by selecting weighting information for image synthesis
according to the position of an obstacle, it is possible to reduce
the unnaturalness of the juncture formed between the camera images
on a synthesized image and thus generate a synthesized image with a
smoother juncture.
[0133] FIG. 19 illustrates yet another example of the image
synthesis parameter group in this embodiment. The image synthesis
parameter group of FIG. 19 is equivalent to that of FIG. 18 except
that one weight reference number indicating a combination of sets
of weighting information is given, in place of sets of weighting
information, for each pixel.
[0134] In the example shown in FIG. 19, the weight reference number
for the coordinates (x0, y0) in the image synthesis parameter group
is "3". By referring to another table, it is found that the weight
reference number "3" indicates that the weighting ratio of the
pixels to be combined is 0.7:0.3 (weighting information A) or
0.2:0.8 (weighting information B).
[0135] While the image synthesis parameter group of FIG. 18 needs
to hold four values as weighting information for each coordinate
position, the image synthesis parameter group of FIG. 19 can hold
only one weight reference information value. This indicates that
the data amount of the image synthesis parameter group of FIG. 19
is smaller as the number of pixels in the image synthesis parameter
group is greater or the number of combinations of sets of weighting
information is smaller.
[0136] Thus, the data amount of the image synthesis parameter group
can be reduced by holding the weight reference number indicating a
combination of a plurality of sets of weighting information, in
place of holding a plurality of sets of weighting information, for
each coordinate position in the image synthesis parameter
group.
[0137] FIGS. 20A and 20B illustrate other examples of the image
synthesis parameter groups in this embodiment, which are generated
using a dithering method based on the image synthesis parameter
group of FIG. 18.
[0138] The image synthesis parameter group A of FIG. 20A is
generated by a so-called dithering method, where, based on the
camera numbers and the weighting information A for each pixel in
the image synthesis parameter group of FIG. 18, one of the two
camera images is selected for each pixel in the vicinity of a given
pixel so that the ratio of the number of pixels using one camera
image to that using the other camera image is close to the
specified weighting ratio. Likewise, the image synthesis parameter
group B of FIG. 20B is generated based on the camera numbers and
the weighting information B for each pixel in the image synthesis
parameter group of FIG. 18.
[0139] In the weighting information A in the image synthesis
parameter group of FIG. 18, at the coordinates (x0, y0), the
weights are 0.7 for the camera 1 and 0.3 for the camera 2. In the
image synthesis parameter group A of FIG. 20A, the camera number is
selected so that the ratio of the number of pixels using the camera
1 to that using the camera 2 is close to 0.7:0.3 in the vicinity of
the pixel at the coordinates (x0, y0). Thus, in the example shown
in FIG. 20A, although the camera 1 is selected for the coordinates
(x0, y0), the ratio of the number of pixels using the camera 1 to
that using the camera 2 is about 0.7:0.3 at and around (x0,
y0).
[0140] The dithering method described above is widely known and
disclosed in "Television image information engineering handbook",
ed. by The Institute of Image Information and Television Engineers,
Ohmsha, Ltd., and the like. Detailed description on the method for
selecting camera images is therefore omitted here.
[0141] The parameter selection section 108 selects the image
synthesis parameter group A or B according to the output of the
vehicle status detection section 200.
[0142] Thus, by selecting one from a plurality of dithered image
synthesis parameter groups according to the position of an obstacle
for image synthesis, it is possible to reduce the unnaturalness of
the juncture formed between the camera images on a synthesized
image and thus generate a synthesized image with a smoother
juncture.
[0143] The total data amount of the image synthesis parameter group
A of FIG. 20A and the image synthesis parameter group B of FIG. 20B
is smaller than the data amount of the image synthesis parameter
group of FIG. 18 by the amount of weighting information.
[0144] Moreover, when image synthesis is performed using the image
synthesis parameter group A of FIG. 20A or the image synthesis
parameter group B of FIG. 20B, the number of pixels read from the
frame memories 103 by the image synthesis section 104 is smaller,
and the amount of calculation by the image synthesis section 104 is
smaller by the addition of the weighting, compared with the image
synthesis using the weight-added image synthesis parameter group of
FIG. 18.
[0145] Thus, the use of the image synthesis parameter group
generated by the dithering method provides the effect of reducing
the processing amount required for image synthesis, in addition to
the effect of reducing the data amount of the image synthesis
parameter group, compared with the use of the weighting
information-added image synthesis parameter.
[0146] In the embodiments described above, the display 106 displays
interlaced scanning images, and the parameter selection section
108, 320 reads an image synthesis parameter group by interlaced
scanning. The same effects as those described in the respective
examples can also be obtained by adopting non-interlaced scanning
in both operations.
[0147] In the embodiments described above, each image synthesis
parameter group is data in the form of a table indicating the
correspondence between the coordinates in camera images and the
coordinates in a synthesized image. Alternatively, it may be
described in other forms such as a mathematical expression
including variables and a program as long as the correspondence
between camera images and a synthesized image is clarified. When
such forms are adopted, the image synthesis section may be
constructed to obtain corresponding coordinates using an image
synthesis parameter group described in the form of a mathematical
expression including variables or a program. Japanese Patent
Application No. 10-217261, for example, discloses a method for
generating image synthesis parameter groups described in the forms
of a mathematical expression including variables and a program.
[0148] In the embodiments described above, it was assumed that the
image synthesis parameter group included elements having one-to-one
correspondence with the coordinates of pixels in a one-frame
synthesized image and had the same size as that of the synthesized
image. The size of the image synthesis parameter group is not
necessarily the same as that of the synthesized image. For example,
an image synthesis parameter group having a size larger than the
synthesized image may be prepared, and a portion having the same
size as the synthesized image may be cut out from this image
synthesis parameter group to be used for image synthesis.
[0149] In the above embodiments, the monitoring system and the
image processor of the present invention were described as being
applied to cars. Alternatively, they may be applied to other types
of vehicles such as shipping and airplanes. Otherwise, cameras may
be placed on a non-mobile object to be monitored such as a store, a
house, a showroom, and the like.
[0150] The positions of a plurality of cameras and the number of
cameras are not limited to those described above.
[0151] The function of the image processor of the present invention
may be implemented by hardware entirely or partially using
exclusive equipment, or may be implemented by software. It is also
possible to use a recording medium or a transmission medium storing
a program programmed to enable a computer to execute the entire or
partial function of the image processor of the present invention.
For example, a computer as shown in FIG. 21 may be used, where the
processing means such as the image synthesis section and the
parameter selection section may be implemented by a software
program executed by a CPU 510 and the results may be stored in a
ROM 520 or a RAM 530.
[0152] Thus, according to the present invention, it is possible to
reduce the unnaturalness of a synthesized image observed when the
vehicle is moving and also prevent the reduction in resolution
observed when the vehicle is standing still. In addition, aliasing
distortion on a synthesized image can be effectively suppressed.
Moreover, it is possible to reduce the unnaturalness of the
juncture formed between camera images on a synthesized image.
[0153] While the present invention has been described in a
preferred embodiment, it will be apparent to those skilled in the
art that the disclosed invention may be modified in numerous ways
and may assume many embodiments other than that specifically set
out and described above. Accordingly, it is intended by the
appended claims to cover all modifications of the invention that
fall within the true spirit and scope of the invention.
* * * * *