U.S. patent application number 15/619630 was filed with the patent office on 2017-09-28 for method for enhancing vehicle camera image quality.
The applicant listed for this patent is MAGNA ELECTRONICS INC.. Invention is credited to Marcel Kussel.
Application Number | 20170280061 15/619630 |
Document ID | / |
Family ID | 47996325 |
Filed Date | 2017-09-28 |
United States Patent
Application |
20170280061 |
Kind Code |
A1 |
Kussel; Marcel |
September 28, 2017 |
METHOD FOR ENHANCING VEHICLE CAMERA IMAGE QUALITY
Abstract
A method of image enhancement for a vehicle vision system
includes capturing multiple frames of image data with a camera and
executing a first brightness transfer function on a current frame
of captured image data to generate a first enhanced image frame. At
least one previously captured frame of image data is retrieved, and
a second brightness transfer function is executed on the at least
one previously captured frame of image data to generate a second
enhanced image frame. The first and second enhanced image frames
are blended to generate a blended enhanced image frame, and these
steps are repeated so as to generate multiple blended enhanced
image frames. The presence of an object is detected via processing
of blended enhanced image frames by a processor, and an output is
generated responsive to detection of the object.
Inventors: |
Kussel; Marcel;
(Aschaffenburg, DE) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
MAGNA ELECTRONICS INC. |
Auburn Hills |
MI |
US |
|
|
Family ID: |
47996325 |
Appl. No.: |
15/619630 |
Filed: |
June 12, 2017 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
14343937 |
Mar 10, 2014 |
9681062 |
|
|
PCT/US2012/057007 |
Sep 25, 2012 |
|
|
|
15619630 |
|
|
|
|
61539049 |
Sep 26, 2011 |
|
|
|
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
B60R 2300/307 20130101;
H04N 5/243 20130101; B60R 1/00 20130101; B60R 2300/8053
20130101 |
International
Class: |
H04N 5/243 20060101
H04N005/243; B60R 1/00 20060101 B60R001/00 |
Claims
1. A method of image enhancement for a vehicle vision system, said
method comprising: (a) providing a camera at a vehicle so that the
camera has an exterior field of view; (b) providing a processor
operable to process image data; (c) capturing multiple frames of
image data with the camera; (d) executing a first brightness
transfer function on a current frame of image data captured by the
camera to generate a first enhanced image frame; (e) retrieving at
least one previously captured frame of image data; (f) executing a
second brightness transfer function on the at least one previously
captured frame of image data to generate a second enhanced image
frame, wherein the second brightness transfer function is different
from the first brightness transfer function; (g) blending the first
and second enhanced image frames to generate a blended enhanced
image frame; repeating steps (d)-(g) so as to generate multiple
blended enhanced image frames; (h) detecting, via processing of
blended enhanced image frames by the processor, presence of an
object in the field of view of the camera; and (i) generating an
output responsive to detection of the object present in the field
of view of the camera.
2. The method of claim 1, wherein the step of executing the first
brightness transfer function enhances contrast of the current frame
of image data captured by the camera.
3. The method of claim 1, comprising executing tone mapping of
multiple frames of captured image data to enhance detection of the
object present in the field of view of the camera.
4. The method of claim 1, comprising classifying the detected
object present in the field of view of the camera.
5. The method of claim 4, comprising generating an output
responsive to classification of the detected object.
6. The method of claim 1, comprising determining a low visibility
driving condition and, responsive to determination of the low
visibility driving condition, increasing contrast of features in
captured image data by brightening brighter areas of captured image
data and dampening darker areas of captured image data.
7. The method of claim 6, comprising increasing contrast of
features in captured image data over multiple successive frames of
captured image data.
8. The method of claim 7, comprising tracking, via processing by
the processor of multiple successive frames of captured image data
during the determined low visibility driving condition, image flow
caused by movement of the vehicle to enhance detection and
identification of objects present in the field of view of the
camera.
9. The method of claim 1, wherein the at least one previously
captured frame of image data is retrieved from memory.
10. The method of claim 1, comprising determining a low visibility
driving condition via processing of captured image data.
11. The method of claim 1, comprising determining that fog is
present in the field of view of the camera via processing of
captured image data.
12. The method of claim 1, comprising providing blended enhanced
image frames of image data to a video display screen that is
disposed in the vehicle at a location viewable by a driver of the
vehicle when operating the vehicle.
13. The method of claim 1, wherein capturing multiple frames of
image data with the camera comprises capturing multiple frames of
image data with the camera at a frame rate of at least 30 frames
per second.
14. The method of claim 1, wherein blending the first and second
enhanced image frames comprises blending the first and second
enhanced image frames to generate a blended enhanced image frame
that is up to 20 percent derived from the first enhanced image
frame.
15. The method of claim 1, wherein executing a second brightness
transfer function on the at least one previously captured frame of
image data comprises executing the second brightness transfer
function on a plurality of previously captured frames of image data
to generate the second enhanced image frame.
16. The method of claim 1, comprising providing the generated
output to a driver assistance system of the vehicle.
17. The method of claim 16, wherein the driver assistance system of
the vehicle comprises a system selected from the group consisting
of a lane change assist system of the vehicle, a lane departure
warning system of the vehicle, a blind spot detection system of the
vehicle, an adaptive cruise control system of the vehicle, a
collision avoidance system of the vehicle, a traffic sign
recognition system of the vehicle, and a vehicle headlamp control
system of the vehicle.
18. The method of claim 1, comprising tracking the detected object
over successive frames of captured image data to determine if the
detected object is an object of interest in the field of view of
the camera.
19. The method of claim 1, wherein processing of captured image
data by the processor is responsive at least in part to steering of
the vehicle.
20. The method of claim 1, comprising distinguishing, via
processing of captured image data by the processor, moving objects
from non-moving objects.
21. The method of claim 20, wherein distinguishing moving objects
comprises distinguishing moving objects responsive at least in part
to at least one of (i) speed of the vehicle and (ii) steering of
the vehicle.
22. The method of claim 1, comprising disposing the camera at a
rear portion of the vehicle with an exterior field of view rearward
of the vehicle, and wherein said method comprises providing a
plurality of cameras at the vehicle so as to have respective
exterior fields of view, and wherein the plurality of cameras
comprises the camera at the rear portion of the vehicle.
23. The method of claim 22, comprising providing a display for
displaying images derived, at least in part, from image data
captured by the camera at the rear portion of the vehicle and
derived, at least in part, from image data captured by other
cameras of the plurality of cameras.
24. The method of claim 1, comprising disposing the camera at a
rear portion of the vehicle with an exterior field of view rearward
of the vehicle, and comprising providing a display for displaying
images derived, at least in part, from image data captured by the
camera during a reversing maneuver of the vehicle.
25. A method of image enhancement for a vehicle vision system, said
method comprising: (a) providing a camera at a vehicle so that the
camera has an exterior field of view; (b) providing a processor
operable to process image data; (c) capturing multiple frames of
image data with the camera; (d) executing a first brightness
transfer function on a current frame of image data captured by the
camera to generate a first enhanced image frame; (e) retrieving at
least one previously captured frame of image data; (f) executing a
second brightness transfer function on the at least one previously
captured frame of image data to generate a second enhanced image
frame, wherein the second brightness transfer function is different
from the first brightness transfer function; (g) blending the first
and second enhanced image frames to generate a blended enhanced
image frame; repeating steps (d)-(g) so as to generate multiple
blended enhanced image frames; (h) detecting, via processing of
blended enhanced image frames by the processor, presence of an
object in the field of view of the camera; (i) tracking the
detected object over successive frames of captured image data to
determine if the detected object is an object of interest in the
field of view of the camera; (j) generating an output responsive to
determination of the detected object being an object of interest;
and (k) providing the generated output to a driver assistance
system of the vehicle.
26. The method of claim 25, wherein the driver assistance system of
the vehicle comprises a blind spot detection system of the
vehicle.
27. The method of claim 25, wherein the driver assistance system of
the vehicle comprises a collision avoidance system of the
vehicle.
28. The method of claim 25, wherein the driver assistance system of
the vehicle comprises an adaptive cruise control system of the
vehicle.
29. The method of claim 25, wherein executing a second brightness
transfer function on the at least one previously captured frame of
image data comprises executing the second brightness transfer
function on a plurality of previously captured frames of image data
to generate the second enhanced image frame.
30. The method of claim 29, wherein blending the first and second
enhanced image frames comprises blending the first and second
enhanced image frames to generate a blended enhanced image frame
that is up to 20 percent derived from the first enhanced image
frame.
31. A method of image enhancement for a vehicle vision system, said
method comprising: (a) providing a camera at a rear portion of a
vehicle with an exterior field of view rearward of the vehicle; (b)
providing a processor operable to process image data; (c) capturing
multiple frames of image data with the camera; (d) executing a
first brightness transfer function on a current frame of image data
captured by the camera to generate a first enhanced image frame;
(e) retrieving at least one previously captured frame of image
data; (f) executing a second brightness transfer function on the at
least one previously captured frame of image data to generate a
second enhanced image frame, wherein the second brightness transfer
function is different from the first brightness transfer function;
(g) blending the first and second enhanced image frames to generate
a blended enhanced image frame; repeating steps (d)-(g) so as to
generate multiple blended enhanced image frames; and (h) providing
blended enhanced image frames of image data to a video display
screen that is disposed in the vehicle at a location viewable by a
driver of the vehicle when operating the vehicle.
32. The method of claim 31, comprising determining a low visibility
driving condition via processing of captured image data.
33. The method of claim 32, comprising, responsive to determination
of the low visibility driving condition, increasing contrast of
features in captured image data by brightening brighter areas of
captured image data and dampening darker areas of captured image
data.
34. The method of claim 31, comprising determining that fog is
present in the field of view of the camera via processing of
captured image data.
35. The method of claim 31, comprising providing a plurality of
cameras at the vehicle so as to have respective exterior fields of
view, wherein the plurality of cameras comprises the camera at the
rear portion of the vehicle, and wherein the video display screen
displays images derived, at least in part, from image data captured
by the camera at the rear portion of the vehicle and derived, at
least in part, from image data captured by other cameras of the
plurality of cameras.
36. The method of claim 35, wherein the plurality of cameras is
part of a surround view multi-camera system of the vehicle, and
wherein images displayed by the video display screen derived, at
least in part, from image data captured by the camera at the rear
portion of the vehicle and derived, at least in part, from image
data captured by other cameras of the plurality of cameras comprise
a bird's-eye view.
Description
CROSS REFERENCE TO RELATED APPLICATIONS
[0001] The present application is a continuation of U.S. patent
application Ser. No. 14/343,937, filed Mar. 10, 2014, now U.S. Pat.
No. 9,681,062, which is a 371 national phase filing of PCT
Application No. PCT/US2012/057007, filed Sep. 25, 2012, which
claims the filing benefit of U.S. provisional application Ser. No.
61/539,049, filed Sep. 26, 2012, which is hereby incorporated
herein by reference in its entirety.
FIELD OF THE INVENTION
[0002] The present invention relates to imaging systems or vision
systems for vehicles.
BACKGROUND OF THE INVENTION
[0003] Use of imaging sensors in vehicle imaging systems is common
and known. Examples of such known systems are described in U.S.
Pat. Nos. 5,877,897; 5,796,094; 5,670,935 and/or 5,550,677, which
are hereby incorporated herein by reference in their
entireties.
SUMMARY OF THE INVENTION
[0004] The present invention provides a vision system or imaging
system for a vehicle that utilizes one or more cameras to capture
images exterior of the vehicle, such as forwardly or rearwardly of
the vehicle, and provides for enhanced image processing to detect
objects in poor visibility conditions, such as in dense fog or the
like.
[0005] The vision system may enhance the image processing by
amplifying the contrast in the captured images by brightness
transfer function filtering and exposure stacking and tracking
contrast thresholds or features within the captured images, such as
on a frame-by-frame basis as the vehicle travels along a road.
[0006] These and other objects, advantages, purposes and features
of the present invention will become apparent upon review of the
following specification in conjunction with the drawings.
BRIEF DESCRIPTION OF THE DRAWINGS
[0007] FIG. 1 is a plan view of a vehicle with a vision system and
forward facing imaging sensor or camera that provides a forward
exterior field of view in accordance with the present
invention;
[0008] FIG. 2 shows images captured by the forward facing camera
and processed by the vision system of the present invention;
[0009] FIGS. 3(a) and 3(b) show graphs showing histograms of
luminance distribution for the vision system, with FIG. 3(b)
showing the histogram of the original image's graph of FIG. 3(a) as
spread into the highest possible dynamic range of the target system
which equates to a contrast amplification of the present
invention;
[0010] FIGS. 4(a) and 4(b) show graphs of brightness transfer
functions A (FIG. 4(a)) and B (FIG. 4(b)) which find use in the
flow chart of FIGS. 5 and 6, with the brightness transfer function
A enhancing the brighter areas and dampening the darker ones, and
with the brightness transfer function B decreasing the medium
illuminated areas, and with the upper end at less of 100, whereby
the overall illumination becomes decreased by this transfer
function;
[0011] FIG. 5 shows a flow chart of the image enhancing and
processing steps according the invention, and by mapping/stacking
an illumination reduced image scene on top of a contrast enhanced
image the dynamic range of the image increases: overexposed areas
appear less bright and underexposed more bright, which leads to
acknowledge details in the scene easier;
[0012] FIG. 6 shows a flow chart of the image enhancing and
processing steps according the invention as to be used in a vehicle
vision system, supporting machine and human vision driver assistant
algorithms; and
[0013] FIG. 7 shows an example of how images may be altered when
processed according to the flow chart of FIG. 5, whereby it becomes
apparent that the process turns out more contrasts of possible
objects on foggy weather conditions when comparing Image(t.sub.0)
and Image.sub.h(t.sub.0).
DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0014] Referring now to the drawings and the illustrative
embodiments depicted therein, a vehicle 10 includes an imaging
system or vision system 12 that includes at least one imaging
sensor or camera 14 (such as forward facing camera at the front (or
at the windshield) of the vehicle), which captures images exterior
of and forwardly of the vehicle (FIG. 1). The imaging system 12 is
operable to process (such as via an image processor) image data
captured by the camera 14 to present these on a display or to
detect objects and/or headlights of approaching vehicles and/or
taillights of leading vehicles in the field of view of the camera
(such as for use in an object detection system of the vehicle or
collision avoidance system of the vehicle or headlamp control
system of the vehicle or adaptive cruise control system of the
vehicle or lane change departure warning system of the vehicle or
traffic sign recognition system of the vehicle or driver assistance
system of the vehicle the like).
[0015] The image processor of the vision system 12 is operable to
process captured image data, such as to detect and identify objects
forward (and optionally sideward and/or rearward) of the vehicle
during normal operation of the vehicle. In poor visibility
conditions, such as foggy conditions and/or heavy snow fall
conditions or the like, objects may be difficult for the driver to
see and may be difficult even for the image processor to detect,
even when image processing algorithms for lens pollution detection
(such as similar to that described in U.S. provisional application
Ser. No. 61/616,126, filed Mar. 27, 2012, which is hereby
incorporated herein by reference in its entirety) come into use.
For example, and with reference to image "A" in FIG. 2, during low
visibility conditions, such as fog conditions as shown, it is
difficult for the driver of the vehicle to detect the person and
dog at the side of the road ahead of the vehicle and beyond the
principal illumination area of the vehicle headlamps (set at low
beams for seeing in the fog). The image processor may process the
image to detect objects, but, and with reference to image "B" in
FIG. 2, normal image processing may not detect the object of
interest (the person and dog in this example) due to the poor
visibility conditions. Typically, the object detection may not work
feasibly when the image contrast falls under a certain level. In
order to increase the detectability of such objects in poor
visibility conditions, the vision system of the present invention
is operable to enhance or increase the contrast of the captured
images so that any objects in the field of view of the camera are
darkened to enhance the detectability of the objects by the image
processor or to ease the visibility of objects to the driver of the
vehicle.
[0016] As can be seen with reference to images "C" through "F" in
FIG. 2, as the contrast is increased, the side markers or posts
along the side of the road and the object of interest (the person
and dog in this example) become darker and, in this example, the
object moves relative to other fixed objects in the captured images
(see images B-F in FIG. 2 and note that the person and dog approach
the fixed road marker in the captured images), and thus the image
processor can detect the presence of the fixed and moving objects
and determine if they are objects of interest to the driver of the
vehicle and generate the appropriate signal responsive to such
detection and determination or identification. For example, the
system, responsive to such an object detection, may generate an
alert to the driver or may adjust the headlamps accordingly or may
display the detected object on a display screen for viewing by the
driver (particularly for backup assist systems where the object is
detected rearward of the vehicle during a reversing maneuver).
Thus, by increasing the contrast in captured images, the vision
system can enhance detection of objects in the camera's field of
view that may otherwise go undetected. The system may be operable
to increase the contrast in the captured images responsive to a
user input or to a detection or determination of a low visibility
condition, such as responsive to a signal from a rain sensor or the
like that is indicative of detection of a foggy condition or such
as responsive to image processing of the captured images to
determine that the vehicle is in foggy driving conditions (such as
by utilizing aspects of the vision systems described in U.S. Pat.
Nos. 4,973,844; 5,796,094; 5,877,897 and 6,353,392, which are
hereby incorporated herein by reference in their entireties).
[0017] It is known to provide image contrast enhancing for
photographs (such as photographs taken by hand held digital cameras
or astronomical telescopes or the like), and such enhancements may
be done by known computer based tools for editing images. Today,
nearly every operating system, library, presenting program and/or
the like provides at least basic image editing functions.
Professional photo editing programs like CoralDRAW.RTM., Gimp.RTM.
or Adobe Photoshop.degree. provide a wide range of image editing
and enhancing features. Typically used for contrast enhancing is
the editing of the contrast histogram. This can be used to expose
objects stronger. A function used especially to do this is
"Contrast Enhancement through Localized Histogram Equalization"
(see Cromwell-intl.com: http://www.cromwell-intl.com/3d/histogram/,
which is hereby incorporated herein by reference in its entirety).
Even night images can become contrast enhanced in a way that low
illuminated objects turn out more visible. Such algorithms used in
consumer computer programs for image enhancing are typically used
in individual pictures, and are not meant to be used in real time
applications.
[0018] Image quality improvement in poor visibility conditions is
known from airborne weather surveillance pictures for reworking
pictures taken in cloudy (foggy) situations. The best results were
achieved by Oakley et al. when contrast enhancement algorithm in
conjunction with a temporal filters came into use (see Image
Processing, IEEE; "Improving Image Quality in Poor Visibility
Conditions Using a Physical Model for Contrast Degradation,"
http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=660994, by
Oakley, J. P. and Satherley, B. L., February 1998, which is hereby
incorporated herein by reference in its entirety). The base was a
physical model on fog reflection.
[0019] Attempts have been made to do video contrast enhancements
such as in "Contrast Enhancement Using Brightness Preserving
Bi-Histogram Equalization" by Yeong-Taeg Kim (Consumer Electronics:
IEEE: "Contrast Enhancement Using Brightness Preserving
Bi-Histogram Equalization," by Yeong-Taeg Kim, February 1997, which
is hereby incorporated herein by reference in its entirety). This
requires real time processing. Demand for this was and is in
applications for the likes of television images, images providing
medical devices, military engineering and/or the like, and Kim et
al. suggested "Partially Overlapped Sub-Block Histogram
Equalization" to be used in cameras (Circuits and Systems for Video
Technology, IEEE: "Partially Overlapped Sub-Block Histogram
Equalization"
http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=915354, by
Joung-Youn Kim, Lee-Sup Kim and Seung-Ho Hwang, April 2001, which
is hereby incorporated herein by reference in its entirety). Also,
Marsi et al. were able to simplify algorithms by attempting
recursive rational filters (Imaging Systems and Techniques, 2004;
IEEE International Workshop: "Real Time Video Contrast Enhancement
by Using Recursive Rational Filter,"
http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=1397276, by
Marsi, S., Ramponi, G. and Carrato, S., May 14, 2004, which is
hereby incorporated herein by reference in its entirety), and Wang
et al. suggested the use of weighted thresholded histogram
equalization for fast processing (Consumer Electronics, IEEE: "Real
Time Video Contrast Enhancement by using Weighted Thresholded
Histogram Equalization"
http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=4266969, by
Qing Wang and Ward, R. K., May 2007, which is hereby incorporated
herein by reference in its entirety). Another challenge is the
noise, a common problem on electronic cameras; Starck et al.
published a procedure to do noise reduction by curvelet transforms
in 2003 (Image Processing, IEEE: "Gray and Color Image Contrast
Enhancement by the Curvelet Transform,"
http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=1208320, by
Starck, J.-L., Murtagh, F., Candes, E. J. and Donoho, D. L., Jun.
2003, which is hereby incorporated herein by reference in its
entirety).
[0020] It is also known to use infrared systems or low light
amplifying systems in vehicles. Earlier systems have used infrared
cameras alone, and some systems additionally use infrared
headlights to light up the area in front of the vehicle (invisible
for the human eye) which makes that area easier to detect with the
infrared camera. Infrared cameras may provide enhanced performance
in object detection in dense fog conditions due to its physical
principal and the detected wave length have the intrinsic property
to interfuse fog, so objects in fog can be detected and/or
visualized.
[0021] State of the art automotive driver assistance systems
typically provide the driver with useful information of the
vehicle's environment, including the traffic or objects in front
of, to the side of and rearward of the vehicle. Typically, there
are additional warnings or image overlays for highlighting hazards,
especially those in the driving direction of the vehicle and in the
anticipated path of travel of the vehicle. Obstacles or pedestrians
that are in the way or path of the vehicle or tend to step into the
path of the vehicle may be highlighted. Systems which also do
active interventions such as braking or collision avoidance
maneuvers are also known. For distinguishing pedestrians from other
objects and for predetermining their walking direction and speed,
the detected objects need to be tracked over a certain time. Also,
analyzing shapes or markers of walking or standing pedestrians is
known in the field of automotive vision systems and image
processing. Due to the vehicle's own movement, the objects in the
captured images flow or move over successively captured images
(optical flow). For example, external or outside objects (even
stationary objects) move through the images taken from a front
facing vehicle camera as the vehicle travels along the road.
Algorithms for tracking objects under driving conditions are also
known. When a vehicle drives through a turn, the optical flow also
behaves in a turned manner. That turn can be anticipated by the
knowledge of the steering wheel's angle and a kinematic model of
the vehicle's curve behavior. The optical flow speed directly
translates from the vehicle's ground speed given by the odometer.
Alternatively known algorithms may determine the optical flow
direct from the image flow without the previous mentioned input
from the vehicle.
[0022] For enabling the above mentioned pedestrian and obstacle
acknowledging and tracking algorithm to work properly, especially
to be able to highlight a hazard or warn the driver or intervene
(such as via braking or cruise control adjustment or the like), it
is necessary to receive sufficient images. In foggy driving
conditions or during heavy snow fall driving conditions, cameras in
the visible spectrum deliver images of insufficient quality. The
present invention provides enhanced image quality of visible
spectrum cameras, especially the dynamic range of the resulting
image, so that the driver assist system algorithms can work
properly and/or display the processed image to the driver as an
improvement to his or her view in such limited visibility
conditions. This is achieved without the need of additional cameras
using different light spectrums (such as infrared sensitive cameras
or the like) or other sensors for the same purpose or high dynamic
range (HDR) cameras.
[0023] The present invention thus provides enhanced image quality
in poor visibility conditions captured by a non HDR camera by
amplifying the contrast details in the captured images by
generating a pseudo HDR image out of current and historical image
components by tone mapping. The system then tracks the contrast
thresholds/features within the captured images with respect to the
image flow caused by the vehicle's movement. This process is
repeated on a frame-by-frame basis to detect and identify objects
in the camera's forward field of view, as can be seen in FIGS. 5
and 6. At every loop the historically (previously enhanced) image
(Image.sub.h(t.sub.0-n))) passes two individual image transfer
functions and then becomes superpositioned (or mapped, merged,
blended or stacked) by the currently captured frame (Image
t.sub.0)). This tone mapping method is called image stacking,
exposure fusion or exposure blending. The mapping ratio of how much
of the historical image (Image.sub.h(t.sub.0-n))) becomes kept and
how much of the current image (Image t.sub.0)) becomes mapped in is
freely selectable between 0 and 1. In the example in FIG. 5,
20%/80% was chosen for a data frame rate of 30 frames per second.
Slower frame rates might require a shift into a stronger influence
of (Image t.sub.0)). The used image enhancements shall not be
limited to these shown in the example of FIGS. 5 and 6.
[0024] The brightness transfer function A (FIG. 4(a)) enhances the
brighter areas and dampens the darker ones. This equates to a
histogram spread (Dynamic Range Increase), such as shown in FIGS.
3(a) and 3(b), of the historically image (Image.sub.h(t.sub.0-n))).
The brightness transfer function B (FIG. 4(b)) decreases the medium
illuminated areas of the currently captured image (Image t.sub.0)).
The upper end is at less than 100%. The overall illumination
becomes decreased by this transfer function. FIG. 7 shows that
already after this step the discrimination between the object
(person with dog) and surrounding (foggy) area is improved. By
mapping/stacking an illumination reduced image scene (currently
captured images) on top of a contrast enhanced image (historical
image), the dynamic range of the image increases as to be seen in
FIGS. 3(a) and 3(b). Overexposed areas appear less bright and
underexposed more bright which leads to acknowledge details in the
scene easier (see FIG. 7). After consecutive loops it may come to a
blooming effect or halo at the borderline of areas with high
contrast. This effect may be enhanced by some blurring which is
caused by unavoidable inaccuracy of the distorting, turning,
cropping and moving of the currently captured image to the
historical scene.
[0025] The result of this image processing and tracking of the
features with respect to the optical flow and the vehicle movement
is shown in principle in FIG. 2 (and discussed above). The
algorithm based on already established image processing procedures
(non-automotive, image enhancements of photographs and `image
registration` and the like), such as tonal value
splitting/-buckling/-limiting, histogram equalization and the like,
as simplified can be seen with reference to FIGS. 3(a) and
3(b).
[0026] Because the yet to be processed images are captured by a
camera on a moving vehicle, it is necessary that the optical flow
and the according information or data of objects (both steady or
moving) moving through the images, including the vehicle speed, the
steering angle of the vehicle and the like, be taken into account.
There may be a model of the vehicle's cinematic mathematical
equations. Its results may be stored in a look up table. The
camera's or cameras parameters as like mounting position and
viewing angle optical properties may be reflected in that
(combined) look up table or in another mathematical model or table.
The moving objects/obstacles can thus be distinguished from steady
objects relative to the movement of the vehicle that is equipped
with the camera system or vision system of the present invention.
Object classification may work on further distances by feeding
enhanced image data. Further algorithms may process the image data
and may indicate hazards or the like, and/or may actively intervene
to avoid collisions and the like. The image enhancing algorithm may
find use in processing multiple camera images separate or by
processing a stitched image which may be arranged as a vehicle top
view image or the like.
[0027] The imaging sensor and its photosensor array may comprise
any suitable camera or sensing device, such as, for example, an
array of a plurality of photosensor elements arranged in 640
columns and 480 rows (a 640.times.480 imaging array), with a
respective lens focusing images onto respective portions of the
array. The photosensor array may comprise a plurality of
photosensor elements arranged in a photosensor array having rows
and columns. The logic and control circuit of the imaging sensor
may function in any known manner, such as in the manner described
in U.S. Pat. Nos. 5,550,677; 5,877,897; 6,498,620; 5,670,935;
5,796,094 and/or 6,396,397, and/or U.S. provisional applications,
Ser. No. 61/696,416, filed Sep. 4, 2012; Ser. No. 61/682,995, filed
Aug. 14, 2012; Ser. No. 61/682,486, filed Aug. 13, 2012; Ser. No.
61/680,883, filed Aug. 8, 2012; Ser. No. 61/678,375, filed Aug. 1,
2012; Ser. No. 61/676,405, filed Jul. 27, 2012; Ser. No.
61/666,146, filed Jun. 29, 2012; Ser. No. 61/653,665, filed May 31,
2012; Ser. No. 61/653,664, filed May 31, 2012; Ser. No. 61/648,744,
filed May 18, 2012; Ser. No. 61/624,507, filed Apr. 16, 2012; Ser.
No. 61/616,126, filed Mar. 27, 2012; Ser. No. 61/615,410, filed
Mar. 26, 2012; Ser. No. 61/613,651, filed Mar. 21, 2012; Ser. No.
61/607,229, filed Mar. 6, 2012; Ser. No. 61/605,409, filed Mar. 1,
2012; Ser. No. 61/602,878, filed Feb. 24, 2012; Ser. No.
61/602,876, filed Feb. 24, 2012; Ser. No. 61/600,205, filed Feb.
17, 2012; Ser. No. 61/588,833, filed Jan. 20, 2012; Ser. No.
61/583,381, filed Jan. 5, 2012; Ser. No. 61/579,682, filed Dec. 23,
2011; Ser. No. 61/570,017, filed Dec. 13, 2011; Ser. No.
61/568,791, filed Dec. 9, 2011; Ser. No. 61/567,446, filed Dec. 6,
2011; Ser. No. 61/559,970, filed Nov. 15, 2011; and/or Ser. No.
61/552,167, filed Oct. 27, 2011, and/or PCT Application No.
PCT/CA2012/000378, filed Apr. 25, 2012, and published Nov. 1, 2012
as International Publication No. WO 2012/145822, and/or PCT
Application No. PCT/US2012/056014, filed Sep. 19, 2012, and
published Mar. 28, 2013 as International Publication No. WO
2013/043661, and/or PCT Application No. PCT/US2012/048800, filed
Jul. 30, 2012, and published Feb. 7, 2013 as International
Publication No. WO 2013/019707, and/or PCT Application No.
PCT/US2012/048110, filed Jul. 25, 2012, and published Jan. 31, 2013
as International Publication No. WO 2013/016409, and/or U.S. patent
application Ser. No. 13/534,657, filed Jun. 27, 2012, and published
Jan. 3, 2013 as U.S. Publication No. US-2013-0002873, which are all
hereby incorporated herein by reference in their entireties. The
system may communicate with other communication systems via any
suitable means, such as by utilizing aspects of the systems
described in PCT Application No. PCT/US10/038477, filed Jun. 14,
2010, and/or U.S. patent application Ser. No. 13/202,005, filed
Aug. 17, 2011, now U.S. Pat. No. 9,126,525, and/or U.S. provisional
applications, Ser. No. 61/650,667, filed May 23, 2012; Ser. No.
61/579,682, filed Dec. 23, 2011; Ser. No. 61/565,713, filed Dec. 1,
2011, which are hereby incorporated herein by reference in their
entireties.
[0028] The imaging device and control and image processor and any
associated illumination source, if applicable, may comprise any
suitable components, and may utilize aspects of the cameras and
vision systems described in U.S. Pat. Nos. 5,550,677; 5,877,897;
6,498,620; 5,670,935; 5,796,094; 6,396,397; 6,806,452; 6,690,268;
7,005,974; 7,123,168; 7,004,606; 6,946,978; 7,038,577; 6,353,392;
6,320,176; 6,313,454 and 6,824,281, and/or International
Publication No. WO 2010/099416, published Sep. 2, 2010, and/or PCT
Application No. PCT/US10/47256, filed Aug. 31, 2010, and/or U.S.
patent application Ser. No. 12/508,840, filed Jul. 24, 2009, and
published Jan. 28, 2010 as U.S. Pat. Publication No. US
2010-0020170; and/or PCT Application No. PCT/US2012/048110, filed
Jul. 25, 2012, and published Jan. 31, 2013 as International
Publication No. WO 2013/016409, and/or U.S. patent application Ser.
No. 13/534,657, filed Jun. 27, 2012, and published Jan. 3, 2013 as
U.S. Publication No. US-2013-0002873, which are all hereby
incorporated herein by reference in their entireties. The camera or
cameras may comprise any suitable cameras or imaging sensors or
camera modules, and may utilize aspects of the cameras or sensors
described in U.S. patent applications, Ser. No. 12/091,359, filed
Apr. 24, 2008 and published Oct. 1, 2009 as U.S. Publication No.
US-2009-0244361; and/or Ser. No. 13/260,400, filed Sep. 26, 2011,
now U.S. Pat. No. 8,542,451, and/or U.S. Pat. Nos. 7,965,336 and/or
7,480,149, which are hereby incorporated herein by reference in
their entireties. The imaging array sensor may comprise any
suitable sensor, and may utilize various imaging sensors or imaging
array sensors or cameras or the like, such as a CMOS imaging array
sensor, a CCD sensor or other sensors or the like, such as the
types described in U.S. Pat. Nos. 5,550,677; 5,670,935; 5,760,962;
5,715,093; 5,877,897; 6,922,292; 6,757,109; 6,717,610; 6,590,719;
6,201,642; 6,498,620; 5,796,094; 6,097,023; 6,320,176; 6,559,435;
6,831,261; 6,806,452; 6,396,397; 6,822,563; 6,946,978; 7,339,149;
7,038,577; 7,004,606; 7,720,580 and/or 7,965,336, and/or PCT
Application No. PCT/US2008/076022, filed Sep. 11, 2008 and
published Mar. 19, 2009 as International Publication No. WO
2009/036176, and/or PCT Application No. PCT/US2008/078700, filed
Oct. 3, 2008 and published Apr. 9, 2009 as International
Publication No. WO 2009/046268, which are all hereby incorporated
herein by reference in their entireties.
[0029] The camera module and circuit chip or board and imaging
sensor may be implemented and operated in connection with various
vehicular vision-based systems, and/or may be operable utilizing
the principles of such other vehicular systems, such as a vehicle
headlamp control system, such as the type disclosed in U.S. Pat.
Nos. 5,796,094; 6,097,023; 6,320,176; 6,559,435; 6,831,261;
7,004,606; 7,339,149 and/or 7,526,103, which are all hereby
incorporated herein by reference in their entireties, a rain
sensor, such as the types disclosed in commonly assigned U.S. Pat.
Nos. 6,353,392; 6,313,454; 6,320,176 and/or 7,480,149, which are
hereby incorporated herein by reference in their entireties, a
vehicle vision system, such as a forwardly, sidewardly or
rearwardly directed vehicle vision system utilizing principles
disclosed in U.S. Pat. Nos. 5,550,677; 5,670,935; 5,760,962;
5,877,897; 5,949,331; 6,222,447; 6,302,545; 6,396,397; 6,498,620;
6,523,964; 6,611,202; 6,201,642; 6,690,268; 6,717,610; 6,757,109;
6,802,617; 6,806,452; 6,822,563; 6,891,563; 6,946,978 and/or
7,859,565, which are all hereby incorporated herein by reference in
their entireties, a trailer hitching aid or tow check system, such
as the type disclosed in U.S. Pat. No. 7,005,974, which is hereby
incorporated herein by reference in its entirety, a reverse or
sideward imaging system, such as for a lane change assistance
system or lane departure warning system or for a blind spot or
object detection system, such as imaging or detection systems of
the types disclosed in U.S. Pat. Nos. 7,881,496; 7,720,580;
7,038,577; 5,929,786 and/or 5,786,772, and/or U.S. provisional
applications, Ser. No. 60/628,709, filed Nov. 17, 2004; Ser. No.
60/614,644, filed Sep. 30, 2004; Ser. No. 60/618,686, filed Oct.
14, 2004; Ser. No. 60/638,687, filed Dec. 23, 2004, which are
hereby incorporated herein by reference in their entireties, a
video device for internal cabin surveillance and/or video telephone
function, such as disclosed in U.S. Pat. Nos. 5,760,962; 5,877,897;
6,690,268 and/or 7,370,983, and/or U.S. patent application Ser. No.
10/538,724, filed Jun. 13, 2005 and published Mar. 9, 2006 as U.S.
Publication No. US-2006-0050018, which are hereby incorporated
herein by reference in their entireties, a traffic sign recognition
system, a system for determining a distance to a leading or
trailing vehicle or object, such as a system utilizing the
principles disclosed in U.S. Pat. Nos. 6,396,397 and/or 7,123,168,
which are hereby incorporated herein by reference in their
entireties, and/or the like.
[0030] Optionally, the circuit board or chip may include circuitry
for the imaging array sensor and or other electronic accessories or
features, such as by utilizing compass-on-a-chip or EC
driver-on-a-chip technology and aspects such as described in U.S.
Pat. Nos. 7,255,451 and/or 7,480,149; and/or U.S. patent
applications, Ser. No. 11/226,628, filed Sep. 14, 2005 and
published Mar. 23, 2006 as U.S. Publication No. US-2006-0061008,
and/or Ser. No. 12/578,732, filed Oct. 14, 2009, now U.S. Pat. No.
9,487,144, which are hereby incorporated herein by reference in
their entireties.
[0031] Optionally, the vision system may include a display for
displaying images captured by one or more of the imaging sensors
for viewing by the driver of the vehicle while the driver is
normally operating the vehicle. Optionally, for example, the vision
system may include a video display device disposed at or in the
interior rearview mirror assembly of the vehicle, such as by
utilizing aspects of the video mirror display systems described in
U.S. Pat. No. 6,690,268 and/or U.S. patent application Ser. No.
13/333,337, filed Dec. 21, 2011, now U.S. Pat. No. 9,264,672, which
are hereby incorporated herein by reference in their entireties.
The video mirror display may comprise any suitable devices and
systems and optionally may utilize aspects of the compass display
systems described in U.S. Pat. Nos. 7,370,983; 7,329,013;
7,308,341; 7,289,037; 7,249,860; 7,004,593; 4,546,551; 5,699,044;
4,953,305; 5,576,687; 5,632,092; 5,677,851; 5,708,410; 5,737,226;
5,802,727; 5,878,370; 6,087,953; 6,173,508; 6,222,460; 6,513,252
and/or 6,642,851, and/or European patent application, published
Oct. 11, 2000 under Publication No. EP 0 1043566, and/or U.S.
patent application Ser. No. 11/226,628, filed Sep. 14, 2005 and
published Mar. 23, 2006 as U.S. Publication No. US-2006-0061008,
which are all hereby incorporated herein by reference in their
entireties. Optionally, the video mirror display screen or device
may be operable to display images captured by a rearward viewing
camera of the vehicle during a reversing maneuver of the vehicle
(such as responsive to the vehicle gear actuator being placed in a
reverse gear position or the like) to assist the driver in backing
up the vehicle, and optionally may be operable to display the
compass heading or directional heading character or icon when the
vehicle is not undertaking a reversing maneuver, such as when the
vehicle is being driven in a forward direction along a road (such
as by utilizing aspects of the display system described in PCT
Application No. PCT/US2011/056295, filed Oct. 14, 2011 and
published Apr. 19, 2012 as International Publication No. WO
2012/051500, which is hereby incorporated herein by reference in
its entirety).
[0032] Optionally, the vision system (utilizing the forward facing
camera and a rearward facing camera and other cameras disposed at
the vehicle with exterior fields of view) may be part of or may
provide a display of a top-down view or birds-eye view system of
the vehicle or a surround view at the vehicle, such as by utilizing
aspects of the vision systems described in PCT Application No.
PCT/US10/25545, filed Feb. 26, 2010 and published on Sep. 2, 2010
as International Publication No. WO 2010/099416, and/or PCT
Application No. PCT/US10/47256, filed Aug. 31, 2010 and published
Mar. 10, 2011 as International Publication No. WO 2011/028686,
and/or PCT Application No. PCT/US11/62755, filed Dec. 1, 2011 and
published Jun. 7, 2012 as International Publication No. WO
2012-075250, and/or PCT Application No. PCT/US2012/048993, filed
Jul. 31, 2012, and published Feb. 7, 2013 as International
Publication No. WO 2013/019795, and/or PCT Application No.
PCT/CA2012/000378, filed Apr. 25, 2012, and published Nov. 1, 2012
as International Publication No. WO 2012/145822, and/or U.S. patent
application Ser. No. 13/333,337, filed Dec. 21, 2011, now U.S. Pat.
No. 9,264,672, and/or U.S. provisional applications, Ser. No.
61/615,410, filed Mar. 26, 2012; Ser. No. 61/588,833, filed Jan.
20, 2012; Ser. No. 61/570,017, filed Dec. 13, 2011; Ser. No.
61/568,791, filed Dec. 9, 2011; Ser. No. 61/559,970, filed Nov. 15,
2011; Ser. No. 61/540,256, filed Sep. 28, 2011, which are hereby
incorporated herein by reference in their entireties.
[0033] Optionally, the video mirror display may be disposed
rearward of and behind the reflective element assembly and may
comprise a display such as the types disclosed in U.S. Pat. Nos.
5,530,240; 6,329,925; 7,855,755; 7,626,749; 7,581,859; 7,338,177;
7,274,501; 7,255,451; 7,195,381; 7,184,190; 5,668,663; 5,724,187
and/or 6,690,268, and/or in U.S. patent applications, Ser. No.
11/226,628, filed Sep. 14, 2005 and published Mar. 23, 2006 as U.S.
Publication No. US-2006-0061008; and/or Ser. No. 10/538,724, filed
Jun. 13, 2005 and published Mar. 9, 2006 as U.S. Publication No.
US-2006-0050018, which are all hereby incorporated herein by
reference in their entireties. The display is viewable through the
reflective element when the display is activated to display
information. The display element may be any type of display
element, such as a vacuum fluorescent (VF) display element, a light
emitting diode (LED) display element, such as an organic light
emitting diode (OLED) or an inorganic light emitting diode, an
electroluminescent (EL) display element, a liquid crystal display
(LCD) element, a video screen display element or backlit thin film
transistor (TFT) display element or the like, and may be operable
to display various information (as discrete characters, icons or
the like, or in a multi-pixel manner) to the driver of the vehicle,
such as passenger side inflatable restraint (PSIR) information,
tire pressure status, and/or the like. The mirror assembly and/or
display may utilize aspects described in U.S. Pat. Nos. 7,184,190;
7,255,451; 7,446,924 and/or 7,338,177, which are all hereby
incorporated herein by reference in their entireties. The
thicknesses and materials of the coatings on the substrates of the
reflective element may be selected to provide a desired color or
tint to the mirror reflective element, such as a blue colored
reflector, such as is known in the art and such as described in
U.S. Pat. Nos. 5,910,854; 6,420,036 and/or 7,274,501, which are
hereby incorporated herein by reference in their entireties.
[0034] Optionally, the display or displays and any associated user
inputs may be associated with various accessories or systems, such
as, for example, a tire pressure monitoring system or a passenger
air bag status or a garage door opening system or a telematics
system or any other accessory or system of the mirror assembly or
of the vehicle or of an accessory module or console of the vehicle,
such as an accessory module or console of the types described in
U.S. Pat. Nos. 7,289,037; 6,877,888; 6,824,281; 6,690,268;
6,672,744; 6,386,742 and 6,124,886, and/or U.S. patent application
Ser. No. 10/538,724, filed Jun. 13, 2005 and published Mar. 9, 2006
as U.S. Publication No. US-2006-0050018, which are hereby
incorporated herein by reference in their entireties.
[0035] The display or displays may comprise a video display and may
utilize aspects of the video display devices or modules described
in U.S. Pat. Nos. 6,690,268; 7,184,190; 7,274,501; 7,370,983;
7,446,650 and/or 7,855,755, and/or U.S. patent application Ser. No.
10/538,724, filed Jun. 13, 2005 and published Mar. 9, 2006 as U.S.
Publication No. US-2006-0050018, which are all hereby incorporated
herein by reference in their entireties. The video display may be
operable to display images captured by one or more imaging sensors
or cameras at the vehicle.
[0036] Changes and modifications to the specifically described
embodiments may be carried out without departing from the
principles of the present invention, which is intended to be
limited only by the scope of the appended claims as interpreted
according to the principles of patent law.
* * * * *
References