U.S. patent application number 12/654432 was filed with the patent office on 2010-09-30 for video processing apparatus.
This patent application is currently assigned to Fujitsu Limited. Invention is credited to Toshiaki Gomi, Jun Kawai, Hiroshi Yamada, Katsutoshi Yano.
Application Number | 20100245607 12/654432 |
Document ID | / |
Family ID | 42312826 |
Filed Date | 2010-09-30 |
United States Patent
Application |
20100245607 |
Kind Code |
A1 |
Kawai; Jun ; et al. |
September 30, 2010 |
Video processing apparatus
Abstract
The video processing apparatus includes: a pattern information
storing unit that stores use part information and location
information; a plurality of video inputting units that input each
of a plurality of video data; a video data generating unit that
generates a plurality of types of generated video data; a generated
video pattern selecting unit that selects a pattern of generated
video data to be generated; a process unit that stores the use part
of each video data in a storing unit; and a video outputting unit
that coordinate-converts the generated combined video data with
coordinate conversion data, generates the generated video data, and
outputs the generated video data to the display unit. The use part
information indicates a use part of each video data used for the
generated video data, and the location information includes
reduction ratio information of the use part of the each video
data.
Inventors: |
Kawai; Jun; (Kawasaki,
JP) ; Yano; Katsutoshi; (Kawasaki, JP) ; Gomi;
Toshiaki; (Kawasaki, JP) ; Yamada; Hiroshi;
(Kawasaki, JP) |
Correspondence
Address: |
STAAS & HALSEY LLP
SUITE 700, 1201 NEW YORK AVENUE, N.W.
WASHINGTON
DC
20005
US
|
Assignee: |
Fujitsu Limited
Kawasaki
JP
|
Family ID: |
42312826 |
Appl. No.: |
12/654432 |
Filed: |
December 18, 2009 |
Current U.S.
Class: |
348/218.1 ;
348/E5.024 |
Current CPC
Class: |
G06T 11/00 20130101;
G06T 11/003 20130101; G06T 3/4038 20130101; B60R 2300/607 20130101;
G06T 3/60 20130101; B60R 2300/303 20130101; H04N 7/181 20130101;
B60R 1/00 20130101; B60R 2300/105 20130101; H04N 13/398 20180501;
G06T 3/0062 20130101; H04N 13/373 20180501; G06T 11/60 20130101;
H04N 13/189 20180501 |
Class at
Publication: |
348/218.1 ;
348/E05.024 |
International
Class: |
H04N 5/225 20060101
H04N005/225 |
Foreign Application Data
Date |
Code |
Application Number |
Mar 31, 2009 |
JP |
2009-088370 |
Claims
1. A video processing apparatus capable of cooperating with a
plurality of capturing unit and a display unit, the video
processing apparatus comprising: a pattern information storing unit
that stores use part information and location information; a
plurality of video inputting units that input each of a plurality
of video data captured with the plurality of capturing units; a
video data generating unit that generates a plurality of types of
generated video data; a generated video pattern selecting unit that
selects a pattern of generated video data to be generated in the
video data generating unit; a process unit that stores the use part
of each video data in a storing unit after reducing the use part of
each video data on the basis of the plurality of video data and
location information according to a selected pattern of generated
video data and generates the combined video data; and a video
outputting unit that coordinate-converts the generated combined
video data with coordinate conversion data preliminarily calculated
with respect to each pattern of the plurality of types of generated
video data, generates the generated video data, and outputs the
generated video data to the display unit, wherein the use part
information indicates a use part of each video data used for the
generated video data, and the location information includes
reduction ratio information of the use part of the each video
data.
2. The video processing apparatus according to claim 1, further
comprising: a coordinate value calculating unit that calculates the
use part information that indicates a use part of each video data
used for the generated video data with reference to the coordinate
conversion data; and a location calculating unit that calculates
the location information including the reduction ratio information
of the use part of the each video data so that a pixel expansion in
generating the generated video data by coordinate-converting the
combined video data is small and within a given size of the
combined video data, wherein the process unit stores the plurality
of video data in the storing unit after reducing a use part of each
video data on the basis of the use part information and the
location information, and generates the combined video data.
3. The video processing apparatus according to claim 1, wherein the
location information is calculated by calculating a reduction ratio
and a location on the combined video data of a use part of each
video data so that one of an average value, an intermediate value,
and a maximum value or a minimum value of a pixel expansion ratio
of processing units of the coordinate conversion or pixel units is
equivalent between the use parts of each video data and within a
given size of the combined video data when the combined video data
is coordinate-converted to the generated video data.
4. The video processing apparatus according to claim 2, wherein the
location information is calculated by calculating a reduction ratio
and a location on the combined video data of a use part of each
video data so that one of an average value, an intermediate value,
and a maximum value or a minimum value of a pixel expansion ratio
of processing units of the coordinate conversion or pixel units is
equivalent between the use parts of each video data and within a
given size of the combined video data when the combined video data
is coordinate-converted to the generated video data.
5. The video processing apparatus according claim 1, wherein the
location information is calculated after a reduction ratio of a use
part of a given high-priority video is fixed.
6. The video processing apparatus according to claim 2, wherein the
location information is calculated after a reduction ratio of a use
part of a given high-priority video is fixed.
7. The video processing apparatus according to claim 3, wherein the
location information is calculated after a reduction ratio of a use
part of a given high-priority video is fixed.
8. The video processing apparatus according to claim 4, wherein the
location information is calculated after a reduction ratio of a use
part of a given high-priority video is fixed.
9. The video processing apparatus according to claim 1, further
comprising: a generated video pattern selection inputting unit that
determines a pattern of generated video data to be generated in the
video data generating unit; a combined video pattern selection
notification unit that notifies the video processing apparatus of a
pattern of the combined video data corresponding to a determined
pattern of the generated video data; and a video processing unit
that generates the generated video data by using the combined video
data input from the video processing apparatus.
10. The video processing apparatus according to claim 2, further
comprising: a generated video pattern selection inputting unit that
determines a pattern of generated video data to be generated in the
video data generating unit; a combined video pattern selection
notification unit that notifies the video processing apparatus of a
pattern of the combined video data corresponding to a determined
pattern of the generated video data; and a video processing unit
that generates the generated video data by using the combined video
data input from the video processing apparatus.
11. The video processing apparatus according to claim 3, further
comprising: a generated video pattern selection inputting unit that
determines a pattern of generated video data to be generated in the
video data generating unit; a combined video pattern selection
notification unit that notifies the video processing apparatus of a
pattern of the combined video data corresponding to a determined
pattern of the generated video data; and a video processing unit
that generates the generated video data by using the combined video
data input from the video processing apparatus.
12. The video processing apparatus according to claim 4, further
comprising: a generated video pattern selection inputting unit that
determines a pattern of generated video data to be generated in the
video data generating unit; a combined video pattern selection
notification unit that notifies the video processing apparatus of a
pattern of the combined video data corresponding to a determined
pattern of the generated video data; and a video processing unit
that generates the generated video data by using the combined video
data input from the video processing apparatus.
13. The video processing apparatus according to claim 5, further
comprising: a generated video pattern selection inputting unit that
determines a pattern of generated video data to be generated in the
video data generating unit; a combined video pattern selection
notification unit that notifies the video processing apparatus of a
pattern of the combined video data corresponding to a determined
pattern of the generated video data; and a video processing unit
that generates the generated video data by using the combined video
data input from the video processing apparatus.
14. The video processing apparatus according to claim 6, further
comprising: a generated video pattern selection inputting unit that
determines a pattern of generated video data to be generated in the
video data generating unit; a combined video pattern selection
notification unit that notifies the video processing apparatus of a
pattern of the combined video data corresponding to a determined
pattern of the generated video data; and a video processing unit
that generates the generated video data by using the combined video
data input from the video processing apparatus.
15. The video processing apparatus according to claim 7, further
comprising: a generated video pattern selection inputting unit that
determines a pattern of generated video data to be generated in the
video data generating unit; a combined video pattern selection
notification unit that notifies the video processing apparatus of a
pattern of the combined video data corresponding to a determined
pattern of the generated video data; and a video processing unit
that generates the generated video data by using the combined video
data input from the video processing apparatus.
16. The video processing apparatus according to claim 8, further
comprising: a generated video pattern selection inputting unit that
determines a pattern of generated video data to be generated in the
video data generating unit; a combined video pattern selection
notification unit that notifies the video processing apparatus of a
pattern of the combined video data corresponding to a determined
pattern of the generated video data; and a video processing unit
that generates the generated video data by using the combined video
data input from the video processing apparatus.
Description
CROSS-REFERENCE TO RELATED APPLICATION
[0001] This application is based upon and claims the benefit of
priority of the prior Japanese Patent Application No. 2009-088370,
filed on Mar. 31, 2009, the entire contents of which are
incorporated herein by reference.
FIELD
[0002] A certain aspect of the embodiments discussed herein is
related to a video processing apparatus.
BACKGROUND
[0003] There is provided an apparatus that supports driver's safe
driving, as the apparatus captures videos of an area around a
vehicle that is out of driver's line of vision by using a camera
mounted on the vehicle and displays captured videos on a monitor
mounted in the vehicle interior.
[0004] Japanese Laid-Open Patent Publication No. 2002-027448
discloses an art to generate a synthesized video with captured
images captured with multiple cameras.
SUMMARY
[0005] According to an aspect of the present invention, there is
provided a video processing apparatus capable of cooperating with a
plurality of capturing unit and a display unit, the video
processing apparatus including: a pattern information storing unit
that stores use part information and location information; a
plurality of video inputting units that input each of a plurality
of video data captured with the plurality of capturing units; a
video data generating unit that generates a plurality of types of
generated video data; a generated video pattern selecting unit that
selects a pattern of generated video data to be generated in the
video data generating unit; a process unit that stores the use part
of each video data in a storing unit after reducing the use part of
each video data on the basis of the plurality of video data and
location information according to a selected pattern of generated
video data and generates the combined video data; and a video
outputting unit that coordinate-converts the generated combined
video data with coordinate conversion data preliminarily calculated
with respect to each pattern of the plurality of types of generated
video data, generates the generated video data, and outputs the
generated video data to the display unit, wherein the use part
information indicates a use part of each video data used for the
generated video data, and the location information includes
reduction ratio information of the use part of the each video
data.
[0006] The object and advantages of the invention will be realized
and attained by means of the elements and combinations particularly
pointed out in the claims.
[0007] It is to be understood that both the foregoing general
description and the following detailed description are exemplary
and explanatory and are not restrictive of the invention, as
claimed.
BRIEF DESCRIPTION OF DRAWINGS
[0008] FIG. 1 is a diagram illustrating a composition of a video
processing apparatus;
[0009] FIG. 2 is a diagram illustrating locations on which cameras
are mounted in a vehicle;
[0010] FIGS. 3A and 3B are diagrams illustrating an example of
generated video data displayed on a display unit;
[0011] FIGS. 4A and 4B are diagrams illustrating an example of
generated video data displayed on a display unit;
[0012] FIGS. 5A and 5B are diagrams illustrating an example of
generated video data displayed on a display unit;
[0013] FIGS. 6A through 6H are diagrams illustrating patterns of
combined video data used for generating a generated video;
[0014] FIG. 7 is a diagram illustrating compositions of a capturing
unit, a video inputting unit, and a process processing unit;
[0015] FIG. 8 is a diagram illustrating a composition of a video
combining unit, a video outputting unit, and a control unit of a
capture preprocessing unit;
[0016] FIGS. 9A and 9B are diagrams illustrating examples of
setting information recorded in a setting holding unit of a capture
preprocessing unit;
[0017] FIGS. 10A through 10C are diagrams illustrating a process of
a process processing unit;
[0018] FIGS. 11A through 11C are diagrams illustrating a process
continued from FIGS. 10A through 10C of a process processing
unit;
[0019] FIG. 12 is a diagram illustrating a composition of an video
generating unit;
[0020] FIG. 13 is a diagram illustrating a composition of a control
unit of an video generating unit;
[0021] FIGS. 14A and 14B are diagrams illustrating examples of
tentative generated video converting pattern data and generated
video converting pattern data;
[0022] FIGS. 15A and 15B are flowcharts illustrating a process of a
control unit;
[0023] FIGS. 16A through 16C are flowcharts illustrating a process
of a control unit to generate location information and generated
video converting pattern data;
[0024] FIG. 17 is a diagram to explain a shape of video projection
plane, a viewpoint vector, and a display range;
[0025] FIG. 18A is a diagram illustrating a partial video range
representing a use part of a front video, FIG. 18B is a diagram
illustrating a partial video area representing a use part of a rear
video, FIG. 18C is a diagram illustrating a partial video area
representing a use part of a left video, and FIG. 18D is a diagram
illustrating a partial video area representing a use part of a
right video;
[0026] FIG. 19 is a diagram illustrating a pixel expansion ratio
calculated with respect to each polygon;
[0027] FIG. 20 is a diagram illustrating a representative value of
a pixel expansion ratio of each camera video;
[0028] FIG. 21 is a diagram illustrating a determination of whether
a partial video data is within the size of combined video data;
[0029] FIG. 22A illustrates partial video data of front, back,
right, and left video data, and FIG. 22B is a diagram illustrating
how to locate partial video data to come to an aspect ratio of the
combined video data;
[0030] FIGS. 23A through 23D are diagrams illustrating a process of
changing a reduction ratio of partial video data to be enlarged to
one and relocating partial video data within combined video data;
and
[0031] FIG. 24 is a diagram illustrating combined video data in the
case that partial video data is clipped free-form.
DESCRIPTION OF EMBODIMENTS
[0032] As previously described, there is provided an apparatus that
supports driver's safe driving, as the apparatus captures videos of
an area around a vehicle that is out of driver's line of vision by
using a camera mounted on the vehicle and displays captured videos
on a monitor mounted in the vehicle interior.
[0033] Japanese Laid-Open Patent Publication No. 2002-27448
discloses an art to generate a synthesized video with captured
images captured with multiple cameras.
[0034] However, a pattern of the synthesized video synthesized with
videos captured with multiple cameras is one given pattern.
Therefore, it is difficult to switch the synthesized video in
response to, for example, driver's driving situation.
[0035] A description will now be given of an embodiment of the
present invention with reference to the accompanying drawings.
[0036] A composition of this embodiment will be described with
reference to FIG. 1. As illustrated in FIG. 1, a video processing
apparatus 1 of this embodiment is provided with a capturing unit
100, a video processing unit 200, an operation unit 351, and a
display unit 352. The capturing unit 100 includes a first capturing
unit 110, a second capturing unit 120, a third capturing unit 130
and a fourth capturing unit 140. The video processing unit 200
includes a video inputting unit 210, a process processing unit 220,
and a video generating unit 270. A capture preprocessing unit 250
of the process processing unit 220, and the video generating unit
270 are mounted on a same board.
[0037] A description will now be given of the capturing unit
100.
[0038] The first capturing unit 110 through the fourth capturing
unit 140 are provided with a camera mounted on a vehicle, and
outputs videos captured with each camera to the video inputting
unit 210 respectively. Hereinafter, the first capturing unit 110
through the fourth capturing unit 140 will be called cameras 110
through 140 respectively. In this embodiment, as illustrated in
FIG. 2, the camera 110 is installed in front of the vehicle, and
captures the video of the front of the vehicle. The camera 120 is
installed on the left side of the vehicle, and captures the video
of the left side of the vehicle. The camera 130 is installed on the
right side of the vehicle, and captures the video of the right side
of the vehicle. The camera 140 is installed in the rear of the
vehicle, and captures the video of the rear of the vehicle. In this
embodiment, four cameras are installed in the vehicle, but the
number of cameras is not limited to four, and can be three, five,
or six. Cameras can be preliminarily mounted on the vehicle, if
they can work in concert with each other.
[0039] The operation unit 351 receives operation inputs from an
operator. A generated video generated from videos captured with
cameras 110 to 140 is displayed on the display unit 352. The
operator performs an operation to switch the pattern of the
generated video displayed on the display unit 352.
[0040] Display examples (patterns) of generated video displayed on
the display unit 352 will be illustrated in FIGS. 3A through 5B.
FIGS. 6A through 6H illustrate patterns of combined video made with
combination of four videos captured with cameras 110 to 140.
[0041] Videos made with combination of two videos as illustrated in
FIGS. 6A through 6C; videos made with three videos as illustrated
in FIGS. 6D and 6E; videos made with four videos as illustrated in
FIGS. 6F and 6G; and a video made with a single video as
illustrated in FIG. 6H are video-processed by the video processing
unit 200, and displayed on the display unit 352. The video
displayed on the display unit 352 is a generated video processed to
be a stereoscopic display. The stereoscopic generated video is
generated as multiple videos captured with cameras 110 to 140 are
combined and coordinate-transformed (details will be described
later). The combined video illustrated in FIGS. 6A through 6C is
used to generate the generated video with only two videos of the
left and the rear to check for a collision in left turn, for
example. The video made with combination of three videos as
illustrated in FIGS. 6D and 6E is used to generate the generated
video from three camera videos except the front camera video to
look behind the vehicle in the lane change. The video made with
combination of four videos as illustrated in FIGS. 6F and 6G is
used to generate the video that enables the driver to check all
around the vehicle. However, the divisions are changed, as which
camera video is used more to generate the generated video depends
on the viewpoint location of when the generated video is
generated.
[0042] The video inputting unit 210 will now be described. The
video inputting unit 210 is provided with a first video inputting
unit 211, a second video inputting unit 212, a third video
inputting unit 213, and a fourth video inputting unit 214.
[0043] The first video inputting unit 211 is a block receiving a
video captured with the camera 110. The first video inputting unit
211 executes A/D conversion to the received video, and outputs the
A/D converted video data to a first video process unit 231 of a
video process unit 230.
[0044] The second video inputting unit 212 is a block receiving a
video captured with the camera 120. The second video inputting unit
212 executes A/D conversion to the received video, and outputs the
A/D converted video data to a second video process unit 235 of the
video process unit 230.
[0045] The third video inputting unit 213 is a block receiving a
video captured with the camera 130. The third video inputting unit
213 executes A/D conversion to the received video, and outputs the
A/D converted video data to a third video process unit 241 of the
video process unit 230.
[0046] The fourth video inputting unit 214 is a block receiving a
video captured with the camera 140. The fourth video inputting unit
214 executes A/D conversion to the received video, and outputs the
A/D converted video data to a fourth video process unit 245 of the
video process unit 230.
[0047] Then, a description will be given of the process processing
unit 220 with reference to FIG. 7. The process processing unit 220
is provided with the video process unit 230, and the capture
preprocessing unit 250.
[0048] The video process unit 230 is provided with the first video
process unit 231, the second video process unit 235, the third
video process unit 241, and the fourth video process unit 245. The
first video process unit 231 is provided with a first video
clipping unit 232, and a first zooming unit 233. The second video
process unit 235 is provided with a second video clipping unit 236,
and a second zooming unit 243. The third video process unit 241 is
provided with a third video clipping unit 242, and a third zooming
unit 237. The fourth video process unit 245 is provided with a
fourth video clipping unit 246, and a fourth zooming unit 247. The
first video process unit 231 through the fourth video process unit
245 have a same composition. Therefore, hereinafter, the first
video process unit 231 will be described as a representative of
them.
[0049] The first video clipping unit 232 receives the video data
output from the first video inputting unit 211. The first video
clipping unit 232 receives an indication signal that indicates a
clipping position of the video from a capture video partial
management unit 253 described later. The first video clipping unit
232 clips a given region of the video according to the indication
signal, and outputs the clipped video (hereinafter, called partial
video data) to the first zooming unit 233.
[0050] The first zooming unit 233 receives the partial video data
output from the first video clipping unit 232. The first zooming
unit 233 receives an indication signal that indicates the
enlargement ratio or the reduction ratio of the video from the
capture video partial management unit 253. The first zooming unit
233 enlarges or reduces the partial video data with the enlargement
ratio or the reduction ratio indicated by the capture video partial
management unit 253. The first zooming unit 233 outputs the
enlarged or reduced partial video data to a writing control unit
254.
[0051] In this embodiment, the number of the video inputting unit
210 and the video process unit 230 is four in accordance with the
number of cameras 110 to 140 mounted on the vehicle. But the number
of the video inputting unit 210 and the video process unit 230 is
not limited to four, and may be six in the system using six
cameras, for example. The video data captured with four cameras 110
to 140 may be input to and processed by one video inputting unit
210 and one video process unit 230, one by one.
[0052] A description will now be given of the capture preprocessing
unit 250 with reference to FIG. 8.
[0053] The capture preprocessing unit 250 is provided with a host
IF unit 251, a setting holding unit 252, the capture video partial
management unit 253, the writing control unit 254, a buffer
switching management unit 255, a combined video frame buffer 256
(in this embodiment, three combined video frame buffers 256A
through 256C), a retrieving control unit 257, and a video
outputting IF unit 258. A dashed arrow in FIG. 8 illustrates a
control signal line, and a solid arrow illustrates a data line.
[0054] The host IF unit 251 acts as an interface with a video
generating unit 260. The host IF unit 251 receives an instruction
to change the generating pattern of the combined video data
generated with combination of the partial video data, from the
video generating unit 260. The host IF unit 251 outputs the
received instruction to change the generating pattern, to the
capture video partial management unit 253. The host IF unit 251
outputs a switching completion notice, which is output from the
capture video partial management unit 253 in the case that the
change of the generating pattern is completed, to the video
generating unit 260 (a control unit 271).
[0055] Multiple setting information for generating combined video
data by combining videos captured with cameras 110 to 140 are
recorded in the setting holding unit 252 with respect to each
generating pattern. Examples of the setting information will be
illustrated in FIG. 9A. The setting information including video
clipping position information, a reduction ratio, and a write start
coordinate of the combined video frame buffer 256 is recorded with
respect to each camera of cameras 110 to 140. This setting
information is set with respect to each generating pattern of the
generated video data.
[0056] The setting information may include a video clipping
position and a write start coordinate with respect to each camera
of cameras 110 to 140 as illustrated in FIG. 9B. The setting
information illustrated in FIG. 9B is set with respect to each
generating pattern of each generated video data.
[0057] The capture video partial management unit 253 acquires
indication information of the generating pattern from the host IF
unit 251. The capture video partial management unit 253 extracts
the setting information according to the acquired indication
information of the generating pattern from the setting holding unit
252. The capture video partial management unit 253 outputs the
indication signal indicating the video clipping position to the
first video clipping unit 232 through the fourth video clipping
unit 246 on the basis of the extracted setting information. The
capture video partial management unit 253 outputs the indication
signal indicating the enlargement ratio or the reduction ratio of
the partial video to the first zooming unit 233 through the fourth
zooming unit 247 on the basis of the extracted setting information.
The capture video partial management unit 253 indicates addresses
of the combined video frame buffers 256A through 256C to which the
partial video is written, to the writing control unit 254.
[0058] The writing control unit 254 writes the partial video input
from the first video process unit 231 through the fourth video
process unit 245 in the writing address indicated by the capture
video partial management unit 253.
[0059] The buffer switching management unit 255 switches the
combined video frame buffers 256A through 256C to which the partial
video is written according to the control of the writing control
unit 254. In addition, the buffer switching management unit 255
switches the combined video frame buffers 256A through 256C from
which the combined video data is retrieved according to the control
of the retrieving control unit 257.
[0060] The combined video frame buffers 256A through 256C are
provided with three buffers. The first video inputting unit 211
through the fourth video inputting unit 214, and the first video
process unit 231 through the fourth video process unit 245 process
each video captured with cameras 110 to 140 respectively
asynchronously. Therefore, two combined video frame buffers to
which the writing control unit 254 writes the video are
provided.
[0061] The retrieving control unit 257 retrieves the latest
combined video data, which is already written to the combined video
frame buffer, by requesting the buffer switching management unit
255 to retrieve, and outputs it to the video outputting IF unit
258.
[0062] The video outputting IF unit 258 is coupled to the video
generating unit 270. The video outputting IF unit 258 outputs the
partial video data retrieved by the retrieving control unit 257 to
a video capture unit 275 of the video generating unit 270.
[0063] A description will now be given of a process of the process
processing unit 220 executed according to the instruction from the
capture video partial management unit 253 with reference to FIGS.
10A through 11C.
[0064] Front video data, rear video data, left video data, and
right video data captured with cameras 110 to 140 respectively, are
illustrated in FIG. 10A. FIG. 10B illustrates setting the clipping
range of the partial video to each video data according to the
indication signal from the capture video partial management unit
253. FIG. 10C illustrates clipping the partial video data according
to the set clipping range. FIGS. 11A and 11B illustrate reducing
the partial video according to the indication signal from the
capture video partial management unit 253. FIG. 11B illustrates
reducing the size of the rear video in the X-axis direction to
half. FIG. 11C illustrates generating the combined video data by
combining four partial video data. The combined video data
illustrated in FIG. 11C is generated in the combined video frame
buffer 256 as the writing control unit 254 writes each partial
video data to the address of the combined video frame buffer 256
indicated by the capture video partial management unit 253.
[0065] A description will now be given of the video generating unit
270 with reference to FIG. 12.
[0066] The video generating unit 270 is provided with the control
unit 271, a drawing processing unit 272, a tentative generated
video converting pattern data holding unit 273, a generated video
converting pattern data holding unit 274, the video capture unit
275, a video capture buffer 276, a buffer management unit 277, a
generated video buffer 278, and a video outputting unit 279. In
FIG. 12, a dashed arrow illustrates a control signal line, and a
solid arrow illustrates a data line.
[0067] The control unit 271 is provided with a CPU (Central
Processing Unit) 301, a ROM (Read Only Memory) 302, a RAM (Random
Access Memory) 303, and an input and output unit 304 as a hardware,
as illustrated in FIG. 13.
[0068] Programs that the CPU 301 uses for the control are stored in
the ROM 302. The CPU 301 reads programs stored in the ROM 302, and
executes processes according to the read programs. Data that the
CPU 301 uses for the calculation, and the calculation result are
stored in the RAM 303.
[0069] The input and output unit 304 receives inputs from the
operation unit 351, and receives the tentative generated video
converting pattern data described later from the external and
records it in the tentative generated video converting pattern data
holding unit 273 by the control of the CPU 301 as an input and
output port, for example.
[0070] The control unit 271 is coupled to the capture preprocessing
unit 250, the drawing processing unit 272, the buffer management
unit 277, and the video outputting unit 279 with the control signal
line.
[0071] When the control unit 271 receives the instruction to change
the generating pattern of the generated video through the operation
unit 351, the control unit 271 instructs the drawing processing
unit 272 and the capture preprocessing unit 250 to change the
generating pattern of the generated video. The control unit 271 may
acquire the vehicle information from the vehicle through the input
and output unit 304, and instruct the drawing processing unit 272
and the capture preprocessing unit 250 to change the generating
pattern of the generated video on the basis of the acquired vehicle
information. For example, when the gear is got into reverse, the
control unit 271 enlarges the area of the rear video in the
combined video, and ensures the resolution of the rear video
displayed wide, as it instructs the drawing processing unit 272 and
the capture preprocessing unit 250 to switch the generated video to
the generated video which displays the rear video wide.
[0072] In addition, the control unit 271 manages the generating
timing of the generated video data with respect to each frame. When
the generated video data for a frame is written to the generated
video buffer 278 by the drawing processing unit 272, the control
unit 271 receives a frame process completion notice from the
drawing processing unit 272. When receiving the frame process
completion notice from the drawing processing unit 272, the control
unit 271 instructs the drawing processing unit 272 to start drawing
the generated video data for the next frame (details of this
process will be described later with reference to a flowchart).
[0073] In addition, the control unit 271 outputs the instruction
signal to switch the generated video buffer 278 to which the
generated video data is written to the buffer management unit 277.
In this embodiment, the generated video buffer 278 is provided with
two generated video buffers 278A and 278B. The control unit 271
selects the generated video buffer to which the drawing processing
unit 272 writes the generated video data. In addition, the control
unit 271 outputs the instruction signal to switch the generated
video buffer 278 from which the generated video data is retrieved
to the buffer management unit 277. In addition, when the video
outputting unit 279 outputs the generated video for a frame, the
control unit 271 receives a frame output completion notice that
indicates that the video outputting unit 279 completes outputting
the generated video for a frame.
[0074] The combined video data output from the video outputting IF
unit 258 of the capture preprocessing unit 250 is input to the
video capture unit 275. The video capture unit 275 receives the
combined video data output from the video outputting IF unit 258,
and outputs it to the video capture buffer 276.
[0075] The video capture buffer 276 captures the video by storing
the combined video data output from the video capture unit 275. The
combined video data stored in the video capture buffer 276 is
output from the video capture buffer 276 to the drawing processing
unit 272 according to the retrieve instruction of the drawing
processing unit 272.
[0076] The tentative generated video converting pattern data
(coordinate conversion data) illustrated in FIG. 14A is stored in
the tentative generated video converting pattern data holding unit
273. The tentative generated video converting pattern is data for
the coordinate conversion to convert each camera video to generated
video data. This data is generated by an information processing
apparatus such as a personal computer, and is recorded in the
tentative generated video converting pattern data holding unit 273
by the control of the control unit 271. The tentative generated
video converting pattern data is data to generate the generated
video data from the video before each camera video is clipped,
reduced, and combined by the capture preprocessing unit 250. The
data of which the coordinate value is corrected by the control unit
271 is recorded in the generated video converting pattern data
holding unit 274 as generated video converting pattern data. The
tentative generated video converting pattern data includes, as
illustrated in FIG. 14A, the polygon number, the vertex number of
polygons that the polygon number indicates, the coordinate value
information of each vertex before the conversion, and coordinate
value information of the generated video data of each vertex after
the conversion, for example. Polygons are partitions on the video,
and processing units for a coordinate conversion when the combined
video data is converted to the generated video data. The polygon
number is the number to identify the polygon. The polygon number
and the vertex number are not always necessary.
[0077] In addition, the tentative generated video converting
pattern may be the pattern data illustrated in FIG. 14B. The
tentative generated video converting pattern illustrated in FIG.
14B includes the polygon number, the vertex number of the polygon
that the polygon number indicates, the coordinate value information
on the video data of each vertex before the conversion, the
coordinate value on the video projection plane defined in the
three-dimensional video processing space, and viewpoint vector data
of the three-dimensional video processing space. The viewpoint
vector data is data to determine where the video projection plane
is viewed from to generate the generated video data (see FIG.
17).
[0078] Referring to the generated video converting pattern stored
in the generated video converting pattern data holding unit 274,
the drawing processing unit 272 converts the combined video data
retrieved from the video capture buffer 276 to the generated video
data. A method such as a texture mapping method is used as a
converting method, for example. The drawing processing unit 272
records the generated video data that the drawing processing unit
272 has generated in the generated video buffer 278.
[0079] The generated video data recorded in the generated video
buffer 278 is retrieved by the video outputting unit 279, and
displayed on the display unit 352.
[0080] A description will now be given of the process of the
control unit 271 to generate the generated video data from the
combined video data, with reference to flowcharts illustrated in
FIGS. 15A and 15B.
[0081] The control unit 271 determines whether the instruction to
change the generating pattern of the generated video data is input
from the operation unit 351. (step S1). When the instruction to
change the generating pattern is input (step S1: YES), the control
unit 271 instructs the drawing processing unit 272 to change the
generating pattern (step S2). Then the control unit 271 outputs the
instruction to change the generating pattern of the combined video
to the capture preprocessing unit 250 (step S3). The control unit
271 determines whether it receives the change completion notice of
the generating pattern of the combined video from the capture
preprocessing unit 250 (step S4). The control unit 271 holds the
next procedure till it receives the change completion notice from
the capture preprocessing unit 250 (step S4: NO). A capture video
partial management unit 253 of the capture preprocessing unit 250
extracts the setting information corresponding to the generating
pattern indicated by the control unit 271 from the setting holding
unit 252. Referring to the setting information, the capture video
partial management unit 253 indicates the used range of each video
data used for the indicated generating pattern to the first video
clipping unit 232 through the fourth video clipping unit 246
respectively. Referring to the setting information, the capture
video partial management unit 253 indicates the reduction ratio of
the indicated generating pattern to the first zooming unit 233
through the fourth zooming unit 247.
[0082] When the control unit 271 receives the change completion
notice from the capture preprocessing unit 250 (step S4: YES), the
control unit 271 determines whether it receives the process
completion notice of the generated video data for a frame from the
buffer management unit 267 (step S5). The control unit 271 holds
the next procedure till it receives the process completion notice
from the buffer management unit 277 (step S5). When the control
unit 271 receives the process completion notice from the buffer
management unit 277 (step S5: YES), it determines whether it
receives the output completion notice of the generated video data
for a frame from the video outputting unit 279 (step S6). When the
control unit 271 receives the output completion notice from the
video outputting unit 279 (step S6: YES), it outputs the
instruction to switch the generated video buffer 278, to the buffer
management unit 277 (step S7). When the control unit 271 outputs
the instruction to switch the generated video buffer 278, to the
buffer management unit 277, the control unit 271 instructs the
drawing processing unit 272 to start generating the generated video
data for the next frame (step S8).
[0083] A description will now be given of a process of the control
unit 271 to generate the generated video converting pattern data
and the setting information stored in the setting holding unit 252
of the capture preprocessing unit 250 by using the tentative
generated video converting pattern data, with reference to
flowcharts illustrated in FIGS. 16A through 16C. The process
described hereinafter, is not executed by inputting actual camera
video data, but is executed by the calculation of the control unit
271, or with a personal computer on the case that the generated
video is designed.
[0084] For the pre-preparation, the generated video is designed by
an application software installed in an information processing
apparatus such as a personal computer.
[0085] The user inputs data such as shape data of the video
projection plane, the viewpoint vector, and a display range of the
video to the application software. The application software
generates the coordinate conversion data (tentative generated video
converting pattern data) to convert the camera video to the
generated video on the basis of the input data. The tentative
generated video converting pattern data generated by the
application software is loaded into the control unit 271 through
the input and output unit 304, and recorded in the tentative
generated video converting pattern data holding unit 273 by the
control of the control unit 271.
[0086] The shape data of the video projection plane is data to
indicate the shape of the projection plane onto which the multiple
videos captured with cameras 110 to 140 are projected, and data to
convert the camera video to the generated video (see FIG. 17). The
viewpoint vector data is data to indicate where the video
projection plane is seen from to generate the generated video. The
ratio in displaying each part of video captured with each camera of
cameras 110 to 140 changes in accordance with these video
projection plane and viewpoint vector data. The data of the display
range of the video is data to indicate the range to display the
video as illustrated in FIG. 17.
[0087] The control unit 271 calculates the coordinate value of the
use part of each camera video, referring to the tentative generated
video converting pattern data stored in the tentative generated
video converting pattern data holding unit 273 (step S11).
[0088] As the tentative generated video converting pattern data is
the conversion data to convert multiple camera videos to the
generated video, the control unit 271 calculates the used range
(the coordinate value) of the camera video from the tentative
generated video converting pattern data. More specifically, on the
basis of the tentative generated video converting pattern data, the
control unit 271 detects the vertex coordinate of each camera video
corresponding to the display range of the generated video data, and
calculates the maximum and the minimum values of the vertex
coordinate (Xmin, Ymin). The control unit 271 outputs the
coordinate value of the rectangle of which the diagonal vertex is
the calculated maximum and the calculated minimum values of the
vertex coordinate (Xmin, Ymin) as the used range. FIGS. 18A through
18D illustrate determining each used range of the front video, the
rear video, the left video, and the right video. As illustrated
with a dashed line, when the used range is extracted free-form, the
used range is calculated as a shape made by connecting the most
outer sides of the used range.
[0089] The control unit 271 calculates the pixel expansion ratio
when each camera video is converted to the generated video (step
S12).
[0090] The control unit 271 executes the procedure of the step S12
to each camera video. The control unit 271 compares all video
processing units (polygons) composing the generated video with the
coordinate value of the polygon on the camera video, and calculates
the pixel expansion ratio in the coordinate conversion (step S12).
Polygons are processing units (pixels) in the coordinate
conversion, and the control unit 271 calculates the pixel expansion
ratio by calculating the coordinate values before and after the
coordinate conversion of the polygon. The method to calculate the
pixel expansion ratio may be a method to calculate the expansion
ratio of the camera video in the X-axis direction and the Y-axis
direction respectively, and a method to calculate the expansion
ratio with the ratio between the area of the polygon of the camera
video before the conversion and the area of the polygon of the
generated video after the conversion, for example. Examples of the
calculation results are illustrated in FIG. 19.
[0091] The control unit 271 determines the representative value of
the pixel expansion ratio of each camera video (step S13). Using
the pixel expansion ratio calculated with respect to each polygon
in the step S12 above, the control unit 271 determines the
representative value of the pixel expansion ratio of each camera
video. The control unit 271 selects the maximum one from the pixel
expansion ratios of each polygon, and determines it as the
representative value of the pixel expansion ratio of the
appropriate camera video. The control unit 271 may select the
minimum one from the pixel expansion ratios of each polygon, and
determine it as the representative value of the pixel expansion
ratio; or the control unit 271 may calculate the average value or
the intermediate value of the pixel expansion ratios of each
polygon, and determine this average value or intermediate value as
the representative value of the pixel expansion ratio.
[0092] Examples of the representative values of the pixel expansion
ratio of each camera video are illustrated in FIG. 20. In examples
illustrated in FIG. 20, the representative value of the pixel
expansion ratio in the X-direction and the Y-direction is
calculated with respect to each camera video.
[0093] Using the coordinate value of the use part of each camera
video extracted in the step S11 (called a partial video, and not
called the partial video data as it is not an actual video data),
the control unit 271 reduces the partial video of the camera video
of which the representative value of the pixel expansion ratio is
less than one by the representative value of the pixel expansion
ratio times, which means that the control unit 271 reduces the
partial video so that the representative value of the pixel
expansion ratio becomes one (step S14). More specifically, the
control unit 271 multiplies the partial video by the representative
value of the pixel expansion ratio, reduces the partial video by
the representative value of the pixel expansion ratio times, and
calculates the number of the pixels of the partial video multiplied
by the representative value of the pixel expansion ratio in the
X-axis direction and the Y-axis direction (.DELTA.Xtemp,
.DELTA.Ytemp) respectively. The control unit 271 does not execute
any process to the camera video of which the representative value
of the pixel expansion ratio is more than or equal to one, and
calculates the number of pixels in the X-axis direction and the
Y-axis direction without process.
[0094] The control unit 271 determines whether all partial videos
are within the regulation size of the combined video (step S15).
The combined video is a video made with combination of multiple
camera videos. To generate the generated video by executing the
conversion process to this combined video in the video generating
unit 270, the size of the combined video is determined as the
regulation size (given size). The control unit 271 locates all
partial videos within the combined video as illustrated in FIG. 21,
and determines whether the partial videos can be located within the
combined video.
[0095] When the partial videos can be located within the combined
video (step S16: YES), the control unit 271 records the image size
information of the partial video multiplied by the representative
value of the pixel expansion ratio, and goes to a step S20.
[0096] When the partial videos can not be located within the
combined video (step S16: NO), the control unit 271 enlarges the
partial video of the camera video of which the representative value
of the pixel expansion ratio is more than or equal to one the
representative value of the pixel expansion ratio times (step S17).
This means that the control unit 271 enlarges the partial video so
that the representative value of the pixel expansion ratio of the
use part of the video after the enlargement becomes one. The
control unit 271 calculates the number of pixels of the partial
video enlarged the representative value of the pixel expansion
ratio times in the X-axis direction and the Y-axis direction
respectively (step S17).
[0097] The control unit 271 calculates the combination of partial
videos, of which the pixel aspect ratio is nearest to the aspect
ratio of the combined video and the reduction ratio in the
reduction to the image size of the combined video is smallest among
all combinations of partial videos, and its reduction ratio.
[0098] The control unit 271 sorts the partial videos reduced by the
representative value of the pixel expansion ratio calculated in the
step S14 times, and the partial videos enlarged the representative
value of the pixel expansion ratio calculated in the step S17 times
(hereinafter, these are called tentative-sized partial videos)
within the image size of the combined video as illustrated in FIGS.
22A and 22B. The control unit 271 calculates the combination of
partial videos, of which the pixel aspect ratio is nearest to the
aspect ratio of the combined video and the reduction ratio in the
reduction to the image size of the combined video is smallest, and
its reduction ratio. FIG. 22A illustrates tentative-sized partial
videos of the front, rear, left, and right directions. FIG. 22B
illustrates sorting these tentative-sized partial videos.
[0099] The control unit 271 calculates the actual reduction ratio
of each partial video on the basis of the reduction ratio
calculated in the step S18 and the representative value of the
pixel expansion ratio (step S19). More specifically, the control
unit 271 calculates the product of the reduction ratio and the
representative value of the pixel expansion ratio with respect to
each partial video.
[0100] The control unit 271 determines whether the partial video of
which the reduction ratio calculated in the step S19 is less than
one (it means the enlargement) exists (step S20). When the partial
video of which the reduction ratio is less than one exists, the
control unit 271 replaces the reduction ratio of the partial video
of which the reduction ratio is less than one with one, and
relocates partial videos without changing the combining position on
the combined video. When the space is created within the image size
of the combined video by relocation, the control unit 271 equally
changes the reduction ratio so that the reduction ratio of the
partial videos of which the reduction ratio is more than or equal
to one (it means the reduction) becomes near to one (step S21). In
examples illustrated in FIGS. 23A through 23D, as the reduction
ratios of partial videos A and B are more than or equal to one, the
control unit 271 executes the size conversion to make reduction
ratios of these partial videos one, and relocates these partial
videos. The result of the size conversion and relocation is
illustrated in FIGS. 23C and 23D.
[0101] The control unit 271 determines the combined video pattern
data after procedures above. The control unit 271 outputs the
determined combined video pattern data to the capture preprocessing
unit 250 as the setting information, and records it in the setting
holding unit 252 (step S22).
[0102] The control unit 271 corrects the tentative generated video
converting pattern data by using the location information of the
camera video in the combined video (setting information) (step
S23). The control unit 271 stores the corrected tentative generated
video converting pattern data in the generated video converting
pattern data holding unit 264 as the generated video converting
pattern data. When correcting the tentative generated video
converting pattern data to the generated video converting pattern
data, the control unit 271 may also make a correction to deal with
the installation condition of the camera by using the installation
condition information of the camera to the vehicle and the camera
characteristics data. The installation condition information of the
camera includes the position coordinate of the camera (X, Y, Z),
and mounting angles (a yaw angle, a depression angle, and a roll
angle). The camera characteristics data includes the number of
pixels of the camera video (horizontal and vertical), field angles
(horizontal and vertical), and lens distortion data.
[0103] In the steps S14 and S17, the image size of the partial
video of which the representative value of the pixel expansion
ratio is less than one, and the size of the partial video of which
the representative value of the pixel expansion ratio is more than
or equal to one, are changed so that the representative value of
the pixel expansion ratio becomes one. Thus, in the actual data
processing, the representative value of the pixel expansion ratio
of the partial video data enlarged according to the coordinate
conversion from the combined video data to the generated video data
becomes equivalent between each partial video.
[0104] When priority is set on camera videos, the reduction ratio
of the high-priority camera video may be determined first, and then
the reduction ratio of other camera videos may be determined. For
example, when the front video is designated as a priority video,
the partial video data is clipped from the front camera video, and
the size of this partial video data is not changed and fixed. Then,
the size of the partial video data clipped from other camera videos
is reduced so as to be within the size of the combined video data.
After the partial video data is clipped from each camera video and
the reduction ratios of all partial video data are calculated, the
reduction ratio of the high-priority video data is reduced by fifty
percent, the reduction ratio of rest of partial video data may be
changed so as to be within the size of the combined video data.
[0105] According to this embodiment, if the coordinate conversion
data for generating the generated video is provided, it is possible
to generate generated video data by calculating the clipped range
of the camera video and the reduction ratio with the video
processing apparatus 1 and generating the combined video data from
the actual input camera video. If multiple coordinate conversion
data are provided, it is possible to generate generated video data
of which the video pattern is changed in response to the running
condition of the vehicle.
[0106] As the size of each partial video is adjusted so that the
representative value of the pixel expansion ratio of the partial
video are equivalent between each partial video, it is possible to
enlarge the partial video almost evenly in executing the coordinate
conversion to generate the generated video.
[0107] As the size of the partial video is adjusted so that the
high-priority video data is not reduced, the high-priority video
data is not reduced in generating the generated video.
[0108] All examples and conditional language recited herein are
intended for pedagogical purposes to aid the reader in
understanding the invention and the concepts contributed by the
inventor to furthering the art, and are to be construed as being
without limitation to such specifically recited examples and
conditions, nor does the organization of such examples in the
specification relate to a showing of the superiority and
inferiority of the invention. Although the embodiment of the
present invention has been described in detail, it should be
understood that the various change, substitutions, and alterations
could be made hereto without departing from the spirit and scope of
the invention.
[0109] For example, in the above embodiment, although the pixel
expansion ratios of all polygons are calculated, it is not
necessary to calculate the pixel expansion ratios of polygons in
the area away from the vehicle. As videos near the vehicle are
important for drive assist, it is not necessary to visualize other
videos at high resolution, and it is possible to make them out of
the processing object.
[0110] When the partial video is clipped from the camera video as
illustrated in FIG. 24, the partial video may be clipped free-form.
FIG. 24 illustrates an example of the combined video data made with
combination of the partial videos clipped free-form.
* * * * *