U.S. patent application number 11/543482 was filed with the patent office on 2007-04-12 for image pickup device of multiple lens camera system for generating panoramic image.
Invention is credited to Benjamin Kuo, Christine Lin, Tatsumi Mitsushita, Patrick Pan.
Application Number | 20070081091 11/543482 |
Document ID | / |
Family ID | 37910771 |
Filed Date | 2007-04-12 |
United States Patent
Application |
20070081091 |
Kind Code |
A1 |
Pan; Patrick ; et
al. |
April 12, 2007 |
Image pickup device of multiple lens camera system for generating
panoramic image
Abstract
The present invention aims to simplify stitching algorithm which
generates horizontal panoramic image. The image pickup device of
the present invention comprises a plurality of lenses and
positioning means. Said positioning means positions each lens so
that the FOV (Field Of View) intersection points of all lenses are
aligned in vertical direction. Accordingly, the horizontal parallax
does not exist in the image picked up by the camera system and the
stitching point remains the same for the objects at different
distances.
Inventors: |
Pan; Patrick; (Taipei,
TW) ; Mitsushita; Tatsumi; (Fukuoka, JP) ;
Lin; Christine; (Taipei, TW) ; Kuo; Benjamin;
(Taipei, TW) |
Correspondence
Address: |
FROMMER LAWRENCE & HAUG LLP
745 FIFTH AVENUE
NEW YORK
NY
10151
US
|
Family ID: |
37910771 |
Appl. No.: |
11/543482 |
Filed: |
October 5, 2006 |
Current U.S.
Class: |
348/335 ;
348/E5.053 |
Current CPC
Class: |
A61B 1/00165 20130101;
G02B 13/06 20130101; G03B 37/04 20130101; H04N 5/2624 20130101 |
Class at
Publication: |
348/335 |
International
Class: |
G02B 13/16 20060101
G02B013/16 |
Foreign Application Data
Date |
Code |
Application Number |
Oct 7, 2005 |
TW |
94135314 |
Claims
1. An image pickup device of multiple lens camera system,
comprising: N lenses, wherein the horizontal field of view for each
lens is HFOV.sub.i (i=1, 2, . . . , N); positioning means, wherein
said positioning means positions each lens on top of the other by
rotation of .theta..sub.i degrees, where
0<.theta..sub.i<HFOV.sub.i, i=1, 2, . . . , N-1, in
horizontal direction, and said positioning means positions each
lens so that the FOV intersection points of all lenses are aligned
in vertical direction.
2. The image pickup device of claim 1, wherein said positioning
means tilts each lens with an angle of .phi..sub.i degrees, where
0<.phi..sub.i<VFOV.sub.i, i=1, 2, . . . , N, in vertical
direction.
3. The image pickup device of claim 1, wherein
.theta..sub.1=.theta..sub.2=.theta..sub.3= . . .
.theta..sub.N-1.
4. The image pickup device of claim 1, wherein the total field of
view obtained by said N lenses is equal to i = 1 N .times. HFOV i -
i = 1 N .times. ( HFOV i + 1 - .theta. i ) . ##EQU13##
Description
FIELD OF THE INVENTION
[0001] The present invention relates generally to an image pickup
device. More specifically, the present invention relates to an
image pickup device of multiple lens camera system for generating
panoramic image. The image pickup device can position a plurality
of lenses in a multiple camera system so that a simple stitching
algorithm is implemented in an ASIC (Application Specific
Integrated Circuit) solution.
DESCRIPTION OF THE PRIOR ARTS
[0002] The generation of a panoramic image usually requires taking
pictures concurrently by a plurality of cameras and then composing
an image by an image processor. On the other hand, a static
panoramic image may be formed by using a single camera combined
with a panning motor to shoot multiple times and then stitching the
images captured each time. For example, Japan Patent No. 11-008845
and No. 11-018003 involve panning motors to capture wide angle
images. However, the panning motor increases the cost and size of
the camera system. Accordingly, it is desired to generate a
panoramic image by a simpler mechanism and a simpler stitching
algorithm.
SUMMARY OF THE INVENTION
[0003] The image pickup device of the invention aligns the FOV
(Field Of View) intersection points of all lenses to provide a
system with fixed stitching points of the captured image so that
simple stitching algorithm can be implemented in a low-cost ASIC
solution to generate panoramic video.
[0004] To achieve the above purpose, the present invention provides
an image pickup device of multiple lens camera system, comprising:
N lenses, wherein the horizontal field of view for each lens is
HFOV.sub.i (i=1, 2, . . . , N); positioning means, wherein said
positioning means positions each lens on top of the other by
rotation of .sub.idegrees (0<.sub.i<HFOV.sub.i, i=1, 2, . . .
, N-1) in horizontal direction, and said positioning means
positions each lens so that the FOV intersection points of all
lenses are aligned in vertical direction.
[0005] According to an aspect of the present invention, the
above-mentioned positioning means tilts each lens with an angle of
.phi..sub.i degrees (0<.phi..sub.i<VFOV.sub.i, i=1, 2, . . .
, N) in vertical direction.
[0006] According to another aspect of the present invention, the
above-mentioned .sub.1=.sub.2=.sub.3= . . . =.sub.N-1.
[0007] According to yet another aspect of the present invention,
the total field of view obtained by the above-mentioned N lenses is
equal to i = 1 N .times. HFOV i - i = 1 N .times. ( HFOV i + 1 - i
) . ##EQU1##
BRIEF DESCRIPTION OF THE DRAWINGS
[0008] FIG. 1 is an illustrative diagram of the N lenses of an
image pickup device according to the present invention.
[0009] FIG. 2 is an illustrative diagram of lens rotation and HFOV
(horizontal field of view).
[0010] FIG. 3 is a diagram showing the FOV intersection point of a
single lens.
[0011] FIG. 4 is a diagram showing the horizontal parallax caused
by the misalignment of FOV intersection points.
[0012] FIG. 5(a) and FIG. 5(b) are diagrams showing overlapping
portions of the images of near objects and far objects,
respectively, in the case of misalignment.
[0013] FIG. 6 is a diagram showing the case in which the FOV
intersection points are aligned.
[0014] FIG. 7 is a diagram showing the image shift without tilting
the camera in vertical direction.
[0015] FIG. 8 is a diagram showing the case in which the images are
aligned by tilting the camera in vertical direction.
[0016] FIG. 9 shows a block diagram of the multiple lens camera
system according to the present invention.
DETAILED DESCRIPTION OF THE INVENTION
[0017] FIG. 1 shows an image pickup device according to the present
invention by the examples of 2, 3 and N lenses. This lens
arrangement is achieved by positioning means according to the
present invention. This positioning means can be a part of a video
phone system which creates wide angle images beyond the angle
limitation of a single lens. This multiple lens camera system
together with a simple ASIC where a simple stitching algorithm is
implemented are adapted to provide a low-cost, small-size and
wide-angle camera system.
[0018] The principle of the present invention is described with
reference to FIGS. 2-8 as follows.
[0019] The image pickup device according to the present invention
comprises N lenses and positioning means. Said positioning means
positions each lens on top of the other by rotation of
.sub.idegrees (0<.sub.i<HFOV.sub.i, i=1, 2, . . . , N-1) in
horizontal direction.
[0020] FIG. 2 is an illustrative diagram of lens rotation and HFOV
(horizontal field of view). Providing N lenses with horizontal view
angle of HFOV.sub.i for each lens (i=1, 2, 3, . . . , N) and lens
rotation angle of .theta..sub.i (i=1, 2, . . . , N-1) in the camera
system, the total HFOVt of the system is equal to i = 1 N .times.
HFOV i - i = 1 N .times. ( HFOV i + 1 - .theta. i ) . ##EQU2## In
case the HFOV.sub.i of each lens is equal to HFOV and all rotation
angles .theta..sub.i are equal to .theta., the total HFOVt of the
system will be equal to HFOV*N-(HFOV-.theta.)*(N-1). For example,
N=2, HFOV1=HFOV2=60, and .theta..sub.1=30.degree. result in a total
HFOVt=90.degree.; and N=11 (11 lenses in total),
HFOV.sub.i=60.degree. (i=1, 2, 3, . . . 11) and
.theta..sub.i=30.degree. (i=1, 2, 3, . . . 10) result in a total
HFOVt=360.degree..
[0021] The importance of the invention is to capture images for a
simple stitching algorithm which can be implemented in a low-cost
ASIC for video stitching. The alignment of the FOV intersection
point of each lens provides constant stitching point for the
objects at different distance and the rotation angle between each
lens is fixed for the camera system. Hence the stitching point can
be calculated during camera calibration. It is not necessary for
the ASIC to calculate the stitching point dynamically at every
frame due to the distance change of the objects. Therefore the
computation power for stitching can be much reduced and the ASIC
cost can be saved.
[0022] In the following description, the relation between the
stitching point and the FOV intersection point alignment is
explained.
[0023] FIG. 3 shows the FOV intersection point of a single lens.
FIG. 4 shows the stitching problem caused by the misalignment of
FOV intersection points. In the figure, stpn represents the
stitching point of near objects; stpf represents the stitching
point of far objects; Dn represents the distance between the FOV
intersection point and near objects; Df represents the distance
between the FOV intersection point and far objects; Dth represents
the distance between the FOV intersection point and the FOV cross
point; Wn represents viewable width of near objects; Wf represents
viewable width of far objects; .alpha. represents the angle between
overlapped boundary and the stitching point; and HFOV represents
horizontal field of view. As shown in FIG. 4, in the case of
misalignment, there is no image overlapping for the objects within
the distance of Dth. Providing the definition of stitching point is
center of the overlapped images, the stitching points shift when
the distance between the object and the camera changes.
[0024] FIG. 5(a) and FIG. 5(b) show overlapping portions of the
images of near objects and far objects, respectively, in the case
of misalignment. Comparing these two figures, it can be seen that
the overlapping portion (shadowed portion) of the images of near
objects in FIG. 5(a) is obviously smaller than the overlapping
portion (shadowed portion) of the images of far objects in FIG.
5(b).
[0025] The stitching point change can be derived from the following
equations:
[0026] For near objects: stpn = 2 .times. .times. Dn * tan .times.
.times. ( HFOV i 2 ) - ( Dn - Dth ) * tan .times. .times. .alpha.
##EQU3## Wn = 2 .times. Dn * tan .times. .times. ( HFOV i 2 )
##EQU3.2##
[0027] The stitching point percentage of near objects within the
image is: stpn Wn = 2 .times. Dn * tan .times. .times. ( HFOV i 2 )
- ( Dn - Dth ) * tan .times. .times. .alpha. 2 .times. Dn * tan
.times. .times. ( HFOV i 2 ) ##EQU4##
[0028] For far objects: stpf = 2 .times. .times. Df * tan .times.
.times. ( HFOV i 2 ) - ( Df - Dth ) * tan .times. .times. .alpha.
##EQU5## Wf = 2 .times. Df * tan .times. .times. ( HFOV i 2 )
##EQU5.2##
[0029] The stitching point percentage of far objects within the
image is: stpf Wf = 2 .times. .times. Df * tan .times. .times. (
HFOV i 2 ) - ( Df - Dth ) * tan .times. .times. .alpha. 2 .times.
Df * tan .times. .times. ( HFOV i 2 ) ##EQU6##
[0030] Therefore, stpn Wn .noteq. stpf Wf .times. .times. ( since
.times. .times. Dth .noteq. 0 ) ##EQU7##
[0031] FIG. 6 shows the case in which the FOV intersection points
are aligned. In this case, the stitching points remain the same
regardless of the object distances. This can be explained by the
following equations:
[0032] For near objects: stpn = 2 .times. .times. Dn * tan .times.
.times. ( HFOV i 2 ) - Dn * tan .times. .times. .alpha. ##EQU8## Wn
= 2 .times. Dn * tan .times. .times. ( HFOV i 2 ) ##EQU8.2##
[0033] The stitching point percentage of near objects within the
image is: stpn Wn = 2 .times. Dn * tan .times. .times. ( HFOV i 2 )
- Dn * tan .times. .times. .alpha. 2 .times. Dn * tan .times.
.times. ( HFOV i 2 ) = 2 .times. .times. tan .times. .times. ( HFOV
i 2 ) - tan .times. .times. .alpha. 2 .times. .times. tan .times.
.times. ( HFOV i 2 ) ##EQU9##
[0034] For far objects: stpf = 2 .times. .times. Df * tan .times.
.times. ( HFOV i 2 ) - Df * tan .times. .times. .alpha. ##EQU10##
Wf = 2 .times. Df * tan .times. .times. ( HFOV i 2 )
##EQU10.2##
[0035] The stitching point percentage of far objects within the
image is: stpf Wf = 2 .times. .times. Df * tan .function. ( HFOV i
2 ) - Df * tan .times. .times. .alpha. 2 .times. Df * tan
.function. ( HFOV i 2 ) = 2 .times. .times. tan .function. ( HFOV i
2 ) - tan .times. .times. .alpha. 2 .times. .times. tan .function.
( HFOV i 2 ) ##EQU11##
[0036] Therefore, stpn Wn = stpf Wf ##EQU12##
[0037] Besides, the images captured by each lens are shifted due to
the vertical displacement of FOV FIG. 7 explains the image
non-coinciding caused by the FOV displacement. The non-coinciding
portions have to be cropped in the final panoramic image. The
larger the N is, the more portions are cropped. To solve this
problem, the present invention provides positioning means for
tilting each lens by .phi..sub.i degrees
(0<.phi..sub.i<VFOV.sub.i, i=1, 2, . . . , N) in vertical
direction. FIG. 8 explains the result obtained by tilting each lens
in vertical direction. It should be noted that the FOV intersection
points are always aligned while tilting the lenses.
[0038] Accordingly, the image pickup device of the present
invention is able to provide the images with constant stitching
points, thereby simplifying the complexity of the stitching
algorithm.
[0039] In the following, an embodiment of the multiple lens camera
system according to the present invention is described with
reference to FIG. 9. For conciseness, the following description is
focused on the lens part and the related image processing procedure
with the detailed description of other parts of the camera system
omitted.
[0040] As shown in FIG. 9, a lens part 110 includes three lenses
110A, 110B and 110C, wherein the lens 110B is arranged on top of
the lens 110A with a counterclockwise rotation of .theta. degrees
(not shown in the figure) in horizontal direction; and the lens
110C is arranged on top of the lens 110B with a further
counterclockwise rotation of .theta. degrees in horizontal
direction. The image signals captured by the lenses 110A, 110B and
110C are passed through FFC (Flexible Flat Cable) 120A, 120B and
120C, respectively, to an image processing logic block 130 for
further processing. The image processing logic block 130 includes a
multi-lens ISP (image signal processor) 131, stitching logic 132,
an ISP 133, a video encoder 134, a MPEG encoder 135 and a network
interface 136.
[0041] At first, the multi-lens ISP 131 performs preliminary
processing of the image signals passed from the lenses 110A, 110B
and 110C so that the differences between the images captured by
respective lenses are reduced. The image signals after the
preliminary processing are respectively passed to the stitching
logic 132. The stitching logic 132 performs transformation and
positional calculation on the image signals so that the images are
put seamlessly together as one single image. Said one single image
is then passed to the ISP 133 for traditional image processing. At
this point, the processed image can be encoded by the video encoder
134 and then displayed on any display device. Alternatively, the
processed image can also be compressed for storing in any storage
device. Further, the compressed image data can be passed through
the network interface 136 to the Internet.
Effects of the Invention
[0042] The stitching algorithm is the part which consumes most
computational power when generating a panoramic image. For high
frame rate video (e.g. 30 fps), a low-cost ASIC solution is not
powerful enough to achieve the performance of updating stitching
point for every 1/30 second. The present invention discloses a
simple and feasible mechanism for positioning multiple lenses to
capture images with constant stitching points, and thus provides a
low-cost, small-size and wide-angle camera system.
* * * * *