U.S. patent application number 13/691632 was filed with the patent office on 2013-06-06 for apparatus and method for video image stitching.
This patent application is currently assigned to STEALTH HD CORP.. The applicant listed for this patent is Stealth HD Corp.. Invention is credited to Bill Banta, Geoff Donaldson.
Application Number | 20130141526 13/691632 |
Document ID | / |
Family ID | 48523714 |
Filed Date | 2013-06-06 |
United States Patent
Application |
20130141526 |
Kind Code |
A1 |
Banta; Bill ; et
al. |
June 6, 2013 |
Apparatus and Method for Video Image Stitching
Abstract
A method includes collecting a first image from a first video
camera of a test pattern and a second image from a second video
camera of the test pattern. The first image and the second image
have an overlap region. The overlap region is evaluated to generate
calibration parameters that accommodate for any vertical,
horizontal or rotational misalignment between the first image and
the second image. Calibration parameters are applied to video
streams from the first video camera and the second video
camera.
Inventors: |
Banta; Bill; (San Francisco,
CA) ; Donaldson; Geoff; (San Francisco, CA) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Stealth HD Corp.; |
San Francisco |
CA |
US |
|
|
Assignee: |
STEALTH HD CORP.
San Francisco
CA
|
Family ID: |
48523714 |
Appl. No.: |
13/691632 |
Filed: |
November 30, 2012 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
61566269 |
Dec 2, 2011 |
|
|
|
Current U.S.
Class: |
348/38 |
Current CPC
Class: |
H04N 21/21805 20130101;
H04N 21/2393 20130101; H04N 21/23439 20130101; H04N 21/47202
20130101; H04N 21/4728 20130101; H04N 2101/00 20130101; H04N 21/816
20130101; H04N 5/23238 20130101; H04N 21/234363 20130101; H04N
21/4223 20130101; H04N 21/4668 20130101 |
Class at
Publication: |
348/38 |
International
Class: |
H04N 5/232 20060101
H04N005/232 |
Claims
1. A method, comprising: collecting a first image from a first
video camera of a test pattern and a second image from a second
video camera of the test pattern, wherein the first image and the
second image have an overlap region; evaluating the overlap region
to generate calibration parameters that accommodate for any
vertical, horizontal or rotational misalignment between the first
image and the second image; and applying the calibration parameters
to video streams from the first video camera and the second video
camera.
2. The method of claim 1 further comprising collecting camera
distortion values from the first video camera and the second video
camera to generate calibration parameters.
3. The method of claim 1 further comprising collecting alignment
parameter between the first video camera and the second video
camera to generate calibration parameters.
4. The method of claim 1 further comprising collecting overlap
values between the first video camera and the second video camera
to generate calibration parameters.
5. The method of claim 1 further comprising collecting twist
parameters between the first video camera and the second video
camera to generate calibration parameters.
6. The method of claim 1 further comprising collecting blend width
parameters between the first video camera and the second video
camera to generate calibration parameters.
7. The method of claim 1 further comprising collecting field of
view overlap parameters between the first video camera and the
second video camera to generate calibration parameters.
8. The method of claim 1 further comprising collecting individual
camera illumination parameters from the first video camera and the
second video camera to generate calibration parameters.
9. The method of claim 1 further comprising collecting system
relative illumination parameters from the first video camera and
the second video camera to generate calibration parameters.
10. The method of claim 1 further comprising applying cropping
lines to the video streams to correct for warping.
11. The method of claim 1 further comprising: identifying a faulty
video camera in a stream of video from multiple video cameras; and
maximizing the field of view of video cameras adjacent to the
faulty video camera.
12. The method of claim 1 further comprising: identifying an offset
video camera in a stream of video from multiple video cameras;
evaluating the offset; forming updated alignment parameters; and
combining video based upon the updated alignment parameters.
Description
CROSS-REFERENCE TO RELATED APPLICATION
[0001] This application claims priority to U.S. Provisional Patent
Application 61/566,269, filed Dec. 2, 2011, entitled "Panoramic
video Camera System and Related Methods", the contents of which are
incorporated herein by reference.
FIELD OF THE INVENTION
[0002] This invention relates generally to video image processing.
More particularly, this invention relates to panoramic video image
stitching utilizing alignment and calibration parameters for
dynamic video image compensation.
BACKGROUND OF THE INVENTION
[0003] Panoramic video feeds should be combined in a seamless
manner. Existing techniques that endeavor to achieve this have high
power requirements and excessive processing times, particularly in
the context of mobile devices, which are constrained by these
parameters.
SUMMARY OF THE INVENTION
[0004] A method includes collecting a first image from a first
video camera of a test pattern and a second image from a second
video camera of the test pattern. The first image and the second
image have an overlap region. The overlap region is evaluated to
generate calibration parameters that accommodate for any vertical,
horizontal or rotational misalignment between the first image and
the second image. Calibration parameters are applied to video
streams from the first video camera and the second video
camera.
BRIEF DESCRIPTION OF THE FIGURES
[0005] The invention is more fully appreciated in connection with
the following detailed description taken in conjunction with the
accompanying drawings, in which:
[0006] FIG. 1 illustrates a calibration system utilized in
accordance with an embodiment of the invention.
[0007] FIG. 2 illustrates processing operations associated with an
embodiment of the invention.
[0008] FIGS. 3 and 4 illustrate image correction operations
performed in accordance with an embodiment of the invention.
[0009] FIGS. 5 and 6 illustrate error correction operations
performed in accordance with an embodiment of the invention.
[0010] Like reference numerals refer to corresponding parts
throughout the several views of the drawings.
DETAILED DESCRIPTION OF THE INVENTION
[0011] An embodiment of the invention enables image stitching at
high frame rates to produce an output video data stream created
from more than one input video stream. Test and calibration
processes using still photographs of charts and video capture of
moving targets enable alignment of the cameras and determine the
known overlap between cameras.
[0012] FIG. 1 illustrates a calibration technique utilized in
accordance with an embodiment of the invention. A first camera 100
with a viewing angle 102 takes an image of a test pattern 104.
Similarly, a second camera 106 with a viewing angle 108 takes an
image of the test pattern 104. The viewing angles have an overlap
region 110. The cameras 100 and 106 have fixed positions in mount
112. Each camera delivers image data to an image processor 114. The
image processor 114 may be a computer with a central processor,
graphics processing unit and/or an additional processor.
[0013] Still photographs are used to determine the fixed overlap
area 110 from one image sensor to the adjacent image sensor based
on alignment within the final system, or based on a standardized
calibration fixture. Sharp lines of the test pattern 104 allow the
image processor 144 to program the known overlap at the pixel
level. That is, the image processor 114 accommodates for any
vertical, horizontal or rotational misalignment. The test pattern
104 also allows the image processor 114 to determine any focus
errors or areas of soft focus for each image sensor so that image
correction processing can be applied later to produce uniformly
sharp images.
[0014] The test pattern 104 may be in the form of a grid. In this
case, the field of view of each image capture device will have
areas where the image has some distortion caused by either the lens
or some other irregularity in the system. To produce an output
video stream from more than one video input, the images should
appear to have no distortion or uniform distortion along the edge
where the input streams are joined. By calculating a known
distortion for each image capture device, distortion for individual
cameras is corrected. Since resulting distortions for points in the
image plane will be known for each image capture device, the
distortion between images will be corrected by image processing to
create an image with minimal distortion between camera data
streams.
[0015] The test pattern 104 may also be in the form of a color
chart and/or uniform grey chart. Such a test pattern allows the
image processor 114 to analyze any potential differences in color
accuracy, relative illumination, and relative uniformity between
cameras. These differences are stored as correction factors
utilized by the image processor 114 to reduce noticeable
differences between image streams.
[0016] This calibration process allows for an output video stream
to be assembled from multiple video streams with the viewer being
unable to perceive any meaningful change in image quality
throughout an entire field of view, including up to 360 degrees
along multiple axes.
[0017] FIG. 2 illustrates processing operations associated with an
embodiment of the invention. The image processor 114 simultaneously
captures frames from the first camera 100 and the second camera
106, as shown with block 200. Stored known camera distortion
parameters 202 are applied by the image processor 204. The frames
may then be converted from rectangular coordinates to cylindrical
or spherical coordinates 206. It is then determined whether
alignment has been calculated 208. If not, alignment is calculated
210 and the alignment calculation is stored 212 and processing
proceeds to block 214. At block 214, adjacent frames are stitched
and blended based on the alignment calculation 214. Adjacent frames
are then applied as output 216 to a display.
[0018] Thus, through the calibration process, camera distortion and
alignment parameters are obtained. These parameters are
subsequently utilized as additional frames are received.
[0019] The following is an example of lens distortion correction
and perspective correction applied to an image in accordance with
an embodiment of the invention. Consider the following parameters.
[0020] Image Size: 1280 px.times.720 px [0021] Image Center: (640,
360) [0022] Distortion Size: 1 (Scaling factor, 0-1) [0023]
Distortion: 0.03 (Distortion correction polynomial coefficient)
[0024] Distant Param: 103.00 px (Focal length in pixels)
[0025] The following code implements operations of FIG. 2.
TABLE-US-00001 vec2 tc = gl_TexCoord[0].st; // Lens Distortion
Correction vec2 P = (tc / imageCenter) - 1.; // to normalized image
coordinates P /= distortionSize; vec2 Pp = P / (1. - distortion *
dot(P, P)); P *= distortionSize; tc = (Pp + 1.) * imageCenter; //
back to pixel coordinates // Cartesian coordinates tc -= vec2(
imageCenter.x, imageCenter.y ); // Sphere(FishEye)-to-Erectangular
float phi = tc.s / distanceparam; float theta = -tc.t /
distanceparam + (M_PI/2.); if (theta < 0.0) { theta = -theta;
phi += M_PI; } if (theta > M_PI) { theta = M_PI - (theta -
M_PI); phi += M_PI ; } float s = sin(theta); vec2 v = vec2(s *
sin(phi), cos(theta)); float r = length(v); theta = distanceparam *
atan(r, s * cos(phi)); tc = v * (theta / r); //
Erectangular-to-Cylindrical tc.t = distanceparam * atan(tc.t /
distanceparam ); //Pixel coordinates tc += vec2( imageCenter.x -
0.5, imageCenter.y - 0.5);
[0026] In one embodiment, the process of automated alignment uses
processing techniques implemented in OpenCV to determine the offset
between two images that have overlapping regions. For example,
feature matching and homography calculations may be used.
In one embodiment, manual alignment may be used to manually
position image frames in a panoramic frame until the overlapping
regions are seamless. When dealing with a 360-degree panoramic
frame, the beginning and end of the frame need to be aligned and
stitched together. To accomplish this there are three parameters
that need to be calibrated. [0027] Overlap--The amount of pixel
overlap between the beginning and end of the frame. [0028]
Twist--The amount of vertical error between the beginning and end
of the frame. [0029] Blend Width--The width of the blending area
across the overlapping region.
[0030] Those skilled in the art will appreciate that the invention
may be used in the production of video camera systems that create
ultra wide-angle video streams or panoramic video. The invention
can also be used post-production to address camera performance
issues in the field. If cameras are knocked out of alignment during
use, the calibration and alignment steps can re-calibrate the
system, preventing the unnecessary replacement of certain parts or
the whole system.
[0031] This invention can also be used when creating camera systems
in multiple axes. For example, if camera systems are stacked along
a vertical axis to create a "tall" cylindrical video stream, the
alignment and calibration process can be used to ensure that camera
performance is set within a pre-defined standard so there is not a
wide variation in image performance between the separate video
camera systems.
[0032] The invention also allows multiple cameras to be calibrated
within a specific performance range. If there are multiple camera
systems being used at the same event, it is critical that they all
have image performance within a defined range to reduce variation
in image performance/quality between camera systems. The invention
allows for calibration across a group of panoramic or wide-angle
camera systems to ensure video outputs appear consistent to the end
user.
[0033] A reduction in bandwidth can be achieved by calibrating the
Lens Distortion Parameter and the Camera Field of View (FOV) for
each camera in the system. When the system processes each camera's
video stream frame by frame it will use the Lens Distortion
Parameter and Camera FOV to correct the inherent lens
distortion/perspective in each frame. Correcting the video frames
with these parameters warps the pixels to a certain degree, causing
the frame to be cropped in order to get rid of the warping affects.
For example, FIG. 3 illustrates distortion 300 between individual
images 302 and 304. As shown in FIG. 4, cropping lines 400 and 402
may be used to compensate for this warping affect and provide a
more uniform viewing area that is pleasing to the eye.
[0034] The image processor 114 may execute software with calibrated
parameters that reduce the overall time to stitch camera frames
together. These parameters may include FOV overlap, image
alignment, individual camera illumination correction and system
relative illumination.
[0035] Calibrating the image alignment and exposure values for the
system can reduce the overall per-frame system processing time by
approximately .about.400 ms and .about.450 ms respectively. This
improvement is critical for live viewing applications, or
applications when the camera may be turned on and off
frequently.
[0036] Image alignment is the parameter that defines the offset
between each camera in order to stitch and produce a complete
360.degree. panoramic video. Each camera's exposure settings should
be set to be the same and change at the same rate to eliminate
under-exposed and over-exposed areas in the 360-degree image.
Dynamic range software can help compensate for cameras that are
under or over exposed during difficult lighting conditions when the
system is set to a single exposure level.
[0037] The alignment and calibration techniques may be used for
initial system set up and initial video streaming. Further, they
may be used once the system is operative and is otherwise streaming
video. In such a context, real world problems occur, such as camera
faults and subsequent camera displacement or offset. Techniques of
the invention address these issues.
[0038] FIG. 5 illustrates processing associated with one such
technique. The image processor 114 streams video from multiple
cameras 500. The image processor 114 is configured to check for a
faulty camera 502 (e.g., a lost signal). If a faulty camera exists,
the image processor 114 maximizes the field of view of adjacent
cameras 504. As shown in FIG. 1, an overlapping region 110 exists
between cameras. So for example, if camera 100 is faulty, the full
field of view 108 of camera 106 would be utilized.
[0039] FIG. 6 illustrates processing associated with a camera
misalignment after initial alignment and calibration. The image
processor 114 streams video from multiple cameras 600. The image
processor 114 is configured to check for an offset camera 602
(e.g., a camera misaligned from its original position). If an
offset camera is identified, the offset is evaluated 604. For
example, the displacement between matching pixels of adjacent video
streams may be evaluated. Also, outside sensors may be used to
determine this displacement, such as accelerometers or digital
gyroscopes. This displacement is then used to form updated
alignment parameters (e.g., the original alignment parameters are
compensated to incorporate the identified displacement) 606. Video
streams are subsequently stitched based on the updated alignment
parameters 608.
[0040] An embodiment of the present invention relates to a computer
storage product with a computer readable storage medium having
computer code thereon for performing various computer-implemented
operations. The media and computer code may be those specially
designed and constructed for the purposes of the present invention,
or they may be of the kind well known and available to those having
skill in the computer software arts. Examples of computer-readable
media include, but are not limited to: magnetic media such as hard
disks, floppy disks, and magnetic tape; optical media such as
CD-ROMs, DVDs and holographic devices; magneto-optical media; and
hardware devices that are specially configured to store and execute
program code, such as application-specific integrated circuits
("ASICs"), programmable logic devices ("PLDs") and ROM and RAM
devices. Examples of computer code include machine code, such as
produced by a compiler, and files containing higher-level code that
are executed by a computer using an interpreter. For example, an
embodiment of the invention may be implemented using JAVA.RTM.,
C++, or other object-oriented programming language and development
tools. Another embodiment of the invention may be implemented in
hardwired circuitry in place of, or in combination with,
machine-executable software instructions.
[0041] The foregoing description, for purposes of explanation, used
specific nomenclature to provide a thorough understanding of the
invention. However, it will be apparent to one skilled in the art
that specific details are not required in order to practice the
invention. Thus, the foregoing descriptions of specific embodiments
of the invention are presented for purposes of illustration and
description. They are not intended to be exhaustive or to limit the
invention to the precise forms disclosed; obviously, many
modifications and variations are possible in view of the above
teachings. The embodiments were chosen and described in order to
best explain the principles of the invention and its practical
applications, they thereby enable others skilled in the art to best
utilize the invention and various embodiments with various
modifications as are suited to the particular use contemplated. It
is intended that the following claims and their equivalents define
the scope of the invention.
* * * * *