U.S. patent application number 12/959137 was filed with the patent office on 2011-09-08 for object detection and rendering for wide field of view (wfov) image acquisition systems.
This patent application is currently assigned to TESSERA TECHNOLOGIES IRELAND LIMITED. Invention is credited to Petronel Bigioi, Peter Corcoran, Alexandru Drimbarean, Mihnea Gangea, Piotr Stec.
Application Number | 20110216157 12/959137 |
Document ID | / |
Family ID | 44530984 |
Filed Date | 2011-09-08 |
United States Patent
Application |
20110216157 |
Kind Code |
A1 |
Bigioi; Petronel ; et
al. |
September 8, 2011 |
Object Detection and Rendering for Wide Field of View (WFOV) Image
Acquisition Systems
Abstract
An image acquisition device having a wide field of view includes
a lens and image sensor configured to capture an original wide
field of view (WFoV) image with a field of view of more than
90.degree.. The device has an object detection engine that includes
one or more cascades of object classifiers, e.g., face classifiers.
A WFoV correction engine may apply rectilinear and/or cylindrical
projections to pixels of the WFoV image, and/or non-linear,
rectilinear and/or cylindrical lens elements or lens portions serve
to prevent and/or correct distortion within the original WFoV
image. One or more objects located within the original and/or
distortion-corrected WFoV image is/are detectable by the object
detection engine upon application of the one or more cascades of
object classifiers.
Inventors: |
Bigioi; Petronel; (Galway,
IE) ; Drimbarean; Alexandru; (Galway, IE) ;
Gangea; Mihnea; (Bucuresti, RO) ; Stec; Piotr;
(Galway, IE) ; Corcoran; Peter; (Claregalway,
IE) |
Assignee: |
TESSERA TECHNOLOGIES IRELAND
LIMITED
Galway
IE
|
Family ID: |
44530984 |
Appl. No.: |
12/959137 |
Filed: |
December 2, 2010 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
61311264 |
Mar 5, 2010 |
|
|
|
Current U.S.
Class: |
348/36 ;
348/E5.024 |
Current CPC
Class: |
G06K 9/6257 20130101;
G06T 2207/30201 20130101; H04N 5/23219 20130101; H04N 5/23238
20130101; G02B 21/22 20130101; G06T 5/006 20130101 |
Class at
Publication: |
348/36 ;
348/E05.024 |
International
Class: |
H04N 5/225 20060101
H04N005/225 |
Claims
1. An image acquisition device having a wide field of view,
comprising: at least one non-linear lens and image sensor
configured to capture an original wide field of view (WFoV) image
with a field of view of more than 90.degree., wherein the
non-linear lens is configured to project a center region of a scene
onto the middle portion of the image sensor such as to directly
provide a rectilinear mapping of the center region; a control
module; an object detection engine comprising one or more cascades
of regular object classifiers; a WFoV correction engine configured
to correct distortion within the original WFoV image; wherein the
WFoV correction engine processes raw image data of the original
WFoV image including applying a cylindrical projection of outer
pixels of the original WFoV image, and combining center pixels and
modified outer pixels to generate a distortion-corrected WFoV
image; and wherein one or more objects located within the center or
outer pixels, or both, of the distortion-corrected WFoV image are
detectable by the object detection engine upon application of the
one or more cascades of regular object classifiers.
2. The device of claim 1, further comprising: a full frame buffer
coupled with the image sensor for acquiring raw image data; a
mixer; and a zoom and pan engine.
3. The device of claim 1, further comprising an object tracking
engine.
4. The device of claim 1, wherein the object classifiers comprise
face classifiers.
5. The device of claim 1, wherein the regular object classifiers
comprise rectangular object classifiers.
6. An image acquisition device having a wide field of view,
comprising: at least one non-linear lens and image sensor
configured to capture an original wide field of view (WFoV) image
with a field of view of more than 90.degree., wherein the
non-linear lens is configured to project a center region of a scene
onto the middle portion of the image sensor such as to directly
provide a rectilinear mapping of the center region; a control
module; an object detection engine comprising one or more cascades
of modified object classifiers; wherein the modified object
classifiers comprise a subset of cylindrical classifiers to be
applied to objects appearing in outer pixels of the WFoV image; and
wherein one or more objects located within the center or outer
pixels, or both, of the original WFoV image are detectable by the
object detection engine upon application of the one or more
cascades of modified object classifiers, including a subset of
regular classifiers and the subset of cylindrical classifiers,
respectively.
7. The device of claim 6, further comprising: a full frame buffer
coupled with the image sensor for acquiring raw image data; a
mixer; and a zoom and pan engine.
8. The device of claim 6, further comprising an object tracking
engine
9. The device of claim 6, further comprising a WFoV correction
engine configured to correct distortion within the original image;
and wherein the WFoV correction engine processes raw image data of
the original WFoV image including applying a cylindrical mapping of
outer pixels of the original WFoV image, and combining center
pixels and modified outer pixels to generate a distortion-corrected
WFoV image.
10. The device of claim 6, wherein the object classifiers comprise
face classifiers.
11. The device of claim 6, wherein the regular object classifiers
comprise rectangular object classifiers.
12. A method of acquiring wide field of view images with an image
acquisition device having at least one lens and image sensor
configured to capture an original wide field of view (WFoV) image
with a field of view of more than 90.degree., wherein the method
comprises: acquiring the original WFoV image, including utilizing
at least one non-linear lens to project a center region of a scene
onto a middle portion of the image sensor such as to directly
provide a rectilinear mapping of the center region; correcting
distortion within the original WFoV image by processing raw image
data of the original WFoV image including applying a cylindrical
projection of outer pixels of the original WFoV image, and
combining center pixels and modified outer pixels to generate a
distortion-corrected WFoV image; and detecting one or more objects
by applying one or more cascades of regular object classifiers to
one or more objects located within the center or outer pixels, or
both, of the distortion-corrected WFoV image.
13. The method of claim 12, further comprising tracking one or more
detected objects over multiple sequential frames.
14. The method of claim 12, wherein the object classifiers comprise
face classifiers.
15. The method of claim 12, wherein the regular object classifiers
comprise rectangular object classifiers.
16. A method of acquiring wide field of view images with an image
acquisition device having at least one lens and image sensor
configured to capture an original wide field of view (WFoV) image
with a field of view of more than 90.degree., wherein the method
comprises: acquiring the original WFoV image, including utilizing
at least one non-linear lens to project a center region of a scene
onto a middle portion of the image sensor such as to directly
provide a rectilinear mapping of the center region; applying one or
more modified object classifiers, comprising applying a subset of
cylindrical classifiers to objects appearing in outer pixels of the
WFoV image; and applying a subset of regular classifiers to objects
appearing in center pixels of the WFoV image; detecting one or more
objects located within center or outer pixels, or both, of the
original WFoV image by the applying of the one or more cascades of
modified object classifiers, including the applying of the subset
of regular classifiers and the applying of the subset of
cylindrical classifiers, respectively.
17. The method of claim 16, further comprising tracking one or more
detected objects over multiple sequential frames.
18. The method of claim 16, further comprising correcting
distortion within the original WFoV image by processing raw image
data of the original WFoV image including applying a cylindrical
mapping of outer pixels of the original WFoV image, and combining
center pixels and modified outer pixels to generate a
distortion-corrected WFoV image.
19. The method of claim 16, wherein the object classifiers comprise
face classifiers.
20. The method of claim 16, wherein the regular object classifiers
comprise rectangular object classifiers.
21. One or more processor-readable media having embedded therein
code for programming a processor to perform a method of processing
wide field of view images acquired with an image acquisition device
having an image sensor and at least one non-linear lens to project
a center region of a scene onto a middle portion of the image
sensor such as to directly provide a rectilinear mapping of the
center region to acquire an original wide field of view (WFoV)
image with a field of view of more than 90.degree., wherein the
method comprises: correcting distortion within the original WFoV
image by processing raw image data of the original WFoV image
including applying a cylindrical projection of outer pixels of the
original WFoV image, and combining center pixels and modified outer
pixels to generate a distortion-corrected WFoV image; and detecting
one or more objects by applying one or more cascades of regular
object classifiers to one or more objects located within the center
or outer pixels, or both, of the distortion-corrected WFoV
image.
22. The one or more processor-readable media of claim 21, wherein
the method further comprises tracking one or more detected objects
over multiple sequential frames.
23. The one or more processor-readable media of claim 21, wherein
the object classifiers comprise face classifiers.
24. The one or more processor-readable media of claim 21, wherein
the regular object classifiers comprise rectangular object
classifiers.
25. One or more processor-readable media having embedded therein
code for programming a processor to perform a method of processing
wide field of view images acquired with an image acquisition device
having an image sensor and at least one non-linear lens to project
a center region of a scene onto a middle portion of the image
sensor such as to directly provide a rectilinear mapping of the
center region to acquire an original wide field of view (WFoV)
image with a field of view of more than 90.degree., wherein the
method comprises: applying one or more modified object classifiers,
comprising applying a subset of cylindrical classifiers to objects
appearing in outer pixels of the WFoV image; and applying a subset
of regular classifiers to objects appearing in center pixels of the
WFoV image; detecting one or more objects located within center or
outer pixels, or both, of the original WFoV image by the applying
of the one or more cascades of modified object classifiers,
including the applying of the subset of regular classifiers and the
applying of the subset of cylindrical classifiers,
respectively.
26. The one or more processor-readable media of claim 25, wherein
the method further comprises tracking one or more detected objects
over multiple sequential frames.
27. The one or more processor-readable media of claim 25, wherein
the method further comprises correcting distortion within the
original WFoV image by processing raw image data of the original
WFoV image including applying a cylindrical mapping of outer pixels
of the original WFoV image, and combining center pixels and
modified outer pixels to generate a distortion-corrected WFoV
image.
28. The one or more processor-readable media of claim 25, wherein
the object classifiers comprise face classifiers.
29. The one or more processor-readable media of claim 25, wherein
the regular object classifiers comprise rectangular object
classifiers.
Description
PRIORITY
[0001] This application claims the benefit of priority under 35 USC
.sctn.119 to U.S. provisional patent application No. 61/311,264,
filed Mar. 5, 2010. This application is one of a series of
contemporaneously-filed patent applications including United States
patent application (Atty. Docket FN-353A-US, FN-353B-US, and
FN-353C-US), each of which are incorporated by reference.
BACKGROUND
[0002] Face detection methods have become very well established
within digital cameras in recent years. This technology brings a
range of benefits including enhanced acquisition of the main image
and adaptation of the acquisition process to optimized image
appearance and quality based on the detected faces.
[0003] More recently, newer consumer cameras have begun to feature
wide field of view (WFOV) imaging systems and as the benefits of
obtaining a wider scene become apparent to consumers, it is
expected that further growth will ensue in such imaging systems
along with an ability to achieve even wider fields of view over
time. In professional cameras, such WFOV imaging systems are better
known, the most well known being the fish-eye lens. WFOV imaging
systems are also used in a range of applications including Google's
"street-view" technology and for some video-phone systems where
they enable a number of people sitting at a table to be imaged by a
single sensor and optical system.
[0004] Now mapping a WFOV image onto a rectilinear image sensor is
non-trivial and a wide range of different techniques are available
depending on the exact form of the WFOV lens and associated optical
elements. The desired image perspective is also important.
[0005] Unfortunately due to the complexity of WFOV imaging systems
the benefits of face detection technologies have not been
successfully applied to such systems. In particular, faces near the
center of a WFOV camera appear closer to the camera and experience
some geometrical distortions. Faces about mid-way from the center
appear at approximately the correct distances from the camera and
experience less significant distortions. Faces towards the edge
experience very significant geometrical distortions. The exact
nature of each of these types of perspective and geometrical
distortion depend on the nature of the lens and optical system.
[0006] Clearly a conventional face detection or face tracking
system employing rectangular classifiers or integral image
techniques cannot be conveniently applied directly to such faces.
Accordingly methods are desired to adapt and compensate for image
distortions within such WFOV imaging systems so that face detection
technologies can be successfully employed in devices like digital
cameras and video phone systems.
[0007] The following is from
http://www.panorama-numerique.com/squeeze/squeeze.htm, where it is
referred to as "Correcting wider than 90.degree. rectilinear images
to print or to display architecture panoramas," by Georges Lagarde.
The indicated point is to remove stretching near the sides of a
wide angle shot. Mr. Lagarde indicates that one simply has to "just
squeeze your panos!" However, in practice, there are greater
complexities that than. This application provides several
embodiments after this introduction for displaying panoramas
without all the inherent distortion.
[0008] Mr. Lagarde points out that 180.degree. panoramic images
require large screen real-estate. Reduced to a more usual size, Mr.
Lagarde presents the examples illustrated at FIGS. 1A-1G. While
panoramic images are typically difficult to appraise, displaying
them in a narrow window has generally been avoided, and instead a
1280.times.1024 screen or "larger" and a fast Internet connection
may be typically recommended.
Mr. Lagarde points out that the exact same source images of FIGS.
1A-1G (showing the Prefecture building in Grenoble, France) were
used in a previous tutorial: Rectilinear/cylindric/equirectangular
selection made easy, and that different but acceptable panoramic
images can result from stitching the same source images and then
using different projection modes is implied here and there.
[0009] FIG. 1A illustrates Piazza Navona, Roma by Gaspar Van
Wittel, 1652-1736 (Museo Thyssen-Bornemisza, Madrid).
[0010] Mr. Lagarde indicates that most photographers restrict
themselves to subjects which can be photographed with a rectilinear
lens (plane projection). A small number of them sometimes use a
fisheye lens (spherical projection) or a rotating lens camera
(cylindrical projection) or a computer (stitcher programs make use
of various projection modes), but when the field of view
(horizontal FOV and/or vertical FOV) is higher than 90 degrees (or
about, this actually depends on the subject) they are disturbed by
the "excessive wide-angle distortion" found in the resulting
images.
[0011] Adapting the usual projection modes to the subject and/or
using multiple local projections to avoid this distortion is a
violation of the classical perspective rules, but escaping
classical perspective rules is exactly what sketchers and painters
always did to avoid unpleasant images. Mr. Lagarde points out that
this was explained by Anton Maria Zanetti and Antonio Conti using
the words of their times ("Il Professore m'entendara") when they
described how the camera ottica was used by the seventeenth century
Venetian masters. Because the field of view of the lenses available
then was much lower than 90.degree., that a camera oscura was not
able to display the very wide vedute they sketched and painted is
evident: the solution was to record several images and to stitch
them onto the canvas to get a single view (strangely enough, that
the field of view is limited to about 90 degrees when one uses
classical perspective--aka rectilinear projection on a vertical
plane--is not handled in most perspective treatises.)
[0012] Equivalent "tricks" can be used for photographic images:
[0013] Use of several projection planes--their number and location
depending of the subject--for a single resulting image. This is the
method explained by L. Zelnik-Manor in Squaring the Circle in
Panoramas (see references.) [0014] Use of several projection
modes--the selected modes depending of the subject--for a single
resulting image. This is the method proposed by Buho (Eric S.) and
used by Johnh (John Houghton) in Hybrid Rectilinear &
Cylindrical projections (see references.) [0015] Use of an "altered
rectilinear" projection (thus no more rectilinear) where the
modification is a varying horizontal compression, null in the
center, high near the sides). This is the method proposed by
Olivier_G (Olivier Gallen) in Panoramas: la perspective classique
ne s'applique plus! (see references.) [0016] Use of "squeezed
rectilinear" projection (neither an actual rectilinear one) where
the modification is a varying horizontal and vertical compression,
null near the horizon (shown as a red line in the examples), null
near a vertical line which goes through to the main vanishing point
(shown as a blue line in the examples), increasing like tangent
(angle) toward the sides (where angle correspond to the angular
distance between the point and the line.)
[0017] If photographers like the results, no doubt they will use
that.
Example 1
Cylindrical--180.degree.
[0018] In a first example, referring now to FIG. 1B, an image is
shown that is a 180.degree. panorama where cylindrical projection
mode is used to show a long building viewed from a short distance.
Most people dislike images like this one, where except for the
horizon, every straight horizontal line is heavily curved.
Example 2
Rectilinear--155.degree.
[0019] The next image shown in FIG. 1C illustrates an attempt to
use the rectilinear projection mode: every straight line in the
buildings is rendered as a straight line. But, while rectilinear
projection works well when field of view is lower than 90 degrees,
it should never be used when field of view is larger than 120
degrees. In this image, though the field of view was restricted to
155 degree (original panorama corresponds to 180.degree.), the
stretching is too high in the left and right parts and the result
utterly unacceptable.
Example 3
Squeezed Rectilinear--155.degree.
[0020] Referring to FIG. 1D, because digital images can be squeezed
at will, rather than discarding this previous rectilinear image,
one can correct the excessive stretching. The result is no more
rectilinear (diagonal lines are somewhat distorted) but a much
wider part of the buildings now have an acceptable look. The
variable amount of squeezing I used is shown by the dotted line
near the top side: the more close the dots are, the more compressed
was the corresponding part of the rectilinear original.
Example 4
Edges, from the 180.degree. Cylindrical Version
[0021] Referring to FIG. 1E, the rendering of the main building is
much better. Note that this view looks like it were taken from a
more distant point of view than in the cylindrical image: this is
not true, the same source images were used for both panoramas.
Example 5
Center, from 155.degree. Squeezed Rectilinear Version
[0022] Referring to FIG. 1F, the left most and right most parts of
the squeezed image are improved, but they are still not very
pleasant. Here is a possible solution, where I used the edge parts
of the cylindrical version in a second layer:
Example 6
Squeezed Rectilinear (Center)+Cylindrical (Left and Right
Edges)-180.degree.
[0023] And finally, referring to FIG. 1G: This view can be compared
with the example of FIG. 1B on the top of this page: each one shows
exactly the same buildings and cars, and each comes from exactly
the same source images.
[0024] The pictured buildings in FIGS. 1B-1G are located on the
sides of a large square but, because there are many large trees on
this square, standing back enough for a large field of view is not
possible. The image shown in FIG. 1B illustrates photos that were
actually taken at a rather short distance from the main building,
while FIG. 1G suggests the viewer being much more distant from this
building.
BRIEF DESCRIPTION OF THE DRAWINGS
[0025] FIGS. 1A-1G illustrate various conventional attempts to
avoid distortion in images with greater than 90.degree. field of
view.
[0026] FIG. 2 schematically illustrates a wide field of view (WFOV)
system that in one embodiment incorporates a face tracker.
[0027] FIG. 3(a) illustrates a wide horizontal scene mapped onto a
full extent of an image sensor.
[0028] FIG. 3(b) illustrates a wide horizontal scene not mapped
onto a full extent of an image sensor, and instead a significant
portion of the sensor is not used.
[0029] FIG. 4 illustrates the first four Haar classifiers used in
face detection.
[0030] FIGS. 4(a)-4(c) illustrate magnification of a person
speaking among a group of persons within a WDOF image.
[0031] FIGS. 5(a)-5(c) illustrate varying the magnification of a
person speaking among a group of persons within a WDOF image,
wherein the degree of magnification may vary depending on the
strength or loudness of the speaker's voice.
DETAILED DESCRIPTIONS OF THE EMBODIMENTS
[0032] An image acquisition device having a wide field of view is
provided. The device includes at least one lens and image sensor
configured to capture an original wide field of view (WFoV) image
with a field of view of more than 90.degree.. The device also
includes a control module and an object detection engine that
includes one or more cascades of regular object classifiers. A WFoV
correction engine of the device is configured to correct distortion
within the original image. The WFoV correction engine processes raw
image data of the original WFoV image. A rectilinear projection of
center pixels of the original WFoV image is applied. A cylindrical
projection of outer pixels of the original WFoV image is also
applied. Modified center and outer pixels are combined to generate
a distortion-corrected WFoV image. One or more objects located
within the center or outer pixels, or both, of the
distortion-corrected WFoV image are detectable by the object
detection engine upon application of the one or more cascades of
regular object classifiers.
[0033] The applying of the rectilinear projection to center pixels
may also include applying a regular rectilinear projection to an
inner portion of the center pixels and a squeezed rectilinear
projection to an outer portion of the center pixels. The applying
of the squeezed rectilinear projection to the outer portion of the
center pixels may also include applying an increasingly squeezed
rectilinear projection in a direction from a first boundary with
the inner portion of the center pixels to a second boundary with
the outer pixels.
[0034] Another image acquisition device having a wide field of view
is provided. The device includes at least one lens and image sensor
configured to capture an original wide field of view (WFoV) image
with a field of view of more than 90.degree., a control module, and
an object detection engine that includes one or more cascades of
modified object classifiers. The modified object classifiers
include a first subset of rectilinear classifiers to be applied to
objects appearing in center pixels of the WFoV image, and a second
subset of cylindrical classifiers to be applied to objects
appearing in outer pixels of the WFoV image. One or more objects
located within the center or outer pixels, or both, of the original
WFoV image are detectable by the object detection engine upon
application of the one or more cascades of modified object
classifiers, including the first subset of rectilinear classifiers
and the second subset of cylindrical classifiers, respectively.
[0035] The first subset of rectilinear classifiers may include a
subset of regular rectilinear classifiers with which objects
appearing in an inner portion of the center pixels are detectable,
and a subset of squeezed rectilinear classifiers with which objects
appearing in an outer portion of the center pixels are detectable.
The subset of squeezed rectilinear classifiers may include subsets
of increasingly squeezed rectilinear classifiers with which objects
appearing in the outer portion of the center pixels are
increasingly detectable in a direction from a first boundary with
the inner portion of the center pixels to a second boundary with
the outer pixels.
[0036] The device may also include a WFoV correction engine
configured to correct distortion within the original image. The
WFoV correction engine may process raw image data of the original
WFoV image. A rectilinear mapping of center pixels of the original
WFoV image may be applied. A cylindrical mapping of outer pixels of
the original WFoV image may also be applied. Modified center and
outer pixels may be combined to generate a distortion-corrected
WFoV image.
[0037] A method is provided for acquiring wide field of view images
with an image acquisition device having at least one lens and image
sensor configured to capture an original wide field of view (WFoV)
image with a field of view of more than 90.degree.. The method
includes acquiring the original WFoV image. Distortion is corrected
within the original WFoV image by processing raw image data of the
original WFoV image. A rectilinear projection is applied to center
pixels of the original WFoV image and a cylindrical projection is
applied to outer pixels of the original WFoV image. Modified center
and outer pixels are combined to generate a distortion-corrected
WFoV image. One or more cascades of regular object classifiers are
applied to detect one or more objects located within the center or
outer pixels, or both, of the distortion-corrected WFoV image upon
application of the one or more cascades of regular object
classifiers.
[0038] The applying a rectilinear projection to center pixels may
include applying a regular rectilinear projection to an inner
portion of the center pixels and a squeezed rectilinear projection
to an outer portion of the center pixels. The applying of a
squeezed rectilinear projection to the outer portion of the center
pixels may include applying an increasingly squeezed rectilinear
projection in a direction from a first boundary with the inner
portion of the center pixels to a second boundary with the outer
pixels.
[0039] A further method is provided for acquiring wide field of
view images with an image acquisition device having at least one
lens and image sensor configured to capture an original wide field
of view (WFoV) image with a field of view of more than 90.degree..
The method includes acquiring the original WFoV image. One or more
cascades of modified object classifiers are applied. A first subset
of rectilinear classifiers is applied to objects appearing in
center pixels of the WFoV image, and a second subset of cylindrical
classifiers is applied to objects appearing in outer pixels of the
WFoV image. One or more objects located within the center or outer
pixels, or both, of the original WFoV image is/are detected by the
applying of the modified object classifiers, including the applying
of the first subset of rectilinear classifiers and the applying of
the second subset of cylindrical classifiers, respectively.
[0040] The applying of the first subset of rectilinear classifiers
may include applying a subset of regular rectilinear classifiers
with which objects appearing in an inner portion of the center
pixels are detectable, and/or applying a subset of squeezed
rectilinear classifiers with which objects appearing in an outer
portion of the center pixels are detectable. The applying of the
subset of squeezed rectilinear classifiers may include applying
subsets of increasingly squeezed rectilinear classifiers with which
objects appearing in the outer portion of the center pixels are
increasingly detectable in a direction from a first boundary with
the inner portion of the center pixels to a second boundary with
the outer pixels.
[0041] The method may include correcting distortion within the
original image by processing raw image data of the original WFoV
image including applying a rectilinear mapping of center pixels of
the original WFoV image and a cylindrical mapping of outer pixels
of the original WFoV image, and combining modified center and outer
pixels to generate a distortion-corrected WFoV image.
[0042] One or more processor-readable media having embedded therein
code for programming a processor to perform any of the methods
described herein.
[0043] Another image acquisition device having a wide field of view
is provided. The device includes at least one non-linear lens and
image sensor configured to capture an original wide field of view
(WFoV) image with a field of view of more than 90.degree.. The
non-linear lens is configured to project a center region of a scene
onto the middle portion of the image sensor such as to directly
provide a rectilinear mapping of the center region. The device also
includes an object detection engine including one or more cascades
of regular object classifiers. A WFoV correction engine of the
device is configured to correct distortion within the original WFoV
image. The WFoV correction engine processes raw image data of the
original WFoV image. A cylindrical projection of outer pixels of
the original WFoV image is applied. Center pixels and modified
outer pixels are combined to generate a distortion-corrected WFoV
image. One or more objects located within the center or outer
pixels, or both, of the distortion-corrected WFoV image are
detectable by the object detection engine upon application of the
one or more cascades of regular object classifiers.
[0044] Another image acquisition device having a wide field of view
is provided. The device includes at least one non-linear lens and
image sensor configured to capture an original wide field of view
(WFoV) image with a field of view of more than 90.degree.. The
non-linear lens is configured to project a center region of a scene
onto the middle portion of the image sensor such as to directly
provide a rectilinear mapping of the center region. An object
detection engine includes one or more cascades of modified object
classifiers including a subset of cylindrical classifiers to be
applied to objects appearing in outer pixels of the WFoV image. One
or more objects located within the center or outer pixels, or both,
of the original WFoV image are detectable by the object detection
engine upon application of the one or more cascades of modified
object classifiers, including a subset of regular classifiers and
the subset of cylindrical classifiers, respectively.
[0045] The device may include a WFoV correction engine configured
to correct distortion within the original image. The WFoV
correction engine processes raw image data of the original WFoV
image. A cylindrical mapping of outer pixels of the original WFoV
image is performed. Center pixels and modified outer pixels are
combined to generate a distortion-corrected WFoV image.
[0046] Another method is provided for acquiring wide field of view
images with an image acquisition device having at least one lens
and image sensor configured to capture an original wide field of
view (WFoV) image with a field of view of more than 90.degree.. The
method includes acquiring the original WFoV image, including
utilizing at least one non-linear lens to project a center region
of a scene onto a middle portion of the image sensor such as to
directly provide a rectilinear mapping of the center region.
Distortion is corrected within the original WFoV image by
processing raw image data of the original WFoV image. A cylindrical
projection of outer pixels of the original WFoV image is applied.
Center pixels and modified outer pixels are combined to generate a
distortion-corrected WFoV image. One or more objects are detected
by applying one or more cascades of regular object classifiers to
one or more objects located within the center or outer pixels, or
both, of the distortion-corrected WFoV image.
[0047] A further method is provided for acquiring wide field of
view images with an image acquisition device having at least one
lens and image sensor configured to capture an original wide field
of view (WFoV) image with a field of view of more than 90.degree..
The method includes acquiring the original WFoV image, including
utilizing at least one non-linear lens to project a center region
of a scene onto a middle portion of the image sensor such as to
directly provide a rectilinear mapping of the center region. One or
more modified object classifiers are applied. A subset of
cylindrical classifiers is applied to objects appearing in outer
pixels of the WFoV image, and a subset of regular classifiers is
applied to objects appearing in center pixels of the WFoV image.
One or more objects located within center or outer pixels, or both,
of the original WFoV image are detected by the applying of the one
or more cascades of modified object classifiers, including the
applying of the subset of regular classifiers and the applying of
the subset of cylindrical classifiers, respectively.
[0048] The method may include correcting distortion within the
original WFoV image by processing raw image data of the original
WFoV image, including applying a cylindrical mapping of outer
pixels of the original WFoV image, and combining center pixels and
modified outer pixels to generate a distortion-corrected WFoV
image.
[0049] One or more processor-readable media having embedded therein
code is/are provided for programming a processor to perform any of
the methods described herein of processing wide field of view
images acquired with an image acquisition device having an image
sensor and at least one non-linear lens to project a center region
of a scene onto a middle portion of the image sensor such as to
directly provide a rectilinear mapping of the center region to
acquire an original wide field of view (WFoV) image with a field of
view of more than 90.degree..
[0050] Another image acquisition device having a wide field of view
is provided. The device includes a lens assembly and image sensor
configured to capture an original wide field of view (WFoV) image
with a field of view of more than 90.degree.. The lens assembly
includes a compressed rectilinear lens to capture a center region
of a scene onto a middle portion of the image sensor such as to
directly provide a rectilinear mapping of the center region. The
device also includes a cylindrical lens on one or both sides of the
compressed rectilinear lens to capture outer regions of the scene
onto outer portions of the image sensor such as to directly provide
a cylindrical mapping of the outer regions. An object detection
engine of the device includes one or more cascades of regular
object classifiers. One or more objects located within the center
or outer pixels, or both, of the original WFoV image is/are
detectable by the object detection engine upon application of the
one or more cascades of regular object classifiers.
[0051] Another image acquisition device having a wide field of view
is provided. The device includes a lens assembly and image sensor
configured to capture an original wide field of view (WFoV) image
with a field of view of more than 90.degree.. The lens assembly
includes a lens having a compressed rectilinear center portion to
capture a center region of a scene onto a middle portion of the
image sensor such as to directly provide a rectilinear mapping of
the center region. The lens also includes cylindrical outer
portions on either side of the compressed rectilinear portion to
capture outer regions of the scene onto outer portions of the image
sensor such as to directly provide a cylindrical mapping of the
outer regions. An object detection engine of the device includes
one or more cascades of regular object classifiers. One or more
objects located within the center or outer pixels, or both, of the
original WFoV image is/are detectable by the object detection
engine upon application of the one or more cascades of regular
object classifiers.
[0052] Another image acquisition device having a wide field of view
is provided. The device includes multiple cameras configured to
capture an original wide field of view (WFoV) image with a field of
view of more than 90.degree.. The original wide field of view image
includes a combination of multiple images captured each with one of
the multiple cameras. The multiple cameras include a first camera
having a first image sensor and a compressed rectilinear lens to
capture a center region of a scene onto the first sensor such as to
directly provide a rectilinear mapping of the center region, and a
second camera having a second image sensor and a first cylindrical
lens on a first side of the compressed rectilinear lens to capture
a first outer region of the scene onto the second image sensor such
as to directly provide a cylindrical mapping of the first outer
region, and a third camera having a third image sensor and a second
cylindrical lens on a second side of the compressed rectilinear
lens to capture a second outer region of the scene onto the third
image sensor such as to directly provide a cylindrical mapping of
the second outer region. An object detection engine of the device
includes one or more cascades of regular object classifiers. One or
more objects located within the original wide field of view image
appearing on the multiple cameras of the original WFoV image is/are
detectable by the object detection engine upon application of the
one or more cascades of regular object classifiers.
[0053] Another image acquisition device having a wide field of view
is provided. The device includes multiple cameras configured to
capture an original wide field of view (WFoV) image with a field of
view of more than 90.degree.. The original wide field of view image
includes a combination of multiple images captured each with one of
the multiple cameras. The multiple cameras each utilize a same lens
and include a first camera having a first image sensor utilizing a
compressed rectilinear portion of the lens to capture a center
region of a scene onto the first sensor such as to directly provide
a rectilinear mapping of the center region, and a second camera
having a second image sensor utilizing a first cylindrical portion
of the lens on a first side of the compressed rectilinear portion
to capture a first outer region of the scene onto the second image
sensor such as to directly provide a cylindrical mapping of the
first outer region, and a third camera having a third image sensor
utilizing a second cylindrical portion of the lens on a second side
of the compressed rectilinear portion to capture a second outer
region of the scene onto the third image sensor such as to directly
provide a cylindrical mapping of the second outer region.
[0054] An object detection engine of the device includes one or
more cascades of regular object classifiers. One or more objects
located within the original wide field of view image appearing on
the multiple cameras of the original WFoV image is/are detectable
by the object detection engine upon application of the one or more
cascades of regular object classifiers.
[0055] Any of the devices described herein may include a full frame
buffer coupled with the image sensor for acquiring raw image data,
a mixer, and a zoom and pan engine, and/or an object tracking
engine, just as any of the methods described herein may include
tracking one or more detected objects over multiple sequential
frames. Any of the object classifiers described herein may include
face classifiers or classifiers of other specific objects. Any of
the regular object classifiers described herein may include
rectangular object classifiers.
[0056] Exemplary face region images distorted in a manner like the
building frontages of FIGS. 1B-1G, i.e. distorted by a WFOV, might
have rectilinear distortion similar to FIG. 1C at the edges, and as
in FIG. 1B cylindrical projection.
[0057] The system shown in FIG. 2 includes a wide field of view
(WFOV) lens of for example 120 degrees; a sensor, for example of 3
megapixels or more; a full frame buffer (e.g., from Bayer); a WFOV
correction module; a face detector and face tracker; a zoom and pan
engine; a mixer and a control module. The WFOV system illustrated
at FIG. 1 incorporates lens assembly and corresponding image sensor
which is typically more elongated than a conventional image sensor.
The system further incorporates a face tracking module which
employs one or more cascades of rectangular face classifiers.
[0058] As the system is configured to image a horizontal field of
>90-100 degrees or more, it is desired to process the scene
captured by the system to present an apparently "normal"
perspective on the scene. There are several approaches to this as
exemplified by the example drawn from the architectural perspective
of a long building described in Appendix A. In the context of our
WFOV camera this disclosure is primarily directed at considering
how facial regions will be distorted by the WFOV perspective of
this camera. One can consider such facial regions to suffer similar
distortions to the frontage of the building illustrated in this
attached Appendix. Thus the problem to obtain geometrically
consistent face regions across the entire horizontal range of the
WFOV camera is substantially similar to the architectural problem
described therein.
[0059] Thus, in order to obtain reasonable face regions, it is
useful to alter/map the raw image obtained from the original WFOV
horizontal scene so that faces appear undistorted. Or in
alternative embodiments face classifiers may be altered according
to the location of the face regions within an unprocessed (raw)
image of the scene.
[0060] In a first preferred embodiment the center region of the
image representing up to 100' of the horizontal field of view (FOV)
is mapped using a squeezed rectilinear projection. In a first
embodiment this may be obtained using a suitable non-linear lens
design to directly project the center region of the scene onto the
middle 2/3 of the image sensor. The remaining approximately 1/3
portion of the image sensor (i.e. 1/6 at each end of the sensor)
has the horizontal scene projected using a cylindrical mapping.
Again in a first preferred embodiment the edges of the wide-angle
lens are designed to optically effect said projection directly onto
the imaging sensor.
[0061] Thus, in a first embodiment, the entire horizontal scene is
mapped onto the full extent of the image sensor, as illustrated at
FIG. 3(a).
[0062] Naturally the form and structure of such a complex hybrid
optical lens may not be conducive to mass production thus in an
alternative embodiment a more conventional rectilinear wide-angle
lens is used and the squeezing of the middle 2/3 of the image is
achieved by post-processing the sensor data. Similarly the
cylindrical projections of the outer regions of the WFOV scene are
performed by post processing. In this second embodiment the initial
projection of the scene onto the sensor does not cover the full
extent of the sensor and thus a significant portion of the sensor
area does not contain useful data. The overall resolution of this
second embodiment is reduced and a larger sensor would be used to
achieve similar accuracy to the first embodiment, as illustrated at
FIG. 3(b).
[0063] In a third embodiment some of the scene mappings are
achieved optically, but some additional image post-processing is
used to refine the initial projections of the image scene onto the
sensor. In this embodiment the lens design can be optimized for
manufacturing considerations, a larger portion of the sensor area
can be used to capture useful scene data and the software
post-processing overhead is similar to the pure software
embodiment.
[0064] In a fourth embodiment multiple cameras are configured to
cover overlapping portions of the desired field of view and the
acquired images are combined into a single WFOV image in memory.
Preferably, this plurality of cameras are configured to have the
same optical center, thus mitigating perspective related problems
for foreground objects. In such an embodiment techniques employed
in panorama imaging may be used advantageously to join images at
their boundaries, or to determine the optimal join line where a
significant region of image overlap is available. The following
cases assigned to the same assignee relate to panorama imaging and
are incorporated by reference: Ser. Nos. 12/636,608, 12/636,618,
12/636,629, 12/636,639, and 12/636,647, as are US published apps
nos. U.S. patent application US20060182437, US20090022422,
US20090021576 and US20060268130.
[0065] In one preferred embodiment of the multi-camera WFOV device
three, or more standard cameras with a 60 degree FOV are combined
to provide an overall horizontal WFOV of 120-150 degrees with an
overlap of 15-30 degrees between cameras. The field of view for
such a cameras can be extended horizontally by adding more cameras;
it may be extended vertically by adding an identical array of 3 or
more horizontally aligned cameras facing in a higher (or lower)
vertical direction and with a similar vertical overlap of 15-30
degrees offering a vertical FOV of 90-105 degrees for two such WFOV
arrays. The vertical FOV may be increased by adding further
horizontally aligned cameras arrays. Such configurations have the
advantage that all individual cameras can be conventional
wafer-level cameras (WLC) which can be mass-produced.
[0066] In an alternative multi-cameras embodiment a central WFOV
cameras has its range extended by two side-cameras. The WFOV
cameras can employ an optical lens optimized to provide a 120
degree compressed rectilinear mapping of the central scene. The
side cameras can be optimized to provide a cylindrical mapping of
the peripheral regions of the scene, thus providing a similar
result to that obtained in FIG. 3(a), but using three independent
cameras with independent optical systems rather than a single
sensor/ISP as shown in FIG. 3(b). Again techniques employed in
panorama imaging to join overlapping images can be advantageously
used (see the Panorama cases referred to above herein).
[0067] After image acquisition and, depending on the embodiment,
additional post-processing of the image, we arrive at a mapping of
the image scene with three main regions. Over the middle third of
the image there is a normal rectilinear mapping and the image is
undistorted compared to a standard FOV image; over the next 1/3 of
the image (i.e. 1/6 of image on either side) the rectilinear
projection becomes increasingly squeezed as illustrated in FIGS.
1A-1G; finally, over the outer approximately 1/3 of the image a
cylindrical projection, rather than rectilinear is applied.
[0068] FIG. 3(a) illustrates one embodiment where this can be
achieved using a compressed rectilinear lens in the middle,
surrounded by two cylindrical lenses on either side. In a practical
embodiment all three lenses could be combined into a single lens
structure designed to minimize distortions where the rectilinear
projection of the original scene overlaps with the cylindrical
projection.
[0069] A standard face-tracker can now be applied to the WFOV image
as all face regions should be rendered in a relatively undistorted
geometry.
[0070] In alternative embodiments the entire scene need not be
re-mapped, but instead only the luminance components are re-mapped
and used to generate a geometrically undistorted integral image.
Face classifiers are then applied to this integral image in order
to detect faces. Once faces are detected those faces and their
surrounding peripheral regions can be re-mapped on each frame,
whereas it may be sufficient to re-map the entire scene background,
which is assumed to be static, only occasionally, say every 60-120
image frames. In this way image processing and enhancement can be
focussed on the people in the image scene.
[0071] In alternative embodiments it may not be desirable to
completely re-map the entire WFOV scene due to the computational
burden involved. In such embodiment, referring to U.S. Pat. Nos.
7,460,695, 7,403,643, 7,565,030, and 7,315,631 and US published app
no. 2009-0263022, which are incorporated by reference along with
US20090179998, US20090080713, US 2009-0303342 and U.S. Ser. No.
12/572,930, filed Oct. 2, 2009 by the same assignee. These
references describe predicting face regions (determined from the
previous several video frames). The images may be transformed using
either cylindrical or squeezed rectilinear projection prior to
applying a face tracker to the region. In such an embodiment, it
may be involved from time to time to re-map a WFOV in order to make
an initial determination of new faces within the WFOV image scene.
However, after such initial determination only the region
immediately surrounding each detected face need be re-mapped.
[0072] In certain embodiments, the remapping of the image scene, or
portions thereof, involves the removal of purple fringes (due to
blue shift) or the correction of chromatic aberrations. The
following case is assigned to the same assignee is incorporated by
reference and relates to purple fringing and chromatic aberration
correction: US20090189997.
[0073] In other embodiments a single mapping of the input image
scene is used. If, for example, only a simple rectilinear mapping
were applied across the entire image scene the edges of the image
would be distorted as in FIG. 1C and only across the middle 40% or
so of the image can a conventional face tracker be used.
Accordingly the rectangular classifiers of the face tracker are
modified to take account of the scene mappings across the other 60%
of image scene regions: Over the middle portion of the image they
can be applied unaltered; over the second 30% they are selectively
expanded or compressed in the horizontal direction to account for
the degree of squeezing of the scene during the rectilinear mapping
process. Finally, in the outer 1/3 the face classifiers are adapted
to account for the cylindrical mapping used in this region of the
image scene.
[0074] In order to transform standard rectangular classifiers of a
particular size, say 32.times.32 pixels, it may be advantageous in
some embodiments to increase the size of face classifiers to, for
example, 64.times.64. This larger size of classifier would enable
greater granularity, and thus improved accuracy in transforming
normal classifiers to distorted ones. This comes at the expense of
additional computational burden for the face tracker. However we
note that face tracking technology is quite broadly adopted across
the industry and is known as a robust and well optimized
technology. Thus the trade off of increasing classifiers from
32.times.32 to 64.times.64 for such faces should not cause a
significant delay on most camera or smartphone platforms. The
advantage is that pre-existing classifier cascades can be re-used,
rather than having to train new, distorted ones.
[0075] Having greater granularity for the classifiers is
advantageous particularly when starting to rescale features inside
the classifier individually, based on the distance to the optical
center. In another embodiment, one can scale the whole 22.times.22
(this is a very good size for face classifiers) classifier with
fixed dx,dy (computed as distance from the optical center). Having
larger classifiers does not put excessive strain on the processing.
Advantageously, it is opposite to that, because there are fewer
scales to cover. In this case, the distance to subject is
reduced.
[0076] In an alternative embodiment an initial, shortened chain of
modified classifiers is applied to the raw image (i.e. without any
rectilinear or cylindrical re-mapping). This chain is composed of
some of the initial face classifiers from a normal face detection
chain. These initial classifiers are also, typically, the most
aggressive to eliminate non-faces from consideration. These also
tend to be simpler in form and the first four Haar classifiers from
the Viola-Jones cascade are illustrated in FIG. 4 (these may be
implemented through a 22.times.22 pixel window in another
embodiment).
[0077] Where a compressed rectilinear scaling would have been
employed (as illustrated in FIG. 1F, it is relatively
straightforward to invert this scaling and expand (or contract)
these classifiers in the horizontal direction to compensate for the
distortion of faces in the raw image scene. (In some embodiments
where this distortion is cylindrical towards the edges of the scene
then classifiers may need to be scaled both in horizontal and
vertical directions). Further, it is possible from a knowledge of
the location at which each classifier is to be applied and,
optionally, the size of the detection window, to perform the
scaling of these classifiers dynamically. Thus only the original
classifiers have to be stored together with data on the required
rectilinear compression factor in the horizontal direction. The
latter can easily be achieved using a look-up table (LUT) which is
specific to the lens used.
[0078] This short classifier chain is employed to obtain a set of
potential face regions which may then be re-mapped (using, for
example, compressed rectilinear compression and/or cylindrical
mapping) to enable the remainder of a complete face detection
classifier chain to be applied to each potential face region. This
embodiment relies on the fact that 99.99% of non-face regions are
eliminated by applying the first few face classifiers; thus a small
number of potential face regions would be re-mapped rather than the
entire image scene before applying a full face detection
process.
[0079] In another embodiment, distortion may be compensated by a
method that involves applying geometrical adjustments (function of
distance to optical center) when an integral image is computed (in
the cases where the template matching is done using II) or
compensate for the distortion when computing the sub-sampled image
used for face detection and face tracking (in the cases where
template matching is done directly on Y data).
[0080] Note that face classifiers can be divided into symmetric and
non-symmetric classifiers. In certain embodiments it may be
advantageous to use split classifier chains. For example right and
left-hand face detector cascades may report detection of a
half-face region--this may indicate that a full face is present but
the second half is more or less distorted than would be expected,
perhaps because it is closer to or farther from the lens than is
normal. In such cases a more relaxed half, or full-face detector
may be employed to confirm if a full face is actually present or a
lower acceptance threshold may be set for the current detector. The
following related apps assigned to the same assignee are
incorporated by reference: US2007/0147820, US2010/0053368,
US2008/0205712, US2009/0185753, US2008/0219517 and 2010/0054592,
and U.S. Ser. No. 61/182,625, filed May 29, 2009 and U.S. Ser. No.
61/221,455, filed Jun. 29, 2009.
[0081] In certain embodiments, when a face is tracked across the
scene it may be desired to draw particular attention to that face
and to emphasize it against the main scene. In one exemplary
embodiment, suitable for applications in videotelephony, there may
be one or more faces in the main scene but one (or more) of these
is speaking. It is possible, using a stereo microphone to localize
the speaking face.
[0082] This face regions, and the other foreground regions (e.g.
neck, shoulders & torso) are further processed to magnify them
(e.g., in one embodiment by a factor of x1.8 times) against the
background; in a simple embodiment this magnified face is simply
composited onto the background image in the same location as the
unmagnified original
[0083] In a more sophisticated embodiment the other faces and the
main background of the image are de-magnified and/or squeezed in
order to keep the overall image size self-consistent. This may lead
to some image distortion, particularly surrounding the "magnified"
face, but this helps to emphasize the person speaking as
illustrated in FIGS. 4(a)-4(c). In this case the degree of
magnification is generally <x1.5 to avoid excessive distortion
across the remainder of the image.
[0084] In another embodiment, one can do a background+face mix or
combination using an alpha map without worrying about distortions.
Then, the face that speaks can be placed at the middle of the
frame. In an another variation on this embodiment, the degree of
magnification can be varied according to the strength or loudness
of a speaker's voice, as illustrated at FIGS. 5(a)-5(c).
[0085] In other embodiments based on the same scene re-mapping
techniques, the rendering of the face region and surrounding
portions of the image can be adjusted to emphasize one or more
persons appearing in the final, re-mapped image of the captured
scene. In one embodiment within a videophone system, a stereo
microphone system triangulates the location of the person speaking
and a portion of the scene is zoomed by a factor greater than one.
The remaining portions of the image are zoomed by a factor less
than one, so that the overall image is of approximately the same
dimension. Thus persons appearing in the image appear larger when
they are talking and it is easier for viewers to focus on the
current speaker from a group.
[0086] The present invention is not limited to the embodiments
described above herein, which may be amended or modified without
departing from the scope of the present invention.
[0087] In methods that may be performed according to preferred
embodiments herein and that may have been described above, the
operations have been described in selected typographical sequences.
However, the sequences have been selected and so ordered for
typographical convenience and are not intended to imply any
particular order for performing the operations.
[0088] In addition, all references cited above herein, in addition
to the background and summary of the invention sections, are hereby
incorporated by reference into the detailed description of the
preferred embodiments as disclosing alternative embodiments and
components. Moreover, as extended depth of field (EDOF) technology
may be combined with embodiments described herein into advantageous
alternative embodiments, the following are incorporated by
reference: US published patent applications numbers 20060256226,
20060519527, 20070239417, 20070236573, 20070236574, 20090128666,
20080095466, 20080316317, 20090147111, 20020145671, 20080075515,
20080021989, 20050107741, 20080028183, 20070045991. 20080008041,
20080009562, 20080038325, 20080045728, 20090531723, 20090190238,
20090141163, and 20080002185.
* * * * *
References