U.S. patent application number 13/127474 was filed with the patent office on 2011-09-01 for system and method for creating interactive panoramic walk-through applications.
Invention is credited to Gerard Crittin, David Lindemann, Pierre-Alain Lindemann.
Application Number | 20110211040 13/127474 |
Document ID | / |
Family ID | 41666646 |
Filed Date | 2011-09-01 |
United States Patent
Application |
20110211040 |
Kind Code |
A1 |
Lindemann; Pierre-Alain ; et
al. |
September 1, 2011 |
SYSTEM AND METHOD FOR CREATING INTERACTIVE PANORAMIC WALK-THROUGH
APPLICATIONS
Abstract
System and method of the invention provides for creating,
storing and broadcasting interactive panoramic walk-through
applications. The combination of images is determined by the user's
choice of direction of displacement at each intersection point and
from each view point or geographical coordinate, in order to
provide a complete view from a first person's point of view. The
system provides a visual perspective comparable to a human visual
experience.
Inventors: |
Lindemann; Pierre-Alain;
(Bex, CH) ; Lindemann; David; (Verossaz, CH)
; Crittin; Gerard; (Port-Valais, CH) |
Family ID: |
41666646 |
Appl. No.: |
13/127474 |
Filed: |
November 5, 2009 |
PCT Filed: |
November 5, 2009 |
PCT NO: |
PCT/IB2009/007335 |
371 Date: |
May 4, 2011 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
61111346 |
Nov 5, 2008 |
|
|
|
Current U.S.
Class: |
348/36 ; 345/419;
348/E5.024 |
Current CPC
Class: |
G01S 19/14 20130101;
G01S 19/49 20130101; G01C 11/02 20130101; G03B 37/06 20130101; G06F
16/29 20190101; G06T 3/0062 20130101 |
Class at
Publication: |
348/36 ; 345/419;
348/E05.024 |
International
Class: |
H04N 5/225 20060101
H04N005/225; G06T 15/00 20110101 G06T015/00 |
Claims
1. A method for creating panoramic images for use in walk-through
applications comprises the steps of: (a) capturing panoramic images
using a panoramic optic system (30, 30'); (b) unwarping images,
comprising the sub-steps of: (i) centering and aligning the image
(210) relative to a grid; (ii) opening up the images after slicing
a section from the image center to an edge to form an image (211)
having a circular arc shape; and (iii) unwarping the circular arc
shape image to form an unwarped image (212) of rectangular shape;
(c) vertical stretching (322) of the image proportional to the
divergence of the field of view to the horizon of the optical
system; and (d) expanding horizontal edges (332), comprising the
sub-steps of (i) cropping and copying a section of the image (222)
located on the left or right edge of the image; and (ii) pasting
said section on the opposite edge of the image to form an expanded
image (232), so that two identical sub-sections of the image appear
on opposite sides of the image.
2. The method of claim 1 wherein the vertical stretching of image
step (322) comprises the sub-steps of: (i) setting up a data buffer
corresponding to a destination image (222); (ii) using a function
curve to determine correspondence between coordinates in the source
image (212) and destination image (222); and (iii) copying pixel
values of source image (212) to pixel values at corresponding
coordinates on destination image (222) on a pixel per pixel basis,
using an image sampling method selected from a group consisting of
"median", "summed area", "bilinear" and "trilinear" image sampling
methods.
3. A method for computing a virtual 360 degree panoramic image from
a virtual 360 degree scene (240), for use in walk-through
applications, comprising the steps of: (a) rendering (253) the
scene in four sections (244), each having a 90.degree. horizontal
field of view, each section being oriented, from the nodal point
(241), at a view angle of 90.degree., from the preceding and
following sections, said sections being represented in a single
image (250) of rectangular shape; (b) vertically stretching (352)
the single image, section per section, along a sinusoidal curve;
and (c) removing (354) upper and lower edges sections to provide a
resulting image (254) of rectangular shape.
4. The method of claim 3, further comprising the additional step of
expanding (332), horizontal edges comprising the sub-steps of: (a)
cropping and copying a section of the image (254) located on the
horizontal edge of the image; and (b) pasting the section on the
opposite edge of the image to form an elongated image (232) which
then comprises the section on both of its opposite sides.
5. The method of claim 3 wherein the vertical stretching step (352)
comprises the sub-steps of: (i) setting up a data buffer
corresponding to a destination image (252); (ii) sinusoidal
stretching (352) using a function curve to compute the vertical
coordinates in the destination image (252) to find the
corresponding coordinates in the source image (250); and (iii)
copying pixel values of source image (250) to pixel values at
corresponding coordinates on destination image (252) on a pixel per
pixel basis, using an image sampling method selected from a group
consisting of "median", "summed area", "bilinear" and "trilinear"
image sampling methods.
6. The method of claim 2 wherein the pixel values comprise values
selected from a group consisting of "Hue, Lightness, Saturation
values", "CMYK values" and "pixel RGB values".
7. A method for computing a virtual 360 degree panoramic image
based on two points perspective, for use in assembling and
broadcasting of walkthrough applications, providing panoramic
images closer to human vision during panoramic motion, comprising
the steps of: (a) dividing the viewable portion (234) of a given
panoramic image (232), in vertical slices (235); (b) stretching
said vertical slices (235) along sinusoidal lines to recover a two
points perspective, wherein a group of vertical slices is in a
horizontal fixed position relative to a viewport; (c) providing
vertical panoramic interactive motion by vertical translation of
said vertical slices (235) without affecting sinusoidal distortion,
wherein said group of vertical slices are vertically compressed
proportional to the angular divergence of the field of view to the
horizon, in order to minimise vertical distortion of two points
perspective.
8. The method of claim 7, wherein said vertical stripes are of
non-uniform width.
9. The method of claim 7, wherein said steps are performed in real
time and broadcasted by a web server (82) to a user terminal, in
response to user motion in a walkthrough space.
10. The method of claim 1, wherein the image modification steps
further include the step of automatic blurring of portion of
images, such as faces or numerical car plates.
11. The method of claim 1, wherein image modification steps
processes images in two resolutions; low resolution for interactive
walkthrough panoramic motion, and high resolution for interactive
panoramic motion.
12. A system implementing the method of claim 1, comprising (a) an
image capture apparatus (20) for capturing panoramic images,
further comprising a camera (40), an optic system (30, 30')
connected to the camera, the optic system providing a substantially
360.degree. field of view, and a memory device (50); (b) a
communication device for transferring data from the memory device
to a computer (80); (d) instructions executable on a computer for
performing image modification steps; and (e) a database for storing
image related data.
13. The system of claim 12, wherein the image capture apparatus
further comprises a memory device to store data including images,
geographic coordinate information related to said images, and
distance information.
14. The system of claim 12, wherein the image capture system
comprises a location measurement device (GPS) and a distance
measurement device.
15. The system of claim 12, wherein the image capture system
comprises an inertial measurement unit for measuring rates of
acceleration and changes in rotational attributes of the vehicle,
fixation means or camera.
16. The system of claim 12, wherein the image related data includes
the image, date and time of image capture, geographical coordinate
information, project or group related information, so as to
identify for example a particular district or street, and camera
settings such as aperture, speed and ISO.
17. The system of claim 1, wherein the computer which executes
instructions for performing image modification steps, processes
images in two resolutions; low resolution for interactive
walkthrough panoramic motion, and high resolution for interactive
panoramic motion.
18. The method of claim 5 wherein the pixel values comprise values
selected from a group consisting of "Hue, Lightness, Saturation
values", "CMYK values" and "pixel RGB values".
19. A system implementing the method of claim 2, comprising (a) an
image capture apparatus (20) for capturing panoramic images,
further comprising a camera (40), an optic system (30, 30')
connected to the camera, the optic system providing a substantially
360.degree. field of view, and a memory device (50); (b) a
communication device for transferring data from the memory device
to a computer (80); (d) instructions executable on a computer for
performing image modification steps; and (e) a database for storing
image related data.
20. A system implementing the method of claim 3, comprising (a) an
image capture apparatus (20) for capturing panoramic images,
further comprising a camera (40), an optic system (30, 30')
connected to the camera, the optic system providing a substantially
360.degree. field of view, and a memory device (50); (b) a
communication device for transferring data from the memory device
to a computer (80); (d) instructions executable on a computer for
performing image modification steps; and (e) a database for storing
image related data.
Description
[0001] This application claims the benefit of U.S. Provisional
Application No. 61/111,346, entitled SYSTEM AND METHOD FOR CREATING
AND BROADCASTING INTERACTIVE PANORAMIC WALK-THROUGH APPLICATIONS,
filed Nov. 5, 2008.
FIELD OF THE INVENTION
[0002] The present invention relates generally to virtual tours.
More specifically, the present invention relates to virtual
walk-through applications using panoramic images, 3D images or a
combination of both.
BACKGROUND OF THE INVENTION
[0003] A virtual tour (or virtual reality tour) is a virtual
reality simulation of an existing location, which is usually built
using contents consisting principally of 2D panoramic images,
sequence of linked still images or video sequences, and/or
image-based rendering (IBR) consisting of image-based models of
existing physical locations, as well as other multimedia content
such as sound effects, music, narration, and text. A virtual tour
is accessed on a personal computer (typically connected to the
Internet) or a mobile terminal. Although not replacing real travel,
virtual tours aim at evoking an experience of moving through the
represented space. Virtual tours can be especially useful for
universities and in the real estate industry, looking to attract
prospective students and tenants/buyers, respectively, eliminating
for the consumer the cost of travel to numerous individual
locations.
[0004] The word panorama indicates an unbroken view, so
essentially, a panorama in that respect could be either a series of
photographic images or panning video footage. However, the terms
`panoramic tour` and `virtual tour` are generally associated with
virtual tours created using stills cameras. Such virtual tours
created with still cameras are made up of a number of images taken
from a single view point. The camera and lens are rotated around
what is referred to as a nodal point (the exact point at the back
of the lens where the light converges). These images are stitched
together using specialist software to create a panorama
representing a near 360 degree viewing angle, as viewed from a
single "view point"; the panoramas are each resized and configured
for optimal on-line use. Some `panographers` will then add
navigation features, such as hotspots (allowing the user to "jump"
from one viewpoint or panorama to the next) and integrate
geographic information such as plans or maps.
[0005] Current virtual tours photographic techniques suffer from
several limitations. A seamless panoramic image can not be created
from still images whenever such still images are captured from
different nodal points or, for two consecutive images, from a
single nodal point but with different focal lengths or focus
distances. Images captured from a single camera rotating on its
nodal point can be stitched seamlessly but this solution can not be
used for applications involving axial translation, where, for
example, images are captured from a vehicle in motion.
[0006] Catadioptric optical systems provide images having a
360.degree. horizontal field of view and near 180.degree. vertical
field of view. The resulting panoramic images are of a annular
shape and generally must be sliced and open and "unwarped" to
create a panoramic image of a rectangular shape. The unwarping step
causes image distortion which, together with the optical
distortions caused by the Catadioptric optics having unevenly
distributed angles along its radial axis (vertical axis of the
view), must be compensated by specialised application software.
[0007] Patent document US 2007/0211955 to Pan discloses a
perspective correction method allowing e-panning without image
distortion, wherein image correction step is performed on image
slices (horizontal section of the wide-angle image) by
repositioning each pixel to a corresponding point on a cylindrical
surface. This method consumes significant processing power and
bandwidth for respectively correcting and transmitting the images
whenever fast user motion is involved during navigation, and is
therefore not optimal for providing a seamless navigation
experience at relatively high user directed panning speed.
[0008] Also, with current image capture solutions, objects near the
camera are responsible for occlusion on distant objects;
"occlusion" meaning, with regard to 2D images, the non-projection
of a surface to a point of observation, and with regard to a 3-D
space, the effect of one object blocking another object from view.
Limitations of current virtual tour technology, such as object
occlusion, have had the detrimental result on virtual tours never
materializing outside of the real estate industry.
[0009] Virtual Walk-Through ("VWT") applications constitute an
evolution over virtual tours. This technology eliminates the
occlusion limitation by enabling user to travel to a point where
distant objects are no longer occluded.
[0010] Commercial online walk-through products such as Google
"Street View" provide virtual outdoor walk-through of cities using
images captured from a camera mounted on a road vehicle which
circulates on motorways and roads at speeds ranging from 30 kmh to
80 kmh. These products are limited to outdoor views wherein any two
consecutive points of view are positioned at a relatively long
distance from each other. Typically, StreetView and applications of
the like provide visualisation at road-view level, that is,
visiting a city as viewed from a car, wherein the user follows a
pathway formed by a plurality of panoramas accessible along main
streets, the user following the pathway by "jumping" (generally
from a graphical interface allowing clicking of on screen icons)
from a panorama or point of view to the next distant point of view.
This application uses a series of standard photographic images,
taken by multiple cameras systems mounted to produce images
representative of multiple view angles; panoramic images are
produced by computation (stitching) of still images from the
multiple cameras. Such panoramic images can not provide accurate
representation of geometric objects, of for example buildings, due
to the inherent discontinuity (break) of such panoramic images,
this discontinuity is due to the physical impossibility of
superposing a single nodal point from multiple cameras and view
angles. Furthermore, "STREET VIEW" products and the like provide
images which suffer from trapezoidal distortion whenever the view
angle is not pointing toward the horizon; this distortion is due to
the perspective. Although geometrically correct, "STREET VIEW"'s
images do not reflect human vision behaviours, which keep vertical
lines mostly parallel whenever a viewer tilts its view gently above
or below the horizon.
[0011] Google "STREET VIEW" also creates ground plane distortion
where planar ground seems to be inclined due to the unwanted motion
of the cameras caused by the inertial force. Other current
walk-through products, such as "EVERYSCAPE" (www.everyscape.com by
Everyscape, Waltham, Mass.) and "EARTHMINE" (www.earthmine.com by
Earthmine inc., Berkeley, Calif.) also produce trapezoidal
distortion, which makes them unfit for applications requiring
continuous undistorted images (i.e. images which more closely
correspond to human vision), as for example, for virtual
shopping.
[0012] The trapezoidal distortion drawback is inherent also to
virtual walk-through applications based on 3D virtual images, which
can be used for example for visiting a virtual building using
real-time 3D engine, such as Second Life (www.secondlife.com by
Linden Research Inc, San Francisco, Calif.) or video games.
[0013] "EARTHMINE" provides commercial online walk-through for
applications such as management of buildings and assets, telemetric
measurement and other cadastral works. The product combines high
resolution still images and 3D mesh information to provide pathways
wherein user jumps from one distant view point to another.
[0014] "EVERYSCAPE" provides commercial online panoramic products
wherein motion between two consecutive view points is simulated by
video postproduction effects. This product does not allow user to
pan and tilt its viewing angle during its displacement along the
travel path. During the travel motion effect, images are no longer
panoramic unless the fields of view of images representative of the
next fixed point, are constrained to the motion axis.
[0015] In sum, current virtual walk-through applications and
systems suffer several important limitations. Travel along pathways
is achieved by jumping from one view point to another, instead of
by a fluid travel motion. Views suffer from high occlusion rate, as
lots of objects are never visible at all along pathways. Generally,
images are not standard panoramic images but rather patchy assembly
of 2D images where many discontinuities are found on each
image.
[0016] The prior art describes several techniques whose purpose is
reducing the bandwidth associated with the transmission of
panoramic images and applications between a server and the user
remote terminal, allowing a user to navigate a walkthrough space
while downloading data.
[0017] The use of a predefined pathway has been widely adopted to
prevent storage and transmission of redundant image data.
Predefined pathways have the additional benefit of simplifying user
navigation, notably by preventing the user from searching available
paths or from hitting objects repetitively during motion, as would
be the case when user tries to walk through walls or door
images.
[0018] U.S. Pat. Nos. 6,388,688 and 6,580,441, both to
Schileru-Rey, disclose a computer system and method that allow
interactive navigation and exploration of spatial environments,
wherein pathways are represented by branches, and intersections in
the real environment are represented by nodes. User selects which
path to follow from the node. A branch represents a video sequence
or animation played during motion between two adjacent view points.
Virtual objects can be integrated to specific branches or nodes,
without assigning a geographic coordinate to virtual objects; each
object is linked to at least one branch or node and displayed when
user is travelling on said branch or node.
[0019] U.S. Pat. No. 7,103,232 to Kotake discloses an IBR system
with improved broadcasting performance, where a panoramic image is
created by stitching several images from several cameras,
preferably video cameras, pointing to distinct points of view, the
cameras being synchronised by use of a single time code. The '232
system provides panoramic images divided in six images sections of
60.degree. horizontal field of view, and broadcast typically only
two of the six sections' image (providing a 120.degree. field of
view) at any giving time with an aim to reduce processing power and
communication bandwidth. The '232 solution is not optimized however
for walk through applications allowing fast movement across the
horizontal plane beyond 120.degree.; moreover, the '232 patent does
not disclose broadcasting images of different image resolution,
meaning that it only covers broadcasting of image of the highest
possible image resolution.
[0020] U.S. Pat. No. 6,633,317 to Jiang Li discloses a data
transfer scheme, dubbed spatial video streaming, allowing a client
to selectively retrieve image segments associated with the viewer's
current viewpoint and viewing direction, rather than transmitting
the image data in the typical frame by frame manner. The method of
patent 317' divides walkthrough space in a grid; each cell of the
grid is assigned to at least one image of the surrounding scene as
viewed from that cell; images are characterised similar to a
concentric mosaic in that each cells is represented by a sequence
of image columns. Method of patent 317' allows transmission of part
of the images (compressed or not) needed in an attempt to
anticipate viewer's change of view point within the walkthrough
space, starting with image data corresponding to viewpoints
immediately adjacent the current viewpoint, with subsequent image
data associated with viewpoints radiating progressively out from
the current viewpoint. Patent 317' is well suited to open
walkthrough spaces where user can move in any direction using
multiple source of image data (simple 2D image, panoramic image or
concentric mosaic), such as in a typical 3D environment. However,
this method is not suited to the optimal transmission of full
panoramic images in situations where user travels along predefined
pathways consisting of several view points in a linear arrangement
within a network of pathways within the walkthrough space.
Additionally, being view direction sensitive, this method is not
optimized, in terms of response time, to allow user to change his
travel plan, for example, by making a U-turn or to travel along
another pathway. Finally as this method allows travel in any
direction (along a predefined pathway) the amount of data download
to represent a given view point is greater and therefore less
suited for fast and responsive viewing experience on the Internet
or other network media having limited bandwidth.
[0021] Consequently, no system of the prior art provides a system
optimized for seamless broadcasting of fluid motion where the user
can orientate (pan and tilt) the field of view during motion and
where the user can stop the motion anywhere along the travel path,
in order to discover local objects in detail without occlusion.
[0022] Integration of virtual objects (such as images, icons, etc)
to panoramas has been limited to the integration of two dimensional
objects in specific view point images wherein said objects are not
visible from distant view points.
[0023] U.S. Pat. No. 6,693,649 to Lipscomb discloses a solution for
non-linear mapping between media and display allowing "hotspots",
defined as an outline of two points connected by straight lines, to
be used in the context of panoramas. Such "hotspots" are referenced
to each image using two angle's or two pixel's coordinates; such
values are only valid for each distinct image.
[0024] Allocation of a third dimension value for virtual objects
and determination of precise geographic location information for
each view point are prerequisite for the seamless integration of
hotspots and other virtual objects in panoramas, where said object
would be visible from any point having a direct line of sight to
the object. Consequently, no prior art system provides a system and
advanced features based on geographical information such as the
ability to pin an element of information on any location in a view,
such element staying spatially fixed to the point during the
travel. The present method and system for creating and assembling
interactive walkthrough applications overcomes these shortcomings
as will now be described.
[0025] Given the market need for immersive online applications,
what is needed therefore is a system and method for providing
seamless, quality, fluid walk-through navigation using any
combination of 2D panoramic images, virtual 3D, and optionally,
virtual objects or any images or a combination thereof.
[0026] What is needed is an optimized system or method for the
real-time construction and broadcasting of panoramic walkthrough
applications, which allow the user, from each view point or
geographical coordinate along a network of pathway, to have a
complete view from a first person point of view, the view covering
substantially 360.degree. in field of view. What is needed is such
a system or method that combines high rate fluid panoramic imaging
broadcasting and the possibility of seamlessly providing higher
quality images in which the visual perspective perception based on
human vision is preserved.
[0027] What is needed is a system or method for creating and
broadcasting interactive panoramic walk-through applications that
can combine still panoramic images that can be captured from either
indoor or outdoor locations, and virtual scenes.
[0028] Finally, what is needed is a system or method for creating
and broadcasting interactive panoramic walk-through applications
that can make genuinely interactive functions available and
accessible in the images.
SUMMARY OF THE INVENTION
[0029] A system, apparatus and method for creating interactive
panoramic walk-through applications having a 2D image acquisition
system is provided. The system includes, comprising a holding
device such as a vehicle equipped with a camera connected to a
catadioptric optical system providing near 360.degree. field of
view, a memory device adapted to store data including images and
geographic coordinate information related to these images,
communication device for transferring data from the memory device
to a computer, a fixation and stabilisation system connecting the
camera to the holding means for aligning the camera perpendicular
to the horizontal plane; a processor and associated software for
performing image modification steps; and optionally, 3D virtual
images.
[0030] The image capture system includes a location measurement
device (GPS), a distance measurement device (odometer) and an
inertial measurement unit (IMU) for measuring rate of acceleration
and changes in rotational attributes (attitude) of the vehicle, the
fixation device or camera.
[0031] The stored data includes the image, date and time of image
capture, geographical coordinate information, and other
information, notably image reference and image group, so as to
identify, for example, a particular district or street, as well as
camera settings such as aperture, and image-related information
such as camera model, speed and ISO reference.
[0032] The image modification steps include the steps for providing
panoramic images based on two points perspective, of unwarping
images, vertical stretching of image proportional to the divergence
of the field of view to the horizon of the optical system, and
expanding horizontal edges. The image modification steps optionally
include the step of automatic blurring of portion of images, such
as faces or numerical car plates.
[0033] Software operates a computer to perform image modification
steps such as processes images in two resolutions, where low
resolution is used for interactive walkthrough panoramic motion,
and high resolution for interactive panoramic motion.
[0034] According to another preferred embodiment of the present
invention, preservation of visual perspective based on human vision
is provided, notably by use of two points perspective that does not
produce the trapezoidal distortion that is inherent with standard
360.degree. environment interactive applications.
[0035] It is an object of the invention to provide a system and
method for providing immersive, interactive and intuitive
walk-through applications using 2D panoramic true images, virtual
3D images or a combination of both, that can provide seamless
quality walk-through navigation and high quality imaging.
[0036] It is another object of the present invention to provide a
system or method that combines high rate panoramic imaging
broadcasting and the possibility of seamlessly providing higher
quality images in which visual perspective based on human vision is
preserved.
[0037] It is another object of the present invention to provide a
system or method for creating and broadcasting interactive
panoramic walk-through applications that can combine indoor and
outdoor images, based on 2D panoramic images and virtual 3D
images.
[0038] It is another object of the present invention to provide a
system or method for creating and broadcasting interactive
panoramic walk-through applications that can provide genuinely
interactive functions accessible in the images.
BRIEF DESCRIPTION OF THE DRAWINGS
[0039] FIG. 1 is a schematic side view of a catadioptric
(mirror-based) panoramic optical system.
[0040] FIG. 2 is a schematic view of a panoramic image made using
the catadioptric based panoramic optical system of FIG. 1.
[0041] FIG. 3 is a schematic side view of a lens based panoramic
optical system.
[0042] FIG. 4 is a schematic view of a panoramic image made using
the lens based panoramic optical system of FIG. 3.
[0043] FIG. 5 is a flow chart showing the unwarping modification
steps.
[0044] FIG. 6A-6C are schematic views of a panoramic image, made
from either systems of FIG. 1 and FIG. 2, following each of the
unwarping modification steps of FIG. 5.
[0045] FIG. 7 is a schematic view of a panoramic image before and
after modification steps to compensate for vertical distortion.
[0046] FIG. 8 is a side view showing the vehicle, image capture
system, measurement means and attached odometer over a schematic
ruler representing distance of vehicle' travel along a road.
[0047] FIG. 9 is a schematic view of the data storage apparatus of
the present invention.
[0048] FIG. 10 is a floor plan view showing the division of a
panoramic virtual 3D image into four sections to be compatible with
standard field of view (<179.9.about..degree.) in standard 3D
applications.
[0049] FIG. 11 is a schematic view showing four images resulting
from the rendering of a 360.degree. field of view in a standard 3D
application by the rendering of section step, providing the
division of a 360.degree. panoramic field of view (FOV) on four 3D
images of 90.degree. of FOV of FIG. 10.
[0050] FIG. 12 is a schematic view showing the steps of assembly
and panoramic distortion over the images of FIG. 11, wherein image
250 shows an expansion of horizontal edges, image 252 shows a
distortion of image to compensate for standard perspective where
image are projected on a plane, and image 254 shows a cut of image
bulb, discontinuous section of the image, at top and bottom of the
image.
[0051] FIG. 13 is a schematic view of a panoramic image of FIG. 7
after modification steps to expand horizontal edges. Same step is
typically applied to image 254 of FIG. 12.
[0052] FIG. 14 is a flow chart showing the two points perspective
distortion steps.
[0053] FIG. 15 is a schematic view showing the steps of modifying a
panoramic image in order to provide a resulting image based on a
two points perspective similar to human vision.
DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS
Image Capture System
[0054] Referring now to FIGS. 1-4 and 8-9, an image capture system
20 consists of a panoramic optic 30, 30', a camera 40 and a memory
device 50 such as a computer, mounted on a vehicle 70 or other
portable holding device.
[0055] The panoramic optic 30, 30' is a physical panoramic optic
providing 2D panoramic images. The optic 30, 30' including either,
a "lens and mirror" based optic system (catadioptric system) 32,
38, 42 as shown in FIG. 1, or a physical optical panoramic system
33 (consisting of an ultra wide angle lens or fisheye system with
lens providing more than 200.degree. of continuous vertical field
of view), without mirror, as shown in FIG. 3. Both systems 30, 30'
are commercially available and reflect the substantially 360 degree
panoramic field of view into the lens based optics connected to
camera 40.
[0056] The mirror shape and lens used is specifically chosen and
disposed such that the effective camera 40 maintains a single
viewpoint. Such a lens is available from Nikon, Canon, and other
vendors for example Bellissimo Inc, Carlson Nev. (www.0-360.com).
The single viewpoint means the complete panorama is effectively
imaged or viewed from a single point in space. Thus, one can simply
warp the acquired image into a cylindrical or spherical
panorama.
[0057] Optionally in the catadioptric system, the image capture
system can use a panoramic lens optic (not shown) commercially
available or specifically designed to be adapted to the image
capture system 20 of the present invention. This panoramic lens
optic is composed of a lens assembly that distorts the field of
view (FOV), wherein the FOV is expanded (on the geometric opposite)
to cover an additional 90.degree. in all directions, from the
camera's original FOV, providing in total a FOV of at least
180.degree. of vertical FOV, and ideally at least 240.degree. of
vertical FOV. Such a panoramic lens allows more vertical field of
view than a catadioptric system, but accentuates chromatic
aberration and is more vulnerable to dust, drops and lens flare
artefacts.
[0058] The most significant advantage of catadioptric systems 32
and physical optical panoramic systems 33 (mono camera system) over
multi-camera image capture systems, is to provide image almost free
of chromatic aberrations or discontinuities (breaks). Moreover,
since a complete panorama is obtained on each image shot, dynamic
scenes can be captured.
[0059] A first advantage of a physical optical panoramic and
catadioptric system over multiple camera systems is that the former
avoids the need to stitch multiple images to create a full
panoramic image and image color and exposure are consistent inside
one point of view over the 360.degree. range. A second advantage is
that the geometric nodal point does not need to be simulated, as is
the case with stitched motion images. Moreover, with a physical
optical panoramic or catadioptric system, the accuracy of an
object's geometry in the image is not relative to it's distance
from the camera. In a multiple camera system where the nodal point
is simulated by software techniques, objects located proximate to
the camera are discontinuous and produce ghosts images and
artefacts over the resulting panoramic image.
[0060] The camera 40 used in the system 30, 30' can be any kind of
imaging device (e.g., a conventional camera with chemical
exposition film, video camera, etc), but is typically a high
resolution digital camera, having CCD or CMOS captors of typically
12 Megapixel of resolution or more, with controllable aperture and
fast response time of typically 4.5 images/second or more.
[0061] Fast response time is required to obtain still images during
image acquisition while the vehicle 70 is in motion. Speed of
displacement of the vehicle varies; it is typically of 10 km/h and
2.5 km/h for respectively outdoor and indoor image acquisition
applications; this provides for a resolution of three to four or
more images per meter (at 2.5 km/h) for indoor applications to one
image per meter for outdoor applications (at 10 Km/h). Typical
speed and number of image per meter disclosed in this document are
provided by example and do not constitute a limitation to the
applicable field of the 1.5 present invention. Images can be
captured at higher vehicle velocity; during which, satisfactorily
images can be captured using a camera with a better sensitivity or
a lower image resolution or at a lower capture rate allowing fewer
view points along a pathway. Identical or a higher number of view
points may be captured using a faster capture device. Higher or
lower density (images per meter) may be achieved based on
requirements of the specific application field, and on hardware
evolution.
[0062] For a low light environment, image bracketing and high
dynamic range (HDR) image processing is used to enhance the dynamic
spectrum of the final image. Image bracketing and HDR is a
technique used to mix multiple images of the same viewpoint, each
image having different exposure time. Image bracketing and HDR
require the immobility of the vehicle during image acquisition, the
drawback being a slower image capture process.
[0063] The digital camera 40 is coupled to a catadioptric optic
system 32 by an optic apparatus 42 such as is commercially
available from manufacturers such as Nikon and Cannon, then by a
standard connector 38 provided by the catadioptric lens
manufacturer for each proprietary optic mounting format. The
digital camera 40 could also be coupled with the panoramic lens 33
by an optic apparatus 43 that is commercially available from the
above-mentioned manufacturers
[0064] The memory device, such as computer 50, receives and stores
the images transferred from the camera 40, together with other
information received from measurement device 60 such as geographic
coordinates (including altitude) related to the images as well as
geometric orientation, acceleration, rate of rotation on all three
axes (attitude) and travel distance information of the capture
vehicle and/or the measurement device. Memory device 50 is
typically a computer installed with an operating system,
proprietary software and a logical device with multiple processing
cores and/or CPU arrangement. The proprietary software manages the
distribution of the load of image acquisition to multiple threads
on multiple CPUs, processing cores or logical processing units. The
distribution of load is achieved by attributing the processing work
of each subsequent image to another CPU core or logical processing
unit. In a computer 50 having four logical processing units, a
typical image capture sequence according to the present invention
would be processed in sequential order as follows: (i) first
image's acquisition processing work is performed on "logical
processing unit 1", (ii) second image's acquisition processing work
is performed on "logical processing unit 2", (iii) third image's
acquisition processing work is performed on "logical processing
unit 3", (iv) fourth image's acquisition processing work is
performed on "logical processing unit 4", (v) fifth image
acquisition processing work is performed on "logical processing
unit 1", (vi) sixth image acquisition processing work is performed
on "logical processing unit 2". The distribution of acquisition
processing across multiple CPUs increases the rate of image
acquisition in a given time, resulting in better image acquisition
performance and accrued image acquisition reliability.
[0065] Images are distributed from memory device 50 to multiple
storage devices 52 to achieve the high data bandwidth required by
the in-motion image capture method of the present invention. Memory
device 50 and multiple storage devices 52 are located onboard
vehicle 70 or remote thereto. A communication device 54 allows
transfer of data from the memory device 50 to a central computer
80. Data is stored in a source database 400 on the central computer
80, wherein each image has a unique identification. Each unique
image ID is associated with a specific time reference being the
database 400, the time reference is the time when the image is
captured. Because the time reference needs high precision, the time
reference is given as universal time reference, provided by the GPS
unit, which is typically more precise than the internal computer
clock. With the time reference, the image capture location can
easily be retrieved.
Data Measurement Means
[0066] The measurement device 60 mounted on the vehicle 70
comprises a GPS tracking device 62 or a similar device able to
determine geographic coordinate information from a satellite
signal, radio signal or otherwise. Each image is recorded in the
memory device 50 or on a central computer 80 with the associated
geographic coordinate information, namely of the location of image
capture, which is stored either on a dedicated recording device or
on the memory device 50 such as an on-board computer or on a remote
central computer 80. Data is transferred using a communication
protocol such as USB, Bluetooth, Ethernet, WiFi, and stored on the
destination apparatus in any standard database format.
[0067] Geographic coordinates, also referred herein as "GPS data"
162, are stored with specific GPS universal time reference to
images, allowing the determination of the exact geographic location
at which each image was taken. Memory device 50 is synchronised to
the GPS clock to store universal time reference with any stored
data. A system and method for the integration of virtual objects in
an interactive panoramic walk through application, using a method
for determining geographic coordinates with increased precision is
described in PCT application entitled SYSTEM AND METHOD FOR THE
PRECISE INTEGRATION OF VIRTUAL OBJECTS TO INTERACTIVE PANORAMIC
WALK-THROUGH APPLICATIONS, by Lindemann et al. which is being
concurrently filed with the instant application and is incorporated
by reference hereto.
[0068] Because GPS devices have limited precision in altitude
tracking, other devices such as an altimeter or any altitude
tracking device can be use in adjunction to GPS device to enhance
the precision of altitude tracking of images. Further, because GPS
devices have limited precision in direction tracking, direction can
be obtained from an electronic compass or other direction tracking
device, thereby enhancing the precision of the recorded path of
image.
[0069] An odometer 66 is connected to vehicle 70 for indicating
distance traveled between any two image locations, thus improving
the precision of the geographic coordinates associated with each
image. Odometer 66 may be an electronic or mechanical device. An
Inertial Measurement Unit--IMU device 67 on board vehicle 70
detects the current rate of acceleration of the vehicle as well as
changes in rotational attributes (attitude), including pitch, roll
and yaw. Such data may be used to correct image inconsistencies
caused by travel over uneven surfaces. It should be noted that the
vehicle's acceleration or speed does not affect the capture
density, number of image along a pathway, because successive images
are automatically triggered as a function of the distance value
between any two successive images, as provided, for example, by GPS
data, odometer data 166 and/or IMU data 167.
Vehicle
[0070] The host vehicle 70, for image capture of outdoor locations,
can be any vehicle adapted for circulation on roads, namely cars,
trucks, or any vehicle adapted to limited circulation areas or
indoor circulation such as golf carts, electric vehicles and
mobility scooters (scooters for the disabled), etc. FIG. 8 shows
vehicle 70 as a small car. For typically smaller, steep roads as
well as image capture of indoor locations, remote controlled
vehicles, unmanned vehicles, robots, and stair-climbing robots, can
be used as the host vehicle 70 for the image capture system 20. In
a miniaturized version, the Image capture system 20 can also be
carried by a human and for some special applications, small animals
such as rats. Where the terrain is difficult, a flying machine can
be used, in which case an odometer is emulated by the use of GPS
data or/and triangulation using radio signals. Triangulation
techniques using radio signals require that two or more emitters be
located at known positions.
Logical Means for Performing Image Modification Steps
[0071] According to a preferred embodiment of the present
invention, source images 210 (2D panoramic images) from the image
capture system 20 are modified using a computer with a logical
device such as central computer 80. Images modification steps
comprise the steps of unwarping 312, compensation of vertical
distortion 322, expansion of horizontal edges 332 and the
resolution of two point perspective distortion 342, in order to
obtain release images 280 that can be broadcast by web server
82.
[0072] As shown in FIGS. 1, 3, 5 and 6A-6C, source images 210
obtained by a panoramic optic 30, 30' are typically circular in
shape, referred in the art as annular images. Unwarping 312 of
source images 210 is achieved using conventional software
techniques, so as to form cylindrical images. The unwarping
operation is typically performed in three consecutive operations:
in a first operation 313, the image is centered and aligned
relative to a grid consisting typically of geographic direction
(North-East-South-West axis) or corresponding degrees of the 360
degrees panorama, where the North axis direction is referred to as
0 degree for convenience; in a second operation 314, the circular
shaped image is opened up after "slicing" a section from the image
center to an edge, typically the bottom point of the image on the
"South" or 180 degree direction the resulting image 211 is in the
shape of a circular arc; and in third operation 315, the circular
arc is further unwarped to form an unwarped image 212 of
rectangular shape. For ease of understanding, images of FIGS.
6A-6C, 7 and 13 show section marks on the vertical axis where the
"A" mark indicates the South or the 180 degree direction, "B"
indicates the East or the 90 degree direction, "C" indicates the
North or 0 degree direction, and "D" indicates West or the 270
degree direction.
[0073] As shown in FIGS. 2 and 4, the field of view 34 of
catadioptric optic is typically un-evenly distributed along the
vertical axis of the mirror, causing optical deformations, where
the image 212 appears compressed near the upper and lower edges of
the image. To correct this vertical unevenness, compensation of
vertical distortion 322 is performed as shown in FIG. 7, and
involves modifying each unwarped image 212 by software operation to
compensate for the compression of the upper and lower edges of the
images, to provide a resulting image 222 that is typically larger
along the vertical axis, compared to the original unwarped image
212.
[0074] Compensation of vertical distortion step 322 is performed by
applying a function curve to affect pixel distribution along the
vertical axis. In a first sub-step, an empty data buffer
corresponding to a destination image 222 is set-up, resulting in a
blank image. In a second sub-step, correspondence between
coordinates in the source image 212 and destination image 222 is
determined using a function curve. In a third sub-step, pixel
values (color, hue, intensity, CMYK or pixel RGB values) at given
coordinates of source image 212 are copied to pixel values at
corresponding coordinates on destination image 222 on a pixel per
pixel basis. On the third sub-step, at least one image sampling
method such as "median", "summed area", "bilinear" or "trilinear"
known image sampling methods is applied to obtain sub pixel data
whenever a pixel's coordinates, size and shape on the original
image 212 do not match resulting pixel's coordinates, size and
shape on the destination image 222. One resulting pixel can match a
source pixel exactly or can be a variable portion of one or several
source pixels), thereby allowing the destination image to have
different vertical resolution (number of pixels) as compared to
source image 212.
[0075] The function curve is determined by either measuring the
curvature of different optic elements of the panoramic optic 30,
typically the main lens or mirror, or by measuring the vertical
distortion produced by said optic elements. Vertical distortion is
measured by acquiring the image of a calibration model (a 2D image)
at a given distance with the image panoramic optic 30 and camera 40
in a calibration room having visible demarkations (for example, a
geometrically precise grid painted on a room wall). This helps
determine distance discrepancies between the image and the
calibration model. In other words, on the destination image 222, a
measurement unit (for example 1 vertical meter) at a given distance
(for illustration purpose a measurement unit of 1 vertical meter at
the distance between a tree, 213 on both images 212 and 222, and
the position of the capture optic 30) fills the same number of
pixel near the horizon and near the top or bottom of the image.
This measurement unit which fills a number of pixels N near the
horizon of the image, will fill the same number of pixels N at the
top or bottom of the image. On the unwarped image 212, same
measurement unit at same distance will fill fewer pixels near the
top and bottom of image compared as to the horizon (image vertical
middle).
Use of Virtual 3D Images Instead of Real Camera Image.
[0076] The system 20 of the present invention also allows the use
of 3D virtual images, alone or in combination with 2D (optical)
panoramic images, for the purpose of creating and broadcasting
interactive panoramic walk-through applications. Current
commercially available 3D rendering software engines used in CAD
("Computer aided-design"), 3D modeling and gaming applications are
not meant to provide panoramic images having near 360 degree
horizontal field of view. As shown in FIGS. 10 to 15, the
computation of a virtual 360 degree panoramic image is thus
typically achieved by transforming a 3D scene 240, originating from
a 3D modeling, for example, into a panoramic 254 image equivalent
to the resulting image 222. Because standard 3D software is not
meant to render image that have an horizontal field of view larger
than .about.179.9.degree., the computation of a virtual 360 degree
panoramic image is thus achieved by dividing the 360.degree. field
of view in 4 sections of 90.degree. fields of view, which in FIG.
10 is represented by numbers 1, 2, 3 and 4.
[0077] According to a preferred embodiment of the present invention
purpose is to increase image broadcasting speed and overall system
performance, computing a virtual 360 degree panoramic image is
performed, for each 3D scene 240, in a first step 253 by rendering
four images, called sections 244 (numbered 1 to 4 on each of FIGS.
10-12 and 15), having a 90.degree. horizontal field of view from a
single nodal point 241 and each section 244 pointing toward one of
the four right angle directions (front, left, back and right). From
the nodal point 241 each section 244 is thus oriented, at a view
angle of 90.degree., from the preceding and following sections. The
combination of the four original sections 244 can be represented by
a single rectangular shape image 250. In 3D space, the combination
of the four 90.degree. view sections can be used to define the
inner faces of a box; consequently, image 250 represents the
juxtaposition over the same plane of the four inner faces of a box
viewed from a central position inside this box.
[0078] As disclosed by US 2007/0211.955 to Pan with respect to
panoramic imaging, and as is known in 3D imaging, the projection of
wide angle or panoramic images over a cylindrical field of view
reduces the number of computing steps necessary to provide
real-time compensation to image distortion, and thus eases the
creation of interactive panoramic motion. This is due to the fact
that, in this kind of image, each image section has the same
proportion ratio independently of the angle of viewing angle.
[0079] In a second step, according to a preferred embodiment of the
present invention, as shown in FIG. 12 image 250 is modified as if
image 250 was then projected over a continuous surface of
cylindrical shape. This image modification is done by stretching
vertically the resulting image 250, section per section, along a
sinusoidal curve using software which affects the sinusoidal
stretching logical step 352, and resulting in modified image 252.
This sinusoidal stretching logical step 352 is described in detail
below.
[0080] In a first sub-step an empty data buffer corresponding to
the destination image 252 is setup (resulting in a blank
image).
[0081] In a second sub-step, the sinusoidal stretching logical step
352 is performed by evaluating resulting images pixels using a
sinusoidal function curve that points on the original image pixels.
This function curve computes the vertical coordinates in the
destination image 252 to determine the corresponding coordinates in
the source image 250. This sinusoidal function curve decreases its
amplitude for vertical coordinates that are farther from the top
edge of the section, the amplitude reaching a null value (0) at the
vertical center of the section, then the amplitude continues to
decrease as negative number and reaches the exact opposite value at
the bottom edge of the section. This sinusoidal function curve is
calculated from the start of an image section 244 to the end of the
same section. The same function curve is used horizontally for each
section, which results in an image plane 252 that contains 4
sections stretched in the same way regarding to the section local
coordinate system.
[0082] In a third sub-step, pixel values at a given coordinate of
source image 250 are copied according to pixel color value at
corresponding coordinates (given by said sinusoidal function curve
of the second sub step) on the destination image 252, on a pixel
per pixel basis. During the third step, at least one image sampling
method such as a, "median", "summed area", "bilinear" or
"trilinear" known image sampling method is applied to obtain sub
pixel data, whenever the pixel's coordinates, size and shape, on
the source image 250 do not match the resulting pixel's
coordinates, size and shape on the destination image 252 (one
resulting pixel can match a source pixel exactly or can be a
variable portion of one or many source pixels), allowing the
destination image 252 to have different vertical resolution (number
of pixels per mm) compared to source image 250. Pixel values can be
either color, hue, intensity values, CMYK (Cyan, Magenta, Yellow,
Key) values or pixel RGB (Red, Green, Blue) values.
[0083] The sinusoidal stretching step 352 is performed to
compensate the distance difference (i) between the view point and
the pixels laid over a flat surface (the face of a box discussed
above) and (ii) between the view point and same pixel laid over a
cylindrical surface (the inner surface of a cylinder as viewed from
within). In geometric perspective, the area, filled with the
projection of an object, grows in the inverse proportional manner
as the distance between said object and the view point. The center
of an edge of a flat surface (any face of the box) is located
closer to the viewpoint, compared to the center of an edge of an
equivalent cylindrical surface (as if the box would be contained
within the arc of a cylinder). In the geometric projection of a box
to a cylinder, the upper and lower edges of each face of a box
projected from the common center of the box and the cylinder become
an arc that follow a portion of a sinus curve. Any pixel of the
image is then positioned in the resulting image 252 relative to its
vertical distance between said pixel and top and bottom edge of the
box, at the same relative distance to the top and bottom edge (arc)
of the cylindrical projected box. Top and bottom edges of the
projected faces of the box to a cylinder are comparable to top and
bottom edges in the modified image 252.
[0084] In a third step, a logical step 354 is performed which
removes the circular area on the upper and lower edges of each
image's section of image 252, so that ultimately a single
rectangular shape image 254 is created.
[0085] In order to increase image broadcasting speed and overall
system performance, each image 222 (at this stage, image 254 and
image 222 can be processed by the system in the same manner), which
is either a panoramic image from the image capture system 30 or an
image resulting of the modification of a 3D image, is further
modified by the expansion of horizontal edges logical step 332
shown in FIG. 13. Image 222 is typically divided in two sections.
The left edge of image 222 is cropped, copied and pasted on the
opposite edge of the image to form expanded image 232. In other
words, the left border section of each image is sliced in a
sub-section on the vertical axis, and the edge sub-section is
cropped and displaced on the opposite edge of the image, so that
two identical sub-sections of the image appear on both opposite
sides of the image. The expansion of horizontal edges step 332
reduces the processing workload necessary for the real-time
performance of the two points perspective distortion step 342,
allowing an output rate typically of 15 or more images per second.
This rate is sufficiently fast to create the illusion of a seamless
motion, without jerky movements, during displacement (image
panoramic motion or rotation of the view) within the image.
Increase of system performance is achieved because the expansion of
horizontal edges 332 avoids need for the image broadcasting system
to manipulate two images simultaneously. Otherwise, this is the
case every time the interactive panoramic reaches a vertical image
split at 0.degree. or 360.degree. image borders. The system of the
present invention achieves perpetual panoramic motion without
slowdown: when the border of an image is reached, the image
broadcasting system simply displays the other side of the same
image, in an invisible manner to the user.
[0086] A virtual walk-through application providing a convincing
natural, immersive navigation environment should be visually as
close as possible to human vision during panoramic motion (view
point rotation, horizontal displacement inside one panoramic
image), or translation motion (view point moving in space forward
or backward on the pathway), irrespective of motion speed.
[0087] According to a preferred embodiment of the present
invention, a panoramic image closer to human vision during
panoramic motion, with optimised calculation performance is
provided by the use of two-point perspective distortion. During
vertical panoramic motion, the image is vertically compressed
depending on vertical elevation (distance to the horizon) in order
to compensate for significant horizontal distortions which occur
further from the horizon. Such distortion is systematically caused
by the cylindrical projection of the image and is more severe
toward the vertical lower and upper parts of the image. The present
invention provides compensation of horizontal distortion similar to
the projection of a cylindrical view to a planar view. In order to
speed up image modification steps to real time, the present
invention departs from the traditional projection of a plane inside
a cylinder by assembling an image from the juxtaposition of several
vertical image strips 235, each of them optionally having a
different width, and by stretching said strips. This avoids the
need to recalculate the position of each pixel of the image
according to the projection of a plane inside a cylinder. Only a
selected vertical and a horizontal portion of the image are
displayed and visible to the user at any given time. The method of
the present invention described below works in a "staircase-like"
manner (visible on the zoomed image 236 of FIG. 15) whereby the
width of each stripe is controlled and reduced and the number of
stripes increased to provide the illusion of a progressive
curvature that is a perspective closer to human vision.
[0088] For this purpose, the panoramic images 232 are further
modified by the two points perspective distortion step 342 in order
to provide images based on a two points perspective. The two points
perspective distortion 342 step is achieved by applying two basic
operations, as shown in FIG. 15, allowing vertical parallel lines
in the real world to appear as parallel lines on screen during the
panoramic motion and interactive navigation.
[0089] In a first operation 343 (shown in FIG. 14), the viewable
portion 234 of any image 232 viewed by a user on screen, is divided
into vertical slices 235 (stripes) in real time, which are each
vertically stretched along sinusoidal lines in real time to recover
a two points perspective. The sinusoidal virtual canvas (the
vertical slices) is in a fixed position relative to the viewport
which is the rectangular window by where the image is viewed by the
user. As the user pans his view during navigation, the viewable
portion 234 travels across fixed vertical slices 235, during which
time only a portion of the panoramic image is visible on
screen.
[0090] In a second operation 345, vertical panoramic interactive
motion is achieved by simple vertical translation of vertical
slices 235 without affecting sinusoidal distortion (that is the
relative position and relative size of each stripe regarding each
other stripe). All the slices 235 are then vertically compressed
proportional to the angular divergence of the field of view to the
horizon, in order to minimise vertical distortion of the two points
perspective. The two points perspective distortion 342 avoids
trapezoidal distortion found on a regular three point perspective
used in 3D computer graphics.
[0091] The image modification steps of the present invention
processes images in two resolutions; low resolution for interactive
walkthrough panoramic motion (view point translation), and high
resolution for interactive panoramic motion (view point rotation)
applications.
[0092] Additional image modification steps are possible, through
the use of a logical processor.
[0093] In order to ensure protection of privacy and to avoid
infringement to privacy rights and right to images, the system of
the present invention can include logical means for the automatic
blurring, masking or removal of objects appearing on images, such
as human faces and car plates. Automatic blurring of faces is
achieved by a face recognition algorithm. Using image tracking
software known in the art, each individual face is identified and
tracked throughout any sequence of images. Based on the location of
faces or image-based coordinates provided by the tracking software
mechanism, software blurs the images on the coordinates
corresponding to recognizable faces. Automatic blurring of car
plates is achieved by a car plate recognition algorithm. Using
image tracking software known in the art, each individual car plate
is identified and tracked throughout any sequence of images. Based
on the location of car plates or image-based coordinates provided
by the tracking software mechanism, a software blurs car plates on
the coordinates corresponding to recognizable car plates. Identical
processes can be used to mask objects from images. The processes
above are not limited to faces or car plates but can be used to
blurred or mask any objects specified by the system's operator,
with corresponding software adaptation.
Management of Images
[0094] According to a preferred embodiment of the present
invention, panoramic resulting images 232 are stored on the central
computer 80, each image being associated with tridimensional (XYZ)
geographic coordinates indicative of the image's location of
capture and optionally other references such as date of capture,
with day and time; project or name of location; image group, so as
to identify for example a particular district or street; digital
photography settings, such as aperture setting, speed, ISO (measure
of the light sensitivity of the digital sensor), exposure, light
measure, camera model; identified for mounted photos filters such
as UV, neutral gray, etc.; vehicle information such as vehicle
speed and model; and camera operator identification.
[0095] Precise reference to the geographic coordinates indicative
of the image's location of capture allows, among other things, the
combination or the superposition of panoramic images of the present
invention with any other heterogeneous geo-referenced digital
information. Views of a street can therefore be combined with data
of a GIS ("Geographic Information System") application, such as,
for example, layers of the utilities' services infrastructure
(water, cable, electric distribution infrastructure), or contact
information associated with commercial establishments located in
the vicinity, PCT application No. ______, to Lindemann et al,
entitled SYSTEM AND METHOD FOR CREATING AND BROADCASTING
INTERACTIVE PANORAMIC WALK-THROUGH APPLICATIONS, filed concurrently
with the instant application and incorporated by reference hereto,
discloses a method for the assembly and broadcasting of interactive
panoramic walk through applications, using an object-based
database. Still further, PCT application No. ______, to Lindemann
et al, entitled SYSTEM AND METHOD FOR THE PRECISE INTEGRATION OF
VIRTUAL OBJECTS TO INTERACTIVE PANORAMIC WALK-THROUGH APPLICATIONS,
filed concurrently with the instant application and incorporated by
reference hereto, discloses a method for the integration of virtual
objects to interactive panoramic walk through applications, using a
method for determining geographic coordinates with increased
precision. The foregoing PCT applications provide functionality
which enhances the user experience or increases the efficiency of
the system 20 of the invention.
3D Computer Graphic Layering
[0096] According to a preferred embodiment of the present
invention, the system and method for creating panoramic
walk-through applications herein provide image modification steps
that enable the seamless and transparent integration of panoramic
images (from a panoramic optic 20) with virtual (3D) images,
wherein virtual images and panoramic images can be combined in a
plurality of ways, for example, to mask or super-impose each other.
It is therefore possible to combine panoramic images, virtual
images and 3D objects without limitation in any panoramic view; the
resulting view having a seamless appearance during walkthrough
interactive motion and panoramic motion.
[0097] The system of the present invention positions each image and
virtual objects on a 3D space using absolute geographic
coordinates. Any panoramic image is edited from either a source
image 210 or a virtual image 240 has geographic coordinates in 3D
space. Due to the use of absolute coordinates, the system of the
present invention allows the addition (pining) of virtual 3D
objects to any panoramic image or virtual image.
[0098] Masking operations are achieved by the system in which a
mask is defined, that is, the representation in shape and volume of
a real object appearing on the panoramic images is defined. This
technique is well known in the art and used to integrate 3D objects
in photos in the real-estate 3D industry, wherein said technique is
limited to static view. Effectively, the mask position and
orientation are correct for a given 3D object only for a given view
point.
[0099] According to an embodiment of the present invention, the use
of absolute 3D space coordinates for referencing both panoramic
images and virtual images inside the 3D space allows any given 3D
mask to be used on any panoramic image in a pathway. The mask
becomes visible from any view points that have a view on the mask's
geographical coordinates.
[0100] The 3D mask is designed to exist at a position and a shape
identical to respectively, the 3D position and shape of a real
physical object viewable on one or more panoramic images. When the
3D mask applied to cover a real physical object visible on a
panoramic image is located closer to the viewpoint of the panoramic
image, compared to the 3D object (i.e., located further from the
viewpoint), the 3D mask pixels will occlude the pixel of the 3D
object; in which case, pixels of the panoramic image representative
of the physical object may become visible when the mask is rendered
(the mask is "transparent" to an extent where pixels of the 3D
object behind the mask pixels also become transparent, revealing
the pixel of background panoramic image). In some cases, the mask
can be displayed with a virtual shadow from, or cast virtual shadow
to, other 3D objects to enhance the realism of the integration of
3D in the real image.
[0101] A 3D mask can be occluded by a 3D object when the pixels
representing said 3D object are located, relative to the view line,
in front (closer to viewpoint) of pixels of said mask. As the
masking process is determined on a pixel per pixel basis, complex
situations may arise. For example, a 3D object can be partially
occluded by a 3D mask that is partially occluded by another 3D
object. The flexibility of the masking process allows a near
visually perfect integration of 3D object inside real image used in
interactive panoramic walk-through applications.
[0102] According to a preferred embodiment of the present
invention, 3D computer graphic layering using masking operations
will produce new panoramic 3D layered images based on real world
panoramic image composed of pixels from 3D objects and 3D masks.
These panoramic 3D layered images are stored in the database 400
with geographic coordinates of the 3D space equivalent to
coordinates of the source panoramic image and tagged with a
reference to the 3D computer graphic layering program. Said 3D
computer graphic layering program reference can be a 3D application
source, or a file name, a project name or an application name or
both. 3D computer graphic layering can be used to mix 3D images
from different 3D rendering packages, or to add objects in the 3D
space without the need to re-render the full 3D panoramic images
(typically necessary when inserting new objects in already rendered
3D panoramic images). 3D computer graphic layering can therefore be
used for insertion and management of advertizing, wherein a virtual
advertisement can be inserted by defining a 2D (flat) or 3D object
and superimposing an advertising image on said object.
[0103] Image is available in alternative version broadcasted
depending on the context: typically, one panoramic representation
of a location is edited without any advertising, and several other
copies are made available to different advertisers; a software
based management tool allows advertisers to select an advertising
to be imported in the system; selection and display of advertising
to be broadcast can be dependent upon, for example, available user
information such as location and language, and the identity of the
web portal from which the user is accessing the panoramic
application.
[0104] Other characteristics and modes of execution of the
invention are described in the appended claims.
[0105] Further, the invention should be considered as comprising
all possible combinations of every feature described in the instant
specification, appended claims, and/or drawing figures which may be
considered new, inventive and industrially applicable.
[0106] Multiple variations and modifications are possible in the
embodiments of the invention described here. Although certain
illustrative embodiments of the invention have been shown and
described here, a wide range of modifications, changes, and
substitutions is contemplated in the foregoing disclosure. While
the above description contains many specifics, these should not be
construed as limitations on the scope of the invention, but rather
as exemplifications of one or another preferred embodiment thereof.
In some instances, some features of the present invention may be
employed without a corresponding use of the other features.
Accordingly, it is appropriate that the foregoing description be
construed broadly and understood as being given by way of
illustration and example only, the spirit and scope of the
invention being limited only by the claims which ultimately issue
in this application.
* * * * *