U.S. patent application number 08/872525 was filed with the patent office on 2001-08-02 for panoramic camera.
Invention is credited to DRISCOLL JR, EDWARD.
Application Number | 20010010555 08/872525 |
Document ID | / |
Family ID | 26693273 |
Filed Date | 2001-08-02 |
United States Patent
Application |
20010010555 |
Kind Code |
A1 |
DRISCOLL JR, EDWARD |
August 2, 2001 |
PANORAMIC CAMERA
Abstract
Most camera systems only record an image from a limited viewing
angle. A new panoramic camera apparatus is disclosed that
instantaneously captures a 360 degree panoramic image. In the
camera device, virtually all of the light that converges on a point
in space is captured. Specifically, in the camera of the present
invention, light striking this point in space is captured if it
comes from any direction, 360 degrees around the point and from
angles 50 degrees or more above and below the horizon. The
panoramic image is recorded as a two dimensional annular image.
Furthermore, various different systems for displaying the panoramic
images and distributing the panoramic images. Specifically, methods
and apparatus for digitally performing a geometric transformation
of the two dimensional annular image into rectangular projections
such that the panoramic image can be displayed using conventional
methods such as printed images and televised images.
Inventors: |
DRISCOLL JR, EDWARD;
(PORTOLA VALLEY, CA) |
Correspondence
Address: |
BE HERE CORPORATION
20300 STEVENS CREEK BLVD
SUITE 100B
CUPERTINO
CA
95014-2241
US
|
Family ID: |
26693273 |
Appl. No.: |
08/872525 |
Filed: |
June 11, 1997 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
60020292 |
Jun 24, 1996 |
|
|
|
Current U.S.
Class: |
348/335 ;
348/143; 348/39; 348/E5.055; 348/E7.087; 359/725; 359/726;
359/727 |
Current CPC
Class: |
H04N 7/183 20130101;
G08B 13/19628 20130101; G03B 37/00 20130101; H04N 5/2628 20130101;
G08B 13/19682 20130101; G08B 13/19626 20130101; G02B 13/06
20130101; G06T 3/0062 20130101 |
Class at
Publication: |
348/335 ; 348/39;
348/143; 359/725; 359/726; 359/727 |
International
Class: |
H04N 009/47; H04N
005/225; G02B 013/06 |
Claims
We claim:
1. An apparatus for capturing panoramic images, said apparatus
comprising: a convex mirror, said convex mirror reflecting light
from 360 degrees around said convex mirror; an astigmatism
correction lens, said astigmatism correction lens correcting
astigmatism in said light reflected from said convex mirror; an
objective lens for focusing said light reflected from said convex
mirror; and an image capture mechanism, said image capture
mechanism capturing said light reflected from said convex
mirror.
2. The apparatus as claimed in claim 1 further comprising: a field
flattening lens, said field flattening lens flattening a field of
optimal focus.
3. The apparatus as claimed in claim 1 wherein image capture
mechanism comprises photographic film.
4. The apparatus as claimed in claim 1 wherein image capture
mechanism comprises an electronic image capture mechanism.
5. The apparatus as claimed in claim 4 wherein said electronic
image capture mechanism comprises a charged coupled device
array.
6. An apparatus for capturing panoramic images, said apparatus
comprising: a convex mirror, said convex mirror reflecting light
from 360 degrees around said convex mirror; an objective lens for
focusing said light reflected from said convex mirror; a field
flattening lens, said field flattening lens flattening a field of
optimal focus; and an image capture mechanism, said image capture
mechanism capturing said light reflected from said convex
mirror.
7. The apparatus as claimed in claim 6 wherein image capture
mechanism comprises photographic film.
8. The apparatus as claimed in claim 6 wherein image capture
mechanism is housed within.
9. The apparatus as claimed in claim 8 wherein said image capture
mechanism comprises an electronic image capture device.
10. The apparatus as claimed in claim 8 wherein said electronic
image capture device comprises at least one charged coupled
device.
11. An apparatus for capturing panoramic images, said apparatus
comprising: a convex mirror, said convex mirror reflecting light
from a 360 degree band around said convex mirror, said 360 degree
band having a horizon line; and an image capture mechanism, said
image capture mechanism capturing an annular shaped image created
by said light reflected from said convex mirror; wherein said
convex mirror is shaped such that an angle of elevation from said
horizon line is linearly proportional to a radial distance of the
center of said annular shaped image.
12. The apparatus as claimed in claim 11 further comprising: a
field flattening lens, said field flattening lens flattening a
field of optimal focus.
13. The apparatus as claimed in claim 11 further comprising: an
astigmatism correction lens, said astigmatism correction lens
correcting astigmatism in said light reflected from said convex
mirror.
14. An apparatus for capturing panoramic images, said apparatus
comprising: a convex mirror, said convex mirror reflecting light
from a 360 degree band around said convex mirror, said 360 degree
band having a horizon line; a beam splitter, said beam splitter
dividing said reflected light into a first annular image and a
second annular image; a first electronic image capture mechanism,
said first electronic image capture mechanism capturing at least
half of said first annular image created by said light reflected
from said convex mirror; and a second electronic image capture
mechanism, said second electronic image capture mechanism capturing
at least half of said second annular image created by said light
reflected from said convex mirror.
15. The apparatus as claimed in claim 14 wherein said first
electronic image capture mechanism and said second electronic image
capture mechanism comprise charged coupled devices.
16. An apparatus for capturing panoramic images, said apparatus
comprising: a convex mirror, said convex mirror reflecting light
from a 360 degree band around said convex mirror; a second mirror,
said second mirror reflecting said light from said convex mirror
into an aperture in said convex mirror; and an image capture
mechanism, said image capture mechanism capturing said light
reflected from said convex mirror and said second mirror.
17. The apparatus as claimed in claim 16 wherein said image capture
mechanism is housed within said convex mirror.
18. A panoramic camera apparatus, said panoramic camera apparatus
comprising: an image capture mechanism; a main reflector, said main
reflector reflecting light from a 360 degree panorama onto said
image capture mechanism; and a protective transparent block; said
protective transparent block covering said main reflector.
19. The apparatus as claimed in claim 18 wherein said protective
transparent block is shaped such that light rays that will be
reflected onto said image capture mechanism enter said protective
transparent case at a normal to the surface of said protective
transparent block.
20. The apparatus as claimed in claim 18 further comprising: a
second reflector, said second reflector reflecting light from said
main reflector down through an opening in said main reflector.
21. The apparatus as claimed in claim 20 wherein said protective
transparent block ends at said opening in said main reflector.
22. The apparatus as claimed in claim 20 wherein an lens system is
housed within an indentation of said protective transparent
block.
23. A method of recording a panorama, said method comprising the
steps of: reflecting light from a 360 degree band around a convex
mirror to an image plane; correcting astigmatism in said light
reflected from said convex mirror; and capturing an annular
representation of said panorama on said image plane with an image
capture mechanism.
24. A method of recording a panorama, said method comprising the
steps of: reflecting light from a 360 degree band around a convex
mirror to an image plane; flattening a field of optimal focus for
said reflected light; and capturing an annular representation of
said panorama on said image plane with an image capture
mechanism.
25. The method as claimed in claim 24 further comprising:
correcting astigmatism in said light reflected from said convex
mirror.
26. The method as claimed in claim 24 wherein said step of
capturing an annular representation comprises capturing an annular
image on photographic film.
27. The method as claimed in claim 24 wherein said step of
capturing an annular representation comprises capturing an annular
image on a charged coupled device array.
28. The method as claimed in claim 24 further comprising: focusing
said light reflected from said convex mirror with an objective
lens.
29. A method of presenting panoramic images, said method comprising
the steps of: recording an annular representation of a panorama;
storing said annular representation of said panorama as a digitized
annular representation; geometrically transforming said digitized
annular representation of said panorama into a rectangular
projection of said panorama; and displaying said rectangular
projection of said panorama.
30. The method as claimed in claim 29 wherein said step of
mathematical transforming further comprises sampling said digitized
annular representation with a dynamically changing sampling shape
wherein said sampling shape is dependent upon a viewing angle.
31. The method as claimed in claim 29 wherein said step of
geometrically transforming further comprises sampling said
digitized annular representation with a dynamically changing
sampling shape orientation wherein said sampling shape orientation
is dependent upon a horizontal viewing angle.
32. A method of presenting panoramic images, said method comprising
the steps of: recording an annular representation of a panorama;
storing said annular representation of said panorama as a digitized
annular representation; geometrically transforming said digitized
annular representation of said panorama into a cylindrical
representation of said panorama; and displaying said cylindrical
representation of said panorama using a computer program.
33. The method as claimed in claim 32 wherein said step of
displaying said cylindrical representation of said panorama using a
computer program comprises running a QuickTime VR program.
34. A method of presenting panoramic images of a remote location,
said method comprising the steps of: capturing a digitized annular
representation of a panorama at said remote location; transmitting
said digitized annular representation to a local computer;
geometrically transforming said digitized annular representation
into a rectangular projection of said panorama using said local
computer; and displaying said rectangular projection of said
panorama.
35. A method of presenting panoramic images of a remote location,
said method comprising the steps of: capturing a digitized annular
representation of a panorama at said remote location; geometrically
transforming said digitized annular representation into a
rectangular representation of said panorama; transmitting said
rectangular representation of said panorama to a local receiver;
and displaying said rectangular representation of said panorama on
said local receiver.
36. The method of presenting panoramic images of a remote location
as claimed in claim 35 wherein said step of geometrically
transforming said digitized annular representation into a
rectangular representation of said panorama comprises geometrically
transforming a portion of said digitized annular representation
into rectangular projection.
37. The method of presenting panoramic images of a remote location
as claimed in claim 36 further comprising the steps of:
transmitting a request from said local receiver to said remote
location; and selecting said portion of said digitized annular
representation dependent upon said request.
38. A method of presenting panoramic images of a remote location,
said method comprising the steps of: capturing a digitized annular
frame of a panorama at said remote location; geometrically
transforming said digitized annular frame into a rectangular
projection frame of said panorama; transmitting said rectangular
projection frame of said panorama to a local receiver; displaying
said rectangular representation of said panorama on said local
receiver; capturing a stream of digitized annular frames of a
panorama at said remote location; geometrically transforming a
particular portion of said stream of digitized annular frames into
a stream of rectangular projection frames; and displaying said
stream of rectangular projection frames on said local receiver.
39. The method of presenting panoramic images of a remote location
as claimed in claim 38 further comprising the steps of:
transmitting a request from said local receiver to said remote
location; and selecting said particular portion of said digitized
annular representation dependent upon said request.
40. The method of presenting panoramic images of a remote location
as claimed in claim 38 further comprising the steps of: capturing
audio at said remote location; and transmitting said audio to said
local receiver.
41. A method of presenting images from a remote location, said
method comprising the steps of: capturing a digitized annular
representation of a panorama at said remote location; geometrically
transforming said digitized annular representation into a
rectangular representation of said panorama; selecting a viewport
from said rectangular representation of said panorama; transmitting
said viewport from said rectangular representation of said panorama
to a local receiver; and displaying said viewport on said local
receiver.
42. The method of presenting images from a remote location as
claimed in claim 41, said method further comprising the steps of:
capturing at least one audio stream at said remote location;
transmitting said audio stream to said local receiver; and
generating audio at said local receiver using said audio
stream.
43. The method of presenting images from a remote location as
claimed in claim 41 wherein said steps are repeated such that said
viewport on said local receiver comprises a motion image.
44. The method of presenting images from a remote location as
claimed in claim 41 wherein said step of capturing at least one
audio stream at said remote location comprises, said method further
comprising the steps of: capturing multiple directional audio
sources; mixing said multiple directional audio sources based upon
said selected viewport to create a mixed audio stream; transmitting
said audio stream to said local receiver; and generating audio at
said local receiver using said audio stream.
45. The method of presenting images from a remote location as
claimed in claim 41, said method further comprising the steps of:
transmitting at least one command from said local receiver to said
remote location; and selecting said viewport dependent upon said
command.
46. A method of presenting panoramic images of a remote location,
said method comprising the steps of: capturing a digitized annular
representation of a panorama at said remote location; selecting a
viewport from said digitized annular representation of said
panorama; geometrically transforming said selected viewport from
said digitized annular representation into a rectangular
representation of said viewport; transmitting said rectangular
representation of said viewport to a local computer; and displaying
said rectangular representation of said viewport on said local
computer.
47. The method of presenting images from a remote location as
claimed in claim 46 wherein said steps are repeated such that said
viewport on said local receiver comprises a motion image.
48. The method of presenting images from a remote location as
claimed in claim 46, said method further comprising the steps of:
capturing at least one audio stream at said remote location;
transmitting said audio stream to said local receiver; and
generating audio at said local receiver using said audio
stream.
49. The method of presenting images from a remote location as
claimed in claim 46 wherein said step of capturing at least one
audio stream at said remote location comprises, said method further
comprising the steps of: capturing multiple directional audio
sources; mixing said multiple directional audio sources based upon
said selected viewport to create a mixed audio stream; transmitting
said audio stream to said local receiver; and generating audio at
said local receiver using said audio stream.
50. The method of presenting images from a remote location as
claimed in claim 46, said method further comprising the steps of:
transmitting at least one command from said local receiver to said
remote location; and selecting said viewport dependent upon said
command.
51. A camera apparatus, said camera apparatus comprising: a
panoramic camera for capturing a digitized annular representation
of a panorama at said remote location; a computer for geometrically
transforming said digitized annular representation into a
rectangular representation of said panorama, said computer
selecting a viewport from said rectangular representation of said
panorama; a communication device for transmitting said viewport
from said rectangular representation of said panorama to a local
receiver; and a local receiver for receiving said viewport and
displaying said viewport.
52. The camera apparatus as claimed in claim 51, said camera
apparatus further comprising: at least one microphone for capturing
at least one audio stream at said remote location, said audio
stream transmitted to said local receiver; and audio generator for
generating audio at said local receiver using said audio
stream.
53. The camera apparatus as claimed in claim 51, said camera
apparatus further comprising: at least two microphones at said
remote location for capturing multiple audio sources; a mixer at
said remote location, said mixer combining said multiple audio
sources into a mixed audio stream, said mixed audio stream
transmitted to said local receiver; and audio generator for
generating audio at said local receiver based upon said mixed audio
stream.
54. The camera apparatus as claimed in claim 53 wherein said mixer
mixes said multiple audio sources based upon said selected
viewport.
55. The camera apparatus as claimed in claim 51 wherein said local
receiver transmits at least one command to said computer to
indicate a desired viewport.
56. The camera apparatus as claimed in claim 51 wherein said
communication device continuously transmits viewports such that a
motion image is displayed at said local receiver.
57. A camera apparatus, said camera apparatus comprising: a
panoramic camera for capturing a digitized annular representation
of a panorama at said remote location; a computer for selecting a
viewport from said digitized annular representation of said
panorama, said computer geometrically transforming said digitized
viewport into a rectangular representation of said panorama; a
communication device for transmitting said viewport from said
rectangular representation of said panorama to a local receiver;
and a local receiver for receiving said viewport and displaying
said viewport.
58. The camera apparatus as claimed in claim 57, said camera
apparatus further comprising: at least one microphone for capturing
at least one audio stream at said remote location, said audio
stream transmitted to said local receiver; and audio generator for
generating audio at said local receiver using said audio
stream.
59. The camera apparatus as claimed in claim 57, said camera
apparatus further comprising: at least two microphones at said
remote location for capturing multiple audio sources; a mixer at
said remote location, said mixer combining said multiple audio
sources into a mixed audio stream, said mixed audio stream
transmitted to said local receiver; and audio generator for
generating audio at said local receiver based upon said mixed audio
stream.
60. The camera apparatus as claimed in claim 59 wherein said mixer
mixes said multiple audio sources based upon said selected
viewport.
61. The camera apparatus as claimed in claim 57 wherein said local
receiver transmits at least one command to said computer to
indicate a desired viewport.
62. The camera apparatus as claimed in claim 57 wherein said
communication device continuously transmits viewports such that a
motion image is displayed at said local receiver.
Description
CLAIM OF PRIORITY
[0001] This application claims the benefit of the U.S. provisional
patent application with Ser. No. 60/020,292 that was filed on Jun.
24, 1996.
FIELD OF THE INVENTION
[0002] The present invention relates to the field of film and video
photography. In particular the present invention discloses a camera
device that captures a 360 degree panoramic image and display
systems for displaying the panoramic image captured by the camera
device.
BACKGROUND OF THE INVENTION
[0003] Most cameras only provide a small viewing angle. Thus, a
typical conventional camera only captures an image in the direction
that the camera is aimed. Limited view cameras force viewers to
look only at what the camera operator chooses to focus on. Some
cameras use a specialized wide angle lens to capture a wider
panoramic image, but such panoramic cameras still have a limited
field of view.
[0004] It would be desirable to have a camera system that would
capture the light from all directions such that a full 360 degree
panoramic image can be created. A full 360 degree panoramic image
would allow the viewer to choose what she would like to look at.
Furthermore, a full 360 degree panoramic image allows multiple
viewers to simultaneously view the world from the same point, with
each being able to independently choose their viewing direction and
field of view.
[0005] At the present time, there are some known methods of
creating 360 degree panoramic images. However, most current methods
are subject to limitations due to their physical movements and
mechanical complexity. For example, some of the current methods
operate by combining a series of individual photographs taken in
different directions into a single panoramic image. Some panoramic
cameras spin a lens and film to capture a panoramic view in a
single sweeping motion.
[0006] There is a market for panoramic photos to be used in
multimedia applications, typically provided on CD-ROMs. In the last
few years, some software manufacturers have introduced standards
for digital storage and computer playback of panoramic datasets.
One example is QuickTime.RTM. VR, introduced by Apple.RTM.
Computer, Inc. Apple.RTM. Computer's QuickTime.RTM. VR standard
governs the file storage format and the playback software needed to
view their datasets.
[0007] Currently, Apple Computer recommends and provides software
tools to implement a labor-intensive process for capturing these
panoramic datasets. In the Apple QuickTime.RTM. VR (QTVR) process a
standard 35 mm camera is mounted vertically on a leveled tripod and
equipped with an extreme wide angle lens (e.g. 15-18 mm focal
length). A sequence of twelve or more overlapping still photographs
is taken at roughly 30 degree intervals as the camera is turned on
the tripod around a vertical axis. These photographs are developed,
digitized and then fed into a semi-automated software program
called a "stitcher" that merges the overlapping still photographs
into one long panoramic strip.
[0008] The labor intensive process suffers from a number of
shortcomings. First, the process is time-consuming since many steps
requiring human intervention and guidance. Furthermore, the
recommended process is prone to temporal artifacts since it
captures each individual photo at a different time. This means that
the "stitched" pan image is not instantaneous but rather is made up
of individual photos taken at different times. The time change
during the series of photographs makes it nearly impossible to
create panoramic images in changing scenes containing shorelines,
urban crowds and traffic, windblown trees, etc. Finally, it is
difficult to see how the image capture method recommended by Apple
QuickTime.RTM. VR (QTVR) can be extended from a single still
panoramic image into a continuous frame, or motion picture
panoramic image capture.
SUMMARY OF THE INVENTION
[0009] The present invention discloses a camera device that
instantaneously captures a 360 degree panoramic image. Furthermore,
the present invention discloses various different systems for
displaying the panoramic images.
[0010] In the camera device, virtually all of the light that
converges on a point in space is captured. Specifically, in the
camera of the present invention, light striking this point in space
is captured if it comes from any direction, 360 degrees around the
point and from angles 50 degrees or more above and below the
horizon as illustrated in FIG. 1.
[0011] Other objects, features and advantages of present invention
will be apparent from the company drawings and from the following
detailed description that follows below.
BRIEF DESCRIPTION OF THE DRAWINGS
[0012] The objects, features and advantages of the present
invention will be apparent to one skilled in the art, in view of
the following detailed description in which:
[0013] FIG. 1 illustrates the panoramic surroundings that are
captured by the panoramic camera system of the present
invention.
[0014] FIG. 2a illustrates a schematic diagram of the panoramic
camera system of the present invention.
[0015] FIG. 2b illustrates a schematic diagram of the panoramic
camera system of the present invention with a parabolic mirror.
[0016] FIG. 2c illustrates an annular image captured by the
panoramic camera system of the FIG. 2b wherein the incident angle
is linearly proportional to the radial distance of the annular
image.
[0017] FIG. 3a illustrates an example of an annular image captured
by the panoramic camera system of the present invention.
[0018] FIG. 3b illustrates a rectangular panoramic image after the
captured annular image is transformed from polar coordinates to
rectangular coordinates.
[0019] FIG. 4a illustrates photographic film used to capture the
annular panoramic image.
[0020] FIG. 4b illustrates a Charged Coupled Device array used to
capture the annular panoramic image.
[0021] FIG. 5 illustrates an alternate embodiment of the camera
system of the present invention wherein a beam splitter is used to
allow the annular image to be captured on two image planes.
[0022] FIG. 6a illustrates a first embodiment of two image planes
used to capture different portions of a single annular panoramic
image.
[0023] FIG. 6b illustrates a second embodiment of two image planes
used to capture different portions of a single annular panoramic
image.
[0024] FIG. 7 illustrates an embodiment of the panoramic camera
wherein some of the optical elements are housed within the
parabolic mirror.
[0025] FIG. 8a illustrates a first embodiment of the panoramic
camera that uses a solid transparent block to surround the
parabolic mirror.
[0026] FIG. 8b illustrates a second embodiment that uses a solid
transparent block to surround the parabolic mirror which houses
other optical elements.
[0027] FIG. 9a illustrates an embodiment that panoramic camera that
supports the convex mirror with a central post that is out of the
annular field of view.
[0028] FIG. 9b illustrates an embodiment that panoramic camera that
divides the convex mirror in quarters and supports the mirror using
posts between the four quarters.
[0029] FIG. 10 graphically illustrates how the annular image is
sample to product a rectangular panoramic image.
[0030] FIG. 11a graphically illustrates how an image is stored in
Apple.RTM. Computer's QuickTime.RTM. VR format.
[0031] FIG. 11b graphically illustrates how viewports are created
from Apple.RTM. Computer's QuickTime.RTM. VR format.
[0032] FIG. 11c illustrates a flow chart that lists how the
panoramic camera system can be used to create images in Apple.RTM.
Computer's QuickTime.RTM. VR format.
[0033] FIG. 12a illustrates a graphical user interface for a client
program used to view panoramic still images created by the
panoramic camera system.
[0034] FIG. 13a illustrates a graphical user interface for a client
program used to view panoramic video created by the panoramic
camera system.
[0035] FIG. 13b illustrates a networked computer arrangement used
to view panoramic video created by the panoramic camera system.
[0036] FIG. 14a illustrates a side view of one embodiment of the
panoramic camera system that includes microphones.
[0037] FIG. 14b illustrates a side view of one embodiment of the
panoramic camera system that includes microphones.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT
[0038] A method and apparatus for a camera device that
instantaneously captures 360 degree panoramic images is disclosed.
In the following description, for purposes of explanation, specific
nomenclature is set forth to provide a thorough understanding of
the present invention. However, it will be apparent to one skilled
in the art that these specific details are not required in order to
practice the present invention. For example, the present invention
has been described with reference to Charge Coupled Devices.
However, the panoramic camera system can easily be implemented with
other types of electronic image capture systems.
The Basic Panoramic Camera Design
[0039] The panoramic camera design of the present invention
captures light from all directions within 50 to 60 degrees above
and below the horizon simultaneously. FIG. 1 graphically
illustrates the cylindrical panoramic view of which the panoramic
camera system captures an image. To capture all the light of the
panorama and generate a two dimensional representation that may
easily be recorded, the present invention uses a carefully designed
and engineered collection of mirrors and lenses. The basic design
of the panoramic camera of the present invention is illustrated in
FIG. 2a. Each element of the panoramic camera will be described
individually.
[0040] The Mirror
[0041] Referring to FIG. 2a, the panoramic camera of the present
invention collects light using a convex mirror 210 that is in the
approximate shape of parabolic cone. In one embodiment of the
present invention, the tip of the convex mirror 210 is pointed
downward as illustrated in FIG. 2a. When the convex mirror 210 is
viewed from below, the parabolic mirror 210 presents an annular
image of the surrounding panorama as illustrated in FIG. 3a.
However, the annular image is distorted and suffers from optical
defects due to the shape of the convex mirror 210.
[0042] The distortion in the image is partly due to the fact that
the convex mirror 210 of the imaging system effectively converts
the surrounding panorama to a polar coordinate system. By adjusting
the shape of the convex mirror 210, the mapping of the elevation
angle of incoming light to radial distance in the annular image,
can be controlled.
[0043] In a preferred embodiment, the convex mirror 210 is a
parabolic mirror that creates an annular image wherein the radial
distance from the center of the annual image is linearly
proportional to the angle of incident light. A panoramic camera
system with a parabolic mirror is illustrated in FIG. 2b. Note that
in the image plane of FIG. 2b, the distance from the center is
linearly proportional to the angle of incident light. This concept
is more clearly illustrated in FIG. 3c, wherein the concentric
circles represent different angles of incident light.
[0044] The Astigmatism Correction Lens
[0045] The convex mirror of the present invention introduces other
image defects that require careful correction. One particular
problem is astigmatism. Specifically, the light reflected downward
from the convex mirror 210 of the present invention will not meet
at a single focal point. To correct for this problem, an
astigmatism correction lens 220 is added to correctly focus the
light from the convex mirror 210.
[0046] The astigmatism correction lens 220 comprises a group of 2
or more lenses whose group focal length is long but with individual
elements of strong and opposite power. Thus, the astigmatism lens
group may be made of the same optical material without introducing
significant lateral color. Since the beam size associated with any
object point in space to be imaged is quite small compared to the
field of the beam, the strong elements tend to introduce
deleterious amounts of spherical aberration or coma into the final
image.
[0047] The Objective Lens
[0048] The next component is a standard camera objective lens 230.
The standard camera objective lens 230 forms an image using the
astigmatism-corrected, reflected light from the convex mirror 210.
In the present embodiment, a standard off-the-shelf camera lenses,
that are optimized for cost and performance in the conventional
photography market. The current embodiment relies upon a
pre-defined focal length.
[0049] The focal length of the standard objective lens is selected
based on two factors. The first factor is the maximum angular field
of view present by the convex mirror and astigmatism correction
lens group. This factor is determined by the largest angle away
from the horizon of an object to be captured. The second factor is
the maximum diameter of the circular image to be recorded. In an
embodiment that uses 35 mm film, this value would not exceed 24 mm.
In an embodiment that uses Charged Coupled Device arrays, the
objective lens must keep the circular image within the bounds of
the CCD array.
[0050] For one preferred embodiment, the appropriate focal length
is 22 mm. Since there are many objective lenses available with
focal lengths in the 18 mm to 24 mm range, this focal length
provides many off-the-shelf lens choices.
[0051] To allow a standard off-the-shelf camera lens to be used,
the present invention "false focuses" the image beyond the normal
focal plane. This allows the next optical element (field flattening
lens) to fit between the objective lens 230 and the image
plane.
[0052] The Field Flattening Lens
[0053] Another optical problem created by the parabolic mirror is a
curved image field that is created by the curve of the parabolic
mirror. The curved image field problem is solved by add yet another
lens 240. This final lens is a "field flattening" lens, that
flattens the field of optimal focus to a flat two dimensional image
plane. The field flattening lens 240 must be kept as close to the
image plane as practical to eliminate the need for a focal plane
shutter.
[0054] In one embodiment, the material SFL6 is used to create the
field flattening lens 240. Due to its high index of refraction,
SFL6 allows the field flattening lens 240 to be approximately 2
millimeters thick. If the field flattening lens 240 was created
using more traditional materials, the field flattening lens 240
would be approximately 4.5 millimeters thick.
[0055] The Image Capture System
[0056] The final major component of the panoramic camera design is
the image capture mechanism 250. The image capture mechanism 250 is
placed at the image plane just beneath the field flattening lens
240. This mechanism captures the optimized two dimensional annular
image of the surrounding panorama. An example of a captured
panorama stored as a two dimensional annular representation is
shown in FIG. 3a.
[0057] In one embodiment of the present invention, the image
capture mechanism can be a frame of photographic film as
illustrated in FIG. 4a. Using convention photography techniques,
several successive frames can be used to record series of images.
The series of images may be several distinct still images taken
from different locations. Alternatively, the series of images may
be a set of successive images recorded used to create a panoramic
motion picture. The image that is recorded onto photographic film
is then later converted into a digital image for digital image
processing as will be described in later sections of this
document.
[0058] In the preferred embodiment of the present invention, a high
resolution digital image capture system is used to capture the
annular image created by the optical elements. In one embodiment of
the present invention a Charged Coupled Device (CCD) array 450 is
placed in the image plane to capture the image as illustrated in
FIG. 4b. Control circuitry 460 coupled to the CCD array 450
captures the image directly into a digital format. The use of a
digital image capture system allows an immediate read-out of the
digitized raw annular image. The digitized raw annular image can be
stored into a storage device 470 such as flash memory or a hard
disk drive. The CCD array 450 may be color (RGB) or black &
white, depending on the intended application.
[0059] To generate an annular image of sufficient quality to be
used in the Apple QuickTime.RTM. VR market, it has been determined
that the image plane must be sampled with an array having at least
2K by 2K elements. To meet this requirement, one embodiment of the
present invention uses a CCD array produced by Loral-Fairchild,
Inc. However, the high resolution CCD array sold by
Loral-Fairchild, Inc. adds a significant cost to the panoramic
camera of the present invention. Furthermore, large CCD arrays such
as the Loral-Fairchild array have difficulty handling the extreme
differences in light intensity that are produced optical system of
the present invention. Specifically, one area of the image may have
direct sunlight and other areas may receive comparatively little
light.
[0060] To reduce the production cost of the panoramic camera,
alternate embodiments of the present invention use a set of lower
resolution CCD arrays. Specifically, consumer grade CCD devices
that are targeted at the consumer electronics market are used.
Consumer electronics grade CCD arrays have the distinct advantages
of lower cost, more highly-integrated support circuitry
availability, high speed read-out, robustness to extreme lighting
and other environmental conditions.
[0061] No individual consumer grade CCD array meets the high
resolution requirements needed by the present invention (at least
2K by 2K elements). Therefore, a method of obtaining a greater
image resolution is required if consumer grade CCDs are used.
[0062] One method of creating an acceptable image capture mechanism
using consumer grade CCD arrays is to use multiple low resolution
CCD chips to cover the image plane using a mosaic pattern.
Referring to FIG. 5, the basic panoramic camera configuration
described in the previous sections is illustrated except the last
stage has an added beam-splitter 545 that directs a second image to
a second field flattening lens 541 and a second image plane 551.
The beam-splitter 545 may comprise a half-silvered mirror or a
prism arrangement as is known in the art. The two image planes
(image plane 551 and image plane 553) each capture a portion of the
whole annular image. To construct a complete image, the camera
optically composites the images from the two different image planes
into a single image.
[0063] FIGS. 6a and 6b each illustrate one possible embodiment of
the dual image plane image capture system. FIG. 6a illustrates a
mosaic pattern create with four consumer grade CCD devices. As
illustrated in FIG. 6a, the two image planes capture the whole
annular image while each image plane leaves room for the chip lead
frames and support circuitry. FIG. 6b illustrates an alternate
embodiment that six consumer grade CCD devices. An additional
advantage of this scheme is that the CCD array chips can
potentially share some supporting circuitry since the signals each
independent chip is requiring are often identical.
[0064] A disadvantage of the mosaic technique is the image capture
variation that will exists between the different CCD chips. The
image variation can be compensated for by having overlapping CCD
array coverage. The overlapping area is used to cross calibrate the
image variation between adjacent CCD arrays.
[0065] Folded Optics Configuration
[0066] FIG. 7 illustrates an alternative embodiment of the
panoramic camera system. In the embodiment of FIG. 7, the camera
subassembly is housed within the parabolic mirror. Referring to
FIG. 7, the convex mirror 710 is inverted and a hole is cut into
the tip. A second mirror 715 is placed above the convex mirror 710,
directing the light from the surrounding panorama into the hole in
the top of the convex mirror 710. The remainder of the optical
path, including the astigmatism correction lens 720, the objective
lens 730, the field flattening lens 740, and the image capture
mechanism 750, are all housed inside the inverted convex mirror
710. It is apparent from the diagram of FIG. 7 that the "folded
optics" configuration protects the optical path and mechanical
parts of the panoramic camera system.
[0067] Transparent Block Configuration
[0068] Another alternative embodiment is shown in FIG. 8a. In this
alternative, the convex mirror is formed as the internal space of a
curved block of transparent material such as glass or plastic. The
mirror surface 810 is formed by the inner surface of hole that is
milled or cast in the top of the transparent material 805. The
shape of the outer surface approximates a sphere centered on the
virtual focal point of the convex mirror 810. The outer surface of
the transparent material is a polished surface that forms the
outside skin of the camera. The bottom tip of the transparent block
is optically mated to the other optical parts of the camera system.
The bottom tip may be polished flat or molded into a shape that
contributes to the astigmatism lens group.
[0069] The solid transparent block approach has a number of
significant advantages. First, the mirrored inner surface of the
transparent block material can be well protected. This technique
overcomes the disadvantages of front surface mirrors. Specifically,
when front surface mirrors are exposed to the outside world they
are susceptible to damage and degradation. In the above described
embodiment, the mirrored surface is fully protected since it is
encased between a protective backing material and the transparent
block material. Another advantage of the solid block approach is
that the skin of the camera is incorporate into the optical system.
Thus only one surface would need to be multicoated to prevent
internal reflections.
[0070] The transparent block technique can also be implement using
the folded optics scheme described in the previous section.
Specifically, FIG. 8b illustrates an inverted solid transparent
block used to implement a panoramic camera system. In this case,
the camera components are contained within the mirror cavity. Note
that the outside surface at the top of the block is no longer an
exit path bus is instead a mirrored surface that directs the image
light down into an optical path inside the parabolic block.
[0071] Different methods can be used to construct a transparent
block panoramic camera system. One method would be to create the
transparent block, then polish the transparent block, and finally
add a mirrored surface where appropriate. An alternate method of
constructing a transparent block panoramic camera system would
start with the convex mirror. Then, the convex mirror would be
encapsulated within the transparent block. This method would be
simpler to construct since a concave surface would not require
polishing. Furthermore, the convex mirror would be protected by the
transparent block.
[0072] Center Support Configuration
[0073] Another alternative embodiment addresses the problem of how
to align and support the optical elements of the panoramic camera
illustrated in FIG. 2a. It is possible to use the protective,
transparent block technique as described in the previous section to
provide structure, stability and alignment. However, the
transparent block technique requires multicoating of the surfaces
or else undesired internal reflections will be visible. FIG. 9a
discloses an alternate embodiment wherein a central post 903 is
used to support the parabolic mirror 910. The remainder of the
optical system is below the parabolic mirror 910 and the central
post 903. The center support scheme takes advantage of the fact
that the center of the annular image is discarded since it contains
only an image of the camera itself. Therefore, the center portion
of the annular image can be used for support of the parabolic
mirror 910.
[0074] External Support Configuration
[0075] Another scheme for supporting the parabolic mirror above the
optical elements below is to use several side supports. This can be
accomplished by splitting the parabolic mirror into "pie-pieces" by
cutting the parabolic mirror after fabrication. For example, the
parabolic mirror can be quartered as illustrated in FIG. 9b. The
four sections of parabolic mirror 1021, 1022, 1023, and 1024 can be
spread apart slightly, allowing for the introduction of supporting
elements 1031, 1032, 1033, and 1034 that will not obstruct the
fields of view.
[0076] If the parabolic mirror is split into four sections, then
annular image will appear as four quadrants at the image plane. To
correct for this, the gaps can be removed during the
polar-to-rectangular coordinate conversion, thereby restoring the
continuity of the panoramic image. The gaps between the mirror
sections should be kept as small as possible, however, since the
optical system is degraded by the loss of rotational symmetry.
Panoramic Image Presentation
[0077] As illustrated in FIG. 3a, the panoramic camera system of
the present invention records a two dimensional annular
representation of the surrounding panorama. However, the annular
representation is not of much interest to most viewers. Therefore,
to display the panoramic images captured by the panoramic camera of
the present invention, several different display systems are
disclosed.
[0078] Still Image Presentation as a Rectangular Panoramic
Image
[0079] The most common method of displaying a panoramic image is to
display the image as a rectangle where the horizontal direction
represents the view angle. An example of this type of panoramic
image presentation is illustrated in FIG. 3b. Such rectangular
panoramic images are commonly displayed in nature magazines. As
stated in the background, the prior art method of creating such
rectangular panoramic images was to take several convention
photographs at different angles and then stitch those photographs
together somehow.
[0080] With the panoramic camera system of the present invention,
such rectangular panoramic images can easily be created. First the
panoramic camera system of the present invention is used to capture
an annular image of the surrounding panorama. Then the annular
image is digitized and loaded into a computer system. (The image
will already be in digital form if a CCD version of the panoramic
camera system was used to capture the image.)
[0081] A custom conversion program is then executed on the computer
system. The custom conversion program scans around the annular
image starting at an arbitrarily chosen sampling line 310. Points
along the sampling line 310 are sampled and then their position
changed using polar coordinate to rectangular coordinate
conversion. FIG. 10 illustrates how two different points on the
annular image are sampled and then placed into rectangular
coordinates. As illustrated in FIG. 10, the orientation of the
sampling pattern changes as the coordinate transform program
rotates around the annular image. The resulting rectangular image
is illustrated in FIG. 3b.
[0082] While sampling the annular image, it is important to sample
the image differently depending on where the annular image is being
sampled. The following three rules must be observed:
[0083] 1. The sampling shape is dynamically changing depending on
the viewing angle (both in the horizontal and vertical).
[0084] 2. The sampling shape size proportional to the radius
(vertical viewing angle); and
[0085] 3. The sampling shape orientation is different depending on
the horizontal viewing angle.
[0086] Since there is a greater resolution around the outer
perimeter of the annular image, the corresponding rectangular image
portion will have better image clarity. The outer perimeter of the
annular image may be the top or the bottom of the rectangular image
depending on the optical path. (Compare FIG. 2a with FIG. 7). In
FIG. 10, the lower portion of the rectangular image will have a
better image clarity since the it is from the outer perimeter of
the annular image. One embodiment of the present invention takes
advantage of this fact by using the outer perimeter of the annular
image for the ground since the ground in a panoramic scene is
generally more detailed than the sky.
[0087] Once the panoramic image has been converted from an annular
image to a rectangular image on a computer system, then the
rectangular image can be presented to viewers in a number of
different formats. For example, the rectangular image may be
distributed electronically as a JPEG image and viewed with JPEG
image viewers. Alternatively, the rectangular image can be printed
out with a color printer. It should be noted that since the
rectangular image is in digital form, it can quickly be added to a
publication being created with a Desktop Publishing Layout Program
such QuarkXpress or Adobe's PageMaker.
[0088] Image Presentation as a Virtual Reality Image
[0089] Apple Computer introduced a standard known as QuickTime.RTM.
VR for storing and displaying virtual reality images. Apple
Computer's QuickTime.RTM. VR standard governs the data storage
format and the playback software needed to view the QuickTime.RTM.
VR datasets. The camera system of the present invention can be used
to quickly create QuickTime.RTM. VR datasets.
[0090] The QuickTime.RTM. VR format stores the image as cylindrical
image as illustrated in FIG. 11a. Specifically, the viewpoint is at
the center of the cylinder and the inner surface of the cylinder
represents the stored QuickTime.RTM. VR image. Note that trapezoid
shaped patches must be sampled to generate an image if the user is
looking up or down as illustrated in FIG. 11b.
[0091] FIG. 11c illustrates a flow diagram that lists the steps
required to produce a QuickTime.RTM. VR dataset using the panoramic
camera system of the present invention. First, at step 1110, a
panoramic image is recorded with the panoramic camera. Then, at
step 1120, the recorded image is digitized and loaded into a
computer system. If the panoramic camera recorded the image on a
piece of film, then a print of the film can be scanned into the
computer system using a flatbed scanner. Alternatively, a film
image can be commercially transformed into the well known
PhotoCD.RTM. format produced by Kodak.RTM.. If the panoramic camera
recorded the image with a CCD array and stored the image digitally,
then the digital image is just copied from the camera's storage
system into the computer system 's storage system.
[0092] After the digital version of the annular image is available
on the computer system, a transformation program is then executed
on the computer system at step 1130 in order to transform the
digitized annular image into a QuickTime.RTM. VR dataset. The
annular image produced by the camera system of the present
invention stores the panoramic image information in a polar
coordinate system. Conversely, Apple.RTM.'s QuickTime.RTM. VR uses
a cylindrical coordinate system as illustrated in FIG. 11a. Thus,
the transformation program converts the annular image from its
polar coordinate system into the QuickTime.RTM. VR cylindrical
coordinate system. After transforming the image into the
QuickTime.RTM. VR cylindrical coordinate system, then a file is
created using the QuickTime.RTM. VR file format at step 1135.
[0093] Once the coordinate transform is complete, the transformed
image can be view using Apple's QTVR player program as stated in
step 1140.
[0094] Still Image Presentation on a Computer Network
[0095] Since the present invention can store the annular image in
digital form, a very useful method of distributing panoramic images
is through a computer network. In particular, the hypertext
transport protocol (http) of the World Wide Web (WWW) on the
Internet can be used to distribute still annular images. The still
annular images would be stored on a World Wide Web server. To
access the still annular images, any user coupled to the Internet
would used a World Wide Web browser program.
[0096] One method of transporting the images would be to define a
new panoramic image annular data format. The images could then be
downloaded as stored in the panoramic image annular data format. A
helper application would then display the images once
downloaded.
[0097] A better method of displaying images using the hypertext
transport protocol (http) of the World Wide Web (WWW) would be to
implement a "plugin" application that would work with the browser
program. FIG. 12a illustrates a graphical user interface for one
possible client panoramic image presentations system. On the right
side of FIG. 12a, a set of different panoramic images to display is
available. To display one of those images, the user selects the
image with a cursor control device. On the upper left of the
graphical user interface of FIG. 12a is viewport for displaying a
portion of a panoramic image. Pan arrows on either side of the
viewport allow the user to pan left and right.
[0098] Image Presentation as Video
[0099] One of the most interesting presentation systems for the
present invention is a video presentation system. FIGS. 13a and 13b
illustrate one possible video presentation system.
[0100] Referring to FIG. 13a, a CCD version of the panoramic camera
system 1205 of the present invention is illustrated coupled to a
computer system 1200. The CCD version of the panoramic camera
system 1205 is coupled through a panoramic camera interface 1210.
The panoramic camera interface 1210 receives a digital stream of
annular images. To interface with computer systems, one embodiment
of the panoramic camera interface 1210 is the FireWire system that
is described in the IEEE 1394 standard.
[0101] After being received through the panoramic camera interface
1210, the digitized annular images are stored in an Annular "Video"
Storage system 1230. The Annular "Video" comprises a series of a
consecutive annular images taken with a CCD version of the
panoramic camera system 1205.
[0102] To display the Annular Video as normal video, the annular
frames must be converted from the annular image format into normal
video images. In one embodiment of the present invention, only a
portion of the annular image is converted into normal video. One
reason for this is that the aspect ratio of video does not allow
for good viewing of wide but short rectangular panoramic images.
Furthermore, by only transforming a portion of the annular image
into normal video, the transformation can be done in real-time
without requiring exceedingly fast computer equipment. The
transformation of annular video to normal video is done by annular
to video conversion units 1240 and 1243.
[0103] To display the normal video, existing video streaming
software 1260 and 1263 can be used. For example, using a standard
transmission protocol like MPEG or a proprietary protocols such as
StreamWorks produced by Xing Technology Corporation of Arroyo
Grande, Calif. or VDOLive produced by VDOnet Corporation of Santa
Clara, Calif., the video can be provided to computer users coupled
to a network.
[0104] FIG. 13b illustrates one possible embodiment of a graphical
user interface (GUI) for accessing the annular video. In the GUI of
FIG. 13b, a video viewport 1340 is used to display the video. A
smaller still panoramic image 1310 is used to illustrate a static
version of the full panoramic video. A locator window 1315 is used
to identify the view angle that the video window 1340 is displaying
within the full panoramic view that is available.
[0105] To change the view angle, the user can select a pan right
arrow 1347 or a pan left arrow 1343 with a cursor 1320.
Alternatively, the user can simply move the position of the locator
window 1315 within the still panoramic image 1310. In the
embodiment of FIG. 13b, the entire vertical image aspect of the
image is compressed into the video viewport 1340, however in other
embodiments the,
[0106] Referring back to FIG. 13a, user input processing routines
1250 and 1253 processing the user's commands. When a the user
requests a viewpoint change, the new viewpoint is communicated to
the respective annular to video conversion units 1240 or 1243 such
that it will begin converting images from the new user viewpoint.
In an alternate embodiment, the user input processing routines are
place within the client program on the client computer system. For
example, in an embodiment of a WWW browser program, a plug-in
program can process the user commands and simply pass the location
of the video viewport to the server.
[0107] Referring back to FIG. 13b, a parameter window 1350 is also
available to the viewer. The parameter window 1350 allows the user
to adjust some of the viewing parameters such as Image Brightness
1352, Image Tint 1353 and Image Contrast 1355. When a user adjusts
these parameters, the changes will be processed by the user input
processing routines 1250 and 1253 provided to the annular to video
conversion units 1240 or 1243 or the video streaming software 1260
or 1263 such that video quality is changed.
Telepresence: Video and Audio
[0108] To more completely convey the experience of being at a
different location, the present invention can be combined with a
three dimensional sound system. Referring to FIGS. 14a and 14b, an
embodiment of the camera system is illustrated with four
directional microphones 1441, 1442, 1443, and 1444. The four
directional microphones 1441, 1442, 1443, and 1444 capture sound
emanating from four cardinal directions.
[0109] To add three dimensional sound, the sound from the various
directional microphones is mixed depending on the viewing angle
that a user has selected. For example, if a viewer that is seeing a
real-time image from camera 1400 of FIG. 14a is viewing straight
out of the page, then the left speaker will receive information
from microphone 1443 and the right speaker will receive information
from microphone 1441. The sound can be provided to users on a
computer network using audio streaming software such as RealAudio
by Progressive Networks, Inc. As the viewer adjusts the viewing
angle, the sound from the directional microphones will be adjusted
accordingly.
[0110] By adding sound to the system, the user is provided with a
cues as to which direction they should be viewing. For example, if
the user hears a sound from "behind", then the user can change the
view angle to look backward.
[0111] The foregoing has described a camera device that captures
360 degree panoramic images and presentation systems for displaying
such images. It is contemplated that changes and modifications may
be made by one of ordinary skill in the art, to the materials and
arrangements of elements of the present invention without departing
from the scope of the invention.
* * * * *