U.S. patent application number 14/593495 was filed with the patent office on 2015-04-30 for critical alignment of parallax images for autostereoscopic display.
The applicant listed for this patent is VISION III IMAGING, INC.. Invention is credited to Michael Burgess MARTIN, Christopher Alan MAYHEW.
Application Number | 20150116466 14/593495 |
Document ID | / |
Family ID | 32393334 |
Filed Date | 2015-04-30 |
United States Patent
Application |
20150116466 |
Kind Code |
A1 |
MAYHEW; Christopher Alan ;
et al. |
April 30, 2015 |
CRITICAL ALIGNMENT OF PARALLAX IMAGES FOR AUTOSTEREOSCOPIC
DISPLAY
Abstract
A method is provided for generating an autostereoscopic display.
The method includes acquiring a first parallax image and at least
one other parallax image. At least a portion of the first parallax
image may be aligned with a corresponding portion of the at least
one other parallax image. Alternating views of the first parallax
image and the at least one other parallax image may be
displayed.
Inventors: |
MAYHEW; Christopher Alan;
(Oakton, VA) ; MARTIN; Michael Burgess;
(Germantown, MD) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
VISION III IMAGING, INC. |
Reston |
VA |
US |
|
|
Family ID: |
32393334 |
Appl. No.: |
14/593495 |
Filed: |
January 9, 2015 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
13298824 |
Nov 17, 2011 |
8953015 |
|
|
14593495 |
|
|
|
|
10536005 |
May 20, 2005 |
8081206 |
|
|
PCT/US2003/037203 |
Nov 20, 2003 |
|
|
|
13298824 |
|
|
|
|
60427961 |
Nov 21, 2002 |
|
|
|
Current U.S.
Class: |
348/51 |
Current CPC
Class: |
H04N 13/282 20180501;
H04N 13/271 20180501; H04N 13/302 20180501; H04N 13/122 20180501;
H04N 13/261 20180501; H04N 13/327 20180501; H04N 13/296 20180501;
G02B 30/27 20200101; H04N 13/20 20180501; H04N 13/133 20180501;
H04N 13/139 20180501 |
Class at
Publication: |
348/51 |
International
Class: |
H04N 13/04 20060101
H04N013/04 |
Claims
1. A method of generating an autostereoscopic display, comprising:
acquiring a first parallax image and at least one other parallax
image; aligning at least a portion of the first parallax image with
a corresponding portion of the at least one other parallax image;
and displaying alternating views of the first parallax image and
the at least one other parallax image.
2. The method of claim 1, wherein the aligning step is performed
during a time period over which the displaying step is also
performed.
3. The method of claim 1, wherein the aligning step includes
applying a transformation to at least one of the first parallax
image and the at least one other parallax image, the transformation
including at least one of translation, rotation, and scaling.
4. The method of claim 3, wherein the transformation is performed
with sub-pixel resolution.
5. The method of claim 1, further comprising storing alignment
parameters associated with at least one of the first parallax image
and the at least one other parallax image, and applying the
alignment parameters to the at least one of the first parallax
image and the at least one other parallax image.
6. The method of claim 1, wherein the displaying step includes
displaying alternating views of the first parallax image and the at
least one other parallax image in a randomly selected order.
7. The method of claim 1, wherein the alternating views are
displayed on at least one of a computer monitor, a television, a
projection screen, and a moving image display.
8. The method of claim 1, wherein the first parallax image and the
at least one other image were captured by two different cameras
viewing a common scene.
9. The method of claim 1, wherein the first parallax image and the
at least one other image were captured from a video image
stream.
10. The method of claim 1, wherein the first parallax image and the
at least one other image were captured using a single camera whose
position was moved between capturing the first parallax image and
the at least one other image.
11. The method of claim 1, wherein the first parallax image and the
at least one other image were generated by a computer.
12. A system for generating a set of aligned parallax images,
comprising: a computer; and an application running on the computer,
the application configured to display alternating views of two or
more parallax images at a desired viewing rate and manipulate the
two or more parallax images such that at least a portion of a first
one of the parallax images is aligned with at least a portion of a
second one of the parallax images.
13. The system of claim 12, wherein the application is further
configured to apply a set of transformations to at least one of the
two or more parallax images.
14. The system of claim 12, wherein the set of transformations
includes at least one of translation, rotation, and scaling.
15. The system of claim 12, wherein the application is further
configured to accept an input from an operator designating one of
the two or more parallax images as a reference image and to accept
transformation parameters from the operator that affect the
manipulation of the two or more parallax images.
16. The system of claim 12, wherein the application is further
configured to accept inputs from an operator identifying
convergence points in the two or more parallax images and to
calculate transformation parameters for performing the manipulation
of the two or more parallax images.
17. The system of claim 12, wherein the application is further
configured to perform pattern matching to determine whether any
significant rotational disparities exist among the two or more
parallax images, a degree of the rotational disparities, a point of
rotation, and rotational translations needed to correct for the
rotational disparities.
18. The system of claim 12, wherein the application is further
configured to measure an amount of apparent shift associated with a
point appearing in each of the two or more parallax images and to
calculate a quantitative position value for the point.
19. The system of claim 12, wherein the application is further
configured to compute a depth map for objects appearing in the at
least two parallax images.
20. A method of generating an autostereoscopic display, comprising:
capturing a first image at a first point of view with an image
capture device; using the image capture device to capture at least
one other image from a second point of view different from the
first point of view; displaying alternating views of the first
image and the at least one other image at a desired viewing rate;
and generating a set of aligned images by manipulating at least one
of the first image and the at least one other image such that at
least a portion of the first image is aligned with at least a
portion of the at least one other image.
21-24. (canceled)
Description
I. FIELD OF THE INVENTION
[0001] The present invention relates to the visual arts field and
more particularly to autostereoscopic imaging methods for producing
two-dimensional images that, upon display, can be perceived to be
three-dimensional without the use of special viewing aids.
II. BACKGROUND
[0002] The production of two-dimensional images that can be
displayed to provide a three-dimensional illusion has been a
long-standing goal in the visual arts field. Methods and apparatus
for producing such three-dimensional illusions have to some extent
paralleled the increased understanding of the physiology of human
depth perception, as well as, developments in image manipulation
through analog/digital signal processing and computer imaging
software.
[0003] Binocular (i.e., stereo) vision requires two eyes that look
in the same direction, with overlapping visual fields. Each eye
views a scene from a slightly different angle and focuses it onto
the retina, a concave surface at the back of the eye lined with
nerve cells, or neurons. The two-dimensional retinal images from
each eye are transmitted along the optic nerves to the brain's
visual cortex, where they are combined, in a process known as
stereopsis, to form a perceived three-dimensional model of the
scene.
[0004] Perception of three-dimensional space depends on various
kinds of information in the scene being viewed including monocular
cues and binocular cues, for example. Monocular cues, include
elements such as relative size, linear perspective, interposition,
light, and shadow. Binocular cues include retinal disparity,
accommodation, convergence, and learned cues (e.g., familiarity
with the subject matter). While all these factors may contribute to
creating a perception of three-dimensional space in a scene,
retinal disparity may provide one of the most important sources of
information for creating the three-dimensional perception.
Particularly, retinal disparity results in parallax information
(i.e., an apparent change in the position, direction of motion, or
other visual characteristics of an object caused by different
observational positions) being supplied to the brain. Because each
eye has a different observational position, each eye can provide a
slightly different view of the same scene. The differences between
the views represents parallax information that the brain can use to
perceive three dimensional aspects of a scene.
[0005] Parallax information does not have to be presented to the
brain simultaneously. For example, left and right eye depth
information can be presented alternately to the left and right
eyes, resulting in depth perception as long as the time interval
does not exceed 100 msec. The brain can extract parallax
information from a three-dimensional scene even when the eyes are
alternately covered and uncovered for periods of up to 100 msec
each. The brain can also accept and process parallax information
presented to both eyes simultaneously if the parallax information
is sequenced. For example. two or more views of the same scene
taken from different observational viewpoints may be shown to both
eyes in a sequence (e.g., each one of the views may be shown to
both eyes for a short amount of time before showing the next view
in the sequence).
[0006] Several three-dimensional image display methods have been
proposed and/or implemented. These methods may be divided into two
main categories of stereoscopic display methods and
autostereoscopic display methods. Stereoscopic techniques including
stereoscopes, polarization, anaglyphic, Pulfrich, and shuttering
technologies require the viewer to wear a special viewing apparatus
such as glasses, for example. Autostereoscopic techniques such as
holography, lenticular screens, and parallax barriers produce
images with a three-dimensional illusion without the use of special
glasses, but these methods generally require the use of a special
screen.
[0007] Other systems have been proposed, however, that require
neither special glasses nor special viewing screens. These systems
include autostereoscopic television and motion picture systems that
utilize alternately displayed views of a scene recorded by two
cameras from different points of view. For example, the devices
described in U.S. Pat. No. 4,006,291 to Imsand; U.S. Pat. No.
4,303,316 to McElveen; U.S. Pat. No. 4,429,328 to Jones et al.; and
U.S. Pat. No. 4,815, 819 to Mayhew et al., all utilize two
carefully aligned cameras to record horizontally, vertically, or a
combination of horizontally and vertically displaced views of a
scene. While these systems deal mainly with techniques of image
acquisition for autostereoscopic display using standard screens.
the cameras must be carefully matched and aligned to capture
appropriate images. Further, once the images from the cameras have
been captured, the alignment of the images cannot be
readjusted.
[0008] In yet another approach, U.S. Pat. No. 5,510,831 issued to
Mayhew describes a method of autostereoscopic display of parallax
images using a slit scanning technique. In this technique, two
cameras are carefully aligned to capture stereoscopic images. These
images may be displayed by providing a first image as a background
image and overlaying a second image onto the first image in the
form of a scanning slit.
[0009] While each of these described methods and systems can be
used to capture images for three-dimensional image display, there
are problems associated with each. For example, many of the methods
require the use of at least two carefully aligned cameras to
capture images having parallax information. Aligning multiple
cameras at a common scene is cumbersome. Not only are there
multiple cameras to carry and to position, but proper alignment and
color/luminance matching of the cameras can be difficult. Even
after alignment, the cameras still may not provide a desired degree
of image alignment for later display. Further, many of the prior
art methods require special camera or lens mechanisms, video
switching equipment, special viewing glasses, and/or special
screens to create the three-dimensional illusion. Also, none of
these three-dimensional display methods are suitable for use with
randomly acquired images or with images extracted from a
conventional video image stream (e.g., sequence) or images with
parallel views, for example.
[0010] The present invention is directed to overcoming one or more
of the problems associated with the prior art three-dimensional
image display systems and methods.
SUMMARY OF THE INVENTION
[0011] A first aspect of the invention includes a method for
generating an autostereoscopic display. The method includes
acquiring a first parallax image and at least one other parallax
image. A portion of the first parallax image may be aligned with a
corresponding portion of the at least one other parallax image.
Alternating views of the first parallax image and the at least one
other parallax image may be displayed.
[0012] A second aspect of the invention includes a system for
generating a set of aligned parallax images. The system includes a
computer and an application running on the computer. The
application is configured to display alternating views of two or
more parallax images at a desired viewing rate and to manipulate
the two or more parallax images such that at least a portion of
first one of the parallax images is aligned with at least a portion
of a second one of the parallax images.
BRIEF DESCRIPTION OF THE DRAWINGS
[0013] FIG. 1 illustrates exemplary camera positions for generating
parallax images in accordance with an exemplary embodiment of the
invention;
[0014] FIG. 2 provides a flowchart representing a method for
critically aligning parallax images in accordance with an exemplary
embodiment of the invention;
[0015] FIGS. 3a-3d illustrate a transformation process for aligning
parallax images in accordance with an exemplary embodiment of the
invention; and
[0016] FIGS. 4a-4d illustrate various sequence patterns for display
of parallax images during and after alignment in accordance with an
exemplary embodiment of the invention.
DETAILED DESCRIPTION
[0017] One exemplary embodiment of the present invention includes a
method for creating an autostereoscopic display by manipulating
parallax images to create a resultant moving image. The resultant
moving image may provide an autostereoscopic display and may be
viewed on a conventional screen (e.g., a TV, computer monitor, a
projection screen, moving image display, or any other type of
display on which a moving image may be shown) As discussed above,
parallax images include two or more images with overlapping visual
fields but different points of view. For example, as illustrated in
FIG. 1, a camera 10 may capture a first set of images and a camera
12 may capture a second set of images of a common scene 14 while
being displaced from one another. The resulting sets of images from
cameras 10 and 12 will be parallax images. That is, the set of
images from camera 10 and the set of images from camera 12 will
include some duplicated visual information by virtue of the fact
that cameras 10 and 12 capture images of the same scene 14. The
sets of images from cameras 10 and 12, however, will also include
some different visual information resulting from the different
points of view. These parallax images may serve as a basis for
generating an autostereoscopic display consistent with the present
invention.
[0018] It should be noted that cameras 10 and 12 may capture
parallax images simultaneously or alternatingly. Parallax images
may even be generated by a single camera 10 that captures a first
image of scene 14 before moving to a new position (e.g., the
position of camera 12 in FIG. 1) and capturing a second image of
scene 14. Further, any length of time may pass between capturing
parallax images of scene 14. For example, after capturing a first
image of scene 14, a second image from a different point of view
may be captured at any later time (1 second, 1 hour, 1 day, 1 year,
etc.). Additionally, cameras 10 and 12 need not be in any special
alignment configuration to produce suitable parallax images for use
with the present invention.
[0019] An exemplary method of the present invention may involve the
steps of acquisition and selection of source images, critical
alignment of the images, and display of the images. In one
embodiment, as illustrated in FIG. 2, the method may include
acquiring source images at step 20, loading source images into
alignment software at step 22, adjusting alignment parameters at
step 24, saving/storing aligned images at step 26, and viewing
aligned images at step 28.
Acquisition and Selection
[0020] The parallax images used to generate the autostereoscopic
display may be acquired from a variety of imaging sources such as
digital still cameras, digital video cameras, conventional film
cameras and conventional video cameras (followed by subsequent
digitization), computer generated graphics sources, and any other
suitable imaging source. Additionally, the parallax images may be
taken from a single image stream or from multiple image streams.
Multiple image streams could be the output of a video stereo camera
pair, or more generally, any two or more image sources with
overlapping views of the same scene, including overlapping image
sequences with parallel points of view. The parallax images may
also be generated by a computer (as with 3D rendered graphics) or
false-color images produced by RADAR, SONAR, etc.
Critical Alignment
[0021] The alignment process includes displaying alternating views
of parallax images, at a desired viewing rate (i.e., a frequency at
which the parallax image views are changed), and then manipulating
the alternating views to match alignment. While the alternating
views may be displayed at any desired viewing rate, in one
embodiment, the viewing rate may be from about 3 Hz to about 6 Hz.
The term "match alignment" refers to a condition in which a region
of interest in an image to be aligned (i.e., converged) is
positioned such that it occupies the same location within the frame
of the image to be aligned as the corresponding region in a
reference image frame. The region of interest may be all or part of
the image to be aligned.
[0022] The alignment matching process begins by selecting a
reference image 30, as shown in FIG. 3a, from a set of parallax
images. Once reference image 30 has been selected, other images 32,
as shown in FIG. 3b, from the parallax image set can be aligned to
reference image 30. While only a single unaligned image 32 is shown
in FIG. 3b, unaligned image 32 may represent a plurality of N
images. One or more of the plurality of N images may be selected
and aligned with respect to reference image 30. In certain
situations, the stability of an autostereoscopic display consistent
with the present invention may increase as the number of parallax
images with differing parallax positions increases.
[0023] Reference image 30 may include a region of interest 34. The
same region of interest 34', albeit as viewed from a different
point of view, may appear in unaligned image 32. Unaligned image 32
may be manipulated, as shown in FIG. 3c, for example, until region
34' matches alignment with region 34, as illustrated in FIG. 3d.
The manipulation process may be represented by an affine
transformation including translation, rotation, scaling, and/or any
other desired transformation. In addition, the point about which
unaligned image 32 is rotated can also be adjusted to a position
other than the center of the image.
[0024] The critical alignment process may be performed by a
computer. For example, a set of parallax images may be loaded into
a software application that enables a user to select a reference
image. For example the set of parallax images may be loaded into
open graphics language (OGL) software or other software suitable
for manipulating image data. The computer may then automatically
perform alignment of one or more of the remaining parallax images
in the set. Alternatively, however, the software may enable an
operator to input transformation parameters for one or more of the
remaining parallax images in the set.
[0025] In one exemplary embodiment, a user may select a convergence
point in the reference image and in one or more of the unaligned
images. A computer can perform appropriate translation(s) to align
the convergence points in the images based on calculated
differences between the selected convergence points in the images.
The computer may further perform pattern matching or feature
extraction algorithms to determine, (a) whether any significant
rotational disparities exist among two or more selected images, (b)
the degree of the rotational disparities, (c) a point or rotation
about which one or more of the selected images can be rotated, and
(d) what rotational translation(s) would be required to match
alignment of regions of interest in the selected images at or near
the selected convergence points. Thus, the computer may align the
images based on the convergence points selected and rotate the
images to match alignment.
[0026] In another embodiment, the computer may control an even
greater portion of the alignment process. For example, either an
operator or the computer may select a convergence point in
reference image 30. Next, the computer may use pattern-matching
algorithms to compute an estimate for a matching region in
unaligned image 32 that corresponds to the region around the
convergence point in reference image 30. Any appropriate pattern
matching algorithm known in the art may be used to perform this
calculation. For example, a block of pixels from each of images 30
and 32 may be chosen and compared for similarity. This process may
be repeated until a best match is chosen. Repetition of this
process with increasingly smaller displacements may be performed to
refine the translation value (e.g., to provide transformation
parameters of sub-pixel resolution). Rotation may also be handled,
as described above.
[0027] In yet another embodiment, the computer may enable an
operator to input transformation parameters for one or more
parallax images. Thus, for each image to be aligned, a user may
manually enter and vary transformation parameters to align the
parallax images. The alignment software may include, for example, a
graphical user interface (GUI) where the user may enter
transformation parameters such as translation parameters, scaling
parameters, rotation values, a rotational pivot point, and any
other parameters associated with image transformations. Additional
features may include alignment guides to assist in qualitatively
identifying matching areas, the ability to zoom in/out, and the
ability to mask off (i.e., obscure) parts of an image outside the
region of interest.
[0028] Regardless of the degree of automation, the transformation
parameters in each process may be continuously adjusted until
critical alignment is achieved. Critical alignment corresponds to a
condition where the degree of alignment is sufficient to achieve a
stable autostereoscopic display. Stability of the whole image may
not be required, as long as at least a particular region of
interest in the autostereoscopic display is stable.
[0029] One of the key elements of the disclosed alignment process
is the use of parallax image manipulations Of sub-pixel resolution
to achieve critical alignment. Specifically, the transformations
for achieving critical alignment may proceed to a sub-pixel level
where one image is moved with respect to another image by an amount
less than an integral number of pixels. That is, the
transformations may include displacements of an integral number of
pixels plus or minus any fraction of one pixel dimension. These
sub-pixel manipulations may help to maximize the stability of the
autostereoscopic display. To achieve sub-pixel alignment, image
interpolation methods such as bicubic rescaling, bilinear
rescaling, or any other appropriate image interpolation method may
be employed.
Display
[0030] The parallax images, and alternating views thereof, may be
displayed before, during, or after critical alignment of the
parallax images. Displaying alternating views of the parallax
images during the critical alignment process may aid in determining
when one or more images match alignment with a reference image. For
example, as the alternating views of the parallax images are
displayed, a user may intermittently enter transformation
parameters, as described above, to align two or more parallax
images. One advantage of displaying the parallax images during the
alignment process is that the user may see, in real time, the
effect that the entered transformation parameters have on the
alignment of the images. In this way, a user may progress
incrementally toward a match alignment condition by entering
transformation parameters. observing the alignment condition of the
parallax images, and reentering transformation parameters to refine
the alignment condition of the parallax images.
[0031] Once the parallax images have been aligned, the aligned
images may be stored as a set of image data. Storing image data in
this manner may be useful for displaying the aligned parallax
images in a stand-alone operation after alignment has been
completed. For example, the aligned parallax images may be stored
and later displayed in a video format. Further, the stored, aligned
parallax images may be reloaded into the alignment software for
viewing or further processing, including, for example, aligning the
images with respect to a new region of interest.
[0032] Alternatively, a record of the transformations used to align
the images (i.e., image alignment parameters) may be stored. In a
later process, the stored transformations may be retrieved and
reapplied to the set of parallax images to regenerate the match
alignment condition of the images. In one embodiment, the image
alignment parameters may be stored and used to align higher
resolution versions of the same images. This process may be useful,
for example, to speed processing of high resolution images. Rather
than performing the critical alignment process on high resolution
images, which may require significant processing resources and may
slow or prevent real-time manipulation of the images, the
manipulations may be performed on low resolution versions of the
high resolution images. Then the alignment parameters determined
for the low resolution images may be applied to the higher
resolution versions of the images.
[0033] Unlike stereoscopic displays, the autostereoscopic images
consistent with the invention can be viewed as a sequence of images
on conventional two-dimensional displays (e.g., screens), such as a
television, computer monitor, a projection screen, moving image
display, or any other type of display on which a moving image may
be displayed. A set of aligned images can be displayed in
sequential order, a randomly selected order, or any other desired
order. For example, FIG. 4a represents a set of six parallax images
(e.g., three right-left pairs) in matched alignment. FIG. 4b
illustrates a sequential playback pattern in which the aligned
parallax images in the set are displayed serially in a repeating
sequence. FIGS. 4c and 4d demonstrate two possible random playback
sequences. As noted above, the frequency with which the views in
the sequence are changed (i.e., the viewing rate) may be any
desired frequency. In one embodiment, however, the viewing rate may
be between about 3 Hz and about 6 Hz. Furthermore, the viewing rate
need not be constant, but may be varied over time.
Analysis
[0034] In addition to or instead of displaying the aligned parallax
images, computational analysis may be performed on the images. For
example, certain quantitative information may be extracted from the
aligned parallax images. As a result of the parallax information
contained in the images, an apparent shift of an object may exist
between different views. The apparent shift refers to the distance
a point in an image appears to move between images taken from
different points of view. By measuring the amount of apparent shift
of a point in two or more parallax images, quantitative position
values may be computed for the point in relation to objects in the
image if certain other information, such as the distance between
the camera and a point in the image, is known. For example, by
knowing the distance between the camera and the ground in an image
captured from the air, and by measuring the apparent shift of the
top edge of a building between two or more parallax images, the
height and/or volume of the building may be calculated.
[0035] Additionally, quantitative positional information for scene
points may be calculated based on known quantities appearing in the
image. For example, if a certain model of automobile appears in the
image and dimensional data is available for that automobile, then
positional values may be calculated for other scene points by
measuring the apparent shift of one or more points in the scene
associated with the automobile.
[0036] Further, by determining position values for enough scene
points in an image, a depth map for objects in the scene can be
computed. This depth map can be used to create views corresponding
to intermediate parallax angles. This allows for interpolation of
views from the originally captured images.
* * * * *