U.S. patent application number 11/835188 was filed with the patent office on 2008-04-10 for methods and systems for 2d/3d image conversion and optimization.
Invention is credited to William Hopewell, Rolf-Dieter Naske, Steven M. Schwartz.
Application Number | 20080085049 11/835188 |
Document ID | / |
Family ID | 46281827 |
Filed Date | 2008-04-10 |
United States Patent
Application |
20080085049 |
Kind Code |
A1 |
Naske; Rolf-Dieter ; et
al. |
April 10, 2008 |
METHODS AND SYSTEMS FOR 2D/3D IMAGE CONVERSION AND OPTIMIZATION
Abstract
A method that improves compensation for hard cuts and/or
vertical motion effects in the conversion of 2D images to 3D
images, displays 2D images to one eye and computed images to the
other eye. Generating the computed images includes using an
approximation variable that represents some relationship between
the 2D images, such as the speed of motion across two images.
Compensating for the hard cut or vertical motion effect includes
temporarily setting the approximation variable to a value that
makes the left and right eye images more similar (if identical,
displaying a 2D image). The approximation variable can be pre-set
to a value based on the speed of motion in a scene to be displayed.
A method of displaying images in either 2D or 3D may include
converting 3D images to 2D images by displaying images computed
from images intended for either the left eye or the right eye.
Inventors: |
Naske; Rolf-Dieter;
(Kakenstorf, DE) ; Schwartz; Steven M.; (Brooklyn,
NY) ; Hopewell; William; (New Milford, CT) |
Correspondence
Address: |
RENNER OTTO BOISSELLE & SKLAR, LLP
1621 EUCLID AVENUE
NINETEENTH FLOOR
CLEVELAND
OH
44115
US
|
Family ID: |
46281827 |
Appl. No.: |
11/835188 |
Filed: |
August 7, 2007 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
10338778 |
Jan 8, 2003 |
7254265 |
|
|
11835188 |
Aug 7, 2007 |
|
|
|
10240556 |
Oct 1, 2002 |
7254264 |
|
|
PCT/EP01/03707 |
Apr 1, 2001 |
|
|
|
10338778 |
Jan 8, 2003 |
|
|
|
Current U.S.
Class: |
382/154 ;
348/E13.008; 348/E13.02 |
Current CPC
Class: |
H04N 13/359 20180501;
H04N 13/221 20180501; H04N 13/261 20180501; H04N 13/128 20180501;
H04N 13/211 20180501 |
Class at
Publication: |
382/154 |
International
Class: |
G06K 9/00 20060101
G06K009/00 |
Foreign Application Data
Date |
Code |
Application Number |
Apr 1, 2000 |
DE |
10016074.3 |
Claims
1-63. (canceled)
64. A method for preparing 3D image data, comprising the steps of:
generating 3D image data for display from a first sequence of 2D
images, including the step of computing a second sequence of images
from at least two images of the first sequence using an
approximation variable that changes based on a relationship between
at least two images of the first sequence; comparing at least two
images of the first sequence to detect rapid change; and setting
the approximation variable to a value that provides a low non-zero
degree of parallax between respective images of the first and
second sequences of images in response to the detection of a rapid
change in images of the first sequence.
65. The method of claim 64, wherein the setting step includes
resetting the approximation variable step-wise over a predetermined
time.
66. The method of claim 65, wherein the resetting step adjusts the
approximation variable over a number of images based on the
magnitude of the detected change.
67. The method of claim 64, further comprising the step of
regulating the amount of each change of the approximation variable
in the computation of successive images of the second sequence.
68. The method of claim 67, further comprising, prior to the
setting step, the step of storing the value of the approximation
variable in a memory.
69. The method of claim 68, comprising the step of repeating the
regulating step until the approximation variable returns to the
value of the approximation variable just prior to the setting
step.
70. The method of claim 64, further comprising, promptly after the
setting step, the step of allowing the approximation variable to
change based on a relationship between at least two images of the
first sequence.
71. The method of claim 70, further comprising the step of
regulating the amount of each change of the approximation variable
in the computation of successive images of the second sequence
until the approximation variable reaches a target value.
72. The method of claim 64, wherein the setting step includes
temporarily setting the approximation variable to zero.
73. The method of claim 72, further comprising, after the resetting
step, the step of allowing the approximation variable to change
based on a relationship between at least two images of the first
sequence.
74. The method of claim 72, further comprising, prior to the
setting step, the step of storing the value of the approximation
variable in a memory.
75. The method of claim 64, wherein the comparing step includes
detecting at least one of a hard cut and a vertical disparity.
76. The method of claim 75, wherein the setting step includes the
step of reducing the value of the approximation variable.
77. The method of claim 76, wherein the reducing step includes
reducing the approximation variable upon the vertical disparity
exceeding a predetermined threshold.
78. The method of claim 76, wherein the reducing step includes the
step of reducing the approximation variable upon the occurrence of
a vertical disparity over a number of images of the first sequence,
wherein the number of images exceeds two.
79. The method of claim 76, further comprising, promptly after the
reducing step, allowing the approximation variable to change based
on a relationship between at least two images of the first
sequence.
80. The method of claim 76, further comprising, prior to the
setting step, the step of storing the value of the approximation
variable in a memory.
81. The method of claim 76, further comprising the step of
regulating the amount of each change of the approximation variable
in the computation of successive images of the second sequence
until the approximation variable returns to the value of the
approximation variable just prior to the reducing step.
82. The method of claim 76, comprising the steps of determining the
amount of vertical disparity and using the amount of vertical
disparity to determine the value of the reduction of the
approximation variable.
83. An apparatus for preparing 3D image data, comprising means for
generating 3D image data for display from a first sequence of
images and a second sequence of images computed from at least two
respective images of the first sequence by using an approximation
variable that changes based on a relationship between at least two
images of the first sequence, means responsive to occurrence of a
rapid change in images of the first sequence for setting the
approximation variable to a value that provides a low non-zero
degree of parallax between the respective images of the first and
second sequences of images.
Description
[0001] This application incorporates by this reference the entire
disclosures of pending U.S. patent application Ser. No. 10/240,556,
filed Oct. 1, 2002, and its predecessors, namely, PCT Patent
Application No. PCT/EP01/03707, filed Apr. 1, 2001, and German
Patent Application No. 10016074.3, filed Apr. 1, 2000.
[0002] This patent application also incorporates by this reference
the entire disclosure of PCT Patent Application No. PCT/EP01/13674,
filed Nov. 24, 2001.
FIELD OF THE INVENTION
[0003] The invention relates to methods and devices for the
generation of three-dimensional (referred to as "3D" from this
point forward) images, for the generation of 3D images from
two-dimensional (referred to as "2D" from this point forward)
images, for the generation of 2D images from 3D images, and for
improvements to such image generation.
BACKGROUND AND SUMMARY OF THE INVENTION
[0004] Three-dimensional imaging is often used to analyze objects,
particularly in the fields of medicine and science. Various methods
with which television pictures in particular can be produced in
three dimensions have also been developed for general consumer
applications. Three-dimensional images also are used for
entertainment, such as television programs, movies, videos, video
games, etc.
[0005] Among the methods of 3D imaging, there is a basic
distinction between sequential image transmission, in which the
images for the right eye and the left eye are transmitted
alternately one after the other or saved to a storage medium, and
parallel transmission, in which the images are transmitted on two
separate channels.
[0006] One particular disadvantage of sequential image transmission
in connection with conventional television systems is the fact that
the refresh rate is reduced to twenty-five images per second for
each eye. This creates an unpleasant flickering for the viewer. Of
course, this limitation does not occur when the image sequences are
each transmitted on their own channel (left or right). However,
problems may still arise with synchronizing both channels, as well
as with the requirements placed on the receiver, which must be able
to receive and process two channels simultaneously. This is not
possible for most systems generally available on the market.
[0007] Signal transmission and processing likely will be entirely
digital in future television systems. In such systems, every image
is broken down into individual pixels which are transmitted in
digitized format. In order to reduce the bandwidth required for
this process, the appropriate compression methods are used;
however, these create problems for stereo transmission.
[0008] For example, using block coding methods with a reasonable
rate of compression, generally it is not possible to reconstruct
every individual line of an image precisely. In addition,
interframe coding techniques, such as MPEG-2, do not allow
transmitting or saving stereo images in a sequential image format,
because image information from one image still is contained in
another image, creating the so-called "crosstalk effect", which
makes clear separation of the right image from the left image
difficult or impossible.
[0009] Other methods for generating a 3D image sequence from a 2DI
image sequence are disclosed in DE 35 30 610 and EP 0 665697. An
autostereoscopic system with an interpolation of images is
disclosed in EP 0 520 179, whereas in "Huang: Image Sequence
Analysis" (published in Springer Verlag) problems of the
recognition of motion areas in image sequences are discussed.
[0010] Therefore, one problem addressed by the invention is the
creation of a method and a device with which it is possible to
generate images that provide a very natural 3D image impression
even if using the above-mentioned transmission and/or compression
methods.
[0011] Several examples are mentioned in which techniques have been
used to convert 2D images to 3D images. One conversion technique
according to an aspect of the invention is described in detail
below. Two-dimension to 3D conversion is advantageous when a given
television program, movie, video, video game, or other sequence of
images, etc. (hereinbelow collectively referred to as "video" for
brevity) has been prepared in a 2D format. Therefore, if a viewer
(person) were to obtain a 2D video, e.g., on a DVD (digital video
disk), compact disk (CD), video tape, or some other storage medium
(hereinbelow collectively referred to as "DVD" for brevity), the
individual may be able to view the video in 3D format using a 2D to
3D conversion system.
[0012] Sometimes a video may be prepared in both 2D and 3D formats
and stored in a DVD for subsequent display in the selected 2D or 3D
format, as the viewer may select. From time to time a person
viewing a 3D format movie may decide to watch the same in 2D
format. This would be possible by switching from the 3D format
stored on the DVD to the 2D format stored on the DVD, but the
switching may take time because usually it is not possible to
switch from one scene in 3D format to the same scene in 2D format;
rather it would be necessary to restart the 2D format video and to
search for the scene where the 3D format of the video was stopped.
Meanwhile, the interest, emotion, suspense or intrigue created by
watching the video may be lost while taking the time to make the
format switch. Some videos are created and stored only in a 3D
format. If such were the case, a viewer would not be able to switch
between the 3D format and a 2D format. Accordingly, there is a need
to provide such ability to switch from a 3D format movie to a 2D
format. There also is a need to improve the versatility of such
display systems to allow the receiving of 2D format or 3D format
input image data/images and to provide outputs for display of the
images represented by the image data in either 2D or 3D format.
[0013] The term "hard cut" is a term frequently used in the video
field to indicate that an image of one scene very quickly switches
to another scene. For example, one scene may be of a person sitting
at a table while eating, and the next scene may show a view outside
the window of the room in which that person is sitting. The two
scenes are entirely different. It is possible that a portion of one
scene may be included in the other scene, but still there is a
substantial difference overall between the two scenes.
[0014] In the techniques used for converting 2D format images,
e.g., movies, etc., to 3D format images for display, the techniques
used may involve a determining of relationship between two images
and making the appropriate adjustments (compensation) to left or
right images, for example, so that the viewer perceives a 3D visual
effect. However, upon occurrence of a hard cut, the relationship
between two sequential scenes may be almost non-existent. Absent
such relation, it may be difficult, or impossible, to obtain a
conversion to 3D format or at least to a sequence of images that
may be comfortably viewed by a viewer. Thus, there is a need to
provide hard cut compensation when converting from a 2D format to a
3D format.
[0015] The occurrence of a hard cut may disrupt smooth conversion
of images from 2D format to 3D format. Such disruption may lead to
a less than optimum 3D viewing experience for the viewer.
Therefore, there is a need to detect occurrence of hard cuts and to
compensate or adjust for the same in a 2D to 3D conversion
system.
[0016] Vertical motion effect in images, e.g., movies, etc., that
are converted from 2D format to 3D format can be somewhat
disadvantageous. For example, most depth perception (3D or stereo
effect in viewing a scene, for example) is obtained in a generally
horizontal plane rather than in a vertical plane because the
viewer's eyes are spaced apart usually in a generally horizontal
plane and respective views are seen according to the stereo base of
the distance between the viewer's two eyes. Vertical motion or
disparity between a pair or more than a pair of sequential images
of a movie may tend to be construed by 2D to 3D conversion systems
as motion indicative of depth, which, of course, may be
incorrect.
[0017] Accordingly, viewing images with vertical motion in 3D
format may present a less than optimal 3D viewing experience for
the viewer and also may inaccurately represent the actual depth
information. Accordingly, there is a need to detect vertical motion
effect and to reduce the same or to eliminate it in systems and
methods for converting 2D format image data to 3D format image data
and associated images provided therefrom.
[0018] Some types of movies, television shows, videos, etc., are
more active than others, e.g., greater action, movement, scene
changes, etc. For example, a war movie usually would have much more
action than a television talk show. If the same 2D to 3D conversion
technique, variables, values, etc., were used to convert an action
movie as would be used to convert a television talk show from 2D
format to 3D format, substantial effect of the 3D images may be
lost during at least part of the program. For example, in an action
movie there is a great deal of motion and usually a great deal of
depth information that can be detected as a result of that motion;
therefore, if the conversion from 2D format to 3D format relied on
the motion, then substantial depth would be encountered. However,
if the same approached were used convert a talk show program from
2D format to 3D format, the depth or 3D effect may be much less
than in the action movie or, in any event, it may take much longer
for the depth or 3D effect to become evident to a viewer from the
3D format images of the talk show program. Accordingly, there is a
need to accommodate the different characteristics of the type of
program, e.g., action movie, television talk show program, wildlife
video, etc., when converting to 2D format representations thereof
to 3D format.
[0019] It is noted here that reference may be made to television,
video, movie, video game, etc., all of which are obtained by a
sequence of images. The sequence of images provide information
indicating the characteristic of a given scene at one moment in
time and the characteristic of that scene in another moment of
time. Reference herein to television, video, movie, video game,
etc., may be construed to include all of the same and other formats
of information conveyance, e.g., by a sequence of images that are
presented for viewing. The viewing may be via a television, a
computer monitor, a flat panel display, a projector, glasses or
goggles, such as, for example, those used in connection with
virtual reality type displays, game displays, etc., which provide
images directly to the eyes of a viewer, and so forth. Furthermore,
reference herein to images may also include image data, i.e.,
signals, data, etc., that may be stored, may be transmitted, or may
be created, synthesized, interpolated, or otherwise obtained to
represent the images. Thus, in the description below, reference to
an image may be construed to mean the image itself or the data
representing the image, whereby, if that data is appropriately
provided to a display device, so the image can be viewed by a
viewer (reference to viewer typically would mean a person who would
view the image). 2D means two-dimensional images, sometimes
referred to as mono, monoscopic, etc. type of images. 3D (sometimes
written as 3-D) means three-dimensional images or images that
appear to have not only height and width characteristic, but also
depth characteristics. Sometimes 3D is referred to as stereo,
stereoscopic, etc. type of images. These definitions (and others
presented herein) are not intended to be limiting; standard or
conventional definitions of the terms used herein also apply.
[0020] Another aspect relates to a method for the generation of 3D
images represented by 3D image data that includes at least two
channels of data for respective left and right eye views from a
sequence of 2D images represented by 2D image data, comprising the
steps of:
[0021] i) assigning the 2D image data to a first channel;
[0022] ii) determining a measure of similarity (.delta..sub.k)
between sets of 2D image data representative of sequential 2D
images;
[0023] iii) comparing the measure of similarity (.delta..sub.k) to
predetermined threshold values
(.delta..sub.0<.delta..sub.1<.delta..sub.2) where [0024] (1)
if .delta..sub.0<.delta..sub.k<.delta..sub.2 and
.delta..sub.k-.delta..sub.k-1.ltoreq.-.delta..sub.1 and an
approximation variable (.alpha.) is less than or equal to Ks, the
approximation variable is set to .alpha.:=.alpha.+s, [0025] (2) if
.delta..sub.0<.delta..sub.k<.delta..sub.2 and
.delta..sub.k-.delta..sub.k-1.gtoreq..delta..sub.1 and
.alpha..gtoreq.s, the approximation variable (.alpha.) is set to
.alpha.:=.alpha.-s, and [0026] (3) if
.delta..sub.0.gtoreq..delta..sub.k, the approximation variable
(.alpha.) is set to .alpha.:=0, where the approximation variable
(.alpha.) determines the width of a stereo base, and s denotes a
step width;
[0027] iv) calculating 3D image data for a second channel from the
2D image data using the approximation variable (.alpha.); and
[0028] v) assigning the first channel and the second channel to
respective left and right eye views for display.
[0029] Another aspect relates to an apparatus for generating 3D
images represented by 3D image data that includes at least two
channels of data for respective left and right eye views from a
sequence of 2D images represented by 2D image data, comprising:
[0030] i) means for assigning the 2D image data to a first
channel;
[0031] ii) means for determining a measure of similarity
(.delta..sub.k) between sets of 2D image data representative of
sequential 2D images;
[0032] iii) means for comparing the measure of similarity
(.delta..sub.k) to predetermined threshold values
(.delta..sub.0<.delta..sub.1<.delta..sub.2) where [0033] (1)
if .delta..sub.0<.delta..sub.k<.delta..sub.2 and
.delta..sub.k-.delta..sub.k-1.ltoreq.-.delta..sub.1 and an
approximation variable (.alpha.) is less than or equal to Ks, the
approximation variable is set to .alpha.:=.alpha.+s, [0034] (2) if
.delta..sub.0<.delta..sub.k<.delta..sub.2 and
.delta..sub.k-.delta..sub.k-1.gtoreq..delta..sub.1 and
.alpha..gtoreq.s, the approximation variable (.alpha.) is set to
.alpha.:=.alpha.-s, and [0035] (3) if
.delta..sub.0.ltoreq..delta..sub.k, the approximation variable
(.alpha.) is set to .alpha.:=0, where the approximation variable
(.alpha.) determines the width of a stereo base, and s denotes a
step width;
[0036] iv) means for calculating 3D image data for a second channel
from the 2D image data using the approximation variable (.alpha.);
and
[0037] v) means for assigning the first channel and the second
channel to respective left and right eye views for display.
[0038] Another aspect relates to a system for the generation of 3D
images represented by 3D image data that includes at least two
channels of data for respective left and right eye views from a
sequence of 2D images represented by 2D image data, comprising:
[0039] i) means for determining a measure of similarity
(.delta..sub.k) between sets of 2D image data representative of
sequential 2D images;
[0040] ii) means for comparing the measure of similarity
(.delta..sub.k) to predetermined threshold values
(.delta..sub.0<.delta..sub.1<.delta..sub.2) where [0041] (1)
if .delta..sub.0<.delta..sub.k<.delta..sub.2 and
.delta..sub.k-.delta..sub.k-1.ltoreq..delta..sub.1 and an
approximation variable (.alpha.) is less than or equal to Ks, the
approximation variable is set to .alpha.:=.alpha.+s, [0042] (2) if
.delta..sub.0<.delta..sub.k<.delta..sub.2 and
.delta..sub.k-.delta..sub.k-1.gtoreq..delta..sub.1 and
.alpha..gtoreq.s, the approximation variable (.alpha.) is set to
.alpha.:=.alpha.-s, and [0043] (3) if
.delta..sub.0.ltoreq..delta..sub.k, the approximation variable
(.alpha.) is set to .alpha.:=0, where the approximation variable
(.alpha.) determines the width of a stereo base, and s denotes a
step width;
[0044] iii) an image generator for calculating 3D image data for a
second channel from the 2D image data using the approximation
variable (.alpha.).
[0045] Another aspect relates to a method for generating 3D images,
comprising providing a first sequence of images for display,
computing a second sequence of images for display from one or more
images of the first sequence using an approximation variable
representative of a relationship between at least two images of the
first sequence, allowing the approximation variable to change in
response to a relationship between at least two images of the first
sequence, checking for occurrence of a hard cut between images of
the first sequence, and in response to occurrence of a hard cut,
setting the approximation variable to a value that provides a low
degree of parallax between respective images of the first and
second sequences of images or the appearance of a 2D image.
[0046] Another aspect relates to an apparatus for generating 3D
images, comprising means for providing a first sequence of images
for display, means for computing a second sequence of images for
display from one or more images of the first sequence using an
approximation variable representative of a relationship between at
least two images of the first sequence and allowing the
approximation variable to change in response to a relationship
between at least two images of the first sequence, means for
checking for occurrence of a hard cut between images of the first
sequence, and means operative in response to occurrence of a hard
cut for setting the approximation variable to a value that provides
a low degree of parallax between respective images of the first and
second sequences of images or the appearance of a 2D image.
[0047] Another aspect relates to a method of compensating 3D image
data in response to hard cut detection, comprising preparing 3D
image data for display from a first sequence of images and a second
sequence of images computed from at least two respective images of
the first sequence by using an approximation variable that changes
based on a relationship between at least two images of the first
sequence, in response to occurrence of a hard cut in images of the
first sequence, setting the approximation variable to a value that
provides a low degree of parallax between respective images of the
first and second sequences of images or the appearance of a 2D
image.
[0048] Another aspect relates to an apparatus for compensating 3D
image data in response to hard cut detection, comprising means for
preparing 3D image data for display from a first sequence of images
and a second sequence of images computed from at least two
respective images of the first sequence by using an approximation
variable that changes based on a relationship between at least two
images of the first sequence, means responsive to occurrence of a
hard cut in images of the first sequence for setting the
approximation variable to a value that provides a low degree of
parallax between respective images of the first and second
sequences of images or the appearance of a 2D image.
[0049] Another aspect relates to a method for generating 3D images,
comprising preparing 3D image data from a first sequence of images
and a second image sequence of images computed from at least two
images of the first sequence using an approximation variable that
changes based on a relationship between at least two images of the
first sequence, and reducing the value of the approximation
variable upon the occurrence of vertical disparity between at least
two images of the first sequence.
[0050] Another aspect relates to an apparatus for generating 3D
images, comprising means for preparing 3D image data from a first
sequence of images and a second image sequence of images computed
from at least two images of the first sequence using an
approximation variable that changes based on a relationship between
at least two images of the first sequence, and means for reducing
the value of the approximation variable upon the occurrence of
vertical disparity between at least two images of the first
sequence.
[0051] An aspect of the invention relates to a method of converting
3D image data for a 3D image that includes image data
representative of respective left and right eye views to 2D image
data for a 2D image, comprising
[0052] i) using the 3D image data representative of one of the left
and right eye views to provide first image data representative of a
2D image, and
[0053] ii) computing computed image data representative of a
subsequent 2D image from the first image data.
[0054] Another aspect relates to a device for converting 3D image
data, which includes image data representative of respective left
and right eye views of a 3D image, to 2D image data, comprising
[0055] i) an output providing the image data that is representative
of one of the respective left and right eye views of the 3D image
data as 2D output image data for a 2D image, and
[0056] ii) a computation device to compute from said image data
representative of said one of said respective left and right eye
views computed image data that is representative of a subsequent 2D
image.
[0057] Another aspect relates to a method of providing images for
display, comprising
[0058] i) receiving image data in either of 2D format or 3D format,
and
[0059] ii) selectively outputting in either 2D format or 3D format
output image data for display, and wherein selectively outputting
2D format output image data from 3D image data includes outputting
at least some output image data computed from one eye view of the
3D format image data.
[0060] Another aspect relates to a system for use in displaying
images, comprising
[0061] i) an input for receiving image data in either of 2D or 3D
format,
[0062] ii) a converter to convert 2D image data to 3D image data
and to convert 3D image data to 2D image data, and
[0063] iii) a selector operable to select the format of output
image data, including [0064] 2D output image data in response to 2D
input image data, [0065] 3D output image data in response to 3D
input image data, [0066] 2D output image data converted by the
converter from 3D input image data, and [0067] 3D output image data
converted by the converter from 2D input image data.
[0068] Another aspect relates to a method for generating 3D images,
comprising preparing 3D image data from a first sequence of images
and a second sequence of images computed from at least two images
of the first sequence using an approximation variable that changes
based on a relationship between at least two images of the first
sequence, and presetting the initial value of the approximation
variable according to the anticipated amount of motion across at
least two images of the first sequence.
[0069] To the accomplishment of the foregoing and related ends, the
invention, then, comprises the features hereinafter fully described
in the specification and particularly pointed out in the claims,
the following description and the annexed drawings setting forth in
detail certain illustrative embodiments of the invention, these
being indicative, however, of but several of the various ways in
which the principles of the invention may be suitably employed.
[0070] Other systems, methods, features, and advantages of the
present invention will be or become apparent to one with skill in
the art upon examination of the following drawings and detailed
description. It is intended that all such additional systems,
methods, features, and advantages be included within this
description, be within the scope of the present invention, and be
protected by the accompanying claims.
[0071] Although the invention is shown and described with respect
to one or more embodiments, it is to be understood that equivalents
and modifications will occur to others skilled in the art upon the
reading and understanding of the specification. The present
invention includes all such equivalents and modifications, and is
limited only by the scope of the claims.
BRIEF DESCRIPTION OF THE DRAWINGS
[0072] Additional details, features, and advantages of the
invention may be seen from the following description of a preferred
embodiment with reference to the annexed drawings. They show:
[0073] FIGS. 1 and 2 are schematic illustrations showing respective
images or image data--FIG. 1 represents actual images and FIG. 2
represents synthesized images;
[0074] FIGS. 3 and 4 are schematic illustrations of images or image
frames showing respective image portions or sub-frames used to
determine a similarity value d.sub.L;
[0075] FIG. 5 is a schematic block diagram representing a system in
accordance with the invention;
[0076] FIGS. 6 and 7 are schematic illustrations of parts of
respective lines of image data (or images) used do demonstrate the
method of converting 3D images to 2D images--it is noted here that
mentioning of 3D images herein usually indicates that the 3D effect
or 3D images are obtained by a sequence of images;
[0077] FIG. 8 is a block flow chart diagram representing an example
of a process for hard cut detection and compensation;
[0078] FIGS. 9a and 9b are schematic illustrations of images or
image frames showing respective image portions or sub-frames used
to determine whether there is vertical disparity or vertical
motion;
[0079] FIGS. 10 and 11 are block flow chart diagrams representing
an example of a process for detecting and compensating for vertical
disparity;
[0080] FIG. 12 is a schematic block diagram of circuitry according
to the invention;
[0081] FIG. 13 is a graphical representation of an actual image
sequence and of a scanned image sequence;
[0082] FIG. 14a-c are schematic representations of phase control in
sequential images; and
[0083] FIG. 15 is a schematic block diagram of one imaging
application of the invented device.
DETAILED DESCRIPTION
Introduction
[0084] The present invention concerns improvements for converting
2D format movies, videos, television programs, video games, etc. to
3D (three-dimensional or stereo) format to obtain a 3D effect when
observed by a viewer. The invention also concerns converting 3D
format movies, videos, television programs, video games, and the
like to 2D format.
[0085] An example of an approach for converting 2D format movies to
3D format is schematically illustrated in FIGS. 1 and 2 and also is
described in greater detail further below in this specification at
the section entitled 2D TO 3D CONVERSION USING APPROXIMATION
VARIABLE .alpha. and in the German patent application and two PCT
patent applications identified at the very beginning of this patent
application.
[0086] Briefly, as is illustrated in FIGS. 1 and 2, there is shown
at 10 a first sequence of k images 0, 1, 2, 3 . . . K. The 0.sup.th
image is the one currently being shown/displayed or is ready to be
shown/displayed--sometimes referred to as the "native" image; the
1.sup.th image is the image that was shown/displayed before the
0.sup.th image; the 2.sup.th image is the one that was
shown/displayed before the 1.sup.th image, etc. From the first
sequence of images the invention prepares a second sequence of k
images, which are shown at 11, and are designated 0', 1', 2', . . .
K'. The images in the second sequence of images are synthesized
from the first sequence of images. Without going into detail on
that procedure to synthesize the second sequence of images, which
is described in greater detail below, suffice it to say here that
each of the images in the second sequence of images is prepared
using information from images in the first sequence, e.g., the
0.sup.th image (the current image), and one or more of the images
that preceded it, e.g., the 1.sup.st image, the 2.sup.nd image,
etc.
[0087] The respective sequential images from the first and second
sequences are shown on a display and are viewed by a person who is
wearing shutter glasses. The shutter glasses are synchronized to
open and close, respectively, so that when a given image from the
first sequence of images is displayed, the person may see that
image using his/her left eye; and when a given image from the
second sequence of images is displayed, the person may see that
image using his/her right eye. If the images of the respective
sequences (for the respective left and right eye viewing) are
related such that there is a parallax relation to them, e.g., one
is shifted slightly relative to the other, the viewer construes
this as depth information and sees the image as a 3D or stereo
image.
[0088] The invention also may be used in connection with autostereo
display systems, which allow viewing of 3D images without the need
for the viewer to wear shutter glasses or to use polarizers or some
other means to discriminate between left eye and right eye
images.
[0089] In the description hereof the terms image, frame and field
may be used; they are intended to be equivalent--meaning
image--unless context indicates otherwise. For example, in the
television display techniques, a frame is composed of sequential
fields; the present invention does not necessarily have to rely on
such a distinction. Rather, the present invention is described as
using images (or image data, e.g., signals representing an image)
without regard to whether the image is a single field frame, a two
field frame, a field itself, or otherwise. Thus, absent an
indication by context, the terms image, frame, and field mean the
same.
[0090] Referring to FIGS. 3 and 4, an example of a process, as was
mentioned above, to obtain images of the second sequence, the
vertical column j.sub.m that is at the horizontal center of each of
the X.sup.0 images, e.g., the respective 0.sup.th images referred
to in FIG. 1, that are shown at 12 and the next image, the X.sup.1
image, for example, that is shown at 13, are determined, and a
horizontal distance or area M in the X.sup.0 image and a horizontal
distance or area N in the X.sup.1 image are determined arbitrarily;
therefore, the areas M and N, N greater than or equal to M, are
respective sub-image portions of the respective images X.sup.0 and
X.sup.1. Then, the two sub-image portions M and N are searched for
the areas of greatest similarity d.sub.L and for the respective
locations of those areas of greatest similarity relative to the
vertical column (centerline) j.sub.m in each of those respective
images X.sup.0 and X.sup.1. If the area of greatest similarity
d.sub.L has moved quite far from one image to the next, this
indicates fast motion occurring in the images; conversely, if the
area of greatest similarity d.sub.L has moved a small distance,
then this indicates slow motion occurring in the images. As is
described in greater detail below in the mentioned section, if the
motion is less than a predetermined minimum or is nonexistent, then
the described process may use the X.sup.0 image and an image
X.sup.2, X.sup.3, etc. Preceding the X.sup.1 image until adequate
motion has been found to allow suitable synthesizing of the next
image of the second sequence 11.
[0091] Based on finding the areas of greatest similarity d.sub.L
and their relative locations in the respective images or
sub-portions M and N of the sequential images, an approximation
variable alpha is obtained. The two images X.sup.0, X.sup.1 can be
related, compared, etc., to compute image data for a synthesized
image with adjustments based on the approximation variable alpha
.alpha.. The approximation variable alpha .alpha. can increase,
decrease or stay the same as the relation between areas of greatest
similarity move closer or farther from each other in sequential
frames. Furthermore, although the above summary concerns use of two
sequential images in the first sequence, it is possible that the
invention may use from the first sequence of images two or more
images that are not sequential, e.g., the native image X.sup.0 and
an image that was displayed or otherwise preceded the native image
2, 3 or more images (X.sup.2, X.sup.3, etc.) prior to the native
image.
[0092] The above summarizes a technique of 2D to 3D conversion that
will be described in greater detail in the 2D To 3D Conversion
section below.
Conversion of 3D Images to 2D Images and Ability to Display Either
2D or 3D Images Regardless of Whether the Input Signals Represent
2D OR 3D Images
[0093] FIG. 5 is a schematic block diagram of a system 20 of the
invention. The system 20 includes a reduction unit 21, which
converts a 3D series of images to 2D, as will be explained further
below.
[0094] The system 20 includes an image buffer 22 that receives
either 2D or 3D images in the form of respective signals or data
from some source, such as, for example, a DVD, a VHS video tape, a
cable TV connection, a satellite connection, a camera, etc. The
data from the image buffer 22 is provided to three different
systems, as follows:
[0095] a) To a conversion unit 23, which is described below.
[0096] b) To a video unit 24, which provides output image signals
(images) as 2D or 3D format output directly to a display, e.g., a
monitor, television, projector, etc. 25, as the image signals are
received from the image buffer 22. If the output is 2D, the images
can be viewed without shutter glasses. If the output is 3D, the
images are viewed on the display 25 via shutter glasses 26 that
sequentially open and close in synchronism with the respective left
and right eye images being delivered to and shown by the display
25. Therefore, what comes in to the image buffer 22 goes out to the
display 25.
[0097] c) To the reduction unit 21 that is able to reduce 3D images
to 2D images.
[0098] A control unit 27 may be operated by a user by pressing
respective buttons 27a, 27b to indicate what kind of signal is
being received (2D or 3D) and what kind of display is to be
provided (2D or 3D). The control unit activates the respective
systems a), b) and c) that are mentioned above. The control unit
could be a remote control, icons on a display or similar user
interface devices.
[0099] Turning to the reduction unit 21 that is the component which
carries out the 3D to 2D conversion for display of 3D images in 2D
format on the display 25:
[0100] 1. The user knows that 3D images are being received by the
image buffer 22, and 2D images are to be sent to the video unit 24
and the display 25 by the reduction unit 21. Therefore, the user
presses the appropriate buttons 27a, 27b to set up for 3D input and
2D output.
[0101] Let us assume that the term "frame" includes two fields, as
is conventional in television systems, for example. In an
interlaced television system, the data in one field (let's say the
"odd field," is provided to the odd numbered lines of the
television (CRT, for example), and the data in the even field is
provided to the even numbered lines. Typically the image provided
by the odd field is displayed by the television and immediately
thereafter the image provided by the even field is displayed.
[0102] Let us also assume that the odd field of a given frame
provides an image for the left eye of a 3D image and the even field
of the same frame provides an image for the right eye of the 3D
image. This mode of operation is referred to as interlaced images
for 3D display.
[0103] 2. The image buffer 22 sequentially receives respective left
eye and right eye fields. The REDUCTION UNIT 21 skips alternate
fields, e.g., it only will use the right eye fields and will not
use the left eye fields. The "not used" fields effectively are
skipped, ignored, etc. Therefore, the vertical resolution of the
output image provided to the video unit 24 and display 25 would be
halved because half of the fields are being skipped. Therefore, the
reduction unit 21 will add new fields that are SYNTHESIZED from the
respective sequential ones of the "used" fields. The new fields are
obtained by a "de-interlacing" process, as follows:
[0104] 2a. The left eye field with lines 1, 3, 5, 7, etc., is
provided to the video unit 24 and display 25.
[0105] 2b. The image data from that left eye field, the odd lines
in this example, is used to compute data for the even lines of the
display 25. This computation is carried out on a line by line/pixel
by pixel basis to obtain image data for the even lines 2, 4, 6,
etc. of the display 15. Therefore, an averaging is carried out.
Specifically, the respective pixels in line 1 are averaged with the
corresponding pixels in line 3 to obtain image data for the pixels
in line 2; and the pixels in line 3 are averaged with the pixels in
line 5 to obtain image data for the pixels in line 4; and so forth
to synthesize the second field for delivery to the video unit 24
and display 25. More specifically, pixel 1 in line 1 and pixel 1 in
line 3 are averaged in terms of r,g,b values or Y,U,V values; and
the resulting average is provided to pixel 1 in line 2. See FIG. 6
for odd field 28 and FIG. 7 for even field 29.
[0106] 2c. Using the above process the even line field is prepared
and is displayed subsequent to the preceding odd line field of the
same frame.
[0107] 2d. Subsequently the next left eye field (odd line field of
the next frame) is displayed; then the next synthesized field (even
line field of the same next frame) is prepared and displayed.
[0108] Note that the right eye field is unnecessary and the data
pertaining to the right eye field is discarded.
[0109] The above process may be carried out discarding only one eye
view, e.g., the right eye view and using only the other eye view,
e.g., the left eye view to obtain sequential fields for inclusion
in a given frame of a two field frame. Alternatively, the right eye
view may be retained and used and the left eye view discarded.
[0110] Furthermore, although the above process of converting 3D
format images to 2D images uses as an example a frame that includes
two fields in which one field provides a left eye image and the
other field provides the right eye image, it will be appreciated
that the process may be carried out using sequential images that
are not necessarily provided in respective fields of a frame.
Moreover, although the 3D to 2D process was described with respect
to an interlaced format in which the frame is composed of one field
provided by odd lines of the frame and the other field provided by
the even lines of the frame, it will be appreciated that other
types of formats may be used in connection with the process wherein
the 3D image data is stored in over/under format, side by side
format, page flip format, sequential format or some other
format.
Hard Cut Detection and Compensation in a 2D to 3D Display
System
[0111] A hard cut is an immediate change in a scene. For example,
assume that a scene in a movie or video shows a close-up of a
person's face, and then the scene immediately switches to an
outdoor view of a forest. The change in scene can happen from one
image (for example, from image X.sup.1 to the next image X.sup.0).
Since the two images are quite different, the similarity factor
d.sub.L will show a great difference, and this would tend to cause
the approximation variable alpha .alpha. to grow very large, which
is undesirable because it is not accurately reflective of motion
information or depth information (3.sup.rd dimension for stereo
images).
[0112] The present invention detects when the similarity value
d.sub.L exceeds a predetermined threshold value; such detection is
considered to represent a hard cut. (See the Introduction section
above and the section below entitled 2D To 3D Conversion Using
Approximation Variable .alpha. for further description of the
similarity value d.sub.L and its use.) Upon detecting that the
similarity value d.sub.L has exceeded the predetermined threshold
value, the invention resets the approximation variable alpha
.alpha. to a value, generally a value or possibly zero, that causes
the next two (or more) images, e.g., the X.sup.0 and the
synthesized image in the 2D to 3D conversion method, which is
described elsewhere herein, to provide in effect an image with a
low level of 3D depth or a 2D image (an image with low parallax).
Thus, the left and right eye images would be the same or nearly the
same. If the approximation variable .alpha. were not set to a
predetermined value upon such hard cut detection, .alpha. might
tend to achieve a level that would be representative of large or
fast motion due to dissimilarity between the two images at the hard
cut. This would present an inaccurate indication of depth (third
dimension) and also would cause an unsatisfactory or uncomfortable
view for the viewer. It is noted that although hard cut detection
is described here as being detected when the similarity value
d.sub.L is such as to indicate a large disparity between two
sequential images of a first image sequence, such hard cut
detection may be based on more than two images. Also, the
approximation variable a may be set to a predetermined value
immediately upon the detection of the occurrence of a hard cut, or
it may be set step-wise over time, such as over the time during
which several images are prepared for display or the time during
which several images are displayed, to provide a more gradual or
smooth transition.
[0113] As an example, by setting the approximation variable to the
indicated value upon detecting occurrence of a hard cut, e.g.,
setting .alpha. to zero, there would be no difference between the
left and right eye images for the next pair of sequential image
images, so the image is seen in 2D for those two images.
[0114] However, after displaying the pair of sequential images
respectively to the left and right eyes of the viewer immediately
following the hard cut having been detected and the approximation
variable .alpha. having been set to the predetermined value, e.g.,
zero (0) or a value yielding small degree of parallax, thereafter
the approximation variable .alpha. is allowed to adjust
automatically as is described elsewhere herein. Therefore, as the
images of the first sequence of images are received and are
displayed and as the images of the second sequence of images are
synthesized and displayed interspersed with the images of the first
sequence of images, 3D images again are displayed and may be seen
by the viewer via the shutter glasses. Such adjustment of the
approximation variable .alpha. may occur in several different ways
that are described just below.
[0115] The approximation variable .alpha. may simply be allowed to
adjust automatically, as is described in the Introduction and in
the 2D To 3D Conversion sections below. Alternatively, the
approximation variable .alpha. may be allowed to adjust as just
indicated, except that the change in the approximation variable
.alpha. may be regulated, e.g., in increments of 0.1, 0.2, 0.5,
0.75, or some other amount or functional relationship, until it
reaches a target value, such as that computed per the description
in the 2D To 3D Conversion section below. As a still further
alternative, the prior value of the approximation variable .alpha.
that existed just prior to the detecting of the hard cut, i.e.,
before it is set to the predetermined value, could be stored; and
the approximation variable .alpha. could be allowed to increase
toward that prior value until it were to reach that (target) value
or near that value or some other value--thereafter, the
approximation variable .alpha. would be allowed to adjust
automatically, as is described in the 2D To 3D Conversion section
below. In any event, regulating the rate of change and target goal
for the approximation variable .alpha. tends to provide a smooth
transition to the increased value thereof and also will provide for
a smoother return of the actually viewed images by an observer to
3D viewed format as compared to an instantaneous rather large
increase or jump to 3D. As the approximation variable .alpha.
changes, the apparent depth of the 3D images gradually but rapidly
changes to the desired depth determined by the target value of the
approximation variable. This provides a more pleasant, comfortable,
less tiring viewing experience.
[0116] Turning to FIG. 8, a flow chart 40 indicating the method
associated with detecting the occurrence of hard cut is
illustrated. At block 41a similarity value d.sub.L is received from
comparison of two images (or more than two images) in a sequence of
images or from portions of two images. At block 42 a comparison is
made to determine whether the similarity value exceeds a threshold.
If yes, then this indicates occurrence of a hard cut, and at block
43 the approximation variable .alpha. (alpha) is set to a
predetermined value, e.g., zero or a value, as to provide a 2D or
limited 3D viewing experience and then a gradual increase back to
full 3D viewing experience, as was described above. Also, at block
43, if desired, the value of the approximation variable .alpha.
prior to resetting may be stored for possible use in connection
with the automatic adjusting of the value of the approximation
variable .alpha., as was mentioned above. At block 44 the next
synthesized image is computed using the then existing value of the
approximation variable .alpha., as is described in the 2D To 3D
Conversion section below. The X.sup.0 image then is displayed at
block 45, and then the associated synthesized image that had been
computed is displayed at block 46.
[0117] A loop 47 then is followed back to block 41 via block 47
calling for repeating the foregoing for the next image frame
X.sup.0.
[0118] If at block 42 the similarity value d.sub.L is less than the
threshold (or equal to it), the approximation variable .alpha. is
computed at block 48. If a hard cut had not been detected at block
42, then the computation of the approximation variable .alpha. may
be as is described in the 2D To 3D Conversion section below.
However, if a hard cut had been detected at block 42, then a flag
may be set so that when the computing occurs at block 48 the change
in the approximation variable .alpha. may be regulated until it
reaches the prior value that had been stored, other target value or
a substantially stable value, e.g., on account of the computing no
longer calling for any changes in the approximation variable
.alpha. for the next image or several of them, as may be desired to
be set by the user.
[0119] The computing of the approximation variable .alpha. also is
described in the immediately following section, which also may
apply in the method of the instant hard cut section and be
considered a part of the description just above.
Elimination of Vertical Motion Effect in Converting From 2D to 3D
in a 2D to 3D Display System
[0120] See FIGS. 9-11.
[0121] The occurrence of vertical disparity between two image
frames is undesirable both because it creates in a sense an
artificial astigmatism (one eye sees the image vertically displaced
from the other) and also because the sensing of vertical motion
possibly could cause an undesirable tendency to increase the
approximation variable alpha .alpha. and, thus, incorrectly cause
an improper synthesized image.
[0122] Therefore, this feature of the invention is to detect
vertical motion, i.e., vertical disparity L* between two similar
portions of the images in successive image frames and to compensate
for that vertical motion by reducing the approximation variable
alpha .alpha..
[0123] The invention detects vertical disparity L* in a way similar
to that described in the Introduction above and also in the 2D To
3D Conversion Section below for detecting motion by detecting
horizontal disparity between successive image frames using the
areas of greatest similarity in the successive image frames.
[0124] Referring to FIGS. 9a and 9b, rather than simply using areas
of greatest similarity in successive frames, the present invention
looks for respective edges of objects shown in the successive image
frames or in sub-frame or sub-image portions O and P centered on
the horizontal image midpoint line or centerline i.sub.m of the
successive image frames X.sup.0 and X.sup.1, for example, that are
shown at 60 and 61. Assume, for example, that the edges are those
of a the eaves of a roof of a house; in one image frame the edge
line of the eaves is at one vertical location L.sub.a in the image
frame and in the prior image frame the edge line of the eaves is at
a different vertical location L.sub.b in the image frame. The
relative locations could be determined by making a comparison
similar to the comparison made with the image portions M and N
described above in the Introduction and in the 2D To 3D Conversion
section; but now the comparison is made using horizontal swaths
(image frame portions) O and P across the respective image frames,
as is illustrated in FIGS. 9a and 9b. The relative locations could
be found by comparing the entire extent of the eaves line or
corresponding portions of the eaves lines in the respective frames
X.sup.0 and X.sup.1.
[0125] The vertical disparity between two successive image frames
will be designated L* for this discussion. Thus, L* is defined as
the difference between L.sub.a and L.sub.b. If the vertical
disparity L* does not exceed a predetermined value or threshold,
for example, a value of plus or minus 1, then we can assume that
there is no significant vertical disparity. (Exceeding the
threshold means larger than plus 1 or smaller than minus 1 in this
example.) Therefore, there is no adjustment made of the
approximation variable alpha. Such threshold value may represent
one or more horizontal lines of an exemplary display or image data
to be displayed, if desired. The threshold value may be other than
plus or minus 1, e.g., it may be more positive, more negative, or
both, e.g., the absolute value may be larger or smaller than 1.
[0126] However, if the vertical disparity L* exceeds plus or minus
1, then we can assume that there is some vertical disparity, and
the extent of the vertical disparity would be generally
proportional to the value of L*.
[0127] Referring to FIG. 10, at block 62 if there is vertical
disparity L* greater than a predetermined threshold (e.g., smaller
than minus 1 or greater than plus 1), then the approximation
variable alpha .alpha. is set to a predetermined value, e.g., zero
(or some other value), either instantly or step-wise. This results
in the viewer seeing in effect a 2D image or a low parallax image
for at least a moment or longer. At block 63, after such vertical
disparity greater than the threshold no longer exists, the
approximation variable alpha is allowed to be adjusted, i.e., to
vary as is described, for example, in the previous section and
below.
[0128] Referring to FIG. 11, where the method 70 of FIG. 10 is
shown in greater detail, at block 71 assume there is vertical
disparity L* greater than plus or minus 1. Therefore, the invention
intends to reduce the magnitude of the approximation variable alpha
.alpha.. This reduction may occur over the occurrence of several
image frames, say 3 to 5 image frames, whereby the approximation
variable alpha .alpha. is decreased from its current value to, for
example, zero, in steps that are proportional to its initial value
when the vertical disparity was detected divided by the number of
frames over which it is being reduced. Thus, for example, if the
value of the approximation variable alpha .alpha. were 2.5 and it
is to be reduced to zero over 5 image frames, each step would be
0.5.
[0129] As the approximation variable alpha .alpha. moves toward
zero, the difference between the image frame X.sup.0 and the
synthesized image (see the 2D To 3D Conversion section below) would
be reduced so that the 3D effect or stereo appearance of the images
as viewed through shutter glasses, for example, would be reduced to
reduce the depth effect. If the approximation variable alpha
.alpha. were zero, the viewer would see a 2D image.
[0130] Another aspect of this invention is the changing of the
number of steps (number of image frames) over which the
approximation variable alpha .alpha. is reduced to zero depending
on the magnitude of the vertical disparity L*. Therefore, if the
vertical disparity L* is relatively small (i.e., larger or smaller
than plus or minus 1, but not, for example, 7, then the
approximation variable alpha .alpha. would be reduced over, for
example, an interval of five images, i.e., in five steps. However,
if the vertical disparity L* were relatively larger or smaller than
plus or minus 1, for example if it were 4, then the rate at which
the approximation variable alpha .alpha. would be reduced would be
much quicker, for example, over two images (2 steps) or even
immediately.
[0131] After reaching zero or other target value, the approximation
variable alpha .alpha. would be kept at the target value until the
vertical disparity L* would revert to a number between plus and
minus 1.
[0132] Now, consider, how does the approximation variable alpha
.alpha. would grow or alter after the vertical disparity had
reverted to below the threshold. It is undesirable for the
approximation variable alpha .alpha. to revert to a large value
instantaneously because that would cause a jerky image going from
2D or a low parallax image to an large depth 3D image (high
parallax image), for example, that would be unpleasant for the
viewer to see.
[0133] Referring to FIG. 11, a flow chart 70 is illustrated.
Vertical disparity L* smaller than minus 1 or greater than plus 1
is detected as existing at block 71. Therefore, the current value
of the approximation variable alpha .alpha. is stored at block 72.
Thereafter, at block 73 the approximation variable alpha .alpha. is
set step-wise to zero or other target value (e.g., over a number of
frames, the number depending on the magnitude of the vertical
disparity L*) or, if the magnitude of the vertical disparity L* is
large, e.g., exceeds some predetermined value, then the
approximation variable alpha .alpha. immediately is set to the
target value or zero.
[0134] At block 74 the image X.sup.0 is displayed and the next
synthesized image is computed and displayed. The display of those
images is coordinated with the shutter glasses so one eye sees
image X.sup.0 and the other eye sees the synthesized image.
[0135] An inquiry is made at block 75 whether the vertical
disparity L* exceeding the threshold still exists; if yes, then at
block 76 the approximation variable alpha .alpha. continues being
reduced or is kept at a target value, and at block 74 the next
image X.sup.0 and the associated synthesized image are displayed,
and so on.
[0136] However, if the vertical disparity L* no longer exceeds the
threshold (block 75), then the approximation variable alpha .alpha.
is increased toward the previously stored value or other target
value thereof until it reaches that value. The occurrence of each
of those increases corresponds to the display of an image X.sup.0
and the immediately following synthesized image. The magnitude of
each increase can be set by the user or can be determined by the
system of the invention, e.g., a stepwise or some other fixed or
variable step or functional increase. As the approximation variable
alpha .alpha. is increased, the 3D stereo effect increases for a
given sequence of images
[0137] After the approximation variable alpha .alpha. has returned
to the previously stored value or other target value, the
approximation variable .alpha. is allowed to adjust automatically.
Therefore, as the images of the first sequence of images are
received and are displayed and as the images of the second sequence
of images are synthesized and displayed interspersed with the
images of the first sequence of images, 3D images again are
displayed and may be seen by the viewer via the shutter glasses,
for example. TABLE-US-00001 ADJUSTMENT FOR PROGRAM-TYPE (MOVIE OR
SCENE TYPE) IN 3D TO 2D CONVERSION SYSTEM See the following chart:
Pre-set Values of Parameters for 2D to 3D Conversion Software Scene
Approximation Variable Type alpha Parallax Warp Action: 2.1 5
Asymmetric Talk Show: 4.3 6 Asymmetric Wildlife: 6 4 Tilt Music
Video: 1.1 4 Lenticular
[0138] An approximation variable alpha (.alpha.) is used in the
computing of synthesized images for display to one eye sequentially
following the displaying of the actual image X.sup.0 to the other
eye, as is described elsewhere herein. The instant feature of the
invention relates to a pre-setting of the approximation variable
alpha .alpha. to a specific level according to the type of scene,
movie, video or the like (collectively referred to as movie below)
that is being displayed. Throughout adjustment processes, these
initial values also are used as target values, such as for the
approximation variable .alpha. in the adjustment processes
described above. While the exemplary values shown in the above
chart are suitable for the display of several initial images, these
target values are exemplary only and other values may be equally or
better suited for a given scene depending on the input data stream.
Other target values may be more suitable for other types of
scenes.
[0139] For example, the faster the movement typically found in the
movie, the smaller the value of approximation variable alpha
.alpha. ordinarily would be. Therefore, if a scene in a movie had
more or faster action, the value of approximation variable alpha
.alpha. would automatically tend to be reduced by the 2D to 3D
conversion software used in the invention, as was briefly described
in the Introduction (above) and in the 2D To 3D Conversion section
(below); likewise as motion slows, the value of approximation
variable alpha .alpha. would tend to increase.
[0140] In accordance with the instant feature or aspect of the
present invention the user may pre-select the type of movie that is
going to be watched, such as, for example, an action movie, talk
show, wildlife movie, or music video. As usually is the case,
action movies and music videos have relatively fast action in the
various scenes thereof. Therefore, the approximation variable alpha
.alpha. would tend to be relatively small. Also, in the case of
television talk shows and wildlife movies (in which an animal is
watched for a while) the amount of motion is relatively slow in the
various scenes; and, therefore, the value of approximation variable
alpha .alpha. would tend to be relatively large.
[0141] The present invention allows the user to pre-select the type
of movie that is going to be watched. Such pre-selection sets an
initial value for approximation variable alpha .alpha.. It also may
set an initial value for parallax and warp, and these parameters
and the approximation variable .alpha. may be set and used in the
2D to 3D conversion process described in the Introduction and in
the section below. The approximation variable .alpha., parallax and
warp parameters are described in the U.S., German and two PCT
patent applications described above at the beginning of this patent
application. I do not describe the details of parallax or of warp;
suffice it to say here that they are parameters that may be set and
used in the 2D to 3D conversion software.
[0142] Such setting is shown by respective inputs 80-82 to the
conversion unit 23 in the system 20 of FIG. 5. Accordingly, the
conversion unit may be used to provide those settings to the video
unit 24. After the approximation variable alpha .alpha. is pre-set
according to the type of movie that will be watched, the
approximation variable alpha (.alpha.) may be adjusted
automatically according to the actual motion that is detected, as
is described elsewhere herein.
[0143] The other parameters, e.g., parallax, warp, and/or others,
similarly could be pre-set according to the type of movie. Also, as
was described with respect to the approximation variable alpha
(.alpha.), one or more of such other parameters may adjust
automatically and/or may be set to a prescribed value in response
to detection of a given occurrence, such as a hard cut, vertical
motion, etc.
2D to 3D Conversion Using Approximation Variable .alpha.
[0144] This section is referred to by the above sections as the 2D
To 3D Conversion section. In this section the symbol O is used to
represent a zero (0) value to try to avoid conflicting with the
letter "o" that is used in several of the variables, formulas,
etc.
[0145] The basic components of a device according to the invention
and their interconnections are schematically represented in FIG.
12. The system comprises a first input E1, through which the
two-dimensional images generated by a camera and transmitted across
a transmission path are directed to an A/D converter 110 and
digitized. The digitized images are then directed to an image
storage device 111 and a phase selector 116. The phase selector
determines whether the average motion in the image sequence is
going to the left or right. The images saved in the image storage
device 111 are analyzed by a phase analyzer 112, the input of which
is connected to the image storage device 111, and the output of
which is connected to the phase selector 116. The phase analyser
112 analyzes whether the average motion is going to the left or
right; based on that analysis or information the phase selector 116
determines whether the image goes to (is directed to) the right eye
or to the left eye. In addition, a long-term storage device 113 is
connected to the image storage device 111 for storing images from
this storage device and the output of which is connected to an
image generator 115.
[0146] Furthermore, the image generator 115 is also connected to
another output of the image storage device 111 and of a motion
analyzer 114, to which images from the image storage device 111 are
directed. In addition, the device comprises a second input E2 for
manual motion control connected to the image generator 115, as well
as a third input E3 for manual phase control attached to the phase
selector 161. The manual motion control may be operated to select
or to toggle between stereo and inverse stereo (also sometimes
referred to as pseudo stereo). A right or left stereo image
B.sub.L, B.sub.R is attached to two outputs of the phase selector
116, which are connected to a first or second output A1, A2 of the
device. These outputs A1, A2 may be coupled to a computer display,
monitor, television, projector or any other display to provide
respective images.
[0147] A second image sequence is generated by this device based on
a (first) image sequence recorded in two-dimensions. Together with
the first image sequence, the second sequence makes it possible to
view the originally two-dimensional images in three dimensions when
the first and second image sequences are transmitted to the left or
right eye. The second image sequence is defined according to the
following description based on image information resulting from the
motion in the first image sequence. The following definitions
apply:
[0148] X.sub.ij is a digitized image at time t with vertical
resolution I and horizontal resolution J. For convenience and
standardization in describing the invention herein, with regard to
the symbols, as used herein, upper case letters, e.g., X.sub.ij,
represent a grouping, such as, for example a digitized image that
is formed by a number of pixels, and lower case letters, e.g.,
x.sub.ij, represent a portion of the grouping, e.g., a pixel of the
image. The scan interval is .DELTA.t, so that the following formula
is defined for an image scanned at time k and saved in the image
storage device 111: X.sub.k:=X.sub.ij(t-.DELTA.tk)
[0149] The above formula concerns an image stored in the image
storage device 111. The above formula is a definition. The symbol
:=means "is defined as". Also, the above formula indicates that the
X image that previously had been received and remains stored in the
system as 1, 2, 3, . . . etc. images/frames/fields ago, is defined
as the image X.sub.ij that is a function of or has occurred at the
current or immediate time t minus the scan interval .DELTA.t times
the number of the 0 time (image). "K" represents the number of
images or frames that currently are stored in the system and "k"
represents the number of the particular image that had been stored,
e.g., one image ago, two images ago, etc. X.sup.0 is defined as the
current image.
[0150] The most recent K images are located in the image storage
device 111 with length K. 0.ltoreq..alpha..ltoreq.k is a real
number representing the time interval of a given image X.sup.k, at
which a (synthetic also referred to herein as "synthesized") image
of the second image sequence is generated. The approximation
variable, .alpha., may be an integer, e.g., 1, 4, etc., or it may
be a decimal or fraction, e.g. 0.2, 1.4, 4.3, etc. In addition,
B.sub.L represents the given displayed left image and B.sub.R the
given displayed right image. The letter "X" is upper case just
above because it is a "given image" rather than a pixel--a pixel
would be represented by the lower case letter "x."
[0151] As a non-limiting example, the number (K) of images stored
in the image storage device 111 may be four or six; these would be
the last (e.g., the most recent) four or six images that were
stored. The number K of such images may be other than four or six.
Thus, each stored image may be represented X.sup.1, X.sup.2, . . .
X.sup.6, that is X.sup.k. The respective images X.sup.k are shifted
in the image storage device 111 as new images are provided.
[0152] It is assumed that a fixed initial value is given to
.alpha.. The images X.sup.k in the image storage device 111 are
viewed as sample values (scanned image sequence according to curve
b in FIG. 13) of a constant function (actual image sequence
according to curve a in FIG. 13). Various methods of approximation
may be applied to this function. The following explanations
relating to FIG. 13 refer to a linear spline approximation.
However, other methods of approximation may be used as appropriate;
for example, higher-level or polynomial approximations.
[0153] FIG. 13 shows an image sequence in two-dimensional (I/J-)
space. The second image sequence is calculated by the image
generator 15 as follows: first, .alpha..sub.u is calculated as the
largest whole number (integer) which is smaller than .alpha.. Next,
.alpha..sub.o is calculated as the smallest whole number (integer)
which is larger than .alpha.. So: B.sub.L:=X.sup.O
B.sub.R:=X.sup..alpha..sup.o(.alpha.-.alpha..sub.o)+X.sup..alpha..sup.u(1-
-.alpha.+.alpha..sub.u) In the above equations the symbols X**
actually mean the images (or image data) X that is associated with
number .alpha..sub.o or number .alpha..sub.u. In the symbology used
herein the number zero is shown as to O, that is a "0" with a line
through it to facilitate distinguishing the number zero from the
letter "o". Also, upper case letter X is used in the respective
paragraphs just above and just below since it represents an image,
not a pixel. Note that the letter "o" is used in the right image
equation because it represents "ober" (the German language word for
more than or greater than) and where used the letter "u" means
"unter" (the German language word for less than).
[0154] The image sequence B.sub.L for a left viewing channel for
(left eye viewing) is calculated by the given actual images of the
first image sequence X.sup.O, X.sup.1, etc., and the (second) image
sequence B.sub.R is calculated by approximation for a right viewing
channel (for right eye viewing). Note that X.sup.O is the "native"
image, e.g., the one being shown or displayed at the current time
or the one being readied to be shown or displayed.
[0155] This calculation is performed separately by the image
generator 15 for all of the pixels x.sub.ij in a selected color
space for examples (RGB or YUV); that is (for YUV): B R := b i
.times. .times. j ( Y , U , V ) := ( x i .times. .times. j .alpha.
o .function. ( Y ) .times. ( .alpha. - .alpha. u ) + x i .times.
.times. j .alpha. u .function. ( Y ) .times. ( 1 - .alpha. +
.alpha. u ) , x i .times. .times. j .alpha. o .function. ( U )
.times. ( .alpha. - .alpha. u ) + x i .times. .times. j .alpha. u
.function. ( U ) .times. ( 1 - .alpha. + .alpha. u ) , x i .times.
.times. j .alpha. o .function. ( V ) .times. ( .alpha. - .alpha. u
) + x i .times. .times. j .alpha. u .function. ( V ) .times. ( 1 -
.alpha. + .alpha. u ) ) . ##EQU1## b.sub.ij is a pixel (member) of
the total image B.sub.R. b.sub.ij.sup.(Y,U,V) is calculated in YUV
color space as is shown above. A similar calculation could be made
for RGB color space.
[0156] RGB color space typically is used for computers; YUV color
space typically is used for television. Other color space variables
representing color, gray scale, etc. also may be used, as will be
appreciated.
[0157] In addition, automatic phase control is performed by the
phase analyzer 12 to determine movements in sequential images of
the first image sequence. I is the number of rows (lines) and J is
the number of columns of an image or of a display in which an image
is expected to be or is being shown. It is assumed that
j.sub.m:=J/2 is the horizontal midpoint of an image along a column
j.sub.m, as is illustrated in FIG. 14a, so x.sub.ijm.sup.O with
O.ltoreq.i.ltoreq.I is the middle column of the image X.sup.O at
time t. Furthermore, M<j.sub.m, where M is a selected integer.
Then: X.sub.ij.sup.O3:=X.sub.ij.sup.O with O<i<I and
j.sub.m-M<j<j.sub.m+M will be defined as a scanned image,
shown in vertical stripes in FIG. 14a. The symbol "s" indicates a
scanned image; it is a portion or subimage of the image
X.sub.ij.sup.O. (Also, note that the symbol "s" here is a subscript
to the O, e.g., as in the case of .alpha..sub.o above.) Said image
comprises 2M+1 columns around the horizontal midpoint j.sub.m of
the image X.sup.O.
[0158] Now, N is a fixed number (since N and M represent numbers of
columns, both N and M are integers) with N>M, so:
X.sub.ij.sup.1s with O.ltoreq.i.ltoreq.I and
j.sub.m-N.ltoreq.j.ltoreq.j.sub.m+N are defined as the search
region (see FIG. 14b) in image X.sup.1, in which the partial image
with the greatest similarity to the scanned image X.sub.ij.sup.Os
is sought. X.sub.ij.sup.1s ("s" is a subscript to the digit "1") is
a subimage that is generated out of image X.sup.1. d.sub.L is the
similarity of the scanned image to a partial image of equal size
from the search region with a displacement position L, where
-N.ltoreq.L.ltoreq.+N.
[0159] If cross-correlation is chosen as a measure of similarity,
d.sub.L is the result for the displacement position L: d L := 1 - i
= 0 I .times. j = J m - m j m + m .times. x i .times. .times. j O /
x i .function. ( j - 1 ) 1 i = 0 I .times. j = J m - m j m + m
.times. ( x i .times. .times. j O / ) 2 i = 0 I .times. j = J m - m
j m + m .times. ( x i .function. ( j - 1 ) 1 ) 2 Formula .times.
.times. ( 1 ) ##EQU2## Here, the value of L ranges from -N to +N,
where L represents the given displacement position of a partial
image in the search region.
[0160] As an alternative to cross-correlation, a Euclidean distance
or an absolute amount may also be chosen as a measure of
similarity.
[0161] Thus, with this method, as indicated in FIGS. 14a and 14b,
the scanned image Xs (FIG. 14a) runs like a scanner across the
search region (FIG. 14b) of the image X.sup.1 (previous image) and
looks for the region with the greatest similarity d.sub.L to the
scanned image, where the similarity d.sub.L is calculated for every
displacement position L.
[0162] In addition, a whole number (integer) .epsilon. is defined,
which may be called the moment of inertia and with which blurring
is defined according to FIG. 14c. This is used to allow for camera
movement that should not be considered displacement of the image.
For the value of .epsilon.-1.ltoreq..epsilon..ltoreq.1
approximately.
[0163] This analysis is performed substantially as follows. First,
all measures of similarity d.sub.L for -N.ltoreq.L-+N are
calculated by the phase analyzer 12. Next, the measure of
similarity d.sub.min is chosen with the smallest value
(d.sub.min:=min d.sub.L) and the index I.sub.min of this measure of
similarity is determined. (I.sub.min is the position where the most
(greatest) similarity exists; as an example only, I.sub.min may be
five columns to the left or to the right.) The values I.sub.min and
.epsilon. are compared by the phase selector 116, and the phase
selector 116 switches as a function of the results of the
comparison as follows:
[0164] If I.sub.min<.epsilon., this means that the region of
greatest similarity in the search region is displaced to the left,
and thus the predominant direction of movement in sequential images
X.sup.1, X.sup.O of the first image sequence is indicated from left
to right. (X.sup.O is the current image, and X.sup.1 is the
immediate prior image.) This may result from the movement of an
object in the images from left to right or from the panning of the
camera from right to left (sometimes referred to as moving,
trucking or other motion of the camera). In this case, for the left
image B.sub.L:=X.sup.O (i.e., the given image of the image
sequence) and a calculated synthetic (sometimes referred to herein
as "synthesized") image (second image sequence) is selected for the
right image B.sub.R. In addition, a "shift" indicator is set to
"left" in the phase selector 116.
[0165] If I.sub.min>.epsilon., this means that the region of
greatest similarity in the search region is displaced to the right,
and thus the predominant direction of movement in sequential images
X.sup.1, X.sup.O of the first image sequence is indicated from
right to left. This may result from the movement of an object in
the images from right to left or from the panning of the camera
from left to right. In this case, a calculated synthetic image
(second image sequence) is selected for the left image B.sub.L and
for the right image B.sub.R:=X.sup.O (i.e., the given image of the
image sequence). In addition, the "shift" indicator is set to
"right"
[0166] If |I.sub.min|<.epsilon. and the indicator is set to
"right", then a calculated synthetic image is selected for the left
image B.sub.L (second image sequence) and for the right image
B.sub.R:=X.sup.O (i.e., the given image of the image sequence).
Finally, if |I.sub.min|<.epsilon. and the indicator is set to
"left", then for the left image B.sub.L:=X.sup.O and a calculated
synthetic image is selected for the right image (second image
sequence).
[0167] The next image is then accessed (e.g., digitized) and the
same process is repeated for this image, beginning with the
calculation of the location of a maximum degree of similarity.
[0168] This automatic phase control or selection may also be
switched off and, for example, replaced by manual switching using a
keyboard, or remote, or icons, via the device's third input.
[0169] Furthermore, the embodiment shown in FIG. 12 comprises the
motion analyzer 114, which uses dynamic motion control or motion
calculation to prevent the stereo base from becoming too large when
there are large movements. In addition, this ensures that a certain
minimum width of the stereo base is maintained during very slow
movements before it disappears in images without any motion. The
long-term storage device 113, from which images are accessed and
used as images of the second image sequence when the movements are
too slow, has been provided for this last purpose. Thus, the
invention may take and use in computations an image from the long
term storage device if no movement (motion) is detected between,
say, adjacent (immediately sequential) images; otherwise the value
for the approximation variable a would get too large. The measure
of similarity .delta..sub.k at time t.sub.k is defined as follows:
.delta. k := 1 - i = 0 I .times. j = 0 J .times. x i .times.
.times. j k x i .times. .times. j k + 1 i = 0 I .times. j = 0 J
.times. ( x i .times. .times. j k ) 2 i = 0 I .times. j = 0 J
.times. ( x i .times. .times. j k + 1 ) 2 Formula .times. .times. (
2 ) ##EQU3## Therefore, this measure of similarity is a function of
the extent to which the entire contents of the next image in an
image sequence differ from the contents of the previous image, and
thus represents a measure of the speed of motion in the images.
[0170] Threshold values
.delta..sub.0<.delta..sub.1<.delta..sub.2 are defined for the
analysis of said measure of similarity, where in the ideal case the
measure of similarity .delta..sub.k=0 for an unchanged (constant)
image at time t.sub.k in comparison to the previous image at time
t.sub.k+1. The values of the "deltas" may be arbitrary. However,
because there is always a certain amount of background noise during
digitization, it should be assumed that
.delta..sub.k<.delta..sub.0 for an unchanged image.
[0171] A Euclidean distance or an absolute amount may of course be
chosen for the calculation instead of the cross-correlation
described. The individual color values of the selected color space
RGB or YUV must always be processed separately.
[0172] To analyze the value of the measure of similarity
.delta..sub.k (k=0, 1, . . . K), it is first stored in the motion
analyzer 114 and then compared to the threshold values.
[0173] If .delta..sub.k<.delta..sub.0, this means that the
movements in the sequential images are very slow or nil. In this
case, the transfer of the values of X.sup.k to the long-term
storage device 113 is stopped so that images will be available
which have a sufficient motion differential. In addition, images
stored in the long-term memory device are used to generate the
second image sequence in order to maintain the minimum stereo base
width, for example, the difference between two images as viewed by
a viewer (person), such as sixty-five millimeters distance between
the viewer's eyes as but one example.
[0174] In order to rapidly set the best value of the approximation
value, it is advantageous to know whether the motion is increasing
(speeding up) or decreasing (slowing). If
.delta..sub.k>.delta..sub.0, the value of the approximation
variables .alpha. will change as a function of the size of the
measure of similarity .delta..sub.k relative to the threshold
values .delta..sub.0, .delta..sub.1, .delta..sub.2, as follows. The
objective is to increase or to decrease the value of the
approximation variable .alpha. as may be appropriate.
[0175] If .delta..sub.0<.delta..sub.k<.delta..sub.2 and
.delta..sub.k-.delta..sub.k-1.ltoreq..delta..sub.1 and as long as
.alpha..ltoreq.K-s, then the approximation variable is set at
.alpha.:=.alpha.+s. ("s" is defined below.)
[0176] If .delta..sub.0<.delta..sub.k<.delta..sub.2 and
.delta..sub.k-.delta..sub.k-1.ltoreq..delta..sub.1 and as long as
.alpha..gtoreq.s, then the approximation variable is set at
.alpha.:=.alpha.-s. Ordinarily it is undesirable for .alpha. to
equal s, because that would bring .alpha. to zero, and then the
left and right images would be the same (producing a 2D image).
[0177] The character "s" denotes a step width which may be 0.1,
however, it can have other values as well such as, for example,
0.5, 0.75 or another calculated value.
[0178] If .delta..sub.0<.delta..sub.k<.delta..sub.2 and
-.delta..sub.1<.delta..sub.k-.delta..sub.k-1<.delta..sub.1,
then the approximation variable will remain at .alpha.:=.alpha.
because the motion velocity is substantially constant. In this
case, no adjustment is necessary. In this case there is no
adjustment to .alpha. but there is constant motion, for
example.
[0179] Finally, if .delta..sub.2<.delta..sub.k, this means that
the movement is very fast and the stereo base width would be too
large. In this case, the approximation variable may be set at
.alpha.:=s times .delta..sub.2/.delta..sub.k.
[0180] This dynamic motion control can also be switched off like
the automatic phase control and replaced by manual entry; for
example, using a keyboard via the device's second input.
[0181] The method described will preferably be implemented using a
data processing program on a computer, in particular a digital
image processing system for the generation of a three-dimensional
depiction of television pictures transmitted or stored in a
two-dimensional format. Reference herein to television means, for
example, broadcast signals, video, signals, DVD, CD and/or tape
recorded signals or other signals representing respective images or
image information. Also, reference herein to image or images may
refer to image data for use ultimately in providing an image for
display or to a display device, such as a computer monitor,
television, etc., as will be evident from the context of the
description.
[0182] In the following, an example with specific values shall be
given for the above embodiment. In case of application of the known
PAL standard the vertical resolution is I=576 and the horizontal
resolution is J=768, whereas for the NTSC standard, I=480 and J=640
are prescribed.
[0183] Generally it is sufficient to store the last five images in
the image storage device 111 which means K:.ltoreq.5. As an initial
value .alpha..sub.0, the approximation variable is set to
.alpha..sub.0:=2.1. For an adequate analysis of motion in
sequential images the value of M is set to 1 or 2. The value of N
should be chosen such that even in case of fast motions the
scanning image is still within the search region. For this, a value
of N of 20.ltoreq.N.ltoreq.30 (for example, N:=25) is adequate.
However, the value of N can as well comprise the complete original
image so that N:=J/2.
[0184] For defining the blurring, a value of .epsilon.:=1 is
proposed whereas for evaluating the measure of similarity the
following values for the threshold values are preferably chosen:
.delta..sub.0:=0.05, .delta..sub.1:=0.6 and .delta..sub.2:=0.8.
[0185] With an embodiment realized with these values a very natural
three-dimensional reproduction could be obtained for image
sequences with very differently moving contents.
[0186] Finally, FIG. 15 shows a block diagram of a device for the
generation and depiction of 3-D images which are calculated based
on a sequence of 2-D images transmitted over a transmission path or
accessed from a storage medium.
[0187] The device comprises a first input 121, to which the 2-D
images transmitted across a transmission path and demodulated or
decompressed according to known techniques are connected. In
addition, there is a second input 122, which is connected to a DVD
player, a video recorder, or another source of images, for
example.
[0188] Both of these inputs are connected to the invented device
123 according to FIG. 15, with which 3-D images are calculated
based on the sequence of 2-D images according to the detailed
explanation above. The outputs A1, A2 of this device, to which a
sequence of left or right images B.sub.L, B.sub.R is connected, are
connected to a stereo storage device 124, 125, in which the images
are stored for each channel.
[0189] Finally, different driver levels (these could be software
driven or hardware driven) can be selected via a third input 126 by
activating a selector switch 127, by means of which a corresponding
image generator is controlled. For example, a driver 128 for
simulator goggles 129 (also known as "shutter glasses"), a driver
130 for an autostereoscopic monitor 131, and a driver 132 for a
stereo projector 133 are shown here.
[0190] This device is preferably part of a digital image processing
system for viewing 3D images from generated or stored 2D
images.
[0191] Although the invention has been shown and described with
respect to certain illustrated embodiments, equivalent alterations
and modifications will occur to others skilled in the art upon
reading and understanding the specification and the annexed
drawings. In particular regard to the various functions performed
by the above described integers (components, assemblies, devices,
compositions, etc.), the terms (including a reference to a "means")
used to describe such integers are intended to correspond, unless
otherwise indicated, to any integer which performs the specified
function (i.e., that is functionally equivalent), even though not
structurally equivalent to the disclosed structure which performs
the function in the herein illustrated embodiments of the
invention. In addition, while a particular feature of the invention
may have been described above with respect to only one of several
illustrated embodiments, such a feature may be combined with one or
more other features of the other embodiment, as maybe desired and
advantageous for any given or particular application.
* * * * *