U.S. patent application number 14/476585 was filed with the patent office on 2014-12-25 for method and apparatus for making intelligent use of active space in frame packing format.
The applicant listed for this patent is Intel Corporation. Invention is credited to Sunil K. Jain.
Application Number | 20140375766 14/476585 |
Document ID | / |
Family ID | 45021784 |
Filed Date | 2014-12-25 |
United States Patent
Application |
20140375766 |
Kind Code |
A1 |
Jain; Sunil K. |
December 25, 2014 |
METHOD AND APPARATUS FOR MAKING INTELLIGENT USE OF ACTIVE SPACE IN
FRAME PACKING FORMAT
Abstract
Active Space separates the Left Frame from the Right Frame in a
frame packing format. A source device is expected to transmit a
constant pixel value. A sink device is expected to ignore all data
received during the Active space regardless of the value.
Embodiments of the present invention involve the Source device to
embed intelligent information in the Active Space in lieu of the
recommended fixed pixel value. Then the Sink device may read the
embedded information from the Active Space, and infer the
transition between the left frame and the right frame which is
useful, for example, to synchronize eyewear.
Inventors: |
Jain; Sunil K.; (Portland,
OR) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Intel Corporation |
Santa Clara |
CA |
US |
|
|
Family ID: |
45021784 |
Appl. No.: |
14/476585 |
Filed: |
September 3, 2014 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
12827461 |
Jun 30, 2010 |
8842170 |
|
|
14476585 |
|
|
|
|
61350297 |
Jun 1, 2010 |
|
|
|
Current U.S.
Class: |
348/43 |
Current CPC
Class: |
H04N 13/341 20180501;
G09G 5/005 20130101; H04N 2213/008 20130101; G09G 2370/12 20130101;
G09G 2370/047 20130101; H04N 13/161 20180501; G09G 3/003 20130101;
G09G 2370/10 20130101; G09G 2370/042 20130101; G09G 2370/22
20130101; H04N 13/398 20180501; H04N 13/178 20180501 |
Class at
Publication: |
348/43 |
International
Class: |
H04N 13/04 20060101
H04N013/04 |
Claims
1. A method, comprising: providing a video stream having a left
active video frame and a right active video frame separated by an
active space; providing data in the active space, wherein the data
comprises unique device identification data; and inferring a
position of the left active video frame and a position of the right
active video frame by detection of data in the active space.
2. The method as recited in claim 1 wherein the video stream
comprises a high definition multimedia interface (HDMI) signal.
3. The method as recited in claim 1 wherein the data in the active
space comprises unique device identification data.
4. The method as recited in claim 2 where the data in the active
space comprises a temporal stamp.
5. The method as recited in claim 2 wherein the data in the active
space comprises a spatial stamp.
6. The method as recited in claim 2 further comprising: providing
one or more pairs of active shutter glasses to sync to a display by
the step of inferring the position in the video stream of the left
active video frame and the right active video frame by detection of
the data in the active space.
7. A system comprising: a source device to provide a video stream
having a left active video frame and a right active video frame
separated by an active space, wherein the source device provides
embedded data in the active space, and a position of the left
active video frame and a position of the right active video frame
is to be inferred by detection of data in the active space.
8. The system as recited in claim 7, further comprising a sink
device to receive the video stream and read the embedded data in
the active space.
9. The system as recited in claim 8, wherein the sink device infers
a position of the left active vide frame and a position of the
right active video frame from the position of the embedded
data.
10. The system as recited in claim 8 wherein the source device is a
high definition multimedia interface (HDMI) device and the sink
device is a HDMI device.
11. The system as recited in claim 10 wherein the source device
comprises a computer with a display.
12. The system as recited in claim 11 wherein the sink device
comprises active shutter glasses.
13. The system as recited in claim 12 wherein the active shutter
glasses synchronize to the display using the inferred position of
the left active vide frame and the inferred position of the right
active video frame from the position of the embedded data.
14. The system as recited in claim 8 wherein the embedded data in
the active space comprises unique device identification data.
15. The system as recited in claim 8 where the embedded data in the
active space comprises a temporal stamp.
16. The system as recited in claim 8 wherein the embedded data in
the active space comprises a spatial stamp.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application is a Continuation of U.S. application Ser.
No. 12/827,461, filed on Jun. 30, 2010, which claims priority to
U.S. Provisional Application Ser. No. 61/350,297, filed on Jun. 1,
2010, the contents of which are herein incorporated by
reference.
FIELD OF THE INVENTION
[0002] Embodiments of the present invention are directed to
stereoscopic systems and, more particularly, to eyewear systems
useful for three-dimensional (3D) viewing.
BACKGROUND INFORMATION
[0003] Humans have what is known as binocular vision owing to the
fact that we have two eyes separated by a couple of inches. Each
eye views the same scene from a slightly different perspective view
each providing the brain with slightly different information. These
two views are combined by the brain such that we perceive depth and
see the world in three-dimensions (3D).
[0004] Electronically stored or transmitted visual images, such as
pictures or video, is typically displayed on a two dimensional
medium such as a television screen or other type of monitor or
projected on to a screen. Both eyes view the same information. The
brain is thus left to use other visual cues from the
two-dimensional (2D) image, such as relative sizes of objects,
shadow, perspective lines, or horizons, to name a few, to sense
depth. However, the picture still looks flat and not like we see
the real world.
[0005] Stereoscopy refers to any of various processes and devices
for giving the illusion of depth from two-dimensional images. We
say illusion because true 3D may be more like a hologram where you
could walk around the image and change your perspective. However,
when done correctly, stereoscopy can trick the brain into thinking
objects are jumping out of the screen at you.
[0006] In its simplest form, two cameras, or one camera with two
lenses, spaced a few inches apart, are used to capture two 2D
images. Each 2D image, of course, is from a slightly different
perspective such that when the left eye views one image and the
right eye views the other, the brain combines the views and we see
the combined image as three-dimensional (3D).
[0007] Big screen stereoscopic motion pictures or "3D movies", as
is the term more commonly used, are becoming quite popular again.
In addition, 3D technologies are now available for home video with
the so-called 3D TVs, video games, and streaming and recorded video
content for computer monitor viewing.
[0008] There are several types of stereoscopic or "3D" technology
available. Most require the viewer to wear special glasses or
goggles. Some require active components in the glasses, others do
not. Some require special monitors or drivers. Each has it pros and
cons and, depending on the situation, may or may not make sense for
a specific task.
[0009] Regardless of the technology used, the end goal is primarily
to separate what the left and the right eye sees. Early
technologies involved physical separation where a viewer looked
into a binocular-like device, with a lens for each eye to
physically separate the left and right views. This technique which
may be the oldest, works quite well and a close variation of this
technique is still used in modern virtual reality goggles or
head-mounted displays. However, this is only good for one person or
individual viewing and may be expensive or impractical for more
than a couple viewers.
[0010] One of the first left/right (L/R) separation technologies
good for the masses was spectral separation. The technical term is
"color anaglyph" and involved each viewer wearing a pair of glasses
with a red filter for one eye and a blue filter for the other. The
left and right images were likewise blue or red encoded and
displayed simultaneously. This technique was popular for producing
3D movies in the 1950s and even works to some degree with standard
color televisions or monitors. While providing a novelty for its
day, it left much to be desired aesthetically. The end result
tended to be monochromatic, and had a lot of ghosting (i.e. the L/R
separation was not clean). On the pro side, it was inexpensive to
produce and the glasses were passive and very inexpensive.
[0011] Similar to spectral separation, the next most common
technique is spatial separation and involves the viewers wearing
polarized glasses, with each eye lens being polarized at 45
degrees, for example, to the other or circularly polarized in
opposite directions. This is the technology used most often today
in movie theaters. It works pretty well with the L/R separation
being fairly complete, but usually requires two projectors or a
special projector in a theatre setting or a few additional layers
in a monitor which adds cost. Also, each eye only sees half
resolution which may degrade the viewing experience. On the pro
side, the polarized glasses are again passive and therefore
relatively inexpensive.
BRIEF DESCRIPTION OF THE DRAWINGS
[0012] The foregoing and a better understanding of the present
invention may become apparent from the following detailed
description of arrangements and example embodiments and the claims
when read in connection with the accompanying drawings, all forming
a part of the disclosure of this invention. While the foregoing and
following written and illustrated disclosure focuses on disclosing
arrangements and example embodiments of the invention, it should be
clearly understood that the same is by way of illustration and
example only and the invention is not limited thereto.
[0013] FIG. 1 is a platform, such as a laptop or other device for
viewing 3D video;
[0014] FIG. 2 is a timing diagram showing the vertical and
horizontal timing parameters plotted along their respective axis
along an illustration of an active video display area;
[0015] FIG. 3 is a timing diagram showing the video timing
parameters, similar to FIG. 2, plotted along the same time
axis;
[0016] FIG. 4 are High Definition Multimedia Interface (HDMI)
timing diagrams illustrating 2D frames and frame stacking for 3D
video; and
[0017] FIG. 5 is a block diagram showing features added to a
graphics driver to discover if content is 2D or 3D and apply
content to display format matching for best viewing quality.
DETAILED DESCRIPTION
[0018] Reference throughout this specification to "one embodiment"
or "an embodiment" means that a particular feature, structure, or
characteristic described in connection with the embodiment is
included in at least one embodiment of the present invention. Thus,
the appearances of the phrases "in one embodiment" or "in an
embodiment" in various places throughout this specification are not
necessarily all referring to the same embodiment. Furthermore, the
particular features, structures, or characteristics may be combined
in any suitable manner in one or more embodiments.
[0019] As discussed above, both spectral separation and spatial
separation techniques use passive glasses having various filter
technologies to separate the left and right eye images. Temporal
separation involves active glasses technology. Active glasses may
be glasses that alternately black out the left and right eyes as
the display alternately displays the left and right images.
[0020] For example, active glasses may be called shutter glasses or
shutter goggles and may have liquid crystal displays (LCDs) in each
eye that are caused to turn black every other frame to block, say
the left eye, from seeing the right image and vice-versa for the
next frame. These work well, but you get half the frames in one
eye, and half the frames in the other. This can lead to headaches
as your eyes try to compensate for the on/off light, low frame
rates for gamers and synch problems between the display and the
glasses.
[0021] FIG. 1 illustrates one system for 3D viewing using active
glasses. As shown a computer 100 or other monitor has a display
screen 101 for displaying 3D video content using temporal
separation. The video content source may be recorded such as on CD
or stored in a memory, streamed over a network, such as the
internet, over the air or cable broadcasts, or any other suitable
video delivery method.
[0022] Active glasses 102 typically use a transmitter 104 that is
synced to the frame rate of the display 101. This transmitter 104
may use a number of wireless 106 technologies to communicate with
the eyewear or glasses 102, such as infra red (IR), Radio frequency
(RF), Bluethooth, etc. The transmitter 104 may be tethered to the
computer 100 by a cable 108, such as a universal serial bus (USB)
or USB2 cable.
[0023] In stereoscopic systems, it is desirable that eyewear
shuttering be tightly coupled to the time when image is stable on
the display 101. Video Electronics Standards Association (VESA)
standard timings may not allow sufficient stable image time if the
image is changing at each frame.
[0024] Prior solutions help a bit by allowing for the extended
vertical blank timing. This may improve the image stability to a
maximum period of about 32%. Further, due to small duty cycle,
average perceived brightness on the display viewed through active
shutter glasses is pretty low. For smaller screen size devices such
as note books or netbooks, the problem of 3D viewing quality gets
exacerbated.
[0025] FIGS. 2 and 3 both illustrate video timing parameters. FIG.
2 shows the active video are 200 comprising a border or margin 202
and addressable video area (Addr Time) 204 with the vertical timing
parameters fashioned along the vertical axis if the active video
200 and the horizontal timing parameters fashioned along the
horizontal axis of the active video 200 for ease of illustration.
FIG. 3 is similar to FIG. 2 but that it shows essentially the
vertical and horizontal video timing parameter along the same axis
as they may occur in real time.
[0026] The video timing parameters of either FIG. 2 or FIG. 3 may
be defined as follows:
[0027] Horizontal Addressable Video is the time between the end of
the Left Border and the beginning of the Right Border.
[0028] Horizontal Blanking is the time between the end of the Right
Border and the beginning of the Left.
[0029] A Border includes the Horizontal Front Porch time, the
Horizontal Sync Pulse Width time and the Horizontal Back Porch
time.
[0030] Horizontal Front Porch is the time between the end of the
Right Border and the beginning of the Horizontal Sync Pulse.
[0031] Horizontal Left Border is the time between the end of the
Horizontal Blanking period and the beginning of the Horizontal
Addressable Video region.
[0032] Horizontal Right Border is the time between the end of the
Horizontal Addressable Video region and the beginning of the
Horizontal Blanking period.
[0033] Horizontal Sync Pulse Width is the time between the end of
the Horizontal Front Porch and the beginning of the Horizontal Back
Porch.
[0034] Horizontal Back Porch is the time between the end of the
Horizontal Sync Pulse and the beginning of the Left Border.
[0035] Horizontal Active Video is the sum of the Horizontal Left
Border time, the Horizontal Addressable Video time and the
Horizontal Right Border time.
[0036] Vertical Addressable Video is the time between the end of
the Top Border and the beginning of the Bottom Border.
[0037] Vertical Blanking is the time between the end of the Bottom
Border and the beginning of the Top Border and may include the
Vertical Front Porch time, the Vertical Sync Pulse Width time and
the Vertical Back Porch time.
[0038] Vertical Front Porch is the time between the end of the
Bottom Border and the beginning of the Vertical Sync Pulse.
[0039] Vertical Top Border is the time between the end of the
Vertical Blanking period and the beginning of the Vertical
Addressable Video region.
[0040] Vertical Bottom Border is the time between the end of the
Vertical Addressable Video region and the beginning of the Vertical
Blanking period.
[0041] Vertical Sync Pulse Width is the time between the end of the
Vertical Front Porch and the beginning of the Vertical Back
Porch.
[0042] Vertical Back Porch is the time between the end of the
Vertical Sync Pulse and the beginning of the Top Border.
[0043] Vertical Active Video is the sum of the Vertical Top Border
time, the Vertical Addressable Video time and the Vertical Bottom
Border time.
[0044] Referring now to FIG. 4 there is shown timing diagrams
comparing 2D video and 3D video frame stacking according to one
High Definition Multimedia Interface (HDMI) standard. A 2D frame is
shown on the left and a 3D frame is shown on the right. Note the 3D
horizontal total pixel is equal the 2D horizontal total pixel. The
3D vertical total line is twice the 2D vertical total line. The 3D
pixel clock frequency is twice the 2D pixel clock frequency. This
structure may be applied to the progressive video format.
[0045] Obtaining accurate information about Left/Right (L/R) frame
placement and frame change has been a technological challenge so
far for the conventional stereoscopic-3D systems. That is why, the
conventional active liquid crystal shutter based 3D Eyewear (as
shown, for example in FIG. 1) typically provides a switch on the
glasses 102 that manually flips between left and right frame
synchronization to avoid ghosting. It would be desirable to
eliminate this switch and provide this function automatically.
Embodiments of this invention solves this problem and provides a
low cost mechanism not only to provide this function, but also
provide more information that can be useful for many other
purposes.
[0046] As shown in FIG. 4, by definition the HDMI Active Space 400
separates the Left Frame from the Right Frame. Per standard
specifications, during the Active space 400, an HDMI source (e.g. a
PC or other viewing device) is expected to transmit a constant
pixel value. HDMI sink (e.g. the active shutter glasses) is
expected to ignore all data received during the Active space 400
regardless of the value. Embodiments of the present invention
involve the Source device to embed intelligent information in the
Active Space in lieu of the recommended fixed pixel value. Then on
the Sink side, this invention causes the Sink device to read the
embedded information from the Active Space, and infer to cause
further useful actions, such as synching of eyewear of panel
optics, etc.
[0047] Typically Active Space 400 is one or more vertical lines
with large number of horizontal pixels depending upon the frame
resolution, plenty of useful information such as unique device id,
temporal and spatial stamps, etc. may be embedded in these packets,
opening a plethora of applications in
multi-transmitter-multi-viewer scenarios that are not possible for
enjoying personal 3D today (e.g. multiple 3D laptop users in a room
with individual glasses looking at their own screen, projector and
their neighbor's screen).
[0048] Because the placement of Active Space 400 is fixed by the
HDMI spec (in between the left and the right frames), intelligence
embedded in the Active Space implicitly also conveys position of
the left and right frame deterministically (even though that is not
the original intent of the Spec). Thus this innovation presents a
very powerful infrastructure for creating high quality
Stereoscopic-3D systems with and without eyewear.
[0049] In one implementation, high quality Stereoscopic 3D eyewear
is possible that is not only synched to the frame timing but also
to left/right placement.
[0050] In another embodiment it is possible to aid the optical
elements of autosterescopic displays (without needing special
eyewear) to synch with left/right frame info or the info about
multiple views that can also be embedded in the active space.
[0051] In yet another embodiment, it is possible to transmit a
predetermined custom pixel color that can be decoded by the eyewear
directly.
[0052] With this invention, deterministic left/right frame
placement information can be provided in a novel in-band method,
that substitutes USB peripheral with a low cost embedded emitter,
eliminates the switch on the eyewear, and makes
multi-transmitter-multi-viewer with eyewear scenario possible. The
invention puts the Active Space 400 mandated by the HDMI spec to a
useful purpose, that was otherwise being ignored as the spec
suggests. Compliance to the spec doesn't break because this
information is picked, such as by a garbage collector, and recycled
for useful purpose.
[0053] In yet another embodiment, stereoscopic content playback
generally involves special players that combine the two video
streams into a desired output 3D format. Many such players may
become available in the market as stereoscopic playback catches
market interest. However because these players are special, these
add extra cost to the 3D PC, and perhaps more importantly these
limit the choices of end user to enjoy 3D content to specific
hardware with specific software. Validation and compliance of many
such commercial solutions is yet another costly vector. According
to one embodiment, this invention may enable almost any
conventional 2D media player 3D playback capable on a mainstream
PC.
[0054] FIG. 5 shows a block diagram according to one embodiment.
Embodiments may include a PC comprising CPU cores 502 and LLC 504
and SA 506 and a 2D 508 connected to a display 210. A Media App 512
may be connected to a DirectShow codec 514 which may feed into a
DXVA 516 and a GFX driver 518. The GFX driver may connect to a
media pipe 520 which may include a GT EU or LRBx Cores 522. A DRAM
524 may communicate with the PC 500. A process with may be stored
in the DRAM 524 may include Encrypted/compressed video data in
block 526 which is decrypted and decoded to a frame in block 528.
There after the frame may be processed into a post processed frame
in block 530 and then a App buffer after composition at clock 532.
Block 532 and a frame buffer (desktop) 534 may be combined into a
blended frame buffer at block 536. A font frame buffer 538 may also
be included.
[0055] An embodiment may be realized as middleware that can be most
effective if implemented as an extension to the graphics driver 518
(however stand-alone middleware application is also possible). The
GFX driver 518 may be modified to 1) Auto detector of 2D/3D content
2) Content to Display format matching-adapting algorithm 3)
Multiplexer for the stereo video streams.
[0056] In short, embodiments automatically discover if the content
if content to be displayed 2D or 3D. Detection may be simple, for
example, a typical windows media player on mainstream PC will open
two independent windows if stereo content is played back. Then the
PC system configuration and content formats being played back are
discovered. Typical Gfx driver has this information from display
EDID and media player parameters. Then via a video stream combiner
that is implemented as an extension of the Gfx driver, apply
content-to-display format matching algorithms and adapt the content
for best viewing quality for the user. This method can be applied
to most general purpose media players in mainstream PC.
[0057] By definition, since high level media player applications
are typically designed to be hardware agnostic, these are not
optimized to specific hardware. In contrast, embodiments of this
invention are designed to reside in the graphics driver instead,
and thus make use of the hardware level information available to
the driver. Being middleware this invention is unique to take
advantage of best of both sides: codec issues are handled by high
level media player while display format and quality issues are
handled by the driver and its hardware underneath. The middleware
thus makes a typical 2D media player behave as a high quality
stereo video player rendering to 3D display.
[0058] The above description of illustrated embodiments of the
invention, including what is described in the Abstract, is not
intended to be exhaustive or to limit the invention to the precise
forms disclosed. While specific embodiments of, and examples for,
the invention are described herein for illustrative purposes,
various equivalent modifications are possible within the scope of
the invention, as those skilled in the relevant art will
recognize.
[0059] These modifications can be made to the invention in light of
the above detailed description. The terms used in the following
claims should not be construed to limit the invention to the
specific embodiments disclosed in the specification and the claims.
Rather, the scope of the invention is to be determined entirely by
the following claims, which are to be construed in accordance with
established doctrines of claim interpretation.
* * * * *