U.S. patent application number 14/377500 was filed with the patent office on 2016-01-21 for method and device for inserting a 3d graphics animation in a 3d stereo content.
The applicant listed for this patent is THOMSON LICENSING. Invention is credited to Valter DRAZIC, Anita ORHAND, Alain VERDIER.
Application Number | 20160019724 14/377500 |
Document ID | / |
Family ID | 46458218 |
Filed Date | 2016-01-21 |
United States Patent
Application |
20160019724 |
Kind Code |
A1 |
VERDIER; Alain ; et
al. |
January 21, 2016 |
METHOD AND DEVICE FOR INSERTING A 3D GRAPHICS ANIMATION IN A 3D
STEREO CONTENT
Abstract
The invention concerns a method and a device for inserting 3D
graphic animation in a 3D image, each 3D graphic element of the
graphic animation being defined in size and in depth for the
insertion in a determined insertion zone of said 3D image. The
method comprises the step of determining for the graphic element to
be inserted a depth range with a maximum allowed depth value,
replacing the out of range depth values by the maximum allowed
depth value when depth values of the graphic element are out of
range and compensating the depth difference between the depth
values of the graphic element and the maximal allowed depth value
in reducing the graphic element in size proportionally to the
reduction of depth for the graphic element.
Inventors: |
VERDIER; Alain; (Vern Sur
Seiche, FR) ; ORHAND; Anita; (Rennes, FR) ;
DRAZIC; Valter; (Betton, FR) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
THOMSON LICENSING |
Issy de Moulineaux |
|
FR |
|
|
Family ID: |
46458218 |
Appl. No.: |
14/377500 |
Filed: |
February 6, 2013 |
PCT Filed: |
February 6, 2013 |
PCT NO: |
PCT/EP13/52300 |
371 Date: |
August 7, 2014 |
Current U.S.
Class: |
345/419 |
Current CPC
Class: |
H04N 13/275 20180501;
H04N 13/128 20180501; G06T 13/20 20130101; G06T 19/20 20130101;
G06T 2200/04 20130101; H04N 13/183 20180501; H04N 13/122 20180501;
H04N 13/156 20180501 |
International
Class: |
G06T 19/20 20060101
G06T019/20; G06T 13/20 20060101 G06T013/20 |
Foreign Application Data
Date |
Code |
Application Number |
Feb 13, 2012 |
EP |
12305157.5 |
Jun 28, 2012 |
EP |
12174023.7 |
Claims
1. A method of inserting graphic animation in a 3D image, each 3D
graphic element of the graphic animation being defined in size and
in depth for the insertion in a determined insertion zone of said
3D image; wherein the method comprises the steps of Determining for
the graphic element to be inserted a maximal allowed depth value
corresponding to the minimum depth value of the image in the
determined insertion zone so that the graphic element is not
overlapped by the corresponding part of the 3D image in said
predetermined insertion zone; When a depth value of the graphic
element to be inserted is superior to the maximal allowed depth
value, keeping the depth value of the graphic element by the
maximal allowed depth value; and compensating the depth difference
between the depth value of the graphic element to be inserted which
is superior to the maximal allowed depth value and the maximal
allowed depth value in reducing the graphic element in size
proportionally to the reduction of depth for the graphic
element.
2. The method of claim 1 wherein it comprises furthermore the step
of determining a bounding box arounding the graphic element so that
the size and the depth value of the bounding box replace the size
and depth value of the graphic element.
3. The method of claim 2 wherein the size of the graphic element is
determined by the maximal length and width values of the graphic
element.
4. The method of 1 wherein the insertion is applied to post
production environment.
5. The method of claim 1 wherein the insertion is applied to
consumer products.
6. A device for inserting graphic animation in a 3D image, each 3D
graphic element of the graphic animation being defined in size and
in depth for the insertion in a determined insertion zone of said
3D image wherein the device comprises: means for determining for
the graphic element to be inserted a maximal allowed depth value
corresponding to the minimum depth value of the image in the
determined insertion zone so that the graphic element is not
overlapped by the corresponding part of the 3D image in said
predetermined insertion zone; means for keeping the depth value of
the graphic element by the maximal allowed depth value when a depth
value of the graphic element to be inserted is superior to the
maximal allowed depth value, and means for compensating the depth
difference between the depth value of the graphic element to be
inserted which is superior to the maximal allowed depth value and
the maximal allowed depth value in reducing the graphic element in
size proportionally to the reduction of depth for the graphic
element.
Description
[0001] The present invention relates to a method and device for
inserting a 3D graphics animation in a 3D stereo content. The
invention applies mainly to post-production environment as it
concerns graphic insertion but also to consumer products including
STB, BD-player, TV sets, Smart phones, Tablets with 3D
capabilities. The invention can be used every time a 3D Graphics
content is generated to be displayed on a 3D display.
[0002] The invention aims at resolving the problem of depth
perception conflicts when overlaying animated graphics on top of 3D
video layer. Indeed, if the occlusion clues do not match the depth
cues, it creates a visual discomfort, cause of headache or nausea.
When the 3D graphics is animated in the depth direction it is even
more likely that a conflict would occur.
[0003] There are a great number of patents and articles describing
how to avoid overlap when mixing two stereo contents: most of the
time, one is video (real image, primary), the other is graphics
(virtual image, secondary). Usually, it is proposed to shift one of
the two contents so there is no more overlap. Sometimes it is
suggested to compress the depth range because it is not always
possible to keep the two incoming contents within the comfort range
(convergence/accommodation issue). WO2008038205A2(Philips) and
US20110199459NV describe such method.
[0004] The invention aims to improve the use case where there is an
animation that makes a graphic element to appear or disappear. With
2D display it is quite simple, because in 2D display a visual
perception conflict can never occur, a 2.5D animation (3D animated
but projected onto a single 2D window) gives good results: the
graphics disappear because of the depth that goes to infinite
during the 3D animation. On 3D display, it is likely that the
element will conflict with other objects of the real scene, at
least with the background (probably not at infinite) but also
likely from time to time with other objects of the real scene that
might be at screen level or even popping out of the screen.
[0005] It is an object of the invention to address the above
mentioned problems of overlapping of graphic animation with objects
of the 3D scene.
[0006] The invention consists in a method of inserting graphic
animation in a 3D image, each 3D graphic element of the graphic
animation being defined in size and in depth for the insertion in a
determined insertion zone of said 3D image. The method comprises a
step of determining for the graphic element to be inserted a
maximal allowed depth value corresponding to the minimum depth
value of the image in the determined insertion zone so that the
graphic element is not overlapped by the corresponding part of the
3D image in said predetermined insertion zone further, the method
comprises a step of, when a depth value of the graphic element to
be inserted is superior to the maximal allowed depth value, keeping
the depth value of the graphic element by the maximal allowed depth
value and compensating the depth difference between the depth value
of the graphic element to be inserted which is superior to the
maximal allowed depth value and the maximal allowed depth value in
reducing the graphic element in size proportionally to the
reduction of depth for the graphic element.
[0007] The use of the invention will show-up a change in the
graphics size in the 2D space while the disparity is kept
constant.
[0008] In a preferred embodiment, the method comprises furthermore
the step of determining a bounding box arounding the graphic
element so that the size and the depth value of the bounding box
replace the size and depth value of the graphic element. In a
preferred embodiment a bounding box arounding different graphic
elements moving together is defined.
[0009] In a preferred embodiment the maximum allowed depth value is
determined so that the overlapping of graphic element by objects of
the 3D image is avoided.
[0010] In a preferred embodiment the maximum allowed depth value is
selected by a spectator.
[0011] In a preferred embodiment the method of inserting graphic
animation in a 3D image is applied to post production
environment.
[0012] In a preferred embodiment the method of inserting graphic
animation in a 3D image applies to consumer products.
[0013] In a preferred embodiment the maximum allowed depth for at
least a graphic element to be inserted is determined so that
overlapping of the 3D image is avoided.
[0014] The invention concerns, too, a device for inserting graphic
animation in a 3D image, each graphic element of the graphic
animation being defined in size and in depth for the insertion in a
determined insertion zone of said 3D image. The device comprises
means for determining for the graphic element to be inserted a
maximal allowed depth value corresponding to the minimum depth
value of the image in the determined insertion zone so that the
graphic element is not overlapped by the corresponding part of the
3D image in said predetermined insertion zone; means for keeping
the depth value of the graphic element by the maximal allowed depth
value when a depth value of the graphic element to be inserted is
superior to the maximal allowed depth value, and means for
compensating the depth difference between the depth value of the
graphic element to be inserted which is superior to the maximal
allowed depth value and the maximal allowed depth value in reducing
the graphic element in size proportionally to the reduction of
depth for the graphic element.
[0015] The invention has the advantage that there are no more
visual perception conflicts on appearing or disappearing graphics.
By the way it preserves the original stereo video content as no
shift is done to prevent overlapping of the video content.
[0016] The features and advantages of the invention will be further
explained upon reference to following drawings in which
[0017] FIG. 1 illustrates an implementation of the invention;
[0018] and FIG. 2 illustrates plane and associated depth to explain
the invention.
[0019] The invention proposes a method that changes the geometrical
transformation issued from the initial animation scenario into a
different one keeping the graphics element within a safe depth
range assuring no overlap with real scene depth budget while giving
the impression to have the graphic element going from/to the
infinite (size=0). This is achieved by updating the scale
parameters together with the translation parameters. Scale
parameters are defined as the maximal length and maximal width of
the graphic element. Translation parameter is the depth value. The
graphic element is thus defined in size with the scale
parameters.
[0020] The 3D graphics are usually modeled by a 3D modeling tool.
Different modeling tools such as Blender, Cinema4D, Maya and
AutoDesk are well known.
[0021] These authoring tools can export a model of the virtual
scene in format as per example, xml, x3d, vrml, dae, dxf, fbx or
obj.
[0022] The model is basically described as a tree containing a
number of elements, each of them is defined as a basic shape
associated to a color or texture and a transformation function such
as translation, rotation or scaling.
[0023] Implementation of the invention, as represented by FIG. 1,
takes as inputs: [0024] a model file which contains the
specification of the 3D graphics. [0025] an animation file:
contains the specification of the animation, basically the
transform between entry point and ending point such as translation,
rotation or scaling, the number of frames and the speed of the
animation. [0026] the incoming stereo video .
[0027] It outputs the composited video formed by the combination of
the 3D graphics and the incoming stereo video.
[0028] It contains the following modules: [0029] a x3d file parser
module: this module reads the file and build the graphics object
element by element, using OpenGL functions (vertex building and
transform operations). [0030] A disparity estimator module: this
module computes a sparse disparity map and issue the minimum value
over a programmable area. This area can be the full screen but more
likely the part of the screen where the graphics is supposed to be
inserted. [0031] An animation builder module: this module check
whether the animated graphics depth will overlap with the real
video or not. If not, the parameters included in the animation file
are used as is, otherwise the parameters are changed. [0032] A
graphics renderer module: this is the module responsible for the
rendering. It takes the animation script from the animation builder
and executes OpenGL operations accordingly. [0033] A composition
module permits to combine the corrected graphic element to the 3D
stereo video image
[0034] As the estimator runs on the incoming left and right 2D
images, it does not provide directly the depth value but rather a
value of disparity between left and right pictures. The relation
that links the disparity value to the depth value is well known for
a person skilled in the art.
[0035] If we know the setting of virtual graphics in term of focal
length, baseline and depth of convergence plane we can get the
resulting disparity for every graphic element. If we know the farer
depth z (obtained after vertex building and transformation) then we
can get the equivalent disparity and compare with the one provided
by the disparity estimator.
[0036] The correction method is the method used by the Animation
Builder to change the initial animation script so as to prevent for
visual perception conflicts. For every frame the Animation Builder
shall check over the resulting disparity for the graphics.
[0037] In case the depth (disparity) is bigger than the one from
the depth (disparity) estimator, then a correction is needed.
[0038] As illustrated with FIG. 2, instead of pushing backward the
graphic element, it is rather reduced in size determined by the
length and the width of the graphic element and kept at the maximum
allowed depth corresponding to zmax (z=4) on the figure. The depth
range is thus limited by the maximum depth value zmax.
[0039] The following diagram shows how to get the illusion of an
object going far away by reducing the size of the object. The
relation between size on the screen zconv (z=3) and actual size and
depth z (z=7) is the following:
l/L=Zmax/Z
[0040] With: [0041] L: initial object length size [0042] I:
corrected length size [0043] Zmax: maximum allowed depth
[0044] The width of the object will be adapted on the same way.
[0045] The two rectangles at zmax (z=4) represent a smaller object
that would have been perceived with the same size on the 2D screen
from respectively the left (circle at z=0) and right eye (circle at
z=0) compared to the original object located at a depth of z (z=7)
Obviously only one object shall be rendered, centered with respect
to the object located at zmax but with a scale factor of
zmax/z.
[0046] In an embodiment of the invention, for example, at
post-production level, graphic inserts can add a logo or indicate a
score/player name in case of sports or also name/title for a
singer.
[0047] At consumer device level, this can be the user interface
that requires graphic insertion, either to browse a menu, or to get
additional data linked to the content (EPG) or to the user
preference as for example: widget for social network, weather
forecast, stock market.
[0048] In an embodiment of the invention, a script relative to the
graphic insertion and its own depth can determine the depth of this
insertion depending of various functions or various scenarios. In
an embodiment of the invention, the graphic should disappear in the
horizon.
[0049] In another embodiment, the depth of a graphic can evolve in
function of the time. It means that the depth is a linear (or not)
and function of the time.
[0050] The method consists to render an animated 3D graphics
element by changing the element size instead of the depth value, if
the depth value is over the maximum allowed depth. Thus a
compensation of the depth difference between the depth of the
graphic element and the maximal allowed depth value is done in
reducing the graphic element in size proportionally to the
reduction of depth for the graphic element.
[0051] The determination of a bounding box arounding the graphic
element will permit to consider the depth value, the length value
and the width value of the bounding box at the place of the graphic
element. There is a possibility for defining a bounding box for
different graphic elements moving together inside of the bounding
box and to consider the insertion of this bounding box at the place
of the different graphic elements.
[0052] The depth, length and width values are respectively maxima
value of the depth, length and width values of the graphic element
or a volume surrounding the different graphic elements or a volume
surrounding part of the graphic elements.
[0053] The depth, length and width values of the graphic element or
of the bounding box are determined data imported with the data
corresponding to the graphic element.
[0054] The maximal depth value of a 3D element corresponds to the
depth value of the pixel representing the most distant point of the
element. As the element being defined in size by its length, width
and depth values, modification of the maximal depth value are
calculated so that the proportion in size of the graphic element is
respected.
[0055] Thus if a scenario indicates that a graphic element is
moving in a 3D image, as soon as the maximal depth value reach the
given limits, the size of this element will be modified in
proportion to the depth value for the graphic element indicated by
the scenario.
[0056] The depth value is fixed relative to the 3D depth of the
screen displaying the 3D image to a spectator.
* * * * *