U.S. patent application number 12/753032 was filed with the patent office on 2010-10-14 for web platform for interactive design, synthesis and delivery of 3d character motion data.
This patent application is currently assigned to Mixamo, Inc.. Invention is credited to Stefano Corazza, Edilson de Aguiar, Emiliano Gambaretto.
Application Number | 20100259547 12/753032 |
Document ID | / |
Family ID | 43973908 |
Filed Date | 2010-10-14 |
United States Patent
Application |
20100259547 |
Kind Code |
A1 |
de Aguiar; Edilson ; et
al. |
October 14, 2010 |
WEB PLATFORM FOR INTERACTIVE DESIGN, SYNTHESIS AND DELIVERY OF 3D
CHARACTER MOTION DATA
Abstract
Systems and methods are described for animating 3D characters
using synthetic motion data generated by motion models in response
to a high level description of a desired sequence of motion
provided by an animator. In a number of embodiments, the synthetic
motion data is streamed to a user device that includes a rendering
engine and the user device renders an animation of a 3D character
using the streamed synthetic motion data. In several embodiments,
an animator can upload a custom model of a 3D character or a custom
3D character is generated by the server system in response to a
high level description of a desired 3D character provided by the
user and the synthetic motion data generated by the generative
model is retargeted to animate the custom 3D character.
Inventors: |
de Aguiar; Edilson; (El
Cerrito, CA) ; Gambaretto; Emiliano; (San Francisco,
CA) ; Corazza; Stefano; (San Francisco, CA) |
Correspondence
Address: |
KAUTH , POMEROY , PECK & BAILEY ,LLP
2875 MICHELLE DRIVE, SUITE 110
IRVINE
CA
92606
US
|
Assignee: |
Mixamo, Inc.
San Francisco
CA
|
Family ID: |
43973908 |
Appl. No.: |
12/753032 |
Filed: |
April 1, 2010 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
12370269 |
Feb 12, 2009 |
|
|
|
12753032 |
|
|
|
|
61166117 |
Apr 2, 2009 |
|
|
|
Current U.S.
Class: |
345/474 |
Current CPC
Class: |
G06T 13/20 20130101;
G06T 13/40 20130101; G06T 2200/16 20130101 |
Class at
Publication: |
345/474 |
International
Class: |
G06T 15/70 20060101
G06T015/70 |
Claims
1. An animation system, comprising: a server system connected to a
communication network and configured to communicate with a user
device that is also connected to the communication network; wherein
the server system is configured to generate a user interface that
is accessible via the communication network; wherein the server
system is configured to receive a high level description of a
desired sequence of motion via the user interface; wherein the
server system is configured to generate synthetic motion data using
at least one motion model resident on the server system based on
the high level description of the desired sequence of motion
received via the user interface; wherein the server system is
configured to stream the synthetic motion data to a rendering
engine on the user device; and wherein the user device is
configured to render a 3D character animated using the streamed
synthetic motion data.
2. The animation system of claim 1, wherein: the server system
comprises an application server and a web server that are
configured to communicate; the application server is configured to
communicate with a database of motion data; the web server is
connected to the communication network; the application server is
configured to generate the at least one motion model using the
motion data and provide the at least one motion model to the web
server; the web server is configured to generate the user interface
that is accessible via the communication network; the web server is
configured to receive the high level description of the desired
sequence of motion via the user interface; the web server is
configured to use the at least one motion model to generate
synthetic motion data based on the high level description of the
desired sequence of motion; and the web server is configured to
stream the synthetic motion data to a rendering engine on the user
device.
3. The animation system of claim 1, wherein the motion data
includes actual motion data obtained via motion capture.
4. The animation system of claim 3, wherein the motion data is
obtained via marker based motion capture.
5. The animation system of claim 3, wherein the motion data
includes meshes obtained using markerless motion capture.
6. The animation system of claim 1, wherein the motion data
includes manually generated motion data.
7. The animation system of claim 1, wherein the web server is
configured to generate synthetic motion by: determining a time base
based on the high level description of the desired sequence of
motion; normalizing a number of different motions to the time base;
determining weightings for each of the motions based on the high
level description of the desired sequence of motion; and
interpolating the normalized motions using the determined
weightings.
8. The animation system of claim 7, wherein: the synthetic motion
data is based upon a standard model for a 3D character; the server
system is configured to receive a model of a 3D character from the
user device via the communication network; the server system is
configured to retarget the synthetic motion data to the received
model of a 3D character; and the synthetic motion data streamed by
the server system is the retargeted synthetic motion data.
9. The animation system of claim 7, wherein: the motion data is
based upon a standard model for a 3D character; the server system
is configured to generate a model of a 3D character from a high
level description of a 3D character received via the user
interface; the server system is configured to retarget the
synthetic motion data to the generated model of a 3D character; and
the synthetic motion data streamed by the server system is the
retargeted synthetic motion data.
10. The animation system of claim 7, wherein the server system is
configured to impose environmental constraints on the synthetic
motion data.
11. The animation system of claim 10, wherein the server system is
configured to modify the synthetic motion data to constrain a
certain aspect of the motion to occur during a particular period of
the time base.
12. The animation system of claim 10, wherein the server system is
configured to modify the synthetic motion data to prevent portions
of the 3D character from penetrating a floor during the motion.
13. The animation system of claim 7, wherein the server system is
configured to use the motion model to generate synthetic motion
data from any description within the continuum of the scale of the
high level characteristic.
14. The animation system of claim 1, wherein the high level
description of the desired sequence of motion includes at least a
motion type, a trajectory for the motion, and at least one motion
styling.
15. The animation system of claim 14, wherein the motion styling is
a description of a physical characteristic or an expressive
characteristic of the motion.
16. The animation system of claim 14, wherein the motion system is
expressed as one of a number of discrete types of motion.
17. The animation system of claim 14, wherein the trajectory of the
motion is specified including at least a start point and an end
point.
18. The animation system of claim 14, wherein the motion styling is
expressed using at least one value from a continuous scale that
corresponds to a high level characteristic that was used to
describe differences between repeated sequences of motion in the
motion data used to generate the motion models.
19. The animation system of claim 1, wherein: the server system is
configured to receive an updated high level description of the
desired sequence of motion over the communication network via the
user interface; the server system is configured to use the motion
model to generate a second set of synthetic motion data based on
the updated high level description of the desired sequence of
motion; and the server system is configured to stream the second
set of synthetic motion data to a rendering engine on the user
device.
20. The animation system of claim 1, wherein the rendering engine
is a plug-in to a browser application on the user device that is
used to interact with the user interface generated by the server
system.
21. The animation system of claim 1, wherein: the motion data is
based upon a standard model for a 3D character; the server system
is configured to receive a model of a 3D character from the user
device via the communication network; the server system is
configured to retarget the synthetic motion data generated by the
motion model based upon the high level description of the desired
sequence of motion to animate the 3D character received from the
user device; and the synthetic motion data streamed by the server
system is the retargeted synthetic motion data.
22. The animation system of claim 1, wherein: the motion data is
based upon a standard model for a 3D character; the server system
is configured to generate a model of a 3D character from a high
level description of a 3D character received via the user
interface; the server system is configured to retarget the
synthetic motion data generated by the motion model based upon the
high level description of the desired sequence of motion to animate
the 3D character generated by the server system; and the synthetic
motion data streamed by the server system is the retargeted
synthetic motion data.
23. A method of animating a 3D character, comprising: receiving a
high level description of the motion characteristics for a desired
motion sequence from a user device via a user interface generated
by a server system; generating synthetic motion data using the
server system based upon the at least one motion model and the high
level description of the motion characteristics for a desired
motion sequence; streaming the synthetic motion data from the
server system to the user device; and rendering an animation of a
3D character using the client device based upon the streamed
synthetic motion data.
24. The method of animating a 3D character of claim 23, wherein
generating synthetic motion data based upon the at least one motion
model and the high level description of the motion characteristics
for a desired motion sequence further comprises: determining a time
base based on the high level description of the motion
characteristics for a desired motion; normalizing the motion data
of a number of different motions included in the at least one
motion model to the time base; determining weightings for the
motion data of each of the motions based on the high level
description of the motion characteristics for a desired motion; and
interpolating the normalized motion data using the determined
weightings to generate synthetic motion data.
25. The method of animating a 3D character of claim 24, further
comprising: obtaining a custom model of a 3D character; and
retargeting the synthetic motion data to the user model prior to
streaming the synthetic motion data to the user device.
26. The method of animating a 3D character of claim 25, wherein the
custom model of a 3D character is uploaded to the server system
from the user device.
27. The method of animating a 3D character of claim 25, wherein the
custom model of a 3D character is generated by the server system in
response to a high level description of the 3D character obtained
from the user device via a user interface generated by the server
system.
28. The method of animating a 3D character of claim 25, further
comprising imposing environmental constraints on the retargeted
synthetic motion data.
Description
RELATED APPLICATION
[0001] The current application is a continuation-in-part of U.S.
patent application Ser. No. 12/370,269 filed Feb. 12, 2009 and
claims priority to U.S. Provisional. Application No. 61/166,117,
filed Apr. 2, 2009 the disclosure of which is incorporated herein
by reference.
FIELD OF THE INVENTION
[0002] The present invention relates generally to the generation of
video animation and more specifically to the generation of
animation using character motion data.
BACKGROUND
[0003] Three dimensional (3D) character animation has seen
significant growth in terms of use and diffusion in the
entertainment industry in the last decade. In most 3D computer
animation systems, an animator defines a set of animation
variables, or Avars that form a simplified representation of a 3D
character's anatomy. The Avars are often organized in a
hierarchical model and, therefore, the collection of Avars for a 3D
character can be referred to as its hierarchical model. Motion of
the 3D character can be defined by changing the values of Avars
over time. The value of an Avar over time is referred to as the
Avar's motion curve, and a sequence of motion can involve defining
the motion curves for hundreds of Avars. The motion curves of all
of a 3D character's Avars during a sequence of motion are
collectively referred to as motion data.
[0004] An animator can directly animate a 3D character by manually
defining the motion curves for the 3D character's Avars using an
off-line software tool. Motion capture of a human or animal during
a desired sequence of motion can also be used to generate motion
data. Motion capture is a term used to describe a process of
recording movement and translating the movement onto a digital
model. A 3D character can be animated using the motion capture
process to record the movement of points on the human or animal
that correspond to the Avars of the 3D character during the motion.
Motion capture has traditionally been performed by applying markers
to the human or animal that can be mapped or retargeted to the
Avars of the 3D character. However, markerless techniques have
recently been developed that enable the animation of 3D characters
using mesh based techniques. Markerless motion capture using mesh
based techniques is described in U.S. Patent Publication No.
2008/0031512 entitled "Markerless Motion Capture System" to
Mundermann, Corazza and Andriacchi, the disclosure of which is
incorporated by reference herein in its entirety.
[0005] Animating a 3D character manually or using motion capture
can be time consuming and cumbersome. As discussed above, the
manual definition of a character's motion can involve a laborious
process of defining and modifying hundreds of motion curves until a
desired motion sequence is obtained. Motion capture requires the
use of complex equipment and actors. In the event that the captured
motion is not exactly as desired, the animator is faced with the
choice of repeating the motion capture process, which increases
cost, or attempting to manually edit the motion curves until the
desired motion is obtained, which is difficult. The inability of
animators to rapidly and inexpensively obtain complex motion data
for a 3D character can represent a bottleneck for the generation of
3D animations.
SUMMARY OF THE INVENTION
[0006] Systems and methods in accordance with embodiments of the
invention are described for animating 3D characters using synthetic
motion data generated by motion models which use pre-defined
character motion data to generate new motion data leveraging
analogies between the pre-defined motions and combining together
one or more of them. The motion data is generated in response to a
high level description of a desired sequence of motion provided by
an animator. In a number of embodiments, the motion data is
generated on a shared server system that utilizes the ability of
motion models to generate synthetic motion data across a continuum
to enable multiple animators to effectively reuse the same set of
previously recorded motion capture data to produce a wide variety
of desired animation sequences. In several embodiments, an animator
can upload a custom model of a 3D character to the server and the
synthetic motion data generated by the generative model is
retargeted to animate the custom 3D character. In many embodiments,
the synthetic motion data is streamed to a rendering engine located
on the animator's local computer. In this way, the processing
overhead associated with rendering animations of 3D characters
using synthetic motion generated by the shared server can be
distributed across a number of local computers.
[0007] One embodiment includes a server system connected to a
communication network and configured to communicate with a user
device that is also connected to the communication network. In
addition, the server system is configured to generate a user
interface that is accessible via the communication network, the
server system is configured to receive a high level description of
a desired sequence of motion via the user interface, the server
system is configured to generate synthetic motion data using at
least one motion model resident on the server system based on the
high level description of the desired sequence of motion received
via the user interface, the server system is configured to stream
the synthetic motion data to a rendering engine on the user device,
and the user device is configured to render a 3D character animated
using the streamed synthetic motion data.
[0008] In a further embodiment, the server system comprises an
application server and a web server that are configured to
communicate, the application server is configured to communicate
with a database of motion data, the web server is connected to the
communication network, the application server is configured to
generate the at least one motion model using the motion data and
provide the at least one motion model to the web server, the web
server is configured to generate the user interface that is
accessible via the communication network, the web server is
configured to receive the high level description of the desired
sequence of motion via the user interface, the web server is
configured to use the at least one motion model to generate
synthetic motion data based on the high level description of the
desired sequence of motion, and the web server is configured to
stream the synthetic motion data to a rendering engine on the user
device.
[0009] In another embodiment, the motion data includes actual
motion data obtained via motion capture.
[0010] In a still further embodiment, the motion data is obtained
via marker based motion capture.
[0011] In still another embodiment, the motion data includes meshes
obtained using markerless motion capture.
[0012] In a yet further embodiment, the motion data includes
manually generated motion data.
[0013] In yet another embodiment, the web server is configured to
generate synthetic motion by determining a time base based on the
high level description of the desired sequence of motion,
normalizing a number of different motions to the time base,
determining weightings for each of the motions based on the high
level description of the desired sequence of motion, and
interpolating the normalized motions using the determined
weightings.
[0014] In a further embodiment again, the synthetic motion data is
based upon a standard model for a 3D character, the server system
is configured to receive a model of a 3D character from the user
device via the communication network, the server system is
configured to retarget the synthetic motion data to the received
model of a 3D character, and the synthetic motion data streamed by
the server system is the retargeted synthetic motion data.
[0015] In another embodiment again, the motion data is based upon a
standard model for a 3D character, the server system is configured
to generate a model of a 3D character from a high level description
of a 3D character received via the user interface, the server
system is configured to retarget the synthetic motion data to the
generated model of a 3D character, and the synthetic motion data
streamed by the server system is the retargeted synthetic motion
data.
[0016] In a further additional embodiment, the server system is
configured to impose environmental constraints on the synthetic
motion data.
[0017] In another additional embodiment, the server system is
configured to modify the synthetic motion data to constrain a
certain aspect of the motion to occur during a particular period of
the time base.
[0018] In a still yet further embodiment, the server system is
configured to modify the synthetic motion data to prevent portions
of the 3D character from penetrating a floor during the motion.
[0019] In still yet another embodiment, the server system is
configured to use the motion model to generate synthetic motion
data from any description within the continuum of the scale of the
high level characteristic.
[0020] In a still further embodiment again, the high level
description of the desired sequence of motion includes at least a
motion type, a trajectory for the motion, and at least one motion
styling.
[0021] In still another further embodiment again, the motion
styling is a description of a physical characteristic or an
expressive characteristic of the motion.
[0022] In a still further additional embodiment, the motion system
is expressed as one of a number of discrete types of motion.
[0023] In still another additional embodiment, the trajectory of
the motion is specified including at least a start point and an end
point.
[0024] In a yet further embodiment again, the motion styling is
expressed using at least one value from a continuous scale that
corresponds to a high level characteristic that was used to
describe differences between repeated sequences of motion in the
motion data used to generate the motion models.
[0025] In yet another embodiment again, the server system is
configured to receive an updated high level description of the
desired sequence of motion over the communication network via the
user interface, the server system is configured to use the motion
model to generate a second set of synthetic motion data based on
the updated high level description of the desired sequence of
motion, and the server system is configured to stream the second
set of synthetic motion data to a rendering engine on the user
device.
[0026] In a yet further additional embodiment, the rendering engine
is a plug-in to a browser application on the user device that is
used to interact with the user interface generated by the server
system.
[0027] In yet another additional embodiment, the motion data is
based upon a standard model for a 3D character, the server system
is configured to receive a model of a 3D character from the user
device via the communication network, the server system is
configured to retarget the synthetic motion data generated by the
motion model based upon the high level description of the desired
sequence of motion to animate the 3D character received from the
user device, and the synthetic motion data streamed by the server
system is the retargeted synthetic motion data.
[0028] In a further additional embodiment again, the motion data is
based upon a standard model for a 3D character, the server system
is configured to generate a model of a 3D character from a high
level description of a 3D character received via the user
interface, the server system is configured to retarget the
synthetic motion data generated by the motion model based upon the
high level description of the desired sequence of motion to animate
the 3D character generated by the server system, and the synthetic
motion data streamed by the server system is the retargeted
synthetic motion data.
[0029] An embodiment of the method of the invention includes
receiving a high level description of the motion characteristics
for a desired motion sequence from a user device via a user
interface generated by a server system, generating synthetic motion
data using the server system based upon the at least one motion
model and the high level description of the motion characteristics
for a desired motion sequence, streaming the synthetic motion data
from the server system to the user device, and rendering an
animation of a 3D character using the client device based upon the
streamed synthetic motion data.
[0030] In a further embodiment of the method of the invention
generating synthetic motion data based upon the at least one motion
model and the high level description of the motion characteristics
for a desired motion sequence further includes determining a time
base based on the high level description of the motion
characteristics for a desired motion, normalizing the motion data
of a number of different motions included in the at least one
motion model to the time base, determining weightings for the
motion data of each of the motions based on the high level
description of the motion characteristics for a desired motion, and
interpolating the normalized motion data using the determined
weightings to generate synthetic motion data.
[0031] Another embodiment of the method of the invention also
includes obtaining a custom model of a 3D character, and
retargeting the synthetic motion data to the user model prior to
streaming the synthetic motion data to the user device.
[0032] In a still further embodiment of the method of the invention
the custom model of a 3D character is uploaded to the server system
from the user device.
[0033] In a yet further embodiment of the method of the invention
the custom model of a 3D character is generated by the server
system in response to a high level description of the 3D character
obtained from the user device via a user interface generated by the
server system.
[0034] Yet another embodiment of the invention also includes
imposing environmental constraints on the retargeted synthetic
motion data.
BRIEF DESCRIPTION OF THE FIGURES
[0035] FIG. 1 is a semi-schematic diagram of an animation system
configured to generate synthetic motion data in accordance with an
embodiment of the invention.
[0036] FIG. 2 is a flow chart illustrating a process for generating
synthetic motion data in accordance with an embodiment of the
invention.
[0037] FIG. 3 is a flow chart illustrating a process for generating
synthetic motion data and retargeting the data to animate a user
defined 3D character in accordance with an embodiment of the
invention.
[0038] FIG. 4 is a conceptual illustration of a user interface for
obtaining a high level description of a desired sequence of motion
in accordance with an embodiment of the invention.
DETAILED DESCRIPTION OF THE INVENTION
[0039] Turning now to the drawings, animation systems and methods
for real time interactive generation of synthetic motion data for
the animation of 3D characters are illustrated. The term synthetic
motion data describes motion data that is generated by a machine.
Synthetic motion data is distinct from manually generated motion
data, where a human animator defines the motion curve of each Avar,
and actual motion data obtained via motion capture. Animation
systems in accordance with many embodiments of the invention are
configured to obtain a high level description of a desired motion
sequence from an animator and use the high level description to
generate synthetic motion data corresponding to the desired motion
sequence. Instead of directly editing the motion data, the animator
can edit the high level description until synthetic motion data is
generated that meets the animator's needs.
[0040] In a number of embodiments, the animation system distributes
processing between a user's computer and one or more remote
servers. The user can upload a custom 3D character to a server and
provide a high level description of a desired motion sequence to
the server via a web based user interface. In several embodiments,
the server is configured to generate synthetic motion data based
upon the high level description of the desired motion sequence
provided by the user using one of a number of motion models that
the server can retarget to the uploaded 3D character. The motion
data can then be streamed to software resident on the user's
computer, which can render an animation of the 3D character using
the motion data. In this way, a single set of motion models and
retargeting software can be shared amongst different users without
the servers bearing the processing burden of rendering each 3D
animation using the generated synthetic motion data. The processing
burden of rendering the 3D animations is instead borne by the
users' computers.
Animation System
[0041] An animation system that can generate synthetic motion data
in accordance with an embodiment of the invention is shown in FIG.
1. The animation system 10 includes a storage device 12 connected
to a server system, which in the illustrated embodiment includes an
application server 14 connected to a web server 16. The web server
is connected to the Internet 18 and a number of user devices 20 can
access the web server via the Internet using a browser
application.
[0042] The storage device contains motion data that is used by the
application server to create one or more motion models. A motion
model is a model that can generate synthetic motion data
corresponding to a high level description of desired motion
characteristics. A variety of techniques can be used to construct
motion models from the motion capture training data including but
not limited to: motion blending models; motion graphs; combining a
finite number of pre-defined motions; time-warping algorithms;
genetic algorithms or any other algorithm suitable for the optimal
combination of pre-defined motions; stochastic motion blending
models, which provide uniqueness and variability to the generated
data; extended motion modes, which represent mathematically
calculated extensions of pre-defined motions; and generative
models. In a number of embodiments, the motion models are built by
blending, interpolating, extrapolating, generalizing, linearly or
non-linearly combining, and/or transforming what is called a motion
data grid. A motion data grid is an abstraction defining a set of
motions which are covering through a discrete grid the space of
all, or part of the, possible combinations of the high-level
descriptors of the motion defined by the user. The motion models
enable the creation of synthetic motion that is similar or
identical to the observed motion in the training set data, new
arbitrary motions derived from the training data, or new motions
that are generated by combining user interactive input with the
pre-defined motion data.
[0043] In one embodiment, the motion models involve generating
synthetic motion data by interpolating linearly or non-linearly
time warped motion capture data. These motion models are
constructed by capturing a number of related motions that form the
basis of the model and then manually labeling each motion according
to different high level characteristics such as, but not limited
to, speed (e.g., quick/slow), emotion (e.g., aggressive/tired),
and/or direction (e.g., high/medium/low). In many embodiments, the
high level description includes one or more values representing
different characteristics of the motion. In several embodiments,
each value is selected on a continuous scale and the continuous
scale can be used by motion models to generate motion data
elsewhere on the scale that is a blend of the observed motion data.
In addition to using labels to provide a high level description of
each motion, salient points of the motion are identified. The
salient points are used in time warping processes to normalize each
motion with respect to the other motions used in the motion
model.
[0044] The steps described above can be illustrated by way of an
example. A motion model can be generated for "punches". Different
punches could be labeled with characteristics such as the height of
the punch (e.g., face, stomach, knee), and the level of fatigue
(e.g., sharp/quick, slow/tired). The labels are typically assigned
using a numerical value on a predetermined scale. The high level
description can also include the length of the wind up to the punch
(e.g., no wind up to a long wind up). The salient features of the
punches can then be marked such as the starting point, the end of
the windup, the strike, and the recoil. When a punch has no recoil,
essentially the initial point of the motion is marked as two
salient points (i.e., the beginning and the end of the windup). The
marking of salient features can be manual or automatic using points
at which the velocity of different Avars or groups of Avars pass
through zero (i.e., change direction) to determine the salient
points. In many embodiments, processes that use different criterion
to automatically identify salient points are used.
[0045] Once the motion data has been labeled and the salient points
marked in the manner described above, a motion model in accordance
with embodiments of the invention can combine the motion data from
several motions to generate synthetic motion data in accordance
with a high level description of a desired motion. In several
embodiments, the high level description can include one or more
values, where each value specifies the magnitude of a particular
characteristic within a continuous scale. The basic process of
combining the motions involves using the high level description of
the desired motion to determine a time base for each of the
different portions of motion bounded by the salient points. Each
motion that is combined by the motion model is then normalized to
the time base for the desired motion. In its simplest form, the
normalization can be thought of as a piecewise linear time warping,
where each linear section corresponds to a portion of the motion
bounded by the salient points. In many embodiments, a nonlinear
time warping is applied to normalize each motion to provide smooth
transitions between each portion of the motion bounded by the
salient points. The high level description then provides
information concerning the weightings that are applied to each of
the normalized motions to produce the desired synthetic motion
data. In a number of embodiments, a quaternion layer interpolation
involving the combination of n different motions is used to combine
the normalized motions. In circumstances where a user has uploaded
or generated a custom 3D character model, the synthetic motion data
is retargeted to the custom 3D character model. Finally,
environmental constraints can be imposed on the synthetic motion
data such as marking were feet should plant during the motion and
imposing constraints such as preventing the 3D character's limbs
from stepping "through" the floor.
[0046] The generation of synthetic motion data using the process
described above can be illustrated by way of example. A variety of
"punching" motions can form the basis of a motion model. At one
extreme, the motion includes a punching motion with a 20 frame
windup and at the other extreme a punching motion with no wind up.
A user interface can be provided allowing the user to define the
extent of the windup desired (e.g., a slider) and the user could
indicate a desire for a punch with a windup half way between the
two extremes. The motion model would then choose a time base
including 10 frames reserved for the windup (i.e., hall the number
between 20 and 0 frames). The windup of each punching motion would
then be normalized with respect to the new time base and then the
two motions interpolated. Due to the fact that one of the motions
involves only one static frame (no windup motion), the
interpolation provides an averaging effect between the full range
of the windup and stationary motion. Therefore, the range of the
windup motion is approximately half that of the "full" windup.
[0047] Another category of motion model that can be utilized by an
animation system in accordance with embodiments of the invention is
statistical models of motion that are trained using supervised
learning. Supervised learning involves training a statistical model
using motion data for a number of sequences of motion, which can be
referred to as training sequences. The training is supervised,
because each training sequence is described using a high level
description. In this way, the statistical model builds
relationships between high-level descriptions and actual motion
data. Once a generative model has been created, synthetic motion
data can be generated by providing the generative model with a high
level description indicative of a desired sequence of motion. In a
number of embodiments, the high level description represents one or
more expressive characteristics of the motion as values on a
continuous scale and, when provided with a sufficient number of
training sequences from throughout the scale, the generative model
can produce synthetic motion data anywhere on the continuum and
that is potentially unlike any of the motion sequences observed
during training.
[0048] In several embodiments, supervised learning is used to train
a Continuous Restricted Boltzmann Machine (CRBM) that can then be
used as a generative model. Training of a CRBM using motion data is
an extension of the method described in the conference paper by
Taylor et al. entitled "modeling Human Motion Using Binary Latent
Variables" presented at the Twentieth Annual. Conference on Neural.
Information Processing Systems in Whistler, Canada (available at
http://www.cs.toronto.edu/.about.gwtaylor/), the disclosure of
which is incorporated by reference herein in its entirety. In other
embodiments, any of a variety of statistical models that can be
trained using supervised learning to form a generative model, which
can generate synthetic motion data from a high level description of
desired motion can be used.
[0049] Although any of a variety of motion models can be utilized
in accordance with embodiments of the invention to generate
synthetic motion data based upon a high level description of a
desired motion including but not limited to the motion models
described above, the performance of motion models utilized in
accordance with embodiments of the invention in generating desired
synthetic motion data is typically improved by using motion data
during training that includes multiple repetitions of the same
sequence of motion and by accurately assigning the high level
characteristics that explain the differences between each
repetition. Furthermore, loading the motion models into the memory
(i.e., RAM) of the servers can also significantly improve the
performance of the animation system. The size of the motion models
is such that a large amount of memory is required. Repeatedly
loading motion models from hard disk drives into memory as they are
required to process user requests to generate synthetic motion
data, however, would introduce significant latency into the system.
Therefore, building a single set of servers in which the motion
models can reside in the RAM of the servers and be shared by
hundreds, thousands, or more animators can be significantly more
efficient than requiring each animator have enough memory on a
local machine to store the motion models for each motion. In
addition, the amount of RAM required continuously increases as
ideally new motion models are added to the animation system
continuously.
High Level Descriptions of Motion
[0050] A high level description of motion can take any of a number
of different forms appropriate to the application. In many
embodiments, each training sequence is described using a common set
of arbitrary characteristics. The arbitrary characteristics can
relate to any aspect of the motion from a simple descriptor of the
type of motion (e.g., running or walking) to a complex expressive
quality of the motion (e.g., happiness, tiredness, urgency, or
intoxication). The term expressive quality is used herein to
describe aspects of motion that communicate information about a
character and/or the character's state of being. When a motion
model is generated using a sufficiently large number of training
sequences labeled to reflect the expressive qualities of the
training sequences, the generative model can be used to generate
synthetic motion data possessing specific expressive qualities. In
a number of embodiments, simple characteristics of motion such as a
type of motion are expressed by labeling each training sequence
with one of a number of discrete values. In several embodiments,
complex characteristics of motion such as the extent of a specific
expressive quality (i.e., the extent to which the motion conveys a
particular idea or emotion) or a physical characteristic of the
motion (e.g., walking speed, height and/or distance of a jump) are
expressed by assigning each training sequence with a score on a
continuous scale. When a high level characteristic is expressed on
a continuous scale, the motion model can be used to generate
synthetic motion data anywhere on the continuum.
Generation of Synthetic Motion Data
[0051] Once a motion model has been trained, the application server
deploys the motion model to the web server, which can use the
motion model to create synthetic motion data from a high level
description. In the illustrated embodiment, the web server creates
a web based user interface that can be accessed via a user device
configured with an appropriate browser application. The web based
user interface enables an animator to provide a high level
description of desired motion, which is typically in terms of the
arbitrary characteristics used in the training of the motion model.
The web server uses the high level description to generate
synthetic motion data in real time that can animate a 3D character.
The 3D character can be resident on the server, generated by the
server from a high level description in accordance with the process
described in U.S. patent application Ser. No. 12/625,553 to Corazza
et al. entitled "Real. Time Generation of Animation-Ready 3D
Character Models" filed Nov. 24, 2009, the disclosure of which is
incorporated by reference herein in its entirety, or uploaded to
the server from the user's computer. When a generated or uploaded
3D character model is animated, the synthetic motion data is
retargeted to the 3D character model. In many embodiments, the
synthetic motion data is streamed to the animator's browser in real
time and the browser includes a rendering engine plug-in that
renders the animation of the 3D character using the synthetic
motion data. The animator can review the rendered animation to
determine whether the synthetic motion data produces a desired
animation sequence. In the event that the animator wishes to edit
the sequence of motion, the animator can modify the high level
description originally provided to the web server via the user
interface and the application server can generate a new set of
synthetic motion data in response to the new high level
description, retarget the motion data (if necessary), and stream
the new synthetic motion data to the user's browser for rendering.
The editing process is analogous to providing modified directions
to an actor and reshooting a motion capture sequence. However, the
use of a generative model can eliminate the need for an actor and
motion capture equipment during the editing process. Use of a
generative model can also provide the animator with greater control
over the final motion data.
[0052] In many embodiments, the motion data (e.g., curves and/or
joint angles) are compressed for streaming to the user device.
Although, the system is described above with reference to a browser
plug-in, the motion data could be streamed to a variety of client
applications including a video game, a browser plug-in, or a third
party software application.
[0053] Provision of a motion model in a distributed architecture in
accordance with embodiments of the invention enables efficient use
of motion capture data. Instead of each animator separately
performing motion capture for each 3D character, a large bank of
motion capture data can be built over time and then the same motion
capture data can be used by multiple animators via motion models.
Despite the likelihood that the bank of motion capture data does
not contain the specific sequence of motion desired by the
animator, the motion model enables the generation of synthetic
motion data matching the high level description of the desired
motion provided by the animator and unlike any sequence in the bank
of motion capture data. Therefore, much of the efficiency is
obtained by providing multiple animators with access to the same
system for generating synthetic motion and by providing a system
that is capable of generating synthetic motion and not simply
retrieving motion sequences from a library. Furthermore, placing
the processing burden associated with rendering 3D animations on
each user's local computer significantly reduces the number of
application servers required to support the creation of a large
number of 3D animations using the system.
[0054] Although a specific architecture is shown in FIG. 1, a
number of animation systems in accordance with embodiments of the
invention are implemented using a single server, using additional
servers, as off-line software packages that include one or more
previously trained motion models and/or provide the ability to
download one or more motion models. In addition, the functions
performed by different aspects of the system can vary. For example,
embodiments of the invention could utilize a web server that
generates a user interface and performs the generation of synthetic
motion data using a motion model. Furthermore, the specific
architecture shown in FIG. 1 is not essential to the delivery of
motion data to a remote user. In a number of embodiments, the
animation system is implemented using any of a variety of
architectures for providing remote access to a system capable of
generating synthetic motion data using a motion model including an
architecture based around the use of cloud computing.
Generating Synthetic Motion Using a Motion Model
[0055] Processes for generating synthetic motion data for animating
a 3D character using a motion model in accordance with an
embodiment of the invention is shown in FIG. 2. The process 40
includes obtaining (42) motion data to use in the building of a
motion model. The motion data can be actual motion data obtained
via motion capture (either marker based or markerless), or can be
synthetic motion created manually by an animator using a
conventional off-line animation software application. For the
motion data to be used in the training of the motion model, each
sequence of motion data is described using a set of high level
characteristics appropriate to the application. The motion data is
then used to build (44) a motion model. As discussed above, any
model appropriate to the application that is capable of generating
synthetic motion data from a high level description of a desired
motion sequence can be used. In a number of embodiments, the motion
model uses a combination of interpolation and linear or non-linear
time warping of motion capture data to generate synthetic motion
data. In several embodiments, the motion model is a statistical
model trained using supervised learning. The motion models are then
loaded into fast storage on a server. An initial set of high level
motion characteristics is then defined (48) by an animator
(typically via a web interface) and the motion model generates (50)
synthetic motion data based upon the high level motion
characteristics. The motion data is streamed to a rendering engine
located on an animator's local computer so that an animator can
make a determination (52) concerning whether the synthetic motion
data is satisfactory. In the event that the synthetic motion data
does not animate the 3D character in the manner desired by the
animator, the high level description of the desired motion can be
modified (54) and the modified description used by the motion model
to generate new synthetic motion data. When the animator is
satisfied with the manner in which the generated synthetic motion
data animates the 3D character, the synthetic motion data can be
downloaded for storage by the animator and/or stored in a user
account.
[0056] Although a specific process for generating synthetic motion
data in accordance with an embodiment of the invention is
illustrated in FIG. 2, other embodiments that enable the editing of
a high level description of desired motion in order to produce
synthetic motion data can also be used.
Building an Animation Sequence
[0057] Animation systems in accordance with embodiments of the
invention support the animation of user defined models of 3D
characters (User Model). The animation typically involves building
a motion model using a first model, a so-called Standard Model, and
then determining how the Standard Model maps to a User Model. When
hierarchical models are used as both the Standard Model and the
User Model, the mapping includes determining the relationships
between Avars in each hierarchical model. In this way, variations
in the proportions and shape of a User Model compared to a Standard
Model used during the generation of the motion model can be
accommodated without the need to train a new motion model using the
User Model for which training sequences may not be available.
[0058] A process for generating synthetic motion data to animate a
user defined 3D character in accordance with an embodiment of the
invention is illustrated in FIG. 3. The process 60 includes
obtaining (62) a User Model. In web based systems, the User Model
can be uploaded to a web server in a standard file format or a
custom 3D model can be automatically generated in response to a
high level description provided by an animator. Once the User Model
is obtained, the User Model can be compared to the Standard Model
used in the generation of the motion model. Through careful
comparison, mappings of the motion data of the Standard Model to
the User Model can be determined. The animator can then provide
(66) an initial high level description of a motion sequence, which
the motion model can use to generate synthetic motion data. The
motion model generates motion data suitable for animation the
Standard Model. A retargeting process is applied to the synthetic
motion data to provide synthetic motion data that is suitable for
animating the User Model. In many embodiments, the animation is
displayed to the animator in real time by streaming the synthetic
motion data via the Internet to a rendering engine on the
animator's computer. The animator can determined (72) whether
additional modifications (74) to the high level description are
required to obtain the desired motion or whether to save (76) the
final motion data.
[0059] Although a specific process is illustrated in FIG. 3, other
processes in accordance with embodiments of the invention can be
used that enable the retargeting of synthetic motion data generated
by a generative model to animate a User Model.
Specifying Desired Motion
[0060] Once a model for a 3D character has been selected (either a
User Model or the Standard Model), systems in accordance with
embodiments of the invention enable the animation of the model of
the 3D character using a high level description of the desired
motion. As discussed above, a variety of characteristics of the
motion including the physical characteristics of the motion and the
expressive characteristics of the motion can be controlled via a
user interface. A user interface that can be used to obtain a high
level description of a sequence of motion from an animator in
accordance with an embodiment of the invention is shown in FIG. 4.
The conceptual illustration of the user interface 100 includes
three different panels that enable the definition of the motion in
three stages. The first panel 102 enables the user to define a
sequence of discrete types of motion. In the illustrated
embodiment, the sequence involves "running" followed by "throwing".
The specific motions can be selected using any of a variety of
common techniques for budding user interfaces. For example, a user
can select options from a pull down list or search for a specific
motion by entering a search term in a dialog box 108. In a number
of embodiments, a single generative model is capable of multiple
different types of motion by providing the type of motion to the
generative model. In several embodiments, a different generative
model is used to generate synthetic motion data for each type of
motion. In many embodiments, additional generative models are used
to generate motion data for the transitions between different types
of motion. The second panel 104 enables the user to define the
trajectory of each different specified type of motion. Where
appropriate, a number of embodiments of the invention enable the
animator to specify a landscape. The animator can then define the
starting point and the ending point of each type of motion. In a
number of embodiments, the animator can also define the transition.
In the illustrated embodiment, the user interface enables the
definition of the starting and end points of each motion using a
drag and drop interface. The third panel 106 enables the definition
of additional physical and/or expressive characteristics of the
motion, which can also be referred to as the motion styling. In the
illustrated embodiment, the user interface includes three sliders
that enable the description of the desired speed of the motion
(110), a description of the athleticism to be expressed by the
motion of the 3D character (112), and a description of the level of
fatigue to be expressed by the motion of the 3D character (114). In
other embodiments, any of a variety of high level characteristics
specified during the training of the generative model can be
controlled via the user interface. Once the animator has completed
describing the characteristics of the motion using the user
interface, the animation system uses one or more motion models to
generate synthetic motion data corresponding to the described
motions and transitions between the described motions. In the
illustrated embodiment, the animation of the 3D character using the
synthetic motion data is presented in the second panel 104 in the
user interface. The animation of the 3D character is performed
locally using a rendering engine located on the animator's computer
using synthetic motion data streamed from an application server.
The animator can review the animation, modify the sequence of
motions, the transition between the motions, the trajectory of each
motion and/or the lengths of transitions and modify the values of
the styling characteristics until the sequence is satisfactory. At
which point, the animator can obtain the synthetic motion data in a
standard format.
[0061] Although a specific user interface is illustrated in FIG. 4,
any user interface that enables the specification of the trajectory
and styling of motion can be used in accordance with embodiments of
the invention to obtain a high level description of desired motion
from an animator.
Improving the Quality of Synthetic Character Motion Data
[0062] In a number of embodiments, synthetic motion data including
motion curves and joint angles can be improved by applying
filtering processes and reducing the number of key frames (i.e.,
complete frames). In several embodiments, motion data is filtered
using a low pass filler with a frequency that is appropriate for
the desired level of smoothing of the motion curves and joint
angles. In addition, relative motion between a character's feet and
a surface such as a floor when foot contact is present can be
removed. In many embodiments, the relative motion is removed using
machine learning techniques (e.g., Support Vector Machines) to
learn the situations in which the foot is expected to be locked
during motion. The relative motion can be eliminated by applying an
inverse kinematics correction to the motion of the feet. The
editing of the motion data can be facilitated by reducing the
number of keyframes on the motion curves. Although specific
processes are outlined above for improving the quality of synthetic
motion data, many fillers and adjustments can be made in accordance
with embodiments of the invention to achieve enhancements to
generated character motion data as appropriate to a specific
character and/or application.
[0063] Although the present invention has been described in certain
specific aspects, many additional modifications and variations
would be apparent to those skilled in the art. It is therefore to
be understood that the present invention may be practiced otherwise
than specifically described, including various changes in the
implementation such as using motion models based upon meshes as
opposed to markers. Thus, embodiments of the present invention
should be considered in all respects as illustrative and not
restrictive.
* * * * *
References