U.S. patent application number 11/832726 was filed with the patent office on 2008-09-11 for method for real time interactive visualization of muscle forces and joint torques in the human body.
This patent application is currently assigned to MOTEK BV. Invention is credited to Antonie J. van den Bogert, Oshri Even Zohar.
Application Number | 20080221487 11/832726 |
Document ID | / |
Family ID | 39739017 |
Filed Date | 2008-09-11 |
United States Patent
Application |
20080221487 |
Kind Code |
A1 |
Zohar; Oshri Even ; et
al. |
September 11, 2008 |
METHOD FOR REAL TIME INTERACTIVE VISUALIZATION OF MUSCLE FORCES AND
JOINT TORQUES IN THE HUMAN BODY
Abstract
A method and system are provided for the visual display of
anatomical forces, that system having: a motion capture system; a
computer, receiving data from said motion capture system; and a
computational pipeline disposed on said computer; that
computational pipeline being configured to calculate muscle forces
and joint torques in real time and visually display those forces
and torques.
Inventors: |
Zohar; Oshri Even;
(Amsterdam, NL) ; van den Bogert; Antonie J.;
(Cleveland, OH) |
Correspondence
Address: |
Vern Maine & Associates
100 MAIN STREET, P O BOX 3445
NASHUA
NH
03061-3445
US
|
Assignee: |
MOTEK BV
Amsterdam
NL
|
Family ID: |
39739017 |
Appl. No.: |
11/832726 |
Filed: |
August 2, 2007 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
60893394 |
Mar 7, 2007 |
|
|
|
Current U.S.
Class: |
600/595 |
Current CPC
Class: |
A61B 5/224 20130101;
A61B 5/4519 20130101; A61B 5/1114 20130101; A61B 5/103 20130101;
A61B 5/4528 20130101; A61B 5/1127 20130101; A61B 5/6804 20130101;
A61B 5/1116 20130101; G09B 23/28 20130101 |
Class at
Publication: |
600/595 |
International
Class: |
A61B 5/103 20060101
A61B005/103 |
Claims
1. A system for the visual display and output of anatomical forces,
the system comprising: A motion capture system; A computer,
receiving data from said motion capture system; A computational
pipeline disposed on said computer; Said computational pipeline
being configured to calculate joint torques, muscle forces, and
joint forces in real time and visually display said forces and
torques.
2. The system of claim 1 further comprising a runtime control
unit.
3. The system of claim 1 wherein said motion capture system
comprises sensors selected from the group of sensors consisting of
optical, magnetic, inertial, and video based sensors.
4. The system of claim 1 further comprising a display device.
5. The system according to claim 1 further comprising a memory
storage device.
6. The system according to claim 1 further comprising an
instrumented treadmill.
7. The system according to claim 1 further comprising a ground
reaction force sensor.
8. The system, according to claim 1 wherein said muscle forces and
said joint torques are displayed on a representation of a human
body.
9. A method for the visual display of anatomical forces, said
method comprising: Placing at least one motion capture marker on a
user; Collecting and recording real time positional data from said
at least one marker; Deriving rotational data from a plurality of
said at least one marker; Computing acceleration and velocity from
said rotational data in real time; Utilizing said acceleration and
velocity to compute forward and inverse dynamics data; Calculating
muscle forces and joint torques in real time from said forward and
inverse dynamics data; and Displaying said muscle forces and joint
torques.
10. The method according to claim 9 wherein said muscle forces and
joint torques are displayed on a 3 dimensional muscle model.
11. The method according to claim 9 further comprising selecting a
body template approximating the build of the user.
12. The method according to claim 9 further comprising displaying
said muscles forces and joint torques with colored animation.
13. A system for diagnosis and treatment of musculoskeletal
disorders; said system comprising: A movement platform; Sensors
disposed within said platform; Motion capture sensors whereby
motion of a patient is captured; A visual display whereby the
location and intensity of joint torques, muscle forces, and joint
forces are displayed in real time on an animated human model.
14. The system according to claim 13 wherein said visual display is
provided to said patient thereby training said patient to improve
movement in an affected limb.
15. The system according to claim 13 wherein said visual display is
configured to provide a clinician with force and torque data
necessary for diagnosis.
16. The system according to claim 15 further comprising a recoding
function whereby said visual display may be replayed for study by
said clinician.
17. The system according to claim 15 further comprising a database
of said visual displays.
18. The system according to claim 17 further comprising a
comparator whereby said database contains visual displays from a
plurality of users.
19. The method according to claim 9 further comprising the use of a
Jacobian matrix to facilitate said step of computing.
20. The method according to claim 9 wherein said step of
calculating further comprises use of a recurrent neural network.
Description
RELATED APPLICATIONS
[0001] This application claims the benefit of U.S. Provisional
Applications No. 60/893,394, filed Mar. 7, 2007. This application
is herein incorporated in its entirety by reference.
FIELD OF THE INVENTION
[0002] This invention most generally relates to a system that
combines motion capture technology and a 3D computational
musculoskeletal model to create a real time display environment
where muscle forces and joint torques are illustrated. More
specifically, various embodiments of the present invention create
real time visualizations of the physical muscle forces and joint
torques in the body during movement.
BACKGROUND OF THE INVENTION
[0003] Currently there is no known system or method available for
visualizing in 3D the muscle forces exerted by the human body in
real time. Most rehabilitation clinics and medical research
institutes use specialized therapeutic programs, based on cause
related classifications of movement disorders, but there is no
known way that they can view the body force arrays in real time as
it usually takes many hours and days of calculations to derive
those parameters and the results are numerical or graphical and not
intuitive to the viewer.
[0004] Motion Capture is a term for a variety of techniques, and
the technology has existed for many years in a variety of
applications. The aim of motion capture is to create
three-dimensional (3D) animation and natural simulations in a
performance oriented manner. In the entertainment industry, motion
capture allows an operator to use computer-generated characters.
Motion capture can be used to create complex motion, using the full
range of human movements and allow also inanimate objects to move
realistically. Some motion capture systems provide real-time
feedback of the data and allow the operator to immediately
determine whether the motion works sufficiently. Motion capture can
be applied to full body motion as well as to hand animation, facial
animation and real time lip sync. Motion capture is also used in
medical, simulation, engineering and ergonomic applications, and in
feature films, advertising, TV and 3D computer games.
[0005] Kinematics is the process of calculating the position in
space of the end of a linked structure, given the angles of all the
joints. Inverse Kinematics does the reverse. Given the end point of
the structure, it calculates the angles of the joints needed to be
in to achieve that end point. This process is used in robotics, 3D
computer animation and some engineering applications.
[0006] Dynamics is the process of calculating the accelerations of
a linked structure in space, given the set of internal and external
forces acting on the structure. Inverse dynamics does the opposite.
Given the accelerations of the structure, and a set of measured
forces, it calculates the unknown internal forces needed to produce
those accelerations. The result is typically provided as a set of
joint torques and resultant joint forces.
[0007] What is needed, therefore, are techniques for creating a
single computational pipeline of all the described steps in real
time. Creating for the first time the capability to view muscle
forces as they occur.
SUMMARY OF THE INVENTION
[0008] One embodiment of the present invention provides a method
for real time display of the array of muscle forces and joint
torques in a human body using color space animation of a 3D human
body muscle model. Data stream coming from a motion capture system
is parsed through a pipeline of specially written algorithms that
derives joint orientations, accelerations and velocities and
forward and inverse dynamics resulting in real time measurements of
muscle forces and joint torques. Those are passed in real time to a
3D human muscle model making the forces and torques visible to the
user as they happen.
[0009] Another embodiment of the present invention provides runtime
interaction by a user or operator.
[0010] A further embodiment of the present invention provides a
combination of motion capture technologies, simulation technology
and custom real time data processing algorithms, using a
combination of hardware and software elements combined with the
authoring and control software to customize the visualization in
real time of forces and torques exerted by the human body.
[0011] Still another embodiment of the invention creates a new
measurement and visualization tool, bearing applications in various
industries. The invention creates the possibility of looking at
muscle force transference in the body for determining, registering
and evaluating human functional performance to a range of given
situations.
[0012] Yet another embodiment of the present invention provides a
new measurement and visualization tool, bearing applications in
various industries. The invention creates the possibility of
looking at muscle forces and joint forces transference in the body
for determining, registering and evaluating human functional
performance to a range of given situations. Other applications
include orthopedic and ergonomic studies and designs.
[0013] A yet further embodiment of the present invention provides a
process that incorporates real time 3D marker data streams coming
from a motion capture system through real-time sets of algorithms
that derive from the 3D markers cloud the joints centers of
rotation, positions and orientations, then derives accelerations
and velocities and converts those into an array of muscle forces
that are passed to the 3D human body muscle model as a data stream
used in the 3D color space visualization of the muscle forces and
joint torques.
[0014] The features and advantages described herein are not
all-inclusive and, in particular, many additional features and
advantages will be apparent to one of ordinary skill in the art in
view of the drawings, specification, and claims. Moreover, it
should be noted that the language used in the specification has
been principally selected for readability and instructional
purposes, and not to limit the scope of the inventive subject
matter.
BRIEF DESCRIPTION OF THE DRAWINGS
[0015] FIG. 1A is a computer generated image illustrating motion
capture points disposed on a user (not shown) configured in
accordance but not limited to one embodiment of the present
invention.
[0016] FIG. 1B is a computer generated image illustrating a
kinematics skeleton configured in accordance with one embodiment of
the present invention.
[0017] FIG. 1C is a computer generated image illustrating a
anatomically correct skeleton configured in accordance with one
embodiment of the present invention.
[0018] FIG. 1D is a computer generated image illustrating a three
dimensional anatomically correct muscle layer configured in
accordance with one embodiment of the present invention.
[0019] FIG. 1E is a computer generated image illustrating a three
dimensional anatomically correct muscle layer disposed on an
anatomically correct three dimensional skeleton configured in
accordance with one embodiment of the present invention.
[0020] FIG. 2 is a computer generated image illustrating pipeline
layer connections configured in accordance with one embodiment of
the present invention.
[0021] FIG. 3 is a computer generated image illustrating a V-Gait
configured in accordance with one embodiment of the present
invention.
[0022] FIG. 4 is a block diagram illustrating a motion capture
system configured in accordance with one embodiment of the present
invention.
[0023] FIG. 5 is a flow chart illustrating a method of motion
capture configured in accordance with one embodiment of the present
invention.
DETAILED DESCRIPTION
[0024] Muscle forces are typically invisible by nature and one can
normally only see the results of applied muscle forces on the
individual's surroundings. One embodiment of the present invention
makes it possible to view simulated muscle forces in the human body
in real-time, in a way that makes clear the force transference in
the human musculoskeletal system. The process of achieving this
functionality relies on fast and accurate real time motion capture
data processing into an IK (inverse kinematics) skeletal layer
containing joint positions and orientations, a further process
deriving accelerations and velocities, a further process deriving
inverse dynamics in real time, a further process deriving muscle
forces from joint torques, and a final process converting the
result streams into 3D visualizations of color and form changes in
a 3D accurate human body muscle model.
[0025] The applicant herein incorporates by reference U.S. Pat. No.
6,774,885 for all purposes.
[0026] One embodiment of the invention is a method for real time
display of the array of muscle forces and joint torques in a human
body using color space animation of a 3D human body muscle model.
Data streams coming from a motion capture system are parsed through
a pipeline of specially written algorithms that derives joint
orientations, accelerations and velocities and forward and inverse
dynamics resulting in real time measurements of muscle forces and
joint torques. Those are passed in real time to a 3D human muscle
model making the forces and torques visible to the eye as they
happen.
[0027] One embodiment of the present invention allows runtime
interaction by a user or operator. Such an embodiment of the
invention can be seen as a combination of motion capture
technologies, simulation technology and custom real time data
processing algorithms, using a combination of hardware and software
elements combined with the authoring and control software to
visualize in real time the forces and torques exerted by the human
body.
[0028] One embodiment of the invention provides a new measurement
and visualization tool, bearing applications in various industries.
One embodiment of the invention creates the possibility of looking
at muscle force transference in the body for determining,
registering and evaluating human functional performance to a range
of given situations. Although at least one embodiment of the
present invention is intended for medical applications, embodiments
of the present invention are adaptable for other market segments
including ergonomics and sports.
[0029] Various embodiments of the present invention provide tools
that are useful in numerous applications, including the sports and
fitness industries. This system allows the visualizations of muscle
forces for any given exercise in real-time. Such a system,
illustrated in FIG. 3 can be used to enhance, optimize and improve
muscle forces, by providing a realistic real time visualization of
the given forces and torques. The system allows the user 30 to see
the force transference to various muscles in the body and achieve
the desired effect. A motion capture system 32 instantly records
the user's motion and provides immediate muscle force
visualizations 34.
[0030] One embodiment of the present invention may be utilized by
the medical community by making it possible to view muscle forces
and torques in real-time. It can assist and improve the quality of
life of many patients and allow the perception of physical movement
and muscle behaviors for those not otherwise capable of such
motion. The system may be useful for victims of traumatic brain
injury, cerebral damage, and spinal damage. The study of motion
recognition supports the notion that the body remembers certain
movements and can even regenerate synoptic paths. By visualizing
the desired muscle force, the body can be re-trained to make that
movement. In the field of orthopedics and prosthetics, embodiments
of the present invention can assist patients in understanding their
present situation, where they lack muscle force and where they are
exerting too much force for compensation reasons. With orthopedics,
prosthetics, and amputees, the system can visualize and track
muscle deficiencies while training and improving movements.
[0031] Yet another embodiment of the present invention combines
muscle forces and resultant joint force into a calculation and
visualization of the forces acting within joints. This is useful as
a training tool to prevent and treat overuse injuries in the
workplace, in ergonomics and in sports.
[0032] In the context of one embodiment of the present invention it
is a first step in the data analysis pipeline illustrated in FIG.
2, taking the data stream from the motion capture system and
calculating the joint angles for every body part, each joint
calculated is drawn as a sphere in this drawing. In one embodiment
of the present invention, Inverse kinematics is used to calculate
the joint orientation from the motion capture data before deriving
the accelerations and velocities of every body part. The next step
in the pipeline is to take the calculated joint angles and to
derive values of accelerations and velocities for every joint
(representing every body part), the acceleration and velocities
values are the base for calculating through the use of inverse
dynamics, the muscle forces and joint torques which are then passed
to the 3D muscle model display as color information.
[0033] One embodiment of the present invention in relation to
medical applications can serve as an example. A development project
called "Virtual Gait Lab" is one embodiment of the system operating
in the real-time domain. Such an embodiment pertains to the
development of a virtual reality system in which the muscle forces
and joint torques of the human body can be seen and evaluated in
real time in a variety of reproducible conditions.
[0034] Among the features of such an embodiment is the ability to
enhance diagnostic and therapeutic activities in a range of medical
fields. The enhancements are defined by allowing a medical expert
team for the first time the opportunity to view and analyze muscle
forces and joint torques patterns as they happen in a controlled
real-time environment.
[0035] Such a system consists of a combination of an instrumented
treadmill that is capable of measuring ground reaction forces, a
large screen or projection system for the display of the forces,
real time motion capture system and the custom computational
pipeline translating the capture data to muscle forces and joint
torques display.
[0036] An embodiment of the present invention seeks to develop an
interactive virtual real-time muscle model, which can provide
patients with means of almost unlimited exploratory behaviors and
at the same time provide medical experts accurate measurement tools
for monitoring the complex array of forces present in the human
body.
[0037] Especially in complex balance tasks, the patterns of muscle
activation determine whether a subject falls or not. These
simulations are aimed at an understanding of normal or pathological
response patterns in certain balance tasks.
[0038] Such an embodiment offers not only a test and learning
environment for patients and doctors, but is also a valuable
research environment for motor control. Such an embodiment opens
the door to a new type of experiments in which real time muscle
force visualization can be offered.
[0039] For example the muscle force tremors as observed in
Parkinson patients are considered to be an enigma by many
clinicians and human movement scientists. In these patients some
visual cues are sufficient to trigger rather normal looking muscle
force patterns (for instance used in walking), while in the absence
of such stimuli a pattern can not even be started. In healthy
subjects, the continuous control of muscle force transference
during walking is possible by having a multi-channel sensory input
onto a vast library of learned motor patterns. Once the possibility
exists to view in real time the muscle force pattern immergence, it
will lead to fundamental improvement in the understanding and
possible treatment of the sickness. Such an embodiment will allow a
new glimpse into the complexity of the natural processes associated
with human motion.
[0040] Other examples can be found among patients with peripheral
disorders, such as partial paralysis or paresis of a limb. In these
situations, gait and balance are compromised both by a partial lack
of sensory input and a lack of muscle coordination. The usual
result of that is that in order to obtain a functional gait and
balance the patients find compensations, resulting in deviant
movement patterns in healthy parts of the body. Making use of the
real time muscle force and joint torques visualization can help to
sort out the distinction between compensation and primary
disorders.
[0041] Another example of an application for one embodiment of the
present invention is the prevention and treatment of low back pain
through teaching of proper lifting techniques. Real-time
calculation and visualization of the forces acting on the
intervertebral discs will provide immediate feedback to the patient
concerning the quality of their movement.
[0042] In many embodiments the muscle forces will be visualized,
but certain training applications may provide audio signals driven
by muscle force values from the computational pipeline. Other
training applications may use muscle force values as input for a
virtual environment, which causes changes in position of virtual
objects, or changes in position of the motion platform on which the
subject is standing.
[0043] The computational pipeline that results in real time muscle
force display is flexible and allows forward dynamics simulations
to be run at any time during runtime of the system. The flow of
movements as an input to the inverse dynamics simulation is stopped
during a sequence and the calculated joint movements are now used
as input, while the movements become output. Thus forward
simulations calculate movements and reaction forces from moments of
force produced around the joints of the subjects. These forward
simulations can be visualized as part of the virtual environment,
and will show what might happen to the patient in hypothetical
situations.
[0044] The forward and inverse dynamic calculations typically
consist of a large set of equations. Depending on the methods used
to form these equations, they are expressed in various ways, such
as Newton-Euler equations, Lagrange's equations, or Kane's
equations. These are called the equations of motion, which contain
the relation between the generalized forces applied at the body and
the generalized movements. "Generalized" in this respect means that
they are formulated along the movement possibilities (or degrees of
freedom) of the human body, rather than in terms of three
dimensional coordinates in the external world. This implies that
most of the generalized forces are actually moments of force (or
torque). Equations can be added describing the kind of interaction
with the environment, such as contacts with the floor. The
equations can be solved simultaneously in a forward simulation,
solved algebraically in an inverse simulation or rearranged and
solved to do a mixed inverse and forward simulation. In one
embodiment of the present invention these computations are all
happening in real time.
[0045] From the dynamic simulation the location of the center of
mass is calculated, which, together with the position of the feet,
can be used to drive the motion of the platform, if this is
required by the virtual environment. The human body model produces
the joint moments of force of the subject. Forward dynamics
simulation can be started to indicate where weak parts in the motor
pattern are located.
[0046] The main tasks of the real time computational pipeline are
processing the input data coming from the motion capture sensors,
mapping the collected data into the above mentioned human body
model, processing the various input and/or computed data depending
on different cases. Other tasks concern the display of real-time 3D
graphic representations of muscle forces and joint torques 28, as
well as driving the output devices such as a treadmill 38 and a
display system 34 as illustrated in FIG. 3.
[0047] The user interface for the operator is implemented as the
means to communicate with the real time 3D muscle model 26 of FIG.
1D through a custom written software program. As an example of
operation, after having decided on the type of motions to execute,
the real time 3D muscle model is projected on the screen in front
of the subject, The user stands on a platform or treadmill, which
can be controlled as part of the system or as a reaction to
movements of the subject. The user wears motion capture markers 20,
as illustrated in FIGS. 1A and 2 of which the positions are
recorded. These are fed into an algorithm that turns them into the
degrees of freedom of the human body model, which is filled with
the segment masses 22 and inertia of the subject and displayed as
color space real time animations of the 3D muscle model of FIG.
1E.
[0048] From the skeleton motion and mass properties, also the
location of the center of mass is calculated, which, together with
the position of the feet, can be used to drive the motion of the
treadmill or platform as required by the environment. The human
body model 26 produces the joint moments of force of the subject,
if necessary; this information can be offered in the projected
image to be used by the subject. Forward dynamics simulation can
also be computed to indicate where weak parts in the motor pattern
are located.
[0049] FIGS. 1A-1E illustrate an overview of one embodiment of the
present invention's computational real time pipeline wherein as
illustrated in FIG. 1A a user is equipped with a number of motion
capture sensors or markers 20 attached at various strategic
locations of the body. The data from the sensors is received by a
motion capture system 32. In a preferred embodiment, the motion
capture data set contains the X axis, Y axis, and Z axis positions
of the user for the full body, and is transmitted at >100 FPS
(Frames per second) to the computer 36. The computer 36
interactively operates with operator's interface 34 and executes
the first step in the computational pipeline converting the
positional data in real time to an inverse kinematics skeleton 22
illustrated in FIG. 1B. This data is typically applied to the
inverse kinematics skeleton 22 to drive a 3D anatomically correct
skeleton 24 in about approximately real time (FIG. 1C). Then a 3D
anatomically correct muscle layer 26 of FIG. 1D is connected to the
human skeleton 24 and the muscle forces and joint torques resulting
from the real time computational pipeline are applied to real time
animations of colors 28 of the respective muscles in the 3D muscle
model of FIG. 1E.
[0050] Referring to FIG. 2, a person is outfitted with markers 20
and a template 22 is processed for an initial or balance position.
The markers 20 are typically used to record the motion. They are
substantially instantaneously captured, and used to process a
complete template. The template 22 utilizes a template matching
algorithm to interpolate for missing or bad marker data. The
Template matching result is passed to the computational inverse
kinematics skeleton 24. Here position data of the markers is
plotted in real time to joint orientations in the computational
skeleton 24. Using Constraint based rigging; the data is in turn
driving a geometry (anatomically correct) skeleton. This skeleton
is the base for the muscle force visualization layer.
[0051] FIG. 3 illustrates an embodiment of the present invention,
wherein the patient 30 on an instrumented treadmill 38 is looking
at the 3D real time interactive muscle model 34 of himself seeing
the muscles in action as muscle force is exerted. This interactive
muscle force model 34 is calculated by a processor 36 using the
method described above using data obtained from optical motion
capture sensors 32 disposed on the patient's body 30, in
combination with sensors disposed in the instrumented treadmill 38.
In one such embodiment, weight sensors may be disposed in the
instrumented treadmill 38 while other sensors such as
accelerometers, speedometers, rotation and position sensors may
also be included.
[0052] FIG. 4 is a block diagrammatic view illustrating the
hardware and software possible interconnections of one embodiment
of the present invention. The hardware platform is based on high
end multi-core Multi-processor workstations.
[0053] In one embodiment the multi-CPU hardware platform 36 is used
as the computer means for processing, memory, and interface. The
various peripherals and communications are accomplished by using
standard high-speed connections using Ethernet, serial, and SCSI
connections to dedicated hosts. The dedicated host can be a
separate personal computer (PC) or an integrated on-board computer
that interfaces with the peripheral equipment. The optical motion
capture system of one embodiment includes six cameras, and the data
acquisition unit of the optical motion capture system translates
the camera input into the desired data set.
[0054] The data set of one embodiment is 3D position information of
the sensors 20 obtained from a person 30 in real time, and is
accessible to a dedicated host that allows for the fast exchange of
data to the CPU 36. Data is, in one embodiment, delivered in a
custom made file format. Though not limited to this type of system,
the chosen main optical capture system of one embodiment is a
realtime passive marker system 32, which is readily configurable
for many setups. This technology is capable of converting and
displaying 3D data coordinates of up to 300 optical markers at
>100 HZ, The instrumented treadmill 38 is interconnected to
dedicated host that connects to the CPU for transferring data and
control information. The treadmill 38 of one embodiment has the
capacity of measurements of real time ground reaction forces by the
use of force sensors under the treadmill belt. It's speed is
interconnected to the computational pipeline for establishing a
feedback loop between the motion capture system 32 and the
treadmill 38 so that the person is remaining at the center of the
treadmill regardless of changes in the walking/running speeds. A
projection device 34 such as a plasma screen or a video projector
and screen is used to display the real time 3D muscle model to the
user.
[0055] FIG. 5 illustrates a flow chart illustrating the operation
of a system configured according to one embodiment of the present
invention. Input from the motion capture system 1 in the form of 3D
marker coordinates is used as input for the Kinematic Solver 6. The
Kinematic solver 6 is also using resource files of a skeleton
definition and marker set templates 3. The Kinematic Solver 6 is
outputting in real-time the current skeleton pose. Real-time
low-pass filtering and differentiation processes the changes in
skeleton pose into velocities and accelerations that are used as
input to the Motion Equations 7. The Kinematic Solver output also
drives the generation of Muscle paths for all respective muscles 5,
and outputs the schematic skeleton used for the visualization 9.
The Motion Equations 7 are also using input from ground reaction
forces and other external forces coming from an array of Force
sensors 2. The Motion Equations 7 also use an input from resource
files that contain the respective body mass properties 4. The
Equations of Motion 7 Output Joint moments to the Optimization
process 8, The Optimization process 8 also uses input of muscle
lengths and moment arms coming from the respective muscle paths 5.
The Optimization process 8 Outputs Muscle forces used in the Real
Time muscle force Visualization 9.
[0056] In one embodiment of the invention, the skeleton pose (i.e.
the set of generalized coordinates) is calculated in real-time by
using the Levenberg-Marquardt nonlinear least-squares algorithm to
solve the global optimization problem. The use of the analytical
Jacobian matrix makes the computations very fast.
[0057] In one embodiment of the invention, equations of motion ave
produced via software that creates C code for the forward
kinematics equations. Those equations generate coordinates of
markers on the body from the generalized coordinates of the
skeleton. The derivatives of the forward kinematics equations,
forming a Jacobian matrix, are generated by via symbolic
differentiation. Finally, one embodiment of the present invention
translates these equations into computer code which is incorporated
into the computational pipeline which executes the calculations at
run time.
[0058] In one embodiment, the muscle forces are the solution of a
static optimization problem, with the general form: minimize the
sum of normalized muscle forces raised to the Nth power, while
requiring that all muscle forces are non-negative, and that the set
of muscle forces multiplied by their respective moment arms, are
identical to the joint torques solved by the inverse dynamics
equations. Normalized muscle force is defined as the muscle force
relative to the maximal force capacity of the muscle. Moment arm is
the distance from the muscle force vector to the instantaneous
center of rotation of a particular joint and is mathematically
calculated as the derivative of muscle length with respect to the
joint's generalized coordinate. Traditional optimization methods
are too slow for real-time applications. For N=2, which is commonly
used in muscle force estimation, a solution is obtained in real
time using the neural network algorithm for quadratic
programming.
[0059] Motion Capture is a phrase used to describe for a variety of
techniques for capturing the movement of a body or object, and the
technology has existed for many years in a variety of applications.
The aim of motion capture is to create three-dimensional (3D)
animation and natural simulations in a performance oriented manner.
In the entertainment industry, motion capture allows an operator to
use computer-generated characters. Motion capture is used to create
complex natural motion, using the full range of human movements and
allow also inanimate objects to move realistically. Some motion
capture systems provide real-time feedback of the data and allow
the operator to immediately determine whether the motion works
sufficiently. Motion capture can be applied to full body motion as
well as to hand animation, facial animation and real time lip sync.
Motion capture is also used in medical, simulation, engineering and
ergonomic applications, and in feature films, advertising, TV and
3D computer games. In the context of the present invention, motion
capture is used to output 3D XYZ marker positions.
[0060] Force sensors are used in many industries such as
Automotive, Robotics and various Engineering applications,
typically a force sensor will measure the total forces applied on
it, those can be vertical force or horizontal and shear force
components. In the context of the present invention, force sensors
are used to measure ground reaction forces from the treadmill a
person is standing, walking or running on. For example, the
treadmill of one embodiment has the capacity of measurements of
real time ground reaction forces by the use of force sensors under
the treadmill belt, It's speed is interconnected to the
computational pipeline for establishing a feedback loop between the
motion capture system and the treadmill so that the person is
remaining at the center of the treadmill regardless of changes in
the walking/running speeds.
[0061] Skeleton definition and marker set Templates 3 are resource
files used in the computational pipeline of the current invention,
people are different in size and weight and a skeleton templates is
selected from a group of skeleton templates to get the best match
for every person. Marker templates are used to define where the 4
markers are placed on the human body. Typically, such markers are
disposed at every joint of the body.
[0062] Body mass properties 4 pertains to the weight of different
body parts of different people. People vary in weight and this has
ramifications on the muscle force they exert to generate specific
motions. The mass properties are used as a resource for the correct
real time force computations.
[0063] Muscle paths 5 are utilized to compensate for differences in
build between users. Variations in length and width between
subjects have ramifications to the force computations as a longer
muscle will exert different force to generate the same motion then
a shorter muscle, also the placement of the ligaments will be
different in different people. In the context of one embodiment of
the present invention, muscle paths are used to assist the
computations of muscle forces and joint torques.
[0064] Kinematic solver 6 provides for the calculation of joint
orientation using inverse kinematics. Kinematics is the process of
calculating the position in space of the end of a linked structure,
given the angles of all the joints. Inverse Kinematics does the
reverse. Given the end point of the structure, what angles do the
joints need to be in to achieve that end point? This process is
used in robotics, 3D computer animation and some engineering
applications. In the context of one embodiment of the present
invention it is a single step in the data analysis pipeline, taking
the data stream from the motion capture system and calculating the
joint angles for every body part. In the context of one embodiment
of the present invention, Inverse kinematics is used to calculate
the joint orientation from the motion capture data, and to thereby
convert XYZ positional values to rotation angles of the joints in
degrees or radians.
[0065] Equations used in the calculation of motion and force are
known to those skilled in the physical sciences, or are readily
derived from equations well known in the field of physics. Motion
Equations 7 are sets of mathematical equations designed to combine
incoming streams of kinematics data with marker and skeleton
templates and convert those to forward and inverse dynamics data.
Those can be lagrangeian equation sets, Casey sets, or Euler-Newton
equation sets. In the context of one embodiment of the present
invention, the motion equations 7 provide the relationship between
generalized forces applied at the body and generalized movements.
"Generalized" in this respect means that they are formulated along
the movement possibilities (or degrees of freedom) of the human
body, rather than in terms of forces in the external world. This
implies that most of the generalized forces are actually moments of
force (or torque). Equations 7 can be added describing the kind of
interaction with the environment, such as contacts with the floor.
The equations 7 can be solved simultaneously in a forward
simulation, solved algebraically in an inverse simulation or
rearranged and solved to do a mixed inverse and forward simulation.
In one embodiment of the present invention these computations are
all happening in real time. In one embodiment, effective delay is
eliminated using efficient algorithms, achieving a minimal sampling
speed in real time to be greater than 30 hz, a standard familiar to
those in the television and broadcast industries. One skilled in
the art will readily appreciate that faster time would likewise be
acceptable or desirable in some applications.
[0066] An optimization process 8 uses the input of muscle lengths
and moment arms coming from the respective muscle paths to output
muscle forces and joint torques. The optimization 8 of the data
contains routines for data normalization and several real time
software filters
[0067] Real Time muscle force visualization 9 is provided by inputs
of muscle forces and joint torques and are used to drive color
animation on the respective muscles displayed as a 3D human body
model on screen. The color brightness and hue correlates with the
muscle force amplitude, gain and activation patterns. The user and
operator can see a real time animation of the muscle forces active
in the human body at any given time
[0068] Various embodiments of the present invention provide
applications adaptable for other market segments. Sports and
fitness is one such market. One embodiment of the present invention
provides a tool that is useful in numerous applications, including
the fitness industry. This system allows the visualizations of
muscle forces for any given exercise in real-time. The system can
be used to enhance and improve muscle forces, by providing a
realistic visualization of the given forces and torques. The
present system allows the user to see the force transference to
various muscles in the body and achieve a desired effect. The
motion capture system instantly records the user's motion and
provides immediate muscle force visualizations.
[0069] One embodiment of the present invention may have an enormous
impact in the medical community by making it possible to view
muscle forces and torques in real-time. It can assist and improve
the quality of life of many patients and allow the perception of
physical movement and muscle behaviors for those not otherwise
capable of such motion. The system is useful for victims of
traumatic brain injury, cerebral damage, and spinal damage. The
study of motion recognition supports the notion that the body
remembers certain movements and can even regenerate synoptic paths.
By visualizing the desired muscle force, the body can be re-trained
to make that movement. In the field of orthopedics and prosthetics,
the present invention can assist patients in understanding their
present situation, where they lack muscle force and where they are
exerting too much force for compensation reasons. With orthopedics,
prosthetics, and amputees, the system can visualize and track
muscle deficiencies while training and improving movements. One
embodiment of the present invention in relation to medical
applications can serve as an example. One development project
called "Virtual Gait Lab" is one embodiment of the system operating
in the real-time domain. This project pertains to the development
of a virtual reality system in which the muscle forces and joint
torques of the human body can be seen and evaluated in real time in
a variety of reproducible conditions. One of the major objectives
of such a project is to enhance diagnostic and therapeutic
activities in a range of medical fields. The enhancements are
defined by allowing a medical expert team for the first time the
opportunity to view and analyze muscle forces and joint torques
patterns as they happen in a controlled real-time environment.
[0070] In one embodiment such as that illustrated in FIG. 3, the
system consists of a combination of an instrumented treadmill 38
that is capable of measuring ground reaction forces, a large screen
or projection system for the display of the forces 34, real time
motion capture system 32 and the custom computational pipeline 36
translating the capture data to muscle forces and joint torques
display.
[0071] Various embodiments of the present invention seek to develop
an interactive virtual real-time muscle model, which can provide
patients with means of almost unlimited exploratory behaviors and
at the same time provide medical experts accurate measurement tools
for monitoring the complex array of forces present in the human
body. Especially in complex balance tasks, the patterns of muscle
activation determine whether a subject falls or not. These
simulations are aimed at an understanding of normal or pathological
response patterns in certain balance tasks. Such an embodiment
offers not only a test-and learning environment for patients and
doctors, but is also a valuable research environment for motor
control. Such an embodiment opens the door to a new type of
experiments in which real time muscle force visualization can be
offered. For example the muscle force tremors as observed in
Parkinson patients are considered to be an enigma by many
clinicians and human movement scientists. In these patients some
visual cues are sufficient to trigger rather normal looking muscle
force patterns (for instance used in walking), while in the absence
of such stimuli a pattern can not even be started. In healthy
subjects, the continuous control of muscle force transference
during walking is possible by having a multi-channel sensory input
onto a vast library of learned motor patterns. Once the possibility
exists to view in real time the muscle force pattern immergence, it
will lead to fundamental improvement in the understanding and
possible treatment of the sickness. Such an embodiment will allow a
new glimpse into the complexity of the natural processes associated
with human motion. Other examples can be found among patients with
peripheral disorders, such as partial paralysis or paresis of a
limb. In these situations, gait and balance are compromised both by
a partial lack of sensory input and a lack of muscle coordination.
The usual result of that is that in order to obtain a functional
gait and balance the patients find compensations, resulting in
deviant movement patterns in healthy parts of the body. Making use
of the real time muscle force and joint torques visualization can
help to sort out the distinction between compensation and primary
disorders.
[0072] One embodiment of the invention is a new principle in real
time visualization, where muscle force is seen and evaluated in a
totally new way. This principle establishes a mechanism to achieve
a visualization state whereby the persons involved can see
immediately which muscles they are using and to what extent.
[0073] One embodiment of the present invention is a muscle force
processing system, comprising a processing means, a motion capture
system connected to the processing means. The motion capture data
is taken from a plurality of motion sensors and is processed in
real-time. There is a computational pipeline connected to the
processing means, wherein resulting data is also processed in
real-time, and wherein resulting data is visualized in real time
through color space changes in a 3D muscle model showing the muscle
forces and joint torques in real time. There is also a means of
interfacing to the muscle model with a runtime control input. A
further embodiment is an instrumented treadmill capable of
measurements of ground reaction forces, wherein the measurements of
said ground reaction forces are integrated in the computational
pipeline resulting in real time view of muscle forces and joint
torques. A further embodiment is a 3D interactive muscle model
further comprising an inverse kinematics skeleton layer, a 3D
geometry anatomically correct skeleton layer and an anatomically
correct muscle model layer. An additional embodiment is a real time
computational pipeline, further comprising a memory means for
recording the motion capture data and processing the data in real
time through the said layers of the processing real time pipeline.
Another embodiment is a method and system for real time
visualization registration, evaluation, and correction of muscle
forces and joint torques in the human body, wherein the full
process is happening in real time.
[0074] The foregoing description of the embodiments of the
invention has been presented for the purposes of illustration and
description. It is not intended to be exhaustive or to limit the
invention to the precise form disclosed. Many modifications and
variations are possible in light of this disclosure. It is intended
that the scope of the invention be limited not by this detailed
description, but rather by the claims appended hereto.
* * * * *